Improve the WorldDiT model card layout
Browse filesReorganizes the WorldDiT card into a compact release dashboard. The rollout showcase now uses a two by two layout, the results and released configuration share one snapshot, and advanced reference material is grouped into expandable sections. Model files, checkpoints, commands, results, and metadata remain unchanged.
README.md
CHANGED
|
@@ -38,91 +38,81 @@ widget:
|
|
| 38 |
<img src="https://pub-2c09ae97630f4932a23e622b450076e0.r2.dev/paris2/model-card/v1/bagel_labs_logo.png" alt="Bagel Labs">
|
| 39 |
</p>
|
| 40 |
|
| 41 |
-
|
| 42 |
|
| 43 |
-
|
| 44 |
|
| 45 |
<p align="center">
|
| 46 |
-
<a href="https://huggingface.co/bageldotcom/worlddit" target="_blank">
|
| 47 |
-
<img src="https://img.shields.io/badge/馃_DOWNLOAD_WORLDDIT_WEIGHTS-FFD21E?style=for-the-badge&logoColor=000000" alt="Download WorldDiT Weights">
|
| 48 |
-
</a>
|
| 49 |
-
<a href="https://github.com/Lifelong-Robot-Learning/LIBERO" target="_blank">
|
| 50 |
-
<img src="https://img.shields.io/badge/馃_LIBERO_BENCHMARK-FF6B6B?style=for-the-badge&logoColor=white" alt="LIBERO Benchmark">
|
| 51 |
-
</a>
|
| 52 |
-
</p>
|
| 53 |
-
|
| 54 |
WorldDiT learns continuous robot action chunks and a future visual target
|
| 55 |
through one shared diffusion transformer. Deployment keeps only the action
|
| 56 |
path.
|
|
|
|
| 57 |
|
|
|
|
| 58 |
This release includes four LIBERO checkpoints, a self contained inference
|
| 59 |
runtime, and an evaluator for reproducing the reported suite results.
|
|
|
|
| 60 |
|
| 61 |
## See WorldDiT act
|
| 62 |
|
| 63 |
The four clips below show successful rollouts from the released checkpoints.
|
| 64 |
Each clip covers a different LIBERO suite and camera view.
|
| 65 |
|
| 66 |
-
<
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 67 |
|
| 68 |
-
|
|
| 69 |
|---|---|---|
|
| 70 |
-
|
|
| 71 |
-
| LIBERO Object | Agent view | [Open MP4](https://pub-2c09ae97630f4932a23e622b450076e0.r2.dev/worlddit/model-card/v1/worlddit_libero_object_agentview_task08_episode01.mp4) |
|
| 72 |
-
| LIBERO Goal | Side view | [Open MP4](https://pub-2c09ae97630f4932a23e622b450076e0.r2.dev/worlddit/model-card/v1/worlddit_libero_goal_sideview_task10_episode01.mp4) |
|
| 73 |
-
| LIBERO Long | Front view | [Open MP4](https://pub-2c09ae97630f4932a23e622b450076e0.r2.dev/worlddit/model-card/v1/worlddit_libero_10_frontview_task06_episode01.mp4) |
|
| 74 |
|
| 75 |
-
|
| 76 |
-
|
| 77 |
-
| Release component | Included artifact |
|
| 78 |
-
|---|---|
|
| 79 |
-
| LIBERO Spatial policy | SafeTensors checkpoint |
|
| 80 |
-
| LIBERO Object policy | SafeTensors checkpoint |
|
| 81 |
-
| LIBERO Goal policy | SafeTensors checkpoint |
|
| 82 |
-
| LIBERO Long policy | SafeTensors checkpoint |
|
| 83 |
-
| Model runtime | `inference.py` |
|
| 84 |
-
| Evaluation runtime | `eval.py` |
|
| 85 |
-
| Frozen encoders | CLIP ViT B 32 and MAE ViT B |
|
| 86 |
-
| Configuration | `config.json` |
|
| 87 |
-
| Environment | Pinned Python requirements |
|
| 88 |
-
|
| 89 |
-
The repository is self contained for WorldDiT inference. LIBERO still provides
|
| 90 |
-
the benchmark environments, assets, task definitions, and initial states.
|
| 91 |
-
|
| 92 |
-
## Reported LIBERO results
|
| 93 |
-
|
| 94 |
-
Across the four released suite checkpoints, WorldDiT records 1,898 successful
|
| 95 |
-
episodes out of 2,000 under the selection aware evaluation protocol.
|
| 96 |
-
|
| 97 |
-
| Suite | Successful episodes | Success rate |
|
| 98 |
-
|---|---:|---:|
|
| 99 |
-
| LIBERO Spatial | 490 of 500 | 98.0 percent |
|
| 100 |
-
| LIBERO Object | 485 of 500 | 97.0 percent |
|
| 101 |
-
| LIBERO Goal | 464 of 500 | 92.8 percent |
|
| 102 |
-
| LIBERO Long | 459 of 500 | 91.8 percent |
|
| 103 |
-
| Selection aware mean | 1,898 of 2,000 | 94.9 percent |
|
| 104 |
|
| 105 |
The released runtime and checkpoints were revalidated from a clean installation
|
| 106 |
-
on eight RTX Pro 6000 Blackwell GPUs.
|
| 107 |
-
|
| 108 |
-
|
| 109 |
-
staged checkpoint selection before the final five hundred episode score was
|
| 110 |
-
assembled.
|
| 111 |
-
|
| 112 |
-
## Model at a glance
|
| 113 |
-
|
| 114 |
-
| Property | Released configuration |
|
| 115 |
-
|---|---|
|
| 116 |
-
| Total parameters | 399.084 million |
|
| 117 |
-
| Trainable parameters | 135.107 million |
|
| 118 |
-
| Observation context | Three frames |
|
| 119 |
-
| Predicted action horizon | Seven actions |
|
| 120 |
-
| Executed before replanning | Three actions |
|
| 121 |
-
| Action dimension | Seven |
|
| 122 |
-
| Visual encoder | MAE ViT B |
|
| 123 |
-
| Language encoder | OpenAI CLIP ViT B 32 |
|
| 124 |
-
| Checkpoint format | SafeTensors |
|
| 125 |
-
| Evaluation environment | Headless LIBERO with EGL |
|
| 126 |
|
| 127 |
## Run a smoke test
|
| 128 |
|
|
@@ -179,7 +169,10 @@ and replans.
|
|
| 179 |
| Future visual supervision is present | No future visual output is requested |
|
| 180 |
| The complete training objective is active | Three actions execute before replanning |
|
| 181 |
|
| 182 |
-
##
|
|
|
|
|
|
|
|
|
|
| 183 |
|
| 184 |
### One GPU
|
| 185 |
|
|
@@ -213,7 +206,10 @@ libero_goal
|
|
| 213 |
libero_10
|
| 214 |
```
|
| 215 |
|
| 216 |
-
|
|
|
|
|
|
|
|
|
|
| 217 |
|
| 218 |
```text
|
| 219 |
.
|
|
@@ -234,7 +230,10 @@ libero_10
|
|
| 234 |
`dependencies/` contains the frozen visual and language encoder weights needed
|
| 235 |
by the released policy. No additional model downloads are required.
|
| 236 |
|
| 237 |
-
|
|
|
|
|
|
|
|
|
|
| 238 |
|
| 239 |
```python
|
| 240 |
from inference import load_model
|
|
@@ -253,7 +252,10 @@ actions = model(primary_images, wrist_images, robot_state, text_tokens)
|
|
| 253 |
|
| 254 |
Evaluation uses the final temporal slot of the predicted action tensor.
|
| 255 |
|
| 256 |
-
|
|
|
|
|
|
|
|
|
|
| 257 |
|
| 258 |
| Component | Specification |
|
| 259 |
|---|---|
|
|
@@ -267,21 +269,14 @@ Evaluation uses the final temporal slot of the predicted action tensor.
|
|
| 267 |
| Evaluation | Headless LIBERO with EGL |
|
| 268 |
| Checkpoint format | SafeTensors |
|
| 269 |
|
| 270 |
-
|
| 271 |
-
|
| 272 |
-
WorldDiT is intended for research on language conditioned robot manipulation in
|
| 273 |
-
the LIBERO simulator. The released checkpoints support reproduction,
|
| 274 |
-
evaluation, and architecture research across the four released suites.
|
| 275 |
|
| 276 |
-
##
|
| 277 |
|
| 278 |
-
|
| 279 |
-
|
| 280 |
-
across embodiments.
|
| 281 |
-
|
| 282 |
-
The present release does not isolate the causal contribution of the future
|
| 283 |
-
visual target. Total parameter count also does not measure training cost,
|
| 284 |
-
deployment latency, or runtime efficiency.
|
| 285 |
|
| 286 |
## Authors and contact
|
| 287 |
|
|
|
|
| 38 |
<img src="https://pub-2c09ae97630f4932a23e622b450076e0.r2.dev/paris2/model-card/v1/bagel_labs_logo.png" alt="Bagel Labs">
|
| 39 |
</p>
|
| 40 |
|
| 41 |
+
<h1 align="center">WorldDiT</h1>
|
| 42 |
|
| 43 |
+
<h2 align="center">One diffusion backbone learns what to do and what comes next.</h2>
|
| 44 |
|
| 45 |
<p align="center">
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 46 |
WorldDiT learns continuous robot action chunks and a future visual target
|
| 47 |
through one shared diffusion transformer. Deployment keeps only the action
|
| 48 |
path.
|
| 49 |
+
</p>
|
| 50 |
|
| 51 |
+
<p align="center">
|
| 52 |
This release includes four LIBERO checkpoints, a self contained inference
|
| 53 |
runtime, and an evaluator for reproducing the reported suite results.
|
| 54 |
+
</p>
|
| 55 |
|
| 56 |
## See WorldDiT act
|
| 57 |
|
| 58 |
The four clips below show successful rollouts from the released checkpoints.
|
| 59 |
Each clip covers a different LIBERO suite and camera view.
|
| 60 |
|
| 61 |
+
<table>
|
| 62 |
+
<tr>
|
| 63 |
+
<td width="50%" valign="top">
|
| 64 |
+
<video width="100%" controls muted loop playsinline preload="metadata"
|
| 65 |
+
src="https://pub-2c09ae97630f4932a23e622b450076e0.r2.dev/worlddit/model-card/v1/worlddit_libero_spatial_frontview_task05_episode01.mp4"></video>
|
| 66 |
+
<strong>LIBERO Spatial</strong><br>
|
| 67 |
+
Task 5, front view.
|
| 68 |
+
</td>
|
| 69 |
+
<td width="50%" valign="top">
|
| 70 |
+
<video width="100%" controls muted loop playsinline preload="metadata"
|
| 71 |
+
src="https://pub-2c09ae97630f4932a23e622b450076e0.r2.dev/worlddit/model-card/v1/worlddit_libero_object_agentview_task08_episode01.mp4"></video>
|
| 72 |
+
<strong>LIBERO Object</strong><br>
|
| 73 |
+
Task 8, agent view.
|
| 74 |
+
</td>
|
| 75 |
+
</tr>
|
| 76 |
+
<tr>
|
| 77 |
+
<td width="50%" valign="top">
|
| 78 |
+
<video width="100%" controls muted loop playsinline preload="metadata"
|
| 79 |
+
src="https://pub-2c09ae97630f4932a23e622b450076e0.r2.dev/worlddit/model-card/v1/worlddit_libero_goal_sideview_task10_episode01.mp4"></video>
|
| 80 |
+
<strong>LIBERO Goal</strong><br>
|
| 81 |
+
Task 10, side view.
|
| 82 |
+
</td>
|
| 83 |
+
<td width="50%" valign="top">
|
| 84 |
+
<video width="100%" controls muted loop playsinline preload="metadata"
|
| 85 |
+
src="https://pub-2c09ae97630f4932a23e622b450076e0.r2.dev/worlddit/model-card/v1/worlddit_libero_10_frontview_task06_episode01.mp4"></video>
|
| 86 |
+
<strong>LIBERO Long</strong><br>
|
| 87 |
+
Task 6, front view.
|
| 88 |
+
</td>
|
| 89 |
+
</tr>
|
| 90 |
+
</table>
|
| 91 |
+
|
| 92 |
+
[Spatial MP4](https://pub-2c09ae97630f4932a23e622b450076e0.r2.dev/worlddit/model-card/v1/worlddit_libero_spatial_frontview_task05_episode01.mp4)
|
| 93 |
+
路 [Object MP4](https://pub-2c09ae97630f4932a23e622b450076e0.r2.dev/worlddit/model-card/v1/worlddit_libero_object_agentview_task08_episode01.mp4)
|
| 94 |
+
路 [Goal MP4](https://pub-2c09ae97630f4932a23e622b450076e0.r2.dev/worlddit/model-card/v1/worlddit_libero_goal_sideview_task10_episode01.mp4)
|
| 95 |
+
路 [Long MP4](https://pub-2c09ae97630f4932a23e622b450076e0.r2.dev/worlddit/model-card/v1/worlddit_libero_10_frontview_task06_episode01.mp4)
|
| 96 |
+
|
| 97 |
+
## Release snapshot
|
| 98 |
+
|
| 99 |
+
| Reported LIBERO result | Released model |
|
| 100 |
+
|---|---|
|
| 101 |
+
| **94.9 percent** selection aware mean<br>**1,898 of 2,000** successful episodes | **399.084 million** total parameters<br>**135.107 million** trainable parameters |
|
| 102 |
+
| **98.0 percent** Spatial<br>**97.0 percent** Object | **Three** observation frames<br>**Seven** predicted actions |
|
| 103 |
+
| **92.8 percent** Goal<br>**91.8 percent** Long | **Three** actions executed before replanning<br>**Seven** action dimensions |
|
| 104 |
|
| 105 |
+
| Checkpoints | Runtime | Encoders and environment |
|
| 106 |
|---|---|---|
|
| 107 |
+
| Spatial<br>Object<br>Goal<br>Long | `inference.py`<br>`eval.py`<br>`config.json` | MAE ViT B<br>OpenAI CLIP ViT B 32<br>SafeTensors and pinned requirements |
|
|
|
|
|
|
|
|
|
|
| 108 |
|
| 109 |
+
The repository is self contained for WorldDiT inference. LIBERO provides the
|
| 110 |
+
benchmark environments, assets, task definitions, and initial states.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 111 |
|
| 112 |
The released runtime and checkpoints were revalidated from a clean installation
|
| 113 |
+
on eight RTX Pro 6000 Blackwell GPUs. The result is selection aware because
|
| 114 |
+
three hundred episodes per suite informed staged checkpoint selection before
|
| 115 |
+
the final five hundred episode score was assembled.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 116 |
|
| 117 |
## Run a smoke test
|
| 118 |
|
|
|
|
| 169 |
| Future visual supervision is present | No future visual output is requested |
|
| 170 |
| The complete training objective is active | Three actions execute before replanning |
|
| 171 |
|
| 172 |
+
## Reference
|
| 173 |
+
|
| 174 |
+
<details>
|
| 175 |
+
<summary><strong>Full evaluation commands</strong></summary>
|
| 176 |
|
| 177 |
### One GPU
|
| 178 |
|
|
|
|
| 206 |
libero_10
|
| 207 |
```
|
| 208 |
|
| 209 |
+
</details>
|
| 210 |
+
|
| 211 |
+
<details>
|
| 212 |
+
<summary><strong>Repository contents</strong></summary>
|
| 213 |
|
| 214 |
```text
|
| 215 |
.
|
|
|
|
| 230 |
`dependencies/` contains the frozen visual and language encoder weights needed
|
| 231 |
by the released policy. No additional model downloads are required.
|
| 232 |
|
| 233 |
+
</details>
|
| 234 |
+
|
| 235 |
+
<details>
|
| 236 |
+
<summary><strong>Inference API and tensor shapes</strong></summary>
|
| 237 |
|
| 238 |
```python
|
| 239 |
from inference import load_model
|
|
|
|
| 252 |
|
| 253 |
Evaluation uses the final temporal slot of the predicted action tensor.
|
| 254 |
|
| 255 |
+
</details>
|
| 256 |
+
|
| 257 |
+
<details>
|
| 258 |
+
<summary><strong>Architecture details</strong></summary>
|
| 259 |
|
| 260 |
| Component | Specification |
|
| 261 |
|---|---|
|
|
|
|
| 269 |
| Evaluation | Headless LIBERO with EGL |
|
| 270 |
| Checkpoint format | SafeTensors |
|
| 271 |
|
| 272 |
+
</details>
|
|
|
|
|
|
|
|
|
|
|
|
|
| 273 |
|
| 274 |
+
## Use and scope
|
| 275 |
|
| 276 |
+
| Intended use | Scope of the release |
|
| 277 |
+
|---|---|
|
| 278 |
+
| Research on language conditioned robot manipulation in the LIBERO simulator. The released checkpoints support reproduction, evaluation, and architecture research across the four released suites. | The results describe LIBERO simulation under the released evaluation protocol. They do not establish real robot reliability, safety, or transfer across embodiments. |
|
| 279 |
+
| The released checkpoints cover all four LIBERO suites. | The release does not isolate the causal contribution of the future visual target. Total parameter count does not measure training cost, deployment latency, or runtime efficiency. |
|
|
|
|
|
|
|
|
|
|
| 280 |
|
| 281 |
## Authors and contact
|
| 282 |
|