Rename model card to Streaming-WAM
Browse files
README.md
CHANGED
|
@@ -11,25 +11,25 @@ tags:
|
|
| 11 |
---
|
| 12 |
|
| 13 |
<div align="center">
|
| 14 |
-
<h1>
|
| 15 |
<h3>Streaming World-Action Models for Robotic Manipulation</h3>
|
| 16 |
|
| 17 |
-
<a href="https://github.com/SJTU-DENG-Lab/
|
| 18 |
-
<a href="https://github.com/SJTU-DENG-Lab/
|
| 19 |
</div>
|
| 20 |
|
| 21 |
-
|
| 22 |
|
| 23 |
-
|
| 24 |
|
| 25 |
-
This repository provides the released LIBERO checkpoints. The corresponding inference and evaluation code is available in the [
|
| 26 |
|
| 27 |
## Released checkpoints
|
| 28 |
|
| 29 |
| Directory | Model | Description |
|
| 30 |
| --- | --- | --- |
|
| 31 |
| `joint-cd/` | FastWAM-Joint-CD | Fast one-step joint world-and-action prediction for LIBERO. |
|
| 32 |
-
| `ac-stream/` |
|
| 33 |
|
| 34 |
Each directory contains:
|
| 35 |
|
|
@@ -48,15 +48,15 @@ We evaluate all methods on LIBERO-10, LIBERO-Spatial, LIBERO-Goal, and LIBERO-Ob
|
|
| 48 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
| 49 |
| FastWAM | 96.20 | 96.20 | 94.20 | 96.20 | 95.70 | 493.0 | 16.31 / 8.25 |
|
| 50 |
| FastWAM-Joint-CD | 97.20 | 99.60 | 98.60 | 100.00 | 98.85 | 114.2 | 6.89 / 3.74 |
|
| 51 |
-
| **
|
| 52 |
|
| 53 |
## Installation
|
| 54 |
|
| 55 |
Clone the code repository and install the environment:
|
| 56 |
|
| 57 |
```bash
|
| 58 |
-
git clone https://github.com/SJTU-DENG-Lab/
|
| 59 |
-
cd
|
| 60 |
|
| 61 |
python -m pip install -U uv
|
| 62 |
uv sync
|
|
@@ -79,14 +79,14 @@ uv run huggingface-cli download Wan-AI/Wan2.2-TI2V-5B \
|
|
| 79 |
Download both released checkpoints:
|
| 80 |
|
| 81 |
```bash
|
| 82 |
-
hf download SJTU-DENG-Lab/
|
| 83 |
-
--local-dir checkpoints/
|
| 84 |
```
|
| 85 |
|
| 86 |
The resulting layout is:
|
| 87 |
|
| 88 |
```text
|
| 89 |
-
checkpoints/
|
| 90 |
βββ joint-cd/
|
| 91 |
β βββ model.pt
|
| 92 |
β βββ dataset_stats.json
|
|
@@ -95,7 +95,7 @@ checkpoints/streamwam/
|
|
| 95 |
βββ dataset_stats.json
|
| 96 |
```
|
| 97 |
|
| 98 |
-
## Run
|
| 99 |
|
| 100 |
Evaluate all 40 LIBERO tasks once on four GPUs:
|
| 101 |
|
|
@@ -104,9 +104,9 @@ PYTHON_BIN=.venv/bin/python \
|
|
| 104 |
GPU_IDS=0,1,2,3 \
|
| 105 |
BACKBONE_PATH="$PWD/checkpoints/Wan2.2-TI2V-5B" \
|
| 106 |
LIBERO_HOME_PATH="$PWD/third_party/LIBERO" \
|
| 107 |
-
CHECKPOINT_PATH="$PWD/checkpoints/
|
| 108 |
-
STATS_PATH="$PWD/checkpoints/
|
| 109 |
-
bash examples/libero/scripts/
|
| 110 |
--ac-stream-accelerated
|
| 111 |
```
|
| 112 |
|
|
@@ -119,11 +119,11 @@ python examples/libero/multigpu_rollout.py \
|
|
| 119 |
--gpus 0,1,2,3 \
|
| 120 |
--suites libero_spatial,libero_object,libero_goal,libero_10 \
|
| 121 |
--num-trials 1 \
|
| 122 |
-
--config examples/libero/configs/recipes/
|
| 123 |
--checkpoint-format fastwam \
|
| 124 |
-
--checkpoint checkpoints/
|
| 125 |
--backbone-path checkpoints/Wan2.2-TI2V-5B \
|
| 126 |
-
--stats-path checkpoints/
|
| 127 |
--libero-home third_party/LIBERO \
|
| 128 |
--num-steps-wait 30 \
|
| 129 |
--replan-steps 16 \
|
|
@@ -134,13 +134,12 @@ python examples/libero/multigpu_rollout.py \
|
|
| 134 |
--save-video
|
| 135 |
```
|
| 136 |
|
| 137 |
-
For more evaluation options, see the [LIBERO guide](https://github.com/SJTU-DENG-Lab/
|
| 138 |
|
| 139 |
## License
|
| 140 |
|
| 141 |
-
Released under the [Apache License 2.0](https://github.com/SJTU-DENG-Lab/
|
| 142 |
|
| 143 |
## Acknowledgements
|
| 144 |
|
| 145 |
-
|
| 146 |
-
|
|
|
|
| 11 |
---
|
| 12 |
|
| 13 |
<div align="center">
|
| 14 |
+
<h1>Streaming-WAM</h1>
|
| 15 |
<h3>Streaming World-Action Models for Robotic Manipulation</h3>
|
| 16 |
|
| 17 |
+
<a href="https://github.com/SJTU-DENG-Lab/Streaming-WAM"><img src="https://img.shields.io/badge/GitHub-Code-111827?logo=github" alt="GitHub Code"></a>
|
| 18 |
+
<a href="https://github.com/SJTU-DENG-Lab/Streaming-WAM/blob/main/LICENSE"><img src="https://img.shields.io/badge/License-Apache--2.0-6B5BFF" alt="Apache 2.0 License"></a>
|
| 19 |
</div>
|
| 20 |
|
| 21 |
+
Streaming-WAM is a research framework for streaming World-Action Models. It provides a unified testbed for studying and comparing efficient robot-control strategies.
|
| 22 |
|
| 23 |
+
Streaming-WAM uses the actions currently being executed by the robot to guide its next prediction. This allows action execution and model inference to proceed together, reducing the time required to complete a robot task while maintaining strong control performance.
|
| 24 |
|
| 25 |
+
This repository provides the released LIBERO checkpoints. The corresponding inference and evaluation code is available in the [Streaming-WAM GitHub repository](https://github.com/SJTU-DENG-Lab/Streaming-WAM).
|
| 26 |
|
| 27 |
## Released checkpoints
|
| 28 |
|
| 29 |
| Directory | Model | Description |
|
| 30 |
| --- | --- | --- |
|
| 31 |
| `joint-cd/` | FastWAM-Joint-CD | Fast one-step joint world-and-action prediction for LIBERO. |
|
| 32 |
+
| `ac-stream/` | Streaming-WAM | Recommended Streaming-WAM checkpoint for efficient LIBERO evaluation. |
|
| 33 |
|
| 34 |
Each directory contains:
|
| 35 |
|
|
|
|
| 48 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
| 49 |
| FastWAM | 96.20 | 96.20 | 94.20 | 96.20 | 95.70 | 493.0 | 16.31 / 8.25 |
|
| 50 |
| FastWAM-Joint-CD | 97.20 | 99.60 | 98.60 | 100.00 | 98.85 | 114.2 | 6.89 / 3.74 |
|
| 51 |
+
| **Streaming-WAM** | **96.60** | **98.80** | **97.40** | **100.00** | **98.20** | **41.0** | **5.36 / 3.15** |
|
| 52 |
|
| 53 |
## Installation
|
| 54 |
|
| 55 |
Clone the code repository and install the environment:
|
| 56 |
|
| 57 |
```bash
|
| 58 |
+
git clone https://github.com/SJTU-DENG-Lab/Streaming-WAM.git
|
| 59 |
+
cd Streaming-WAM
|
| 60 |
|
| 61 |
python -m pip install -U uv
|
| 62 |
uv sync
|
|
|
|
| 79 |
Download both released checkpoints:
|
| 80 |
|
| 81 |
```bash
|
| 82 |
+
hf download SJTU-DENG-Lab/Streaming-WAM \
|
| 83 |
+
--local-dir checkpoints/streamingwam
|
| 84 |
```
|
| 85 |
|
| 86 |
The resulting layout is:
|
| 87 |
|
| 88 |
```text
|
| 89 |
+
checkpoints/streamingwam/
|
| 90 |
βββ joint-cd/
|
| 91 |
β βββ model.pt
|
| 92 |
β βββ dataset_stats.json
|
|
|
|
| 95 |
βββ dataset_stats.json
|
| 96 |
```
|
| 97 |
|
| 98 |
+
## Run Streaming-WAM
|
| 99 |
|
| 100 |
Evaluate all 40 LIBERO tasks once on four GPUs:
|
| 101 |
|
|
|
|
| 104 |
GPU_IDS=0,1,2,3 \
|
| 105 |
BACKBONE_PATH="$PWD/checkpoints/Wan2.2-TI2V-5B" \
|
| 106 |
LIBERO_HOME_PATH="$PWD/third_party/LIBERO" \
|
| 107 |
+
CHECKPOINT_PATH="$PWD/checkpoints/streamingwam/ac-stream/model.pt" \
|
| 108 |
+
STATS_PATH="$PWD/checkpoints/streamingwam/ac-stream/dataset_stats.json" \
|
| 109 |
+
bash examples/libero/scripts/launch_streamingwam_libero_ac_stream_4gpu.sh \
|
| 110 |
--ac-stream-accelerated
|
| 111 |
```
|
| 112 |
|
|
|
|
| 119 |
--gpus 0,1,2,3 \
|
| 120 |
--suites libero_spatial,libero_object,libero_goal,libero_10 \
|
| 121 |
--num-trials 1 \
|
| 122 |
+
--config examples/libero/configs/recipes/streamingwam_libero_joint_cd_wan22_5b.yaml \
|
| 123 |
--checkpoint-format fastwam \
|
| 124 |
+
--checkpoint checkpoints/streamingwam/joint-cd/model.pt \
|
| 125 |
--backbone-path checkpoints/Wan2.2-TI2V-5B \
|
| 126 |
+
--stats-path checkpoints/streamingwam/joint-cd/dataset_stats.json \
|
| 127 |
--libero-home third_party/LIBERO \
|
| 128 |
--num-steps-wait 30 \
|
| 129 |
--replan-steps 16 \
|
|
|
|
| 134 |
--save-video
|
| 135 |
```
|
| 136 |
|
| 137 |
+
For more evaluation options, see the [LIBERO guide](https://github.com/SJTU-DENG-Lab/Streaming-WAM/blob/main/examples/libero/LIBERO.md).
|
| 138 |
|
| 139 |
## License
|
| 140 |
|
| 141 |
+
Released under the [Apache License 2.0](https://github.com/SJTU-DENG-Lab/Streaming-WAM/blob/main/LICENSE).
|
| 142 |
|
| 143 |
## Acknowledgements
|
| 144 |
|
| 145 |
+
Streaming-WAM builds on ideas and open-source work from [FastWAM](https://github.com/yuantianyuan01/FastWAM), [StarWAM](https://github.com/shaohua-pan/StarWAM), [LIBERO](https://github.com/Lifelong-Robot-Learning/LIBERO), and [Wan2.2](https://github.com/Wan-Video/Wan2.2).
|
|
|