| ---
|
| library_name: pytorch
|
| tags:
|
| - motion
|
| - rvq
|
| - vector-quantization
|
| - human-motion
|
| - safetensors
|
| - humanML
|
| - motion-reconstructor
|
| license: cc-by-nc-4.0
|
| datasets:
|
| - Wojtekb30/HumanML3D-500ms-FPP-descriptions-CoTs-1
|
| ---
|
|
|
| # Motion RVQ (Move Reconstruction)
|
|
|
| This model uses Residual Vector Quantization (RVQ) to reconstruct motion represented as 263-dimensional frame vectors.
|
| It is a custom PyTorch model (not a Transformers `AutoModel`) and is loaded from `safetensors` with `rvq_model.py`.
|
|
|
|
|
| 
|
|
|
| ## Model Summary
|
|
|
| - Architecture: encoder -> 4-level RVQ -> decoder
|
| - Input shape: `(T, 263)` per sequence (frame-major)
|
| - Training window: 100 frames (with crop/pad in dataset loader)
|
| - Output: reconstructed motion sequence in the same 263-dim representation
|
|
|
| ## Repository Files
|
|
|
| - `motion_rvq_weights.safetensors` - main published checkpoint
|
| - `config.json` - model configuration metadata
|
| - `rvq_model.py` - model architecture (`MotionRVQ_VAE`)
|
| - `TestRVQ.py` - inference + 3-panel visualization
|
| - `TrainRVQ.py` - training script
|
| - `rvq_humanml_dataset.py` - training dataset loader
|
| - `Mean.npy`, `Std.npy` - normalization statistics
|
| - `000001.npy`, `000012.npy` - sample motion files
|
|
|
| `motion_rvq_weights.pth` can be treated as a legacy artifact; code uses `motion_rvq_weights.safetensors`.
|
|
|
| ## Install
|
|
|
| ```bash
|
| pip install torch safetensors numpy matplotlib
|
| ```
|
|
|
| ## Inference
|
|
|
| Run the provided visualization script:
|
|
|
| ```bash
|
| python TestRVQ.py
|
| ```
|
|
|
| By default, `TestRVQ.py` uses `000001.npy`. You can change `FILE_TO_TEST` in `TestRVQ.py` to another sequence.
|
|
|
| Minimal loading example:
|
|
|
| ```python
|
| from pathlib import Path
|
| import torch
|
| from safetensors.torch import load_file
|
| from rvq_model import MotionRVQ_VAE
|
|
|
| base = Path(".")
|
| device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
|
|
|
| model = MotionRVQ_VAE().to(device)
|
| state_dict = load_file(str(base / "motion_rvq_weights.safetensors"), device=str(device))
|
| model.load_state_dict(state_dict)
|
| model.eval()
|
| ```
|
|
|
| ## Training From Scratch
|
|
|
| Expected layout:
|
|
|
| ```text
|
| rvq/
|
| TrainRVQ.py
|
| rvq_model.py
|
| rvq_humanml_dataset.py
|
| Mean.npy
|
| Std.npy
|
| new_joint_vecs/
|
| *.npy
|
| ```
|
|
|
| Run training:
|
|
|
| ```bash
|
| python TrainRVQ.py
|
| ```
|
|
|
| Output checkpoint:
|
|
|
| - `motion_rvq_weights.safetensors`
|
|
|
| ## Limitations
|
|
|
| - This model reconstructs motion vectors; it is not a text-to-motion generator.
|
| - Input format must match the same 263-dim representation and normalization scheme used during training. |
