File size: 12,240 Bytes
700dd75 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 | # Configuration Guide
SONIC uses [Hydra](https://hydra.cc/) for hierarchical configuration. This guide
explains the config structure and the most important parameters to tune.
## Config Hierarchy
When you run a training command like:
```bash
python gear_sonic/train_agent_trl.py +exp=manager/universal_token/all_modes/sonic_release
```
Hydra composes the final config from a chain of YAML files:
```
gear_sonic/config/
├── base.yaml # Global defaults (seed, num_envs, paths)
├── base/
│ ├── hydra.yaml # Hydra output directory settings
│ └── structure.yaml # Resolved experiment directory structure
├── algo/
│ └── ppo_im_phc.yaml # PPO hyperparameters
├── manager_env/
│ ├── base_env.yaml # Environment defaults (sim_dt, decimation, episode length)
│ ├── actions/tracking/base.yaml
│ ├── commands/tracking/base.yaml
│ │ └── terms/motion.yaml # Motion library, body names, future frames
│ ├── rewards/tracking/
│ │ └── base_5point_local_feet_acc.yaml # Reward composition
│ │ └── terms/*.yaml # Individual reward terms with weights
│ ├── terminations/tracking/
│ │ └── base_adaptive_strict_ori_foot_xyz.yaml # Termination composition
│ │ └── terms/*.yaml # Individual termination conditions
│ ├── events/tracking/
│ │ └── level0_4.yaml # Domain randomization events
│ └── observations/
│ ├── tokenizer/ # Encoder input observations
│ ├── policy/ # Policy (actor) observations
│ └── critic/ # Critic observations
├── actor_critic/
│ └── universal_token/ # Network architecture (encoders, decoders, quantizer)
├── aux_losses/
│ └── universal_token/ # Auxiliary loss terms
├── trainer/
│ └── trl_ppo_aux.yaml # Trainer config (PPO with aux losses)
├── callbacks/ # Training callbacks (save, eval, W&B, resample)
└── exp/manager/universal_token/all_modes/
├── sonic_release.yaml # Original release experiment config
└── sonic_v1_1.yaml # SONIC v1.1 experiment config
```
The experiment config (`sonic_release.yaml`) sits at the top and overrides
specific values from the base configs. You can further override any value
from the command line with `++key=value`.
## Overriding Config Values
Hydra uses `++` prefix to force-override values (even nested ones):
```bash
# Override a top-level value
python gear_sonic/train_agent_trl.py +exp=... num_envs=16
# Override a nested value (use dots for nesting)
python gear_sonic/train_agent_trl.py +exp=... \
++manager_env.commands.motion.motion_lib_cfg.motion_file=/path/to/data
# Override a reward weight
python gear_sonic/train_agent_trl.py +exp=... \
++manager_env.rewards.tracking_anchor_pos.weight=1.0
```
## Top Parameters to Tune
### Training scale
| Parameter | Default | Location | Description |
|-----------|---------|----------|-------------|
| `num_envs` | 4096 | `base.yaml` | Number of parallel environments. Reduce for debugging (`16`), increase for throughput. |
| `headless` | True | `base.yaml` | Set `False` to open the Isaac Lab viewer for visual debugging. |
| `seed` | 0 | `base.yaml` | Random seed for reproducibility. |
### PPO hyperparameters
| Parameter | Default | Location | Description |
|-----------|---------|----------|-------------|
| `algo.config.actor_learning_rate` | 2e-5 | `ppo_im_phc.yaml` | Actor learning rate. Lower for finetuning, higher for training from scratch. |
| `algo.config.critic_learning_rate` | 1e-3 | `ppo_im_phc.yaml` | Critic learning rate. Usually 10-100x the actor LR. |
| `algo.config.num_learning_epochs` | 5 | `ppo_im_phc.yaml` | PPO epochs per batch of experience. |
| `algo.config.num_mini_batches` | 4 | `ppo_im_phc.yaml` | Mini-batches per PPO epoch. |
| `algo.config.num_steps_per_env` | 24 | `sonic_release.yaml` | Rollout length (steps per env before PPO update). |
| `algo.config.gamma` | 0.99 | `ppo_im_phc.yaml` | Discount factor. |
| `algo.config.lam` | 0.95 | `ppo_im_phc.yaml` | GAE lambda. |
| `algo.config.clip_param` | 0.2 | `ppo_im_phc.yaml` | PPO clip parameter. |
| `algo.config.entropy_coef` | 0.01 | `ppo_im_phc.yaml` | Entropy bonus coefficient. |
| `algo.config.desired_kl` | 0.01 | `ppo_im_phc.yaml` | Target KL for adaptive learning rate schedule. |
| `algo.config.num_learning_iterations` | 100000 | `ppo_im_phc.yaml` | Total training iterations. |
### Simulation
| Parameter | Default | Location | Description |
|-----------|---------|----------|-------------|
| `manager_env.config.sim_dt` | 0.005 | `base_env.yaml` | Physics timestep (200 Hz). Smaller = more stable but slower. |
| `manager_env.config.decimation` | 4 | `base_env.yaml` | Policy runs every `decimation` sim steps (50 Hz policy at 200 Hz sim). |
| `manager_env.config.episode_length_s` | 10.0 | `base_env.yaml` | Episode length in seconds before timeout reset. |
| `manager_env.config.terrain_type` | trimesh | `sonic_release.yaml` | `plane` for flat ground, `trimesh` for rough terrain. |
| `manager_env.config.robot.type` | g1_model_12_dex | `sonic_release.yaml` | Robot type (must match `robot_mapping` in code). |
### Motion data
| Parameter | Default | Location | Description |
|-----------|---------|----------|-------------|
| `manager_env.commands.motion.motion_lib_cfg.motion_file` | — | `sonic_release.yaml` | Path to retargeted robot motion PKLs. |
| `manager_env.commands.motion.motion_lib_cfg.smpl_motion_file` | — | `sonic_release.yaml` | Path to SMPL motion PKLs (or `dummy`). |
| `manager_env.commands.motion.motion_lib_cfg.soma_motion_file` | — | `sonic_bones_seed.yaml` | Path to SOMA motion PKLs (4-encoder config only). |
| `manager_env.commands.motion.motion_lib_cfg.smpl_y_up` | true | `sonic_release.yaml` | Set `true` if SMPL data uses y-up coordinates. |
| `manager_env.commands.motion.motion_lib_cfg.target_fps` | 50 | `motion.yaml` | Target FPS for motion resampling. |
| `manager_env.commands.motion.motion_lib_cfg.asset.assetFileName` | g1_29dof_rev_1_0.xml | `motion.yaml` | MJCF file for motion library FK. Change for different robots. |
### Motion command
| Parameter | Default | Location | Description |
|-----------|---------|----------|-------------|
| `manager_env.commands.motion.num_future_frames` | 10 | `sonic_release.yaml` | Number of future reference frames provided to the policy. |
| `manager_env.commands.motion.dt_future_ref_frames` | 0.1 | `sonic_release.yaml` | Time spacing between future frames (seconds). |
| `manager_env.commands.motion.cat_upper_body_poses` | true | `sonic_release.yaml` | Augment lower-body motions with upper-body from different clips. |
| `manager_env.commands.motion.cat_upper_body_poses_prob` | 0.5 | `sonic_release.yaml` | Probability of upper-body augmentation per episode. |
| `manager_env.commands.motion.freeze_frame_aug` | true | `sonic_release.yaml` | Augment with frozen (static) reference frames. |
### Observation history
| Parameter | Default | Location | Description |
|-----------|---------|----------|-------------|
| `actor_prop_history_length` | 10 | `sonic_release.yaml` | Number of past proprioception frames stacked for actor. |
| `actor_actions_history_length` | 10 | `sonic_release.yaml` | Number of past actions stacked for actor. |
| `critic_prop_history_length` | 10 | `sonic_release.yaml` | Same, for critic. |
| `critic_actions_history_length` | 10 | `sonic_release.yaml` | Same, for critic. |
### Reward weights
All reward terms have a `weight` parameter. Positive weights encourage the behavior,
negative weights penalize it. The default weights for `base_5point_local_feet_acc`:
| Reward term | Weight | Description |
|-------------|--------|-------------|
| `tracking_anchor_pos` | 0.5 | Root position tracking |
| `tracking_anchor_ori` | 0.5 | Root orientation tracking |
| `tracking_relative_body_pos` | 1.0 | Body position tracking (anchor-relative) |
| `tracking_relative_body_ori` | 1.0 | Body orientation tracking (anchor-relative) |
| `tracking_body_linvel` | 1.0 | Body linear velocity tracking |
| `tracking_body_angvel` | 1.0 | Body angular velocity tracking |
| `tracking_vr_5point_local` | 2.0 | 5-point (wrists + head + feet) local tracking |
| `action_rate_l2` | -0.1 | Smooth actions (penalize jerk) |
| `joint_limit` | -10.0 | Stay within joint limits |
| `undesired_contacts` | -0.1 | Penalize non-foot ground contacts |
| `anti_shake_ang_vel` | -0.005 | Penalize wrist/head jitter |
| `feet_acc` | -2.5e-6 | Penalize foot acceleration (smooth stepping) |
Each reward term also has a `std` parameter controlling the Gaussian kernel
sharpness. Smaller `std` = stricter tracking (reward drops faster with error).
Override example:
```bash
++manager_env.rewards.tracking_anchor_pos.weight=2.0
++manager_env.rewards.tracking_anchor_pos.params.std=0.1
```
### Termination thresholds
Terminations end episodes early when tracking error exceeds a threshold. The
adaptive variants use a curriculum that tightens thresholds over training:
| Termination | Threshold | Description |
|-------------|-----------|-------------|
| `anchor_pos` | 0.15 m | Root position deviation |
| `anchor_ori_full` | 0.2 rad | Root orientation deviation |
| `ee_body_pos` | 0.15 m | End-effector position deviation |
| `foot_pos_xyz` | 0.2 m | Foot position deviation |
| `motion_time_out` | — | Episode ends when motion clip finishes |
Looser thresholds (larger values) make training easier initially. The adaptive
terminations automatically tighten as the policy improves.
### Adaptive motion sampling
The motion library supports adaptive sampling — motions the policy fails on are
sampled more frequently:
| Parameter | Default | Description |
|-----------|---------|-------------|
| `adaptive_sampling.enable` | true | Enable adaptive sampling. |
| `adaptive_sampling.bin_size` | 50 | Window size for failure rate tracking. |
| `adaptive_sampling.adp_samp_failure_rate_max_over_mean` | 200 | Max/mean failure rate ratio cap. Prevents one hard motion from dominating. |
### Saving and logging
| Parameter | Default | Location | Description |
|-----------|---------|----------|-------------|
| `algo.config.save_interval` | 500 | `ppo_im_phc.yaml` | Save checkpoint every N iterations. |
| `algo.config.eval_frequency` | 500 | `ppo_im_phc.yaml` | Run evaluation every N iterations. |
| `use_wandb` | false | `base.yaml` | Enable Weights & Biases logging. |
| `base_dir` | logs_rl | `base.yaml` | Root directory for training outputs. |
## Experiment Configs
| Config | Encoders | Use case |
|--------|----------|----------|
| `sonic_release` | G1, teleop, SMPL | Default — matches the released checkpoint |
| `sonic_v1_1` | G1, teleop, SMPL | SONIC v1.1 with heading-normalized targets and wrist-pose augmentation |
| `sonic_bones_seed` | G1, teleop, SMPL, SOMA | Extended training with SOMA skeleton encoder |
| `sonic_h2` | G1, teleop, SMPL | H2 robot (31 DOF) |
## Common Recipes
### Debug a training run visually
```bash
python gear_sonic/train_agent_trl.py +exp=... \
num_envs=4 headless=False \
algo.config.num_learning_iterations=10
```
### Finetune with lower learning rate
```bash
python gear_sonic/train_agent_trl.py +exp=... \
+checkpoint=sonic_release/last.pt \
++algo.config.actor_learning_rate=5e-6 \
++algo.config.desired_kl=0.005
```
### Train on flat ground only
```bash
python gear_sonic/train_agent_trl.py +exp=... \
++manager_env.config.terrain_type=plane
```
### Relax termination thresholds for hard motions
```bash
python gear_sonic/train_agent_trl.py +exp=... \
++manager_env.terminations.anchor_pos.params.threshold=0.3 \
++manager_env.terminations.ee_body_pos.params.threshold=0.3
```
### Increase tracking precision
```bash
python gear_sonic/train_agent_trl.py +exp=... \
++manager_env.rewards.tracking_relative_body_pos.params.std=0.1 \
++manager_env.rewards.tracking_anchor_pos.params.std=0.1
```
|