| [2026-07-15T10:32:23+09:00] waiting for 50000: checkpoint directory not created |
| [2026-07-15T14:31:53+09:00] waiting for 50000: last checkpoint symlink not created |
| [2026-07-15T14:32:23+09:00] validated checkpoint 50000; GPU3 has 99644 MiB free |
| [2026-07-15T14:32:23+09:00] stopping GPU0 baseline PID 201098 with SIGTERM |
| [2026-07-15T14:32:29+09:00] GPU0 baseline process tree stopped cleanly |
| [2026-07-15T14:32:39+09:00] launched tmux session insight_baseline_g3 on GPU3 |
| [2026-07-15T14:33:15+09:00] ERROR: resumed process environment mismatch: {'INSIGHT_LEROBOT_ACTION_SPACE': {'actual': None, 'expected': 'ee_abs_rot6d'}, 'INSIGHT_LEROBOT_PROMPT_SET': {'actual': None, 'expected': 'detailed'}, 'INSIGHT_LEROBOT_SAVE_STEPS': {'actual': None, 'expected': '60000'}} |
| Traceback (most recent call last): |
| File "/home/ext_minje/clvla/benchmarks/INSIGHT/my_scripts/watch_move_baseline_to_gpu3.py", line 475, in <module> |
| raise SystemExit(main()) |
| ^^^^^^ |
| File "/home/ext_minje/clvla/benchmarks/INSIGHT/my_scripts/watch_move_baseline_to_gpu3.py", line 452, in main |
| resume_details = verify_resume( |
| ^^^^^^^^^^^^^^ |
| File "/home/ext_minje/clvla/benchmarks/INSIGHT/my_scripts/watch_move_baseline_to_gpu3.py", line 298, in verify_resume |
| raise RuntimeError(f"resumed process environment mismatch: {mismatches}") |
| RuntimeError: resumed process environment mismatch: {'INSIGHT_LEROBOT_ACTION_SPACE': {'actual': None, 'expected': 'ee_abs_rot6d'}, 'INSIGHT_LEROBOT_PROMPT_SET': {'actual': None, 'expected': 'detailed'}, 'INSIGHT_LEROBOT_SAVE_STEPS': {'actual': None, 'expected': '60000'}} |
|
|
