Robo-ValueRL

[Project Page] [GitHub] [Dataset] [Paper]

This repository contains the model checkpoints for Robo-ValueRL: Reliable Value Estimation for Offline-to-Online Reinforcement Learning.

Robo-ValueRL studies how reliable value estimation can guide robotic policy learning from heterogeneous offline demonstrations and online rollout trajectories. The framework learns a history-conditioned value estimator, converts value differences into action-quality conditions for offline policy pretraining, and uses value-guided rollout filtering for stable online residual adaptation.

Model Description

Robo-ValueRL is an offline-to-online robotic reinforcement learning framework centered on reliable value estimation. Instead of only reporting final task success, Robo-ValueRL explicitly diagnoses whether learned values capture global task progress and local action-level preference, then propagates these value signals into downstream policy learning.

The released model suite includes:

1. History-Conditioned Value Estimator
   Predicts normalized task progress from multi-view robot observations, language instructions, and visual history. The temporal context helps reduce ambiguity caused by occlusions, repeated motions, and visually similar task stages.

2. Quality-Conditioned VLA Policy
   Uses value differences to derive action-quality conditions. These conditions guide a Vision-Language-Action policy during offline pretraining, allowing the policy to prioritize useful behaviors from mixed-quality demonstrations.

3. Online Residual Adaptation Module
   Learns lightweight corrections from value-filtered online rollouts while keeping the pretrained base policy frozen. This enables targeted failure recovery and self-correction without overwriting the offline prior.

Model Hierarchy

1. Value Estimation

   • Learns history-conditioned value functions from heterogeneous robot data.
   • Evaluates value reliability with global-progress and local-preference metrics.

2. Offline Policy Pretraining

   • Converts value differences into action-quality conditions.
   • Trains a quality-conditioned VLA policy on mixed-quality demonstrations.

3. Online Policy Improvement

   • Uses reliable value estimates to filter online rollout data.
   • Trains a lightweight residual adapter for targeted real-world improvement.

Associated Dataset

The models are trained and evaluated with the Robo-ValueRL dataset:

[Robo-ValueRL Dataset]

The dataset contains heterogeneous real-robot demonstrations and online rollout trajectories for chip insertion and block disassembly.

Key Features

• Reliable Value Estimation: Uses visual history to produce smoother progress estimates and sharper error responses.
• Value-Guided Data Utilization: Prioritizes useful demonstrations and rollout segments from heterogeneous robot experience.
• Quality-Conditioned Policy Learning: Conditions the VLA policy on value-derived action quality.
• Stable Offline-to-Online Improvement: Improves real-world performance through residual adaptation while preserving the pretrained base policy.
• Real-Robot Evaluation: Evaluated on precision chip insertion and generalizable block disassembly.

Highlights

• 86% final success on chip insertion
• 84% final success on block disassembly
• +26% offline gain on chip insertion
• +34% offline gain on block disassembly
• 240h offline demonstrations
• 3,000+ online rollout trajectories

Usage

The released assets are organized for reproducing Robo-ValueRL's model pipeline.

Please refer to the GitHub repository for setup instructions, inference scripts, training code, and data-processing utilities.

Citation

If you use Robo-ValueRL in your research, please cite our work. Citation will be updated after the arXiv release.

License

Please refer to the license file in the GitHub repository.

Contact

For questions, please open an issue on our GitHub repository.