Ep#111: SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learning
Philip Schroeder, an MIT computer science PhD student, and Thomas Wang, a research scientist at the Robotics and AI Institute, discuss SOLE-R1 with the RoboPapers hosts and audience.
In this talk
- 1:04
Training a reward model to resist hacking
SOLE-R1 uses video-language reasoning to provide the reward signal for online reinforcement learning across unseen manipulation tasks. The researchers say general VLM reward models can be exploited: a policy might move a camera to make a failed grasp look successful, or slightly open a drawer to earn a high score. SOLE-R1’s predicted rewards tracked ground-truth rewards more reliably in these examples.
- 11:59
Data built around progress and failure
The training mixture combines spatial reasoning data with robot videos from RoboCasa and Open X-Embodiment. In RoboCasa, the team generated suboptimal trajectories by injecting random actions into expert demonstrations and recovering toward later expert states; simulator state provided dense progress estimates. About 90% of roughly 40,000 RoboCasa videos were suboptimal, alongside about 200,000 Open X-Embodiment videos.
- 16:19
Reasoning over time and training the model
For each prediction, the model sees the task, the first video frame, and the previous and current time steps, with wrist and external camera views. It reasons about changes between frames and predicts progress relative to its previous estimate; its generated reasoning text is not carried forward as memory. Training uses supervised fine-tuning followed by reinforcement learning to improve progress prediction, with the reasoning traces supplying much of the SFT signal.
- 34:56
Results, limitations, and what comes next
Across 40 tasks in five settings, SOLE-R1 achieved at least 50% success on 24 tasks, compared with seven for the strongest cited baseline, GPT-5. Ablations showed that removing suboptimal videos or general spatial reasoning increased reward-hacking failures, while removing reasoning led to weaker learning signals. The team also described newer LIBERO experiments using a pretrained policy, with average success of 98% on more complex multistep tasks; they pointed to real-world data, simulation, and additional sensing as complementary directions.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.