Ep#75: TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics
Shay, a fifth-year PhD student at the University of Washington and student researcher at AI2, and Cole Harrison, a machine learning engineer at Amazon and AI2 collaborator, discuss TOPReward with the RoboPapers host.
In this talk
- 1:00
Why robotics needs denser rewards
Shay describes TOPReward as a training-free way to estimate task progress from video, with uses such as success detection and data filtering. The guests argue that sparse rewards and the sim-to-real gap make real-world robot learning difficult: robots need feedback during a task, not only after a full rollout.
- 7:04
Reading progress from token probabilities
The guests contrast TOPReward with GVL, which shuffles video frames and asks a vision-language model to produce numeric progress labels. TOPReward instead prompts the model to judge whether a video completes a task, then uses the log probability of the answer token “true” as a progress signal. They say this avoids relying on the model to produce numbers, which can be unreliable.
- 17:21
Examples, models, and prompt sensitivity
The host shows a bike video where the score rises as the rider approaches a trick and drops after the rider loses balance. A soccer example also tracks changing prospects as Argentina scores against France. The guests report testing Gemini, M2, and Qwen, with two- to three-billion-parameter models performing poorly in some tests, and say task wording can change the result.
- 22:52
Comparing reward methods and interpreting scores
The guests explain that TOPReward's normalized score was used to compare with GVL, while their reinforcement learning experiment used reward differences without normalizing to zero and one. They describe RoboMeter as fine-tuned on robotics data, reporting strong results in some single-arm settings, while TOPReward may generalize to settings outside a method's training data. They also discuss score spikes, saturation on long tasks, and a possible future approach that breaks tasks into subtasks.
- 50:18
Applications and next steps
The guests discuss using TOPReward with advantage-weighted regression and explain that stationary stretches in demonstrations can produce zero advantage and no learning signal. Further reinforcement learning experiments are still in progress. They see potential applications beyond robotics, including video curation and other agents.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.