Fall 2026 GRASP SFI: Jesse Zhang, University of Washington
Jesse Zhang, a PhD student at the University of Washington, presents at the GRASP Lab’s Fall 2026 GRASP SFI about extracting more useful training signal from robot data to help robots keep learning after deployment.
In this talk
- 0:02
Why robot data can underdeliver
Zhang opens with a Pi Zero policy that reaches for a carrot when asked to put a sausage in a pot, even though its vision-language backbone can recognize both. He argues that robot datasets may look visually rich but often have correlated scenes, simple instructions and repetitive action patterns, so training extracts less information than the data contains.
- 5:52
Training policies to predict more
Zhang proposes objectives that draw denser signal from existing robot data, especially during pretraining and reinforcement learning post-training. His FlexPi model predicts future RGB, 3D point maps and DINO features alongside actions, and uses cross-modality masking to make the model learn representations even when some input and output streams are missing. In reported evaluations, FlexPi with half the demonstrations outperformed baselines trained with all the data; on the screwdriver insertion task, full joint prediction reached 76% task completion.
- 27:02
Using context smoothing to help reinforcement learning
Zhang introduces TMRL, which trains policies on varying amounts of noise added to their context, then lets reinforcement learning choose the noise level. The approach is intended to help an overfit policy explore action patterns from its training data, rather than remain stuck in its usual response. In the carrot and sausage example, noising helped a fine-tuned Pi 0 policy redirect its trajectory toward the sausage; Zhang also reports real-world and simulation results on other tasks.
- 41:32
Where robot learning can complement frontier models
Zhang argues that robot learning should keep increasing the training signal available from datasets, while focusing on areas such as dexterity, speed and reinforcement learning improvement. He shows a VLM-guided block task and describes work on zero-shot reward models that turn scene descriptions into feedback for policy training. He also discusses combining vision-language models with world-action models, using language to steer a robot policy.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.