Trustworthy World Models for Safe & Generalist Robots - Anirudha Majumdar 0213
Anirudha Majumdar, associate professor at Princeton University and research scientist on Google DeepMind’s Robotics Safety and Alignment team, discusses action-conditioned video models for robot evaluation, training, and planning at a Robot Perception and Learning Lab talk.
In this talk
- 5:27
Using video models to evaluate policies
Majumdar describes fine-tuning Google’s Veo model to generate multiple camera views conditioned on robot actions, then using those videos to evaluate Gemini Robotics policies. Across eight policy variants, predicted and real success rates were strongly correlated, although success in the video model was about 2.5 times lower. Edited scenes also exposed weaknesses in instruction following and safety, including a policy that tried to close a laptop onto scissors.
- 22:14
Training world models with autonomous play
Majumdar argues that teleoperation demonstrations cover a narrow range of states and tend to emphasize successful behavior. His Princeton group used a vision-language model to propose tasks while a robot collected up to eight hours of play data autonomously, totaling 30 hours across objects. Fine-tuning on that data improved video accuracy and the relationship between simulated and real policy success; it also supported stronger reinforcement learning gains than a model fine-tuned on demonstrations.
- 42:17
Estimating when video predictions are unreliable
Majumdar presents a method that produces a patch-level confidence estimate alongside each generated video. In examples from DROID and Bridge, uncertainty rose around hallucinated objects, disappearing items, and visual distortions. The estimates were calibrated on in-distribution data and remained well calibrated in tested shifts such as altered lighting, clutter, backgrounds, and end-effector appearance.
- 50:00
Open challenges for robotic video models
Majumdar identifies fine-grained physical interaction and long video generation as unresolved problems, noting that five-minute or hour-long rollouts remain out of reach. He also raises the challenge of adding action control without losing a pretrained model’s generalization abilities, and questions whether pixel-level video prediction is the right approach for longer tasks compared with latent world models such as JEPA.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.