Google DeepMind robotics lab tour with Hannah Fry
Hannah Fry, host of Google DeepMind’s podcast, tours Google DeepMind’s California robotics lab with robotics director Kanishka Rao and researchers to explore how multimodal models help robots handle flexible, longer tasks.
In this talk
- 1:00
Models broaden what robots can understand
Kanishka Rao says newer visual backbones help robots generalize across different lighting and backgrounds. Vision-language-action models connect instructions and visual scenes with physical actions, helping robots respond to new situations. He describes version 1.5 as adding an agent that can combine smaller actions into longer tasks, alongside a model that reasons about actions before taking them.
- 4:49
Dexterity learned through demonstrations
A robot attempts to pack a lunchbox, a task that requires careful handling of a small Ziploc bag and fitting bread into a tight space. Fry sees the robot manage several items, while also observing mistakes and retries. A researcher explains that the skill was learned through teleoperation, with people demonstrating the task from the robot’s perspective.
- 7:55
Generalizing to unfamiliar objects and requests
Researchers demonstrate a robot that follows spoken instructions, including moving colored blocks and opening a small pot to place an unfamiliar squishy object inside. A Gemini layer lets people converse with it while it acts. Fry then sees a humanoid robot sort laundry and handle objects that were not included in its training.
- 10:13
Agents coordinate longer tasks
The researchers describe an orchestrator that breaks a high-level request into smaller instructions for a vision-language-action model. In a sorting demonstration, the system looks up San Francisco’s waste categories and directs a robot to tidy items into bins. Another laundry demo uses one end-to-end model that outputs both its reasoning and actions.
- 14:56
Physical interaction data remains a challenge
Fry notes that the robots can be slow and make errors, though their actions often show an understanding of the request. Rao calls current methods foundational and says another major breakthrough is still needed. He identifies the limited supply of robot interaction data as a key constraint, while suggesting that learning from human manipulation videos could help expand training.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.