Ep#5 R+X: Retrieval & Execution from Human Videos
Research scientists Norman and Giorgos, working on the paper at Imperial College’s Robot Learning Lab, discuss learning robot tasks from human videos with RoboPapers hosts.
In this talk
- 1:51
Collecting everyday video
People wear a chest-mounted camera and record several minutes of uninterrupted, unlabeled activity. The researchers stitched clips together to simulate longer recordings, with tasks spread across rooms. Their experiments used depth alongside video.
- 9:00
Retrieving examples and generating actions
Given a language request, Gemini searches the recordings for relevant clips, such as phone pickups for a request to call someone. DINO features provide visual keypoints, while Keypoint Action Tokens represents scenes and robot trajectories as numbers that a language model can use for in-context imitation. The pipeline uses off-the-shelf models without training a new policy.
- 20:35
Generalization and scaling
The researchers tested tasks including picking up cans, handling laundry, wiping, pushing, and turning on lights. They reported advantages over Octo and a language-conditioned diffusion policy in hard spatial and language generalization, including unusual object positions and indirect commands. Retrieval also lets new task demonstrations be added without retraining, though in-context performance tends to plateau after roughly 20 examples.
- 43:48
Limits and next steps
The human-to-robot embodiment gap, keypoint information bottleneck, and dependence on metric depth limit performance, especially for precise tasks and obstacle avoidance. The researchers suggest combining human-video demonstrations with robot experience or reinforcement learning, and improving depth estimation could make videos without depth more useful. They caution that research results do not yet meet the reliability expected of consumer robots.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.