RI Seminar: Saurabh Gupta : Robot Learning by Understanding Egocentric Videos

In this talk

1. 0:12

   Scaling robot learning with observation

   Gupta contrasts the large internet datasets used in vision and language with robotics policies often trained on a small number of objects and settings. He argues that learning from videos of people and other agents could expand the variety of experience available to robots, while acknowledging that observation alone will not solve robot learning.

2. 10:32

   Learning affordances and object states

   For manipulation, Gupta describes using human hand activity in videos to identify interaction locations, likely grasps, and changes in object state. Examples include detecting plate edges and drawer handles, and learning features that distinguish states across different objects. These predictions can guide robot exploration and reward learning.

3. 22:08

   Separating hands from the scene and adding 3D cues

   Human hands provide useful signals but create a gap between human and robot imagery. Gupta describes using segmentation and video-aware diffusion inpainting to separate the agent from its surroundings, then transferring reward signals to a robot; in drawer-opening experiments, this approach learned faster than inpainting alone or using no inpainting. He also presents hand-pose estimation improvements using image crop position and auxiliary labels from in-the-wild data.

4. 43:31

   Handling errors beyond demonstrations

   Gupta explains how behavior cloning can fail when a robot drifts away from the states shown in demonstrations. His group uses a diffusion model to create perturbed robot-camera views and assigns corrective actions based on nearby expert trajectory data. Across four tasks, the augmented data improved on behavior cloning, though Gupta notes that the tests were conducted in the lab and did not show broad transfer across labs.

5. 50:55

   Planning whole-body motions for drawers and cupboards

   Gupta describes a learned planner that uses past experience to turn end-effector poses into whole-body motion plans for a mobile robot. In tests across 13 buildings, the system opened unseen drawers and cupboards with a 61% success rate; perception, rather than control, was the main weakness reported. Gupta frames video-derived knowledge as a useful prior for exploration, alongside the need for robot interaction and experience.