Ep#50: EMMA: Scaling Mobile Manipulation via Egocentric Human Data

In this talk

1. 0:36

   Why mobile robot data is costly

   Lawrence says mobile teleoperation takes much longer than performing the same task as a person, making it hard to collect enough demonstrations. EMMA builds on EgoMimic’s finding that human and robot demonstrations can be combined to increase trajectory coverage. The team captures human data with Project Aria glasses, which provide hand tracking and offline SLAM.

2. 11:49

   Separating navigation from manipulation

   The team’s central idea is to combine mobile human demonstrations with static robot demonstrations, since navigation appears more transferable between people and robots than grasping. They use hand-to-head motion and spatial clustering to identify navigation and manipulation phases, then remove head movement from the manipulation data. To make human paths executable by a differential-drive base, they optimize robot velocities to follow the human waypoints smoothly.

3. 27:39

   Results across tasks and scenes

   EMMA is evaluated on table service, wine handover, and bimanual grocery shopping, with comparisons to robot-only training and mobile teleoperation. In a new scene, half an hour of human data combined with static robot data helps the robot complete a wine handover without collecting robot demonstrations there. In a scaling study, adding human data from 15 to 60 minutes improved full-task success; the 60-minute set contained about 200 human demonstrations.

4. 33:31

   Limits, extensions, and data scale

   Ablations show that skipping phase detection can make the robot move during precise manipulation, while skipping trajectory retargeting can send it toward the wrong direction. The team also reports that human demonstrations improved grasp accuracy when pushing a chair, especially by providing dense examples of approaching and grasping its backrest. They identify a key limitation: people need to look roughly where they are moving, and say human data is intended to extend robot data rather than replace it.

5. 49:13

   Future sensing and data collection

   Pranav points to wider use of smart glasses as a way to make human demonstrations easier to collect at scale. The researchers see shared 3D scene information and future depth sensing as possible ways to reduce the visual gap between human and robot observations. They also discuss eye gaze as a possible signal of intent, though they have not established a task that needs it.