Ep#96: EgoHumanoid: Unlocking In-the-Wild Loco-Manipulation with Robot-Free Egocentric Demonstratio

In this talk

1. 0:56

   Why use human demonstrations

   Robot demonstrations are costly and difficult to collect outside laboratories, while people can record examples in homes, stores, and other varied places. But human and robot demonstrations differ in viewpoint and actions, so combining them requires ways to align those differences.

2. 2:45

   Collecting and aligning the data

   People wore a Pico 4 Ultra headset with motion trackers and a ZED Mini camera; robot demonstrations used a Unitree H1 with a matching camera. To reduce viewpoint differences, the team estimated depth from human video and reprojected the scene from a lower camera position, around the robot’s 1.3-meter height. They also used shared action formats, including end-effector pose changes, walking velocity commands, and hand open or close.

3. 17:56

   Co-training results and trade-offs

   The team compared robot-only training with human-robot co-training across laboratory and more varied environments. Co-training improved performance in generalization tests, and the pillow placement example reached 97.5% in one unseen setting. The best human-to-robot data mix varied by task, with precise manipulation benefiting more from robot examples.

4. 24:04

   Where human data helps next

   The analysis found navigation harder to improve than manipulation, although navigation behaviors appeared to transfer more readily between people and robots. Modi sees large, diverse human datasets as useful for learning whole-body behavior, such as moving through narrow spaces. Fine manipulation and tactile skills remain difficult to transfer because human hands and robot sensors differ.