Embodied Intelligence from Human Experience - Simar Kareer

In this talk

1. 4:26

   Learning from embodied demonstrations

   Kareer presents EgoMimic, which trains a shared policy on robot demonstrations and human hand motion captured with Project Aria glasses. Its tests include toy sorting, grocery packing, and laundry folding. In controlled comparisons, adding human data improved performance, and results rose as more human demonstrations were included.

2. 16:24

   Building a dataset for scale

   Kareer describes EgoVerse, a dataset and collaboration infrastructure for robot learning. Its flagship collection includes about 100 hours across six controlled tasks, while a free-form collection adds more than 200 hours across roughly 1,500 tasks. The data includes hand and head tracking plus language annotations, and tests across three labs found consistent gains on unfamiliar objects and placements.

3. 26:17

   Testing human-to-robot transfer

   Kareer reports fine-tuning the pretrained π₀.₅ model with an even mix of human and robot data, without explicitly aligning the two sources. Human demonstrations improved results on unfamiliar scenes, objects, and task instructions, including sorting eggs by color and cleaning up unseen spaces. The benefit grew with the strength and embodiment diversity of robot pretraining.

4. 40:42

   What transfers and what comes next

   Ablations showed that both predicting hand motion and predicting language subtasks contributed to transfer. In the egg-sorting example, human data conveyed which items belonged where, while robot data supplied the motion skills needed to handle them. Kareer argues that some robot data is needed to make broader human data useful, and sees large, varied human collections as especially valuable for pretraining.