Ep#17 EgoZero: Robot Learning from Smart Glasses

In this talk

1. 1:28

   Why collect demonstrations from people

   Vincent argues that robot teleoperation is costly to scale, while handheld demonstrations can be slow and unnatural. EgoZero aims to use people’s everyday, egocentric activity as a source of robot training data, with a lightweight setup that works across environments.

2. 3:41

   A shared 3D representation

   Building on Point Policy, EgoZero represents human and robot demonstrations as ordered 3D points describing objects and end effectors. The authors say this makes the representation morphology-agnostic and allows straightforward 3D augmentation. Their proof of concept maps the midpoint between a person’s thumb and index finger to a two-finger gripper.

3. 11:34

   Reconstructing motion from smart glasses

   The Aria glasses provide camera pose and hand and wrist tracking, but their first generation lacks ground-truth depth and useful stereo overlap. EgoZero triangulates object points across the wearer’s movement, assuming objects stay stationary during the trajectory; fingertip motion remains closed loop, while object state is not tracked frame by frame. The authors describe roughly 60 to 70 percent task success and say measurement quality mattered more than adding data beyond their 20 minutes of demonstrations.

4. 24:38

   Where the system struggles and could improve

   The approach can transfer to changes such as a different towel color, but the team had not tested multiple towel distractors. At inference, object segmentation and point correspondence can fail, and low-resolution iPhone depth can make points jump, especially near object edges. The authors suggest eventually learning geometric behavior without requiring fixed point correspondences.

5. 32:50

   Scaling data and robot learning

   The authors favor natural hand use because gloves and other equipment can constrain movement and reduce the fidelity of demonstrations. They envision wider data collection through useful, comfortable smart glasses, while acknowledging battery, cost, and privacy concerns. They see human demonstrations as a way to bootstrap robot skills, with autonomous exploration as a longer-term goal that still faces practical challenges such as resets and failures.