Professor Danfei Xu of Georgia Tech gives the Conference on Robot Learning early career keynote on learning robot skills from embodied human data.
Xu contrasts internet-scale AI training data with the slow collection of robot demonstrations. Teleoperation provides useful links between what a robot sees and how it acts, but it requires a robot and deliberate operator effort. He proposes collecting human sensorimotor data without a robot in the loop.
MimicPlay uses human videos to predict hand movement as a signal of task intent, while teleoperation data trains the robot’s low-level controller. The resulting hierarchical policy was demonstrated on household tasks including table cleanup and towel folding. Xu notes that reliance on robot data still limits the low-level policy.
EgoMimic captures first-person video and hand tracking with Project Aria glasses, then pairs human data with data from a lightweight bimanual robot using the same glasses as its vision sensor. The system stabilizes hand trajectories against head movement using device tracking and trains a shared policy on human and robot data. On an object-flipping task, one hour of human data produced a marked performance gain; the system also showed transfer to new scenes.
Xu describes work that records human tactile and audio signals with a GelSight sensor and contact microphone, then transfers learned policies to robot insertion tasks. He argues that future datasets should include robot-suitable activities and annotations such as hand tracks and device poses. He also points to potential uses in human-robot collaboration, where sensorimotor data could help models predict human motion and intent.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.
Danfei Xu, a robotics researcher and faculty member at Georgia Tech, presents his work on generative task and motion planning at a robotics seminar; the host introduces him, and no other venue is stated.
