Toward Agile and Generalizable Humanoid Interaction Skills from Human Videos - 0227 Yinhuai Wang

In this talk

1. 1:20

   From motion imitation to interaction

   Wang distinguishes local motion imitation from interacting with objects in dynamic settings. Teleoperation-based data collection and task-specific rewards can support interaction, but he says they have limits in robustness and generality. HumanX builds on interaction imitation, learning from both human and object behavior.

2. 9:36

   Turning human videos into interaction data

   HumanX’s XGen pipeline converts human videos into humanoid interaction trajectories without explicitly modeling human-object interaction data. It uses tools such as GM and GMR to estimate and retarget motion, then derives object trajectories from an object mesh and hand-object anchor configuration. The pipeline adjusts contact stages and simulates non-contact motion, such as a flying ball.

3. 23:37

   Training one policy for varied interactions

   X-Mimic first trains a teacher using privileged information, then distills it into a student policy for real-world deployment at 100 Hz. Its shared imitation reward includes body motion, object motion, human-object relative motion, and contact. Wang describes disturbed object initialization, interaction-based early termination, and data augmentation as ways to improve generalization.

4. 31:49

   Testing generalization in simulation and the real world

   Simulation tasks include basketball catch-and-shoot, badminton hitting, and cargo pickup. Wang reports that the method outperformed SkillMimic, OmniRetarget, and HDMI in the presented comparisons, and that skills learned from a single video generalized across different basket positions, shuttlecock positions, and cargo locations. Real-world demonstrations include basketball, repeated football kicks, cargo pickup, and responses to a person’s actions; continuous interaction requires motion-capture sensing, while proprioception alone loses track when contact ends.