Fall 2025 GRASP SFI -Tairan He, Carnegie Mellon University & NVIDIA

In this talk

1. 0:53

   A framework for humanoid skills

   He organizes humanoid learning around control, perception, interaction, and evaluation. He compares approaches by real-world transfer, control complexity, and task diversity, with the goal of combining perception, whole-body control, and manipulation across many tasks.

2. 4:41

   Learning control from human motion

   H2O turns AMASS motion capture into humanoid-feasible motions through shape fitting and physics-based retargeting, then trains a reinforcement learning policy to track them. OmniH2O uses Apple Vision Pro to provide three command points and distills a teacher policy into a student that can control the full body. HOVER extends this idea by masking commands during training so one policy can respond to interfaces such as VR, motion capture, and joysticks.

3. 24:27

   Improving sim-to-real transfer

   ASAP collects real robot trajectories and trains a delta-action model to make a hybrid simulator better match real dynamics, then fine-tunes the policy in that simulator. He says the learned corrections were largest around the knees and ankles. ABS combines an agile policy with a safety recovery policy and uses depth images to predict obstacle distances for collision avoidance.

4. 33:29

   Perception, manipulation, and interaction

   PDC trains a visual reinforcement learning policy in simulation for whole-body manipulation, controlling body and finger motors across kitchen tasks. He also describes WaCoCo, which learns whole-body control from sequential contact commands, and Falcon, which uses separate lower-body and upper-body agents for forceful tasks. Examples include pushing an 85-kilogram cart and opening a heavy door.

5. 37:27

   Future directions and evaluation

   He wants to combine visual perception, control, and dexterous manipulation in scalable sim-to-real systems. He also proposes using simulation to evaluate real-world policies, provided simulated performance corresponds closely to hardware results. In the discussion, he argues better hardware can reduce transfer effort, while data and teleoperation can bring more people into robot skill collection.