Ep#47: ResMimic: From General Motion Tracking to Humanoid Whole-Body Loco-Manipulation via Residual

In this talk

1. 1:17

   Why humanoid loco-manipulation is hard

   Suhang says humanoids can use coordinated whole-body movement to handle tasks in human-built spaces, but precise loco-manipulation is harder than locomotion or tabletop manipulation. Human demonstrations are easier to collect for general motion tracking than human-object interaction data, and retargeting can create contact errors such as floating hands or object penetrations.

2. 5:46

   ResMimic adds a residual policy

   ResMimic first trains a general motion-tracking policy, then freezes it and trains a residual policy with humanoid and object information. The two policies’ outputs are added together, and the residual actor starts near zero so the pretrained policy can guide early behavior. A virtual object controller stabilizes training before gradually handing control to the robot; point-based object tracking and discretized contact rewards encourage accurate object movement and human-like contact.

3. 10:53

   Real-world tasks and comparisons

   The team compares ResMimic with direct motion tracking, training from scratch, and fine-tuning the pretrained policy. In demonstrations, the humanoid carries boxes on its back, kneels to lift, handles a 4.5 kg box and chair, and manipulates objects up to 5.5 kg using whole-body contact. The speakers report about 92.5% mean task success for ResMimic versus about 10% for the base tracking policy; they also say residual training can converge in about 30 minutes on a 4090 GPU.

4. 23:53

   Evaluation and design lessons

   The team trains in Isaac Gym and evaluates sim-to-sim transfer in MuJoCo. They say training from scratch fails on the evaluated MuJoCo tasks, while fine-tuning can work in Isaac Gym but loses transfer performance. Ablations suggest the virtual object controller helps overcome imperfect demonstrations, while contact rewards encourage torso and arm contact that transfers better than relying on the wrists alone.

5. 37:49

   Limitations and possible extensions

   The current system uses motion capture for human and object data and struggles with large changes in object pose or size; the team reports about 10% size variation in one example. They suggest reconstructing human and object motion from video, and discuss replacing motion capture object input with depth sensing or a separate 6D pose estimator. More varied demonstrations or motion-style methods could help the system choose different pickup strategies for different objects.