Ep#20 VideoMimic

In this talk

1. 1:04

   Learning humanoid skills from video

   VideoMimic aims to teach humanoids terrain locomotion, stair climbing, and sitting or standing by learning from human demonstrations. The pipeline turns monocular video into a 3D scene and human motion, retargets that motion to a robot, then uses simulation to train a policy. The goal is to reduce reliance on manually tuned locomotion rewards.

2. 13:03

   Reconstructing people and their surroundings

   The team records people with an iPhone and reconstructs both human motion and the static environment from video. They align an SMPL body model with scene point maps using 2D pose detections, then retarget the motion to a Unitree G1 and create a mesh for simulation. The authors describe limitations including noisy dynamic human point clouds, holes in reconstructed scenes, and a 30 to 50 percent pass rate for their own recorded videos.

3. 29:04

   Training and distilling a controller

   Training begins with motion capture pretraining, followed by tracking roughly 120 video clips alongside terrain height maps. The team then distills the reference-conditioned tracker into a policy that uses a root direction command and terrain observations, followed by reinforcement learning fine-tuning. This lets a joystick command guide the robot without supplying a precise whole-body motion at every step.

4. 45:47

   What the approach can and cannot scale

   The authors say the work demonstrates that scene-aware locomotion can be learned from video, rather than proving that internet-scale data already works. They tried internet videos but did not use them for training because reconstruction and curation were difficult; recording targeted clips was more reliable. They see richer depth input, more data, and improved reconstruction of dynamic motion as possible next steps.