Ep#73: VideoManip: Dexterous Manipulation Policies from RGB Human Videos via 3D Hand-Object Traject
Hongyi Chen, a PhD researcher at Carnegie Mellon’s Robotics Institute, explains VideoManip on RoboPapers with host Mike, showing how RGB human video can train dexterous robot manipulation.
In this talk
- 0:22
From human video to robot actions
Chen introduces VideoManip, which aims to train robot skills from ordinary RGB footage without teleoperation or wearable sensors. The pipeline estimates depth, object masks and geometry, hand shape, and object pose, then maps the human hand motion to a robot hand. It aligns the reconstruction with the table’s vertical direction because in-the-wild footage lacks a camera-to-robot calibration.
- 6:42
Fixing contact and expanding demonstrations
Reconstruction errors can leave gaps or collisions between the hand and object meshes. Contact optimization adjusts the estimated human hand pose to better match intended touch points, and a distance-based model learns grasps from the resulting hand-object relationships. For manipulation, the team varies object position and interpolates trajectories so one recorded example can yield many training demonstrations.
- 13:19
Results and failure cases
In grasping tests on 20 objects, 15 worked well or fairly well; five initially failed, often because of pose errors, occlusion, or unstable grasps. Adding two videos for each failed object improved results. Across seven manipulation tasks, Chen reports strong performance relative to Pi 0.5 and LVP, with no major drop between calibrated lab footage and in-the-wild footage; success still depends on reconstruction quality and the camera remaining mostly still.
- 23:54
Limits and next steps
Moving cameras, heavy occlusion, and inaccurate object poses make video reconstruction difficult, especially in cooking clips. The current system also lacks force feedback, can misapply contact during tasks such as turning a bulb, and may struggle when the robot hand differs in size or shape from a human hand. Chen’s proposed directions include handling moving cameras, broader simulation-based variation, larger video datasets, and combining visual learning with tactile or force information.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.