Ep#22 DexWild: Dexterous Human Interactionsfor In-the-Wild Robot Policies
Mohan, a research engineer at Skild AI, and Tony, a CMU master’s student, discuss DexWild with the RoboPapers hosts, drawing on work with Deepak at CMU.
In this talk
- 1:29
Combining human and robot data
Tony describes the cost and difficulty of collecting teleoperation data across many locations. DexWild uses human demonstrations for scene generalization and task understanding, while robot demonstrations teach low-level control and object interaction. The team also starts from a pretrained vision transformer and reports that straightforward co-training works with separately normalized data.
- 6:26
A portable human data collection system
The setup combines a motion capture glove, two wrist cameras, and a third-person camera with an ArUco marker for wrist tracking. The third-person camera is used during collection, while deployment relies on wrist images and robot arm state. The researchers collected about 9,200 episodes across five tasks and more than 90 environments; the Manus glove was reported to cost about $6,000 per pair with education pricing.
- 31:33
Retargeting and training the policy
The policy takes images from both wrist cameras, arm and gripper state, and proprioceptive history, then predicts end-effector changes. Human hand motions are retargeted to match the robot hand’s action space, allowing the data to transfer across hand embodiments. The team found a 1:2 robot-to-human ratio per training batch worked well, while robot data remained especially important for reliable grasping.
- 34:14
Generalization, limits, and next steps
Human-only policies often approached objects and chose sensible motions but struggled with the final grasp; co-training improved results in scenes with human data and in entirely new scenes. The researchers say adding more human demonstrations improved performance, though they had not seen it plateau, and identify tactile sensing, better retargeting, recovery examples, and additional modalities as promising directions. They also discuss using simulation or data from other robots to reduce the need for robot-specific collection.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.