Ep#25: DexUMI: Using Human Hand as the Universal Manipulation Interface for Dexterous Manipulation

In this talk

1. 1:28

   Matching the exoskeleton to robot hands

   DexUMI uses a wearable exoskeleton designed around the target robot hand’s joint locations, limits, and link lengths. The team moved the thumb backward by nearly five centimeters to make the device easier to wear while preserving fingertip workspace. They built versions for the underactuated Inspire hand and fully actuated XHand, with many reusable components.

2. 10:08

   Closing the visual and tactile gaps

   Although the camera is mounted in the same position on the exoskeleton and robot, their appearances differ because a person wears the exoskeleton. The team replays recorded actions on the robot, uses SAM 2 to segment the hands, and applies flow-based inpainting to replace exoskeleton pixels while preserving visibility and occlusion. Training also uses image augmentations, while tactile readings from the thumb, index, and middle fingers are thresholded to reduce differences caused by stronger human contact.

3. 20:08

   Training and evaluating policies

   Policies take wrist-camera images and tactile readings as input. Across five tasks and two robot hands, the team reports an overall success rate of about 86%, without retargeting or additional robot operation data. They collected roughly 300 to 500 demonstrations per task, with longer tasks such as picking tea with tweezers requiring up to 500.

4. 24:06

   Comparing data collection methods

   In a tool-handling example, DexUMI took about seven seconds, compared with nearly 26 seconds for teleoperation, which required the operator to adjust their view. Mena said DexUMI offers direct fingertip force feedback and can provide data for direct deployment without extra robot data. The discussion also noted that teleoperation can capture different state and action relationships, while human-hand data may be useful for pretraining.

5. 43:51

   Future directions and open hardware

   The researchers are interested in co-designing a comfortable exoskeleton and a robot hand around it, rather than adapting people to existing hand designs. They also discussed adding cameras and making the wired setup more portable. The conversation closed on open-source hardware and the possibility of scaling data collection through contributions from many people.