Ep#25: DexUMI: Using Human Hand as the Universal Manipulation Interface for Dexterous Manipulation
Mena, a PhD student, and Han, an incoming Stanford PhD student, discuss DexUMI, their framework for collecting dexterous manipulation demonstrations with a human-worn exoskeleton, on RoboPapers with the episode hosts.
In this talk
- 1:28
An exoskeleton matched to robot hands
DexUMI uses an exoskeleton designed around a target robot hand’s joint locations, limits, and link lengths. The team moved the palm back nearly five centimeters to make it easier to wear while preserving fingertip workspace. Their devices support the Inspire and XHand designs, with several reusable components.
- 4:10
Capturing demonstrations and matching robot visuals
An iPhone records hand trajectories with ARKit, while a wrist-mounted fisheye camera records interactions and encoders capture finger motion. To reduce the visual gap between exoskeleton and robot, the team replays recorded actions on the robot, segments both hands with SAM 2, and uses flow-based inpainting to replace the exoskeleton with robot-hand imagery. Tactile readings from three fingers also serve as policy input; thresholding converts them to contact or no contact.
- 21:06
Deployment results and data collection trade-offs
The team reports about 86% success across five tasks on the ByHand and XHand, without retargeting or additional robot teleoperation data. They collected roughly 300 to 500 demonstrations per task, with longer tasks requiring more. In one comparison, DexUMI took about seven seconds to perform a task, close to the bare hand and faster than the roughly 26 seconds for teleoperation; it also provides fingertip force feedback.
- 26:43
Where each kind of data fits
Mena explains that DexUMI demonstrations keep state and action closely aligned, while teleoperation commands can differ from the robot’s current state and may include overshoot. The hosts and researchers discuss using broad sources of human data for pretraining and higher-quality DexUMI or teleoperation data for post-training. ARKit drift may accumulate on long recordings, but the team says relative actions should limit its effect on policies.
- 43:51
Future directions: comfort, portability, and co-design
The researchers propose designing a comfortable exoskeleton first and then designing a robot hand around it, which they call co-design. They also discuss adding cameras, improving fit across hand sizes, and replacing cables with compact electronics for portability. The conversation closes with a case for open-source hardware and broader data collection.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.