Ep#97: Cross-Embodiment Robot Manipulation via a Unified Hand Action Space

In this talk

1. 1:17

   Why share robot actions

   Manipulation policies are often tied to the robot they were trained on, which makes it hard to reuse behavior across grippers with different shapes and finger counts. The team wants one policy to work across these embodiments and potentially help combine their data for robot foundation models.

2. 9:35

   Build a shared space from geometry

   The team represents each hand with a sphere positioned and oriented using its palm and fingertips. Points on the hand map to spherical coordinates, creating correspondences across hands. Actions deform the sphere, and cascading inverse kinematics converts those changes into joint targets for each gripper.

3. 35:22

   Train and transfer across hands

   The team evaluated the method on MANO, LEAP, Allegro, and Shadow hands using an in-hand cube reposing task. A single policy trained on multiple hands achieved zero-shot transfer to a held-out Allegro hand, averaging 7.3 successful reposes. The researchers also found that training on multiple embodiments encouraged behavior that worked across their shared capabilities.

4. 48:40

   Test on physical robots

   The team deployed the controller on LEAP and Allegro hands, tracking a cube with a RealSense camera and AprilTags. On LEAP, the multi-hand model averaged 1.8 reposes, compared with 0.9 for zero-shot transfer and 2.0 for a model trained on LEAP; on Allegro, multi-hand and hand-specific models each averaged 2.1. Simulation-to-real transfer remained difficult because of pose estimation errors, contact simulation, and physical hand deformation.

5. 55:30

   Limits and next steps

   The researchers say a new gripper can be added from its URDF and required reference frames, with kinematic analysis handled by scripts. They demonstrated the method with a Wuji hand, but did not measure its success rate; its restricted thumb motion sometimes caused the hand to get stuck during cube manipulation. Future work could add more hands and explore force and touch information in the representation.