Ep#92: Human Universal Grasping

In this talk

1. 1:14

   Learning grasps from human data

   Kevin Ma introduces HUG, a system trained on human grasp data rather than simulation or teleoperation. Its dataset, 1 Million HUGs, contains about one million frames collected with Meta Aria Gen 2 glasses. The model predicts a human grasp from an RGB-D image and a user click, then retargets that grasp to a robot hand.

2. 6:49

   Building the dataset and model

   The team fits MANO hand models to Aria’s tracked landmarks and uses head movement to pair hand labels from grasp frames with earlier images that show the object without a hand. HUG combines DINOv2 image features with a cropped point cloud and uses flow matching to predict hand pose and wrist position. The team notes that hand-tracking errors create label noise, especially when the hand is occluded.

3. 34:37

   Benchmarking grasp quality

   HUG-Bench pairs real objects with metric-scale simulated meshes, including 30 purchasable test objects. In simulation, HUG achieved about 73% success on the test set, compared with about 90% for the human-grasp oracle. The model’s 3D loss and point-cloud crop both mattered: removing the 3D loss lowered success by more than 30 percentage points, while removing the crop lowered it by more than 10.

4. 50:34

   Testing robots and discussing limits

   The team tested HUG with different robot hands, arms, and cameras. It achieved about 66.7% success across 300 tabletop trials and about 62% in a rented home, where the team ran all trials in one morning without tuning the environment. The main limitations include open-loop execution, imperfect retargeting, noisy labels, and fixed right-hand shape; the speakers discuss how feedback, force sensing, or motion planning could address some failures.