Ep#45: HERMES:

In this talk

1. 1:13

   Why human motion needs physical refinement

   Joan says robotics lacks the demonstration data available in vision and language research, while collecting dexterous teleoperation data is expensive. Human videos and motion capture offer a larger source, but hand-to-robot differences and imperfect pose extraction make direct retargeting unreliable. HERMES uses reinforcement learning to refine retargeted motion so it accounts for object interaction and physical constraints.

2. 6:01

   A shared reward for different tasks

   HERMES accepts teleoperation, motion capture, and video trajectories; video poses are extracted with FoundationPose and WiLoR. Its shared reward combines object tracking, contact and penalty terms, and an object-centric chain distance that aligns hand and object directions. The team reports gains over an object-tracking baseline, especially for tasks involving multiple objects or noisy trajectories.

3. 7:56

   Training and transfer to real robots

   The researchers use residual actions to refine reference motions, train with DrM in MuJoCo or PPO in MJX, then distill a state-based expert into a depth-image student using DAgger. They augment depth observations with clipping, Gaussian noise, and images from an in-the-wild dataset. The team says teleoperation trajectories are easiest to track and video-derived trajectories hardest; each task uses one trajectory rather than a mixture of sources.

4. 16:50

   Navigation meets manipulation

   The robot uses ViNT to navigate from RGB images, then applies closed-loop localization with EfficientLoFTR feature matching and PnP to align its pose with the manipulation target. The presenters report localization errors of roughly one to two centimeters, which they consider adequate for manipulation. In the store example, the robot navigates to a table, refines its position, and picks bottles.

5. 30:49

   Limits and next steps for sim-to-real

   The team attributes failures mainly to navigation errors and the perception gap between simulation and real scenes; video pose jitter and inaccurate collision models also make some tasks harder. They identify faster scene setup and better simulation of deformable objects as open challenges. They see simulation as a way to scale data collection, but note that asset creation, physical parameter tuning, and real-world transfer remain difficult.