Ep#24 CLONE: Closed-Loop Whole-Body Humanoid Teleoperation for Long-Horizon Tasks

In this talk

1. 5:02

   Why whole-body teleoperation matters

   The researchers want humanoids to handle tasks at different heights, such as squatting to reach objects on the floor. They argue that current demonstrations often emphasize locomotion or scripted actions, while home and warehouse work requires interaction with objects throughout a 3D space. They see a shortage of whole-body interaction data as a barrier to training those capabilities.

2. 21:47

   CLONE combines motion control and localization

   CLONE uses an Apple Vision Pro to track the operator’s hands and head, providing three points to guide the robot’s 29 degrees of freedom. A teacher policy trained in simulation uses motion tracking and an adversarial motion prior; a distilled student policy with a mixture of experts is used for deployment. Global localization combines headset tracking with robot odometry from lidar, RGB-D, and an IMU to correct drift during extended operation.

3. 23:11

   Demonstrations and data limits

   The demos include squatting, walking while crouched, wrist movements, boxing, table tennis, handovers, and warehouse-style tasks. The team reports lower tracking error with a mixture-of-experts policy and better upper-body tracking when training with its CLONE motion data. That data included about 20 newly captured motion sequences, roughly a minute each, alongside about 130 AMASS sequences; the researchers distinguish this controller-training data from the much larger interactive datasets needed for autonomous policies.

4. 48:33

   Practical constraints and next steps

   The researchers see changing operating height as a potential advantage of humanoids over wheeled robots, but say terrain flexibility, including stair use, still needs development. They have not tested high-force tasks such as opening heavy doors, and note that the G1’s shoulder and elbow motors can overheat during extended operation. Future directions include more flexible control modes, larger models, dynamic tasks, and real-world data collection; operators also need to adapt to headset delay, especially for precise manipulation.