Ep#16 TWIST: Teleoperated Whole-Body Imitation System

In this talk

1. 0:20

   Why whole-body imitation matters

   Yanji says human motion could help humanoids handle complex tasks, but earlier whole-body teleoperation systems showed limited coordination. TWIST aims to map human movement across the body so a robot can combine actions such as kicking while carrying something or closing a door while walking backward.

2. 5:46

   Data collection and retargeting

   TWIST combines about 50,000 motion clips, or 60 hours, from AMASS and OMOMO with 150 clips of lab motion capture. Yanji says a mock-up head and suit provide accurate whole-body tracking, while separate offline and online retargeting stages balance motion quality with timely feedback. Motion capture improves accuracy but adds setup cost and complexity.

3. 11:05

   Training a robust motion tracker

   The student policy tracks the current retargeted motion, while a teacher in simulation can use future motion and privileged information such as gravity and friction. The team samples future motion sparsely, up to 20 steps across two seconds. Yanji says reward-based learning with behavior cloning generalizes better than action-only DAgger on motion capture and real-world data.

4. 17:26

   Capabilities and practical limits

   Yanji says retargeting latency was later reduced from about 0.5 seconds to around 0.1 seconds or less. Demonstrations include deep squats, kicking, and carrying a box while nudging an object with a foot. High motor torque can cause overheating, and limited balancing data means one-foot balance remains difficult.

5. 27:08

   Operator experience and next steps

   Yanji describes challenges with judging grip force and occlusion in third-person teleoperation, while egocentric views can be restricted by robot camera placement. He says the motion capture suit is uncomfortable and hopes to make teleoperation less painful. Future directions include more varied human motion data, video-based pose tracking, and improved operator views.