2025 Winter Robotics Colloquium: Xiaolong Wang (UC San Diego)

In this talk

1. 0:47

   Why humanoid robots and layered control

   Wang opens with a home demonstration of a Unitree robot handling toys, which highlighted how bulky legged robots can be in narrow spaces. He argues that humanoid form can help in tighter environments, and describes a control strategy where vision-language models suggest directions or rough routes while specialized controllers handle precise movement.

2. 9:50

   Teleoperation and learning from people

   Wang describes Open-TeleVision, which streams a robot’s first-person camera view to a VR headset and maps the operator’s hand movements to the robot. Remote control worked across roughly 10,000 kilometers, with latency around 0.3 seconds. Because robot demonstrations are costly and limited, his group also trained on human recordings, including about 30,000 demonstrations gathered over three weeks; mixing these with robot data improved performance at object locations absent from robot training.

3. 46:54

   Humanoid motion and whole-body skills

   Wang presents work on teleoperating humanoids while they walk, then discusses motion imitation using the CMU motion-capture dataset. His team selects a workable range of motions from that dataset before training reinforcement-learning controllers, helping the policy handle more poses. He notes that coordinating walking with accurate upper-body manipulation remains an open challenge.

4. 53:14

   Navigation guided by language

   Wang introduces NaVILA, which pairs a vision-language model with a low-level locomotion controller. The model turns instructions into intermediate commands, such as moving forward a specified distance or turning by an angle, and can run on a desktop while the robot receives commands through a phone. Training on navigation examples alongside language and video data lets the model retain visual question-answering abilities.

5. 1:02:22

   Scaling data and closing gaps

   In discussion, Wang identifies broader generalization as an ongoing challenge and points to more diverse human recordings, real robot data, and simulation as ways to improve it. He says smaller robots have shown a narrower sim-to-real gap in his group’s work. For vision inputs, he sees RGB as practical at scale and suggests models could infer depth instead of relying only on depth sensors.