Ep#043: Attention-based map encoding for learning generalized legged locomotion

In this talk

1. 1:04

   Why end-to-end locomotion needs to generalize

   Jang describes the limits of model-based planning: it can generalize across terrain, but requires tuning, CPU-heavy training, and reliable maps and models. Earlier end-to-end policies could traverse stepping stones and balance beams, but often failed to transfer to different terrain without task-specific training.

2. 7:22

   Using attention to select footholds

   The method encodes a terrain map and uses attention to select useful foothold points based on the robot’s command and joint observations. A CNN adds local features to map points, then 16 attention heads retrieve relevant features for an action-generating network. Jang says this structure acts like a learned planner and helps the controller generalize when terrain inputs change.

3. 17:15

   Results on varied terrain and hardware

   The same training pipeline was used for quadrupeds and humanoids, with demonstrations on stepping stones, beams, and difficult outdoor terrain. The policy handled unseen scenes, recovered from slips and disturbances, and adjusted speed from joystick commands. The deployed model runs in about one millisecond on a CPU, while mapping runs at around 50 Hz.

4. 30:10

   Deployment limits and lessons

   Jang identifies accurate odometry and clean maps as practical requirements, noting that depth cameras can produce artifacts and sensor layouts can cause occlusion. For sim-to-real transfer, he emphasizes matching inertia, contact, and actuation. He argues that direct training with the robot’s available observations can avoid the performance gap introduced by a privileged teacher.

5. 49:55

   Next steps for locomotion

   Jang says upcoming work explores parkour across varied terrain, including humanoid versions with and without upper-body involvement. He also wants to develop a 3D representation and higher-level planning that can make use of controllers able to traverse complex terrain.