Cordelia Schmid: Transformers for Vision-Language Navigation and Manipulation

In this talk

1. 1:03

   Transformers for navigation

   Schmid describes robots following flexible spoken instructions, which requires interpreting language, understanding the surroundings, and planning actions. Her first model combines image views from multiple time steps with an instruction, using a hierarchy of vision and temporal transformers to retain navigation history. It outperformed earlier methods across several navigation benchmarks, with pretraining and added training tasks improving results.

2. 7:05

   Maps for unfamiliar environments

   To help robots remember places they have already explored, Schmid presents two map approaches. One builds a topological graph for backtracking and route planning, while a newer method keeps a compact 5-by-5 implicit map that can be updated as the robot moves. The compact map performed well in simulated object-finding tasks and transferred to real rooms after fine-tuning on 35 demonstrations, reaching 60% success across seven object categories.

3. 18:13

   Language-guided manipulation

   For tasks such as stacking cups, Schmid’s transformer uses language and images from multiple camera views to predict the next position as a heat map. On selected RLBench tasks, the approach improved overall performance by 20% over PerAct; history helped most on longer tasks, while multiple views aided precision and occlusion handling. Training with 100 real demonstrations also enabled real-robot trials, though performance fell on some unfamiliar instructions and scenes.

4. 26:35

   3D perception and remaining challenges

   Schmid’s PerAct approach combines colored point clouds from multiple cameras with language to predict robot positions. It outperformed HiveFormer in the tested settings, and removing distracting background elements helped, especially with noisy real-world depth data. She notes that robots still struggle to recover from unfamiliar failures, and points to better grasp prediction, motion planning, robust perception, and language grounding as open needs.