Spring 2022 GRASP SFI - Stefanie Tellex, Brown University

In this talk

1. 0:00

   Why robots need natural language

   Tellex describes robots moving into warehouses, cars, and everyday settings, while their controls still rely heavily on screens and phone apps. Her group aims to let people give robots flexible spoken instructions, but examples from a forklift study show that people describe the same task in many different ways.

2. 5:53

   A collaborative POMDP framework

   Tellex frames collaboration as a decision-making problem involving the robot’s physical world, the person’s goal, and noisy observations such as speech, camera images, and lidar. The robot may act in the world or ask questions to clarify what the person wants. Its perception should also respond to the task, such as attending to a speck of dust only when asked to find one.

3. 14:38

   Understanding commands at different levels

   In grid-world experiments, Tellex’s group studied both abstract goals and step-by-step movement commands, using hierarchical representations so one model could handle both. For commands that specify a route, such as passing through one room before reaching another, they used linear temporal logic and finite-state automata to preserve the relevant history without storing the full trajectory. Their English-to-LTL model reached just under 60% accuracy on a dataset of about 800 commands, showing both progress and a need for more data.

4. 29:08

   Generalizing language and learning from trajectories

   To handle unfamiliar landmarks, the group’s approach extracts a landmark phrase and matches it to map data, enabling transfer from Providence and Boston training data to a Tulsa simulation. Tellex also describes later work that learns language-to-LTL mappings from positive and negative trajectories without requiring people to annotate LTL expressions, reducing reliance on expert-provided meaning labels.

5. 39:33

   Using language to guide object search

   Tellex describes robots combining language with noisy sensor readings to search for objects, such as using a spoken list of possible rooms to guide where to look for mugs. Object-oriented POMCP factors the state by objects to make planning more efficient; in the reported comparison, it outperformed standard POMCP and random search, and performance improved when language gave more accurate location information. The group extended the approach to 3D and explored spatial descriptions such as an object being behind a building.