RI Seminar: Leslie Kaelbling : The Role of Rationality in Modern Robotics

In this talk

1. 2:26

   Rationality as a design principle

   Kaelbling wants robots that can handle varied tasks, learn efficiently, and remain robust and safe in real homes. She argues that useful learning needs structure: models of objects and their relationships, local action effects, beliefs about the world, and utilities that represent what users want.

2. 9:08

   Object-based world models enable planning

   Kaelbling favors representations that support efficient reasoning and incremental learning, including object-centered models and sparse causal action models. Tip-Top combines image and language input with 3D perception and CuTAMP planning to perform tasks such as sorting hardware and moving a Coke can before placing coffee pods on a tray. She says the system can transfer to another robot with its URDF and camera calibration, though its current execution is open-loop.

3. 21:37

   Demonstrations reveal goals and action structure

   What a robot can learn from a demonstration depends on what it already knows. Kaelbling’s group uses a vision-language model to suggest possible explanations of a person’s intent, then evaluates them with a planner; examples include sorting fruit and utensils and arranging objects by color. In separate work, demonstrations of tool use are segmented into contact-mode changes, then refined through simulation to generalize across tools.

4. 32:14

   Learn causal models from skills and predicates

   Kaelbling describes learning causal action models from demonstrations of low-level skills, using predicates to capture changes in the world. A vision-language model proposes candidate predicates, while data and planning help select useful ones. The approach lets Spot combine learned actions to wipe a table or operate a juicer, without relearning basic skills such as picking up and placing objects.

5. 45:10

   Robots need memory, belief planning, and practice

   Kaelbling emphasizes partial observability: robots need to search for objects, improve their views, and plan around uncertainty. She also describes a robot that practices important skills during downtime, using planning to reset the scene and gather useful experience; after three hours of autonomous practice, Spot improved at placing a ring on a difficult table. She identifies persistent memory, re-identification, and uncertainty handling as open needs for robots working across rooms and over long periods.