Ep#44: From Pixels to Predicates: Learning Symbolic World Models via Pretrained Vision-Language Mod

In this talk

1. 1:07

   Learning a world model from demonstrations

   Kumar’s method learns transition dynamics rather than a policy. It aims to let a robot use search to combine familiar skills in new environments and solve similar tasks from a specified goal.

2. 6:51

   Proposing and selecting predicates

   Predicates are binary concepts such as whether an object is an eraser or is on the floor. A vision-language model proposes candidate concepts from skill-labeled demonstrations and labels them against before-and-after images. A search-based procedure selects a compact set of predicates and learns operators that describe how skills change the world.

3. 26:17

   Testing transfer and replanning

   The evaluation includes two real-robot environments and three environments from earlier work, with held-out tasks that involve more objects, steps, or obstructions. In a cleanup demo, a robot uses learned concepts and operators in a different MIT room to throw away a can, retrieve an item from a container, and wipe a table. It also checks whether each expected change occurred and replans when the table becomes dirty again.

4. 39:59

   Limits, debugging, and next steps

   For a juicing task, the system learned about 12 operators from hundreds of candidate predicates and millions of possible operators, then planned toward a goal of putting juice in a cup. Kumar describes the learning search as nondifferentiable and sensitive to demonstrations and tuning. Because the learned predicates are written in natural language, researchers can inspect errors and decide whether to adjust the demonstrations or regularization.

5. 47:30

   Combining symbolic search with learned skills

   Kumar says it remains an open question how symbolic planning compares with increasingly capable vision-language-action models. He suggests using vision-language models to recognize concepts and search to compare possible plans. Future directions include learning from large human video datasets and combining hierarchical planning with low-level robot skills.