Robotics Seminar: Danfei Xu

In this talk

1. 1:18

   From specialized dexterity to physical reasoning

   Xu contrasts Betty the crow’s ability to solve an unfamiliar tool-use puzzle with robots trained on thousands of demonstrations for a specific task. He argues that language and vision models can produce plausible action sequences but often lack a grounded understanding of physical constraints.

2. 10:04

   Generative planning through skill chaining

   Xu proposes generative task and motion planning, which samples plans from learned distributions instead of searching a large space. In his first approach, separate short-horizon skill models are chained with diffusion so information can flow forward and backward across a plan. Tests include tool use and block arrangement, with reported success rates around 60 to 70 percent on one task; the task sequence is supplied in advance.

3. 40:14

   Factor graphs connect skills and constraints

   Generative factor chaining extends the method from linear skill chains to factor graphs, which can represent temporal and spatial constraints. Xu shows examples including two-arm handovers and hammering, combining learned skill models to produce plans for new arrangements. The training examples are generated with scripted or reinforcement-learning policies, rather than human demonstrations.

4. 49:56

   Stitching trajectories without a task plan

   Xu’s third approach uses one diffusion model trained on short trajectories and chains trajectory chunks without a supplied task plan. It conditions each chunk on noisy samples from its neighbors instead of averaging score functions, which can create unlikely intermediate paths. Maze navigation tests show that chunk count affects the result: too few chunks can leave gaps, while excess chunks can cause unnecessary movement.

5. 58:16

   What foundation models still need

   Xu says foundation models contain broad semantic knowledge, but argues that robots need representations of physical goals, beliefs, and constraints that people do not usually express in language. He points to unpredictable human actions in egocentric video as evidence that visible observations and explicit reasoning traces may not capture the reasoning behind behavior.