2025 Winter Robotics Colloquium: Katsu Ikeuchi (Principal Researcher at Microsoft)

In this talk

1. 3:03

   From demonstrations to robot instructions

   Ikeuchi describes Learning from Observation 2.0, which uses GPT to interpret a demonstration and produce a task model. The system identifies both the action and its relevant parameters, then represents the instructions as a behavior tree. Robot skills execute the tree, with simulation used to check whether a plan is safe to run on hardware.

2. 10:42

   A reusable library of robot skills

   A task system selects and parameterizes skills from libraries for grasping and manipulation. Ikeuchi says the same task model and skill library can work across different robot setups, though changing the gripper may require updates to the grasp skills. The skills use reinforcement learning, with demonstration supplying an initial direction and force feedback helping them handle errors.

3. 15:12

   Designing skills around contact and grasp purpose

   Ikeuchi groups grasps by their purpose: securing an object, allowing active adjustment, or constraining it while permitting some motion, as when holding a door handle. For manipulation, he represents tasks as changes between face-contact states. His team narrows possible transitions to a household skill set that includes actions such as picking, placing, opening drawers, and wiping.

4. 35:14

   Combining skills into longer tasks

   For extended tasks, Ikeuchi combines learned skills into reusable task sequences and uses GPT to assemble them into behavior trees. A symbolic map supports navigation, while GPT can handle decision points such as determining whether a table is dirty. He shows a sequence involving navigation, careful carrying, pouring, and releasing an object.

5. 39:58

   Why separate planning from execution

   Ikeuchi frames learning from observation as selecting useful parts of a human action rather than reproducing its full trajectory. GPT-based planning handles task structure, while hardware-specific learned skills carry out the motions. In the discussion, he says skill libraries need consistent inputs and reward functions, and notes that demonstrations can provide initial conditions for reinforcement learning.