Robots bootstrapped through learning from Experience

In this talk

1. 0:06

   Use prior experience to learn faster

   Learning from scratch can be slow, while supervision may not be available. The Experience Project applies structural bootstrapping: combining what a robot already knows with new observations to fill gaps about objects, actions, and planning.

2. 1:36

   Plan and execute a table-setting task

   ARMAR uses a spoken request to form a planning goal, checks objects and locations against its memory, then creates and executes a plan. For grasping, it chooses a viewing direction using pose uncertainty and uses visual servoing; it checks that a placement spot is clear before releasing an object.

3. 4:08

   Find or replace missing objects

   When an item is missing, ARMAR can look for alternatives using visual shape and affordance features, or common-sense affordances learned from text. For example, it may suggest juice instead of soda because both can be used for drinking, with the human confirming the substitution. It also updates its location knowledge from experience and can ask a person where an item is.

4. 8:06

   Adapt learned actions to new tasks

   When ARMAR cannot reach a whisk, it asks a person to hand it over, then adapts a previously demonstrated wiping action to stirring. Its wiping motion is represented as a periodic movement primitive that can be adjusted for different surfaces, positions, and speeds, with force feedback guiding execution.

5. 10:28

   Apply experience to object handling and movement

   ARMAR estimates where it can approach a chair using its reachability map, then pushes it toward a target while avoiding obstacles. It also learns typical object locations over time and can use those patterns to find items and complete tasks.