2023 Summer Robotics Colloquium: Tyler Westenbroek (UT Austin)

In this talk

1. 0:58

   Why combine control and reinforcement learning

   Westenbroek says robots are hard to deploy reliably because their dynamics and interactions with the world are difficult to model, and their behavior can be unstable. He proposes using feedback control to bring structure from approximate physics models into reinforcement learning, while using real-world data to address what those models miss.

2. 12:12

   Shaping rewards with stability information

   Westenbroek explains how Control Lyapunov Functions, or CLFs, encode a route toward a stable goal as a function that a controller can reduce. Adding a model-based CLF term to the learning objective can shorten the effective planning horizon and reduce the real-world data needed. In a quadruped example, the approach produced useful tracking after three minutes of data; a learned potential also helped a robot perform a swing-up after one real-world trajectory.

3. 37:38

   Using feedback to improve policy gradients

   The second part examines policy gradients for unstable systems, where sensitivity can cause high variance, slow updates, and compounding errors from approximate models. Westenbroek describes a hierarchical architecture in which a learned planner sets a desired trajectory and a low-level controller tracks it. With real-world trajectories and model-based sensitivity estimates, the approach improved control of an RC car and a quadruped using only a few minutes of data.

4. 49:27

   Extending the approach to perception and intent

   Westenbroek identifies perception and human-robot interaction as next challenges. Robots need useful world models for contact, such as estimates of ground conditions and contact forces, and systems need to translate human goals into reliable actions. He suggests that feedback-based architectures could provide interfaces between higher-level intent and low-level control.