Ep#35: Reinforcement Learning with Action Chunking

In this talk

1. 1:06

   Why use action chunks

   An action-chunking policy predicts a short sequence of actions from the current state instead of choosing one action at a time. In imitation learning, chunks can preserve coherent behaviors, such as reaching for and grasping a mug. The researchers ask whether chunked policies can also improve online reinforcement learning.

2. 8:56

   What makes chunked RL work

   A straightforward attempt to apply online RL to chunks performed poorly. The authors say chunked RL needs both an expressive policy, such as a flow or diffusion policy, and a behavior constraint that keeps its actions close to patterns in the offline data. Their Q-chunking method trains a chunk-level critic and samples candidate chunks from the behavior policy, selecting the highest-valued candidate.

3. 25:32

   Online exploration results

   The team evaluated the method on OGBench manipulation tasks, including puzzles, drawer and block sequences, and stacking two to four cubes. Chunking brought little extra benefit on easier tasks where offline data already supported learning, but it improved online learning on harder tasks, especially cube quadruple. They also report benefits on three RoboMimic tasks, both offline and online.

4. 41:37

   Benefits and limitations

   The authors point to two gains: more coherent online exploration and faster value backup from learning a critic over action chunks. An ablation suggests that an n-step return without a chunk policy and critic did not reproduce those gains and often underperformed. Chunking can reduce reactivity on dynamic tasks, and the experiments did not use real robots; the authors also note the high action dimension and millions of environment steps in their experiments.

5. 44:51

   Next research directions

   The guests identify real-robot trials, comparisons with skill-based methods, and more realistic simulations as useful next steps. They also want policies that can choose how much of a chunk to execute before replanning. Improving sample efficiency, including how update-to-data ratio affects learning, remains an open question.