Ep#95: Action-to-Action Flow Matching

In this talk

1. 1:09

   Using recent robot feedback as the starting point

   Jingtao introduces Action-to-Action Flow Matching as an alternative to generating each action from Gaussian noise through many denoising steps. The method uses measured robot feedback from the recent action history as a closer starting distribution for predicting the next action chunk. In the reported setup, the history is eight time steps, one fifth the length of the future chunk.

2. 5:01

   Fast inference and a trade-off in diversity

   The researchers report that one inference step can reach a 90% success rate after fewer training epochs, with latency around one millisecond in some conditions. They also describe tests with altered backgrounds, lighting, and camera angles, plus an unseen color-changing block where the method exceeded 80% success. The discussion acknowledges that using recent actions makes the policy more deterministic and can reduce its ability to represent multiple valid trajectories.

3. 15:11

   Adding multimodality when the task calls for it

   Jianfei describes MARS Policy as a follow-up intended to balance A2A’s speed with the ability to choose among different actions. It adds noise where the model identifies a need for multimodality, such as a choice point during a cup-picking task, while using more deterministic behavior elsewhere. The researchers say this adaptive behavior is learned through a self-supervised loss and can apply at multiple points in longer tasks.

4. 28:12

   Smoother trajectories and future directions

   The guests discuss representing robot trajectories with polynomial coefficients, which can encode a segment compactly and provide velocity signals for the low-level controller. They report smoother movement and roughly twice the speed of A2A for this approach, and say it may help policies run on onboard hardware, though that deployment still needs validation. Their planned work includes adapting action chunk length to the task, with longer predictions for predictable motion and shorter ones for contact-rich situations.