Ep#52: Probe, Learn, Distill: Self-improving Vision-Language-Action Models
Wenley, a Cornell University PhD student who worked on the research during an internship at NVIDIA’s Gear Lab, discusses PLD with the RoboPapers hosts and audience: a method for improving vision-language-action policies through real-world reinforcement learning, trajectory collection, and distillation.
In this talk
- 3:09
PLD’s three-stage approach
PLD combines residual reinforcement learning, hybrid trajectory collection, and supervised fine-tuning. The residual policy adds corrections to a base VLA without changing its policy architecture, which lets the method work across different models. The goal is to improve task performance through real-world practice, then distill the resulting data back into the VLA.
- 5:09
Why probe before distilling
Wenley says human demonstrations can be imperfect, while RL rollouts tend to cluster around optimal trajectories and may leave the distilled policy unable to recover from unfamiliar states. PLD first uses the base policy to reach random but recoverable states, then switches to the RL policy. This yields trajectories that combine recovery examples with successful behavior.
- 11:07
How residual RL works in practice
The team warms up a critic with base-policy rollouts, then uses RLPD and SAC with online and offline buffers. Successful offline trajectories are retained, and each training batch draws half its samples from each buffer. On the GPU insertion task, training ran for about eight hours at 20 Hz, roughly 500,000 steps; the residual is regularized to make small corrections when needed.
- 15:05
Generalization and the role of the base model
On LIBERO benchmarks, Wenley reports that PL data can outperform human data on unseen tasks, which he attributes to its recovery examples. Training on a small set of related tasks also improved performance on other tasks as the PL dataset grew. The method depends on a capable base policy: for GPU insertion, Wenley estimated it needed around 80 to 90 percent success before residual RL could help.
- 36:23
Where RL fits in longer tasks
Wenley suggests decomposing long-horizon tasks and applying RL to the difficult, precise parts, while the VLA handles easier actions and task sequencing. For example, the GPU policy learned insertion, while moving the GPU between the motherboard and table worked with 20 human trajectories and did not need RL. He also notes that a person still monitored training and could stop dangerous behavior.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.