Ep#108: VLS: Steering Pretrained Robot Policies via Vision–Language Models

In this talk

1. 1:30

   Why pretrained policies struggle with new conditions

   Fine-tuning can teach a policy specific combinations of instructions and scenes, but it may fail on a new combination even when it already knows the motor skills. Sha uses stacking different numbers of cubes and then pressing a button as an example. The team focuses on shifts in vision, spatial layout, and language while keeping the underlying task within the policy’s capabilities.

2. 17:23

   How VLS builds guidance

   VLS first uses Gemini to identify relevant objects, Segment Anything to mask them, and DINO features to select useful points on objects. GPT-4o then writes stage-aware, differentiable PyTorch reward functions based on the scene, instruction, and key points. Those functions provide guidance for refining the policy’s proposed actions during denoising.

3. 26:21

   Steering action generation

   The method keeps a diverse batch of action proposals with an RBF repulsion step, applies reward gradients to steer proposals at each denoising step, then resamples using reward scores. The presenters describe gradient guidance as the main contributor: on CALVIN, removing it reduced performance from about 90% to 18%. Resampling helps favor stronger proposals, while diversity gives the system more options to compare.

4. 32:00

   Evaluation and practical limits

   The team evaluated VLS on LIBERO-Pro, LIBERO-Reasoning, CALVIN, and real-world tasks. They report top results against the compared policies and steering methods on the two LIBERO benchmarks, plus a 7.4-fold improvement over a base diffusion policy on mobile objects in CALVIN. They did not compare latency across methods; Sha says VLS is likely slower than DAGger, which can query its trained world model quickly.

5. 47:46

   What the approach may enable

   Sha describes VLS as training-free and usable with diffusion or flow-matching action policies, including VLA policies. The method aims to recombine skills a policy already has when task instructions or observations change. The discussion suggests similar steering could be explored with world action models or inverse dynamics approaches, though those extensions are possibilities rather than results shown here.