Ep#103: Freeform Preference Learning for Robotic Manipulation

In this talk

1. 1:04

   Why single preferences are ambiguous

   The researchers aim to improve low-performing robot policies without relying on dense, hand-designed rewards. A single A-or-B preference can be hard to judge when each trajectory has different strengths, such as placement accuracy, speed, or safety. FPL lets annotators compare trajectories across multiple axes using natural language.

2. 6:44

   Learning rewards from freeform feedback

   FPL uses a pretrained vision-language model to predict a continuous reward for each frame, conditioned on the trajectory and a language-described axis. The reward model learns from preferences for each axis using a Bradley-Terry approach. Researchers collected about 300 trajectory pairs per task, yielding around 1,500 labels and an average of roughly 40 distinct axis descriptions.

3. 10:57

   Conditioning policies on reward

   The team trains a pretrained vision-language-action policy on demonstrations, conditioned on the task and predicted rewards. Across real-robot tasks including folding shorts, plating toast, and setting a table, FPL outperformed the discussed success/failure and single-preference baselines. For toast plating, annotators could label spatula use as hygienic, then request that behavior through reward conditioning.

4. 19:01

   Composing behaviors and limits

   In simulation, a policy conditioned on speed and target location could place an object quickly on a peg even though it had not seen fast demonstrations for that peg. The researchers also showed reward curves that often rose when relevant objects were placed, suggesting improved credit assignment. They noted open challenges: inconsistent language labels, conflicting preferences, choosing reward levels at test time, and the need for broader evaluation.

5. 25:07

   Scaling feedback for longer tasks

   The interface lets annotators provide several labels while reviewing a trajectory once, reducing time per label, though total annotation time is longer than for a single preference. The team used about 20 frames per trajectory and relied on the model to connect whole-episode judgments to relevant moments. The discussion highlighted crowdsourcing, personalization, and longer-horizon tasks such as cleaning a kitchen as areas for future work.