Ep#34: RoboArena

In this talk

1. 0:41

   Why robot policies need a shared evaluation

   Praveen says generalist policies need testing across varied scenes and language-commanded tasks, while one-off evaluations are difficult to compare and scale. Kyle adds that shared evaluation may be more practical now that generalist policies can work across different environments.

2. 3:22

   How RoboArena compares policies

   Policy developers host their models remotely, and evaluators receive two policies at random to test back to back on a task and scene they choose. Evaluators report preferences, and the system aggregates those pairwise results into a leaderboard. The initial benchmark uses the DROID robot setup and dataset, while the comparison and ranking approach could work across platforms.

3. 8:43

   Handling varied tasks and feedback

   The project embraces variation in real-world scenes rather than requiring identical conditions across every evaluation; each pair is compared in the same setting within roughly two minutes. Evaluators score task progress and explain their preference using factors such as success, speed, smoothness, and safety. The team adapted the Bradley-Terry ranking model to account for task variability and compared its rankings with exhaustive evaluations of all policies on each task.

4. 23:08

   What the results can and cannot tell developers

   Developers can review rollout videos and evaluator comments, which the team used with an LLM to summarize policy strengths and weaknesses with links to supporting examples. The benchmark is intended for holistic comparisons, not precise scientific ablations. Evaluations and rollouts are public, and evaluators do not know which policies they are testing.

5. 29:24

   Growing the benchmark

   Remote policy hosting shifts compute costs to policy authors and allows developers without a robot to submit models, though latency may limit future dynamic tasks. The team hopes papers will report RoboArena results and more labs will evaluate policies; its CoRL challenge offered free evaluation credits. Results update hourly, and the discussion estimates that around 100 pairwise comparisons, often three or four days of evaluation, can provide a confident ranking.