Ep#9: AutoEval - Autonomous Evaluation of Generalist Robot Manipulation Policies in the Real World

In this talk

1. 1:37

   Why automate real-world evaluation

   Evaluating generalist policies takes repeated trials across many skills, with each rollout requiring time and access to a physical robot. AutoEval aims to automate outcome detection and scene resets so researchers can submit evaluations remotely and leave the robot running.

2. 4:11

   How AutoEval runs evaluations

   The demo shows two Berkeley WidowX arms with live video, job submission, status tracking, and public reports containing success rates, videos, rollout duration, joint effort, and safety checks. Users host their policy as a server that receives images and language instructions and returns actions for the robot.

3. 12:13

   Resets, outcome checks, and safety

   AutoEval uses a reset policy fine-tuned from OpenVLA with about 50 demonstrations, plus classifiers that judge task outcomes. Paul says reset failures were the main failure mode in a review of 50 runs. Safety measures include workspace limits, force-aware stopping, joint-health monitoring, and human intervention when needed.

4. 23:57

   Results and practical limits

   The team compared OpenVLA, SuSIE, Octo, π₀, and MiniVLA, finding that different policies did better on different tasks. AutoEval scores correlated strongly with human evaluations, with a reported Pearson coefficient of 0.99. A WidowX setup takes about three to five hours to prepare, while adapting to a new robot can take days; the system also still needs occasional human intervention.

5. 38:14

   A distributed benchmark for robotics

   Paul envisions AutoEval stations at multiple institutions, where researchers could send policies to varied real-world scenes. The hosts discuss how differences in hardware, camera angle, and scene setup may change results, and argue that shared evaluations could improve reproducibility and allow more trials than the usual 10.