Jason Ma, a robotics researcher at the University of Pennsylvania who worked on the paper at Google DeepMind, joins RoboPapers hosts Michael and Chris to discuss using vision-language models to assess robot task progress and data quality.
Ma describes prompting a vision-language model with a task description and robot video frames to estimate progress from 0 to 100, without training a new model. The team evaluated the approach on Open X-Embodiment and internal datasets, across varied tasks and robot platforms.
When frames stay in chronological order, models such as Gemini 1.5 Pro and GPT-4o may mistake the passage of time for progress and produce steadily rising scores. Shuffling the frames encourages the model to judge each image, while the original first frame anchors direction for tasks that can be reversed. Ma also proposes that a trajectory’s recoverable ordering can help indicate whether it shows meaningful action.
Ma introduces Value Order Correlation, or VOC, which compares the order of frames ranked by predicted progress with their actual time order. Expert demonstrations tend to score positively, while the tested failure trajectories had broadly scattered scores. The measure offers an offline way to evaluate progress estimates and sort trajectories without running reinforcement learning for every task.
Providing shuffled, labeled examples in the model’s context improved estimates on the more challenging ALOHA 250 tasks; performance rose from weakly positive correlation with no examples to roughly 0.6–0.7 with five examples. Examples could come from a person or from other robot tasks, with smaller gains reported for those settings. Ma suggests that a robot could reuse experience from earlier tasks to help assess a new one.
The team used trajectory scores to compare datasets and select higher-quality demonstrations for behavioral cloning. Ma says human-collected sets such as RT-1, Dobb-E, and Bridge scored well, while some autonomous or poorly framed data scored low; he points to DROID videos that sometimes failed to capture the manipulation. In a reported experiment, filtering to a cleaner subset improved results over random selection with the same data amount, though the discussion notes that overly strict filtering can leave too few examples.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.
Jason Ma, Dyna Robotics co-founder and chief scientist, joins Latent Space on site to discuss Dyna-2, robot learning, hardware, and commercial deployments.
Jason Ma, co-founder and AI research lead at Dyna Robotics, presents Dyna 2’s approach to robot learning and commercial deployment at the Fellows Forum on Day 2.
Jang Ren, a Peking University pre-PhD researcher collaborating with Galbot, Tsinghua University, SG2, Calia, and NVIDIA, discusses LDA-1B and its approach to training robot action models from mixed embodied data with RoboPapers hosts Chris and Michael.
Seonghoon, a PhD student at KAIST and research intern at NVIDIA, explains DreamZero, a world action model for zero-shot robot policies, in a Robo Papers conversation with host Michael.
Jason Ma, cofounder of Dyna Robotics, discusses building reliable robot foundation models for real-world manipulation at the Humanoids Summit.
Arhan, a PhD student at the University of Washington, and Carl, a robotics researcher at Physical Intelligence, discuss PolaRiS and how it uses real-to-simulation methods to evaluate robot policies with RoboPapers’ Chris and other guests.
