Ep#114: Oopsie Data
Arpit, a UT Austin PhD student, Klaas, a UT Austin postdoc, and Paul, a Berkeley student, discuss Oopsie Data with the RoboPapers hosts and guests, covering how to collect and use robot failures across research labs.
In this talk
- 1:37
Why collect robot failures
Klaas describes how a failed ALOHA ball grasp prompted the team to ask how learning systems could benefit from mistakes. He found few large datasets of real robot failures and argued that collecting diverse examples requires contributions from many labs.
- 4:37
What makes data useful
Near misses can show a policy which action to avoid at a critical moment. The team also counts repeated attempts that eventually succeed and successes with side effects, such as knocking over another object, as useful data. They chose to collect broadly and study later which failures help policies, reward models, or world models.
- 12:51
Early contributions and data quality
After the first submission deadline, the team had about 800 hours of data from 40 labs, spanning multiple robots and settings; Franka arms were common, with robotic hand data among the less typical examples. Much of the data came from policy rollouts, while human teleoperation was a smaller share. The team is checking and relabeling submissions because labs used inconsistent names and recording practices.
- 24:53
Making recording and training practical
Next steps include cleaning the dataset, testing how contributors can use it, and developing a training format and data loader. Mixed action spaces, missing camera information, coordinate conventions, and action chunks complicate reuse across labs. The team says recording tools should fit into existing evaluation scripts so labs can save data with little extra effort.
- 30:54
Sustaining shared research and learning from failures
The guests discuss how shared datasets can establish the value of saving evaluation and play data, even if future work moves beyond a single project. Oopsie Data could support data driven failure taxonomies and analysis of human interventions, including DAgger style corrections. The team is also planning a CoRL workshop in Austin with breakout discussions and awards for notable failures.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.