Pathway to Generalist Robots: Scaling Law, Data Flywheel, and Humanlike Embodiment (CoRL 2023)
A researcher from the Robot Perception and Learning Lab presents at CoRL 2023 on building generalist robots through scalable algorithms, continuous data collection, and versatile humanoid hardware.
In this talk
- 0:00
A path from specialist to generalist robots
The goal is a robot that can build broad core abilities and keep gaining expertise in new tasks. The proposed route is to train generalist models on open-ended goals and large datasets, drawing a parallel with language models that learn from diverse text and support many applications.
- 3:49
Scaling skills and task interfaces
The research combines sensorimotor skills as reusable building blocks with multitask models intended to improve as data and compute grow. VILA uses object-focused visual inputs and a Transformer policy; it learned a coffee-making task from 50 demonstrations collected in about 30 minutes. MiMma unifies text and image prompts to select skills, and its tests found more than twice the success rate of baselines on the hardest generalization levels.
- 9:09
Learning through deployment
A data flywheel connects deployment, new experience, model updates, and broader use, but requires safe operation and efficient learning. In SERL, a person can correct a robot when monitoring detects it is stuck; the system saves those interactions, updates the policy in the background, and returns it for further use. In an insertion task, human interventions became less frequent by the third run.
- 13:35
Humanoid hardware for physical skills
The lab is developing a humanoid platform and a VR teleoperation interface to collect demonstrations, including for ramen preparation. The work also addresses balance during physical interaction with a real-time hierarchical controller that turns task commands into joint torques, supporting skill learning in simulation and on hardware.
- 15:37
Three pieces of the approach
The closing framework brings together algorithms that scale and compose skills, deployment data gathered through human-robot collaboration, and humanoid hardware for studying physical intelligence. The intended result is a robot that can improve through real-world use while people help manage imperfect behavior.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.