Stanford University and Google Brain Professor Chelsea Finn presents her RI Seminar at Carnegie Mellon Robotics Institute on scaling data and algorithms for robot learning, introduced by host Wen.
Finn describes robots learning manipulation through trial and error, including placing a block in a sorter and lifting objects with a spatula. Those skills worked in the training setup but failed with a different spatula, bowl, or tablecloth. She argues that broad generalization calls for training with varied experience, rather than starting each task from scratch.
Finn contrasts narrow robot datasets with ImageNet’s range of objects, tasks, environments, viewpoints, and lighting. RoboNet pooled 15 million video frames from seven robot platforms across four institutions, with 113 camera viewpoints. She also describes guided exploration: a few minutes of human-provided example images helped a robot interact more with a target drawer than novelty seeking alone.
Finn presents image-based dynamics models that predict future camera views from current images and planned actions, allowing robots to learn from data without task rewards. A single model could support different goals, such as lifting a stuffed animal, covering an object with a towel, or sweeping objects into a dustpan. Pretraining on RoboNet improved results on new robots in several evaluations, though some results suggested the models underfit the diverse data.
To address underfitting, Finn explores goal-conditioned representations and training video prediction models layer by layer to reduce memory demands. The larger greedy-trained models improved prediction metrics across RoboNet, KITTI, and Human3.6M, and planning success on 100 unseen objects rose from about 50% to roughly 88–90%. She also describes using human videos in robot reinforcement learning by aligning domains, inferring actions, and assigning reward to successful final states; this improved performance over standard reinforcement learning.
Finn concludes that broader robot generalization depends on collecting varied data and building algorithms that can use it without underfitting. In the discussion, she identifies task specification, dataset annotations, and matching human videos to robot experience as unresolved challenges. She suggests that modest amounts of paired data may help establish correspondences, while fully unsupervised matching depends on the data.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.
Chelsea Finn of Physical Intelligence discusses building reliable, general-purpose robots, drawing on office coffee service, factory box assembly, and laundry-folding examples.
Chelsea Finn, cofounder of Physical Intelligence, presents at a Y Combinator event on building reliable, general-purpose robots for real-world work.
Chelsea Finn, co-founder of Physical Intelligence, discusses building general-purpose robots at a Y Combinator event.
Chelsea Finn, co-founder of Physical Intelligence and Stanford associate professor, discusses building general-purpose robots with the No Priors hosts.
Stanford AI and robotics professor Chelsea Finn talks with host Peter on The Robot Brains Podcast about helping AI systems adapt to changing conditions, with examples from language models and robots.
Chelsea Finn, a Stanford professor, discusses how robots can generalize through broader training data and adapt to unfamiliar settings at a CMU Robotics Institute seminar.
