Jay, a research scientist at Google DeepMind, presents his perspective on robot generalization and foundation models at CoRL 2024 in a session chaired by Andrea Bicchi and Shaoan Wang.
Jay points to cheaper, more reliable hardware, progress in language and vision-language models, and investment interest as drivers of recent robotics progress. He argues that robots must work in unfamiliar settings because it is impractical to engineer for every case. Examples include Mobile ALOHA failures, RT-2 struggling with smaller apples, and lab demos affected by changes in lighting.
Jay describes robot foundation models as a promising route to generalizable control and argues that models and data should be developed together. PIVOT samples possible robot actions, represents them as image annotations, and iteratively asks a vision-language model to rank them. It can guide navigation and placing fruit in bowls, but spatial reasoning and fine-grained physical control remain limitations.
SpatialVLM adds spatial question-and-answer data to improve tasks such as estimating distances and deciding whether a robot can fit through a gap. Jay also describes using reinforcement learning in simulation to expand a handful of dexterous-task demonstrations into a larger dataset. An automatic curriculum selects informative starting states, and the team reports simulation data collection can be 100 times faster than teleoperation.
Jay argues that the signals connecting vision-language models to low-level policies affect generalization. RT-Trajectory uses paths drawn in image space as conditioning for a robot policy; the paths can come from people, videos, or foundation models. In the reported evaluation, RT-Trajectory generalized better than RT-1 and RT-2 across several tasks.
Jay says generalization remains unsolved and summarizes three priorities: extract existing capabilities from vision-language models, improve missing capabilities with targeted data, and find effective interfaces for actions. He estimates home humanoids may be five to ten years or more away, and says foundational problems still need substantial work. He favors continuing this research at Google DeepMind, citing its models, robots, simulation, computing resources, and team.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.
YC hosts discuss robot-use agents with Jaime and Vincent of Waddle Labs and Jay of RoboCurve, covering how general-purpose language models can control robots and how to make that control faster and more reliable.
