YC hosts discuss robot-use agents with Jaime and Vincent of Waddle Labs and Jay of RoboCurve, covering how general-purpose language models can control robots and how to make that control faster and more reliable.
Jay describes RT-2 as a language model fine-tuned to output robot end-effector poses. The guests argue that newer models can use tools, write code, and explore an environment in context, giving them more flexibility than directly predicting actions. They suggest that broad pretraining data may help robotics even when it is not robot-specific.
François compares in-context learning with updating model weights: examples can improve performance quickly, but gains may saturate after roughly 20 to 40 examples and depend on the model’s training and context limits. The group discusses turning lessons from a task into reusable programs or skills for future agents. They also consider smaller, faster models specialized by a stronger model and its harness.
A demonstration shows Astra controlling arms to move a block into a bowl using camera feeds and repeated pose commands. The guests describe using code or compiled skills for repetitive steps, while vision-language models handle variation, such as detecting objects or responding to failure. They say model latency was a bottleneck and cite an estimate that it was improving by about two times per month.
The guests connect the Platonic Representation Hypothesis with the idea that stronger general-purpose models may develop representations useful for robotics. They suggest computer-use and CAD data can teach spatial concepts such as top, down, left, and right. A robot interface designed to resemble familiar computer tools may also make it easier for a language model to control a robot.
A guest predicts that robots capable of following natural-language instructions across unfamiliar tasks and environments could arrive within about two years. The discussion identifies slow model calls and the cost of keeping a large model in the loop as practical challenges. One proposed direction is to collect task experience, then compress it into faster policies, updated models, and an organized library of skills.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.
Jay, a research scientist at Google DeepMind, presents his perspective on robot generalization and foundation models at CoRL 2024 in a session chaired by Andrea Bicchi and Shaoan Wang.
