Language-Driven Learning for Interactive Robotics–Siddharth Karamcheti (Stanford University)
Siddharth Karamcheti, a Stanford researcher advised by Dorsa Sadigh and Percy Liang, presents his work on language-driven interactive robotics at a Paul G. Allen School colloquium hosted by Siddharth Srinivasa.
In this talk
- 3:29
Language as a bridge for visual understanding
Karamcheti describes Voltron, which combines language-conditioned image reconstruction with caption generation to learn both fine visual details and broader scene meaning. On referring-expression detection, its representations outperformed prior approaches while using 500,000 images, compared with more than 400 million used to train CLIP.
- 19:31
Mapping vision-language model design
Prismatic is an open toolkit and evaluation suite for testing choices such as language model size, image encoders, and image preprocessing. Combining DINOv2 and SigLIP features improved fine-grained detection by more than 10% on average; controlled comparisons also showed Prismatic models outperforming base LLaVA models with the same data and model capacity.
- 26:11
From vision-language models to robot control
Karamcheti explains how vision-language models can provide a foundation for policies that turn instructions into robot actions. OpenVLA, trained using Open X-Embodiment data, generalized across robot embodiments and could be adapted to new tasks with as few as 70 demonstrations, though fine-tuning still took several hours.
- 30:23
Teaching robots during collaboration
Vocal Sandbox uses language models to translate instructions into interpretable programs, then lets people teach new behaviors or identify objects through conversation and targeted feedback. In an eight-person study, users built more complex behaviors over time and rated the system more predictable and helpful than two baselines; later, they could focus on writing cards while the robot packed gift bags.
- 39:05
A path toward useful, reliable robots
Karamcheti’s future research agenda includes more reactive manipulation using force and touch, plus interactive robot policies that can take human corrections during deployment. He argues for gradually expanding robot skills through focused task curricula, and for making systems transparent enough that people can understand and verify their plans.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.