Jana Kosecka - Finding Objects in the Era of Large Vision and Language Models
Jana Kosecka, a computer vision and robotics researcher at George Mason University, discusses how vision-language models can help robots find and use objects, speaking at an IEEE Robotics and Automation Society event.
In this talk
- 0:48
Object finding as a robotics capability
Kosecka frames object localization as useful for navigation, manipulation, and service tasks such as fetching and delivery. In a table-setting example, a language model can describe the steps, but a robot still needs to identify the objects and determine where to grasp and place them.
- 2:56
How supervision shaped vision models
She traces computer vision training from image-level labels such as ImageNet, through bounding-box detection, to segmentation masks. More detailed annotation took greater effort and produced smaller datasets; earlier robotics systems typically covered only around 10 to 30 object and non-object categories.
- 5:56
From image captions to open-vocabulary models
Self-supervised methods and Transformers enabled training on large image and language collections, while CLIP aligned images with captions and supported zero-shot classification through text prompts. Kosecka notes that caption-trained models struggled with spatial relations and compositional meanings. She describes how curated labels and segmentation data, including about one billion masks for Segment Anything, helped improve grounding and open-vocabulary detection.
- 13:40
Adapting models for robot tasks
Newer vision-language systems connect visual representations to generative language models, allowing people to specify tasks in natural language. Kosecka cautions that zero-shot performance may be inadequate for robotics and points to visual instruction tuning and robotics-specific data as ways to adapt models. She also describes using object co-occurrence knowledge to score navigation frontiers when a target is out of view.
- 19:27
Evaluation, interaction, and measurement
Kosecka sees opportunities for natural-language instructions and human feedback, especially in defined environments, while stressing the need for high-quality robotics data and better evaluation methods. In response to a question about measurement, she says quantitative information can be incorporated into these models when the training data fits the environment.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.