Computer vision researcher Cordelia Schmid presents her work on multimodal video understanding and vision-language-guided robotics at CoRL 2024.
Schmid describes a model that predicts action timestamps and captions together, using visual input and speech. Pretrained on 15 million YT-Temporal-1B videos with ASR-derived event boundaries, it outperforms prior approaches on several captioning benchmarks. She notes that small objects and missing visual grounding remain challenges.
Schmid presents a multi-stage approach that parses a question, grounds relevant events in the video, and reasons over those events to produce an answer. Focusing on the relevant time and location can improve accuracy and make the evidence behind an answer easier to inspect. She points to long videos and linking people or places across moments as important open problems.
For indoor navigation, Schmid combines language instructions with visual observations and action history. A topological graph lets the agent remember visited places and backtrack, while a small recursive implicit map supports navigation in continuous environments. The implicit-map approach improves benchmark performance and was adapted to navigate to objects in unseen real environments using additional demonstrations.
Schmid introduces GenBench, which tests manipulation across novel placements, objects, categories, actions, and long-horizon tasks. Her approach uses an LLM to break instructions into action steps, vision-language models to locate objects, and 3D point-cloud policies to plan motions. Planning helps on harder tasks, but inaccurate object grounding remains a major failure source; precise localization is essential for grasping.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.
Cordelia Schmid, research director at Inria and a researcher at Google Research, discusses transformer methods for language-guided robot navigation and manipulation at NAVER LABS Europe.
