Ming-Yu Liu, who leads research at NVIDIA, discusses how Cosmos 3 combines language, video, audio, and action for physical AI with Tim on Machine Learning Street Talk.
Cosmos 3 can take text, video, audio, and action as input. Liu describes using it to answer questions about a scene, then turning the process around to generate scenes and build closed-loop simulations, including for self-driving tests.
Liu outlines a system that connects a vision encoder to a language model, then uses the resulting vision-language model to initialize a video and action generator. The reasoning component predicts language token by token, while the generator uses diffusion to produce coherent video, action, and audio; action is added during mid-training.
Liu distinguishes forward dynamics, which predicts what follows an action, inverse dynamics, which infers an action from a visual change, and policy, which selects actions to reach a goal. He says training these together encourages shared learning because each task links what a robot sees with what it does. Human egocentric video may also help robot learning because manipulation patterns can transfer across embodiments.
Liu sees neural simulators becoming useful first for comparing policy checkpoints, before they are reliable enough to generate policy training data. A simulator need not reproduce reality perfectly if its policy rankings track real-world results. He acknowledges that policies may exploit simulators and says preventing such exploits remains an open research problem.
Liu says Cosmos can support robotics with better data, environments, and starting points, and cites state-of-the-art pick-and-place results after post-training on the DROID dataset. Cosmos comes in Super, Nano, and Edge sizes: Edge is designed for local devices such as Jetson Thor, Orin, and DGX Spark, avoiding dependence on a data-center connection. Liu says models, code, and some datasets are available through NVIDIA's Cosmos GitHub repository and Hugging Face.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.
Ming-Yu Liu, who leads NVIDIA’s Cosmos Lab, explains how Cosmos models and NVIDIA’s computing platform can support robotics and other physical AI applications at the Fellows Forum.
