Introducing NVIDIA Cosmos 3: The Open Model That Thinks, Generates, and Acts

In this talk

1. 0:01

   Physical AI needs scalable data

   Real-world data is difficult to scale because physical environments are unpredictable. The presentation frames compute as a way to produce the data physical AI needs.

2. 0:14

   How Cosmos combines modalities

   Cosmos is described as an open model built with a mixture of transformer types. It takes in visual, action, audio, and language inputs; one transformer reasons and plans, while another generates future content.

3. 0:35

   Adapt the model to different roles

   Developers can post-train Cosmos for different embodiments and tasks. In a vision-language role, it interprets scenes, explains events, and identifies relevant details.

4. 0:50

   Generate worlds and train policies

   Cosmos can create synthetic video from images, text, or existing video, with the goal of representing physical behavior accurately. It can also simulate environments for policy training and evaluation, and predict future frames from actions.

5. 1:13

   From perception to robot actions

   Further post-training turns Cosmos into a model that perceives, reasons, plans, and produces actions for different kinds of robots. Computer-generated data also serves as a training signal for physical AI.