World Models & General Intuition: Khosla's largest bet since LLMs & OpenAI

In this talk

1. 0:34

   From game clips to action models

   The host describes Medal’s retroactive game recorder and its 3.8 billion clips, which capture moments players choose to save. Py says the footage and action labels let GI train models to map visual input to actions while preserving individual users’ privacy. The company’s approach grew from the idea of predicting actions from a large, diverse collection of interactive data.

2. 2:43

   Agents learn from human play

   Py demos a vision-based agent that predicts actions from game frames, including an early version that can recover after getting stuck with four seconds of memory. Later versions play in real time against people and bots, using imitation learning without game-state access or reinforcement learning. Py says training on selected highlights includes examples of unusually strong play as well as mistakes.

3. 9:13

   Interactive world models and transfer

   GI’s world models generate what happens next in response to actions, rather than simply predicting likely video frames. Py highlights spatial memory, rapid camera movement, and continuity through smoke using only a short video context. He says models trained in games can transfer to real-world video, and argues that interactive game footage provides useful action and visual information for spatial reasoning.

4. 50:30

   From game bots to broader uses

   GI is working with game developers and engine companies on replacing deterministic player controllers with models that take frames and return actions. Py says better bots can keep games populated when few human players are online, and describes work with robotics and manufacturing where systems can use game-controller-style inputs. GI plans to offer an API and custom policies, while aiming to move some pretraining work into post-training for customers.

5. 58:52

   A path from imitation to simulation

   Py describes a goal of making saved clips playable inside world models, which could let researchers train agents on events such as crashes using their recorded lead-up actions. He says this could connect imitation learning to reinforcement learning at the scale of Medal’s clip library. For 2030, he envisions GI models supporting a large share of AI-driven interactions between physical objects, with simulation as an early market.