π0: A Foundation Model for Robotics with Sergey Levine - 719
Sergey Levine, a UC Berkeley professor and Physical Intelligence co-founder, discusses the π0 robotics model with Sam Charrington on The TWIML AI Podcast.
Justin Johnson, co-founder of World Labs and associate professor at the University of Michigan, discusses world models with Sam Charrington on The TWIML AI Podcast.
Johnson says researchers use “world model” for several distinct ideas: implicit knowledge of the real world, models that predict state changes in reinforcement learning, and generative systems that create navigable environments. He favors clearer terminology because these are all valuable research directions, but they are not the same kind of system. He also distinguishes generating realistic observations from building theories that explain how the world works.
Johnson explains the POMDP framework as an agent acting on a world, receiving partial observations, and responding as the world changes. The state is an abstraction suited to the task, and a learned internal representation can behave like a state even when the true physical state is inaccessible. A robot trained by behavior cloning can operate in this loop without being trained with a reinforcement learning reward.
Johnson contrasts explicit 3D representations such as Gaussian splats with models that generate pixels directly. World Labs’ Marble turns text, images, or video into splat-based worlds, while its RTFM research model generates interactive frames without an explicit 3D representation. Marble first generates a 360-degree panorama, then lifts it into 3D; a single image can anchor the visible scene while the model fills in unseen areas.
Johnson groups world models by what they produce in the agent-world loop: renderers output observations, planners output actions, and simulators model how states change. Marble spans rendering and simulation because its splats can be viewed and manipulated as a 3D state. He expects these capabilities to increasingly share a model, with different operating modes for tasks such as controlling a robot, generating a virtual world, or exploring counterfactuals.
Johnson expects transformers to remain useful, while training objectives and methods for handling long contexts continue to evolve. World models may need hundreds of thousands or millions of tokens to represent video or 3D space. Tokens could correspond to video patches, groups of splats, spatial chunks, or abstract latent states; he advises choosing representations that fit the task and remain simple enough to scale.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.
Sergey Levine, a UC Berkeley professor and Physical Intelligence co-founder, discusses the π0 robotics model with Sam Charrington on The TWIML AI Podcast.