2026 Fall Robotics Colloquium: Tal Daniel (Carnegie Mellon University)

In this talk

1. 1:32

   Why visual tokens should represent objects

   Daniel argues that recent gains in vision-language models point to better spatial representations, including object-centered tokens for boxes and segmentations. Unlike fixed image patches, which can split objects across tokens, these representations aim to capture meaningful entities without requiring people to define every object in advance.

2. 10:08

   Deep Latent Particles

   Deep Latent Particles (DLP) encode an image as a set of entities, each with a position, scale, transparency, depth ordering, and learned appearance features. A differentiable decoder reconstructs the scene from these particles and a background representation. Daniel says the local, set-based design can represent more objects at test time than appeared in training, provided the model has enough particles.

3. 29:44

   Object structure for learning robot skills

   Daniel describes using particle representations for reinforcement learning and imitation learning, including policies that must handle more objects than they saw during training. In experiments with multi-cube manipulation, unstructured patch representations struggled as object counts increased. He also presents world action models that predict states alongside actions, which helped with scaling to more objects and compositional generalization.

4. 40:57

   Stochastic world models from particles

   Daniel frames a world model as a dynamics model that predicts possible next states from the current state and conditioning such as an action or language instruction. His LPWM framework learns latent actions for each particle, allowing different entities to have distinct stochastic behavior. In video prediction experiments, the particle model handled object interactions better than a patch-based baseline, whose predictions became blurry during interactions.

5. 57:40

   Extending the approach to 3D and exploration

   Daniel notes that object-centric models still have limits with complex textures and in-the-wild data. His group is extending the approach to voxel-based 3D-DLP and to 3D embeddings learned from multiple 2D camera views; the latter improved results by an average of 20% to 30% on MimicGen and RLBench tasks. Ongoing work explores per-entity intrinsic motivation for discovering interactions and applies the representation to dexterous hands.