RoboBoom
GRASP Lab· November 6, 2026· 1:12:31

Fall 2026 GRASP on Robotics - James Anderson, Columbia University, “How to Control Your JEPA”

James Anderson, a control and robotics researcher at Columbia University, explains how to make JEPA world models more useful for robot control at GRASP Lab.

In this talk

  1. 1:45

    Why world models need better control

    Anderson contrasts strong robot performance in structured tasks, such as doing dishes, with the difficulty of handling unpredictable settings like homes and roads. He frames world models as a way to connect perception with trajectory planning, using compact latent representations instead of familiar physical state variables.

  2. 13:50

    How JEPA predicts in latent space

    A JEPA uses an encoder to compress observations and a predictor to estimate the next representation from the current one and an action. Training compares predictions with encoded future observations in latent space, rather than reconstructing pixels. Anderson uses a frozen DINOv2 vision encoder in the first set of experiments.

  3. 21:11

    Bisimulation filters visual distractions

    Anderson describes how a model can focus on easy-to-predict but irrelevant details, such as an unchanged maze background, making control fragile when the background changes. His group adds a trainable representation head and a bisimulation loss to bring together states with similar task dynamics and separate states with different dynamics. In maze and PushT simulations, the method stayed more robust to background changes than DINO-based baselines, with about 7% average degradation on PushT versus as much as 70% for DINO in some cases.

  4. 46:12

    Inverse dynamics preserves unstable modes

    For systems such as cart-pole and bipedal robots, an encoder may omit unstable modes that a controller needs to stabilize. Anderson says SigReg can achieve zero training loss while missing those modes, and proposes adding an inverse-dynamics objective that predicts which actions produced future states. In simulations, the approach captured control-relevant dynamics and stabilized cart-pole and a bipedal robot where comparison methods failed.

  5. 53:03

    Limits, assumptions, and next steps

    In the discussion, Anderson says reward prediction added little in the reported vision-control experiments because other regularization already prevented representation collapse. He notes that the methods rely on assumptions such as detectability and adequate controllability, and that combining the bisimulation and inverse-dynamics ideas would help when both background changes and unstable dynamics matter. Applying the approach to physical robots was ongoing work with a student visiting Aaron Ames's lab at Caltech.

Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.