Ep#39: MolmoAct: An Action Reasoning Model that reasons in 3D space

In this talk

1. 2:13

   Why add spatial reasoning

   The team presents MolmoAct as a robot policy that predicts depth, a visual trajectory trace, and then actions. The hosts show it completing a breakfast task despite unfamiliar studio lighting, with the model predicting actions in chunks of eight steps. The authors argue that spatial traces can make a robot’s plan easier to understand and guide than language alone.

2. 10:24

   Training data and model pipeline

   MolmoAct is trained on robot demonstrations, multimodal web data, and auxiliary depth and trajectory tasks. Its depth maps are generated with Depth Anything V2 and encoded into 128 discrete tokens; trajectories are represented by five sampled image points. The team describes filtering Open X-Embodiment data and collecting about 22 to 25 hours of curated mobile and household demonstrations, with multiple camera views.

3. 40:37

   Steering and real-world evaluation

   The authors report adapting MolmoAct to single-arm and bimanual Franka tasks, evaluating 25 trials per task against OpenVLA and π₀ Fast. Users can steer the robot by drawing a 2D line over the image; in one evaluation, visual traces achieved about 0.75 average task progression versus about 0.42 for changed language instructions and 0.13 for π₀. The team says depth conditioning can improve grasp success even when the predicted trace looks the same.

4. 1:07:51

   Limitations and next steps

   The authors note that MolmoAct predicts many tokens autoregressively, which makes it slower than some alternatives, and suggest reasoning only when needed as one path to faster systems. They describe the depth tokens and visual traces as an initial approach, with 3D trajectories, richer planning, and better dexterous skills as possible directions. They also point to open evaluation and transparent reporting as useful for future work.