Ep#63: NovaFlow: Zero-Shot Manipulation via Actionable Flow from Generated Videos

In this talk

1. 1:59

   Why use a video-to-action framework

   Ju says robot demonstrations are costly to collect, while tasks can fall outside the training distribution. NovaFlow separates task understanding from execution and uses 3D point flow as the task representation, which can describe rigid, articulated, and deformable motion in scene coordinates.

2. 14:36

   From generated video to robot actions

   The pipeline generates a human-hand video from an RGB image and task instruction, estimates depth, calibrates it using the initial RGB-D image, then tracks 3D points to produce actionable flow. The team generates multiple videos in parallel because video models struggle with robot morphology and contact. A vision-language model ranks candidate flows to filter out hallucinations, tracking failures, and grounding errors.

3. 23:36

   Planning and experimental results

   For rigid objects, NovaFlow combines NVIDIA GraspGen proposals with trajectory optimization that follows the flow while accounting for smoothness, collisions, and joint limits. For deformable objects, it uses a particle model and MPC to track the flow; the rope example shows a mismatch attributed to model error. Across six tasks, the team compares against zero-shot AVDC and VidBot and demonstration-trained policies, and reports that 2D flow, limited training coverage, and generated-to-real video differences constrain those baselines.

4. 40:46

   Extending to multistep tasks and faster execution

   The follow-up work NovaPlan uses a VLM to break longer tasks into subtasks, verify each step, and generate recovery actions after failures. For precision tasks, it tracks human hand poses and converts them into gripper motion; one example uses a generated finger poke to recover a misplaced block. The team says its runtime improved from about 730 seconds for NovaFlow to about 40 seconds for NovaPlan, with video generation still a major bottleneck.

5. 54:33

   Limits and future directions

   The researchers say grasp generation is a current bottleneck, while deformable-object planning depends on having a useful dynamics model and reward. They see potential in action-conditioned world models and in using real execution video for closed-loop failure detection and recovery. For longer tasks, they suggest hierarchical decomposition into subtasks or improved long-video generation.