Edward Johns, director of Imperial College London’s Robot Learning Lab, and PhD student Vitalis Forlini discuss Instant Policy, their graph diffusion approach to in-context imitation learning, with RoboPapers hosts Michael and Chris.
Forlini introduces Instant Policy: a robot uses one or a few demonstrations to perform a task immediately, without task-specific training or network updates. The aim is to reduce the time and effort required by behavior cloning, which can depend on hundreds or thousands of demonstrations and lengthy training.
The method represents demonstrations, the robot’s current state, and future actions together in a graph built from segmented point clouds. Geometry-aware encoders preserve scene information, while relative positions between nodes support generalization across workspace locations. The model uses diffusion to iteratively produce actions, and demonstrations can be dense or subsampled to about 10 waypoints.
The network learns to remove noise from action nodes in the graph, a diffusion process used to predict actions at deployment. For training, the team generates consistent pseudo-demonstrations in simulation: they need to show semantically similar movements across objects, but do not need realistic physics. A real demonstration at deployment supplies the task and its dynamics; adding some realistic training data can further improve performance.
The model repeatedly consults the demonstrations during execution, shifting attention to relevant waypoints as the task progresses. When one demonstration leaves the intended behavior ambiguous, adding another can resolve it, as in the example of opening boxes of different sizes. The researchers identify precision, noisy point clouds, and fast or complex trajectories as challenges; they suggest more varied pseudo-demonstrations and closer or additional cameras as possible improvements.
Forlini describes transferring a demonstration from a tracked human hand to a robot gripper, and a language adapter that can convey simpler tasks without demonstrations. The team trained with one RTX 3080 for about five days and reported inference at roughly 20 to 30 Hz, depending on context length. They see longer tasks, better pseudo-demonstration distributions, dexterous manipulation, and tactile sensing as directions for further work.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.
Edward Johns, director of Imperial College London’s Robot Learning Lab, discusses robot manipulation, imitation learning, and AI with Robot Talk host Claire Asher.
