Spring 2026 GRASP SFI - Jiafei Duan, University of Washington
Jiafei Duan, a robotics researcher whose university affiliation is given as the University of Washington, presents at GRASP Lab on building robotics foundation models that reason before acting.
In this talk
- 0:02
Why robots need reasoning
Duan argues that robots must move beyond teleoperated demonstrations and the structured tasks of factory assembly to work reliably in homes. He describes three gaps in current models: spatial reasoning, memory of past events, and recognizing and recovering from failures.
- 8:26
Grounding spatial instructions
Duan presents RoboPoint, which maps language instructions to 2D points in an image, including relations such as placing an object between two plates. Its training data is generated in simulated homes using 3D assets and spatial annotations. He says the model also points to locations in settings outside its kitchen training data, including a carpool lane.
- 16:10
Reasoning traces before action
Duan describes MoMa as a robotics model that predicts a depth representation and a visual trajectory before generating actions. To train it, his team filtered a large robot dataset down to 190,000 trajectories and added about 10,000 in-house trajectories, along with simulated depth and visual reasoning annotations. He reports that reasoning tokens improved performance, while their fine-grained action tokenization made inference slower.
- 28:54
Giving manipulation policies memory
Duan explains that behavior-cloning policies often treat each observation independently, which can make them forget earlier actions. SENTO combines a multitask manipulation policy with compact memory across camera views. In a task where a robot had to remember where a cube began, the memory-enabled system reached nearly 100% success, compared with about 35% without memory.
- 39:35
Learning from failures and shaping future work
Duan presents AHA, which uses simulated failure cases to train a vision-language model to detect failures and explain their causes. On the reported benchmark, AHA reached nearly 80% success on binary failure detection, while GPT-4o was close to 50%. He identifies world models, scalable data collection, harder evaluations, and additional forms of physical reasoning as directions for future research.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.