RoboBoom
Toronto Machine Learning Society (TMLS)· August 11, 2026· 41:32

Reasoning Robots: Open World Navigation and Memory for Agentic Robots

A Toronto Machine Learning Society presenter describes agentic reasoning, open-world navigation and long-term memory for robots, using Trailbot and warehouse examples in a talk delivered at a conference.

In this talk

  1. 0:10

    Robots that follow natural language

    The presenter frames useful robots as systems that can understand instructions and their surroundings. A Trailbot snack-dispenser project showed the idea in practice: students could ask for chips, and the robot would deliver them before continuing along a trail.

  2. 3:23

    Open-world navigation

    The first research effort extends vision-and-language navigation from scanned homes to outdoor scenes, with tasks such as finding a house on the right or locating a trash can. The system combines image segmentation and captions with LiDAR-based 3D maps, then uses an LLM to update a value map and a conventional planner to choose a route. It works zero-shot across indoor and outdoor settings, while direct LLM route suggestions perform poorly because the models struggle with spatial understanding.

  3. 11:12

    OmniMemory for longer tasks

    The STAR work adds memory for tasks that span larger environments and longer periods. OmniMemory stores a hierarchical 3D scene graph, searchable video captions, and key frames; an agent retrieves relevant evidence through text, time, position, or distance queries. In a warehouse example, it matched Digital Twin boxes to a shelf with an available slot, and it found an EQP-labeled bottle by revisiting key frames after an initial search missed the label.

  4. 20:12

    Results and current limits

    On NavQA, memory improved performance over earlier reasoning methods; on a 200-question warehouse set developed with Clearpath Robotics, the system reached about 67%. Retrieval cost stayed roughly steady across memories ranging from 36 seconds to 35 minutes, and the team estimated a reasonably sized SSD could hold a day of operations. Object detection, spatial relations, and visual counting remain difficult, and real-time mapping runs onboard while many language-model calls rely on an offboard server.

  5. 24:20

    What comes next

    The presenter describes future work on changing environments, including boxes that appear or disappear and people moving through warehouses, as well as memory that prioritizes meaningful events. The team also plans deployments with warehouse operators and hopes to develop generalized task schemas from successes and failures. For now, the system focuses on individual finite tasks, though the presenter says ongoing missions could be built from repeated tasks and monitoring.

Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.