Weiyu Liu on Grounded Semantic Reasoning for Robotic Interaction | Toronto AIR Seminar

In this talk

1. 1:24

   Grounding semantic knowledge in sensor data

   Liu argues that robots need semantic representations to connect common-sense knowledge with sensorimotor data such as point clouds and object poses. Because those data differ greatly in modality and detail, his work uses reasoning to link specific observations with higher-level concepts and generalize to new objects and tasks.

2. 6:00

   Reasoning about object properties and locations

   Liu introduces a representation of relationships among multiple object properties, including class, material, cleanliness, and location. A dataset of 1,457 situated object instances supports a Transformer model that predicts missing properties. On object-search tasks, the model approached human-level performance for retrieving objects from users’ top three likely locations and ran more than 150 times faster at inference than the Markov Logic Network baseline.

3. 15:34

   Selecting task-suitable grasps

   Liu distinguishes semantic grasping from choosing a grasp solely for stability: a robot should select a grasp suited to the object’s state, material, affordances, and intended task. His SG-14,000 dataset captures varied grasp contexts, while TaskGrasp includes more than 250,000 task-oriented grasps and semantic relations between objects and tasks. Experiments suggest that structured knowledge helps generalize to new object categories and tasks.

4. 26:53

   Rearranging objects from language goals

   Liu presents StructFormer, which selects objects from language instructions and predicts their placements in structures such as circles, lines, towers, and table settings. Trained on more than 100,000 simulated rearrangement sequences using over 300 ShapeNet models, the system was evaluated on novel objects and scenes. Liu reports that it also rearranged novel real-world objects on a robot without fine-tuning.

5. 36:10

   Limits and future directions

   In the discussion, Liu explains that rotational errors can be high for symmetric objects, where ground-truth orientation is ambiguous. He says the rearrangement instructions in this work are structured and describes natural language as future work. Other directions include accounting for force and physical properties, improving fine-grained 3D object representations, and using semantic predictions to choose safe exploratory actions.