Fall 2022 GRASP SFI - Sonia Chernova, Georgia Institute of Technology

In this talk

1. 0:47

   Semantics for household robots

   Chernova frames everyday human spaces as difficult robotics environments because people, objects, and preferences vary. She describes semantics as turning sensor data into more abstract concepts, which can help robots reason about objects and communicate with people.

2. 6:24

   Grasping with object and task context

   Chernova’s group trains robots to choose grasps based on both the task and an object’s condition, such as whether a cup is hot or a spatula is empty. Their CAGE approach ranks candidate grasps using object identity, functional parts, material, state, and task information. The team created a dataset of 14,000 labeled semantic grasps across five object classes; Chernova notes that its labels reflect human preferences.

3. 18:42

   Inferring object states and likely locations

   To infer properties such as cleanliness, the group models how features co-occur, since a cup’s location can provide clues about its state. Its LINK dataset contains about 1,457 object instances, and a transformer predicts missing scene details from partial information. In a scavenger hunt, the algorithm found target objects in 88% of cases by the third attempt, compared with 91% for people.

4. 30:11

   Anticipating needs from object movement

   Chernova describes a proactive assistant that learns household routines by tracking object locations over time, rather than monitoring people’s activities. A graph neural network uses time and the current scene to predict future object locations, so a robot might bring out a tennis bag or breakfast items. The group simulated varied routines across 40 days in two households and identified continual learning as an open goal.

5. 41:16

   Explaining robot failures

   The group studies how robots can explain manipulation failures in language that nontechnical users can act on. Ranked semantic scene graphs focus explanations on relevant causes, such as an object being blocked by a pitcher or too close to cups. In a user study, these explanations helped participants identify failures and fixes with success rates above 90%.