Mobile Manipulation Hackathon (IROS 2018) - Robotics.SG Team

In this talk

1. 0:09

   Set up the tray and map

   The team explains that Tiago needs an attachment to carry a tray of items left behind by customers. It recognizes items on the tray and uses a mapped convenience store to plan where they belong.

2. 1:00

   Identify, reach, and place items

   At the shelves, Tiago scans its tray, chooses an item, checks whether it can reach it, then attempts to pick and place it. The team describes unreachable items as a challenge and says the robot may need to move an item or retry when perception leads to a failed pick.

3. 3:30

   Rely on AI perception

   For this demonstration, the team relies on an AI network for object identity and pose rather than combining AI with model-based methods. Lighting or perception errors can make Tiago target empty space, so the system checks and retries. A bounding box and arrow on the monitor show the detected item and the planned approach.

4. 7:07

   Train object recognition

   The team says it used YOLO and labeled more than 15,000 images, often containing multiple objects, to identify items without markers. Tiago compares detections with its database of items and shelf locations to choose what to pick next.

5. 8:45

   Handle navigation and hardware limits

   Tiago navigates to another shelf for the soap, with objects covering the lower shelf area to help laser navigation detect the shelves. The team notes that gripper reach and hardware constraints can require moving an item before attempting a pick. By the demonstration's end, the team reports completing four of five items.