Harold Soh, a robotics researcher from Singapore, discusses a proposed foundation model for human-robot interaction at an IEEE Robotics and Automation Society event.
Soh describes his lab’s work on tactile perception, robot learning, and social intelligence, including models of trust in human-robot collaboration. He proposes that an HRI foundation model should help robots interact with people by representing what those people may do next.
Soh uses an experience with his son Aaron to show how a useful mental model of another person can change how we respond. He suggests robots should build representations of people’s goals, intentions, beliefs, desires, fears, and emotions from sensor data, then use those representations to predict behavior.
Soh proposes HIP, or Human Interaction Pre-training, as a building block inspired by CLIP. It would map robot signals such as visual frames and language into a joint representation predictive of agent behavior. Researchers could use it in learning systems, planning, or memory, either out of the box or after fine-tuning.
Soh identifies synthetic psychology and cognitive science data, annotated online video, and newly collected robot interaction data as potential sources. He favors structured latent-variable models that predict future states through self-supervised learning, while representing different agents’ beliefs, intents, and goals. Evaluation should measure prediction of human behavior and latent states, as well as usefulness and integration in downstream tasks such as social navigation.
In response to questions, Soh says GPT-3.5 showed promise for predicting beliefs and feelings from language, but results were sensitive to prompts and lacked grounding in gestures and other environmental cues. He agrees that video-based learning needs careful annotation: dataset quality and diversity matter, and raw video alone may not reveal what people know or intend.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.
Harold Soh, who leads the Collaborative Learning and Adaptive Robots (CLEAR) Group, discusses ways to connect robotics with generative AI at an IEEE Robotics and Automation Society event.
Harold Soh of the National University of Singapore discusses robot learning and human-robot interaction at the Fall 2022 GRASP SFI, hosted by GRASP Lab.
