Teaching a humanoid robot to play cornhole with Gemini 4 Argon

In this talk

1. 0:04

   Learning from a human example

   The team gave Gemini 4 footage of a person throwing and asked it to practice in the MuJoCo simulator. It reviewed each attempt, identified mistakes, and used trial results to improve its control.

2. 0:46

   Mapping the throw to the robot

   The robot says it analyzed the thrower's arm movement and release, then mapped that motion to its own joints and hands. It also used its two cameras to locate the board in 3D and build a physics simulation of the robot and target.

3. 0:46

   Practicing and adjusting the motion

   In simulation, the robot changed its stance and tuned its swing to clear its hip and send the bag along a smooth arc. On the physical robot, it watched where each bag landed and shifted its release point by a few centimeters.

4. 0:46

   Seven successful shots in a row

   After those adjustments, the robot made seven consecutive shots into the hole. The presenter says Gemini 4 used video understanding to reason about failures and learn from its own trials.