Reinforcement Learning for Robotics Part 3: Deploy AI Agent to Real Robot (Sim-to-Real) | DigiKey

In this talk

1. 0:46

   Prepare the trained actor for deployment

   Training uses an actor network to choose actions and a critic to estimate future rewards; deployment only needs the actor. The presenter exports the best model as ONNX, converts its fully connected layers and tanh activations into C, and runs inference on an ESP32 in about two to three milliseconds.

2. 5:36

   Connect the agent to the robot

   The setup uses an M5Stack BALA2 Fire, Arduino, and an IMU calibration routine that stores calibration data on the ESP32. At each five-millisecond step, firmware combines accelerometer and gyroscope readings, reads wheel encoders, and supplies four observations to the actor: pitch, pitch rate, and both wheel velocities. The two outputs are scaled into motor commands, with a tip threshold that stops the motors.

3. 17:36

   Diagnose the sim-to-real gap

   The robot balances only briefly before driving away and falling, despite performing well in simulation. Possible causes include sensor noise or calibration, incorrect motor torque estimates, environmental noise, and differences between the modeled and physical robot. More accurate hardware modeling and domain randomization are presented as ways to address these mismatches.

4. 18:40

   Tune firmware and prepare for domain randomization

   Firmware adjustments improve balance: a pitch offset corrects the robot’s perceived upright position, a 1.5 motor boost compensates for weaker battery operation, and a deadband suppresses small commands. An IIR filter smooths motor actions, though it adds lag. These fixes help, but require tuning; the next step is training with randomized sensor and motor behavior so the agent can handle variation more robustly.