RoboBoom
RoboPapers· November 10, 2026· 50:30

Ep#116: SimEX: Simulation-Integrated Robotics AutoResearch

Jahung, a UT Austin PhD student, and J, an Amazon FAR research scientist, discuss SimEX and its simulation-assisted approach to improving robot skills on RoboPapers with host Michael.

In this talk

  1. 1:38

    Why bring coding agents into robotics

    Jahung and J describe how coding agents can test ideas and revise programs using feedback. Directly generating robot code leaves little room to learn from failures, while physical trial and error takes time and risks damaging equipment. SimEX uses simulation to make experiments faster and safer.

  2. 5:26

    SimEX’s two-stage process

    SimEX first builds a toolbox of robot skills and guidance through open-ended simulation experiments. Agents propose tasks that expose weaknesses, refine the toolbox, and retain earlier tasks to check that improvements do not undo prior progress. The resulting toolbox is then adapted to a real task using robot trials and simulation-based comparisons of possible fixes.

  3. 15:40

    Results across robot tasks

    The team reports learning barcode scanning, plate placement, and towel folding without demonstrations and with under 10 minutes of real-robot interaction. For towel folding, their simulator lacked visual rendering and realistic gripper contact, yet the agent learned to grasp a corner, fold, and flatten the towel. Other learned behaviors included deliberately flipping objects to find a barcode and changing grasp positions to avoid tote edges.

  4. 22:03

    Human involvement and limits

    The researchers want to reduce human involvement, but people still reset the robot between trials and inspect the initial simulator. They say frontier models can reason about 3D scenes and adapt through contact, while slow inference and hard-to-simulate tasks such as tearing remain obstacles. J also cautions that a simulator built from a picture may omit hidden physical details, making reinforcement learning prone to exploiting errors.

  5. 38:02

    Research, agents, and people’s future roles

    The guests debate whether coding agents will make implementation and idea generation cheaper, and argue that research judgment and problem selection may matter more to PhD students. They imagine humans increasingly handling physical setup and resets while agents run experiments, while noting that real-world robot fleets are difficult to scale to thousands of environments.

Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.