Ep#64: Project Instinct

In this talk

1. 2:00

   A modular toolkit for robot learning

   Project Instinct brings perception, whole-body control, motion references, sensors, and real-robot testing into one toolchain. Its components are designed to be replaceable, so researchers can compare modules or use their own datasets without rebuilding the system. The authors invite the community to help develop reusable evaluation methods, since current humanoid metrics can be hard to measure consistently in real-world tests.

2. 10:55

   From quadruped parkour to humanoid control

   The research began with quadruped parkour, aiming to overcome obstacles larger than the robot, and later moved to humanoid parkour. The team found that humanoids need different terrain representations because their elevated cameras see stairs and platforms differently from a quadruped’s forward-facing view. They describe instinct-level behavior emerging when a policy learned to avoid risky jumps after experiencing failures in simulation.

3. 16:41

   Matching motion to terrain with perception

   The pipeline scans terrain, retargets motion-capture data, matches motions to terrain, and trains many robots in parallel. In tests, depth input helped the robot adjust when its starting position was shifted by as much as 0.5 meters, and it completed motions on unseen terrain. The team uses a forward stereo depth camera with a two-meter view; delayed frames from a weak USB connection sometimes caused jitter.

4. 32:57

   Engineering for scale and reuse

   The team uses virtual obstacles and terrain edge detection to encourage safer, more redundant movements, such as jumping higher than a platform requires. Project Instinct also includes ray casting for crowded training scenes, several learning methods, multiprocess training, and deployment tools that read experiment settings from training logs. To support another robot, researchers can define its hardware configuration alongside the G1 configuration.

5. 41:02

   Whole-body intelligence and next steps

   The guests frame humanoid locomotion as whole-body control that can include climbing, crawling, and using contact, with decisions about when to go over an obstacle or around it. They argue that hardware and software should be developed together, since hardware changes can improve performance but may trade flexibility for rigidity. They want more motion and terrain data, stronger sim-to-real reliability, and community contributions to shared robot configurations.