Why would you use a video game to simulate robots?

In this talk

1. 0:00

   Minecraft as a robotics simulator

   The host compares Minecraft with established options such as Gazebo, Webots, Unity, and Isaac ROS. A mod by Japanese student Kazuza Hashimoto connects Minecraft with ROS 2, making its generated voxel worlds available for robotics experiments.

2. 1:52

   Setting up the ROS 2 environment

   The mod runs in Docker, which lets the host use software built for ROS 2 Humble on a Jazzy machine. Docker Compose launches Minecraft and RViz in separate containers; the first launch downloads and builds the needed components. The host also installs NVIDIA Container Toolkit to resolve a GPU setup error.

3. 5:19

   Sensors and movement through ROS

   Minecraft publishes camera images and provides several LiDAR models with different resolutions and fields of view. Higher data loads slow the host's computer, so the demonstration uses a simpler laser. The mod also accepts velocity commands, letting the host drive the player with ROS 2 keyboard teleoperation.

4. 9:00

   Where the simulation struggles

   The host could not get consistent results from SLAM, possibly because of timing, synchronization, latency, or stuttering. Odometry support appears incomplete, and a provided SLAM example worked only briefly with one laser. The host notes that the IMU, player status, console commands, and entity spawning could add useful capabilities, though some features were unavailable or difficult to use.

5. 9:58

   Teaching experiments to try

   For tasks that tolerate slower updates, students could use OpenCV to find a red target and move toward it. Other suggested exercises include searching for AprilTags and using LiDAR depth to avoid obstacles. The host sees potential for teaching ROS and algorithms in a familiar game, while noting that a stopped control script may leave the player moving.