Ep#29: Exploring Low Cost Robots using LLM Agents
Hobbyist robotics and machine learning practitioner Ilya Lashanka joins RoboPapers to discuss affordable home robotics and his experiments with LeRobot, zero-shot LLM control, and lightweight learned policies.
In this talk
- 2:23
Affordable home robotics with LeRobot
Lashanka introduces Hugging Face’s LeRobot library, which supports robot control, teleoperation data collection, policy training, and running policies on real robots. He describes the OpenArm 100, a five-degree-of-freedom arm with a gripper that costs about $130, and the mobile Lucky EV3. His home setup uses a dining table, cameras, a laptop, and cloud GPUs for larger training runs.
- 10:18
Controlling a robot with LLM agents
Lashanka connects an MCP server to LLM clients so models can inspect robot state and camera images, control the gripper, and issue simple movement commands. In zero-shot trials, Claude, Gemini, and GPT attempt tasks such as grabbing a ball or putting socks in a box; Claude performs best in his tests, followed by Gemini, while GPT is inconsistent. Simple grasps succeed about half the time, and adding steps reduces success. Different camera viewpoints can confuse left and right, while aligned top views help.
- 47:28
A compact behavioral cloning policy
Lashanka presents a decoder-only transformer policy trained through behavioral cloning, using camera images and robot state to predict action sequences. He simplifies the architecture and describes techniques including L1 loss, a short set of recent observations plus one older observation, and sending slightly future actions to trade some precision for speed. In PushT and simulated ALOHA insertion, he reports improvements of roughly 6 to 10 percentage points over comparison policies, with a 2-million-parameter model for the PushT keypoint setup.
- 1:06:55
Running the policy and planning next steps
The learned policy runs on a MacBook M1 without a GPU at about 25 predictions per second and adapts when a cup changes position. Lashanka says he has documented the work and shared code and trained models. He wants to explore vision-language-action policies and combine LLM reasoning with small policies for more precise actions; he also hopes to extend LLM control to the Lucky EV3’s mobile base.
Briefs summarise talks and interviews; timestamps open the video at that moment. Claims in a talk are the speaker's own.