Training the AIs' Eyes: How Roboflow is Making the Real World Programmable, with CEO Joseph Nelson

In this talk

1. 6:02

   Why computer vision remains difficult

   Nelson says visual models are advancing along a path similar to language models, but the variety of real-world scenes makes vision harder to generalize. Some tasks, such as counting people or reading receipts, are becoming reliable, while precise grounding, measurement, and long-tail cases remain challenging. Roboflow’s Vision Checkup collects examples of these limitations, including inconsistent results and slow inference.

2. 36:23

   Choosing and optimizing a model

   Nelson recommends starting with the task’s requirements, including accuracy, response time, privacy, and where the model must run. For real-time applications such as Wimbledon replay, a frontier model can label examples that are then used to train a smaller model for efficient edge deployment. He says the amount of data depends on the task: a controlled manufacturing line may need hundreds of images, while open-road driving has far more variation.

3. 57:14

   Roboflow’s models and architecture search

   Roboflow built RF-DETR for real-time detection and segmentation using a DINOv2 backbone and a shared-weights neural architecture search technique. One training run evaluates many network configurations and yields models across a speed and accuracy tradeoff. Nelson says customers can run the search on their own dataset and select a model suited to their constraints.

4. 1:07:38

   Agents, robotics, and wearables

   Nelson describes Roboflow’s approach as offering accessible tools with deeper options available as needed, including CLIs that coding agents can operate. The company is also developing an agent to help users turn a goal, such as tracking cars crossing a line, into a suitable vision pipeline. He sees world models, vision-language-action systems for robots, and smart glasses as emerging areas, while noting that edge compute and power remain constraints.

5. 1:36:30

   Potential benefits and governance

   Nelson imagines vision systems improving agriculture, food safety, manufacturing, transport, and everyday tasks. He expects privacy and public acceptance to shape how camera-enabled products are used. On regulation, he favors enforcing rules against harmful outcomes and warns that regulating tools or model size could unintentionally block useful applications, such as automating cell counts in a research lab.