Researchers with the DrivingBench project tested whether a general-purpose language model could drive a real car without a specially trained autopilot. Four models were given access to the steering wheel, accelerator, and brakes of a Toyota Corolla, and a route marked out with cones in a parking lot. Only one made it to the finish. Here's how the test was set up and where the others went wrong.
How the experiment works
The test was conducted by Tobias Gessler, Aditya Ramabadran, and Simon Mans. The car had a Comma Four device from comma.ai running the open-source openpilot software. It connects to the car's internal network (CAN) and provides software access to the steering wheel, accelerator, and brakes. The model controlled the car using three tools within a single continuous dialogue: it observed the surroundings, decided what to do, and issued a command.
The conditions were chosen to ensure no one got hurt: an empty parking lot, a strict speed limit, and a person at the wheel ready to brake at any moment. The models themselves ran on remote servers, so every decision required a network call.
Route
About 135 meters (134.7 m, to be precise) between the cones: straight sections and several turns. The first obstacle was a diagonal line of cones, and even that proved a challenge for perception. Each model was given three attempts.
Results
| Model | Best result |
|---|---|
| GPT-6 Astra | the entire route, on its second attempt, in 5 min 22 s |
| Claude Fable 5.1 | about 45% of the distance |
| Grok 4.6 | about 11% |
| GPT-5.6 Sol | about 6% |
GPT-6 Astra's first attempt ended in failure, but on its second try the model learned from its mistakes and made it to the end. Its average speed was about 1.5 km/h (0.94 miles per hour), with a top speed of around 3 km/h. The model requested a new observation roughly every 5–6 seconds.
Over the course of the drive, it used about 6.6 million tokens. Thanks to caching, the inference bill came to $7.74, but that works out to about $92 per mile. The figure is meaningless as a comparison with any production autopilot, but it clearly shows how much “thinking” goes into something a person does reflexively.
Where the models went wrong
- Thinking pauses. In one attempt, Claude Fable 5.1 controlled the car for 31 seconds, then the vehicle sat still for about 160 seconds while the model thought about its next move. Of the 190 seconds in the attempt, the car was moving for less than a fifth of the time.
- Confusion about directions. Models mixed up left and right and misread which way the line of cones went.
- Vehicle dimensions. The car's dimensions were estimated inaccurately, so it either turned too close to the cones or missed the intended path.
- Safety refusals. Some models refused to drive despite assurances that the parking lot was empty. The researchers had to rename the system “DrivingBench Sandbox” to get the models to treat the task as a test.
What this test shows
The gap between reasoning in chat and controlling a physical object is enormous. A model can correctly describe the need to turn left and slow down, but in the real world, reaction time and spatial understanding matter—and language models are still poor at both. A car that sits still for almost three minutes while it “thinks” is more dangerous on the road than one that doesn't move at all.
At the same time, GPT-6 Astra's result is an important benchmark: a general-purpose model with no specialized driving training was able to complete the entire route and correct its mistakes between attempts. For comparing models with one another, this kind of test is more informative than many text benchmarks because it tests perception, planning, and decision-making speed all at once.
Takeaways for those connecting models to physical systems
- Latency matters more than reasoning quality. If a model takes tens of seconds to think, the system it controls must be able to remain safely stationary for that entire time.
- An emergency override outside the model is essential. In the test, this role was played by a person at the brake and a speed limiter. In any real-world application, this must be a device or logic independent of the model.
- Test spatial tasks separately. High scores on text tests say nothing about how a model will understand the positions of objects in an image.
- Account for the cost of long sessions. A continuous “observation — decision — action” loop quickly racks up millions of tokens, even on a short task.
Compare models before you start
The service sets its plans, limits and model catalog. If they differ from this article, contact us so we can update it and record a new review date.
Browse modelsAffiliate link: your price stays the same and the project earns a commission.