Everyone Feels 20% Faster With AI. The Stopwatch Disagrees.

The gap between what developers feel about AI tools and what the data measures has gotten weirder in 2026, not narrower.

In mid-2025, METR — an independent AI evaluation lab — ran a randomized controlled trial with sixteen experienced open-source developers. These weren't bootcamp grads. The average participant had worked on the repo in question for five years. Before the study began, they predicted Cursor, Claude, and similar tools would make them roughly 24% faster. After completing the tasks, they reported feeling about 20% faster.

The stopwatch said they were 19% slower.

That number — a 39-point gap between perception and reality — bounced around the discourse for months. Then, in February 2026, METR published an update saying their experimental design was probably flawed: developers who benefit most from AI refused to participate in any treatment arm without it, even when offered $50 an hour. Selection bias, in other words, may have stripped the very population the result was supposed to describe.

That's a fair caveat. It also doesn't make the bigger picture go away.

The pattern keeps repeating

A National Bureau of Economic Research paper this February surveyed companies actively rolling out AI coding tools and found that 80% reported no measurable impact on output. Not "modest gains." Not "early days." No measurable impact, at the org level, after rollout.

A Microsoft / MIT / Princeton / Wharton field experiment across 4,867 developers landed in similar territory: above-median-tenure engineers saw no statistically significant productivity increase. The juniors did — somewhere in the 10–30% range, depending on the task. The seniors didn't.

Veracode tested more than a hundred LLMs against 80 coding tasks and found that 45% of the AI-generated code introduced one of the OWASP Top 10 vulnerabilities. CodeRabbit's review of merged code found AI-generated changes contained 2.74× the security defects of human-written changes in the same repos.

Meanwhile, in repos that adopted AI coding agents, the throughput numbers shot up: 98% more PRs opened, 91% longer review times, code churn rose from 3.1% to 5.7%. More code, written faster, reviewed slower, rewritten sooner.

That is not the productivity story the industry has been telling.

Why the vibe is right and the data is right

The simplest reconciliation is that "feels faster" and "is faster" are measuring different things, and AI has cleanly separated them.

When you sit down to write a function and an agent drops a working draft in front of you, the typing-and-thinking part of your day shortens. That's the part you notice. Your fingers spent ten seconds. The status bar updated. The feedback loop felt great.

The parts that didn't shorten — and which the data captures — are:

• Validation overhead. You can't trust a draft you didn't write, especially in code you'll have to maintain. So you re-read it line by line, run it, poke at the edges. For an experienced engineer, that re-reading is slower than writing it yourself would have been, because writing is also how you understand.
• Review time. Your colleagues are doing the same thing on every PR you ship. They're slower than you were, because they didn't write any of it. If you double the PR count, you don't just double review time — you compound it.
• Churn. Code that looks plausible and works in the happy path tends to get committed and then quietly rewritten three weeks later, when the second feature touches the same module and the abstractions don't actually line up. That cost is real but doesn't show up on the dashboard until much later.
• Security debt. Same shape as churn, but the dashboard is a CVE.

None of that is a fault of the model. The model wrote what it was asked to write. The bottleneck moved from "how fast can a human type" to "how fast can a human verify."

The senior/junior split is real and probably permanent

Every well-run study I've read this year has found the same split: juniors faster, seniors flat or slower.

The mechanism, once you stare at it, is obvious. A junior engineer's main cost is not typing speed. It's the cost of not knowing — not knowing the standard library, not knowing the framework's idioms, not knowing how a thing is "supposed" to be done. AI removes that ignorance tax in real time. Of course it speeds them up.

A senior engineer's main cost is deciding. Should this be a queue or a stream? Should we couple these two services or split them? Is this the place to introduce a new abstraction, or am I about to bake in three years of pain? Models in 2026 can produce an answer to those questions instantly. They cannot produce the right answer reliably, because the right answer depends on context the model doesn't have and can't easily get. So the senior reads the suggestion, decides it's probably wrong, decides it's possibly right, considers an alternative, prompts again, gets a different answer, decides… and the cost compounds.

That cost is bigger than typing.

Brockman's 80%

OpenAI's president said earlier this year that AI is "writing 80% of the code at OpenAI." That number traveled. It still travels.

It's also doing a lot of work. The most charitable reading is the one Brockman himself walked toward in follow-up: AI is involved in roughly 80% of changes — autocomplete, refactors, generated drafts that humans then re-shape. Under that definition, the number at most well-tooled shops is probably also north of 70%. Mine is.

The less charitable reading — "humans wrote 20% of OpenAI's code" — would mean OpenAI's repositories should be exhibiting the same churn-and-review patterns described above at extreme magnitudes. I don't have access to OpenAI's internal metrics. I'd be very curious if they do.

Either way, "80%" is a measure of AI's share of token output, not a measure of velocity. They are not the same thing. A team can produce 5× the diffs and ship the same number of features.

What I do about it

I'm not interested in being either a doomer or a booster about this. I use AI tools every day. I shipped two products this spring that I could not have shipped alone in that timeframe without them. The numbers above are real and they're also not the whole picture.

What's changed in the way I work, after sitting with this data for a few months:

I treat AI output the way I treat a junior engineer's first draft: skim for shape, ask "what does this not handle," send it back. The minute I find myself accepting suggestions because they look right rather than because I read them, the productivity story collapses into the churn story.

I write the hard parts myself. Not because the model can't do it, but because I'm the one who has to maintain it, and the writing is part of the understanding. The boring parts — adapters, glue, the eighth CRUD endpoint — those I let the model draft, because the cost of being wrong is small and the verification is fast.

I watch for the trap of feeling fast. Every study agrees that the feeling is unreliable. If it's been three days and I "feel productive," that's the cue to look at what actually shipped.

And I've stopped quoting productivity numbers I can't measure. "10× engineer" was always a vibe. "AI makes my team 30% faster" is the same vibe, dressed up in a slide deck.

The data doesn't say AI is a bad tool. It says we don't yet know how to manage it, and the gap between our intuition and our throughput has rarely been this large. The fix isn't another model. The fix is paying attention to the part of the workflow that isn't typing — which, it turns out, was always most of it.

---

Sources for the studies referenced: METR's initial RCT, the METR follow-up, the Veracode LLM security study, and reporting in MIT Technology Review.