AI Productivity Stats Are Lying, Measure It Yourself

The Number That Keeps Changing Its Mind

METR's early-2025 randomized trial put experienced open-source developers on their own repositories and timed them with and without AI assistants. The result: AI made them 19% slower. The developers themselves thought they'd gotten faster.

Then the revision landed. Looking again at returning developers, METR found an 18% speedup. Newly recruited developers on similar tasks? A 4% slowdown. Same tools, same repositories, three different answers.

So which number is real? All three, probably, and that's the uncomfortable part. METR's writeup is careful about it: the effect depends on who's using the tool, on what code, with how much familiarity with the codebase. Around the same time, TechCrunch noted that plenty of developers now refuse to work without AI, a position built on feeling, not data, because almost nobody is measuring.

And here's what keeps bothering me about that. This isn't a story about AI being good or bad. It's a story about how badly most teams measure their own work. The productivity illusion is real precisely because the measurement is so sloppy. If a funded randomized trial can flip its conclusion in twelve months, what chance does your sprint velocity chart have?

Averages Are the Wrong Unit of Analysis

Goldman Sachs found no meaningful economy-wide link between AI adoption and productivity, but roughly a 30% median gain inside specific tasks, especially customer support and software development work. Zero and thirty aren't contradictory. They're different units, and most teams are measuring at the wrong one.

"The economy" isn't a task. Neither is "the engineering org." When you average a 40% gain on writing boilerplate test fixtures with a 60% loss on debugging an unfamiliar payment service, you get a number that describes nothing and predicts nothing.

I watched this play out at a 40-person SaaS company in 2025. Leadership rolled out an AI coding assistant in January and by March was presenting a chart of "PRs merged per engineer," up 22%. Two engineers had quit in that window. The ones who remained were merging more PRs partly because they'd stopped writing tests. Nobody lied. The metric simply measured the wrong loop.

Ask yourself: how many productivity charts have you sat through that nobody in the room actually believed? That isn't a data problem. It's a measurement-design problem, and it costs real money the next time someone decides how many engineers to hire.

The 31% You Never Put on a Timesheet

Roughly 31% of a developer's day now goes to AI-related invisible work: validating generated output, debugging code they didn't write, and switching context between the assistant and the actual system. None of it appears in a demo. All of it appears in your deadlines.

That figure comes from Use, the DevOps platform, and it ran in SD Times in 2026. It matches what most working developers already know in their bones, the prompt takes four seconds, and the verification takes forty minutes.

Invisible work is the term for this, and it's the biggest blind spot in how teams plan. A task that used to read "write the migration script" now quietly contains "read the generated script, figure out which of the three proposed approaches is actually safe, test the edge case the model invented, and rewrite the error handling." The task name didn't change. The task absolutely did.

A platform lead I know at a fintech put it bluntly: her team's tickets look identical to last year's, but every ticket now hides an extra review step that nobody estimated. So they hit their ticket count and missed their launch date, three quarters running.

That's how AI-heavy teams end up with the worst of both worlds: faster first drafts and slower finished features. The bottleneck moved from writing code to reviewing, trusting, and integrating code, and almost no planning ritual accounts for the move.

Half the Output, More Time

By 2026, AI was writing just under half of everything created inside Linear, the issue tracker. And total product development time still went up, because agents added work instead of replacing it.

Read that again, because it's the most useful data point in this whole mess. More artifacts, more time. When an agent can file issues, draft PRs, and spawn sub-tasks faster than a human can read them, your tracker fills with things that all look like progress and none of which are done. Broader industry data points the same direction: AI-assisted coding climbed to roughly 29% of new code in the U.S. by early 2025, while measured productivity gains stayed in the low single digits.

The failure mode isn't laziness or bad engineering. It's arithmetic. If one engineer can now generate three times the candidate work, the coordination cost of deciding what matters, who owns it, and when it ships triples too. And coordination is the part nobody instrumented.

Agent-generated work behaves differently from human work in one specific way: it arrives without context. A human opening a ticket usually carries the conversation, the thread, the hallway comment. An agent doesn't. So every generated artifact costs someone a few minutes of reconstruction, and a few minutes times a few hundred artifacts is a week.

Measure Your Own Team in One Week

You don't need a research budget to find out whether AI is helping your team. You need a stopwatch, one task category, and the discipline to not measure everything at once.

Here's the method I'd use:

1. Pick one recurring task type. Not "engineering." Something like "add a field to an existing API endpoint" that happens at least weekly.
2. Time the whole loop, not the draft. Start when the developer opens the ticket. Stop when it's merged and tests pass. The draft took four minutes; the loop took a day.
3. Log the review tax separately. Have one person note how long they spent validating output. You'll find your invisible 31% inside a day.
4. Re-measure three weeks later with the same people. Returning developers show different results than new ones, METR's revision proved exactly that.
5. Capture observations somewhere that doesn't cost effort. If logging a task takes more than a few seconds, it won't survive a busy Tuesday. This is the one place tooling genuinely matters: keyboard-first capture means task-level measurement happens as a side effect of working, not as an extra job on top of it.

Five tasks per category is enough. Ten is generous. The point isn't statistical rigor, it's replacing a feeling with a number you can act on. Most teams discover their fast path got faster and their slow path got slower, which is a much more useful finding than "AI helped 12%."

Then Shorten Your Plans

Once you have honest numbers, messier and less flattering than the vendor deck suggested, the right response is shorter planning cycles, not longer ones. Six weeks, re-estimated every Friday.

Rolling re-estimation sounds bureaucratic until you remember what it replaces: a twelve-week roadmap written before the tools changed how long anything takes. If your estimates carry a 30% error bar and the underlying work shifts monthly, a fixed quarterly plan isn't a plan. It's a wish with a Gantt chart.

Three practical rules:

• Plan in ranges, not points. "Two to four days" is honest. "Three days" is a coin flip wearing a suit.
• Re-estimate only what's ahead of you. Don't relitigate finished work, that's where these processes go to die.
• Keep every commitment in one place with an owner and a date. Not a doc, not a thread, not someone's memory.

Karea is built around that third rule, and around the idea that capturing a task has to be faster than forgetting it. But the rule matters more than the tool. Teams that write things down ship. Teams that remember things don't.

The Leadership Gap Nobody Wants to Name

Microsoft's 2026 Work Trend Index found that 58% of AI users said they were producing work they couldn't have produced a year earlier. Meanwhile, the conversation about how organizations should be designed around that fact is barely happening.

Those two things can't both stay true for long. If most of your staff believes they're capable of more than they were twelve months ago, and your planning, staffing, and review rituals look identical to 2024, you're leaving the gain on the floor, and blaming the tools for it.

Goldman Sachs researchers found the gains concentrate in narrow, well-defined tasks. Oracle framed AI coding in 2026 as a productivity jump still ahead of us. Investors keep debating whether any of it reaches delivery metrics. The honest answer: it shows up at the task level, unevenly, and only if someone bothers to look. Aggregate claims will keep disappointing you. Local measurements won't.

Which means the next twelve months probably won't be won by whoever has the best model. It'll be won by whoever knows, with numbers they trust, where their own loop actually got faster, and has the nerve to stop defending the parts that didn't.