AI scores 90% on the benchmarks and finishes 2.5% of the real work. Measure the gap that matters.

You keep reading that AI now outperforms people on the hardest tests we have. It writes code that passes the benchmarks, clears professional exams, and tops the reasoning leaderboards. The natural inference, and the one your board may already be making, is that if it can ace the test it can do the job. Then you point it at your actual work and it stalls. The dazzling 80% demo does not become an 80% delivery. That gap is not your implementation falling short. It is the difference between scoring well on a test and finishing a real piece of work end to end, and there is now hard evidence for how wide it really is.

Researchers at Scale AI and the Center for AI Safety built the Remote Labor Index to measure exactly this. They took 240 real, paid freelance projects from Upwork, spanning software development, design, architecture and data, and tested whether AI agents could complete them end to end, to the standard a paying client would actually accept. The best-performing agent completed 2.5%. Not twenty-five percent, two-and-a-half. The same class of models that scores 80 to 90% on research benchmarks performs, in the authors' words, near the floor on real economically valuable work. The benchmark measures isolated skills in ideal conditions; the index measures delivered value in messy reality, and the two turn out to be worlds apart.

Why is there such a gap between the benchmark and the real job?

Because benchmarks test the part of work that is easy to test: bounded problems with a clean, checkable answer. Real projects are the opposite. The brief is ambiguous, the context is missing, the requirements shift, and getting it finished means exercising judgment, filling gaps, iterating with a client and pushing it over the line to a standard someone will pay for. AI is genuinely excellent at the bounded part and genuinely weak at the unbounded part. A high benchmark score tells you the model has raw capability. It tells you almost nothing about whether it can deliver your work, because your work is mostly the unbounded kind.

What does this mean for how I invest in AI?

Stop buying on the leaderboard and start measuring on your own work. The demo that dazzles in a vendor pitch is the benchmark's cousin: a bounded, curated slice chosen to impress. What actually matters is the completion rate on your real tasks, to your real standard. That means running AI against a representative sample of the work you care about and measuring how much of it reaches done-and-acceptable without a person quietly finishing it, rather than how polished the first draft looks. The finish is where the value and the risk both live, and it is the thing a benchmark score will never show you.

So is AI overhyped?

No, and this is the balanced read that matters. The 2.5% is a snapshot of today, and the trajectory is upward: capability is climbing quickly and the completion rate will rise with it. The error is not believing AI is powerful, it plainly is. The error is assuming that benchmark power equals delivered value in your context right now. The sober position takes both seriously at once: treat the capability as real and rising, and treat the delivery gap as real and, for now, wide. Deploy AI fully where it genuinely finishes the job, and keep a human firmly in the loop everywhere the gap is still open, which today is most places.

1. Measure AI on your work, not the leaderboard. Build a test set of real tasks and score end-to-end completion to your standard, not first-draft impressiveness.
2. Separate "impressive draft" from "delivered result." The distance between them is where cost and risk hide. Track the finish, not the demo.
3. Discount vendor benchmarks. A model topping a public leaderboard is a claim about capability, not a promise about your outcomes.
4. Deploy where it finishes, augment where it stalls. Full automation only where completion is genuinely high; human-in-the-loop everywhere else.
5. Re-measure on a schedule. The gap is closing, so today's "keep a human on it" becomes tomorrow's "let it run." Decide on evidence, not fear and not hype.

What does this change for me as a leader?

It moves the question from "how capable is AI" to "how much of my actual work does it finish." The first has an impressive, rising answer. The second has a soberer one that should drive your decisions. The leaders who waste money are the ones who read the benchmark and restructure as if delivery had already arrived. The ones who get ahead measure the real completion rate on their own work, deploy hard where it is high, and keep human judgment in the loop where it is not, revisiting the line as the evidence moves.

This is the same discipline behind why most organisations fail at AI adoption: the constraint is rarely the model's raw power and almost always the gap between that power and delivered value in a real organisation. Close that gap with measurement and honest deployment, and you capture what AI can actually do today. Assume the benchmark is the business, and you will pay for a capability you have not yet got.

About the author

Thomas Green

British technology futurist, AI keynote speaker and advisor. Thirty years across enterprise technology and AI strategy, helping leaders navigate the future of work. The futurist who died.