Beyond the Pilot: Scaling AI That Pays Off

In 2023, the hard question was “can we build a convincing demo?” By 2026 that question is settled: a capable prototype is a weekend, not a quarter. The hard question now is the one that actually decides winners and losers: can you turn that prototype into a durable line on the income statement? For most organisations, the honest answer is still no.

The numbers are sobering. A 2025 MIT study of enterprise adoption found that roughly 95% of generative-AI pilots delivered no measurable impact on profit and loss, a chasm the researchers called “the GenAI divide.”¹ RAND, studying AI projects more broadly, put the failure rate above 80%, about twice the rate of conventional IT projects.² Gartner expects at least 30% of generative-AI projects to be abandoned after proof of concept.³ The technology works. The transfer from lab to ledger does not.

This is not a story about models getting worse; they are getting dramatically better. It is a story about a familiar pattern in the history of general-purpose technologies, and about the unglamorous discipline that separates the few who compound value from the many who accumulate proofs of concept. This piece lays out why pilots stall, and the operating model that gets them to production value, grounded in the work of the researchers and practitioners who have studied it most closely.

01The pilot paradox

Pilots are seductive because the first 80% of a demo is now almost free. A foundation model, a few prompts, a slick interface, and you have something that looks like the future in a boardroom. But the demo optimises for the wrong thing: it proves the technology can produce an impressive output, not that the organisation can produce a reliable outcome, at scale, at acceptable cost and risk, embedded in a workflow people actually use.

None of this is new. Economists have watched it before. The electric motor was commercialised in the 1880s, yet the productivity payoff didn't arrive for roughly four decades, not until factory owners stopped simply bolting motors onto steam-era layouts and redesigned the factory itself around distributed power. Economic historian Paul David told this story in his classic study of “the dynamo and the computer,”⁴ and Erik Brynjolfsson and colleagues formalised the modern version as the “Productivity J-Curve”: general-purpose technologies first cost you, in the intangible work of reinvention, before they pay.⁵

Ng's analogy is quoted so often it has lost its edge, but the edge is exactly the point. Electricity was transformative not because it was electricity, but because of everything organisations rebuilt to use it. AI pilots stall precisely where that rebuilding is skipped.

02Why pilots stall: the four gaps

Across failed initiatives, the same four gaps recur. They are not exotic. They are the parts of the work that a demo lets you ignore.

Gap 1: The model is the easy 5%

A decade ago, a team of Google researchers led by D. Sculley published what remains one of the most quoted diagrams in applied machine learning: a tiny box labelled “ML code” surrounded by vast boxes for data collection, configuration, serving infrastructure, monitoring, and “glue code.” Their point in Hidden Technical Debt in Machine Learning Systems was blunt: the model is a small fraction of a real-world ML system, and the surrounding machinery is where projects live or die.⁶ Generative AI hasn't repealed that law; it has raised the stakes. The prompt is the 5%. Retrieval, evaluation, guardrails, latency, cost control, versioning, and observability are the 95%.

Gap 2: The pilot measures the wrong thing

Demos are judged on whether the output is impressive. Businesses are paid on whether a metric moved: cost per case, cycle time, conversion, revenue, risk. A pilot that reports “90% accuracy” but is never wired to a P&L line is a science project, not an investment. The MIT researchers found the divide correlated less with model sophistication than with whether the deployment was connected to a real workflow with a real feedback loop.¹

Gap 3: The data isn't production-ready

Andrew Ng's data-centric AI movement made the case that, for most applied systems, the fastest path to performance is improving the data, not the model.⁷ Pilots quietly borrow clean, curated examples; production faces messy, drifting, permissioned, and sometimes poisoned data. The gap between the two is where accuracy, and trust, evaporates.

Gap 4: The workflow and the organisation aren't redesigned

This is the deepest gap, and the one the technology can't close for you. Brynjolfsson's research on earlier IT waves showed that the firms which captured value were those that paired technology with complementary organisational change: new processes, roles, incentives, and skills, an “intangible capital” investment far larger than the software itself.⁵ Thomas Davenport and Nitin Mittal, surveying enterprises that succeeded with AI in All-in on AI, reach the same conclusion: the differentiator is organisational and strategic commitment, not access to algorithms.⁸

03The operating model that pays off

If the failure pattern is consistent, so is the success pattern. Organisations that cross the divide don't have better models than everyone else; the frontier models are a commodity available to all. They have a repeatable operating model for turning a promising idea into an industrialised capability. It has five moving parts.

1. Select for value, not novelty. Winners run a portfolio, not a science fair. Each candidate use case is scored on business value and feasibility, sequenced so that early wins fund and de-risk harder bets. McKinsey's and BCG's research on high performers points the same way: value concentrates in a small number of well-chosen, deeply integrated use cases, not a scatter of experiments.^9,10
2. Build the production system, not the demo. Treat the 95% as the actual project. Stand up evaluation harnesses, retrieval and data pipelines, monitoring, cost and latency budgets, versioning, and rollback, the discipline Chip Huyen calls designing the system around the model rather than shipping the model alone.¹¹ The goal is a data flywheel: production usage generates the signal that makes the next version better.
3. Redesign the workflow around the model. Decide where the human sits, what the model is allowed to do autonomously, and how the two hand off. The largest gains come from re-shaping the job, not decorating it, which is why change management, training, and incentives belong in the project plan, not the appendix.
4. Make trust a feature, not a gate. At scale, the binding constraint becomes risk: security, reliability, privacy, and compliance. Teams that treat governance as an enabler (red-teaming against the OWASP LLM Top 10, building controls aligned to frameworks like the EU AI Act) ship faster, because they can put systems in front of customers and regulators with confidence. Trust is what lets a pilot leave the building.
5. Measure in the P&L, and iterate ruthlessly. Instrument the business outcome from day one. Kill the use cases that don't move a number; double down on the ones that do. The point of a pilot is not to prove AI is possible; it is to learn the smallest thing that tells you whether to scale or stop.

04What the evidence shows when it works

Done this way, the returns are real and measurable, not hypothetical. In one of the most rigorous field studies to date, Brynjolfsson, Danielle Li, and Lindsey Raymond examined more than 5,000 customer-support agents given access to a generative-AI assistant embedded in their workflow. Productivity rose about 14% on average, and around 34% for the least-experienced workers, as the system effectively distributed the tacit knowledge of the best agents to everyone else.¹² The gains came not from the model in isolation, but from the model in the workflow, measured against a real outcome.

The macro picture rhymes with the micro one. BCG has found that only about a quarter of companies have built the capabilities to move beyond proofs of concept and generate tangible value, and that this minority pulls further ahead each year.¹⁰ McKinsey's surveys tell the same story: a small cohort of “AI high performers” concentrates the returns, distinguished by how they operate (redesigned workflows, senior ownership, and disciplined measurement) rather than by which models they use.⁹ The divide is not luck. It is a choice about how you run.

05The takeaway

The uncomfortable truth of the last three years is that access to frontier AI confers no lasting advantage; everyone has it. The advantage belongs to the organisations that can reliably convert that access into outcomes: the ones that pick the right problems, build the unglamorous 95%, redesign the work, earn trust, and measure the money.

That is not a model you download. It is a capability you build. Scaling AI that pays off is, in the end, an operating-model transformation wearing a technology costume, and the companies that understand this are quietly compounding, while their competitors accumulate another quarter of impressive pilots that never left the building.

References & further reading

1. MIT (Project NANDA), “The GenAI Divide: State of AI in Business 2025” (2025): the finding that ~95% of enterprise GenAI pilots showed no measurable P&L impact.
2. RAND Corporation, “The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed” (2024): AI project failure rates above 80%.
3. Gartner press release (2024): forecast that at least 30% of generative-AI projects will be abandoned after proof of concept.
4. Paul A. David, “The Dynamo and the Computer: An Historical Perspective on the Modern Productivity Paradox,” American Economic Review (1990).
5. Erik Brynjolfsson, Daniel Rock & Chad Syverson, “The Productivity J-Curve” (2021), and Brynjolfsson's broader work on complementary organisational capital.
6. D. Sculley et al., “Hidden Technical Debt in Machine Learning Systems,” NeurIPS (2015).
7. Andrew Ng, Data-Centric AI (DeepLearning.AI): the case for improving data over models in applied systems.
8. Thomas H. Davenport & Nitin Mittal, “All-in on AI,” Harvard Business Review Press (2023).
9. McKinsey & Company, “The State of AI” (2024–2025): the profile of AI high performers.
10. Boston Consulting Group, “AI Adoption in 2024” (2024): ~26% of firms moving beyond proofs of concept to tangible value.
11. Chip Huyen, “Designing Machine Learning Systems,” O'Reilly (2022).
12. Erik Brynjolfsson, Danielle Li & Lindsey Raymond, “Generative AI at Work,” NBER Working Paper (2023).

This article summarises publicly available research and the published positions of the cited experts; quotations are limited to widely documented public statements, and other views are paraphrased with attribution. It is intended as analysis, not a substitute for tailored professional advice.