The Production Gap: Why AI Pilots Never Ship

One AI lab just raised more money in a single round than most countries spend on infrastructure in a year. The same season, a survey of enterprises found that only about 15 in 100 were actually ready to run the AI they had already bought. That distance, between what AI can raise and what it can ship, is the most expensive gap in business right now, and almost nobody is budgeting for it.

Here is the thing that took me two companies and a lot of wasted quarters to accept. A working demo tells you almost nothing about whether you have a product. It tells you the model can do the task once, in your hands, on the input you chose. Production asks a completely different question: can it do the task a thousand times, on inputs you did not choose, inside a workflow you do not fully control, at a cost you can afford, with someone accountable when it is wrong. The distance between those two questions is what I call the production gap, and it is where most AI work goes to die.

I have shipped AI that saved a team hours a day, and I have killed AI projects that demoed beautifully and could not survive contact with a real user. The difference was never the model. It was whether I had built the boring foundation underneath the clever part. This is a map of that foundation, and a way to see the gap before it swallows your time and money.

The gap nobody budgets for

Start with the numbers, because they are worse than the vibe on your timeline suggests.

MIT’s Project NANDA studied enterprise generative AI in its “GenAI Divide” report and found that about 95% of pilots delivered no measurable return on the profit-and-loss statement. Not a small return. No measurable return. IDC put the share of AI proofs of concept that never reach production at 88%. RAND found more than 80% of AI projects fail outright, roughly twice the failure rate of ordinary software projects. S&P Global’s survey of enterprises found that 42% abandoned most of their AI initiatives in 2025, up sharply from 17% the year before. And when the Composio team looked specifically at AI agents, they found 97% of executives had deployed one in the past year, while only 12% of those agent projects reached production at scale.

Now hold that against what companies are spending. The Fivetran readiness survey found that close to 60% of organizations were investing millions to tens of millions of dollars in agentic AI, while only 15% were fully prepared to run it in production. Money going in at one rate, readiness lagging at a fraction of it. That is not a technology gap. A model good enough to demo is a model good enough to demo. That is an execution gap, and it has a shape you can learn to see.

The cruelest part is the cost curve. A proof of concept is cheap, a few weeks and a small bill. Then the real number arrives. Teams routinely discover that the production version of their pilot costs 50 to several hundred times the pilot baseline once you account for real volume, retries, monitoring, and integration. One documented case turned a 1,500 dollar per month proof of concept into a 1,075,786 dollar per month production system, a 717x jump that no business case built on pilot economics could survive. The project does not get killed at the demo. It gets killed at month 14, when the production forecast lands and the spreadsheet stops making sense.

If you are a founder, the enterprise framing hides how directly this hits you. You are not immune because you are small. You are more exposed, because you probably built the demo yourself in a weekend and felt the rush of it working. That rush is the trap. The weekend built the easy 90%. The gap is the last 10%, and the last 10% is most of the work.

Two tests, not one

The reason smart people keep falling into the gap is that a demo and a production system look like the same thing at different sizes. They are not. They are two different tests measuring two different properties, and passing the first tells you almost nothing about the second.

The demo test asks: can the system produce a good output on an input I selected, one time, while I am watching. It is a test of capability under ideal conditions. You curate the input, you run it until it looks right, you show the good take. Nothing is wrong with that. It is how you find out the idea is worth pursuing.

The production test asks something harder and less flattering. Can the system produce an acceptable output on inputs it has never seen, thousands of times, when no one is watching, at a price that leaves a margin, inside the tool where the work actually happens, and can someone be held responsible when it fails. Every one of those clauses is a place the demo quietly cheated, and every one of them is a wall you have to climb before real users depend on the thing.

I find it helps to lay the two tests side by side, because the demo hides its assumptions so well that you forget they are assumptions at all.

Read down the right column and you have the syllabus for the rest of this piece. Each row is a gate. A pilot passes production only when it clears all of them, and it dies at the lowest gate it never built for.

The Five Gates

The production gap is not one wall. It is five, in a rough order, and the order matters because a pilot fails at the first gate it cannot pass and never reaches the ones after it. You can have a flawless cost model and it will not save you if your data falls apart at gate one. So the useful mental model is a funnel: a hundred pilots walk in, and at each gate a predictable share drop out, until the dozen or so that built every layer walk out the other side into production.

The five gates are Data, Reliability, Workflow, Economics, and Ownership. The first three are about whether the thing works when reality shows up. The last two are about whether the business around it can survive the thing working. Founders love to argue about model choice, which is a decision that sits above all five gates and matters far less than any of them. Let me take each gate in turn, because the fix at each one is specific.

Gate one: the data underneath

This is the gate that kills the most pilots, and it is the least glamorous, which is not a coincidence. In the Fivetran readiness work, the limiting factor was not model quality. It was data readiness. In a separate survey, 60% of chief executives named disconnected or low-quality data as the single biggest barrier stopping their AI from scaling. The model was fine. The fuel was contaminated.

Here is the mechanism. When you build a demo, you feed the system data you chose. It is clean, it is recent, it is formatted the way you like, and it represents the happy path you want to show. Then production arrives, and the real data is nothing like that. It is late. Half the fields are empty. The same customer appears three times under two spellings. The document that was a tidy PDF in your demo is a scanned fax with a coffee stain. The model that looked brilliant on your curated set now confidently produces garbage, because the assumptions it was trained and tested against do not match the enterprise reality it now lives in.

I have watched a support agent that scored beautifully in testing fall apart in its first week live, not because the model got worse, but because real tickets arrived with attachments, half-sentences, and context buried in a thread from four months earlier. None of that was in the demo. All of it was in production.

The fix is unsexy and it is the whole game. Before you fall in love with the clever layer, build the plumbing that gets clean, current, connected data to the model. That means a real pipeline, not a folder of hand-picked examples. It means deduplication, freshness, and a way to handle the missing and the malformed without falling over. This is the operational layer I broke down in the internal AI stack for solo founders, and it is the least fun part of the build and the part that decides everything. If you skip it, you do not have a data problem later. You have a dead pilot later.

A blunt test for whether you have cleared this gate: can you point the system at a random slice of real, untouched production data immediately, with no cleanup, and get an acceptable result. If the honest answer is “well, once we clean it up,” you are still at gate one, no matter how good the demo looked.

Gate two: the worst run, not the best

The demo shows you the best run. Production has to survive the worst one. That single shift, from best case to worst case, is the second gate, and it is where a lot of technically impressive pilots quietly fail.

When you demo, you naturally run the thing a few times and show the take that landed. That is not dishonest, it is human. But it means the number in your head is the ceiling, not the floor. In production, a user does not experience your average. They experience the specific answer they got, and if one in twenty answers is wrong in a way that costs them money or trust, they do not care that the other nineteen were great. The worst run is the product, because the worst run is what breaks the relationship.

This is the reason AI systems that look ready are not, and I wrote a whole piece on the failure modes in why AI agents fail in production. The short version is that reliability is not a vibe you get from a good demo. It is a measurement you earn by running the system many times on your own hard cases and looking at the spread, not the peak.

Which leads to the trap on the other side of this gate: fake reliability. It is tempting to quote the good number, the demo number, the best of many runs, and call it your accuracy. That is the move I called eval-washing, and buyers have gotten good at smelling it. The defense, and the way to clear this gate honestly, is to build your own evaluation set out of the cases that actually matter to your users, run the system against it repeatedly, and hold yourself to the worst acceptable outcome rather than the best possible one.

The test for this gate is simple to state and uncomfortable to pass: what happens on your ten hardest real inputs, run five times each. If you have never done that, you do not know your reliability. You know your demo.

Gate three: the job it has to live in

A demo is a standalone toy that you drive. Production is a step inside a job that someone already does a certain way, and if your AI does not fit that job, it does not matter how good it is. It becomes the extra tab nobody opens.

MIT’s research on why pilots stall put a name on this. They called it the learning gap: tools that do not learn, integrate poorly, or fail to match how the work actually happens. The systems that failed were not dumb. They were brittle and disconnected. They sat beside the workflow instead of inside it, they did not remember what happened last time, and they did not adapt to the way a specific team worked. So people used them twice, found them more effort than they were worth, and drifted back to the old way.

I think of this as the difference between a feature and a detour. A feature removes steps from the job. A detour adds a stop. If your AI requires the user to leave what they were doing, go somewhere else, paste something in, wait, copy the result, and bring it back, you have built a detour, and detours lose to the status quo every time. The bar is not “is the output good.” The bar is “is the whole loop, including getting the input in and the result back to where it is needed, faster and easier than not using it at all.”

This is also where the system has to close the loop and get better with use, which is the real moat I argued for in loop engineering. A tool that produces the same static quality forever, disconnected from feedback, is exactly the brittle thing MIT found dying in pilots. A tool that captures what happened, learns from corrections, and fits tighter to the job each week is the thing that survives, because it earns its place in the workflow instead of demanding one.

The test for this gate: watch a real user do the whole task with your tool in it, and time it against them doing it without. If the AI version is not clearly faster or better for them, in their tool, on their terms, you have not cleared gate three even if the model is excellent.

Gate four: the bill that arrives at scale

Now the business gates. The first three were about whether the thing works. These two are about whether you can afford for it to work. Gate four is the cost that was invisible in the pilot and becomes the whole story at scale.

I said it earlier and it is worth sitting with: production costs routinely land at 50 to several hundred times the pilot baseline, and there is a documented case of a 717x jump. That is not because someone was careless. It is because the pilot ran on a trickle of traffic with no retries, no monitoring, no redundancy, and no long context, and production runs on a flood with all of them. Every retry is a paid call. Every guardrail check is a paid call. Every user who sends ten times the expected volume is ten times the cost, at a flat price you already quoted them.

The reason this kills pilots specifically is that the business case was built on pilot math. Someone modeled the return using the trickle-sized bill, got a healthy margin, and got approval. Then the flood-sized bill arrives, the margin inverts, and the same spreadsheet that justified the project now condemns it. The project does not fail because it stopped working. It fails because it started working, at a price nobody planned for.

Clearing this gate means refusing to model the return on pilot economics. You have to forecast the real thing: cost per action at real volume, including retries and guardrails and the long tail of heavy users. I have argued that the honest unit for AI is not cost per call but cost per correct task, because a cheap call that has to run three times to get a usable answer is not cheap. And the deeper structural point, that AI businesses do not get the fat, forgiving margins the old software playbook promised, is the argument in why the SaaS playbook breaks on AI gross margins. If you price and plan as if inference were free, the economics gate is where your pilot goes to die.

The test for this gate: what does one real unit of value cost you to produce at full volume, and is your price comfortably above it. If you cannot answer that in dollars, you have not modeled production. You have modeled a demo.

Gate five: the person who owns the outcome

The last gate is the quietest and it fails more pilots than anyone admits, because it fails them slowly. A demo has a champion, the person who is excited about it. Production needs an owner, a named person who is accountable for the outcome when it is wrong, who has the authority to fix it, and who is measured on whether it works. Enthusiasm gets you a pilot. Accountability gets you production.

The pattern I have watched too many times: a pilot has a passionate sponsor, it demos well, everyone is excited, and then attention moves on. The sponsor gets busy, no single person is on the hook for the production version, the project drifts, and it dies in what people have started calling pilot purgatory, suspended somewhere around month 14, never officially cancelled, just quietly starved. Analysts who look at the wreckage keep concluding the same thing: the root cause is organizational, not technical. The model was capable. Nobody owned the outcome.

There is a second half to ownership, which is being able to prove what the system did when someone with authority asks. A regulated buyer, an insurer, a customer whose data you touched, a court. If the answer is a shrug, the deal stalls or the incident metastasizes. This is the whole reason I treat governance not as a legal chore but as a build decision, which I laid out in the AI governance stack. An accountable owner with a real record of what the system does is the difference between an AI you can stand behind and one you are hoping nobody looks at too closely.

The test for this gate: name the one person who loses sleep if this system is wrong next Tuesday, and describe what they can show a serious buyer about how it behaves. If you cannot name that person, or they have nothing to show, the pilot has not cleared gate five, and it will drift until it dies.

The build-versus-buy trap

There is a decision that sits across all five gates and quietly decides how many of them you have to climb yourself: build or buy. Founders, especially technical ones, have a strong bias to build, because building is the fun part and because it feels cheaper than paying a vendor. The data says that instinct is often wrong.

The MIT work found a stark split. Companies that bought AI tools from focused vendors and partnered well succeeded far more often than companies that tried to build everything internally, by a margin some read as roughly two to one. The reason is exactly the five gates. A good vendor has already climbed the data, reliability, and workflow gates for their narrow slice, over many customers, across cases you have not seen yet. When you build the same thing from scratch, you are re-climbing all of that alone, with one customer’s worth of edge cases, on a founder’s time budget.

The move that works is not “buy everything” and it is not “build everything.” It is to build only the one thing you uniquely understand, the specific pain point where your insight is the edge, and buy the boring plumbing underneath it. Buy the data pipeline, the evaluation harness, the monitoring, the parts that are the same for everyone. Spend your scarce building energy on the narrow place where being generic would lose. The startups that MIT saw going from zero to real revenue in a year were not the ones that built the most. They were the ones that picked one pain point, executed it well, and did not waste months rebuilding infrastructure they could have rented.

This connects to a principle I keep coming back to, which is that anything you do not control is a dependency, and you want your dependencies to be predictable and your differentiators to be yours. I made that case about model providers in how an AI business survives model churn, and it applies here too. Rent the commodity, own the edge. Building the commodity yourself is how a lot of founders spend their whole runway climbing gates that a vendor would have handed them for a monthly fee.

The contrarian take

Here is the part most people get exactly backwards. The common belief is that a better model closes the production gap. Just wait for the next release, the reasoning goes, and the reliability problems and the workflow friction will melt away. I think the opposite is true. A better model widens the gap, and it does it by making the demo more seductive while leaving the last mile exactly as hard as it was.

Think about what each model jump actually does. It makes the demo easier. It lowers the effort to produce something that looks impressive on a curated input, which means more people build more demos faster and feel more certain they have a product. But none of the five gates got easier. The data is still messy. The worst run still has to pass. The workflow still has to accept the thing. The bill at scale is, if anything, larger, because more capable models often cost more and the more capable demo invites heavier use. So the population of confident demos explodes while the population of shipped systems barely moves. That is the divide the surveys keep measuring, and it grows with capability, not shrinks.

The tell that you are on the wrong side of this is the sentence “we will fix that when the next model comes out.” That sentence is almost always covering for an unbuilt gate. The next model will not clean your data. It will not build your evaluation set. It will not fit itself into your customer’s workflow, or model your unit economics, or become the accountable owner of the outcome. Those are your jobs, and no release schedule does them for you.

To be fair to the other side, better models do help. A more capable model widens the range of tasks worth attempting and can genuinely reduce error rates, which matters at the reliability gate. I am not arguing that model quality is irrelevant. I am arguing that it sits above the gates, not inside them, and that treating a model upgrade as a substitute for foundation work is the single most reliable way to stay stuck in the gap. The founders who ship are not the ones with the best model. They are the ones who did the unglamorous foundation work while everyone else waited for the model to save them. In an era where the demo is nearly free, the foundation is the entire moat.

What to do Monday morning

Enough diagnosis. Here is what I would actually do, in order, if I had an AI pilot and wanted it to become a product instead of a nice memory.

Write the production spec before you build the demo. One page. What acceptable looks like on the worst input, not the best. What real data it has to handle. What one action costs at full volume. Whose job it lives inside. Who owns it when it breaks. If you cannot fill that page in, you are not ready to build, you are ready to daydream. Writing it first turns the five gates from surprises into a checklist.

Start at the gate that kills you first, which is almost always data. Before you tune a single prompt, get a real, untouched slice of production data and run your system on it. Whatever breaks is your actual roadmap. This is unpleasant and it is the highest-value hour you will spend, because it moves the death of the pilot from month 14 to this afternoon, where it is cheap to fix.

Define acceptable as a floor, then measure against it. Pick the ten hardest real cases your users will throw at this, run the system five times on each, and look at the worst outcomes. That is your reliability, not the demo. Build this into a small evaluation set you can rerun every time you change something, so quality becomes a number you watch instead of a feeling you have.

Get the real cost of one correct outcome. Not one call, one correct outcome, including retries and checks. Multiply it by your realistic volume, including the heavy users. If the number scares you, better to be scared now than after you have priced a plan you cannot honor. Decide your pricing and your guardrails against that number, not the pilot’s.

Name the owner and ship a thin slice to real users in days, not months. Pick one person accountable for the outcome. Then put the smallest useful version in front of real users in the next few days, inside the tool where they already work, rather than polishing in private. Real users at gate three teach you in a week what a demo hides for a quarter. Shipping small and early is the same discipline I argued for in the two-speed founder framework: move fast on the reversible parts, and let contact with reality, not your own confidence, tell you what to fix next.

None of these steps require a better model. All of them are available to you already. That is the good news hiding inside the grim statistics. The gap is not a technology problem you have to wait out. It is a set of specific, unglamorous jobs, and the founders who do them ship while everyone else demos.

FAQ

What is the production gap?

The production gap is the distance between an AI demo that works in your hands and an AI system that real users depend on. A demo proves the model can do a task once, on an input you chose, while you watch. Production requires it to do the task reliably, affordably, inside a real workflow, on messy real data, thousands of times, with someone accountable when it is wrong. Those are different tests, and most AI work dies in the space between them, not because the model was weak but because the foundation under it was never built.

Why do most AI pilots fail to reach production?

Because a working demo hides its assumptions. The demo runs on clean data you picked, shows the best of a few runs, sits outside the real workflow, costs almost nothing, and answers to no one. Production breaks every one of those assumptions at once. Surveys bear this out: MIT found about 95% of enterprise generative AI pilots delivered no measurable return, IDC found 88% of proofs of concept never reach production, and analysts consistently trace the cause to organizational and data problems rather than model limitations.

Is the production gap a model problem or a data problem?

Far more often a data problem, followed by workflow, economics, and ownership problems. In the readiness research, data readiness, not model quality, was the limiting factor, and 60% of chief executives named disconnected or low-quality data as the biggest barrier to scaling AI. The model that looked brilliant on your curated demo set produces garbage on real production data that is late, messy, and unlabeled. Fixing the data foundation clears more pilots than any model upgrade.

Should I build my AI product or buy the pieces?

Build the one thing you uniquely understand, and buy the boring plumbing underneath it. Research found companies that bought focused AI tools and partnered well succeeded roughly twice as often as those that tried to build everything internally, because a good vendor has already crossed the data, reliability, and workflow gates for their narrow slice across many customers. Spend your scarce building energy on the specific pain point where being generic would lose, and rent the commodity infrastructure.

How long does it take to go from AI pilot to production?

A proof of concept typically takes 6 to 12 weeks, and a real production deployment takes several more months on top of that, often landing decisions somewhere around month 14. But calendar time is not the real constraint. The constraint is whether you built the data pipeline, the evaluation set, the workflow fit, the cost model, and the ownership. Teams that skip those move fast into a wall. Teams that build them ship slower on paper and actually reach production.

What is the biggest hidden cost between pilot and production?

The inference and infrastructure bill at real volume. Production costs routinely land at 50 to several hundred times the pilot baseline, with one documented case jumping 717 times, from 1,500 dollars a month to over a million. The pilot ran on a trickle with no retries or monitoring, and production runs on a flood with all of them. The danger is that the business case was approved on pilot math, so the real bill does not just hurt, it can invert the margin and kill an otherwise working system.

How do I know if my AI pilot will actually ship?

Run five tests. Point it at a random slice of untouched production data and see if the result is acceptable. Run your ten hardest real cases five times each and look at the worst outcomes. Watch a real user do the whole task with your tool in it and time it against doing it without. Compute what one correct outcome costs at full volume and check it is comfortably below your price. Name the one person accountable when it is wrong. If any of those five has no good answer, that gate is where your pilot will stall.

Does building the foundation early slow me down?

It feels slower and it is faster. The foundation work costs you days early. Skipping it costs you the whole project late, when a missing data pipeline or an unmodeled cost surfaces during a real deployment and turns a launch into a rebuild. The readiness research found that leaders who invested in governance and data foundations moved faster to scale, not slower, because they had the confidence and the plumbing to ship. Early foundation is not a tax on speed. It is what lets you keep the speed once reality shows up.

---

I’m Vikas Malpani, a founder writing about building companies in the AI era. For more on the operational side of shipping AI, start with the AI-native founder playbook, then why AI agents fail in production and how founders should think about AI.