The Evaluation Gap: Why AI Agents Fail in Production

This month the industry declared that AI agents grew up. Google’s agent-to-agent protocol moved under the same neutral governance as Anthropic’s Model Context Protocol, the foundation that owns them now counts more than 250 members, and AWS pushed live web search for its agent runtime to general availability. The message from every keynote was the same: agents are out of the pilot phase and into real work across software, healthcare, logistics, and finance.

The same month, LangChain surveyed more than 1,300 people building these systems and found that only 57% have agents in production at all, and that the single biggest thing stopping the rest is not cost, not speed, not compute. It is quality. Thirty-two percent named it as their number one barrier.

Both things are true. The benchmark says ready. Production says not yet. The distance between those two numbers is the most expensive thing in AI right now, and almost nobody measures it on purpose.

I have shipped agents that scored beautifully in testing and then did something no human would approve on day three of real traffic. I have also killed agents that looked mediocre on paper and turned out to be the only ones I could trust. The difference was never the model. It was whether my evaluation told me the truth about the world I was actually going to run in. This is a piece about that gap, why your score lies to you, and how to build an eval you can actually ship behind.

What this covers

• The month agents graduated, and the number that says otherwise
• What the evaluation gap actually is
• The four leaks: where the score drains out
• Leak one: the distribution lie
• Leak two: the consistency lie
• Leak three: the trajectory lie
• Leak four: the drift lie
• The gap widens as agents get more autonomous
• The two worlds your eval lives in
• What to measure instead: the eval stack
• The eval map: four places a team can be
• Evaluation is the missing layer in the agent stack
• The contrarian take: a higher score can make you ship worse
• What to do Monday morning
• Frequently asked questions

The month agents graduated, and the number that says otherwise

Read the launch posts from this year and you would think the reliability problem is solved. Standards bodies are forming. Cloud providers ship managed agent runtimes with governance baked in. Frontier models cluster at the top of every public leaderboard. If you only looked at benchmark scores, you would green-light a fleet of agents on Monday.

Now read the deployment data. A March 2026 survey of 650 enterprise technology leaders found that 78% are running agent pilots and fewer than 15% have reached production scale. MIT’s NANDA initiative put a harder number on it: roughly 95% of enterprise generative-AI pilots produce no measurable return. Gartner expects 40% of enterprises to demote or decommission agents they already shipped by 2027, most of them because a governance gap surfaced only after a production incident.

These two pictures cannot both be the whole story. A benchmark says the capability is there. The field says the capability does not survive contact with real work. The reflex is to blame the model, so teams chase a higher score, swap in a bigger model, and ship again into the same wall.

Here is the part that took me too long to accept. The root cause of a production failure is almost never model accuracy. It is that the number you trusted was measured in a world you do not operate in. The benchmark was a clean room. Your users are not clean. The task was cooperative. Your traffic is not. The score graded the final answer. Your failure lived three tool calls earlier. You did not ship a worse model than the leaderboard promised. You shipped into a gap you never measured.

What the evaluation gap actually is

The evaluation gap is the distance between what your eval score claims and what production delivers. It is not noise. It is not bad luck. It is a structural difference between the conditions a benchmark measures and the conditions your agent runs in, and it is large enough to invert deployment decisions.

Researchers who study this put the typical benchmark-to-production gap at 20 to 40 percentage points, driven by task distribution shift, dependence on the exact scaffolding the benchmark used, and contamination. One multi-dimensional evaluation framework measured an average gap near 37 points across enterprise agent tasks, along with something even scarier for anyone signing the invoice: up to a 50x cost variation between systems that scored about the same on accuracy. Two agents can post the same headline number and differ by a factor of fifty in what they cost to run. The score you were shown hides both facts.

Think about what that means for a decision. If a model shows 88% on a public benchmark and your production reality is 55%, you did not buy a 12% risk. You bought a 45% risk, and you priced it as if it were 12%. Every downstream choice, how much human review to staff, what you promise a customer, how much autonomy to grant, was made against a number that was wrong by more than a factor of three.

The gap is not a reason to stop using benchmarks. It is a reason to stop treating a benchmark score as a deployment decision. A leaderboard tells you an agent can do a thing under favorable conditions. It says almost nothing about whether it will do that thing, every time, on your data, along a path you would be comfortable defending. Those are different questions, and the whole discipline of agent evaluation is the work of telling them apart. This is a different failure than the one I wrote about in the production gap that kills AI pilots, which is about the organizational distance between a demo and a shipped system. This gap is narrower and meaner: it is the distance inside the number itself.

The four leaks: where the score drains out

A benchmark score does not fall to the production number in one step. It leaks out in four places, and each leak has a different cause and a different fix. I draw it as a waterfall because that is what it feels like to watch a 90 turn into a 53. The score enters high, and four separate drains pull it down before it reaches the world.

The illustration uses round numbers, not a measured average, and the sizes are meant to make the shape memorable rather than to claim your leaks split exactly this way. The point is the anatomy. Distribution and consistency usually take the biggest bites. Trajectory is smaller in headline points but is where the ugliest failures hide, because a wrong path can still produce a right-looking answer. Drift is small on any given day and unbounded over a quarter, because it never stops. Fix them in the wrong order and you polish the small drains while the big ones keep pouring. Let me take each one.

Leak one: the distribution lie

A benchmark is a curated set of tasks. Curated means someone chose inputs that are legible, well-formed, and solvable. The users in a benchmark are cooperative by construction, because a script is generating their turns. Nothing is misspelled unless the test wanted it misspelled. No one pastes half a spreadsheet into the chat, changes their mind in the middle, or asks the agent to do something adjacent to what it was built for.

Your production traffic is the opposite. It is messy, out of distribution, and occasionally hostile. Real inputs arrive malformed. Real users contradict themselves and expect the agent to keep up. A share of your traffic is people actively trying to make the thing misbehave, and another share is people who are confused about what they even want. The benchmark measured the agent on the ten percent of inputs that look like a demo. Production runs it on the other ninety.

This is the leak that inflates every public score, and it is why frontier models cluster so tightly at the top. When top systems on a coding benchmark compress to within five percentage points of each other, that is not proof they are all excellent. It is a sign the benchmark has stopped discriminating, because its curated distribution is easier than the world. The most capable-looking number is often the one measured on the least realistic inputs.

The fix is not a better public benchmark. It is your own data. The only way to close the distribution leak is to build an evaluation set out of your real traffic, including the malformed, adversarial, and half-finished inputs that a curated set would have thrown out. A golden set of fifty tasks pulled from your logs will predict your production behavior better than a benchmark of five thousand tasks pulled from someone else’s. This is the same lesson that shows up in how computer-use agents behave once they touch real software: the demo runs on the tasks the demo chose, and the cliff appears the moment the inputs get real.

Leak two: the consistency lie

This is the leak that changed how I think about the whole problem, so I want to slow down on it.

Almost every score you see is a pass@1 number. It measures whether the agent got the task right on a single try, usually averaged across many tasks. That is a fine metric for a research paper. It is a terrible metric for a system that has to work every single time a customer hits it.

The metric that matches production is pass^k, said as “pass hat k.” It measures the fraction of tasks the agent solves on all k independent runs, not just one lucky one. It is the every-time test. And it is brutal. A model that posts a 90% single-run success rate can drop to roughly 57% consistency across eight runs. Same model, same tasks. The only thing that changed is that you asked it to be right eight times in a row instead of once.

Why does this happen? Because failures are close to independent across runs, and independence compounds. If a single run fails ten percent of the time and those failures do not line up, the chance of eight clean runs in a row is 0.9 to the eighth power, which is about 43%. Real systems land a little above that floor, near 57%, because their failures correlate somewhat. Either way, the every-time number is nowhere near the once number, and production only ever pays you the every-time number.

The public leaderboards make this worse by freezing at flattering snapshots. The best-known tool-and-user benchmark still shows its headline board on a late-2024 model set, where the top system scored about 69% on retail tasks and 46% on airline tasks on a single try. Those are single-try numbers on curated tasks. Ask that same agent to be right eight times in a row, on your data, and you are looking at a very different business.

The fix is to stop reporting pass@1 to yourself. Run every eval task k times, set k to at least five, and gate your ship decision on pass^k, not on the average. A system that is right on average but not right in a row is not a product. It is a demo with good luck. This is the quantitative bones under the softer point I made in the reliability paradox: the more polished an agent looks, the more its rare failures cost, because you and your users have stopped bracing for them.

Leak three: the trajectory lie

A benchmark almost always grades the final answer. Did the database end in the right state, did the returned string match the expected output, did the code pass the test. That is the endpoint. Production failures live along the path.

An agent is a sequence of steps. It reads context, calls a tool, reads the result, calls another tool, updates its state, and eventually produces an answer. Every one of those steps can fail on its own. A tool returns malformed JSON and the agent keeps going on garbage. The reasoning drifts one step off the goal and never recovers. The agent calls the right tools in the wrong order and corrupts a record. It loops. It spends forty dollars of compute to answer a two-dollar question. And here is the trap: some of those broken paths still land on a correct-looking final answer, so an endpoint-only eval scores them as passes.

The measured size of this leak is uncomfortable. Agents evaluated on final-output quality alone pass 20 to 40 percent more test cases than a trajectory-level evaluation reveals. In other words, up to four in ten of your “passes” are agents that reached a plausible answer through a path you would never sign off on if you watched it. The critical failure surface is at the step level: tool-call arguments, state propagation, and goal-alignment drift. None of it shows up in the final answer.

The fix is trajectory evaluation. Grade the path, not just the destination. Score whether the agent called the right tools, in a sane order, with valid arguments, at an acceptable cost, and stayed aligned to the goal at each step. The tooling for this is real now. Trajectory evaluators can match an agent’s exact sequence of messages and tool calls against a reference, or use a model as a judge to assess whether the path was reasonable. You do not need to build it from scratch. You do need to decide that a right answer reached the wrong way is a failure, because in production it is one that has not billed you yet. Keeping the path safe is the same discipline as deploying agents without losing control, just measured before the incident instead of after it.

Leak four: the drift lie

The first three leaks are static. You could, in principle, measure them once and know your true production number today. The fourth leak makes that number expire.

A benchmark is a frozen snapshot. Your world is not. The model provider ships an update and the same prompt behaves differently. An upstream API changes its response shape. Your users learn new tricks. A new category of request appears that did not exist when you built your eval set. Seasonality moves your traffic mix. None of these touch your benchmark score, and all of them move your production reality. A number that was true last quarter is a guess this quarter.

Public benchmarks make this vivid because they visibly rot. A benchmark saturates when frontier models all cluster near the ceiling and score differences stop meaning anything. The most famous knowledge benchmark hit that wall first. The hard reasoning sets are following. And because many benchmark tasks are drawn from public sources, contamination creeps in: a coding benchmark built from public repositories cannot tell you how much of a model’s score is skill and how much is memorization, and the providers do not disclose it. Some sets carry annotation error rates above fifty percent. The number was never as solid as it looked, and it degrades from the day it is published.

The fix is to treat evaluation as a running process, not a launch gate you pass once. You need online evaluation: scorers attached to live production traffic that catch a regression when it happens, not three weeks later when a customer complains. You need to re-run your golden set on every model change and every prompt change. The number you trust has to be refreshed as often as the world that produces it, which for most agents means continuously. An eval you ran in March is telling you about a March that no longer exists.

The gap widens as agents get more autonomous

Here is the trend that should worry you. The four leaks are not fixed in size. They scale with how many steps an agent takes, and the whole industry is pushing agents toward more steps. Longer horizons, multi-agent handoffs, deeper tool chains. Every one of those adds surface area to the exact leaks that already drain the score.

The consistency leak compounds the fastest. A one-step classifier that is right 95% of the time barely notices the every-time test. A twenty-step agent that is right 95% at each step, with independent failures, finishes the whole task cleanly only 0.95 to the twentieth power of the time, which is about 36%. Same per-step quality, and the end-to-end reliability fell off a cliff, purely because you asked it to chain more decisions. The trajectory leak scales the same way, because twenty steps is twenty places for a bad tool call or a wrong order to slip in, and only some of them show up in the final answer.

This is why “the model got better” does not rescue you. Capability per step keeps climbing, but the number of steps per task is climbing faster, and reliability is the product of the steps, not the average. More autonomy is not a smaller eval problem, it is a bigger one, and the agents most likely to be quietly failing are the ambitious multi-step ones that demo the best. It is also why so many shipped agents end up getting pulled: the gap that was invisible at one step becomes a reason to decommission the agent at twenty. And the human answer, more oversight, has its own ceiling, because watching every step of every agent is exactly the decision load that burns a team out. The measurement has to carry the weight, because the humans cannot.

The two worlds your eval lives in

Step back and the four leaks are really one thing: a benchmark measures the wrong world. It is worth seeing the two worlds side by side, because the differences are systematic, not random. Every row below is a place where the benchmark is easy and production is hard, and the gap is just the sum of the rows.

Dimension	Benchmark world	Production world
Inputs	Clean, curated, well-formed	Messy, malformed, out of distribution
Users	Cooperative, scripted	Confused, contradictory, sometimes hostile
Metric	pass@1, averaged (right once)	pass^k (right every time)
Graded on	Final answer only	The whole trajectory, tools and order and cost
Data	Someone else’s public set	Your traffic, your edge cases
Time	Frozen snapshot	Drifting weekly with models and users

When you see it laid out, the surprise disappears. The benchmark was never trying to predict your production behavior. It was trying to compare models under controlled conditions, which is a real and useful job. The mistake is ours: we take a number built to rank models and use it to approve a deployment. Those are different questions, and the eval you need for the second one has to close every row in that table.

What to measure instead: the eval stack

If a single benchmark number cannot green-light a deployment, what can? Not a better number. A stack of them, each catching what the layer below it misses. I think of production-grade evaluation as five layers, and you trust a system only when it clears all five, in order.

Layer	What it catches	What it misses
1. Public benchmark	Gross capability gaps, can it do this at all	Your distribution, consistency, the path
2. Your golden set	Distribution gap on your real tasks	Rare edge cases, consistency if run once
3. Trajectory eval	Bad tool calls, wrong order, runaway cost, unsafe path	Inputs not in your set yet
4. Adversarial and edge	Messy inputs, hostile users, injection, the ugly tail	Genuinely novel drift
5. Online eval	Live regressions, drift, your true failure rate	Anything you forgot to score

Layer one is the public benchmark, and it does exactly one honest job: it disqualifies. If a model cannot clear a relevant public benchmark, it cannot do the task at all, and you can stop. It can never approve a deployment, only rule one out.

Layer two is your golden set, thirty to a hundred tasks pulled from your real traffic, with expected outcomes you defined. This is the layer that closes the distribution leak, and it is the highest-value hour of work in the whole stack. Most teams skip it because building it is tedious. That tedium is the moat.

Layer three is trajectory evaluation, grading the path and not just the answer, which closes the trajectory leak and catches the cost blowouts and unsafe sequences that endpoint grading waves through.

Layer four is deliberate adversarial and edge testing. You write the malformed inputs, the contradictory users, the prompt-injection attempts, and the requests that sit just outside the agent’s job, because your users will supply all of them for free and you would rather meet them in a test.

Layer five is online evaluation. You attach scorers to a sample of live traffic, a model acting as judge plus rule-based assertions, and you shadow or canary every new version before it takes real load. This is the only layer that measures the real world, and it is the one that never turns off. The layers below it tell you what should happen. This layer tells you what is happening. This is also where an agent’s own persistent memory and behavior get audited over time, because drift shows up in the traces long before it shows up in a complaint.

The eval map: four places a team can be

Two questions decide whether your evaluation actually protects you. First, what are you measuring: only the final output, or the full trajectory? Second, where are you measuring it: on a curated benchmark, or on your live traffic? Cross those two and you get four places a team can sit, and only one of them is safe to ship from.

Most teams sit in Benchmark Theater. They have a high public score, a slide with a green number, and no idea what their agent does on their own traffic. It feels like rigor and provides none. Clean-Room Rigor is the team that built careful trajectory evals but only ever runs them on curated tasks, so they are precise about the wrong world. Blind Monitoring is the team that watches production dashboards but only tracks final outcomes, so they see that something is wrong without being able to see where. Only the top-right corner is safe: trajectory-level evaluation, run on your live traffic, every time. Everything else is a way to feel measured while the gap stays open.

Evaluation is the missing layer in the agent stack

Look at where teams have put their money. Better models. Persistent memory. Orchestration and multi-agent coordination. Tool integrations. Guardrails and access control. Every one of those is a layer of the agent stack, and every one of them has a budget line and a vendor. Evaluation is the layer that tells you whether any of the others actually work on your traffic, and it is the one most teams treat as an afterthought. On my own map of where the real AI work is, this is the layer with the least attention and the highest payoff, because it is the only one that turns all the other investments into something you can trust.

It helps to separate evaluation from its neighbors, because the word “eval” gets stretched to cover several different jobs. Using an eval as a gate to choose which model to route a request to is a related but different job: that is a live routing decision between options, not a readiness verdict on your whole system. The human review layer, where people catch the agent’s mistakes, is downstream of this: automation bias erodes that layer over time, which is exactly why you cannot lean on human vigilance as your only detector. And once you accept that agents are becoming the users of your software, the case for a real eval instrument gets stronger, because the thing consuming your product no longer complains in words when something breaks. It just quietly does the wrong thing at scale.

Put simply, evaluation is the instrument that measures the whole stack. Skip it and every other layer is unmeasured spending. Build it and you finally know which of your investments earned their keep.

The contrarian take: a higher score can make you ship worse

Here is the part most people get backwards. They treat a higher benchmark score as strictly good. Ship the model with the bigger number. But a higher score can make you ship a worse product, and the mechanism is not subtle.

A high benchmark number buys confidence, and confidence postpones the only evaluation that matters. When the leaderboard says 92%, the golden set feels like overkill, the trajectory eval feels like process for its own sake, and online monitoring feels like something you will add after launch. The better the flattering number, the more it argues against doing the real work. So the teams with the most impressive benchmark scores are often the ones flying the blindest in production, because the score talked them out of building an instrument. I have watched a worse-scoring agent with a trusted eval quietly outship a better-scoring agent that nobody could see inside.

That flips the whole frame. Evaluation is not a score you report. It is a detector you trust. Score-thinking asks “how high did it get.” Detector-thinking asks one question instead: would this eval have caught the last failure? Pull up your last five production incidents and ask it of each one. If your current evaluation would not have caught them, your score is decoration, no matter how high it is. If it would have, your score is an instrument, even if it is lower than the leaderboard’s. A mediocre agent with a detector you trust beats a state-of-the-art agent you cannot see inside, every time, because the first one tells you before your customer does and the second one does not.

The honest counterweight, because I do not want to leave you thinking benchmarks are useless: they are not. They are a genuine filter for capability. They answer “can this class of model do this class of task at all,” which is a real question, and a model that fails a relevant benchmark should be ruled out fast. The error is using a capability filter as a reliability approval. Use benchmarks to disqualify, never to green-light. The green light only comes from your own traffic, on the full path, run enough times to watch it fail.

That is the law I would tape to the wall. A benchmark score is a promise made in a world that is not yours. The only eval that counts runs on your traffic, on the whole path, enough times to watch it fail. And if you cannot reproduce the failure, you have not fixed it, you have just stopped seeing it.

What to do Monday morning

You do not need an evaluation platform or a research team to start closing the gap. You need about a day and the willingness to stop trusting a number you did not generate. Here is the order I would run it.

First, run the detector test on your history. Pull your last five production failures, the real ones that reached a user or cost real money, and for each ask whether your current evaluation would have caught it before shipping. Write the answer down. This one exercise usually ends the argument about whether you have an eval problem.

Second, build a golden set. Pull thirty to fifty real tasks from your logs, weighted toward the messy and edge cases you would rather not think about, and write down the expected outcome for each. This is your layer two, and it will predict production better than any public benchmark. Do not aim for a thousand tasks. Aim for fifty honest ones this week.

Third, add trajectory grading. For each task, score not just the final answer but the path: did it call the right tools, in a sane order, with valid arguments, at an acceptable cost. Use an existing trajectory evaluator or a simple model-as-judge rubric. Decide, explicitly, that a right answer reached through a bad path is a failure.

Fourth, switch from pass@1 to pass^k. Run every task at least five times and gate your ship decision on the fraction that succeed on all runs, not the average. If that number embarrasses you, that is the point. Better to be embarrassed by an eval than by a customer.

Fifth, turn on one online scorer. Attach a single model-as-judge check to a sample of live traffic, and put every new version behind a shadow or canary run before it takes full load. You do not have to instrument everything on day one. You have to stop shipping blind. This is the same discipline that keeps you from paying the efficiency trap in reverse, spending on speed you cannot verify. A cheaper agent you can see inside is worth more than a faster one you cannot.

Frequently asked questions

What is the evaluation gap in AI agents?

The evaluation gap is the distance between what an agent scores on a benchmark or demo and what it actually delivers in production. Researchers measure it at 20 to 40 percentage points on typical enterprise tasks, driven by task distribution shift, dependence on the benchmark’s scaffolding, and data contamination. It matters because deployment decisions get made against the benchmark number, which can overstate real reliability by more than a factor of three.

Why do AI agents pass benchmarks but fail in production?

Because benchmarks measure a different world. They use clean curated inputs, cooperative scripted users, single-try scoring, and final-answer grading, on a frozen snapshot of tasks. Production has messy inputs, confused and sometimes hostile users, needs the agent to be right every time, fails along the path rather than at the endpoint, and drifts as models and users change. Each of those is a separate leak, and together they turn a high score into a much lower production number.

What is pass^k and why does it matter more than pass@1?

Pass@1 measures whether an agent gets a task right on a single try, usually averaged across tasks. Pass^k measures whether it gets the task right on all k independent runs, which is the every-time consistency production actually requires. The two diverge sharply: a model with a 90% single-run success rate can drop to around 57% across eight runs, because near-independent failures compound. Gate your ship decision on pass^k, because customers experience the every-time number, not the average.

What is trajectory evaluation for AI agents?

Trajectory evaluation grades the full sequence of an agent’s actions, the tool calls, their arguments, their order, the cost, and whether each step stayed aligned to the goal, rather than only checking the final answer. It matters because agents graded on final output alone pass 20 to 40 percent more cases than trajectory evaluation reveals, since a broken path can still land on a correct-looking answer. Tools like trajectory evaluators and model-as-judge scorers make it practical without building from scratch.

Are public benchmarks like SWE-bench and GAIA still useful?

Yes, for one job: disqualifying models that cannot do a task at all. They are unreliable for approving a deployment, because they saturate as frontier models cluster near the ceiling, they suffer contamination when tasks come from public sources, and their curated distribution is easier than production. Use them as a capability filter to rule models out, and never as a reliability green light.

How do you evaluate an AI agent on your own data?

Build a golden set: thirty to a hundred real tasks pulled from your production logs, weighted toward messy and edge cases, each with an expected outcome you defined. Run the agent against them repeatedly, score both the final answers and the trajectories, and treat this set as your source of truth over any public benchmark. Fifty honest tasks from your traffic predict production better than thousands from someone else’s.

What is LLM-as-a-judge and when should you use it?

LLM-as-a-judge uses a model to score another model’s output or trajectory against a rubric, which scales evaluation to cases where a hard-coded check is impractical, like judging whether a reasoning path was sound or a response was helpful. Use it for trajectory grading and for online evaluation on live traffic, paired with rule-based assertions for the checks that can be exact. Validate the judge against human labels on a sample so you know it is measuring what you think.

What is a good ship gate for deploying an AI agent?

A defensible gate has four parts: the agent clears a relevant public benchmark, it passes a pass^k threshold on your golden set with k of at least five, its trajectories are scored for tool use, order, cost, and safety, and one online scorer plus a shadow or canary run is live before it takes full traffic. If any layer is missing, you are shipping on a number that does not describe your world.

One note, since evaluation touches how much people trust automated systems: the goal is not to trust agents less, it is to trust them for the right reasons. A measured agent you can see inside deserves more autonomy than an unmeasured one with a higher score. Build the instrument, then extend the trust.