Evals are the product, not the model

Every week this year another AI agent ships with a demo that looks like the future. A founder types one sentence, the agent books the travel, reconciles the invoices, files the ticket, and the room claps. Then the same product meets real users and quietly falls apart. The booking goes to the wrong city. The invoice reconciliation is off by a digit nobody catches for a month.

The number that stuck with me this year came from MIT’s NANDA initiative. In its 2025 report, The GenAI Divide: State of AI in Business, the group studied 300 public AI deployments and found that about 95 percent delivered no measurable impact on the bottom line. Only around 5 percent reached real revenue. The report was clear that the cause was not model quality. It was a learning gap: the tools did not adapt to the work, and the teams had no way to tell a good output from a bad one at scale.

That is the whole game to me. The distance between the demo and the product is not a better model. It is whether anyone built a way to know, every day, whether the thing is actually working. I call that the eval loop, and I have come to believe it is the real product. The model is a commodity you rent. The eval loop is the asset you own.

Why 95 percent of AI pilots stall

A demo proves a system can succeed once. A product has to succeed on the thousandth try, on inputs nobody imagined, when the model vendor silently ships a new version overnight. Those are different problems, and most teams only build for the first one.

Here is what the MIT work pointed at, and it matches what I see in almost every AI company I talk to, and it is the same enterprise AI gap I have written about before. Teams buy or build a tool, wire it into a workflow, and then have no instrument on the dashboard that tells them if it is getting better or worse. They are reacting to complaints. A user reports a bad answer, someone tweaks a prompt, and nobody can say whether the tweak fixed the real problem or just moved it somewhere they are not looking.

The same blind spot shows up on the engineering side. Sonar’s 2026 State of Code Developer Survey polled more than 1,100 developers and found that 96 percent do not fully trust that AI generated code is functionally correct, yet only 48 percent say they always check it before committing. AI already writes about 42 percent of committed code in that survey, heading for a projected 65 percent by 2027. Werner Vogels, the CTO of Amazon Web Services, has a name for the cost of skipping the check: verification debt. You ship fast, the balance compounds, and one day the interest comes due as an outage or a silent wrong answer that cost a customer money.

Verification debt is the tax on building without evals. And unlike normal technical debt, you often cannot see it, because the output looks right. In that same Sonar survey, 53 percent of developers said code that looks correct but is not reliable was a real drag on their codebase. A plausible wrong answer is more dangerous than an obvious error, because it passes the eye test and goes straight to the customer.

The eval loop is the control system

Think of an AI product as a machine with a feedback controller, not as a clever one shot function. The model generates. Real usage produces outcomes. You measure those outcomes against a standard you defined. The gap feeds the next change. Round and round. That cycle is the eval loop, and it is the one piece of the system a competitor cannot copy by renting the same model you rent.

I draw it like this when I explain it to a team.

None of the four stages is optional, but the one teams skip is the bottom. They build, they ship, they watch a usage chart go up, and they never close the loop with a real measurement. A usage chart tells you people clicked. It does not tell you the answers were right. The eval is the only stage that converts opinion into a number you can argue with.

Anthropic made this concrete in an engineering piece from January 2026 called Demystifying evals for AI agents. Their framing is the cleanest I have seen. Capability evals ask what the agent can do well, and should start at a low pass rate so the team has a hill to climb. Regression evals ask whether the agent still handles what it used to, and should sit near 100 percent. When a capability eval starts passing reliably, it graduates into the regression suite. That graduation is the loop turning: today’s frontier becomes tomorrow’s floor, and you never let the floor drop without knowing.

The reason this is an asset and not overhead is simple. Every eval you write is a piece of judgement about your product that outlives any single model. Swap the model, the evals stay. Hire a new engineer, the evals teach them what good looks like. A year in, your eval set is a written record of everything your product has learned to get right, and no amount of funding lets a competitor skip ahead to owning it.

Reliability is not capability, and it compounds

The most expensive misunderstanding I see is treating a capable model as a reliable one. They are not the same axis. Capability is whether the model can do the task at all. Reliability is whether it does the task every time you ask. A product lives and dies on the second one, and the second one decays fast when tasks have more than one step.

Anthropic names the two measures that matter here. Pass at k is the chance of at least one success in k tries, which is fine when one good answer is enough and a human picks it. Pass to the power of k is the chance that all k tries succeed, which is the one that matters for anything customer facing and unattended. Their example is blunt: a 75 percent per attempt success rate gives you only about 42 percent across three attempts. The math is unforgiving once an agent chains steps, because the whole chain only works if every link holds.

Here is the same arithmetic laid out. If each step in a workflow succeeds with probability p, a chain of n steps succeeds with probability p to the power of n.

Read the bottom right corner. A step that works 90 percent of the time feels great in a demo, and it hands you a 35 percent product once it has to do ten things in a row. Moving each step from 90 to 99 percent is the difference between a toy and a tool, and you cannot do that moving by vibes. You do it by writing an eval for each step, finding the one that fails most, and fixing that one. The eval loop is how you find out which link is weakest instead of guessing.

This is also why I am skeptical of founders who tell me their agent is accurate because they have used it a lot themselves. Using it is pass at k with a human in the seat picking the good run. Your customer gets pass to the power of k with nobody watching. If you have not measured the second number, you do not know your product, you know your luck.

How I would build the first ten evals

The good news is that starting is cheap, and the first version should embarrass you a little. Anthropic suggests 20 to 50 simple tasks drawn from real failures as a starting set. I would go even smaller to get moving: write ten this week. Perfect coverage is a trap that keeps teams from writing any evals at all.

My rule is that every eval comes from a real failure, not from imagination. Go to the transcripts and the complaints you already have. Each bad output you find becomes a test case: this input should produce that kind of result. That keeps your eval set pointed at the problems customers actually hit, instead of the ones that are fun to think about.

A few things I hold to when writing them. Make each task unambiguous enough that two people who know the domain would agree on pass or fail, and give each one a reference answer. Grade the outcome, not the exact path, so the model is not punished for solving the problem a different valid way. Give partial credit on multi part tasks so a near miss does not read the same as a total failure. And test the cases where a behavior should not fire as hard as the cases where it should, or you will optimize one side into the ground and quietly break the other.

Most interesting outputs cannot be graded by a string match, so you will reach for a model as a judge. That works, with guardrails. Calibrate the judge against human verdicts before you trust it, let it answer unknown when it lacks the information to decide, and use a separate judge for each dimension rather than one judge grading everything at once. Then validate it against people on a sample, regularly, because a judge that drifts is worse than no judge, since it lies with a number attached.

The single highest return habit, and the one almost nobody does enough, is reading the raw transcripts. Anthropic put it in two words: read the transcripts. Scores tell you something changed. Only the transcript tells you what and why. I have never seen a team build good evals without someone whose actual job, for part of every week, is to sit and read what the thing really said to users.

Why a passing score can still lie to you

Once you have numbers, the next failure mode is trusting them too much. A score is a measurement, and measurements have bugs.

Anthropic shared an example I keep quoting. Their Opus 4.5 model scored 42 percent on a benchmark called CORE-Bench, which looked like a real ceiling, until they found bugs in the grader. Once the grading was fixed, the same model scored 95 percent. The model had not changed at all. The number had been wrong by more than half. If a frontier model scores zero on many trials of your eval, your first suspicion should be a broken task or a broken grader, not a broken model.

The opposite trap is saturation. An eval that everything passes has stopped telling you anything. Anthropic notes that SWE-bench Verified, a respected coding benchmark, is nearing saturation with scores above 80 percent. When your own eval saturates, it is not a victory, it is a signal to write harder cases that reflect what your best users are now trying to do. A healthy eval set is always a little painful to pass.

Then there is gaming. Any grader can be gamed if passing it does not require actually solving the problem. I design graders on the assumption that the model will find the cheap path through them, because over enough runs it will. If an eval can be passed by pattern matching the format of a good answer rather than producing a good answer, it will eventually be passed that way, and your dashboard will turn green while your product gets worse.

The protection against all three is the same boring discipline: do not trust a score until a human has read the details behind it. The number is a pointer to the transcript, not a replacement for it.

Buying beats building when the discipline is missing

One more finding from the MIT work changed how I advise early teams. Deployments that came from buying a specialized vendor or forming a partnership succeeded roughly 67 percent of the time. Internal builds succeeded at about a third of that rate. The instinct to build everything yourself is often the instinct to own the demo while skipping the discipline that makes it a product.

I read that gap as a question about where your eval loop will actually live. A specialized vendor has usually already paid the cost of building evals and reading transcripts for your use case, across many customers. If you build in house, you are not just building the feature, you are signing up to own the feedback controller too. That is worth doing when the capability is core to your company and you intend to run the loop seriously. It is a bad trade when you build in house to feel in control and then never staff the measurement that control requires.

Evals are a waste of time before you have users

The strongest case against everything above is a speed argument, and I take it seriously. The claim goes: before product market fit, you do not know what good looks like, so writing evals is premature optimization. You should be shipping, talking to users, and changing direction weekly. Formal evals freeze an answer you have not found yet, and the overhead slows the only thing that matters early, which is learning.

There is truth in that. A pre users team writing a hundred formal evals for a product that will pivot in a month is polishing the wrong thing. But I think the conclusion is backwards. The point of early evals is not to lock in an answer, it is to make your learning legible. Ten lightweight evals pulled from your first twenty real conversations do not slow learning down, they are how you find out whether last week’s change helped or hurt. The teams that skip evals early are not learning faster. They are learning by anecdote, and they usually cannot tell the difference between a product that is working and a founder who is persuasive. Start small, but start.

Write ten evals before you write another feature

If you are deciding where to place your bets in AI, a working eval loop is one of them. If you take one thing from this, make it a decision about where your next week goes. Before you add the next feature to your AI product, write ten evals for the feature you already shipped.

Concretely, this week I would do four things. Open your last fifty real user sessions and read every transcript, with no goal except to notice where the output was wrong or weak. Turn the ten worst into test cases, each with an input and a clear statement of what a good result would be. Score your current product against those ten and write the number down, because that number is your honest baseline and it will be lower than you hope. Then fix the single worst case and watch the number move.

That is the whole loop at the smallest size that still works. Do it again next week with ten more. Within a quarter you will have a few hundred evals that encode exactly what your product is supposed to do, a baseline you can defend to a customer or an investor, and the one asset in your company that does not reset to zero every time the model vendor ships a new version. The model is rented. The loop is yours. That is the part worth building.

Questions founders ask me

What is the difference between a capability eval and a regression eval

A capability eval measures whether the product can do a task well, and I expect it to start with a low pass rate so there is room to improve. A regression eval measures whether the product still does a task it already handled, and I expect it to sit near 100 percent. The pattern I follow comes from Anthropic: when a capability eval starts passing reliably, it graduates into the regression suite, so a new skill becomes a protected floor you do not drop. One set tells you where you are growing. The other stops you from quietly going backwards when you change a prompt or a model.

How many evals do I need before I ship

Fewer than you think to start, and more than you want forever. I would not block a first ship on evals, but I would not run a product past its first real users without them. Anthropic suggests a starting set of 20 to 50 tasks pulled from real failures. I tell founders to write ten this week and add ten a week. The goal early is not coverage, it is a habit and an honest baseline number. Coverage grows on its own once reading transcripts and writing cases is part of the weekly rhythm rather than a project you keep postponing.

Can I just use a model to grade the outputs

Yes, with care. A model as a judge is often the only practical way to grade outputs that cannot be checked with a simple match. The guardrails matter though. Calibrate the judge against human verdicts before you trust it, let it answer unknown when it does not have enough information, and use a separate judge for each dimension instead of one judge for everything. Then keep validating it against people on a sample. An uncalibrated judge does not save you work, it gives your mistakes a confident number and sends them to production faster.

My benchmark scores look great, so why does the product still feel unreliable

Two likely reasons. First, you may be measuring pass at k, the chance of at least one good answer in several tries, while your customer experiences pass to the power of k, the chance that every step succeeds with nobody watching. Those numbers diverge fast as tasks gain steps. Second, your eval may have saturated or be gameable, so a green dashboard no longer reflects real quality. The fix for both is the same: measure end to end success unattended, write harder cases, and read the transcripts behind the score instead of trusting the number alone.

Should I build my AI feature in house or buy it

It depends on whether you will actually run the feedback loop. MIT found that bought or partnered deployments succeeded about 67 percent of the time, while internal builds succeeded at roughly a third of that rate. A good vendor has usually already paid the cost of building evals and reading transcripts across many customers. Build in house when the capability is core to your company and you will staff the measurement seriously. Buy when you want the outcome and would not actually commit the people to own the eval loop, because owning a feature you do not measure is how pilots end up in the failing 95 percent.

How do I know when an eval has stopped being useful

When it stops surprising you. An eval that your product passes every time has become a regression check, which is useful as a floor but tells you nothing new about where to improve. When most of your capability evals saturate, that is the signal to write harder cases drawn from what your most demanding users are now attempting. I also retire or rewrite any eval I catch being gamed, where the product passes without really solving the problem. A good capability eval should always be a little uncomfortable to pass, because that discomfort is where the next improvement hides.

I’m a four-time founder building AI-native companies. I write about the founder mindset, AI opportunities, and the mechanics of building with AI at vikasmalpani.com.