Cost Per Correct Task: The Real AI Productivity Metric

Anthropic put a staggering number on the table this month. In its report on building software with its own models, the company said more than 80 percent of the code merged into its production codebase is now written by Claude, with leadership estimating the real figure is north of 90 percent once you count scripts and experiments. The headline that traveled fastest was the throughput one: by the second quarter of 2026, a typical engineer there was merging roughly eight times as much code per day as the same kind of engineer did in 2024.

Eight times the output. It is the kind of number that makes a founder feel behind before breakfast.

Here is the durable version, the part that does not depend on whose model wrote what this quarter. Eight times the output is not eight times the value. When the cost of producing a unit of work falls toward zero, every metric that counts units of work stops measuring anything useful. Lines of code, tokens spent, pull requests opened, tasks attempted, percent of work authored by a machine: all of them inflate at exactly the moment they stop correlating with results. The thing that still costs money, still takes your scarce attention, and still decides whether the business actually moved forward is a different number entirely.

That number is the cost per correct task. Not the cost to attempt a task. The all-in cost to land one task that is right, shipped, and does not come back to bite you. I have run two companies where AI now does most of the first-draft production, and the single most useful shift in how I run them was to stop counting what the machines produce and start counting what survives contact with reality. This is the metric, why it beats every volume number you are being shown, and how to instrument it by Monday.

Table of Contents

• The Output Illusion
• Cost Per Correct Task, Defined
• Why Every Volume Metric Became a Vanity Metric
• Where the Cost Actually Moved
• First-Pass-Correct Rate: The Number Under the Number
• How to Compute Cost Per Correct Task
• Buying and Pricing AI by Correct Task
• The Contrarian Take: Acceptance Rate Is a Trap
• What to Do Monday Morning
• FAQ

The Output Illusion

The most honest study of AI and developer output is also the most uncomfortable. A controlled experiment with experienced open-source developers measured how long real tasks took with AI tools allowed versus disallowed. Before the study, the developers predicted AI would make them about 24 percent faster. They were actually 19 percent slower with the tools on. And the part that should keep every founder up at night: even after living through the slowdown, they still believed AI had sped them up by roughly 20 percent.

Sit with that gap. The same people, doing the same work, were wrong about their own productivity by close to 40 points, in the optimistic direction, while holding the stopwatch. The feeling of speed and the fact of speed had come apart. AI is very good at producing the feeling. A draft appears in seconds. Your screen fills. Something that looked like an hour of work arrives instantly, and your brain books the win.

The fact is hiding one layer down, in the work you do after the draft appears. Across teams that adopted AI heavily, rework rates, the share of developer time spent fixing newly shipped code, climbed 30 to 60 percent within six months. Code review times rose sharply as humans waded through machine-generated changes they had not written and did not fully understand. Incidents per pull request went up even as pull requests per author went up, which is the worst possible combination: more shipping, more breaking, per unit of shipping. One widely cited figure put the first-year cost of heavy AI coding adoption at around 12 percent higher once you add the review overhead, the extra testing burden from more defects, and the churn from code that gets rewritten soon after it lands.

None of that shows up in throughput. Throughput counts the draft and stops counting. The rework, the review, the incident at 2 a.m., the rewrite three weeks later all happen after the metric has already recorded a win. So the dashboard goes up and to the right while the business does not move, and everyone feels faster while being slower. That is the output illusion, and it is not a software problem. It is a measurement problem. You are counting the cheap thing and ignoring the expensive thing.

Cost Per Correct Task, Defined

A correct task is a unit of work that is finished. Not drafted, not attempted, not eighty percent there. Finished: it does what it was supposed to do, it shipped, and it did not generate a downstream cost that erased its value. A support ticket that the customer never reopens. A pull request that lands and stays. A blog post that goes live and does not need a rewrite. A contract clause that holds up. The bar is deliberately strict, because the whole point is to stop crediting work that is not actually done.

Cost per correct task is the total cost of getting to that finished state, divided by the number of finished states you reached.

The total cost is not just the tokens. It is generation cost, which is now small, plus verification cost, which is the human or automated effort to check the work, plus rework cost, which is everything spent on the attempts that were not right the first time, plus failure cost, which is the cleanup when something wrong slips through and reaches a customer. The denominator is only the tasks that ended up correct. Attempts that were abandoned, drafts that were thrown away, and shipped work that later broke do not count toward the bottom of the fraction, even though they add to the top.

The funnel is the whole idea in one picture. A thousand things come in the top almost for free. They pass through a gate where checking them is the expensive part. Some fail and circle back through rework, paying the gate cost again and again without producing anything new. A few hundred come out the bottom finished. And a handful of those break later and have to be subtracted back out. The metric that matters is the cost of the whole machine divided by what comes out the bottom intact.

Once you frame it this way, the eight-times-throughput headline reads differently. Eight times more entering the top of the funnel, with a verification gate that did not get eight times cheaper and a first-pass-correct rate that may have gotten worse, does not mean eight times more out the bottom. It can mean the same number out the bottom at a higher total cost, which is a higher cost per correct task. More activity, less value, and a dashboard that congratulates you the entire time. This is the same trap I wrote about in the AI eval budget: an unmeasured reliability tax that only becomes visible when you decide to measure it.

Why Every Volume Metric Became a Vanity Metric

A vanity metric is a number that goes up reliably, feels like progress, and does not change any decision you make. In the AI era, almost every output metric crossed that line at once, and for the same reason: the input they measure stopped being scarce.

Lines of code used to be a rough proxy for effort because a human had to type and think through each one. Now a model emits a few hundred lines before you finish reading the prompt, so the count tells you about the model’s verbosity, not your progress. Tokens spent measures how much the model talked, which is closer to a cost than an output. Pull requests merged sounds like delivery, but if review time is rising 91 percent and a meaningful share of those merges later cause incidents, the count is measuring motion, not distance. Percent of code authored by AI is the purest vanity metric of the batch: a company can push that number to 90 and learn nothing about whether its software got better, because the metric is about who typed, not whether what got typed was right.

Tasks attempted is the one that fools smart people, because it feels outcome-shaped. An agent ran a hundred workflows today. But a workflow that ran is not a workflow that worked. One large panel of more than 8,000 users found a mean task completion rate around 75 percent, which sounds fine until you notice it averaged away a 21-point spread between the best and worst agents and sat alongside lower user trust than plain manual search. A 75 percent completion rate means one in four tasks did not complete, and you are paying for all of them, including the ones that produced confident garbage you then had to catch.

The test for whether a metric is vanity is simple: would a worse version of your operation also make this number go up? A team that ships sloppier code faster will post more lines, more tokens, more merged PRs, and a higher AI-authored percentage than a careful team. If your worst possible behavior improves the metric, the metric is not measuring quality, it is measuring volume, and volume is the thing that just became free. The table below is the translation guide I keep taped next to my own dashboards.

Where the Cost Actually Moved

To see why cost per correct task is the right unit, follow the money on a single task across two eras. In 2024, when a human did most of the work, the cost of a task was dominated by making it. Someone sat down, thought it through, and produced the thing. Checking it was real but small by comparison, because the person who made it already understood it, and rework was contained because they had reasoned their way to the answer rather than guessing at a plausible one.

By 2026 the shape inverted. Generation, the part that used to cost the most, collapsed toward zero. But the work did not disappear, it moved downstream. Now the expensive parts are verification, because someone has to check output they did not produce and may not fully grasp, and rework, because a confident draft that is subtly wrong takes longer to fix than a blank page takes to fill, and failure, because when wrong work slips through it reaches a customer and the cleanup is the most expensive cost of all.

This migration is why the productivity numbers feel contradictory. Controlled studies of scoped, well-defined tasks really do show large speedups, sometimes 30 to 55 percent, and a survey of thousands of developers found AI cut time on routine coding by close to half. Those gains are real, and they live in the generation layer. But a broad look at adoption found that while 84 percent of developers now use AI tools, organization-level productivity gains often land around 10 to 30 percent, and sometimes vanish. The speedup in generation is real and the disappointment at the business level is also real, because the cost moved to a layer the generation metrics never watched. You sped up the cheap step and slowed down the expensive one.

There is a slower-burning version of this cost too. Teams report that AI can generate code five to seven times faster than anyone can actually understand it, which builds what some now call comprehension debt: software that works today but that no one can confidently modify, debug, or own six to eighteen months from now. That is failure cost with a long fuse. It will not appear on this quarter’s throughput chart. It will appear the day a customer hits a bug in code your team shipped but never understood, and that is when cost per correct task, measured honestly over the life of the work, would have warned you. I dug into the production side of this in why AI agents fail in production.

First-Pass-Correct Rate: The Number Under the Number

If cost per correct task is the headline, first-pass-correct rate is the engine underneath it. It is the share of tasks that come out of the funnel right the first time, with no trip through the rework loop. It is the single biggest lever on your cost per correct task, and almost nobody measures it, because the volume dashboards have no place to put it.

The math is unforgiving in a useful way. Imagine generation is nearly free and the real cost of a task is the verification pass, which you pay once for work that is right and pay again for every lap through rework. If your first-pass-correct rate is 80 percent, most tasks clear the gate once and your cost per correct task stays close to the cost of one verification. If that rate falls to 40 percent, most tasks go around at least twice, some three times, and your cost per correct task can double or triple even though generation got cheaper. The cruel part is that cheaper generation can lower your first-pass-correct rate, because when output is free you ask for more of it, accept rougher drafts, and push more marginal work into the funnel, every piece of which still has to clear the same expensive gate.

This is the link to evaluation. The reason I keep an eval budget, which I broke down in the evals playbook for solo founders, is that evals are how you measure and then raise first-pass-correct rate deliberately instead of hoping. An eval suite is a first-pass-correct meter you can point at a model, a prompt, or an agent before it touches a customer. And it is why benchmark honesty matters so much: a vendor who games a benchmark, which I covered in the benchmark contamination playbook, is selling you a fake first-pass-correct rate, which means a fake cost per correct task, which means a real bill you discover later in rework.

The practical move is to instrument first-pass-correct rate per task type, not in aggregate. A single blended number hides the same way the 75 percent completion average did. Your agent might be 90 percent first-pass-correct on summarizing a call and 35 percent on drafting a legal clause. Averaged, that looks like a mediocre 60 and tells you to do nothing. Split, it tells you exactly where to keep a human in the loop and where to let the machine run unwatched, which is the only decision that actually lowers your cost per correct task.

How to Compute Cost Per Correct Task

You do not need a data team to compute this. You need to be honest about the denominator. Here is a worked example using a single agent workflow over one week, with round numbers chosen to make the arithmetic visible rather than to claim precision for any specific tool.

Say an agent drafts 100 pull requests in that week. Generation is cheap: call it about 50 dollars in total model cost across all 100. Your first-pass-correct rate is 40 percent, so 40 land cleanly and 60 need a human to step in. Each of those 60 takes a senior engineer about 45 minutes to review and fix, and at a loaded rate near 100 dollars an hour that is about 75 dollars each, or 4,500 dollars. So far you have spent 4,550 dollars and produced 100 merged PRs. The vanity dashboard says you shipped 100 things for the price of generation. Now subtract reality: of the 100 merged, 5 cause incidents over the next month, each costing roughly 2,000 dollars in debugging, customer impact, and cleanup, for 10,000 dollars more. Your true correct-task count is 95, and your total cost is 14,550 dollars.

Cost per correct task is 14,550 divided by 95, or about 153 dollars per correct PR. The generation cost, the number that felt like the whole story, was 50 dollars, or about 0.3 percent of the real figure. Everything that mattered lived in verification, rework, and failure, exactly the parts the throughput metric never counted.

Now the same workflow with a higher first-pass-correct rate. Suppose you invest in better evals and a tighter spec, generation rises to 80 dollars because you run more checks, but first-pass-correct climbs to 75 percent. Only 25 PRs need the 75-dollar human pass, for 1,875 dollars, and incidents drop to 1 at 2,000 dollars. Total cost is 3,955 dollars, correct tasks are 99, and cost per correct task falls to about 40 dollars. You spent more on generation and on evals and your cost per correct task dropped by nearly three quarters, because you moved the lever that actually controls the bill. No volume metric would have told you to make that trade. Tokens went up. Lines were the same. Cost per correct task is the only number that pointed the right way.

The formula in plain terms: add up everything you spent getting the work done this period, including the model cost, the time spent checking and fixing, and the cost of anything that broke. Divide by the number of tasks that ended up correct and stayed correct. Track it per task type, watch it over time, and make every AI decision, which model, how much review, build or buy, by whether it moves that number down. This is the operational sibling of the argument I made in the cost-first AI product launch playbook: in AI, cost is a decision you make at the unit level, not a finance problem you defer.

Buying and Pricing AI by Correct Task

The market is already moving to this unit, which is the strongest sign that it is the right one. Through 2026 the fastest-growing AI pricing model is outcome-based: you pay only when the system delivers a measurable result. Support agents have led the way. One vendor charges about 0.99 dollars per resolved customer conversation and bills nothing when the agent fails to resolve the ticket. Another dropped its price to 0.50 dollars per resolved conversation. A third launched at 1.50 dollars per automated resolution on committed volume and 2.00 dollars pay-as-you-go. Across the category, resolved support tickets run roughly 2 to 8 dollars and qualified sales leads run 5 to 25 dollars.

Strip away the marketing and outcome-based pricing is vendors agreeing to be paid in correct tasks. They have every incentive to count honestly, because they only get paid for resolutions that stick, which means they are quietly running their own cost per correct task in the background and pricing above it. When a vendor will not offer outcome-based pricing and insists on per-seat or per-token billing, that is information: either they cannot predict their own first-pass-correct rate, or it is low enough that being paid per correct task would expose it. Either way, the pricing model tells you something the demo will not.

As a buyer, this gives you a clean way to evaluate any AI tool. Translate whatever they are selling, seats or tokens or tasks, into your own cost per correct task for your own work, including the verification and rework you will carry on your side. A tool that looks cheap per seat can be expensive per correct task if it has a low first-pass-correct rate on your specific use, because you pay the gate cost. A tool that looks expensive per resolution can be the cheapest per correct task if it almost never sends you garbage to clean up. The sticker price is in the wrong unit. Convert it before you sign. The same discipline protects you from the supplier risk I described in building an AI business that survives model churn: if you know your cost per correct task by task type, you can swap a model the moment a cheaper one matches it, instead of being locked in by a per-seat contract that hid the real unit all along.

The Contrarian Take: Acceptance Rate Is a Trap

Here is where I disagree with most of the dashboards being sold as quality measures, including some that look sophisticated. The popular replacement for raw volume is acceptance rate: the share of AI suggestions a human accepts. It feels like a quality metric. It is not. It is a vanity metric wearing a quality costume, and it can be the most dangerous number on the board.

Acceptance rate measures whether a human said yes, not whether the work was right. Those are different things, and AI is engineered to widen the gap between them, because models are trained to produce output that looks good to a human reviewer. A confident, fluent, plausible draft gets accepted at a high rate whether or not it is correct. So a rising acceptance rate can mean your work is getting better, or it can mean your reviewers are getting tired, trusting, and fast, rubber-stamping output they no longer fully check. The metric goes up in both cases. The second case is how the perception gap from that developer study happens at the team level: everyone is accepting more, feeling more productive, and the rework rate is climbing in a different report nobody reads next to this one.

Cost per correct task is harder to fool because correct is defined downstream, after acceptance, by whether the work actually held. A task you accepted and then had to redo does not count as correct. A ticket you marked resolved that the customer reopened does not count. The metric only credits work that survived, which means it cannot be gamed by a human getting more agreeable. That is the entire point. Any AI metric that can be improved by lowering your standards is measuring the wrong thing, and acceptance rate fails that test on the first try.

The honest counterpoint, because there always is one: cost per correct task is lagging. You do not know a PR was correct until enough time has passed without an incident, and you do not know a resolution stuck until the customer fails to come back. That delay is real and you should not pretend it away. The answer is to pair a leading indicator, first-pass-correct rate measured by evals before shipping, with the lagging truth of cost per correct task measured after. The evals tell you fast and approximately. The cost per correct task tells you slowly and exactly. Run both, trust the slow one, and use the fast one to steer between measurements.

What to Do Monday Morning

You can start measuring this without a project, a tool, or a meeting. The point is to get one honest number in the next few days and let it start changing decisions.

First, pick the one task your AI does most often. The PR, the support ticket, the drafted email, the research summary, whatever your funnel pushes the most volume through. You want the highest-traffic task because that is where cost per correct task moves your business the most.

Second, define correct in one sentence, in writing, before you measure anything. What makes this task finished and what gets it sent back. If you cannot write it down, you cannot measure it, and the fact that it is hard to write is itself the discovery: your agents have been working without a target, and so has your review.

Third, count the past seven days honestly. How many of these did the AI attempt, how many shipped, and how many of the shipped ones came back, got reopened, or had to be redone. The number that came out correct and stayed correct is your real denominator. Almost everyone is surprised, in the bad direction, the first time they do this.

Fourth, add up the real cost. Model spend, plus the human time spent checking and fixing at a loaded rate, plus the cost of anything that broke. Divide by the correct count. That is your cost per correct task. Write it down. It is the first time most operations have seen the actual price of their AI output.

Fifth, run one experiment to move it. Raise the spec, add an eval gate, or keep a human on the one task type with a low first-pass-correct rate. Measure again in two weeks. If the number went down, you found a real improvement, not a feeling of one. This is the same loop I run across the whole company, and it is the operating discipline behind the agent boss operating system for founders: you delegate the doing freely and you measure the result ruthlessly. The table below is the instrumentation I keep.

One number, tracked over time, beats ten dashboards that only ever go up. If you want the wider context for where this fits, the AI opportunity map lays out how the durable advantages are shifting from producing output to controlling its quality, and cost per correct task is how you keep score in that world.

FAQ

What is cost per correct task?

Cost per correct task is the total cost of getting a unit of work to a finished, shipped, durable state, divided by the number of tasks that reached that state. The numerator includes generation cost, verification cost, the cost of rework on attempts that were wrong the first time, and the failure cost of anything that shipped and later broke. The denominator counts only tasks that ended up correct and stayed correct. It is the AI-era replacement for volume metrics, which stopped measuring value once generation became nearly free.

Why are output metrics like lines of code or tokens misleading for AI work?

Because they measure an input that is no longer scarce. A model can produce hundreds of lines or thousands of tokens in seconds, so counting them tells you about the model’s verbosity, not your progress. The test for a vanity metric is whether a worse, sloppier operation would also push the number up. A careless team posts more lines, tokens, merged pull requests, and AI-authored percentage than a careful one, which means those numbers track volume, not quality. Cost per correct task cannot be improved by working worse, which is why it survives.

What is first-pass-correct rate and why does it matter so much?

First-pass-correct rate is the share of tasks that come out right the first time without needing rework. It is the biggest lever on cost per correct task, because every task that fails the first pass goes back through the expensive verification step again. If the rate is high, most work clears the gate once and costs stay low. If it drops, work circles through rework and cost per correct task can double or triple even when generation got cheaper. Measure it per task type, not in aggregate, so you know where to keep a human in the loop.

Does using AI actually make teams less productive?

It can, when measured honestly. A controlled study of experienced developers found they were 19 percent slower with AI tools while believing they were about 20 percent faster, and they had predicted a 24 percent speedup going in. Broad surveys show 84 percent of developers use AI tools but organization-level gains often land near 10 to 30 percent. The speedup in generation is real, but the cost moved downstream to verification and rework, which the generation metrics never counted. AI raises productivity only when you also measure and manage the correctness step.

How do I calculate cost per correct task for my own work?

Pick your highest-volume AI task and define in one sentence what makes it correct. Count how many the AI attempted last period, how many shipped, and how many of those came back or had to be redone. The ones that stayed correct are your denominator. Add up model spend, the human time spent checking and fixing at a loaded hourly rate, and the cost of anything that broke. Divide that total by the correct count. Track it monthly per task type and make AI decisions based on whether they move it down.

Is acceptance rate a good quality metric for AI?

No. Acceptance rate measures whether a human said yes, not whether the work was right, and models are trained to produce output that looks good to reviewers, which widens the gap between accepted and correct. A rising acceptance rate can mean better work or tired reviewers rubber-stamping plausible drafts, and the metric goes up either way. Any AI metric that improves when you lower your standards is measuring the wrong thing. Cost per correct task defines correct downstream, after the work has held up, so it cannot be gamed by reviewers getting more agreeable.

How does outcome-based AI pricing relate to cost per correct task?

Outcome-based pricing is vendors agreeing to be paid in correct tasks. Support agents now bill roughly 0.50 to 2.00 dollars per resolved conversation and charge nothing when they fail to resolve, which means they are running their own cost per correct task internally and pricing above it. As a buyer, translate any pricing model, seats or tokens or tasks, into your own cost per correct task including the verification and rework you carry. A tool that looks cheap per seat can be expensive per correct task if its first-pass-correct rate on your work is low.

What should a solo founder do first to start measuring this?

Start with one task type and one honest week. Write down what correct means for that task, count how many AI attempts ended up correct and stayed correct, total the model spend plus your review and fix time plus any cleanup cost, and divide. That single number is usually a surprise, and it immediately changes decisions: where to add an eval gate, where to keep a human in the loop, and which tools are actually cheap once you price them in correct tasks instead of seats. Then run one experiment to lower the number and measure again in two weeks.