Loop Engineering: The Real Moat in AI Agents

A founder I follow posted a line this week that stuck with me: “Everyone’s still optimizing prompts. The real moat is loop engineering.” He was pointing at a voice AI company that just crossed 100 million dollars in funding by building software that holds a 40 minute phone call together while a real human wanders, interrupts, and changes their mind three times. A prompt cannot do that. A prompt gets you one good answer to one clean question. The call needs something else.

Around the same time, an Anthropic engineer described how their teams build: roughly 90 percent of them wrap models in loops and let agents verify their own work, sometimes generating their own practice problems to improve. The phrase they keep using is “close the loop.” Give the agent a way to check itself, and then act on what the check says.

Strip the news away and you are left with a claim that has been true for two years and will be true in two more. The model is not the product. The loop around the model is the product. The prompt is the cheapest, most copyable part of your stack. The loop, and one specific edge of it, is the part nobody can copy by reading your system message.

I have built agents that looked brilliant in a demo and fell apart in week two of real use. Every time, the failure was the same shape. I had engineered the prompt and skipped the loop. This is a field guide to the part I skipped, written for founders and builders who are shipping AI products and tired of watching them degrade the moment a customer touches them.

A prompt gets one answer. Production asks a thousand questions.

Here is the math that kills most agent products, and almost nobody puts it on a slide.

An agent that does real work takes many steps in a row. It reads a request, calls a tool, reads the result, calls another tool, writes a draft, checks a field, sends a message. Each step has some chance of being right. Those chances multiply. They do not average.

Say each step is 95 percent reliable. That sounds great. Ship it. Now chain twenty steps together, which is a modest agent workflow, and your success rate is 0.95 to the twentieth power. That is about 36 percent. Two out of three runs fail somewhere. At 99 percent per step, ten steps still only gets you to 90 percent, and twenty steps drops to 82. A seemingly tiny 1 percent error per step, compounded across a long task, lands you near a 63 percent chance of failure by the end. The reliability you measured on a single call is a lie about the system you actually shipped.

This is not a theory I cooked up. Carnegie Mellon built a benchmark called TheAgentCompany that asks agents to do real white-collar tasks inside a simulated company. The best frontier models finished about 24 percent of the tasks on their own. A study of multi-agent systems published in early 2025 looked at more than 1,600 real execution traces across seven popular frameworks and found failure rates running from 41 percent all the way to 87 percent, with coordination breakdowns alone accounting for 37 percent of failures. Some commercial agents working on complex office tasks fail more than 90 percent of the time.

Founders see these numbers and reach for the wrong fix. They wait for a better model. They rewrite the prompt for the fortieth time. They add a second agent, which usually adds a second source of errors. The actual fix is structural. You stop trying to make a single pass perfect, because you cannot, and you start building a loop that catches and repairs the failures a single pass will always produce. I wrote a whole piece on the failure modes themselves in why AI agents fail in production. This piece is about the cure.

The cure has a shape, and the shape has levels. Most products are stuck on the bottom two rungs and blaming the model for it.

The Loop Ladder: five levels of agent reliability

I started drawing this ladder after the third time I shipped an agent that worked in the demo and broke in the wild. I needed a way to tell, at a glance, how much real engineering sat between a model call and a dependable product. The answer was almost never “a better model.” It was almost always “a better loop.”

Every agent sits on one of five rungs. The rung you are on predicts your reliability, your defensibility, and how badly the next model release will hurt or help you. The moat lives on the top two rungs, and most teams never climb past the bottom two.

The single most important idea on this ladder is the jump from L2 to L3. Below that line, the agent is judging itself. Above it, something outside the agent does the judging. That line is where reliability stops leaking and starts compounding, and it is the line almost everyone skips because self-grading is easy to bolt on and an external verifier is real work.

The Verification Gap: the part you can actually own

Underneath the ladder is one core idea that explains why the top rungs are defensible. Call it the verification gap. It is the distance between two events: the moment a model produces an output, and the moment your system knows that output is correct.

Generation is collapsing in price and getting easier every quarter. Anyone can call the same frontier model you call. The gap, the work of closing the distance between “produced” and “known correct,” is the scarce part. It is domain specific. It is yours. When you build a way to know, cheaply and automatically, whether a given output is right, you have built something a competitor cannot lift out of your prompt because it does not live in your prompt. It lives in your tests, your tools, your data, and your judgment about what “right” even means in your domain.

There is a useful asymmetry here that the best loop businesses exploit. For some problems, checking an answer is far cheaper than producing one. Running a unit test is cheaper than writing correct code. Validating a JSON schema is cheaper than extracting the data. Confirming a booking exists is cheaper than navigating the booking flow. Those are the problems where loops pay off the most, because the verifier costs you little and saves you from shipping confident garbage. I will come back to this when we talk about loop economics, and it connects directly to the metric I argued matters most in cost-per-correct-task.

L0 and L1: why retries are not reliability

L0 is a single model call. Prompt in, answer out. It is where every project starts and where it should stay only for things that genuinely take one step, like classifying a message or drafting text a human will read before it goes anywhere. The trouble starts when founders wire an L0 call into a five step process and call it an agent. The compounding math from earlier takes over, and the thing degrades in a way that looks random but is not.

L1 adds a retry. The call fails, times out, or returns malformed output, so the system tries again. This is worth doing. It catches the genuinely random failures: a dropped connection, a rate limit, a momentary glitch. Most teams stop here and believe they have built reliability.

They have not, and the reason is subtle. A blind retry only helps when the failure announces itself. A crash announces itself. A timeout announces itself. A confidently wrong answer does not. The model returns a clean, well formatted, completely incorrect result, the system sees no error, and it sails on to the next step carrying a mistake it will never notice. Retrying a step that did not know it was wrong just gives you a fresh chance to be wrong in a new way. You cannot retry your way out of a problem you cannot detect.

So the question that actually defines your reliability is not “do you retry.” It is “how does your system decide an output was good enough to keep.” That decision is the loop. Everything upstream of it is generation. Everything that matters for reliability is in how you make that one call.

L2: the self-grading trap

The obvious next move, and the one I reached for first, is to ask the model to check itself. Generate an answer, then in a second call ask the same model “is this correct, and if not, fix it.” This is the family of techniques people mean when they say reflection or self-critique. The research literature has a famous version called Reflexion, where an agent writes a verbal critique of its own attempt, stores it, and uses it to do better on the next try. On paper it looks magical. Reflexion pushed a coding benchmark from 80 to 91 percent and lifted a multi-hop question task by around 20 points.

Here is the anatomy of what a real loop looks like, and where the self-check sits inside it.

Now for the trap. When the verify step is just the same model grading its own homework, it fails in a specific and well documented way. A team at Google and the University of Cambridge tested this carefully and found that large models are bad at finding their own reasoning errors. The bottleneck is not fixing mistakes. Given the location of an error, models correct it fine. The bottleneck is noticing the mistake in the first place. A separate, widely cited result put it bluntly in its title: large language models cannot reliably self-correct reasoning yet. Left to critique themselves with no outside signal, models often take a correct answer and talk themselves into a wrong one.

The pattern in all of this research is consistent. Self-reflection helps when three things are true at once: the starting accuracy is low, the task is genuinely hard, and there is some external signal available to ground the critique. Remove the external signal and self-checking becomes a confidence amplifier. The model was sure before. Now it is sure and it wrote a paragraph explaining why. That is not verification. It is a hallucination with a feedback delay.

L2 is not useless. It is just not a moat, and on its own it is not reliability. It is a cheap improvement that works in narrow conditions and quietly hurts you outside them. The teams that get burned are the ones who saw the Reflexion numbers, bolted self-critique onto everything, and assumed they had crossed into reliable territory. They had not crossed the L2 to L3 line at all. To cross it, the check has to come from somewhere the generator cannot fake.

L3: the grounded loop and the verifier moat

L3 is where the check stops being the generator’s opinion and starts being a fact the generator cannot argue with. The verifier is grounded in something real and independent: a test that runs, a tool that returns ground truth, a schema that validates, a database that confirms, or a second model trained specifically to judge rather than to produce.

This is the rung Anthropic describes in their own guidance on building agents. Their evaluator-optimizer pattern is exactly this: one model generates, a different model evaluates against clear criteria, and the loop runs until the evaluation passes. They are careful to say it only works when you have real evaluation criteria and when feedback measurably improves the output. That qualifier is the whole game. The criteria have to be real, and the evaluator has to be independent enough to apply them honestly.

The numbers move when you do this right. Self-consistency, where you sample several independent reasoning paths and take the majority answer instead of trusting one pass, lifted a grade school math benchmark by about 18 points to 74 percent, with consistent gains across every reasoning task it was tried on. Purpose built verifier models do even better. A 2024 approach to generative verifiers, where the judge is trained to reason about correctness rather than just score it, pushed a math benchmark from 73 to 93 percent and lifted an algorithmic task from 5 to 45 percent. Same base generator. The entire jump came from a better check.

A concrete version from my own work. I had an agent that pulled key terms out of contracts. On a single pass it was right about 88 percent of the time, which sounds usable until you remember a customer uploads forty clauses and expects all forty correct. At 88 percent per clause, a forty clause document is correct end to end almost never. The fix was not a smarter extraction prompt. It was a grounded check: every extracted value had to point back to a span of text in the source document, and a cheap second pass confirmed the span actually said what the extraction claimed. Anything that failed got re-extracted or flagged for a human. The generator never got smarter. The product went from a coin flip to something a legal team trusted, because the loop refused to ship a claim it could not trace to the page.

There is a reason this rung is defensible and the rungs below it are not. The verifier encodes what correct means in your specific domain, and that knowledge is hard won and hard to copy. A generic model knows what good code looks like in general. Your loop knows whether this output passes the test suite for your customer’s actual system. That difference is the gap a competitor has to close, and they cannot close it by reading your prompt because the knowledge is in your checks, not your instructions. This is the same reason I argued in how an AI business survives model churn that the durable layer of an AI product is never the model. When the model gets swapped, and it will, your loop is what stays.

Here is the difference between a prompt-first company and a loop-first company, laid out plainly.

If you want the practical build patterns for the verify step, I went deep on them in the evals playbook for solo founders, and on how to tell a real reliability claim from a marketing one in eval-washing. The short version: your verifier is only as honest as it is independent from the thing it checks.

L4: self-improving loops and why the moat compounds

L4 is the rung the Anthropic engineer was describing with the word “dreaming.” Once you have a grounded verifier, you have something more valuable than a quality gate. You have a labeling machine. Every time the loop runs, it produces an example of an output and a verdict on whether that output was correct. That is training data you did not have to buy, generated by your own product, specific to your own domain.

Feed that signal back and the loop improves itself. The verifier gets sharper because it sees more cases. The generator gets better because it is tuned on examples your verifier judged. Some teams go further and have the system generate its own practice problems in areas where it is weak, then solve and verify them, which is the self-improving idea taken to its conclusion. The loop is no longer a static gate. It is a flywheel.

This is where the moat stops being a wall and becomes a slope your competitors have to climb while you keep moving. The longer your loop runs, the more domain-specific verification data you accumulate, and that data is the one asset a better foundation model does not hand your rivals for free.

It also matters because the underlying capability is moving fast. METR, a research group that measures what agents can actually do, found that the length of task an agent can finish on its own has been doubling roughly every seven months, and has accelerated to something closer to every four months recently. Read the fine print, though. That measurement is at 50 percent reliability. The models can attempt longer and longer work, but the reliability gap on that work is exactly the gap a loop closes. As tasks get longer, the compounding math gets more brutal, and the verifier becomes more valuable, not less. The capability rising tide lifts the generators. It does not build your loop.

Loop economics: the cheapest insurance in your stack

The objection I hear most is cost. A loop calls the model more than once. It runs verifiers. It retries. Does that not blow up your inference bill? Founders look at the token meter and flinch.

This is the wrong meter. The number that matters is not cost per call. It is cost per correct result, the metric I made the case for in cost-per-correct-task. An L0 product that ships a wrong answer did not save you money. It cost you a refund, a support ticket, a churned customer, or in a regulated domain, a real liability. A loop that spends three times the tokens to ship a correct answer the first time is not more expensive. It is dramatically cheaper once you count the full cost of being wrong.

The asymmetry of verification is what makes the economics work. When checking is cheaper than generating, and for a huge class of useful problems it is, a verifier is the highest-return spend in your whole system. Running a test suite costs a fraction of a cent. Shipping code that breaks a customer’s production system costs you the account. Pick problems where the check is cheap and the cost of a wrong answer is high, and the loop pays for itself many times over on the first prevented failure.

Put rough numbers on it. Suppose a single high quality generation costs you two cents in tokens, and a verify pass plus an occasional retry triples that to six cents. You flinch at the three times jump. Now price the other side. One wrong answer that reaches a customer costs you a fifteen minute support reply, maybe a credit, and a dent in trust you cannot expense. Call it ten dollars, conservatively. If your single pass is wrong even five percent of the time, every hundred outputs carries roughly fifty dollars of buried failure cost. The loop’s extra four cents per output, four dollars across the same hundred, buys most of that back. You did not spend more. You moved spend from invisible refunds to visible compute, and visible compute is the only kind you can optimize.

Token prices keep falling, which only sharpens this. As generation gets cheaper, spending a little more compute on independent verification becomes the most obvious trade in the stack. The teams still optimizing prompts to save a call are economizing on the wrong line item. The teams winning are spending freely on the check, because the check is what they sell. I sketched how this fits a launch budget in the cost-first AI product launch playbook, and how loops interact with splitting work across agents in agent delegation.

The contrarian take: self-verification is mostly theater

The popular version of loop engineering says: just have the agent check its own work. Add a reflection step. Tell it to be critical. This is the part of the advice that is comforting, easy to implement, and largely wrong.

A model grading its own output is not a verifier. It is the same probability distribution that produced the answer, now asked to bless the answer. The research is clear that this fails in exactly the place you need it most. Models cannot reliably find their own reasoning errors, and when they try to self-correct without an outside signal, they frequently make correct answers worse. The self-check feels like rigor. It is the agent nodding at its own reflection. If your reliability story is “the model double-checks itself,” you do not have a loop. You have a single pass wearing a second hat.

The second thing most people get backwards: they treat the model as the asset and the loop as plumbing. It is the reverse. The model is rented. You and every competitor pull from the same handful of labs, and the contract resets with each release. The loop is owned. So here is the test I now apply to any AI product, my own included. If the next model release could kill your product, you built a prompt. If the next model release just makes your product cheaper and faster to run, you built a loop. Defensibility is not how good your model is today. It is how little a better model helps the person trying to copy you.

And the uncomfortable corollary for founders chasing the frontier: the winning move is usually not the smartest model. It is the cheapest model that clears your verifier. Once you have a real check, you can run a smaller, faster, cheaper generator and let the loop catch its mistakes, because the loop, not the model, is what guarantees the output. The people obsessing over which model is half a point ahead this month are optimizing the part that commoditizes. The people building better checks are compounding the part that does not.

None of this means prompts do not matter. A good prompt is the cheapest reliability win available, and you should still write one. It means the prompt is the floor, not the moat. Spend a day on the prompt. Spend the quarter on the loop.

What to do Monday morning

If you build or own an AI product, here is the concrete sequence. None of it requires a better model.

1. Find your rung. Trace one real task end to end and mark every model call. Where does your system decide an output is good enough to keep? If the answer is “it does not” or “the model decides,” you are on L0 to L2. That is your starting point, and it is fine, as long as you do not stay.
2. Count your steps and do the multiplication. Take your honest per-step reliability and raise it to the number of steps in your longest task. The number you get is your real success rate. It is almost always lower than what you have been telling customers.
3. Pick one place to add a grounded check. Not self-critique. An external check: a test that runs, a schema that validates, a tool call that confirms the world matches the claim, or a second model whose only job is to judge. Start with the step where a wrong answer costs you the most.
4. Choose verification-cheap problems on purpose. When you scope new features, favor the ones where checking is far cheaper than doing. Those are where loops pay off fastest.
5. Switch your dashboard from cost per call to cost per correct result. Once you count the full cost of wrong answers, the loop stops looking expensive and starts looking like the cheapest insurance you have.
6. Turn the loop into data. Log every output and every verdict. That is your private training set and the start of an L4 flywheel that a competitor cannot buy.

Do this and you will feel the product change character. It stops being a clever demo that degrades and becomes a system that holds together under real use. That is the difference between something you showed an investor and something a customer pays for twice. For the mindset shift underneath all of it, the move from chasing model magic to engineering reliable systems, I wrote how founders should think about AI, and the broader map of where these opportunities sit lives on the AI opportunity map.

Frequently asked questions

What is loop engineering in AI?

Loop engineering is the practice of building the control loop around a model rather than just the prompt inside it. Instead of taking one model output and using it, a loop plans, acts, observes the result, verifies whether the result is correct, and either ships it or sends a correction back through. The reliability of an AI product comes from how well this loop is built, especially the verify step.

How is loop engineering different from prompt engineering?

Prompt engineering shapes a single model call to get a better answer. Loop engineering wraps that call in a process that checks the answer and repairs it when it is wrong. A prompt is easy to copy and gives diminishing returns. A loop, and the domain-specific verifier inside it, is hard to copy and is where durable advantage lives. Prompts are the floor; loops are the moat.

Why do AI agents fail more as they take more steps?

Because per-step reliability multiplies across a task, it does not average. An agent that is 95 percent reliable per step is only about 36 percent reliable across 20 steps, since 0.95 to the twentieth power is roughly 0.36. Small per-step error rates compound into large end-to-end failure rates, which is why agents that look solid on one call fall apart on long tasks.

Can an AI model verify its own output reliably?

Usually not on its own. Research shows large models are poor at finding their own reasoning errors, and self-correction without an external signal can turn correct answers into wrong ones. Self-checking helps most when the task is hard, starting accuracy is low, and an outside signal is available. For dependable verification you need a check that is independent of the model that produced the answer.

What is a grounded verifier and why does it matter?

A grounded verifier is a check rooted in something real and independent of the generator: a test that runs, a tool that returns ground truth, a schema that validates, or a separate model trained to judge. It matters because it encodes what correct means in your specific domain, which is hard to copy, and because it lets reliability compound upward instead of leaking away across steps.

Does adding a loop make my AI product too expensive?

It raises cost per call but usually lowers cost per correct result, which is the number that matters. A wrong answer that ships costs you refunds, support, churn, or liability. A loop that spends extra tokens to ship a correct answer the first time is cheaper once you count the full cost of being wrong, particularly when checking is cheaper than generating.

How do I know which loop level my agent is on?

Trace one task and find where your system decides an output is good enough to keep. No decision point means L0. A blind retry on errors is L1. The model grading itself is L2. An external, independent check is L3. A loop that turns its own verified results into training signal and improves over time is L4. Defensibility starts at L3.

Will better foundation models make loop engineering unnecessary?

No. Better models make generation cheaper and let agents attempt longer tasks, but capability gains are measured at partial reliability, and longer tasks make the compounding math worse. The verifier that closes the reliability gap becomes more valuable as models improve, not less, and it stays yours when the model is swapped.