AI Agent Observability: Why Agents Fail Silently

Capital has started repricing agents. Median post-money valuations for agent startups climbed roughly 40 percent in a single quarter, and the money is flowing specifically toward what investors have taken to calling write-path platforms: agents that take actions rather than agents that fetch and summarize. Read-path tools, however polished, are getting treated as features. The pitch that raises now is not “our agent is smart.” It is “our agent replaces this workflow.”

Everyone is reading that as a valuation story. It is a liability story.

The moment an agent stops answering questions and starts taking actions, the cost of being quietly wrong stops being an annoyance and becomes an invoice, a refund, a compliance incident, or a customer who leaves without telling you why. And the uncomfortable part, the part nobody puts in the deck, is that the monitoring stack most teams have in place cannot see that failure at all.

Here is the durable version.

Traditional software fails loudly. It throws an exception, returns a 500, times out, or falls over in a way that generates a stack trace with a line number in it. Forty years of operational tooling was built on that one assumption: that a failure announces itself, and your job is to find out why.

Agents violate the assumption. An agent that fails almost never throws. It completes. It returns HTTP 200. It produces well-formed, confident, grammatically clean output that is wrong. Every dashboard you own goes green while the thing quietly does damage.

I have shipped agents that ran for weeks looking perfectly healthy on every chart I had, and were producing bad output the entire time. Not occasionally. The entire time. I found out from a user, which is the worst way to find out anything.

What follows is the framework I now use to instrument agents so that does not happen again. It has four layers, and almost every team I talk to has built only the first one.

Table of Contents

• The Failure That Never Pages You
• The Observability Inversion
• The Four Traces: A Visibility Framework
• Trace One: Execution, and Why It Is Not Enough
• Trace Two: Decision, or What the Agent Was Thinking
• Trace Three: Outcome, the Layer Everyone Skips
• Trace Four: Consequence, the Layer That Pays
• Detection Lag: The Metric That Actually Matters
• The Silent Failure Taxonomy
• The Loudness Budget: How Much to Instrument
• The Contrarian Take: You Do Not Have an Observability Problem
• What to Do Monday Morning
• Frequently Asked Questions

The Failure That Never Pages You

Start with the numbers, because the numbers explain why so many well-run teams are walking into this.

Gartner published a forecast in June 2025 that over 40 percent of agentic AI projects would be canceled by the end of 2027, and pinned the causes on escalating costs, unclear business value, and inadequate risk controls. That prediction has aged well enough that it keeps getting re-reported. Independent surveys of enterprise deployments put the share of agent pilots that reach production and deliver measurable value at around 10 percent. PwC found that while roughly 79 percent of organizations had adopted agents in some form, most of them could not trace a failure through a multi-step workflow.

Read that last clause again, because it is the whole essay. They adopted the technology. They could not tell you where it went wrong.

One more number, and it is the one I find most useful: roughly 65 percent of enterprise agent failures trace back to context drift rather than architecture defects. The system was built correctly. The plumbing worked. What broke was what the agent knew, or believed it knew, at the moment it decided.

Now do the arithmetic on your own product. Suppose your agent is right 99 percent of the time on individual actions, which is a number most teams would be thrilled to have. If it takes 10,000 actions a day, it is taking 100 wrong actions a day. Not 100 errors. Not 100 alerts. One hundred confident, well-formed, plausible, wrong actions, each of which returned success.

Chain them and it gets worse. I wrote about this compounding effect in the context of loop engineering for AI agents: a five-step agent where each step is 95 percent reliable finishes correctly about 77 percent of the time, and at 85 percent per step you are down near 44 percent. The steps do not report failure. They report completion. The failure lives in the composition, which is exactly the place nothing is watching.

This is why I have stopped calling it a reliability problem. Reliability implies the thing falls over. This is a detection problem. The thing does not fall over. It keeps running, confidently, in the wrong direction, and there is no signal in your infrastructure that distinguishes that from working perfectly.

The Observability Inversion

Here is the conceptual flip that took me embarrassingly long to see, and once I saw it, the tooling decisions got much easier.

In traditional software, you observe in order to diagnose a failure you already know about. Something pages you. An error rate spikes, a latency graph goes vertical, a customer reports an outage. The failure announces itself, and observability is the forensic layer you use to find the cause. Detection is free. Diagnosis is the hard part.

In agent systems, that is inverted. You observe in order to discover that a failure happened at all. Nothing pages you. Detection is the hard part, and it is not free, and it is not a side effect of your existing infrastructure. It is a thing you have to build on purpose, and pay for, and maintain.

I call this the Observability Inversion, and it has one very practical consequence: every dollar and hour most teams put into agent observability goes into the diagnosis layer, because that is what the tooling market sells and what engineers are trained to build. The detection layer, the one that answers “was that answer correct,” gets skipped, because there is no obvious place in a traditional stack where it belongs.

That last row is the one that should worry you. A crashed service stops doing damage the moment it crashes. That is an underrated feature. An agent with a corrupted understanding of its task does not stop. It executes the wrong thing with perfect uptime, and the better your infrastructure, the faster it does it.

Which means for agent systems, uptime and correctness have partially decoupled. You can have five nines of availability on a process that is wrong a third of the time, and every SLA you have ever written would report that as a good month.

The Four Traces: A Visibility Framework

Once you accept that detection is the hard part, the natural next question is what exactly you are supposed to capture. Here is the model I use. Four traces, stacked, each answering a different question, each strictly harder and more expensive than the one below it.

The percentages are illustrative rather than measured, and I want to be direct about that so nobody quotes them as research. They reflect what I have consistently seen when instrumenting agent systems, and the shape is what matters: the layer that catches the majority of silent failures is the layer almost nobody builds.

Notice the trap in that stack. Layer one is the cheapest to build, comes free with most frameworks, produces the most beautiful dashboards, and catches almost nothing that matters. It is genuinely useful for cost control and for catching runaway loops, and I would never ship without it. But a team that builds layer one and stops has bought the feeling of visibility without the substance of it, and that feeling is more dangerous than having no dashboard at all. No dashboard makes you nervous. A green dashboard makes you confident.

Let me take each layer in turn.

Trace One: Execution, and Why It Is Not Enough

The execution trace is the one you already have, or can get in an afternoon. It records that a run happened and what it physically did: the sequence of spans, how long each took, how many tokens went in and out, which tools were called, how many retries fired, and whether anything threw.

The good news is that this layer has genuinely standardized. The OpenTelemetry GenAI semantic conventions gave the industry a shared vocabulary for LLM and agent spans, with agreed attribute names for model calls, token usage, and operation duration. Client spans exited experimental status in early 2026, agent and framework spans have been stable in practice, and major vendors map the conventions natively. If you are starting today, emit OpenTelemetry GenAI spans and stop shopping. The vendor-neutral layer is real now, and standardizing on it means you can change observability vendors later without re-instrumenting your product, which is not a small thing given how fast that market is consolidating.

What this layer buys you is real and worth having. Cost control, primarily. If you are tracking cost per correct task rather than cost per call, the execution trace is where the denominator comes from. It catches infinite loops, which are expensive and embarrassing. It catches latency regressions after a model swap. It catches the agent that started retrying four times on every call because an upstream API began rate limiting you.

What it cannot do is tell you whether the run was any good.

Here is the concrete version. An agent is asked to reconcile a customer invoice. It runs in 3.2 seconds, calls two tools, uses 4,100 tokens, returns 200, and produces a clean structured JSON response. Perfect execution trace. Every metric nominal.

It reconciled against the wrong month.

There is nothing in the execution trace, in principle, that could ever surface that. The run is indistinguishable from a correct one at that layer. You would need to know what the right month was, and the execution trace does not have opinions about rightness. It measures effort, not accuracy, and those are different things that happen to correlate in traditional software and stop correlating in agent systems.

This is the same failure of measurement I described in eval-washing. A number that looks like a reliability number, that behaves like a reliability number, and that is measuring something else entirely. Uptime is not correctness. Latency is not correctness. Token efficiency is definitely not correctness. Yet those three metrics constitute the entire agent dashboard at most companies I have looked at.

Trace Two: Decision, or What the Agent Was Thinking

The decision trace captures not just what the agent did but what it had in front of it when it decided. The exact prompt as assembled, including everything the framework silently injected. The documents that retrieval actually returned, with their relevance scores. The tool schemas visible at that moment. The plan the agent produced before executing it. The conversation state carried in from earlier turns.

This is where the 65 percent figure earns its place. If most agent failures come from context drift rather than architecture defects, then the context is the crime scene, and if you are not recording it you have no case.

Context drift is a specific thing, and it is worth naming its shapes:

• Retrieval decay. The vector search that returned the right three documents in testing now returns two right ones and a plausible wrong one, because the corpus grew. Nothing errored. The agent read the wrong document and reasoned correctly from it.
• Accumulated state poisoning. On turn eleven, the agent is still carrying a mistaken assumption it formed on turn three, and every subsequent step compounds it. Each individual step is defensible given what it believed.
• Silent truncation. The context window filled and the framework dropped the middle. The agent answered from a partial picture and had no way to know it was partial.
• Tool description drift. Someone updated a tool’s description string, and the agent now selects it in situations it was never meant for. No code changed in the agent.
• Model substitution. Your provider routed you to a different checkpoint, or you upgraded, and behavior on your edge cases shifted underneath a prompt that was tuned to the old one. I covered the strategic side of this in the agent identity gap, but the operational version is simpler and meaner: your prompt is now calibrated to a model that no longer exists.

Every one of those produces a clean execution trace. Every one of them is visible in a decision trace, if you kept it.

The practical objection is storage, and it is a fair objection. Full context capture on every run is genuinely expensive, and if your agent processes anything sensitive it is also a compliance problem you have just created for yourself. The pattern I use: hash and index the context on every run, store the full payload on a sample plus every run that any downstream signal flags. That way you can always answer “did the context change” cheaply, and you can answer “what exactly was it” for the runs you care about. Redact at capture time, not at query time.

One more thing about the decision trace, and it matters more than the storage question. Capture the agent’s plan separately from its execution. When an agent produces a plan and then executes it, those are two failure surfaces, not one. A good plan executed badly and a bad plan executed faithfully look identical in the output and require completely different fixes. I learned this the slow way while working through how to split work across one agent versus many, where the plan-versus-execution distinction turns out to be the thing that determines whether adding agents helps or hurts.

Trace Three: Outcome, the Layer Everyone Skips

This is the detection layer. This is the one that catches the confident wrong answer, and it is the one that is missing from the overwhelming majority of agent deployments I have seen.

The outcome trace asks a question the other layers structurally cannot: was the output correct? Answering it requires something none of the lower layers need, which is a definition of correct that exists outside the agent. You cannot derive correctness from the run. You have to bring it.

The mechanism is online evaluation. You score live production output against a correctness definition, continuously, and you alert on the score rather than on the status code. This is the single sentence that separates teams who find their failures from teams whose customers find their failures.

Note the word online. Most teams that have done any eval work at all have done offline evals: a fixed test set, run before deploy, producing a number in CI. That is necessary and I would not ship without it, and I laid out how to build one in the evals playbook. But an offline eval cannot catch a production silent failure, because production silent failures happen on inputs that are not in your test set. That is very close to the definition of why they happen. Offline evals protect you from regressions you anticipated. Online evals detect failures you did not.

The obvious objection: how do you score correctness in production when you do not have ground truth? You mostly do not have ground truth. But you have four cheaper things, and stacked they get you a long way.

Run the cheap ones on everything and the expensive ones on a sample. Deterministic assertions and groundedness checks should fire on 100 percent of runs because they cost nearly nothing. Model-as-judge on 1 to 5 percent, weighted toward high-value or unusual runs, is usually enough to detect a shift in the failure rate even if it cannot catch every individual failure. You are not trying to inspect every output. You are trying to notice when the failure rate moves.

The fourth row deserves special attention because it is the one teams overlook and it is free. Your product almost certainly already collects delayed ground truth and throws it away. Every time a human edits the agent’s draft before sending it, that edit is a correctness signal. Every rejected suggestion is a label. Every reversed transaction, every reopened ticket, every “actually, no” in a chat thread. Most products log those as product events and never connect them back to the run that caused them. Connecting that loop is usually a week of work and it gives you a real correctness signal on a meaningful slice of your traffic, with no additional inference cost at all.

This connects to something I argued in the trust gap: the question is never whether to trust AI output in general. It is what evidence you have about this specific output, at the moment it matters. The outcome trace is how you manufacture that evidence at scale instead of relying on a human to eyeball it.

Trace Four: Consequence, the Layer That Pays

The top layer asks whether anything in the business actually changed, and it exists because of a hard limit on the layer below it.

Every eval you write encodes your current understanding of how the agent could be wrong. Which means evals are structurally blind to failure modes you have not imagined yet. The outcome trace catches the failures you thought of. Something has to catch the rest.

The consequence trace watches the downstream business signals that move when an agent is quietly doing damage, whether or not you knew to look for that particular damage:

• Refund rate and reversal rate on agent-touched transactions versus human-touched ones
• Support ticket volume tagged to workflows the agent participates in
• Human override and edit rate, tracked as a time series rather than an aggregate
• Rework: how often does a completed task get reopened within 30 days
• Churn among accounts with high agent exposure versus matched accounts with low exposure
• Time to completion for the end-to-end workflow, not the agent’s step

The single most valuable number in that list is the human override rate, and it is valuable precisely because it is a leading indicator that costs nothing. When the people who work alongside your agent start correcting it more often, they have detected something before your instrumentation did. They are usually right, and they usually will not file a ticket about it. They will just quietly start double-checking everything, which is exactly the behavior that destroys the productivity case for the agent while every dashboard still shows adoption climbing.

I watched this happen on a support workflow I built. Adoption metrics rose steadily for six weeks. Volume through the agent, up. Time saved per ticket, up. Meanwhile the edit rate on agent drafts had crept from 12 percent to 41 percent, and nobody was looking at that number because it was not on the dashboard. The team had stopped trusting the output and started treating the agent as a first-draft generator, which is a fine thing to be but a completely different product than the one we thought we had shipped. The productivity gain had evaporated a month before anyone noticed, and the reason nobody noticed is that the metric that captured it was not being watched.

That is the pattern I described from a different direction in the AI efficiency trap. The cost does not show up where you are measuring. It shows up as rework, as verification overhead, as the slow accumulation of human distrust that never appears in a single line item.

The consequence trace is also the layer that makes the business case. When a board asks what your agent is worth, the execution trace tells them how many tokens you spent, the outcome trace tells them how often it was right, and the consequence trace tells them whether anything got better. Only one of those three is the actual question.

Detection Lag: The Metric That Actually Matters

If I could get founders to adopt exactly one new number from this essay, it would not be a failure rate. It would be Detection Lag.

Detection Lag is the elapsed time between a silent failure occurring and someone on your team knowing about it.

I prefer it to failure rate for a specific reason: failure rate is largely a property of the model and the task, and you have limited control over it in the short run. Detection Lag is entirely a property of your engineering, and you can cut it by an order of magnitude in a week. It is the part you own.

The arithmetic of Detection Lag is what makes it urgent. Take an agent running 10,000 actions a day at a 1 percent silent failure rate. That is 100 bad actions daily. Detect in 30 days and you have shipped 3,000 wrong actions into your customers’ businesses. Detect in an hour and you have shipped four.

Same model. Same prompt. Same failure rate. A 750x difference in damage, produced entirely by instrumentation.

That ratio is why I think the framing of agent reliability as a model problem is mostly a distraction for founders. You do not control the model’s error rate. You completely control how long that error runs before you catch it. One of those is a research problem and the other is a Tuesday.

Two practical notes on measuring it. First, you can only measure Detection Lag on failures you eventually detected, which means the number is optimistic by construction. It is still directionally useful. Second, measure it per failure class rather than as a single average, because a workflow with strong delayed ground truth might have a lag of hours while a fire-and-forget workflow in the same product has a lag of never. Averaging those tells you nothing.

The Silent Failure Taxonomy

Not all silent failures are the same shape, and the shape determines which trace catches it. This is the working taxonomy I use when deciding where to spend instrumentation effort.

The last row is my favorite because it is so thoroughly invisible. An agent that refuses tasks looks fantastic on an execution trace. Low latency, low token spend, no errors, high throughput. Your cost per run goes down. Your dashboard improves. And the work is not getting done. If your agent’s average token consumption drops without a deliberate change, that is not efficiency, and I would treat it as an incident until proven otherwise.

Notice also that the taxonomy distributes across all four traces. There is no single layer that catches everything, which is why the framework is a stack rather than a choice. But there is a clear priority order, and it is not the order most teams build in.

The Loudness Budget: How Much to Instrument

Full four-layer instrumentation on every agent in your product is not a realistic goal and I would not attempt it. Online evals cost inference. Full context capture costs storage and creates compliance exposure. Consequence tracking costs analytics work that competes directly with shipping features.

So the real question is not whether to instrument. It is how loud to make each agent, and the answer should come from the cost of that agent being quietly wrong, not from how technically interesting it is.

I use a simple two-factor rule I call the Loudness Budget. Score each agent on reversibility and blast radius, then read the required instrumentation off the grid.

The bottom-right quadrant is the one founders under-use. If an agent writes drafts a human reviews before anything happens, you do not need online evals on it. The human is the eval. Instrumenting that agent to the same standard as your payments agent is a waste of budget that you will pay for by under-instrumenting something that matters.

The top-left quadrant is the one founders over-trust. Irreversible plus wide is the combination where a silent failure becomes a company-ending event rather than a bad week, and it is exactly where “the demo was really impressive” is least relevant. This maps directly onto the control framework I laid out in deploying AI agents without losing control: the autonomy you grant should be a function of reversibility, and the instrumentation you build should be a function of the same variable.

One warning about the grid. Agents migrate across it without anyone re-evaluating them. An agent ships as a suggestion tool in the safe green quadrant, then six months later someone wires its output directly into a system of record because the suggestions were good, and it is now in the red quadrant with green-quadrant instrumentation. Nobody makes that decision explicitly. It happens through a series of small reasonable integrations. Re-score every agent whenever its output gets a new consumer, and put that in your review process rather than in your memory.

The Contrarian Take: You Do Not Have an Observability Problem

Everything above is the operational answer, and I stand behind it. Here is what I actually think, which is less comfortable.

Most teams reaching for agent observability tooling do not have an observability problem. They have a specification problem wearing an observability costume.

Try this. Ask the person who owns your agent to write down, in one paragraph, what a correct output looks like. Not “helpful.” Not “accurate.” A definition precise enough that a second person could apply it to a hundred outputs and produce the same verdicts you would.

Most teams cannot do it. I could not do it, for the first two agents I built, and that failure was the actual root cause of everything downstream.

This matters because it explains why the outcome trace is the layer that gets skipped, and the explanation is not budget or engineering time. You cannot instrument a definition you do not have. An eval is a specification with a score attached. If you never wrote the specification, no vendor can sell you the eval, and the tool you buy will quietly degrade into a very expensive execution trace with a nicer interface. That is the actual failure mode of the observability tooling market: teams buy the platform, wire up traces, get beautiful dashboards, and still cannot answer whether the agent is right, because the missing input was never technical.

There is a harder version of this, and it is the thing I would want a founder to sit with. If you cannot specify what correct means for a task, you cannot safely automate that task. Not “you should build better monitoring first.” You cannot automate it, because you have no mechanism to know when it is going wrong, and an unbounded process you cannot evaluate is not automation, it is an experiment running against your customers.

The industry has a comfortable story about this, which is that models will improve until the problem goes away. I do not believe it, for a reason that has nothing to do with model capability. A more accurate model reduces the failure rate but does nothing to your Detection Lag. It makes silent failures rarer and therefore harder to notice, and it makes your team trust the output more, which lengthens the time before anyone checks. Better models make this problem quieter without making it smaller. That is not the same as fixing it, and in a system where the damage scales with time-to-detection, quieter can be worse.

The teams that will still be running agents in production in three years are not the ones with the best models. They are the ones who wrote down what correct means, early, while the problem was still small enough to write down. That is a judgment problem before it is an engineering problem, and it does not get easier by waiting.

What to Do Monday Morning

Concrete sequence. This is roughly two weeks of work for one engineer and it will change what you know about your own product.

1. Inventory and score your agents (60 minutes). List every agent or agentic workflow in production. For each, write two scores: reversibility and blast radius. Place them on the Loudness Budget grid. You will find at least one agent in the red quadrant that you have been treating as green. That is the finding, and it is worth the hour on its own.

2. Write the correctness definition for your highest-risk agent (90 minutes). One paragraph. Precise enough that two people would agree on the verdict for the same output. If you cannot write it, stop instrumenting and go get whoever owns the underlying business process, because they have it in their head and nobody has written it down. This step is the bottleneck. Everything downstream is mechanical.

3. Ship three deterministic assertions (half a day). From that definition, pick the three cheapest things that must be true of every valid output. Schema conformance, IDs resolve, totals reconcile, dates within range. Run them on 100 percent of production runs. Log failures with the full context payload attached. This is the highest return-per-hour work in the entire framework and it requires no vendor and no inference cost.

4. Connect delayed ground truth (2 to 3 days). Find where a human already corrects, edits, rejects, reverses, or overrides the agent’s output. Join that event back to the run ID that produced it. You now have a real correctness signal on real traffic for free. If your product does not capture that event today, capturing it is more valuable than any eval you could write this quarter.

5. Instrument override rate as a time series (half a day). Not the aggregate. The trend, by week, by workflow, with an alert on a relative increase. This is your cheapest early warning that the people closest to the agent have stopped trusting it, and they will notice before your instrumentation does.

6. Add sampled model-as-judge on the red-quadrant agent (2 days). Score 5 percent of production runs against the rubric from step 2. Alert on the score moving, not on any individual low score. Individual scores are noisy. The moving average is not, and the moving average is what tells you something changed.

7. Measure your Detection Lag and put it on the wall (ongoing). For the last three incidents where the agent was wrong, work out how long it took before a human knew. Write those three numbers down. That is your current baseline, and it is almost certainly measured in weeks. Then cut it, and keep measuring it per failure class rather than as one number.

If you do only two of these, do steps 2 and 4. The correctness definition is the thing you cannot outsource, and the delayed ground truth loop is the highest signal you will ever get for free.

For the broader question of what to build agents for in the first place, and where they earn their keep, the AI Opportunity Map covers the strategic layer this operational work sits underneath. And if you are earlier than that, still trying to get a first agent from pilot into production at all, the production gap is the more useful starting point.

Frequently Asked Questions

What is AI agent observability?

AI agent observability is the practice of capturing and evaluating an agent’s full decision path in production so you can tell whether it worked, not just whether it ran. It differs from traditional application monitoring because agents rarely fail with errors. They fail by returning confident, well-formed, wrong output while every conventional metric stays green. Complete agent observability spans four layers: execution (did it run), decision (what did it see and choose), outcome (was it right), and consequence (did anything in the business change).

Why do AI agents fail silently instead of throwing errors?

Language models produce a plausible output for essentially any input rather than signaling that they lack the information to answer. A retrieval step that returns a wrong-but-relevant document, a truncated context window, or a stale assumption carried from an earlier turn all yield fluent output and an HTTP 200. The failure lives in the semantics of the answer, not in the mechanics of the run, and conventional error monitoring only inspects the mechanics. Roughly 65 percent of enterprise agent failures trace to context drift rather than architecture defects, which is why these failures produce clean execution traces.

What is Detection Lag and how do I measure it?

Detection Lag is the elapsed time between a silent failure occurring and someone on your team knowing about it. Measure it retroactively: take your last three known agent failures, identify when the first bad output was produced, and identify when a human first knew. The gap is your baseline, and for teams running only execution traces it is typically weeks. It matters more than failure rate because failure rate is largely a property of the model while Detection Lag is entirely a property of your engineering. An agent taking 10,000 daily actions at a 1 percent failure rate ships 3,000 wrong actions at a 30-day lag and four at a one-hour lag.

Are offline evals enough, or do I need online evals in production?

Offline evals run against a fixed test set before deploy and protect you from regressions you anticipated. They cannot catch production silent failures, because those happen on inputs that are not in your test set, which is close to the definition of why they happen. Online evals score live production output continuously and alert on the score rather than the status code. You need both. Offline evals gate deploys; online evals detect the failures you did not imagine.

How do I evaluate agent correctness in production without ground truth?

Stack four methods by cost. Run deterministic assertions such as schema validation, ID resolution, and arithmetic reconciliation on 100 percent of runs since they are nearly free. Run groundedness checks verifying each claim traces to a retrieved source. Run model-as-judge scoring against a written rubric on a 1 to 5 percent sample, weighted toward high-value runs. And connect delayed ground truth, meaning the human edits, rejections, reversals, and overrides your product already generates, back to the run IDs that caused them. The fourth is free, accurate, and almost always discarded.

Should I use OpenTelemetry or a dedicated agent observability vendor?

Emit OpenTelemetry GenAI semantic convention spans regardless of vendor. The conventions standardized attribute names for model calls, token usage, and operation duration, client spans exited experimental status in early 2026, and major platforms map them natively. Standardizing on the vendor-neutral layer means you can switch observability vendors later without re-instrumenting your product. Vendors add value above that layer through evaluation, scoring, and failure analysis, which is where the actual differentiation sits, not in trace collection.

How much observability does each agent need?

Score each agent on reversibility and blast radius. Reversible and narrow, such as drafts a human reviews, needs only an execution trace, since the human reviewer is the eval. Reversible but wide, such as bulk enrichment, needs execution, decision, and sampled outcome traces with alerting on rate changes. Irreversible but narrow needs full decision capture plus close consequence monitoring. Irreversible and wide, such as payments or deletions, needs all four traces, online evals on every run, and human approval before execution. Re-score any agent whenever its output gains a new consumer, because agents migrate into higher-risk quadrants without anyone deciding to move them.

Will better models make silent failure go away?

A more accurate model lowers the failure rate but does nothing to Detection Lag. It makes silent failures rarer and therefore harder to notice, and it increases team trust in the output, which lengthens the time before anyone checks. Since damage from a silent failure scales with time-to-detection rather than with per-action probability, better models can make this problem quieter without making it smaller. The durable fix is a written definition of what correct means for your task, which is a specification problem rather than a model problem.