The Context Engineering Playbook for AI Builders

Microsoft just canceled most of its Claude Code licenses. Uber blew its entire 2026 AI budget in four months. The cost of one AI token has collapsed about 280-fold in two years. None of those facts are in tension. They are the same fact, told three ways. The discipline that explains it has a name. Most founders do not know it yet. The ones who learn it first will own the next decade of AI products.

The numbers are public. Microsoft deployed Claude Code to 5,000 engineers in late 2025. By April 2026, monthly usage was hitting 84 to 95 percent. Per-engineer API costs were running between 500 and 2,000 dollars a month. Internal forecasting models broke. The cancellation memo went out the same week the Anthropic deal was supposed to be deepening. Uber’s CTO disclosed at an investor briefing that the company’s full annual AI engineering budget, set at 3.4 billion dollars for 2026, was exhausted before the year was half over. AI software prices across the US climbed somewhere between 20 and 37 percent in twelve months. None of this was supposed to be possible. The per-token price was supposed to keep falling. It is falling. It has fallen about 280-fold in two years. That is the paradox.

Every story I hear from founders right now lives inside this paradox. They built a thing. It worked. They put it in production. The bill exploded. They cut features. The bill grew anyway. They switched models. The bill grew anyway. They added eval tooling, observability, three more dashboards. The bill grew. Half of them are about to fire their AI vendor and the other half are about to fire their own product.

None of them have a context engineering discipline. That is the only diagnosis that fits all the symptoms. The math broke because they treated context as free. The math will keep breaking until they treat context as the unit of cost. This post is the playbook for doing that. It is the framework I run on every AI product I build and every one I advise. The post is long because the discipline is not optional any more. The next 18 months of AI is going to separate the teams that ship context-engineered products from the teams that ship beautiful, popular, expensive, bankrupt ones.

Table of Contents

• The Cost Paradox Nobody Wants to Explain
• The Context Stack: A Builder’s Framework
• Layer 1: System Prompt, Cache or Die
• Layer 2: Tool Definitions and the MCP Tax
• Layer 3: Retrieved Context, RAG is Back
• Layer 4: Conversation History, the Quadratic Killer
• Layer 5: Tool Outputs, Where Agents Bloat
• Layer 6: The User Query, the Sacred Layer
• The Quadratic Math Every Builder Must See
• The Contrarian Read: RAG Was Not Dead, It Was Early
• The Cost Floor Audit and Your Monday Plan
• FAQ

The Cost Paradox Nobody Wants to Explain

Here is the math that does not get said out loud. The per-token price you pay an AI provider has fallen by about two orders of magnitude in two years. A 100,000-token prompt that cost a few dollars in 2023 now costs a few cents. The official story is that AI is getting cheaper. The actual bill is the opposite. The reason is that almost nobody is paying per token any more. They are paying per finished task. And the number of tokens it takes to finish a task is exploding faster than the per-token cost is falling.

The mechanism is simple and almost nobody internalizes it the first time they hear it. An LLM API is stateless. Every call sends the entire context window again. A naive agent that runs for 20 steps does not send the original prompt once. It sends it 20 times. The first call costs N tokens of input. The second call costs N plus whatever the first turn added. By turn 20 you are paying for the same system prompt and conversation history 20 times over. The math is closed form: roughly N*T plus K*T squared over 2, where T is the turn count, N is the original prompt size, and K is the average size of what each turn adds. It is quadratic. It does not feel quadratic when you are watching one task run. It feels quadratic at the end of the month when the credit card statement arrives.

This is why a single complex agent task can cost 5 to 8 dollars instead of 5 cents. It is why agents burn somewhere around 50 times more tokens than a one-shot chat. It is why unconstrained agentic coding sessions routinely cost 10 to 100 dollars per session. Industry data shows that re-sent prior context is roughly 62 percent of the average agent bill. The most expensive line item in your AI product is not the work the model is doing. It is the work the model already did, that you keep handing back to it.

I have watched founders build a perfectly functional agent, ship it to ten customers, and discover their per-customer cost is higher than their per-customer revenue. They blame the model. The model is not the problem. They built a context machine and they are paying for context. They were sold a per-token price and they architected a per-loop product. Until they fix what enters the model on each loop, the price collapses do not reach their P&L. They are running the agentic equivalent of a pickup truck that hauls its own gas tank around in the bed of the truck. The gas got cheaper. The truck got heavier.

This is what context engineering fixes. Not at the prompt level. At the architecture level. The product change is what owns the cost curve.

The Context Stack: A Builder’s Framework

Every call to a model carries a stack of context. The stack has six layers. Each layer has its own cost behavior, its own optimization tools, and its own failure mode. The thing nobody teaches you in any AI course is that the stack is the unit of work. You do not optimize an LLM application by writing a better prompt. You optimize it by deciding, for every layer, what enters, when it leaves, and how it is cached. That is the entire discipline. The Context Stack is the way I think about it, and the way I make builders I advise think about it.

Here is what the picture is doing. Every layer above Layer 6 is a tax. You did not get billed for it on a chat product because chat has tiny context, no tools, and no loops. You get billed for it on an agent because agents loop, accumulate, and re-send. The job of context engineering is to decide, layer by layer, what enters, when it leaves, and how it is cached. There is a different lever for each layer, and using the wrong lever on the wrong layer is the most common mistake I see. You cannot summarize a system prompt. You can cache it. You cannot cache a conversation history. You can compress it. You cannot RAG a tool definition. You can prune it. The discipline is layer-specific.

The post is going to walk each layer in turn. By the end you will be able to look at any AI product and read its bill from the architecture. You will also know which layer to attack first. The order is not arbitrary. Layer 4 is where the quadratic math hides, so it is where most of the money goes. Layer 2 is where the largest static waste hides, so it is where the fastest single win lives. Most teams get the order wrong, attack Layer 3 first, and walk away with a 10 percent savings while Layer 2 keeps eating them alive.

Layer 1: System Prompt, Cache or Die

The system prompt is the most boring layer and the easiest one to fix. It is the same on every call. It is supposed to be the same on every call. It is your assistant’s identity, behavior rules, refusal patterns, response format. If you are still paying full price for it on every call, you are leaving 80 to 90 percent of a free win on the table.

Prompt caching is the lever. Anthropic, OpenAI, and Bedrock all support it now. The economics are very specific and worth memorizing. Anthropic charges 1.25 times the standard input rate to write a cache entry with a 5-minute time to live, and 2.0 times the standard input rate for a 1-hour TTL. Every subsequent read inside that TTL costs 0.10 times the standard input rate. That is a 90 percent discount on the cached portion. The math: if your system prompt is 4,000 tokens and your product fires 20 calls within a 5-minute session, the system prompt cost falls from 20 times the full input rate to about 1.25 plus 19 times 0.10, which is 3.15 times the rate. That is a 84 percent savings on Layer 1 alone, every session, forever. One project I work with cut their effective Layer 1 cost from 3.00 dollars to 0.47 dollars across a typical 20-request session. Same product. Same model. Same prompts. Just cached.

The trap people fall into is treating prompt caching as an optimization to add later. It is not an optimization. It is the default. Anything that lives in the system prompt and is identical across calls should be cache-eligible on the first day of the product. Building without it is like writing SQL without indexes and promising to add them “when we scale.” You will not add them. You will just spend the money you should have saved.

There is one nuance. Cache works because the prefix is identical. If your system prompt has any dynamic content baked into it, your hit rate collapses. The fix is structural: put the dynamic stuff in Layer 3, not Layer 1. The user’s name, today’s date, the customer’s plan tier, none of that belongs in the system prompt. It belongs in retrieved context. Mixing layers is how teams pay full price for prompts they think are cached.

Layer 2: Tool Definitions and the MCP Tax

This is the layer that will surprise you. Tool definitions are static. They look small. They are not. Modern LLM tool calling protocols, especially MCP, ship the entire schema of every connected tool on every call. Every parameter. Every description. Every type. Every example. And every conversation turn reloads it, because the API has no memory.

The math is brutal once you connect a few servers. Public reports from production builders show that running 7 active MCP servers consumes about 67,300 tokens before the conversation even starts. That is 33.7 percent of a 200,000-token Claude Sonnet context window, gone, before the user has typed a single character. The GitHub MCP server alone consumes nearly 25 percent of a Sonnet window. One developer reported a full setup eating 143,000 of 200,000 tokens, with 82,000 of those being MCP tool metadata. That is the tail wagging the dog. 72 percent of your context window, paying full price, on every single call, for tool descriptions you may not even use that turn.

I built a quick reference table for this. I look at it every time a founder asks me to review their agent. The MCP Tax is the single largest static line item in most production agent bills, and it is invisible if you do not measure it.

Two levers fix this. Prune and cache, in that order. Pruning means choosing tool exposure per task instead of per agent. A coordinator agent should look at the user query, pick the 1 to 3 MCP servers actually needed for this turn, and only load those tool definitions. Everything else stays out of context. This is the semantic tool discovery pattern that researchers started publishing on in early 2026. It is becoming the standard for production agents that survive their first month of bills. Caching the surviving tool definitions is the second move. Tool schemas almost never change inside a session, so they should be cached the same way the system prompt is cached.

If you remember nothing else from this section, remember this: every MCP server you connect to your agent is a recurring monthly bill, paid in tokens, whether or not the user ever invokes its tools. Treat connecting an MCP server the same way you would treat hiring a vendor. Audit it. Justify it. Cap it. Most teams I see have 6 to 12 MCP servers connected because they sounded useful at install time, with no measurement of what each one costs in tokens per session and no rule for when to drop one. That is not engineering. That is collecting.

Layer 3: Retrieved Context, RAG is Back

Layer 3 is where the long-context-vs-RAG debate of 2024 actually got resolved in 2026. The short version: long context won the headlines, RAG won the bills. The 1 million token context windows are real and they are useful, but they are economically catastrophic if you treat them as a substitute for retrieval. The math is not subtle. Published benchmarks show RAG queries running at around 0.00008 dollars each while pure long-context queries on equivalent corpora run around 0.10 dollars. That is roughly a 1,250-fold difference. Latency matches: end-to-end RAG queries land near 1 second on the same workload where long context configurations need 30 to 60 seconds.

The cost comes from the obvious place. You are paying for input tokens. A 100K-token prompt on a frontier model is about 0.20 dollars of input alone. A 1M-token prompt is around 2 dollars before output. Multiply that by a few hundred thousand user queries a month and you have built a per-document hotel where every guest pays full rent for every other guest’s room. Most providers also moved to tiered pricing in 2026 specifically to discourage this pattern. Gemini 3.1 Pro charges 2 dollars per million input tokens under 200K and 4 dollars per million above it. GPT-5.4 charges 2.50 dollars per million for the first 272K tokens and 5.00 dollars per million after that. Claude removed its long-context surcharge in March 2026, but the underlying compute reality is the same: serving 2 million tokens of input is dramatically more expensive than 200K, and the bill finds you eventually, either through pricing or rate limiting or both.

So the heuristic is: if the user’s query needs a small, specific slice of a large corpus, RAG it. If the user’s query needs the whole document end to end, long-context it. The rare case is the second one. The common case is the first one. Most LLM applications have a corpus that is much larger than what any one query actually needs, which is the textbook RAG setup. RAG is back not because it became better. RAG is back because long context became expensive enough to make the always-stuff-it crowd quiet.

The thing I tell builders to do here is measure recall on real queries, not on benchmarks. Long context benchmarks like needle-in-a-haystack hit 99.7 percent recall, which sounds great until you realize multi-fact recall on real queries lands closer to 60 percent. That gap is not academic. It is the gap between a product that answers correctly and a product that confidently makes things up. RAG with retrieval scoring and reranking lands much higher precision in production, especially for queries that need to pull facts from multiple parts of a corpus. The pattern that is winning right now is hybrid: RAG for the recall, long context as the synthesis surface, with a retrieval scorer that decides per-query which mode to invoke. That is a Layer 3 engineering decision, not a model choice.

Layer 4: Conversation History, the Quadratic Killer

If Layer 2 is the most surprising tax, Layer 4 is the deadliest. This is where the quadratic math lives. Every agent loop re-sends prior turns. Every turn adds to prior turns. Every additional turn pays for every prior turn again. A 20-step loop that adds 2,000 tokens per turn does not cost 20 turns times 2,000 tokens of input. It costs roughly 20 times the system prompt plus the sum of 2,000 times 1, plus 2,000 times 2, plus 2,000 times 3, all the way up to 2,000 times 20. That last term alone is more than the first ten combined. The published cost analyses are consistent: agents typically burn around 50 times more tokens than equivalent chat applications, and the same prompt and conversation history is re-billed by step 20 about 20 times.

The lever is compression or reset. Compression means after every K turns, you replace the raw history with a model-written summary of it. The history shrinks. The next turn pays for the summary, not the raw. The trick is choosing K. Too aggressive and you lose state. Too conservative and the quadratic comes back. In my own builds I have settled on K of about 8 for short-loop agents and K of about 4 for code agents that produce verbose tool outputs. Reset is the harder version. You break the agent into coordinator and specialists, give each specialist a one-shot scope, and reset the conversation history every time the coordinator hands off. The specialists do their work, return a short structured artifact to the coordinator, and the long history never accumulates anywhere. Anthropic’s own published patterns are converging on this shape. It is what the production-grade agent frameworks are quietly doing under the hood.

The behavioral fix that goes with this is harder. Stop measuring agent quality by total tokens or total turns. Measure it by tokens per finished task. A 12-turn agent that finished the task is better than a 6-turn agent that did not. A 12-turn agent that uses 30K tokens total is better than a 12-turn agent that uses 200K. The right unit of economics is the task, not the call. Until your monitoring shows you tokens per finished task at the percentile level, you are flying blind. P50, P95, and P99 of tokens per finished task are the three numbers that should be on the wall behind your engineering team.

Layer 5: Tool Outputs, Where Agents Bloat

Layer 5 is the layer most teams ignore because it does not feel like cost. It is cost. Every time your agent calls a web search, it gets back 10,000 to 40,000 tokens of raw HTML or JSON. Every time it calls a database tool, it might get back 5,000 tokens of rows. Every time it reads a file, the whole file enters context. And then on the next turn, the API re-sends all of that as part of the conversation history. The tool output you got 18 steps ago is in your bill on step 19, on step 20, and on step 21. You are paying for the search results from earlier in the conversation forever, until you compress or reset.

The lever is summarize, and the rule is simple. Raw tool output never re-enters context. The agent reads it, extracts the structured fact it needed, writes a one or two sentence note into the conversation, and discards the rest. If the agent might need the raw output again later, the raw output goes to durable storage with a pointer, not to the conversation. This is the difference between a research agent that costs 50 cents per task and one that costs 8 dollars per task on identical workloads. Same model. Same query. Same answer. The 50 cents agent reads, extracts, discards. The 8 dollar agent reads, keeps, re-reads, keeps, re-reads.

This is also where MCP tool design quality matters more than people admit. A good MCP server returns small, scoped responses. A bad one returns the whole table. Most public MCP servers are bad ones, because they were written before the cost reality landed. When you pick which MCP servers to expose to your agent, return shape is a first-class criterion now. If the server cannot give you a paginated or filtered response, wrap it in a thin proxy that can. The cost of running an unwrapped chatty MCP server inside an agent loop is not abstract. It is a line on your bill that grows quadratically.

Layer 6: The User Query, the Sacred Layer

Layer 6 is the only layer you should let grow without engineering. It is the user’s actual question. It is what they are paying you for. The rule is straightforward: the user’s query enters cleanly, in its raw form, with no fingerprinting and no padding. Every other layer exists to make this layer cheap to serve.

The reason I call it sacred is that I have watched teams pollute it. They will inject a paragraph of context into every user query before sending it to the model. They will add a “for context, you are talking to Sarah, a product manager at Acme who is on the Pro plan” preamble to every turn. This is fine for the first call. It is catastrophic when the user has a 30-turn conversation, because Sarah’s preamble gets re-billed 30 times, even though Sarah does not need to be reintroduced. Personalization belongs in the system prompt cache, not in the user turn. The user turn is the only layer where token growth is acceptable, because it is the only layer where the user is paying for what they are saying.

The Quadratic Math Every Builder Must See

I want you to feel the math. The supporting diagram below shows two curves on the same axes. The horizontal axis is turn count. The vertical axis is cumulative input tokens billed. The red curve is the naive agent loop. The green curve is the same agent with compression at K equals 8 plus cache plus tool output summarization. Both agents do the same work. Both reach the same answer. The red curve is what most production agents look like today. The green curve is what context-engineered agents look like. The gap is the discipline.

The reason this chart matters is that nobody experiences the quadratic in development. A developer runs three or four turns to test, sees a small bill, ships the agent, and falls off a cliff in production where real users take 15 to 25 turns. The bill arrives a month later. By then the architecture is set, the customer expectations are set, and changing the loop is a refactor. The right time to engineer the green curve is the first week, not the fifth month. The cheap moment to install compression and caching is before anyone is using the product. It costs roughly nothing to add and roughly everything to add later.

The Contrarian Read: RAG Was Not Dead, It Was Early

For most of 2024 and 2025, the dominant story in AI engineering was that RAG was dying. Long context windows would replace it. The model would just hold the whole corpus and you would not need retrieval. Smart people wrote thousands of words on this. They were not wrong about the technical capability. They were wrong about the economics. By mid 2026 the economics had spoken, and the verdict is unambiguous: RAG was not dead, it was early. The 1,250-fold cost difference between RAG and pure long context is not a minor optimization. It is the difference between a viable AI product and a venture-funded science project.

The deeper read is even more uncomfortable for the long-context maximalists. RAG is not just cheaper. It is also more accurate on the queries that matter most. Multi-fact retrieval inside a giant context is hard, and recall lands closer to 60 percent than the 99.7 percent that needle-in-a-haystack benchmarks suggest. Production teams I talk to are quietly walking back from “we just stuff the whole thing” to “we retrieve precisely and synthesize tightly.” The pattern that is winning is not RAG OR long context. It is RAG INTO long context, with retrieval doing the filtering and the long context window doing the synthesis. That is the production architecture of 2026. It is what context engineering looks like at the corpus layer.

The other contrarian read is about the model itself. The dominant narrative for the last two years has been “the next moat is the model you can build or fine-tune.” The dominant narrative for the next two years is going to be “the next moat is how few times you have to call the model.” If your agent can solve a task in 6 turns instead of 20, you have a 10x cost advantage at the same per-token price. If your competitor has to call the model more often to do the same job, they are running a structurally inferior unit economics business, regardless of how good their model picker is. Context engineering is how you compound this advantage. The moat is not in the model. It is in the discipline of the calls.

This is the part most founders miss. They think the AI cost paradox is a vendor problem, or a model problem, or a market problem. It is none of those. It is an architecture problem. Microsoft did not cancel Claude Code because Claude is too expensive. Microsoft canceled Claude Code because the loop economics of agentic coding inside Microsoft’s wrappers did not have context engineering, so the per-engineer-per-month bills hit 500 to 2,000 dollars and the product math broke. The model is not the bug. The architecture around the model is the bug. The teams that internalize this win.

The Cost Floor Audit and Your Monday Plan

Here is the part you can run yourself, today. I built a six-question audit you can score on any AI product, including one you have not shipped yet. Each question is worth 0, 1, or 2 points. The total is out of 12. The band tells you how close your architecture is to the cost cliff.

Here is what I do with founders I advise the Monday after they take this audit.

Monday: Score the audit. Be honest. If you do not have measurement in place to answer question 6, write a 1 in that row, not a 2. Then identify your lowest-scoring layer and pick exactly one move to fix it this week. Not five moves. One. The teams that try to fix everything at once fix nothing.

Tuesday: If your weakest layer is Layer 1, ship prompt caching on the single highest-traffic route. If it is Layer 2, run the MCP audit and drop the bottom three servers. If it is Layer 4, install K-equals-8 history compression on the longest-running agent. If it is Layer 5, write the summarize-then-discard rule into your tool output handler.

Wednesday: Deploy the change behind a feature flag at 10 percent of traffic. Measure tokens per finished task at P50 and P95 on the flagged versus unflagged traffic for 24 hours.

Thursday: If P95 tokens per task dropped at least 20 percent and quality held, roll the change to 100 percent. If quality dropped, tune the knob. If neither happened, your fix was on the wrong layer. Go back to the audit and pick the next layer.

Friday: Write the change up in a one-page internal doc. The teams that compound this discipline turn it into institutional muscle. The teams that fix once and walk away regress in two months.

If you are a solo founder, run this loop weekly for the next eight weeks. You will compound the savings. If you are a tech lead with a team, run this loop monthly per agent. Either way, the work is the same: pick a layer, fix it, measure it, lock it in, move to the next layer. The Cost Floor Audit is not a one-time scoring exercise. It is the ongoing operating cadence of an AI product that survives.

FAQ

What is context engineering, in one sentence?

Context engineering is the discipline of deciding, for every layer of context that enters an LLM call, what enters, when it leaves, and how it is cached, so that the cost of finishing a task does not grow quadratically with the work.

How is context engineering different from prompt engineering?

Prompt engineering is what you write inside Layer 1 to get good behavior. Context engineering is the architecture decision about what enters all six layers and how they are billed. A great prompt inside a bad context architecture still produces a quadratic bill. Prompt engineering is local. Context engineering is structural.

Why are my AI bills exploding even though per-token prices keep falling?

Because you are billed per call, every call sends the full context window, and modern agents call the model many times per task. The per-token price has fallen about 280-fold in two years. The number of tokens it takes to finish a complex agent task has risen by more than that. If you do not control the context per call, the price collapses do not reach your P&L.

Is RAG dead in the era of 1 million token context windows?

No. RAG is winning the bills. Production data shows roughly a 1,250-fold cost difference between RAG and pure long-context approaches on equivalent workloads, with RAG also returning faster and often more accurately on multi-fact queries. The right pattern is RAG into long context, where retrieval filters and the long window synthesizes.

What is the single biggest hidden cost in modern AI agents?

Re-sent prior context. Industry breakdowns show roughly 62 percent of agent bills come from the conversation history being re-billed on every turn. Layer 4 compression is the single highest-impact fix for most teams.

How much does MCP cost in tokens, really?

A lot more than founders realize. Public reports show 7 active MCP servers consuming about 67,300 tokens (33.7 percent of a 200K Sonnet window) before any conversation starts. Heavy real-world setups can hit 72 percent of the context window in tool metadata alone. Every connected MCP server is a recurring token bill, paid whether or not the user invokes its tools.

Should I use prompt caching from day one?

Yes. Prompt caching is not an optimization to add later. It is the default. Anthropic, OpenAI, and Bedrock all support it. Cache writes are 1.25x to 2.0x the standard input rate. Cache reads are 0.10x. That is a 90 percent discount on every reused prompt prefix, from day one, with no behavior change to your product.

How do I measure if my AI product is well context-engineered?

Track tokens per finished task at P50, P95, and P99. If you only track total tokens, you are flying blind. If your P95 tokens per task is more than 3x your P50, you have a long-tail context problem. If your P50 is growing month over month at constant work shape, you have a compression problem. The percentile view is the only one that tells you what your bill is going to do.

If this resonated, the related deep dives that go best with it are Why AI Agents Fail in Production, How Founders Should Think About AI, The AI Adoption Maturity Model, Building AI Agents That Make Money, The AI Wrapper Trap, The Data Moat Playbook, and The AI Opportunity Map 2026.