The Context Window Trap: Bigger Isn’t Better

Every frontier lab is racing on the same number right now. OpenAI shipped the GPT-5.6 family on July 9 with a one million token context window on every tier, Luna through Sol. Gemini already advertises two million. Claude Sonnet sits at one million. The spec sheets keep climbing, and the pitch is always the same: fit your whole codebase, your entire knowledge base, a year of customer tickets, all of it, into a single prompt. No more retrieval, no more chunking, no more work. Just paste and ask.

Here is the durable version, the one that will still be true when the number on the spec sheet reads ten million. A bigger context window is not more usable context. The number the lab prints is the size of the box. It is not a promise about how much the model can actually hold in its head at once and still answer correctly. Those are two different quantities, and the gap between them is where founders quietly lose accuracy, money, and speed without ever seeing the bill.

I run two companies where models do most of the production work. The most expensive mistakes I have watched founders make with AI in the last year were not model choices. They were context choices. They stuffed the window because it was there, the demo looked great, and then the thing got slower, more expensive, and quietly wrong in production. This post is the map I wish I had handed them before they filled the window.

Table of contents

• The problem: a window is not a hard drive
• The framework: the usable window
• The reliability tax: context rot is real and measured
• The cost tax: attention is quadratic
• The latency tax: the window you fill is the wait you pay
• The fix: context engineering, not context maxxing
• The curation ladder: what earns a place in the window
• The contrarian take: a bigger window needs more discipline, not less
• What to do Monday morning
• FAQ

The problem: a window is not a hard drive

The context window is the amount of text a model can take as input for a single request. Founders read the number as storage. Storage has one rule: if it fits, it works the same. A one terabyte drive holds a one gigabyte file exactly as faithfully as a one megabyte file. Nothing about the file changes because the drive is big.

A context window does not behave like that. It is not a drive, it is a field of attention, and attention is a budget that gets spread thinner the more you put in front of it. The model does not read your context the way you read a page, one line after another with equal care. It weights every token against every other token and decides, on the fly, what to pay attention to. When you hand it fifty thousand tokens to answer a question that only needed four hundred, you did not give it more help. You gave it forty-nine thousand six hundred tokens of noise to sort through, and you asked it to find the signal on its own.

This is the trap. Filling the window feels like diligence. You gave the model everything, so surely it has what it needs. But the industry has now measured what actually happens, and the results are not close to the storage intuition. In the 2026 State of Context Management survey, 82 percent of IT and data leaders said prompt engineering alone is no longer enough to run AI at scale, and 95 percent of data teams planned to invest in context engineering training this year. The reason is blunt: most agent failures in production are not model failures, they are context failures. The model was capable. The window it was handed was wrong.

The stakes are not abstract. A window packed past what the model can use degrades in three specific, measurable ways at once. It gets less accurate, it gets more expensive, and it gets slower. I call those the three taxes, and every founder pays all three the moment they treat the window as free space. Before the taxes, though, you need the one number that tells you how much of your window is real.

The framework: the usable window

Here is the single idea to take from this post. The advertised window and the usable window are different numbers, and only one of them is on the spec sheet.

The advertised window is the marketing number. One million tokens, two million tokens, whatever the lab prints. It is the maximum input the API will accept without erroring.

The usable window is the fraction of that maximum where the model still holds accuracy on your task. Past the usable window, the model keeps accepting tokens and keeps producing fluent answers, but the answers get less reliable in ways that do not announce themselves. No error is thrown. The response still reads well. It is just more likely to be wrong.

The lever that moves the usable window is not the model. It is what I call relevance density: the share of the tokens in your window that actually bear on the answer. Signal tokens divided by total tokens. If you send four hundred tokens that matter inside a fifty thousand token prompt, your relevance density is under one percent. The model is doing a search problem before it can do your problem, and every irrelevant token is a chance to be pulled off course.

High relevance density is the whole game. It is why a tight four hundred token prompt often beats a fifty thousand token one on the exact same model. Not because less information is better in the abstract, but because the tokens you removed were not helping and were quietly hurting. This connects directly to a metric I have written about before, cost per correct task. A denser window raises the correct rate and lowers the token bill in the same move.

The reliability tax: context rot is real and measured

The first tax is the one founders refuse to believe until they see the numbers, because it contradicts the storage intuition so directly. As you fill the window, the model gets less accurate at using what is in it. This is not a rumor. It has a name now, context rot, and it has been measured across the whole frontier.

In July 2025, Chroma Research published a study titled Context Rot: How Increasing Input Tokens Impacts LLM Performance. Kelly Hong, Anton Troynikov, and Jeff Huber ran careful experiments across 18 frontier models, the same ones you are choosing between, on retrieval and multi-hop reasoning tasks from ten thousand tokens up to the top of their advertised windows. Every single one of the 18 models showed accuracy falling as the input grew. Not one held uniform accuracy across its full advertised window. On needle-in-a-haystack style retrieval, drops of 20 to 50 percent appeared just going from ten thousand to a hundred thousand tokens, and the steepest fall showed up in the hundred thousand to five hundred thousand range. The window that says one million on the box was already coming apart at a fifth of that.

This sits on top of an older, sturdier finding. The Lost in the Middle research showed that models use information best when it is at the very start or the very end of the input, and worst when it is buried in the middle. Accuracy follows a U shape across position, and burying the key fact in the middle of a long context could cost more than 30 percent. The architectural reason is unglamorous: the position encoding most models use, RoPE, has a long-term decay property that lowers the attention weight between tokens that are far apart. Distance in the window literally reduces attention. It is baked into how the model reads.

The Chroma work added two findings that should change how you build. First, semantic similarity drives the decay more than raw length does. When the thing you are looking for stands out from its surroundings, the model finds it even in a long context. When the distractors look a lot like the answer, accuracy drops hard, and it drops harder the longer the context gets. Real business context is full of near-duplicates: three versions of a contract, five similar support tickets, last quarter’s plan next to this quarter’s. That is the worst case, and it is the normal case.

Second, and this one is genuinely counterintuitive, structured coherent text degraded attention more than shuffled text. Across all 18 models, feeding a logically ordered document sometimes hurt more than feeding the same sentences in random order. The neat, well-organized knowledge base you are so proud of is not automatically the easy case for the model. Order that helps a human read can pull a model’s attention in ways that hurt retrieval.

Put the reliability tax in plain founder terms. If you build an agent that reads a large window on every step, and the model loses even a quarter of its retrieval accuracy at the sizes you run at, you have shipped a product that is wrong more often than your demo suggested, and it is wrong silently. This is exactly the failure mode I mapped in AI agent observability: the model does not tell you it lost the thread, so you find out from a customer. Context rot is one of the biggest silent-failure engines there is, and stuffing the window is how you turn it on.

The cost tax: attention is quadratic

The second tax is the one that shows up on your invoice, and it is worse than linear. Most founders assume tokens cost what the price sheet says, so a prompt twice as long costs twice as much. That is true for what you pay the API per token. It is not true for the underlying work, and the underlying work is what sets prices over time.

The attention mechanism at the core of a transformer compares every token to every other token. Double the number of tokens and you roughly quadruple the number of comparisons. Two times the context is four times the attention compute. This is why the labs cannot simply make context free even when they want to, and why the price of very long context does not fall as fast as the price of short context.

The direct bill adds up faster than people expect. Take a retrieval setup that pulls two hundred chunks of about two hundred fifty tokens each into the window, running ten thousand queries a day. That is roughly sixteen hundred dollars a day in inference, or forty-eight thousand dollars a month, for one feature. A large share of those tokens were retrieved just in case, not because the answer needed them. Cut the window to the tokens that actually bear on the answer and the bill falls in direct proportion, while the accuracy goes up because you also cut the distractors. You almost never get to lower cost and raise quality with the same lever. Relevance density is that lever.

This tax compounds with a structural risk I covered in the AI compute subsidy. Today’s token prices are held down by a capital cycle that is spending far ahead of revenue. When that cycle normalizes, the first prices to move will be the expensive ones, and long context is the expensive one because it is quadratic. A business whose margins depend on stuffing huge windows is a business with a borrowed bet on someone else’s capex staying generous. A business built on tight, dense windows is insulated. Same product, very different exposure, and the difference is a build habit.

The latency tax: the window you fill is the wait you pay

The third tax is the one users feel before they can name it. A model answers in two phases. First it reads the entire input, the prefill phase, and only then does it start generating. Time to first token is dominated by prefill, and prefill scales with how much you put in the window. A short prompt starts answering in a blink. A very long prompt makes the user stare at a spinner while the model works through everything you handed it, most of which it did not need.

The degradation is not linear, it accelerates. As inputs get very large, attention operations take over the compute, and the wait grows faster than the token count. At the extreme, generating from a full one million token context is measured in thousands of seconds per token in research settings, with over 96 percent of that time spent on attention. You will not run at that extreme in production, but you feel the same curve at fifty and a hundred thousand tokens: seconds of prefill where you wanted milliseconds. There is a memory cost hiding here too. Holding the intermediate state for a one million token context, the KV cache, runs to over a hundred gigabytes, which is why long context is an infrastructure decision, not just a prompt decision.

For anything a human waits on, the latency tax is a product tax. It shows up as abandoned sessions and a UI that feels sluggish next to a competitor who kept their windows tight. For an agent that loops, it is worse, because you pay the prefill on every step, and a slow step multiplied across a long loop is the difference between an agent that finishes a task in seconds and one that grinds. I wrote about designing those loops in loop engineering for AI agents, and context size is one of the biggest levers on how a loop feels and costs.

A worked example: the support agent that got slower and wronger

Let me make the three taxes concrete with a scenario I have watched play out more than once. A founder builds a support agent. The first version retrieves the three most relevant help articles for each ticket and answers. It is fast, cheap, and about as accurate as the human it replaced. Then someone has the idea that fits the storage intuition perfectly: the window is a million tokens now, so why retrieve at all? Load the entire help center, every past ticket, the product docs, all of it, into the window on every call. The model will just have everything. The demo on a handful of tickets looks flawless.

In production, all three taxes arrive together. The reliability tax hits first and quietest. The help center is full of near-duplicate articles, three versions of the refund policy, five similar troubleshooting guides, and those are exactly the distractors the Chroma study showed drag accuracy down as context grows. The agent starts citing the outdated refund policy because it sits in the middle of the window where attention is weakest, and nobody notices until a customer quotes it back. The cost tax shows up on the monthly invoice, which has gone from a few hundred dollars to five figures, because every ticket now prefills a hundred thousand tokens whether it needed them or not, and attention on that is quadratic. The latency tax shows up in the reviews: replies that used to feel instant now take several seconds, because the agent reads a novel before it writes a sentence.

The founder’s first instinct is to reach for a better model, or to ask the vendor for an even bigger window. Both make it worse. A better model reads the bloated window slightly better and costs more, so the bill climbs while the underlying problem, a window that is one percent signal, stays. A bigger window removes the last natural cap on how much they stuff. The actual fix is to go back to the first version’s discipline: retrieve the three articles that bear on this ticket, keep the window dense, and let the model do the easy job of answering from clean context instead of the hard job of finding the answer in a haystack. Same model, one tenth the cost, higher accuracy, instant replies. This is the pattern behind most reliability claims that fall apart once real traffic hits: the demo ran on a small, clean context, and production ran on a stuffed one.

The lesson generalizes past support. Any time an AI feature gets slower and more expensive as you add data to it, and quietly less accurate, suspect the window before you suspect the model. The symptom set is the signature of context maxxing, and it survives every model upgrade until you fix the curation. It is also why I treat context size as a first-class variable when deciding how a business survives model churn: a product tuned to run on dense, small windows ports cleanly to the next model, while one that depends on a specific giant window is betting the next model prices or behaves the same way.

The fix: context engineering, not context maxxing

The reflex the three taxes punish has a name worth saying out loud, because naming it makes it easier to catch yourself doing it. Context maxxing is the habit of filling the window because it is there. It feels like thoroughness. It is actually the most expensive and least reliable way to give a model information.

The discipline that replaces it is context engineering, and it is the fastest-rising skill in applied AI for a reason. Prompt engineering is about how you phrase the request. Context engineering is about what information is in the window when the model reads it, and in what order, and how it got there. The 2026 shift is that the second question turned out to matter far more than the first. Most teams that think they need a bigger model, or a bigger window, actually need better curation of the window they already have.

Context engineering is a handful of concrete moves, not a mindset. Retrieval, so you pull only the passages that bear on this specific query instead of loading the whole corpus. Compression, so long histories and documents get summarized down to their load-bearing parts before they enter the window. Ordering, so the tokens that matter most sit at the very start and the very end where the model attends best, and the filler, if any survives, sits in the weak middle. Tool scoping, so an agent’s tools return narrow results instead of dumping four thousand search hits into the window. Memory, so state persists across turns without re-pasting everything every time. None of these are exotic. All of them raise relevance density, which is the one number under all three taxes.

This is also the honest answer to a question I get constantly: do we still need retrieval now that windows are huge? Yes, more than before. The bigger the window, the stronger the temptation to skip retrieval and paste everything, and the higher the price when you do. Retrieval is not a workaround for small windows. It is how you keep relevance density high inside a large one. The teams treating the two million token window as a reason to delete their retrieval layer are walking straight into all three taxes at once.

One more move belongs here because it prevents the whole problem: know what correct looks like before you automate. If you cannot say which tokens bear on the answer, you cannot curate the window, and you probably cannot evaluate the output either. That is the same specification discipline behind the evals playbook. A team that can specify correctness can curate context. A team that cannot will stuff the window and hope, which is where agents fail in production.

The curation ladder: what earns a place in the window

Curation sounds like judgment, and it is, but it runs on one repeatable test. For every candidate token, every document, ticket, code file, prior message, ask a single question: does this change the answer? Then place it on the ladder in the diagram above.

Tokens that change the answer and are needed on every call earn a permanent place, and they go at the ends of the window, not the middle, so the model attends to them. Tokens that matter only in gist get summarized first, so you keep the meaning and drop the length. Tokens that are needed sometimes but not this time get pulled in by retrieval when the query calls for them, not loaded by default. Tokens that feel relevant but do not actually change the answer are distractors, and they get cut, because they lower relevance density and, per the Chroma finding, similar-looking distractors are exactly what drags accuracy down as length grows. And tokens that are in the window just in case, where nobody can articulate what they are for, get cut first, before anything else.

Here is the comparison that makes the two philosophies concrete.

And here is the three taxes summary in one place, so you can hand it to whoever owns your AI features.

The contrarian take: a bigger window needs more discipline, not less

The industry sells a bigger context window as a way to do less work. Stop chunking, stop retrieving, stop thinking about what goes in the prompt, the window is big enough now. That is exactly backwards, and it is the most expensive misread in applied AI today.

A bigger window does not reduce the need for curation. It raises it. Every one of the three taxes gets worse with size, not better. The reliability tax is largest at the long end of the window, because that is where context rot bites hardest. The cost tax is quadratic, so the marginal token in a huge window costs more than the marginal token in a small one. The latency tax accelerates with length, so the pain per added token grows. A bigger window is a bigger gun pointed at all three, and the trigger is the temptation to fill it.

So here is the hard version. Most of the time a founder tells me they need a bigger context window, they do not have a capacity problem. They have a curation problem wearing a capacity costume. The window they already have is big enough for the tokens that actually change the answer, several times over. What they are missing is the retrieval and compression and ordering that would put the right four hundred tokens in front of the model instead of the wrong fifty thousand. Buying more window does not fix a curation problem. It makes it more expensive to have.

And the point people reach for to escape this, better models will fix it, does not hold either. Better models genuinely raise the baseline. GPT-5.6, Sonnet 5, the July coding-agent upgrades, they all read long context better than last year’s models did. But every one of the 18 models Chroma tested still degraded with length. Better models lower the failure rate, they do not remove the taxes. Attention stays quadratic no matter how good the model is. Prefill still scales with input. And a smarter model that you trust more is a model you are tempted to hand even more context, which walks you right back into the trap with more confidence and a bigger bill. The discipline is what scales. The window is not the answer, curation is. That is the same lesson underneath the AI trust gap: capability going up does not reduce the work of getting reliability, it moves the work.

What to do Monday morning

This is fixable this week, and most of the fix is subtraction, which is cheap.

Measure your usable window. Take your most important AI feature and find out how many tokens you actually send on a typical call. Most teams have never looked and are shocked. Then run your own small test: take a real task, put the key fact at the front, then in the middle of a long context, then at the end, and see how the answer changes. You will find your usable window is a fraction of the advertised one, and now you have the number.

Compute relevance density. For a real request, estimate how many of the tokens in the window actually bear on the answer. If it is under a few percent, you are paying all three taxes for almost nothing. That single ratio tells you how much room you have to cut.

Cut first, retrieve second. Before you add any retrieval sophistication, delete the just-in-case tokens, the ones nobody can defend on the ladder. Then move whatever is left that is only needed sometimes into a retrieval step so it enters the window on demand instead of by default. Cutting is free and it raises accuracy and lowers cost at the same time.

Put load-bearing tokens at the ends. If a fact must be in the window every call, place it at the very start or the very end of your prompt, never buried in the middle. This is a one-line change to your prompt assembly and it directly fights lost in the middle.

Watch for silent context failures. Because the reliability tax does not throw errors, you have to catch it deliberately. Log the length of the context on each call and check whether accuracy falls as length rises. If it does, you have found your usable window in production, and you know where to cap it. Tie this into whatever gets your pilots to production rather than leaving it as a demo.

None of this requires a new model or a bigger window. It requires treating the window as an attention budget you spend deliberately, which is the whole of context engineering. Do this and the same model gets more accurate, cheaper, and faster in one pass. That is rare, and it is already available, sitting inside the window you already have.

FAQ

What is the context window trap? It is the mistake of treating a model’s context window like storage, where bigger is always safer, and filling it with everything that fits. A context window is an attention budget, not a hard drive. Filling it past what the model can use makes answers less accurate, more expensive, and slower all at once, even though no error is thrown.

What is the difference between the advertised window and the usable window? The advertised window is the maximum input the API accepts, the number on the spec sheet, one million or two million tokens. The usable window is the fraction of that where the model still holds accuracy on your task. Research across 18 frontier models found none held uniform accuracy across their full advertised window, so the usable window is always smaller, often much smaller.

Is context rot real? Yes, and it is measured. Chroma Research tested 18 frontier models in July 2025 and every one lost accuracy as input length grew, with drops of 20 to 50 percent going from ten thousand to a hundred thousand tokens on retrieval tasks. It builds on the earlier lost-in-the-middle finding that models use information best at the start and end of the input and worst in the middle. The root cause is partly architectural, in how position encoding lowers attention between distant tokens.

If windows are so big now, do I still need retrieval? More than before. A bigger window raises the temptation to skip retrieval and paste everything, and raises the price when you do, because cost is quadratic and accuracy falls with length. Retrieval is how you keep relevance density high inside a large window, not a workaround for a small one. Deleting your retrieval layer because the window got bigger is walking into all three taxes.

What is context engineering? It is the discipline of controlling what information is in the window when the model reads it, in what order, and how it got there. It covers retrieval, compression, ordering, tool scoping, and memory. Prompt engineering is how you phrase the request; context engineering is what the model can see when it answers. In 2026 the second turned out to matter more, and most agent failures trace to context, not the model.

Why does a full window cost more than the price sheet suggests? The token price you pay is linear, but the compute underneath is quadratic, because attention compares every token to every other token. Double the context and you roughly quadruple the attention work. That is why very long context does not get cheaper as fast as short context, and why margins built on stuffed windows are fragile if token prices ever normalize.

Will better models make this go away? They lower the failure rate but do not remove the taxes. Every frontier model tested still degraded with length, attention stays quadratic, and prefill still scales with input. A better model you trust more is also one you are tempted to overload, which walks you back into the trap. The discipline of curation is what scales, not the window.

How do I find my usable window? Run a positional test on a real task: put the key fact at the front, the middle, and the end of a long context and watch the answer change. Then log context length against accuracy in production. Where accuracy starts falling as length rises is your usable window, and you cap context below it.

The window is not the product. Judgment about what goes in it is. If you want the bigger picture of where the durable advantages sit as models keep getting cheaper and larger, start with the AI opportunity map, and if you are wrestling with how much of your own thinking to hand the model, the synthesis skill is the companion piece. The context race will keep going. The founders who win it will be the ones spending the window on purpose while everyone else fills it.