Context Rot: Why Bigger Windows Make AI Worse

On September 3, GPT-6 Astra shipped with a 1.05 million token context window, and the timeline did what it always does. People pasted whole codebases into a single call. Someone dropped an entire data room in and asked for a memo. The pitch, roughly, was that you would never have to think about what to give a model again. Just put everything in. The window is big enough now.

I read the same launch notes everyone else did, and the number I stopped on was not 1.05 million. It was 272,000. That is the point where Astra starts billing you double for every token in the request. So the model that promises you never have to curate is also the model that charges you a premium the moment you stop curating. That is not a coincidence. It is the whole story of the last two years of AI, compressed into a pricing table.

Here is the claim I want to defend, and it is going to sound wrong to anyone who just watched the window get eight times bigger. A larger context window can make your product worse. Not slower. Not more expensive, though it is that too. Worse, as in less accurate, less reliable, more likely to miss the one fact that mattered. The size of the window is a ceiling on what the model could theoretically read. It tells you almost nothing about what the model will actually use. And the gap between those two numbers is where most AI products quietly break.

The failures I keep seeing are not the model being dumb

When something I build with a model goes wrong, my first instinct used to be to blame the model. Wrong answer, hallucinated citation, ignored the instruction I put right there at the top. Swap in a smarter model, the thinking goes, and the problem gets better. Sometimes it does. But over the last year I have watched the same failure survive every model upgrade I threw at it, and that is the tell. If a better model does not fix it, the model was not the problem.

The problem was almost always what I handed the model for that particular step. Too much. The wrong slice. The right document buried under nine near-duplicates. A conversation history so long that the instruction from turn two had been drowned out by turn forty. The engineers building production agents figured this out before I did. By the middle of 2025, the more experienced ones had stopped tuning prompt wording and started obsessing over what actually sits in the window at each step. The consensus that formed is blunt: most production failures in 2026 are not the model being dumb, they are context failures. The agent retrieved the wrong document. The relevant fact was there but in a position the model does not read well. The window was full of plausible junk. It is the same shape as the reliability paradox I have written about, where a smarter model does not make the system more dependable, because the hard part was never raw capability.

This matters more now than it did a year ago for a specific reason. When windows were small, you were forced to curate. A 4,000 token limit does not let you paste a data room. You had to decide what went in, and that discipline, annoying as it was, protected you. The model only ever saw a tight, relevant slice because that was all that fit. Now the window is a million tokens and the discipline is optional, which means most people drop it. They fill the window because they can, and they inherit a failure mode that the small window never let them have. The capability that was sold as a convenience removed the guardrail that was keeping the output good.

The advertised window is a container, the effective one is a budget

Here is the mental model I use now, and it is the one thing in this piece I would ask you to keep. Stop thinking of the context window as a container you fill and start thinking of it as a budget you spend.

A container has one property that matters: how much fits. If the window is a container, then a bigger window is strictly better, the same way a bigger hard drive is strictly better. Nothing you already put in gets worse when you add more. That is the intuition the marketing wants you to have, because it makes the big number sound like pure upside.

A budget works differently. Every token you spend has a cost, and the cost is not just money. It is attention. A transformer spreads a fixed amount of attention across everything in the window. Add a thousand tokens of background nobody needs, and you have not just wasted a thousand tokens. You have made every other token in the window a little harder for the model to find and weigh. I call this the dilution tax. It is the reason filling the window is not free even when the tokens are cheap. The junk does not sit quietly in a corner. It competes with your signal for the model’s attention, and sometimes it wins.

Once you hold the budget model in your head, the launch announcements read differently. A ten million token window is not a promise that the model will use ten million tokens well. It is a promise that it will accept them without crashing. Those are very different guarantees, and the marketing works hard to blur them. The useful question was never how big the window is. It is how much of it the model can spend before the answers start to drift. If you are mapping where the real AI opportunities sit, this is one of them hiding in plain sight: the gap between the advertised number and the usable one is a product, not a footnote.

Why more tokens thin out the attention

It helps to know why this happens, because the mechanism tells you it is not a bug that a patch will quietly fix. A transformer works by having every token attend to every other token, and the total amount of attention is fixed. Think of it as a set pool of focus that has to be divided across everything in the window. When the window holds 2,000 tokens, each one gets a meaningful share. When it holds 900,000, that same pool of focus is split hundreds of times thinner, and the tokens that actually matter get a smaller slice of the model’s attention than they would have gotten in a tighter window. Nothing broke. The math just spread the same attention over more things.

This is why the decline is smooth rather than a cliff. There is no token count where the model suddenly stops working. It gets a little less sharp with every block you add, which is exactly what makes context rot dangerous. A hard failure you would notice and fix. A gradual softening you ship, because each individual call still looks fine, and the damage only shows up in aggregate as a slightly higher rate of confident mistakes you cannot pin on any one thing. The cost of a bigger window is not paid in crashes. It is paid in a quiet drop in reliability that hides inside otherwise reasonable output. It is also, separately, paid in money and latency, which is the other half of the inference bill most teams underestimate when they design around the biggest window they can find.

There is a hopeful reading of all this, and it is that the mechanism also hands you the fix. If the problem is that attention gets divided, the answer is not a bigger pool, it is fewer things to divide it among. You cannot change how the model allocates its attention. You can change how much you ask it to spread that attention across. That one lever, how much you put in, sits almost entirely in your hands, and it is the lever most teams never touch because the big window told them they did not have to.

The usable window is often half the number on the box

This is not a vibe. It has been measured, repeatedly, and the numbers are worse than most people building on these models assume.

NVIDIA’s RULER benchmark defines a model’s effective context length as the longest input where it still holds above a fixed quality bar. Run that test and the advertised numbers fall apart. GPT-4 advertised a 128,000 token window. Its effective length under RULER was 64,000, half the sticker. And that is the friendly case. On complex tasks, the ones that require the model to chain several facts together across the window, the effective length can collapse to a small fraction of the advertised maximum, in some measurements dropping by up to 99 percent from the marketed number. You bought a million-token window and, for the kind of multi-step reasoning you actually care about, you are getting the reliability of a model with a window a fiftieth of that size.

In July 2025, Chroma published a study that put a name to this. They tested 18 frontier models, the ones you would actually reach for, GPT-4.1, Claude 4, Gemini 2.5, Qwen3, across a battery of controlled tasks. Every single one degraded as the input got longer. Not some of them. All 18. And the degradation showed up even on tasks that should be trivial, like retrieving a fact you planted or copying text back. The name they gave it, context rot, stuck because it describes the feeling exactly. The information is in there. The model just gets worse at using it the more of it there is.

The benchmark most people cite for long context, needle in a haystack, is part of why the problem stayed hidden so long. You hide a sentence in a long document and ask the model to find it. Models ace it, and the marketing quotes the score. But that test measures one narrow thing: can the model find a literal string it can pattern-match on. Change the test so the needle matches the question by meaning rather than exact words, the way real questions actually work, and scores fall hard. That harder version has a name too, NoLiMa, and the drop between the easy version and the realistic one is a decent measure of how much the headline long-context claims are worth. Which is to say, less than they look.

Models lose the middle of what you give them

Of all the ways context rot shows up, the one that has burned me most is position. Models do not read a long window evenly. They read the beginning well, they read the end well, and they get soft in the middle. Put the fact that decides the answer in the first paragraph or the last, and the model uses it. Bury that same fact halfway through a long window and accuracy on retrieving it can fall by more than 30 percent. Same fact. Same model. Same prompt. The only thing that changed is where in the pile it sat.

Think about what that does to a naive design. You build an agent that keeps a running conversation, or a support bot that carries the whole ticket history, or a research tool that stacks every retrieved document into one call. In every one of those, the important context is exactly where the model reads worst: somewhere in the middle of a growing pile, pushed there by whatever arrived before and after it. The system was not built to fail. It was built to accumulate, and accumulation walks your most important facts straight into the blind spot.

I have a personal rule now that came directly out of getting this wrong. If a fact must be used, it does not go in the middle. It goes at the very end of the context, right before the instruction, or it goes at the very top. The middle is for things I can afford the model to half-read. When I explain this to people, the common reaction is that it feels like a hack, like I am gaming the model instead of using it properly. Maybe. But I would rather build around how the model actually behaves than around how the spec sheet says it should.

A near-miss document hurts more than an empty page

Here is the finding that changed how I do retrieval. You would assume that adding a relevant-looking document to the context can only help, or at worst do nothing. It can hurt. Chroma and others found that distractors, content that is semantically close to what you want but not actually the answer, degrade performance more than plain filler does. A near-miss is more dangerous than a blank.

This is intuitive once you sit with it. Random filler is easy for the model to ignore because it does not look like an answer. A document that is on-topic, uses the right vocabulary, cites the right entities, but is subtly wrong or outdated, is exactly the thing the model is most likely to grab and run with. It looks like signal. The better your retrieval system is at finding topically similar content, the more of these convincing near-misses it feeds the model, and the more chances the model has to answer confidently from the wrong source. A dumber retriever that returned obvious garbage would, perversely, be safer, because the garbage would get ignored.

So the instinct to be generous with retrieval, to grab the top twenty documents instead of the top three because more can’t hurt, is backwards. Each extra document you add past the truly relevant ones is a fresh distractor, a fresh chance for the model to latch onto something that reads right and is wrong. I now treat every document I put in the window as something that has to earn its place, not because of token cost, but because its presence actively taxes the model’s judgment on everything else.

Filling the window is the most expensive way to be wrong

Let me put the two costs together, because founders tend to feel one and ignore the other.

The money cost is obvious once you look. Long context is expensive to serve, and the pricing tells on it. Astra doubling its per-token rate past 272,000 tokens is not a random cash grab. It is the provider telling you, in the only language a pricing table has, that the back half of that window is expensive to run. When you stuff a window because you did not feel like curating, you are choosing to pay a premium for the privilege of degrading your own output. You pay more and you get worse. It is the rare decision that is bad on both axes at once. If you have ever been surprised by how fast agent token costs eat a margin, uncurated context is a large and usually unexamined part of the bill.

The reliability cost is the one that actually kills products, and it is invisible on the invoice. A stuffed window does not throw an error. It returns a fluent, confident answer that is subtly wrong because the model was reading through a fog of its own context. You do not find out at build time. You find out three weeks into production when a customer catches the model citing a policy you deprecated, which was still sitting in the middle of a context blob nobody pruned. The failure is silent, delayed, and hard to trace back to its cause, which is the worst combination of properties a failure can have.

This is why I have come to see “just put it all in the window” as a tax you pay for skipped engineering. It feels like the model is doing the work of figuring out what matters. It is not. It is accepting your unsorted pile and doing its best across all of it, and its best across all of it is worse than its best across the right slice. The work of deciding what matters did not disappear when the window got big. It just moved from build time, where you control it, to inference time, where the model improvises it, badly.

Feed the model, do not fill it

If the window is a budget, the job is not to fill it. It is to spend it well. The practice that has grown up around this even has a name now, context engineering, and it has largely replaced what people used to call prompt engineering. The distinction is worth stating plainly, because it reframes the whole job. Prompt engineering was about wording a single instruction well. Context engineering is about deciding, at every step, what set of tokens the model should see, and just as important, what it should not.

The people doing this in production converge on a small set of moves. I will describe them in plain language rather than dress them up, because they are not mine and they are not complicated.

The first is to keep most of what an agent knows out of the window entirely and pull it in only when a step needs it. Notes, past results, and reference material live outside the context and get retrieved on demand, so the window holds what this step needs and nothing else. The second is to shrink long histories into summaries once they get big, so a forty-turn conversation becomes a tight paragraph of what actually happened instead of forty turns of raw transcript. The third is to hand narrow subtasks to separate calls with their own clean windows, so the deep, messy work of one subtask never pollutes the main line of reasoning. A research subtask can churn through fifty documents in its own context and hand back a three-sentence answer, and the main agent never has to carry those fifty documents at all. This is the same instinct behind routing work across more than one model: the point is to keep each call small and fit for its job rather than to run one giant call that tries to hold everything.

The fourth move is caching, reusing the stable front of a prompt so you are not re-paying to process the same system instructions on every call. That one is mostly about cost and speed rather than accuracy, but it matters when you are running these things at volume. Put together, the pattern is the opposite of the million-token pitch. The frontier of good practice in 2026 is not stuff more in. It is get more out of the window while keeping less in it.

Compaction is maintenance, not a rescue

One habit separates the teams whose agents hold up over long tasks from the ones whose agents fall apart. The good teams compact early, as routine maintenance, and the struggling teams compact only when the window is about to overflow, as an emergency.

Compaction just means periodically replacing a long, raw history with a tighter summary of what mattered. If you wait until the window is nearly full to do it, two bad things have already happened. The rot has already set in, so the summary you generate is being written by a model that is already degraded by the bloated context you are trying to fix. And you have been running every step in between through that same fog. Anthropic’s guidance on this is to trigger compaction well before the window fills, on the order of every 5,000 to 20,000 tokens for lighter workloads and 50,000 to 100,000 for heavier ones, so that trimming the context is a normal part of the loop rather than a last resort. In their own tests, editing the context to keep it clean gave a 29 percent lift on a long task, and pairing that with an external memory the agent could write to and read from pushed it to 39 percent. Those are not rounding errors. That is the difference between an agent that finishes a long job and one that loses the thread halfway through.

The mindset shift is to stop treating the context window as storage and start treating it as a working desk. You do not pile every document you have ever touched on your desk and work on top of the heap. You keep the two or three things this task needs in front of you and everything else in drawers. The model works the same way, whether or not the interface makes that obvious. A clean desk is not a nice-to-have. It is the condition under which the model does its best work.

The window is working memory, not a hard drive

The deepest mistake behind a stuffed window is a category error. People treat the context window like storage, a place to keep everything the model might ever need, when it behaves far more like working memory, the small set of things you are actively holding in mind right now. You would not try to hold a hundred documents in your head at once and reason across all of them at the same time. You hold a few, and you walk back to the shelf for the rest. The window is the head, not the shelf, and building as if it were the shelf is how the trouble starts.

Once you see it that way, the design follows on its own. The shelf, the durable store of everything the agent knows, lives outside the window, and you pull items onto the desk only when a step actually needs them. This is the same reason giving an agent real memory is a different problem from giving it a big window. Memory is about what the agent can retrieve and carry across time. The window is about what it is holding this instant. Conflate the two and you get an agent that has technically been told everything and can reliably use almost none of it.

It also changes what you can safely hand off. The same discipline is what makes handing real work to an agent hold up, because an agent you cannot trust to hold the right context for a step is an agent you cannot trust with the step. When people say an agent is not ready for some task, the diagnosis is usually not that the model is too weak. It is that nobody built the machinery to keep the agent’s window pointed at the right few things while it works.

The job title changed because the bottleneck moved

You can see this whole shift show up in who gets hired. The role called prompt engineer, which had its eighteen months of being the hot title, has largely vanished from 2026 job listings. What replaced it is a set of roles built around managing information flow into models: applied AI engineers, agent engineers, and increasingly people whose title is some version of context engineer. Titles are a lagging signal, but they are an honest one. Companies do not invent a job category for a problem they think is solved.

The money follows the same line. In one recent read of the field, 89 percent of teams said they plan to invest in context management infrastructure within the next twelve months. Sit with that number. It is not 89 percent planning to invest in better models, or bigger windows, or more compute. It is 89 percent planning to spend on the plumbing that decides what goes into the window and what stays out. The industry has quietly concluded that the constraint is no longer the size of the window or the intelligence of the model. It is the discipline of what you feed it. The center of gravity moved from the model to the material.

For founders, that relocation is the actual news, and it is more useful than any window-size headline. It means the durable skill on your team is not prompt-whispering and it is not chasing the top of the model leaderboard. It is the far less glamorous work of building the retrieval, the summarization, and the routing that keep each call lean. That work does not get obsolete when the next model ships, because a better model with a rotted context still gives you a rotted answer.

The best document in your pile can lower your accuracy

If you take one counterintuitive thing from this, take this. Adding your single most relevant document to the context can make the answer worse, and whether it does depends entirely on what else is in there with it.

Most people model context as additive. Each good thing you put in raises the odds of a good answer, each irrelevant thing does nothing, so the safe move is to be generous and let the model sort it out. That model is wrong in a way that costs you. Context is not additive, it is competitive. Everything in the window competes for a fixed pool of attention, so a genuinely useful document dropped into a pile of nine near-misses can lose that competition to a distractor that happens to sit in a position the model reads better. You did everything right, you retrieved the correct source, and the surrounding noise buried it. The quality of any single piece of context is not a property of that piece. It is a property of the whole window it sits in.

This is why I no longer trust the phrase “grounded in your data” when it comes with a claim that more data means more grounding. Grounding is not a volume knob. Past the point where you have given the model the facts it needs, every additional token is not reinforcement, it is interference. It ties back to something I keep coming back to about working with these tools, that the scarce skill is holding a high bar, and here the bar is knowing what to leave out. The teams that understand this ship retrieval systems that are aggressive about relevance and ruthless about cutting, and their answers are better for what they left out. The ones that treat their vector store as a thing to dump into the window as fully as possible are, without knowing it, tuning their product toward confident wrongness.

How to size your product to the effective window

Enough theory. Here is what I would actually do, and mostly do, when building on these models.

Find your model’s effective window before you design around its advertised one. Do not trust the spec sheet. Run your own version of the test: take a real task from your product, plant the fact that decides it at various depths in a growing context, and watch where accuracy starts to slide. That depth, not the marketed maximum, is the number you design to. This is really just the evaluation discipline pointed at context length instead of at the model, and the same rule holds: a number someone else measured on a task that is not yours is close to worthless. It will be uncomfortably small, and knowing it is worth more than any benchmark someone else ran on a task that is not yours.

Put load-bearing facts at the edges. The instruction the model must follow and the fact it must use go at the top of the context or right before the question, never in the middle of a long block. This costs nothing and recovers a chunk of the accuracy that position was quietly stealing.

Cut your retrieval to what earns its place. If you are pulling the top twenty documents, try the top three and measure. In my experience the answer quality holds or improves, and the calls get cheaper and faster as a bonus. Treat every extra retrieved item as a distractor until it proves otherwise, not as free insurance.

Compact on a schedule, not in a panic. Build summarization into the loop so long histories get trimmed as routine, well before the window is full. If your agent runs long tasks, give it an external place to write notes and read them back, so its memory lives outside the window instead of clogging it.

And treat the window like a working desk for the rest of the build. Before every call, the question is not “what could I include,” it is “what does this step actually need,” and everything that fails that question stays out. The provider will keep making the desk bigger and charging you more to use the far end of it. You do not have to take the offer. The teams getting the most out of these models in 2026 are not the ones filling the largest windows. They are the ones who learned to hand the model less and got more back for it.

Questions founders ask about context windows and context rot

What is context rot in simple terms?

Context rot is the measured tendency of a language model to get less accurate as you put more into its context window. It is not about running out of room. Even well below the advertised limit, and even on easy tasks like finding a fact you planted, models degrade as the input grows. A 2025 Chroma study found all 18 frontier models it tested got worse with longer inputs. The practical takeaway is that a fuller window is not a better-informed model, it is often a more confused one.

If context rot is real, why do providers keep shipping bigger windows?

Because window size is a clean number that markets well, and because a bigger window genuinely does let you fit more in without an error. What it does not guarantee is that the model uses the back half of that window as well as the front. Both things are true at once: the window is bigger, and the reliable, effective portion of it is a fraction of the headline. The pricing usually admits this quietly, which is why some models charge a premium above a token threshold.

How do I find my model’s effective context window?

Test it on your own task rather than trusting a benchmark. Take a representative job from your product, plant the fact that determines the correct answer at different depths in a growing context, and track where accuracy starts to fall. The depth where reliability breaks is your effective window. It will usually be well under the advertised maximum, and on multi-step tasks it can be dramatically smaller.

Is retrieval or a bigger context window the better way to give a model knowledge?

For most production uses, retrieving a small, relevant slice beats dumping everything into a large window. A big window makes the model read through everything you gave it, including the parts that do not matter, and that dilutes its attention. Good retrieval keeps the window lean and the signal high. The catch is that retrieval has to be tight, because near-miss documents that look relevant but are not do more damage than empty space.

Does putting a fact in the middle of the context really matter?

Yes, and more than people expect. Models read the beginning and end of a window well and the middle poorly, a pattern often called lost in the middle. Accuracy on retrieving a fact can drop by more than 30 percent when that fact sits in the middle rather than at an edge. If a piece of context is load-bearing, put it at the top or right before your question, not buried in a long block.

What is context engineering and how is it different from prompt engineering?

Prompt engineering is about wording a single instruction well. Context engineering is about deciding, at every step of an agent’s run, exactly what set of tokens the model should see and what it should not. It covers retrieval, summarizing long histories, splitting work across separate calls, and trimming the window as you go. By 2026 it has largely replaced prompt engineering as the job that actually determines whether a system works, because most production failures are context failures, not wording failures.

How often should an agent compact its context?

Earlier than feels necessary. Compaction means replacing a long raw history with a tight summary of what mattered. If you wait until the window is nearly full, the summary itself gets written by an already-degraded model and every step in between ran through a bloated context. Common guidance is to compact on the order of every 5,000 to 20,000 tokens for lighter workloads and 50,000 to 100,000 for heavier ones, as routine maintenance rather than emergency cleanup.

Can adding a relevant document ever make the answer worse?

Yes. Context is competitive, not additive. Everything in the window competes for a fixed pool of attention, so a genuinely useful document can lose out to a near-miss distractor that sits in a position the model reads more strongly. This is why generous retrieval, grabbing many documents to be safe, often lowers quality. Past the facts the model actually needs, extra context is interference, not reinforcement.

I’m a four-time founder building AI-native companies. I write about the founder mindset, AI opportunities, and the mechanics of building with AI at vikasmalpani.com.