AI Capacity Risk: The Ceiling Under Your Product
In April 2026, Meta agreed to pay CoreWeave another 21 billion dollars for compute, bringing its total contract with a single outside vendor to 35 billion. The striking part was not the number. It was the purpose. The money was for inference, the everyday work of serving models to users, not for training the next frontier model. Meta runs some of the largest data centers on earth and is spending its own way into hundreds of billions of capex. And it still went outside to rent serving capacity, because it could not build fast enough on its own.
Sit with that for a second. If the company behind Facebook, Instagram, and WhatsApp has to rent the room its models run in, what does that say about the capacity under your product? You are two or three layers further down the same stack. You call an API. That API runs on a provider. That provider runs on chips that are back-ordered a year. Every layer is renting from the layer above it, and when the room gets crowded, you are the last one in line for a seat.
I have spent the last two years building on top of these APIs, and I have watched founders obsess over the wrong number. We benchmark quality. We fight over price per token. We tune prompts to shave a few cents. Almost nobody asks the question that actually takes a product down in front of real users: when demand spikes, will the call even complete? That question is the capacity risk, and it is the most under-priced risk in AI products today.
Table of Contents
- The Third Number Nobody Budgets For
- The Framework: The Borrowed Stack
- The Priority Ladder: Who Gets Served First
- The Throttle Cliff and the Correlated Spike
- Why This Is Structural, Not a Bad Week
- How Capacity Risk Reshapes the Roadmap
- Wrong vs Absent: Two Failure Modes
- The Capacity Defenses
- Two Founders, One Demand Spike
- The Contrarian Take: Capacity Is a Negotiation, Not a Bug
- The Capacity Map and What to Do Monday
- FAQ
The Third Number Nobody Budgets For
Every AI product lives on three numbers. The first is price: what a request costs you. The second is quality: how good the output is. Both get endless attention because both are easy to measure and easy to argue about. The third number is availability: the odds that a request you send actually comes back with an answer, on time, when your users are all knocking at once. This is the number that decides whether you have a product or a demo.
Availability gets ignored because it looks invisible right up until it isn’t. At 100 users a day, your provider serves you without a hiccup and you conclude the pipes are infinite. At 10,000 users, on the afternoon you get written up somewhere and traffic triples in an hour, you discover the pipes have a diameter. Requests start returning a 429, which means you have hit a rate limit, or a 529, which means the provider itself is overloaded and cannot absorb the volume. Your product does not get slower in a graceful way. It starts handing users errors at the exact moment you finally had their attention.
The receipts are not hypothetical. Independent trackers logged 354 separate outages on OpenAI’s status page from January 2025 onward, roughly 18 a month, including a worldwide outage on July 25, 2026 that took ChatGPT and the API down for close to 50 minutes. Anthropic’s Claude racked up 208 incidents from October 2025, including elevated errors across multiple models for about four hours on the night of August 14, 2026. These are not fringe vendors. They are the two most-used model providers on the planet, and they go dark on a schedule regular enough to set your watch by.
Here is the part founders miss. Those outages are not the main risk. A full outage is loud and rare and everyone forgives it because everyone shares it. The quieter and more common failure is throttling: your traffic specifically getting slowed or rejected while the provider stays technically up. That happens far more often than a headline outage, it happens under load, and it is invisible in the provider’s green status page because the service is running fine. It just is not running fine for you.
The Framework: The Borrowed Stack
To see the capacity risk clearly, you have to see the whole stack your product sits on and notice that you do not own a single layer of it. I call this the borrowed stack. Every layer is rented from the layer above, and the capacity narrows as you go down, until it reaches a chokepoint that is fixed in the short term and that you have no control over.
The law of the borrowed stack is short. You do not own the capacity your product runs on. When that capacity gets scarce, priority goes to whoever paid to skip the line, and by default that is not you. Everything else in this piece follows from that one sentence.
Notice what the stack means for your options. You cannot make the chips arrive faster, because a 52-week lead time on the packaging that bonds memory onto a GPU is a physics-and-factory problem, not a money problem. You cannot make your provider add serving capacity this quarter, because they are stuck behind the same lead time. What you can change is your position in the line and how much you depend on any single line at all. That is the entire game, and most builders are not playing it because they do not know the line exists.
The Priority Ladder: Who Gets Served First
Providers do not ration capacity at random. They sell position in the queue, and the products that pay for a better spot get served while the ones that did not get slowed or dropped. This is the priority ladder, and it is worth understanding rung by rung because your default position is lower than you think.
OpenAI sells this position explicitly. Its Scale Tier lets you buy a set number of input and output tokens per minute upfront, for one model snapshot, with a 30-day minimum, and it promises faster and more consistent speed than pay-as-you-go even during peak demand. Its Reserved Tier sells provisioned throughput priced in dollars per minute that you can spread across standard and priority processing. Its Priority Processing sells a premium per-token rate for tokens that jump the queue. Read those three products together and the message is plain: the front of the line is a paid product, and if you are on standard pay-as-you-go, you are behind everyone who bought it.
Not every provider even offers you a way up the ladder. As of this writing, Anthropic sells rate-limit tiers named Start, Build, Scale, and Custom, but does not offer an equivalent reserved or provisioned-throughput product the way OpenAI does. That matters. On a provider without a reserved option, there is no seat you can buy to guarantee service during a crunch. You are exposed to whatever the shared pool has left. When Anthropic hit its own capacity wall in March 2026, it added peak-hour throttling and paused a major integration, and the fix did not come from customers buying their way out. It came from Anthropic signing a compute partnership that let it double session limits and raise some API limits by up to 1500 percent two months later. Your availability was decided by a supply deal you had no part in.
The Throttle Cliff and the Correlated Spike
The cruelest property of the capacity risk is when it triggers. Your product does not degrade gradually as you grow. It works fine, and works fine, and works fine, and then falls off a cliff at the exact moment a spike arrives. I call this the throttle cliff, and it is nastier than a slow decline because you cannot see it coming in your metrics. Average latency looks healthy right until the afternoon it doesn’t.
Now add the detail that turns a bad afternoon into a pattern. Your demand spikes are correlated with everyone else’s. The moments that send your traffic up, a product launch, a news cycle, a Monday morning, a viral post, a holiday, are the same moments that send the whole market’s traffic up. So the instant you need the most capacity, the shared pool has the least to give, because ten thousand other products are reaching for it at the same time. This is the correlated spike, and it is why capacity fails precisely when it is most expensive to fail. You are not competing for a seat on a quiet Tuesday. You are competing on the one day the theater sold out.
The provider’s defense against the correlated spike is the spillover rule, and you should read the fine print on it. OpenAI’s Scale Tier, as of July 2026, automatically spills overflow traffic into Fast mode rather than failing it, which is a good thing if you paid for Scale Tier and a non-event if you did not. For everyone on standard, spillover means your requests are the ones that get deprioritized so the paying tiers keep flowing. The system is working as designed. It is just not designed around you.
There is a compounding cost here that goes past the failed request. When your product errors during a spike, you lose the highest-intent users you will ever get, the ones who showed up because something made them curious right now. A user who hits a spinner and then an error message during your one moment of attention does not come back later to try again. The capacity failure does not just cost you that request. It costs you the customer that request would have become, and it does it on your best traffic day, not your worst.
Why This Is Structural, Not a Bad Week
It would be comforting to treat all of this as a temporary shortage that clears next quarter. It is not. The capacity risk is baked into the physics of the supply chain, and the numbers say it persists for years, not weeks. If you are building a product you expect to still be running in 2027, you have to plan around a constrained world, not bet on an abundant one.
Start with lead times. Data-center GPUs are running 36 to 52 weeks from order to delivery, because the bottleneck is not the chip itself but the packaging that bonds high-bandwidth memory onto it. TSMC’s CoWoS packaging capacity is booked solid through at least the middle of 2027, and the memory suppliers cannot make HBM fast enough to fill the orders that already exist. You cannot buy your way past a factory that is already running flat out. New serving capacity takes 18 to 24 months to come online from the moment someone decides to build it.
Now the demand side. The five largest hyperscalers committed somewhere between 600 and 630 billion dollars of capital spending in 2026, with roughly three quarters of it aimed straight at AI infrastructure, and one count put the combined figure near 700 billion, about double the prior year. Those forward orders consumed most of Nvidia’s available allocation through the end of 2026 and into 2027. When the giants lock up the supply on multi-year commitments, the smaller clouds and neoclouds that you actually build on get squeezed, and analysts estimate demand outran supply by a factor of 1.4 to 1.6 through 2027. That gap is the capacity risk expressed as a ratio.
The price signal confirms it. During the 2026 crunch, rental contracts for H100 GPUs rose about 40 percent and Blackwell about 48 percent, which is the opposite of what you would expect if the shortage were easing. When the thing your provider needs to serve you keeps getting more expensive and harder to get, the pressure lands somewhere, and where it lands is your rate limits and your spillover priority. This is the structural version of the point I made about the price side in pricing under cheap inference: per-token prices fell for years, but the capacity to actually serve those cheap tokens is the constraint that did not fall.
How Capacity Risk Reshapes the Roadmap
Capacity usually gets filed as an operations concern, something the infrastructure owner deals with after the product is built. That filing is the mistake. Availability shapes what you should build in the first place, and the founders who treat it as a product decision end up with a more defensible product, not merely a more stable one. This is one of the places where the map of AI opportunities I laid out in the AI opportunity map quietly rewrites your feature list, because a feature that cannot survive a spike is not really shipped.
Start with what you promise. If your product tells users they get an instant answer on every request, you have written a promise your provider has to keep for you, and on a spike day it will not keep it. The stronger design decides in advance which parts of the experience must be instant and which can be asynchronous, queued, or served from cache without the user feeling cheated. A feature that tolerates a two-second delay or a slightly stale cached result is a feature that survives a throttle event. A feature that must call a frontier model synchronously, in the foreground, on every keystroke, is a liability dressed up as a differentiator. Deciding that split early is a design choice, not a patch you bolt on after the first outage.
This is also where the capacity risk exposes thin products. A wrapper that does nothing but forward a prompt to one provider has no owned floor at all, which is the deeper version of the problem I described in the wrapper trap: such a product is exposed on price and on capacity at the same time, and it cannot defend either. The move that protects your margins, owning a real piece of the value rather than reselling someone else’s tokens, is the same move that protects your availability. Both come from owning something the provider cannot throttle.
Two more interactions are worth naming. First, chasing raw cost savings can quietly worsen your capacity position: cramming everything onto the single cheapest provider concentrates all your risk in one pool, which is the availability cost hiding inside the pure cost thinking I warned about in the efficiency trap. Second, if your users are increasingly other software rather than humans, the pattern I traced in agents are your customer, then your traffic is machine-paced and burstier than human traffic ever was, and the correlated spike hits harder. Capacity risk is not a corner of your roadmap. It runs through the middle of it.
Wrong vs Absent: Two Failure Modes
Founders lump all model failures into one bucket labeled “the AI broke,” and that mislabeling is why capacity gets ignored. There are two distinct failure modes, and they have different symptoms, different fixes, and different owners. Getting them straight is the difference between fixing the right thing and burning a week on the wrong one.
The first mode is a reliability failure: the model answers, but the answer is wrong, made up, or subtly off. That is the problem I dug into in the reliability paradox and in the production gap. The second mode is an availability failure: the model does not answer at all, because your request was rate-limited, the provider was overloaded, or the service was down. One gives you a bad answer. The other gives you no answer. Wrong versus absent. They feel similar to a frustrated user, but they live in completely different parts of your system.
| Dimension | Reliability failure (Wrong) | Availability failure (Absent) |
|---|---|---|
| What the user sees | A confident answer that is incorrect | A spinner, then an error, then nothing |
| Typical signal | Bad eval score, user correction, hallucination | HTTP 429 (rate limited), 529 (overloaded), timeout |
| When it strikes | On hard or unusual inputs | On demand spikes, when you need it most |
| Who controls it | Mostly you: prompts, evals, guardrails | Mostly the provider: their pool, your tier |
| What fixes it | Better evals, retrieval, verification | Failover, reserved capacity, caching, backoff |
The reason this distinction earns its own section is that the tools you already trust do nothing for the availability side. A better prompt does not help when the request never lands. A sharper eval suite does not help when the provider returns a 529. Retrieval, guardrails, and verification are all reliability tools, and they are worthless against a rate limit. If your entire resilience story is “we tested the outputs,” you have hardened one failure mode and left the other one wide open, which is the one that fails on your biggest day.
The Capacity Defenses
The good news is that the availability side responds to architecture. You cannot widen the chokepoint, but you can stop depending on a single path through it, and you can hold your own reserve for the moments the shared pool runs dry. Here are the five moves that matter, ranked by how much protection they buy for the effort.
| Move | What it costs | What it buys |
|---|---|---|
| Multi-provider failover | A second integration and a shared prompt contract | Two independent pools stacked; the single biggest availability win |
| Reserved / priority capacity | Dollars per minute, paid upfront | A held seat at the front of the line during peaks |
| Response caching | A cache layer and a staleness policy | Fewer calls at the chokepoint, cheaper and faster |
| Queue + backoff + jitter | A small amount of queueing code | Survive short 429 bursts without a thundering herd |
| Owned small model on the critical path | Hosting and tuning a fine-tuned specialist | A floor you control when every external pool is dry |
Failover is the move with the best return, and the math is worth memorizing because it is the argument that gets it prioritized. Two independent providers, each up 99.53 percent of the time, do not average to 99.53 when you can fall over between them. They multiply their downtime, and the combined availability climbs above 99.99 percent. In practical terms that is the difference between roughly 41 hours of expected downtime a year and about 12 minutes. You do not get that from a bigger contract with one vendor. You get it from a second vendor and a switch that reroutes on the first sign of trouble.
Failover has to be built correctly or it makes things worse. When you catch a 429 or a 529, you reroute to the backup provider immediately rather than hammering the one that is already struggling. You retry with exponential backoff and jitter, because synchronized retries from every client at once are a thundering herd that keeps the provider on the floor, and jitter is the one-line fix. You honor the provider’s own Retry-After and rate-limit headers instead of guessing. You cap retries at three to five attempts and then fall through, because a retry loop with no ceiling is not resilience, it is a slow denial-of-service attack you are running against yourself. And you classify errors before retrying, since a 429 usually clears in seconds while a 401 or 403 means your credentials are broken and retrying just wastes time.
The last row, an owned small model on the critical path, is the one founders skip and the one that gives you a floor nobody can take away. You do not need a frontier model for every step. A fine-tuned small model can match frontier quality on narrow tasks like classification and extraction at a fraction of the cost, and because you host it, it is the one path in your product that does not depend on anyone else’s pool. When every external provider is throttling on the same correlated spike, that owned path is the difference between a degraded product and a dead one. This is the same “own the incompressible core” logic I argued in the incompressible core, applied to availability instead of headcount.
One clarification, because it is the most common confusion. Failover is not the same as routing. In multi-model routing I made the case for picking the cheapest capable model per request, and that is a cost-and-quality decision. Failover is an availability decision: the same request, sent somewhere else, because the first place could not serve it. You want both, and they often share plumbing, but do not let a routing layer built for cost convince you that you are covered on capacity. They solve different problems.
Two Founders, One Demand Spike
Picture two founders shipping the same kind of AI product, both on the same primary provider, both on standard pay-as-you-go. On a quiet week they are indistinguishable. Same latency, same bill, same happy users. The difference only shows up on the day it matters.
The first founder wired everything to one provider because it was simplest and it worked. She has a clean codebase, a single API key, no failover, no cache, no reserve. Her product is fast and cheap right up until a launch sends her traffic up 4x in an hour. The provider starts returning 429s on her tier because the whole market spiked at once, her requests queue and time out, and her new users meet an error on their first try. She spends the launch day watching her status dashboard stay green while her own product is unusable, because the provider is up, just not for her. The best traffic day of her quarter converts almost nobody.
The second founder spent two extra days early on. He put a second provider behind a failover switch, cached the responses that repeat, added backoff with jitter, and routed his one truly critical step to a small model he hosts himself. His product is a hair more complex and costs a little more to run on a normal day. When the same 4x spike hits, his primary provider throttles him too, his switch reroutes to the backup, the cache absorbs a third of the load, and his hosted model quietly handles the critical path. His users never learn any of this happened. They just get answers. Same spike, same provider, opposite outcome, and the only difference was that one founder treated availability as the third number and the other never counted it.
The Contrarian Take: Capacity Is a Negotiation, Not a Bug
Here is the reframe that changes how you act. Most founders treat a rate limit as an engineering problem, something to be retried and backed off and optimized around. It is not, at its root. It is a negotiation you have already lost. The provider decided who gets served first, sold the front of the line to whoever paid, and put you where you sit by default. All the backoff code in the world is you being polite in a queue you chose not to skip. Capacity is not primarily a bug to fix in your code. It is a position to buy or a dependency to remove.
Once you see it that way, your options resolve into two honest paths, and both are strategic rather than technical. Path one is to buy your way forward: pay for reserved or priority capacity so you hold a seat that survives the spike. That is a real, purchasable answer, and for a product with predictable, valuable load it is often the right one. Path two is to remove the dependency: architect enough independence, through failover and an owned floor, that no single provider’s line can take you down. Most durable products do some of both. What they never do is pretend that a green status page and a retry loop add up to a capacity strategy.
The honest counterweight is that you can absolutely overpay for capacity insurance you do not need. If your traffic is steady and low, a heavy multi-provider failover setup and a block of reserved capacity is money and complexity spent guarding against a spike that never comes. Resilience has a cost, and buying more of it than your actual demand volatility justifies is its own mistake. The goal is not maximum insurance. It is insurance sized to how spiky your demand really is and how much a failed request on your best day actually costs you. That is a judgment call, and it is the right one to be making, which is more than most founders can say about a risk they have not named.
This is also why I put capacity in the same family as platform risk rather than treating it as a pure infrastructure detail. Platform risk is the danger that your provider changes the deal on you, the price, the terms, the access. Capacity risk is the danger that your provider simply runs out of room. Both are the risk of building your business on a foundation you rent and do not control, and both are answered the same way: by owning the part of your product that cannot be allowed to fail, an idea that runs through the commoditization clock and the whole question of what a founder should still own when the models are rented from someone else.
The Capacity Map and What to Do Monday
Whether you need to act on any of this depends on two things: how spiky your demand is, and how deep your fallbacks go. Plot those two and you get a map that tells you where you stand and where you are trying to move.
The trap quadrant is Fragile: spiky demand, no fallback. That is the founder who is one good day away from an outage that costs them their best users. Exposed is the quiet version of the same weakness, steady demand today with no fallback, safe only until the day your demand stops being steady, which is the day you succeed. Over-insured is a real waste but a survivable one, and Resilient, spiky demand met with real fallback depth, is where you want to be before the spike arrives, not during it.
On Monday morning, run the capacity audit. It is five moves and you can do the first three this week. First, locate your chokepoint: list every external model call on a path a user waits on, and mark which ones have no alternative if the provider says no. Second, classify your failures: pull your logs and separate the wrong answers from the absent ones, the eval misses from the 429s and 529s, so you know which failure mode is actually hurting you. Third, add failover to the single most critical call: a second provider behind a switch, with backoff, jitter, and a retry cap, so your most important path stops depending on one pool. Fourth, decide your position on the priority ladder on purpose: either buy reserved or priority capacity for predictable peak load, or consciously accept standard and cover the gap with architecture, but choose rather than drift. Fifth, set an availability floor: pick the one step in your product that is never allowed to fail, and give it a path you control, whether that is a cache, a hosted small model, or a graceful degraded mode, so that when every external pool is dry you still hand the user something. Do those five and you have moved from hoping the pipes are infinite to knowing exactly how wide they are and what you do when they narrow.
Frequently Asked Questions
What is AI capacity risk?
AI capacity risk is the danger that your product cannot get its requests served because the model provider you depend on has run short of capacity. It shows up as rate limiting, throttling, or overload errors, usually during demand spikes. It is distinct from cost, which is what you pay per request, and from reliability, which is whether the answer is correct. Capacity is about whether the request completes at all.
What is the difference between a 429 and a 529 error?
A 429 means “too many requests,” a rate limit tied to your account or tier, and it usually clears within seconds. A 529 means the provider itself is overloaded and cannot absorb the current volume from anyone. A 429 is often solved by backing off and retrying or failing over to another provider. A 529 signals a broader capacity problem on the provider side and is a strong argument for having a second provider ready.
How do I protect my AI product against provider outages?
The single highest-return move is multi-provider failover: a second model provider behind a switch that reroutes when the first returns a rate limit or overload error. Add exponential backoff with jitter, honor the provider’s Retry-After headers, cap your retries, cache repeatable responses, and give your most critical step a fallback you control. Two independent providers with failover can push combined availability above 99.99 percent.
Is buying reserved capacity worth it?
It depends on how predictable and valuable your peak load is. Reserved or priority capacity buys you a held seat at the front of the line during spikes, which is worth a lot if a failed request on a busy day costs you real customers. If your demand is steady and low, that spend is insurance against a spike that may never come, and architecture-based defenses like failover and caching may cover you more cheaply. Decide based on your actual demand volatility, not fear.
Why is the compute shortage expected to last into 2027?
The bottleneck is manufacturing, not money. Data-center GPUs run 36 to 52 week lead times because the packaging that bonds memory onto the chip, and the high-bandwidth memory itself, are both allocated through at least mid-2027. New serving capacity takes 18 to 24 months to build. Meanwhile the largest cloud companies committed 600 billion dollars or more in 2026 and locked up most of the available supply, leaving demand running ahead of supply by an estimated 1.4 to 1.6 times.
Does multi-model routing solve capacity risk?
Not by itself. Routing picks the cheapest capable model for each request, which is a cost and quality decision. Capacity risk is an availability decision: sending the same request somewhere else because the first provider could not serve it. The two often share infrastructure, but a routing layer built to save money does not automatically fail over when a provider is down. You need failover logic specifically, triggered by the error, not by the cost.
How is capacity risk different from platform risk?
Platform risk is the danger that your provider changes the deal: raises prices, alters terms, or cuts off access. Capacity risk is the danger that your provider simply runs out of room to serve you, even with the deal unchanged. Both come from building on a foundation you rent rather than own, and both are answered by owning the part of your product that cannot be allowed to fail and by not depending on a single provider.
How much availability can I realistically get?
A single provider typically runs somewhere around two to three nines of availability in practice, which means hours of downtime a year. Two independent providers, each around 99.53 percent, combined with automatic failover, can reach above 99.99 percent, cutting expected downtime from roughly 41 hours a year to about 12 minutes. The gain comes from independence, not from a larger contract with one vendor, because a bigger contract does not help when that one vendor is the thing that is down.