AI product margins, not benchmarks, decide who wins

· 17 min read

The price of running a model has collapsed. Epoch AI tracked the cost of reaching GPT-4 level performance on a hard graduate science benchmark and found it fell about 40 times per year over three years, with some tasks dropping anywhere from 9 to 900 times a year (Epoch AI, March 2025). One concrete line from their data: answering questions at the level GPT-4 hit in March 2023 cost 37.50 dollars per million tokens then, and 18 cents on a comparable model by February 2025. If the main input to your business got that much cheaper that fast, you would expect it to be minting money.

It is not. ICONIQ, in its 2026 report on AI companies, put the average gross margin of AI products at 45 percent for 2025 (reported by CloudZero, 2026). A healthy software business runs north of 80 percent. So the input got dozens of times cheaper and the product still earns about half the margin of the ordinary SaaS tool it is meant to replace. That gap is the most important number in AI right now, and almost nobody building is looking at it. The bets I care about sit on my AI opportunity map, but a bet with no margin is a hobby, not a business.

Founders watch the wrong scoreboard. We track benchmark scores, context windows, how a new model ranks on a leaderboard. None of that pays rent. The scoreboard that decides who is still standing in three years is gross margin: how much of each dollar of revenue survives after you pay for the thing that produces the value. I would rather run a product with a boring model and 75 percent margins than a product with the best model on the board and 30 percent margins, because only one of those can fund its own growth.

Why AI margins sit below software

In a normal software business, the cost of serving one more customer is close to nothing. You build the product once, and the hundred-thousandth user costs about the same to serve as the thousandth. That is the whole reason software prints 80 to 90 percent gross margins. The expensive part is fixed, paid once, and spread across everyone who signs up.

AI breaks that. Every answer, every generated draft, every agent run is a fresh bill from a model provider. The cost is variable, it scales with use, and it shows up again on the very next request. Bessemer measured this directly: AI-native companies run at 50 to 60 percent gross margins against 80 to 90 percent for SaaS, and the fastest-scaling cohort, the ones growing most explosively, sit near 25 percent or negative (Bessemer, 2026). The faster they grow, the more inference they buy, the thinner the margin gets. Growth and gross margin point in opposite directions, which is the opposite of how software is supposed to work.

Here is the part that keeps people comfortable when they should not be. Benchmarkit found that the median software gross margin is still above 80 percent and has barely moved in four years (Benchmarkit, 2026). Read quickly, that says AI has not hurt margins at all. Read carefully, it says something sharper: most products calling themselves AI are a thin feature bolted onto a normal software cost base, which is the wrapper trap in a different outfit, so the model bill is a rounding error. The margin pain is concentrated in the companies where the model is the product. A help-desk tool that adds an AI summary button on top of a database still runs at 85 percent. A product whose entire value is an AI that resolves the ticket end to end does not. If the core thing your customer pays for is produced by a model call, you are in the low-margin cohort until you prove otherwise, and you should run your plan as if you are.

Margin per task is the benchmark that matters

The one number I would put on the wall is gross margin per task. Not per token, not per seat, not per month. Pick the unit of value your customer actually buys, one resolved support ticket, one drafted contract, one answered research question, and measure what it costs you to deliver that unit against what you charge for it. That ratio is the real benchmark, and it is the one almost no early team tracks because it is harder to pull than a revenue chart and less fun than a model release.

Per-token thinking is where good founders go wrong. The token price is falling, so the instinct is to assume the cost problem solves itself. It does not, because the number of tokens you burn per task is climbing faster than the price per token is dropping. Cheaper tokens plus a reasoning model that thinks for thirty seconds plus an agent that calls a dozen tools can easily cost more per finished task than the old, dumber, more expensive model that answered in one shot. The unit price fell and the unit cost rose at the same time, and only one of those shows up in a headline.

The picture looks like this: the price of a token falls year after year, the tokens you spend to finish one task climb, and the cost of delivering that task, the thing that actually sets your margin, stays stubborn or drifts up.

Time →Price per token (falling)Tokens per task (rising)Your cost to deliver one task

So the question is never whether tokens are getting cheaper. They are. The question is whether the value you deliver per task is rising faster than the tokens you burn to deliver it. If it is, you have a real business that gets stronger as the model layer improves. If it is not, you are selling dollars for ninety cents and calling the growth traction.

Cheaper tokens make people use more, not spend less

When Satya Nadella reacted to DeepSeek in January 2025, he reached for Jevons paradox: as AI gets more efficient and accessible, he wrote, its use will skyrocket (Satya Nadella, January 2025). DeepSeek had just claimed it trained its R1 reasoning model for under 6 million dollars on older, export-restricted chips, and the app shot to the top of the free charts. The lesson most people took was that intelligence had gotten cheap. The lesson that matters for your margin is what happens next: when a resource gets cheaper, people do not pocket the savings, they consume more of it.

Jevons paradox is a nineteenth-century observation about coal. More efficient steam engines did not reduce coal use, they made coal so useful that total consumption went up. The same thing is happening with tokens. Every drop in price pulls in more usage, more ambitious features, longer context, more autonomous behavior. Your users do more because they can, and your own product does more on their behalf because you can finally afford to. The bill holds steady or grows while the per-token price falls.

This is why I never budget on today’s token price. I budget on what my users and my own system will do once the price stops being a brake. The honest planning assumption is that any efficiency you gain gets spent, not saved, unless you deliberately cap it. Efficiency is not a margin strategy on its own. It is permission to do more, which is a growth strategy wearing a cost-saving costume. If you are building a company where AI does the work, as I argued in my AI-native founder playbook, you have to assume the work expands to fill the cheaper compute.

What agents do to your token bill

The jump from a chatbot to an agent is where margins quietly break. Crusoe, which runs AI infrastructure, put numbers on it: an agentic workflow can burn 10 to 100 times more tokens per task than a single chat exchange (Crusoe, 2026). In one support example they walk through, the final answer the customer sees is about 800 tokens, but the workflow behind it processes more than 50,000 tokens and calls more than a dozen tools to get there. The user reads one tidy reply. You pay for sixty times that under the hood.

Multiply that across retries, reasoning steps, tool calls that fail and rerun, and context that gets re-sent on every hop, and the per-task cost of an agent bears no resemblance to the per-task cost of a prompt. A simple classifier that labels an email might cost a fraction of a cent. The same job wrapped in an agent that reads the thread, checks three systems, drafts a reply, and critiques its own draft can cost a hundred times more for an answer the user cannot tell apart. This is the single biggest reason AI-native margins land where they do. The product got more capable and more expensive in the same move, and the number on the invoice to the customer did not go up to match. Before you ship something that agents handle end to end, you have to know what one completed task costs you.

It helps to see where different kinds of products actually sit.

Product type Typical gross margin Main cost driver
Traditional SaaS 80 to 90 percent Fixed build, near-zero marginal cost (Bessemer, 2026)
AI-augmented SaaS Around 80 percent Model used sparingly, as one feature (CloudZero, 2026)
AI-enabled SaaS 60 to 79 percent Model central to some workflows (CloudZero, 2026)
AI-native product 50 to 60 percent The model call is the product (Bessemer and ICONIQ, 2026)
Fastest-scaling AI cohort About 25 percent or negative Heavy inference, growth chosen over margin (Bessemer, 2025)

The table is not destiny, but it is a map. The further down you are, the more of your attention belongs on cost per task rather than on the next feature. A vertical product can sit higher than its type suggests if it owns the data and workflow around the model, which is the whole case I made for the vertical AI playbook. The model is rarely the moat. What you build around it is.

How I would price and build around variable cost

If your cost scales with usage, your price has to as well, or you are writing your heaviest users a blank check. Flat per-seat pricing on a product with variable inference cost is the classic way to go broke while the dashboard says revenue is up. The accounts that love you most, the ones using the product hardest, are the ones losing you the most money. I would tie price to the unit of value, charge per resolved ticket or per run or per thousand tasks, so revenue and cost move in the same direction. There is more on this in my notes on revenue models for AI products, but the short version is that a meter beats a seat the moment serving costs real money.

Then I would put a ceiling on cost per account. Rate limits, token budgets, and hard caps are not hostile to customers, they are what keeps the product alive long enough to serve them. On the build side, the levers that actually move margin are unglamorous: route the easy 80 percent of tasks to a cheaper model and reserve the expensive one for the cases that need it, cache anything you compute more than once, trim the context you re-send on every call, and cut the retries and tool calls that do not change the answer. None of that shows up in a demo. All of it shows up in the margin.

Most of all, I would measure cost per task in production from the first week, not the first audit. You cannot manage a number you do not see, and by the time a variable cost is large enough to notice in the monthly bill, it is already shaping your business. The teams that route models well also tend to measure quality well, which is why I keep coming back to the idea that evals are the product: you can only send a task to a cheaper model with confidence if you can measure whether the cheaper model still passed.

Maybe margins fix themselves as compute gets cheaper

There is a real case that I am worrying about a problem time will solve. Token prices keep falling. SemiAnalysis reported that Anthropic’s gross margin on its inference infrastructure rose from 38 percent to over 70 percent as it tuned how it serves models (SemiAnalysis, May 2026). ICONIQ projects AI product margins climbing from 45 percent in 2025 to 53 percent in 2026 and 59 percent in 2027 (reported by CloudZero, 2026). The median software margin has not budged. Maybe the low numbers are just the early, inefficient phase, and in two years this reads like worrying about server costs in 2004.

I half believe it. Margins will improve. But look at where the Anthropic gain came from: engineering, scale, and hard work on how models get served, not from sitting still and waiting for prices to drop. Those gains go to the teams that chase them, not to the ones that assume the curve is on their side. And the two forces I described pull the other way at the same time. Usage expands to eat the efficiency, and agents multiply the tokens per task. The ICONIQ projection still lands at 59 percent in 2027, which is better, and still well short of the 80 percent a durable software business takes for granted. Improving is not the same as solved. I would rather hit those numbers on purpose than hope the trend carries me there.

Instrument cost per task before your next raise

Here is what I would do this quarter if I were building an AI product. Tag every model call to a unit of customer value, so you can answer one question: what did it cost to deliver one of the things we sell. Compute gross margin per task every week and put it next to revenue in the same report, because a revenue chart that climbs while margin per task falls is a slow-motion problem that looks like success right up until it does not. Set a target margin for each task type and treat anything chronically below it as either a pricing bug or a product that should not exist at that price.

Then act on what you see. Reprice the tasks that lose money, route the cheap ones to cheap models, and be willing to kill a feature that is beloved and unprofitable. An investor in 2026 who has read a few of these reports will ask about your cost per task and your gross margin trend. Walking in with those numbers measured, and a credible plan to move them, is worth more than one more point on a benchmark. The teams that win the next few years will be the ones that treated margin as a feature and built it in from the start, not the ones that assumed cheaper tokens would quietly hand it to them.

Questions founders ask me about AI margins

How do I calculate gross margin for an AI product

Take your revenue over a period, then subtract the direct cost of delivering the service: model and inference bills, vector database and retrieval costs, and the compute that runs your pipelines. Divide what is left by revenue. The mistake I see most is burying inference in a general cloud line so nobody can see it. Pull it out and tie it to a unit of value, one task or one run, so you know the margin on the thing you actually sell, not just the blended number for the whole account. That per-task view is where the real decisions live, and it is usually worse than the headline figure.

Should I price per seat or per usage for AI features

If the feature has a real variable cost, per usage protects you and per seat exposes you. Flat seats work when serving one more action costs nothing, which was true for classic software and is not true when every action is a model call. I would price on the unit of value, a resolved ticket, a generated report, a batch of runs, so your revenue rises with your cost instead of lagging behind it. Seat pricing can still work as a floor for predictability, but put a usage meter underneath it so your heaviest users do not quietly become your biggest losses.

Will falling token prices fix my margins over time

Partly, and not on their own. Token prices are falling fast, and serving costs do come down as you tune your stack. But two things push the other way. Cheaper tokens pull in more usage, so people spend the savings rather than bank them, and agentic features burn far more tokens per task than simple prompts. Projections have AI margins improving over the next few years, yet still landing below where software businesses sit. I plan as if margin is something I have to build and defend, not something the price curve will deliver to me while I wait and hope.

How much do agents really add to my token bill

A lot more than the output you see suggests. One infrastructure provider measured agentic workflows using 10 to 100 times more tokens per task than a single chat exchange, with a support flow processing more than 50,000 tokens to produce an answer the user reads as a few hundred. Reasoning steps, tool calls, failed retries, and context re-sent on every hop all add up fast. Before you ship an agent, model its token cost per completed task at the volume you expect, not the volume in your demo, because the gap between those two numbers is exactly where margins disappear.

What gross margin should an AI startup aim for

Early on I care more about the trend than the absolute number. AI-native companies commonly run in the 50 to 60 percent range today, below the 80 percent or more that mature software enjoys, so starting there is normal and not a failure. What I want to see is margin per task improving quarter over quarter as you tune models, caching, and routing. A product stuck at low margin with no upward path is a warning. One at 55 percent and climbing, with a clear plan to reach the high sixties or seventies, is a business. Pick a target per task type and chase it on purpose rather than hoping.

When does it make sense to run a cheaper model

More often than most teams assume. A large share of real tasks are easy and do not need your most capable model, and sending them to a smaller one can cut cost sharply with no drop a user would notice. My rule is to route by difficulty: default to the cheapest model that clears your quality bar for that task, and escalate to the expensive one only when the task genuinely needs it. The work is in measuring quality per task type so you can route with confidence instead of guessing. Done well, model routing is one of the few levers that improves margin without touching your price.

I’m a four-time founder building AI-native companies. I write about the founder mindset, AI opportunities, and the mechanics of building with AI at vikasmalpani.com.