The Founder’s Calibration Practice

How to know how much to trust your own judgment, in a year when your tools sound certain about everything.

A team at MIT spent this spring teaching language models to say “I am not sure.” Their finding had a name that should stop every founder cold: the hallucination paradox. The smarter and more persuasive a model gets, the more dangerously confident it becomes in its own errors. The MIT method that fixes it, reinforcement learning with calibration rewards, cut a model’s calibration error by up to 90 percent without making it any less accurate. The machine got no smarter. It just got honest about how sure it should be.

Here is the durable version of that story, the one that outlives this month’s research paper. Your AI does not have a knowledge problem. It has a confidence problem. And so do you.

For most of business history, the founder’s edge was having answers other people did not have. You knew the customer, you knew the market, you had the insight. That edge is gone, or close to it. The model on your laptop can produce a plausible answer to almost any question in four seconds. What it cannot do, and what almost no founder does well either, is tell you how much to trust that answer.

That skill has a name. It is called calibration, and it is the most underrated discipline in entrepreneurship. Calibration is not confidence. It is the match between how sure you feel and how often you turn out to be right. A perfectly calibrated founder who says “I am 70 percent sure this will work” is right about 70 percent of the time. Most founders who say 70 percent are right about 50 percent of the time, and they have no idea, because nobody keeps score.

I have run two companies where AI now does most of the first-draft thinking. The work that used to fill my day, the analysis, the drafting, the modeling, gets done before I have finished my coffee. What is left is the part the machine cannot do for me: deciding how much to believe it. I got that wrong for the better part of a year. This post is the system I built so I would stop getting it wrong, backed by the research that explains why it works.

What this post covers

• The problem: more decisions, faster, on borrowed confidence
• The framework: the calibration curve and the calibration gap
• Why founders are built to be overconfident
• Confidence contagion: how AI makes it worse
• The calibration loop: the actual practice
• Calibration is trainable, and the evidence is strong
• The reversibility filter: how much calibration each call deserves
• The contrarian take: the fix is not trusting AI less
• What to do Monday morning
• FAQ

The problem: more decisions, faster, on borrowed confidence

Two things changed at once when AI entered the founder’s workflow, and they pull in opposite directions.

The first is volume. You make far more decisions per day than you did two years ago, because the cost of producing a decision-shaped artifact collapsed. A pricing model, a hiring rubric, a market sizing, a go-to-market plan, each used to take hours of human work that forced a natural pause. Now they arrive in seconds, and you approve or reject them in seconds. A founder I work with counted her decisions for a week. She was making roughly four times as many judgment calls as she had in 2023, and spending about a third as long on each one.

The second is certainty. The artifacts arrive sounding finished. The model does not hedge unless you make it. It writes the pricing model in the confident register of a McKinsey deck, complete with assumptions stated as facts. And here is the part the research nails down: that confidence is contagious. When you read a confident answer, your own confidence rises to meet it, whether or not the answer is any good.

Put those together and you get the trap. You are making more decisions, faster, on confidence that was manufactured by a tool that does not actually know if it is right. The decisions feel more certain than they have ever felt. They are not more accurate. The gap between how sure you feel and how often you are right, which was always there, just got wider and quieter.

The stakes are not abstract. Around 90 percent of startups fail. Roughly 10 percent die in the first year and about 70 percent in years two through five, right in the scaling phase where the big bets get made. Among venture-backed companies, 60 to 75 percent never return the capital they raised. Most of those deaths are not caused by a single catastrophic mistake. They are caused by a hundred confidently-made decisions that were each a little more wrong than the founder believed, with no mechanism anywhere in the company to catch the drift.

Calibration is that mechanism. It is the discipline of knowing, decision by decision, how much weight your own judgment can actually bear.

The framework: the calibration curve and the calibration gap

The cleanest way to see calibration is a picture that decision researchers have used for decades, called a reliability diagram. I have redrawn it for founders below. It is the single most useful chart I know for thinking about your own judgment.

Read it like this. The horizontal axis is how sure you felt when you made a call, from a coin-flip 50 percent up to dead-certain 100 percent. The vertical axis is how often you actually turned out to be right. A perfectly calibrated founder sits on the diagonal: when they feel 80 percent sure, they are right 80 percent of the time. The distance between your real curve and that diagonal is your calibration gap. For almost everyone, untrained, the curve sags below the line. You feel 90 percent sure and you are right 70 percent of the time.

Three things matter in that picture, and they map to the rest of this post.

The red curve is you, untrained. It is not a character flaw. It is the human default, and founders sit further below the line than almost any other group, for reasons I will get to. The orange curve is you after a year of working alongside confident AI. It sags further, because you have been quietly absorbing the machine’s certainty on top of your own. And the green diagonal is reachable. That is the surprising, hopeful part: calibration is one of the most trainable skills in all of decision science. People move from the red curve toward the green one in a single afternoon of the right practice.

Notice what the chart does not say. It does not say feel less confident. A founder who answered every question with “I have no idea” would sit way above the diagonal, perfectly useless. The goal is not less confidence or more confidence. It is accurate confidence. Your felt certainty should track your real hit rate, up and down, call by call.

To keep the rest of this concrete, hold one distinction in your head. Confidence is a feeling. Calibration is a track record. Here is how they come apart in practice.

Almost every bad founder decision I have watched, including my own, traces back to confusing the left column for the right. The feeling of certainty got treated as evidence of accuracy. They are not the same thing, and the rest of this post is about closing the distance between them.

Why founders are built to be overconfident

If calibration is so useful, why is almost no founder good at it by default? Because the traits that get you to start a company are the same traits that wreck your calibration. The selection pressure runs the wrong way.

Start with the raw bias. Overconfidence is the best-documented cognitive bias in entrepreneurship, and the numbers are not subtle. In one study of founders, people overestimated the value of their intellectual property before product-market fit by 255 percent. A study of Austrian entrepreneurs found that founders express overprecision, a specific flavor of overconfidence where your error bars are far too narrow, and that being a solo founder rather than a co-founder predicted higher overconfidence. When you have no co-founder to say “are you sure,” your guesses get tighter and worse at the same time.

It shows up in timing too. Research from the Startup Genome project found that startups need two to three times longer to validate their market than founders expect. Not 20 percent longer. Two to three times. That is not bad luck. That is a calibration gap baked into the founder personality, applied to the single variable that kills the most companies: runway.

Now add the survivorship story we tell ourselves. Every founder has heard the legend of the one who ignored all the doubters and won. Nobody tells the story of the ninety who ignored all the doubters and lost, because they are not around to tell it. So the culture trains you to read your own stubbornness as vision. The feeling of unshakeable conviction gets reframed as a strength, when half the time it is just a miscalibrated curve sagging hard below the diagonal.

Here is the trap closing. Overconfidence tells the founder they are an early genius. Confirmation bias then sends them out to collect only the evidence that supports the genius story. Sunk cost keeps them in once the bills pile up. Each bias hands off to the next. Calibration is the one practice that interrupts the chain, because it forces you to write down a number before reality has a vote, and then it makes reality vote.

None of this means confidence is bad. You cannot raise money, recruit, or sell while broadcasting doubt. The point is narrower and more useful: the same engine that lets you act under uncertainty also lies to you about how uncertain you actually are. You do not want to turn the engine off. You want a gauge on it.

Confidence contagion: how AI makes it worse

For all of business history, your overconfidence had a natural ceiling. You could only be as sure as your own gut would let you feel. AI removed the ceiling, and the mechanism by which it did is now well documented.

A line of research presented at the 2025 CHI conference, the main venue for human-computer interaction, studied what happens to a person’s own confidence when they work next to a confident AI. The finding was clean and a little disturbing. Human self-confidence aligns with AI confidence. When the model is sure, you become sure. When the model hedges, you hedge. And the alignment is one-directional. Between two people, confidence flows both ways and tends to average out. Between a person and an AI, it only flows from the machine to you. You imitate the model. The model does not imitate you.

That would be fine if the model were well calibrated. It is not. The same MIT work I opened with describes the hallucination paradox: as models get more capable and more persuasive, they get more confidently wrong, not less. A survey of agentic retrieval systems published in May 2026 named “overconfident gap-filling” as one of the four core failure modes, where the system, missing a fact, invents one and presents it in the same assured tone as everything it actually knows. You cannot hear the difference. That is the whole problem. The confident-but-wrong answer and the confident-and-right answer sound identical.

So the chain runs: the model is confidently wrong, your confidence aligns to the model, and now you are confidently wrong too, with the added conviction of having “checked with the AI.” The research on human-AI teams calls the result a reliance failure. People over-rely on overconfident systems and under-rely on under-confident ones, and in both cases the team performs worse than the human would have alone. A 2026 study with 408 participants tracked exactly this erosion of judgment as hallucination levels rose.

The picture below is the mechanism, drawn out.

The thing to take from this is that AI did not create your calibration problem. It found the gap that was already there and poured certainty into it. A founder with a tight, accurate sense of their own hit rate is far harder to push around with a confident wrong answer, because they are running their own gauge instead of borrowing the machine’s. Calibration is the immune system for confidence contagion.

The calibration loop: the actual practice

Everything up to here is diagnosis. This is the treatment, and it is almost embarrassingly simple. Calibration is built by one loop, run over and over, on the decisions you were going to make anyway.

Walk through it once.

Predict. Before you make a call, write down a probability. Not a vibe, a number. “I am 70 percent sure this hire works out.” “I put 40 percent on this feature moving the activation metric.” The number feels silly the first ten times. It is the whole point. A number can be scored. A feeling cannot.

Decide and act. Make the call you were going to make. Calibration does not slow you down or make you timid. You still ship. You just shipped with a number attached.

Record. One line in a running log: the decision, your probability, the date you will actually know the answer, and one note on what would have to be true for you to be wrong. That last note matters more than it looks, because it is the thing your overconfidence skips.

Score. When the outcome lands, score it. The standard tool is the Brier score, and it is simpler than it sounds. You take your probability, subtract the outcome (1 if it happened, 0 if it did not), and square the difference. Predict 70 percent and it happens: your error is (0.7 minus 1) squared, which is 0.09. Predict 90 percent and it does not happen: (0.9 minus 0) squared, which is 0.81, a brutal score for being confidently wrong. The squaring is deliberate. It punishes confident misses far harder than honest hedges, which is exactly the lesson an overconfident founder needs.

Adjust. After twenty or thirty scored predictions, a pattern shows up. Maybe everything you call 90 percent actually lands around 70. Now you have a correction factor for your own brain. When you next feel 90 percent sure, you mentally translate it to 70 and act accordingly. That correction is calibration. You did not get smarter. You got an accurate gauge.

The whole apparatus fits in a spreadsheet. Here is the one I keep.

Five columns. Two minutes per decision. That is the entire practice. The eng-hire row, scoring 0.64, is the kind of line that changes how you operate: you were 80 percent sure and dead wrong, and now you have proof your hiring confidence runs hot. That single data point is worth more than any productivity tip you will read this year.

Calibration is trainable, and the evidence is strong

The reason I am confident this works is that calibration is one of the few judgment skills with a real, replicated training literature behind it. This is not a motivational reframe. It is a measurable skill that responds fast to practice.

Start with the most striking result. Doug Hubbard’s firm has trained well over 1,000 people in calibration using a half-day workshop of repeated estimate-and-score exercises. By the fifth exercise, around 80 percent of participants are ideally calibrated. Half a day. Before training, the same people show the classic pattern: statements they make with 90 percent confidence are true only about 70 percent of the time, and their 75 percent statements come true about 60 percent of the time. After a handful of scored rounds, the curve climbs to the diagonal.

This holds up in serious settings. A 2024 study tested commercial calibration training on 70 intelligence analysts, people whose entire job is judging uncertain situations. Before training they were overconfident on interval estimates. After the course they were measurably better calibrated. If trained analysts start overconfident, you can be sure you do.

Then there is the deepest body of evidence, Philip Tetlock’s Good Judgment Project, the multi-year forecasting tournament that produced the idea of the superforecaster. The headline numbers tell the story. Superforecasters average a Brier score around 0.166, against roughly 0.259 for ordinary forecasters, where lower is better. They were more accurate from the start and they improved faster as new information arrived. And here is the part founders should tattoo somewhere: what made them super was not raw intelligence or domain expertise. It was method. They broke big questions into smaller ones, they reasoned in probabilities instead of yes or no, they updated in small steps as evidence came in, and they kept score. Training and tournaments produced 20 to 40 percent improvements over standard forecasting.

Every one of those habits is available to you for the price of a spreadsheet and the discipline to fill it in. You do not need a tournament. You need to write numbers down before you know the answer, and then go back and check. The machine that beat the experts was not a genius. It was a person who kept an honest scorecard.

The reversibility filter: how much calibration each call deserves

An obvious objection: you make dozens of decisions a day now. You cannot run a five-column journal on what font the landing page uses. Correct. Most decisions do not deserve the ceremony, and treating them all the same is its own failure. The filter that tells you which calls earn the full practice is reversibility.

The cheapest version of this idea comes from Jeff Bezos, who split decisions into two kinds. One-way doors are decisions that are expensive or impossible to undo: betting the company on a market, a senior hire, a pricing architecture, a brand name, an acquisition. Two-way doors are reversible: a feature flag, a landing page test, a vendor you can swap, a campaign you can kill. The mistake almost every founder makes is running both at the same speed. They agonize over reversible calls and rush the irreversible ones, usually because the irreversible ones arrive wrapped in the most exciting story.

Calibration discipline should scale with irreversibility. Here is the filter I run.

The last column is where the AI era and the calibration practice meet. For a reversible call, borrowing the model’s confidence is fine, because reality corrects you cheaply and fast. For a one-way door, the confident AI answer is the most dangerous input in the room, precisely because it is the most persuasive. That is where you slow down, force the model to make the opposite case, bring in an outside human who has no stake in your story, and write your own number down before any of them speak. I keep a small personal board of advisors for exactly the one-way-door calls, people whose job is to be the un-borrowed second opinion my own confidence cannot supply.

The reversibility filter is what keeps calibration from becoming overhead. You run the full loop on the handful of decisions that can actually end the company, and you let the cheap, reversible ones fly. Speed where it is safe, calibration where it is not.

The contrarian take: the fix is not trusting AI less

The standard advice for the AI-overconfidence problem is to trust the AI less. Verify everything. Add a human in the loop. Be skeptical. It sounds responsible, and it is mostly wrong, or at least aimed at the wrong target.

The thing you actually cannot calibrate is the machine. You do not control its training, you cannot see its real confidence, and the next model update can change its behavior overnight. Pouring your energy into second-guessing the AI is a treadmill. The thing you can calibrate, the only thing, is yourself. The whole research record points the same way: calibrating human self-confidence improves human-AI team performance and produces more rational reliance, while trying to fix the human’s trust in the AI by tweaking the AI’s confidence displays barely moves the needle. The power to fix this sits on your side of the keyboard, not the model’s.

So the named enemy here is not the AI. It is the founder’s belief that conviction is a virtue. We have built an entire mythology around the founder who just knew, who ignored the data and the doubters and was vindicated. That story is survivorship bias wearing a cape. For every founder who just knew and was right, there is a graveyard of founders who just knew and were wrong, and the only difference visible in advance was a calibration gap nobody was measuring. Conviction is not the asset. Accurate conviction is. The founder who can say “I am 60 percent on this, here is what would change my mind” will, over a hundred decisions, crush the founder who is sure about everything, because the first one is compounding an honest track record and the second one is compounding errors with a clean conscience.

Now the honest counterpoint, because calibration has a real failure mode. There are genuine cases where the calibrated, base-rate answer is “this almost never works,” and the founder who listens to it never starts anything. Calibration applied to whether to be a founder at all would talk most people out of it, and some of those people would have built something great. The resolution is not to throw out calibration. It is to aim it. Be radically calibrated about the decisions inside the business, the pricing and the hiring and the timing, where being wrong is just expensive. Reserve your uncalibrated, irrational conviction for the single bet that the whole thing is worth doing. One article of faith, surrounded by a hundred honest probabilities. That is the actual founder skill, and almost everyone has the ratio backwards: they are uncertain about the small reversible things and blindly certain about the big irreversible ones.

What to do Monday morning

Skip the philosophy. Here is the install, and it takes less than an hour to start.

Open one spreadsheet. Five columns: decision, your probability, review date, what would make you wrong, outcome plus Brier. That is the whole tool. Do not buy software for this. The friction of a fancy tool kills the habit.

Put a number on your next five decisions. Not the trivial ones. The next five that have a real outcome you will know within a month or a quarter. Write the probability before you act. It will feel arbitrary. Do it anyway, because an arbitrary number you can score beats a confident feeling you cannot.

Take a calibration test this week. There are free ones online: forty trivia questions where you give an answer and a confidence level, and it shows you your curve. Most founders are shocked the first time. You will probably find that your 90 percents come true about 70 percent of the time, exactly the gap in the research. Now you have your personal correction factor, and you have it in an afternoon.

Run the reversibility filter on your current big call. Whatever the biggest decision on your desk is right now, ask: one-way door or two-way door? If it is reversible, stop agonizing and ship it. If it is irreversible, slow down, write your number, force the AI to argue the other side, and call one outside person before you commit.

Score yourself at 30, 60, and 90 days. The loop only works if you go back. Put a recurring block on the calendar to open the sheet, mark the outcomes that have landed, and compute your average Brier. Watch it drop. That falling number is your judgment getting measurably better, which is a thing almost no founder can say with a straight face because almost none of them keep score.

That is it. One sheet, five numbers a week, a quarterly review. The founders who do this will be making decisions on a gauge while everyone else is making them on a feeling that the machine keeps inflating.

Frequently asked questions

What is the difference between confidence and calibration?
Confidence is how sure you feel. Calibration is how often your “sure” turns out to be true. A calibrated founder who says they are 70 percent confident is right about 70 percent of the time. Most untrained people who say 70 percent are right closer to 50 percent and do not know it, because they never check. Calibration is the match between the feeling and the track record, and only the track record is real.

How do I measure my own calibration?
Write a probability on your decisions before you make them, log the date you will know the outcome, and score each one with the Brier method once reality lands: take your probability, subtract 1 if it happened or 0 if it did not, and square the result. After twenty or thirty scored predictions you will see your pattern, usually that your high-confidence calls come true less often than you felt. That gap is your correction factor.

What is a Brier score and what is a good one?
The Brier score measures the accuracy of a probability forecast as (forecast minus outcome) squared, where the outcome is 1 or 0. Lower is better, and 0 is perfect. As a reference point, superforecasters in Philip Tetlock’s research average around 0.166, while ordinary forecasters land near 0.259. You do not need to hit those numbers. You need to watch your own average fall over time, which is the proof the practice is working.

Can calibration actually be trained, or are some people just born with good judgment?
It is strongly trainable, and fast. Doug Hubbard’s firm has trained over 1,000 people with a half-day workshop after which around 80 percent are ideally calibrated. A 2024 study improved the calibration of 70 intelligence analysts through a commercial training course. Tetlock’s superforecasters got there through method, not raw IQ. Good judgment is a skill with a real training literature, not a personality trait.

Why does AI make founder overconfidence worse instead of better?
Research presented at the 2025 CHI conference found that human self-confidence aligns to AI confidence, and the alignment is one-directional: you imitate the model, it does not imitate you. Because models tend to be confidently wrong, a problem MIT calls the hallucination paradox, you absorb certainty that is not backed by accuracy. The confident-and-wrong answer sounds identical to the confident-and-right one, so you cannot tell them apart by ear. The result is a reliance failure where the human-AI team performs worse than the human alone.

Does keeping a calibration journal slow me down?
Barely, if you use the reversibility filter. Cheap reversible decisions get no ceremony at all, you just ship and measure. Only the one-way-door decisions, the pricing models, senior hires, and pivots that can end the company, get the full journal and outside check. Writing a probability takes about two minutes, and you only do it on the calls that matter. Calibration adds speed where it is safe by stopping you from over-deliberating reversible decisions.

Should I just stop trusting AI for important decisions?
No, and that is the common mistake. You cannot calibrate the machine, you do not control it, and the next update can change it. You can only calibrate yourself. For reversible calls, take the AI’s answer at face value because reality corrects you cheaply. For irreversible calls, treat the AI as one biased witness rather than the judge: make it argue both sides, write your own number down first, and bring in an outside human. The fix is a better gauge on your own confidence, not blanket skepticism of the tool.

Where should I start if I only do one thing?
Take a free online calibration test this week, the kind with forty questions where you give an answer plus a confidence level and it plots your curve. It takes twenty minutes and it will show you your real calibration gap immediately. Seeing your own 90 percents come true 70 percent of the time is the moment the whole practice stops being abstract and starts being personal.

---

This is part of an ongoing series on the founder operating system and how to think clearly while building with AI. If it was useful, the related reading below goes deeper on the judgment side of the same system.

Related reading: How founders should think about AI · The founder operating system · Building a personal board of advisors · Why AI agents fail in production · The evals playbook for solo founders · AI agent security: the identity gap · The solo founder AI operating system