Eval-Washing: How to Audit AI Reliability Claims
AI vendors and labs sell reliability scores they cannot back. Eval-washing is how a benchmark hides the truth, and how a founder can audit any AI claim.
Field notes from four reinventions — a $200M exit, two unicorns, and building AI-native. Published as I learn.
AI vendors and labs sell reliability scores they cannot back. Eval-washing is how a benchmark hides the truth, and how a founder can audit any AI claim.
Most founders have one speed. The ones who build durable companies have two. A decision framework for when to move fast and when to slow down — plus the AI trap nobody warned you about.
Generative engine optimization is the new distribution channel. How founders get cited by AI search, win answer-engine traffic, and prepare for agent buyers.
AI made output nearly free, so volume metrics lie. Cost per correct task is the only AI productivity metric that survives. Here is how to measure and use it.
Everyone is becoming an agent boss, handing work to AI. The founders who win guard the one thing they cannot delegate: judgment. Here is the operating system.
Why startups fail: post-mortem data shows founders kill far more startups than competitors do. Get the Failure Ledger framework and a Monday execution audit.
Founders treat evals as overhead until the reliability tax comes due. How to size an AI eval budget that turns invisible failures into a bounded line item.
Founders read other people's postmortems too late. The pre-mortem practice flips it: imagine the failure first, while it is still cheap to prevent. Here is how.
Your AI model gets deprecated, repriced, and verticalized under you. How to build an AI business that survives model churn and own a moat the model cannot.
AI leaderboards are gamed, contaminated, and quietly retired. The Benchmark Contamination Playbook: how to pick the right AI model without getting fooled.
Building something interesting? Looking for an investor who’s been in the trenches? Or just want to say hello? I’d love to hear from you.
One email when I publish. Essays on reskilling, rewiring your mindset, and building your next thing. Free, no spam, unsubscribe anytime.