Eval-Washing: How to Audit AI Reliability Claims
AI vendors and labs sell reliability scores they cannot back. Eval-washing is how a benchmark hides the truth, and how a founder can audit any AI claim.
Essays & insights on ai & technology.
AI vendors and labs sell reliability scores they cannot back. Eval-washing is how a benchmark hides the truth, and how a founder can audit any AI claim.
AI made output nearly free, so volume metrics lie. Cost per correct task is the only AI productivity metric that survives. Here is how to measure and use it.
Founders treat evals as overhead until the reliability tax comes due. How to size an AI eval budget that turns invisible failures into a bounded line item.
AI leaderboards are gamed, contaminated, and quietly retired. The Benchmark Contamination Playbook: how to pick the right AI model without getting fooled.
Your AI agent is an employee with root access and no manager. The Five Gates framework shows builders how to close the agent identity gap before it leaks.
Most AI products ship on vibes and break in production. The Evals Playbook is the 5-layer pyramid that lets solo founders measure what they ship.
Token prices keep falling. Your AI bill keeps climbing. The Context Engineering Playbook is the discipline that decides which AI products survive the math.
Most AI agents work in a demo and break in production. The Production Gap framework shows builders the four reasons agents fail and how to ship reliable ones.
Most AI data moats are a story founders tell themselves. The Data Moat Ladder shows the 5 rungs of real defensibility and how to build one that actually holds.
Prompt engineering is product architecture, not wordsmithing. The 5-layer stack that keeps AI products from breaking when the model or the inputs change.
Building something interesting? Looking for an investor who’s been in the trenches? Or just want to say hello? I’d love to hear from you.
One email when I publish. Essays on reskilling, rewiring your mindset, and building your next thing. Free, no spam, unsubscribe anytime.