The Right Experiment for the Right Claim The Right Experiment for the Right Claim Most ROI cases for enterprise AI are pre/post analyses. A pre/post analysis can prove a metric moved. It cannot prove the model moved it. Get it wrong, and you're signing a number you can't defend. The previous
Spec-Driven Development for AI: What Works, What Doesn't, and What to Do About It Here is the bet most teams are making: write a system prompt that describes what the agent should do, run it against a few test cases, watch it work, and ship. The prompt is the spec. The spec is the documentation. The documentation is the test. One artifact does all
Making Measurement Operational Making Measurement Operational Every technique in this series (holdbacks, stepped-wedge designs, ablation studies, causal lineage maps, Shapley decomposition) requires one thing you cannot retrofit: structure built before deployment. That's not a one-time project. It's a capability. Most organizations treat AI measurement as a project
Inside the System Boundary Inside the System Boundary In early 2024 the Berkeley AI Research group made a point that practitioners had been circling for a while: state-of-the-art results now come from compound systems, not individual models. A production RAG pipeline is a retriever, a reranker, a generator, and a prompt
Checking the Math on the $500 Million AI Spend Story This piece originally ran as a shorter post on my LinkedIn on May 29, 2026. I'm moving it to the blog, with a couple of added charts. I saw the headline, read the article, and couldn't believe it. So I did the math. I still couldn&
From Model Lift to the P&L From Model Lift to the P&L You ran the holdback. You measured the lift. You have a clean $2M incremental-value number. Then someone from finance asks how it shows up in EBITDA, and the number gets complicated. That question is not pedantry. It is the real test
Eval-Driven Development: The Discipline Agent Teams Are Missing Fifty-seven percent of organizations have AI agents running in production right now. McKinsey QuantumBlack tracked this in their latest global AI survey, and the number surprised almost everyone. What didn't surprise anyone paying attention: quality is the number-one barrier to deploying more of them. Most of
Your Agents Are Principals. Treat Them Like It. Ninety-one percent of organizations have already deployed AI agents. Ten percent have a governance strategy for them. That gap is a decision that hasn't been made yet. The window to make it cleanly is closing. I have watched this exact pattern before. A decade ago, service accounts
The Four Layers of Runtime AI Governance The Four Layers of Runtime AI Governance August 2 is five weeks away. That's the date the EU AI Act's Annex III high-risk AI obligations take full effect. Credit decisions, hiring systems, critical infrastructure controls, educational scoring, law enforcement AI: if your organization operates any
The Most Important Person in Your AI Governance Program Isn't in Your C-Suite The Most Important Person in Your AI Governance Program Isn't in Your C-Suite The AI governance org charts I see most often look the same. Board oversight at the top. A CAIO or CTO in the middle. A cross-functional council below that. Legal, InfoSec, HR, a
Your AI Governance Policy Is Not Governance Your AI Governance Policy Is Not Governance I sat in a meeting last month where a CTO walked me through their AI governance framework. Forty-two slides. Acceptable use language, data classification tiers, a numbered list of prohibited applications. Legal had reviewed it. HR had signed off. The CISO looked
Two Failure Modes in LLM-Generated Pull Requests: Capability-Gap and Scope-Discipline Haircuts in a Controlled 216-Run Evaluation Two Failure Modes in LLM-Generated Pull Requests: Capability-Gap and Scope-Discipline Haircuts in a Controlled 216-Run Evaluation Abstract LLM-generated pull requests can fail mergeability review for two structurally distinct reasons. The first is a capability-gap haircut: the model cannot produce output that meets maintainer standards,
Cost-Adjusted Capacity: When Tokens Stop Converting to Mergeable PRs Cost-Adjusted Capacity: When Tokens Stop Converting to Mergeable PRs Most AI coding budgets optimize the wrong variable. They track token spend and model tier as if buying a bigger model and feeding it more tokens reliably produces more shipped software. The evidence says otherwise, and the gap between what
prompt-engineering Using Evaluation Data to Match Model Capability to Task Requirements Evaluation data from your prompt testing reveals which models actually work for your task. Using that data to choose models is more reliable than benchmark scores and more cost-effective than defaulting to the largest available model.
Finding the Causal Root: Measuring Value When No Single Model Owns the Outcome Accuracy doesn't equal value. Most AI projects fail due to misframed problems and missing measurement—focus on causal impact, not just model performance.
POST-AI Judgment Was Built by Proxy. AI Automated the Proxy. Activity was never a guarantee — people still hit plateaus. But it gave you the raw material. AI automated the proxy before we designed the replacement.
fulfillment Fulfillment Is Not Enjoyment Enjoyment fades the moment you chase it. Fulfillment requires the opposite — it lives on the other side of difficulty. The research is clear, and so is what to do about it.
When Small Mistakes Compound: Building Systems That Survive Imperfect AI Small AI mistakes can snowball into big problems when they ripple through automated processes. Learn how to build systems that catch errors before they compound and cost your business.
efficiency The Compounding Tax: Why Small Model Errors Become Large System Failures Small model errors compound rapidly in sequential tasks, turning seemingly reliable systems into failure-prone ones. Don't rely on per-step accuracy—trajectory reliability is what matters, and it degrades faster than you think.
The Machine Is Learning to Build Itself. You Still Have to Decide What to Build. AI is rapidly writing code – Anthropic's engineers now ship 8x more per quarter. But don’t panic; it highlights the crucial need to strategically manage AI adoption, following a proven maturity model.
Proving Competency at Every Stage: The Eight Stages of AI Maturity Feeling advanced and being advanced are different things. This is the measurement guide for each of the eight stages: what proof looks like, what gaps look like, and the signals that tell you whether you are where you think you are.
Before You Architect Anything, Agree on What the AI Actually Does AI projects often fail because teams misunderstand each other's vision. A shared classification system—a taxonomy and ontology—can ensure everyone agrees on what the AI *does* before building it.
labor-compression The Productivity Paradox: When AI Helps Everyone Equally AI boosts productivity most for junior workers, not experts—a surprising twist that reshapes how we think about knowledge work and career progression. Is your role becoming "collapsable" or uniquely valuable?
mentorship Mentorship Is a System, Not an Accident Stop waiting for accidental mentorship! Real career growth happens when mentorship is designed with a specific ask and recurring cadence.
