Finding the Causal Root: Measuring Value When No Single Model Owns the Outcome

Accuracy is not value. A churn model with 0.92 AUC creates zero value if no one acts on its scores. A code copilot with a high acceptance rate creates zero value if the developer would have shipped the same code anyway. A vision system that catches 99% of defects creates zero value if the line was already catching 98% another way. The model is only as valuable as the action it changes, and the action is what you have to measure.

This sounds obvious. The data says it is not. RAND's 2024 study found more than 80% of AI projects fail, roughly twice the rate of non-AI IT projects, and the root causes were misframed problems and missing measurement, not weak algorithms. McKinsey's 2025 State of AI survey found only about 39% of organizations could attribute any EBIT impact to AI, and most of those put it under 5%. BCG found only about 4% of companies are creating substantial value. The widely-cited MIT figure that 95% of generative AI pilots show no P&L return is worth treating with care, since it comes from a non-peer-reviewed report with a contested methodology, but it points the same direction as the better-sourced numbers. These studies measure different things, failure versus abandonment versus value-capture maturity, which is exactly why you should read them side by side rather than collapse them into one scary statistic. The throughline is consistent: value is gated by what the organization does with the model, not by the model's accuracy.

So the question is not "is the model good?" The question is "did this specific thing cause a change in an outcome the business cares about, and would that change have happened anyway?" That is a causal question, and causal questions do not answer themselves from a dashboard.

Three layers people insist on conflating

The most expensive mistake in enterprise AI measurement is treating one layer of measurement as if it were another. There are three, they have different owners and cadences, and none substitutes for the others.

Layer 1 is model verification. Accuracy, precision, recall, calibration, drift, fairness, latency under load. It answers "is the model performing as designed?" It is owned by the ML team and rolls into MLOps. A perfectly calibrated model with zero drift can still create zero value, because Layer 1 says nothing about whether the action it informs would have changed without it.

Layer 2 is causal lift. Did this model create incremental value against an explicit counterfactual? This is the layer that requires real design, randomization, holdback, stepped-wedge rollout, or a defensible quasi-experiment. It is the layer almost everyone skips, and skipping it is why so many ROI claims evaporate under scrutiny.

Layer 3 is systemic monitoring. Revenue, EBITDA, churn, NPS, throughput, working capital, the metrics leadership actually reports. Many models, many initiatives, pricing moves, the macro environment, all push on these at once. Layer 3 cannot attribute movement to a single model. That is its defining limitation, not a flaw to engineer around.

The fallacies follow directly from confusing these. The metric moved, so the model worked, ignores the three other initiatives that shipped that quarter. The model is accurate, so it is valuable, mistakes Layer 1 for Layer 2. We have a dashboard, so we do not need measurement, mistakes telemetry for causal attribution. Each is common enough to deserve a name, and each costs money. When a Layer 3 metric moves favorably and no Layer 2 measurement exists, the honest sentence is "the metric moved, the model is one of several contributors, the causal measurement begins next quarter." Not "the model drove the lift."

You have to design the counterfactual in

Value is a comparison between what happened and what would have happened without the model. That second thing, the counterfactual, does not exist in your data unless you put it there. This is Pearl's ladder of causation in operational dress: association lives on the bottom rung, but intervention and counterfactual reasoning, the rungs you actually need, require structure you build before the data arrives.

The toolkit is well established and ordered by strength of claim. A randomized controlled trial is the gold standard, randomize who receives the model's treatment at the decision level. Holdback or champion-challenger keeps a random fraction on the prior baseline, the standard move for recommenders and production swap-ins. Stepped-wedge rolls out to clusters, sites, regions, segments, in a randomized sequence, so each cluster is its own control before it crosses over. This design has a deep methodological literature in clinical trials, and it fits phased AI rollouts almost perfectly. When you cannot randomize at all, you drop to difference-in-differences, synthetic control, regression discontinuity, or instrumental variables, each weaker and each demanding a defensible assumption. Pre/post analysis, comparing before and after while holding nothing constant, is the weakest and almost never the right answer alone.

The non-negotiable principle is that you design measurement before you train. If the measurement plan shows up after the model is in production, randomization is already impossible, the baseline is contaminated, and you are left with the weak end of the toolkit. The strongest designs depend on choices made before any data is collected: the randomization seed, the holdback selection, the instrumentation. Measure-later is how a defensible number becomes a marketing number.

And every value claim is net of cost. Gross lift minus the full cost of building, running, governing, and monitoring the model, minus the cost of the action you would have taken anyway. A model that saves five million dollars but replaces a process already saving three is a two-million-dollar win, not a five-million one. That last subtraction, the counterfactual cost, is the one optimistic decks always forget.

Now make it multi-model, where the root hides

Here is where it gets genuinely hard, and where most measurement frameworks quietly give up. The systems we ship now are rarely single models. They are compound systems. Berkeley's AI research group made the case plainly in early 2024: state-of-the-art results increasingly come from systems with multiple components, routers, retrievers, multiple foundation models, critics, specialist agents, not from a single monolithic model. Composition is now the architecture, which means the causal root of value is now distributed across components that interact.

A compound system that beats its best single component is doing one of three things, and usually all three at different magnitudes. Component lift is what each model contributes independent of the composition. Composition lift is what the router, gate, or supervisor adds over a static baseline. Interaction lift is the value that emerges from the composition itself, that no single component could produce. The number a CFO signs needs to know which of these is doing the work, because they have very different durability. Component value travels with the foundation-model vendor and can be replicated by anyone. Composition value travels with your engineering and is defensible at renewal.

Measuring this requires designs that go inside the system boundary. Ablation lift replaces one component with a credible baseline, a passthrough, a small default model, a deterministic rule, and runs it in parallel to isolate that component's contribution. Shadow routing runs the production router and a baseline router side by side on the same traffic, acts on the production choice, and measures the outcome difference on the subset where they disagree, weighted by the disagreement rate. Specialist-isolation A/B holds the supervisor and protocol constant while swapping a specialist, to isolate what that specialist is actually worth in an agent network. This is the same logic the compound-systems literature uses: component-level experimentation and cost attribution, not model-level metrics, are how you find the root.

The validity threats multiply too, and they are subtle. Routing coupling: when the same router decides which model acts and which observation gets logged, your per-component sample is router-biased, so component lift on that sample is not componentwise causal lift. Cascade contamination: in a confidence cascade, the second-tier model only ever sees cases the first tier could not handle, so comparing tiers measures the sample, not the models. Critic-induced selection: when a critic gates outputs, apparent quality reflects the critic's strictness as much as the components' quality. Specialist co-evolution: when supervisor and specialists are trained together, you cannot untangle them with ablation alone, the counterfactual had to be designed before joint training. And compositional Goodhart, the one that compounds fastest.

Goodhart's law, when a measure becomes a target it ceases to be a good measure, is not a slogan in these systems. Recent formal work ties it directly to reward hacking and overfitting: optimize a proxy whose causal path to the real goal is indirect, and you degrade the goal. Tune a router or supervisor on a proxy metric and it will eventually break the very components the proxy depends on, and composition-level Goodhart compounds faster than the single-model version because the proxy sits upstream of everything. The mitigation is unglamorous: pair leading metrics with lagging outcome metrics, recalibrate on a cadence, and refuse to auto-tune on proxies.

I have watched a router with no measurable lift over a static rule get defended on engagement, because routing accuracy looked great on a dashboard. Routing accuracy is a Layer 1 property. Router value is the Layer 2 lift the router's choices produce on the business outcome. Both belong in the report. Only the second belongs in the value claim. The right response to a router that adds no lift is to retire it, not to defend it.

The bridge to the number leadership reports

Single-model lift, even component-resolved lift inside a compound system, still does not roll cleanly up to the revenue or churn number leadership reports. That is the Layer 2 to Layer 3 gap, and closing it is a discipline of its own. You combine the individual causal lifts under an explicitly stated additivity regime, because the lifts almost never sum cleanly. Models interact and cannibalize each other. Some lift is captured by intermediate metrics before it reaches the top line. Measurement windows do not align. And the business metric includes large non-AI drivers, pricing, market, reorganizations, that naive summation would silently credit to AI.

The honest version uses a causal lineage map, a directed graph from model outputs to decisions to operational metrics to the reported outcome, built with the client, that surfaces overlapping causal paths before you attribute anything. On top of it you bring techniques sized to the data: marketing mix modeling when you have eighteen-plus months of history, Bayesian structural time series, Google's CausalImpact at portfolio scale, for clean intervention windows, and Shapley-value decomposition to allocate credit among contributors that jointly moved a metric. Shapley is worth dwelling on, because it is the same game-theoretic math that powers SHAP in model explainability, which gives you one vocabulary for attributing value across components of a system and across drivers in a portfolio. And a drivers analysis without confidence intervals is a presentation artifact, not an attribution. The interval widens as you stack measurement error, additivity error, and non-AI estimation error, and showing that honestly is the point.

This is the part most firms will not do, which is exactly why it is worth doing. The market mostly asserts AI ROI and stops at a single-model number, or worse, reads Layer 3 movement as proof. Measuring what others assert is harder, slower, and occasionally produces a number smaller than the one on the pitch deck. It is also the only number that survives an audit, and the only one that earns the right to say you found the causal root rather than the nearest convenient correlation.

Sources and further reading: Zaharia, Khattab et al., "The Shift from Models to Compound AI Systems," Berkeley AI Research, 2024; Brodersen et al., "Inferring causal impact using Bayesian structural time-series models," Annals of Applied Statistics, 2015 (Google CausalImpact); Abadie, Diamond, Hainmueller on synthetic control, JASA 2010; Hussey & Hughes on stepped-wedge designs, 2007; Pearl & Mackenzie, The Book of Why, 2018; RAND, "The Root Causes of Failure for AI Projects," 2024; McKinsey, "The State of AI," 2025; BCG, "Where's the Value in AI?," 2024; recent arXiv formalizations of Goodhart's law (2505.23445, 2410.09638); NIST AI Risk Management Framework (AI 100-1), 2023.