Three Services, Three Different Answers About What Was Live

I ran a subscription-churn model once with three independently operated components sitting around it: a feature store computing churn features, a serving endpoint scoring churn probability, and a drift monitor checking those features against a stored baseline. Each component tracked "which model and feature version is currently live" in its own local config file, updated by its own separate deploy script. On day twenty-five, against a synthetic sixty-day feed of six thousand subscriber accounts, a new model version went live with a redefined engagement-score feature calculation.

The problem

The serving endpoint's config updated immediately, because that was the thing being deployed. The feature store's feature-definition config and the drift monitor's baseline config were supposed to update too, by a separate, uncoordinated script, but they lagged behind for twelve days. For those twelve days, the serving endpoint was scoring churn probability using the new model against features that were still nominally labeled with the old definition, and the drift monitor was comparing the new model's live traffic against the old model's stale baseline. Comparing apples to oranges the entire time, and every drift reading it produced during that window was meaningless: the math itself was fine, but the two things it was comparing no longer meant the same thing. Nobody noticed any of this until a manual audit on day thirty-seven finally caught the mismatched version tags sitting in the logs.

What makes this failure mode uncomfortable is that every individual component was doing its job correctly. The serving endpoint served the right model, the feature store computed features according to whatever definition it had on file, and the drift monitor compared against whatever baseline it had on file: each one correct in isolation. Nothing crashed. Nothing threw an error. Three correctly functioning components produced a systemically wrong answer, because correctness at the component level says nothing about agreement across components, and nothing in the architecture was checking for that agreement.

The pattern

The fix I built removes the possibility of disagreement by construction, instead of trying to catch it faster after the fact. The feature store, the serving endpoint, and the drift monitor all read "current model and feature version" from one shared registry instead of maintaining independent local configs. A deploy becomes a single atomic write to that registry, so there's no window left in which any two components can hold a different answer about what's currently live.

I refactored all three components against the same churn scenario to read from a shared, versioned registry backed by SQLite, in-process, with no external service required. A deploy script performs the atomic registry update, and all three components observe the identical version on their very next read. Measured version-mismatch duration dropped to zero, and the count of meaningless drift readings dropped to zero right alongside it, because there was no window left for the components to disagree in. This architecture already has a name: Databricks and Tecton's feature-store reference architecture describes a canonical five-component model (model, transformation, storage, serving, monitoring, and registry), with the registry explicitly described as the central interface that serving and monitoring both read from. The pattern here is just that architecture applied specifically to the version-agreement problem.

Design considerations

What this pattern actually resolves is narrower than "keep everything in sync" sounds like it should mean. It guarantees agreement about which version is live, while leaving open what happens once a model stops being a static artifact and starts continually updating itself. Once a model is stateful, or updating on some rolling basis, versioning the model as a single fixed thing stops being sufficient on its own. The production data stream that shaped the model's current live state needs its own versioning and replay capability, or the model can't actually be reproduced or rolled back, even with a perfectly synchronized registry correctly telling every component the correct version number. I built the first half of that problem here. The second half is still open, and I don't think a shared registry gets you there by itself.

There's also a real cost to this pattern that's easy to underweight because it sounds purely additive. A shared registry is a new single point of coordination that every component now depends on, and every component that reads from it on every request has added a dependency it didn't have before, even if that dependency is fast and local. If the registry itself becomes unavailable or slow, you've traded three independently-failing components for one component whose failure now blocks all three. I think that tradeoff is worth making almost every time the disagreement it prevents is expensive, twelve days of meaningless drift readings is expensive, but it's a real tradeoff that has to be paid for deliberately, and I'd want the registry itself built to be boring and reliable before I trusted three other services to depend on it atomically.

I'd also flag that atomic agreement about version numbers is a narrower guarantee than agreement about meaning. Two components can correctly read the identical version identifier from the registry and still compute something differently if the underlying code paths that consume that version diverge in some other way I haven't accounted for here. The registry answers "which version," cleanly and completely. Whether everyone who reads that version does the same thing with it is a separate question, with its own discipline behind it.