The Endpoint That Was Never in the Room

The problem

I built a router for a loyalty-program chat assistant serving customers in the United States, Canada, and the European Union. Three model endpoints stood behind it, each with its own cost per thousand tokens, quality score, latency profile, and declared location. The router scored every incoming request against all three on cost and quality and called whichever one won. I seeded the US endpoint as the cheapest and highest quality of the three, so it won most of the time.

That would be a completely defensible procurement decision if customer data were free to sit wherever it happened to land. It isn't, and Quebec and the EU are explicit about it. A meaningful share of the requests I ran through that router, at least a fifth, carried a tag marking the customer as a Quebec resident under Law 25 or an EU resident under GDPR. Those tags sat right in the request payload. Nothing in the router read them before scoring ran. Residency got checked, if it got checked at all, in an after-the-fact log review, well after the US endpoint had already served the request from a data center that was never a legal candidate for the job.

I measured the resulting number as a residency violation rate: the percentage of requests served by an endpoint outside the jurisdiction that request was legally required to stay in. With the cheapest endpoint sitting in neither Quebec nor the EU, and a fifth of the workload tagged as belonging to one or the other, the violation rate ran above 15 percent. From inside the system, every one of those violations looked like an ordinary, successful response. The customer got an answer. Latency was fine. Cost was low. The only thing wrong with the transaction was a question the router never asked: was this even legal to do?

The distinction that trips up more routing logic than it should is that "in Canada" and "in the specific province the requirement names" sound like the same claim until they aren't. Law 25 cares whether a Quebec resident's data stayed under Quebec's own regime, not merely whether it stayed inside Canada in some general sense. A router scoring only on a country-level tag will wave a Quebec request through to a compliant-sounding Canadian endpoint that fails the actual requirement. The failure is an ordering problem dressed up as a coding bug. Cost and quality got to vote on a candidate that should never have been allowed into the room.

The pattern

The fix changes exactly one thing about the order of operations. Before any cost or quality scoring runs, a hard filter removes every endpoint outside the requesting customer's required jurisdiction, at both the country level and the sub-national level where that distinction matters, from the candidate pool entirely. Cost and quality scoring then runs only over whatever survives that filter. A Quebec-tagged request never sees the US endpoint as a candidate, no matter how favorably it would have scored. It's not in the room when the vote happens.

I built this as two genuinely separate steps with real branching logic: a residency-filter stage prunes the candidate list per request, and only afterward does a scoring stage rank what's left. If the filter empties the pool entirely, meaning no compliant endpoint exists for that request at all, the system has to say so explicitly rather than quietly falling back to an out-of-jurisdiction endpoint under load. That detail carries more weight than it looks like it should. A hard gate that softens into a suggestion the moment traffic spikes was never a hard gate.

Run this version against the same workload and the residency violation rate drops to structurally zero, since non-compliant endpoints never enter scoring in the first place. I also log cost and quality for every resolved request, specifically so the price of compliance stays visible instead of getting buried in an aggregate number. Filtering out the cheapest endpoint for a share of requests raises the average cost of serving those requests. I report that delta directly rather than pretending the fix arrived for free.

Design considerations

The tempting shortcut is to treat jurisdiction as a single tag, country only, and skip the sub-national layer entirely. That shortcut is exactly how the original failure happens. Quebec's requirement doesn't collapse into "somewhere in Canada," and a system that only checks the coarser tag will pass its own residency audit while still violating the actual law. Getting the granularity of the jurisdiction tags right, down to the province or state where a real requirement names one, is the calibration question that determines whether this pattern does anything at all.

The cost of doing this correctly is real, and I'd rather name it than wave it away. Removing the cheapest candidate from the pool for a meaningful share of traffic raises the blended cost of serving that traffic. Organizations that only look at the average cost-per-request number will see it go up and read that as the system getting worse, when what actually happened is that the number stopped hiding a liability. The right response is to report the compliance premium as its own line item, not to fold it back into a blended average where it disappears again.

There's also a boundary to this pattern that matters more the longer I sit with it. It filters on physical and legal jurisdiction, where a request's data is required to stay. It has nothing to say about who owns or trained the model running at a compliant-looking endpoint. Two endpoints can sit in the identical, fully compliant jurisdiction, and this pattern alone can't distinguish between them on any basis other than location. Model provenance, vendor dependency, and geopolitical exposure of the entity operating the endpoint are separate questions this pattern was never built to answer, and treating a residency-clean router as having resolved all of them is its own quiet mistake.

Where this pattern doesn't earn its keep is single-jurisdiction operations. If every customer a system serves is subject to the identical data residency requirement, and every candidate endpoint already satisfies it, the filter has nothing to do. Building the machinery anyway adds a layer of branching logic and an explicit failure state for a scenario that structurally can't occur. The pattern's value scales with how many distinct jurisdictions a system's actual customer base spans, and with how serious the consequence is when a residency requirement gets treated as a soft preference instead of a hard constraint. Given how many enterprises are now building their AI stacks around local and regional vendor requirements rather than pure capability comparisons, that second condition is becoming harder to assume away.