Three Tools, One Gate
The problem
A health plan I worked with runs three internal tools that each call a model directly. A claims-adjudication assistant. A member-chat bot. A clinical-notes summarizer. Each tool had its own hardcoded client pointing at its own vendor endpoint, configured independently by whichever team built it. Every request into any of the three carried a flag marking whether it contained protected health information and a data-sensitivity level, and nothing downstream was required to honor either one.
With no central gateway, each tool's routing and logging behavior was whatever that team felt like implementing on a given afternoon. The clinical-notes summarizer, the tool handling the most protected health information of the three, happened to route to an endpoint that wasn't region-appropriate for that data. The member-chat bot logged nothing at all. Three tools, three log formats, and no single place anyone could audit which model handled which sensitivity level even if they went looking.
I put a number on how bad that gap actually was. Central log coverage, the percentage of requests appearing in a single unified compliance log with a consistent schema, came in below 40 percent. Cross-tool policy consistency, the percentage of PHI-tagged requests landing on the same class of endpoint regardless of which tool sent them, failed for the identical reason: nothing enforced consistency across three tools built by three teams with three different sets of priorities. Every one of these systems worked. Each answered its request and returned a plausible result. What was wrong with them never showed up in the metrics anyone was actually watching, because nobody had built a place where those metrics would converge.
The pattern
The fix doesn't touch a line of the three tools' own logic. It removes their direct vendor clients and routes every one of their calls through a single gateway that applies one routing policy and writes one unified log entry per call, regardless of which tool originated it. PHI-tagged requests land on the region-appropriate endpoint class. Everything else routes by cost and quality. Adding a fourth tool later means pointing it at the gateway. No new routing logic to write from scratch.
Run the same workload through this version and central log coverage reaches 100 percent, because every call now passes through the same enforcement point by construction rather than by whatever a given team happened to remember to build. Cross-tool consistency for PHI-tagged requests reaches 100 percent too, for the same reason. A request landing on the wrong endpoint class stops being a possibility that depends on which team wrote that particular tool, and becomes a question the gateway itself either enforces or doesn't.
Two pieces of market evidence from this year back this shape up, even though neither vendor set out to solve this exact problem on purpose. Palo Alto Networks completed its acquisition of Portkey, a standalone AI gateway processing trillions of tokens a month, folding it into a security platform. Read that as validation that one control plane, vendor-independent, policy-enforced, with a unified log, has become infrastructure enterprises are willing to pay security-platform prices for. A few years ago this would have been dismissed as a defensive niche build some paranoid architect insisted on. LiteLLM, Portkey, and OpenRouter becoming the default vendor-neutral abstraction layer for enterprise multi-provider LLM use by the middle of this year points at the same premise from a different angle. The industry converged on one gateway, independent of cloud choice, as the ordinary shape of this problem. Nobody treats it as the exotic choice anymore.
Design considerations
The gateway's value depends entirely on a boundary I want to be precise about. It consumes a PHI flag and a data-sensitivity level to decide where a request routes. It doesn't generate either flag itself. If the upstream classification producing those tags is wrong, silent, or absent for a given call, the gateway will faithfully and consistently route a misclassified request to the wrong endpoint class, complete with the same clean unified log as a correctly classified one. A routing gate is only as trustworthy as the label it gates on. Standing up the gateway without also auditing where those labels come from just moves the failure point one layer upstream and makes it harder to spot, because now there's a clean-looking log sitting on top of it.
The migration cost is also real, and I'd rather name it than gloss over it. Three tools each had their own hardcoded client, their own retry logic, their own assumptions about what a response looked like. Routing all three through a shared gateway means each tool gives up direct control over its own call path, and someone has to own the gateway's policy going forward, including what happens when a new sensitivity category shows up that the original policy never anticipated. Centralizing three tools' routing into one gateway also centralizes the blast radius of a gateway misconfiguration. Get the routing policy wrong at the gateway and every tool behind it inherits the mistake simultaneously, instead of the mistake staying contained to whichever single tool used to make that call directly.
This pattern earns its cost fastest in exactly the scenario I started with: multiple tools, built by multiple teams, over multiple points in time, each making its own routing and logging decisions independently. A single tool with a single vendor relationship doesn't need a shared gateway sitting in front of it. Neither does an organization where every tool already happens to be built by the same team using the same shared library, because the coordination problem the gateway solves never really existed there in the first place. The pattern is for the moment an organization notices it can no longer answer, with confidence, which model handled which sensitive request across its own tools. Once that question can't be answered from a single place, the three separate logs are three separate audit gaps wearing different names.