The Tokenize-and-Restore Sandwich - A Proxy That Never Sees the Real Value

The problem

I built a client-service chatbot for a fictional wealth-management firm, one I called PortfolioAssist, and pointed it at the kind of question relationship managers actually field: what's the balance on account ACCT-004821 for a named client. Account numbers, SSNs, and client names show up directly in the same sentence, because that's how people actually ask.

The first version I built has no proxy layer between the application and the external hosted model at all. Every query, including the ones naming an account number, an SSN, and a client in the same breath, went straight to the external model with those spans sitting in the outbound payload exactly as typed. I seeded 20 fake clients with invented names, fake account numbers in an ACCT-000### pattern, fake SSNs, and a matching set of 40 natural-language queries referencing those identifiers, and measured the percentage of known sensitive spans present in the outbound prompt. It landed at or near 100% across every span type. Nothing in the pipeline touched the payload before it walked out the door, so of course it showed up whole on the other side.

Whatever gets logged, cached, or retained at that external boundary carries the raw identifiers exactly as the client typed them. That's not a hypothetical downstream risk I'm gesturing at. It's the literal content of the outbound call, sitting in whatever storage the receiving system uses, for as long as that system's own retention policy says it should.

The pattern

The fix is an inline proxy that sits between the application and the external model and does exactly two things, in a fixed order. On the way out, it detects sensitive spans, account numbers, SSNs, client names matched against the synthetic roster, and replaces each with a stable placeholder token, something like <ACCT_TOKEN_17> or <SSN_TOKEN_3>. The external model never receives a raw identifier. It sees a token that means nothing outside the active request. On the way back, the proxy reverses the substitution before the response reaches the person who asked, so the client-facing answer still reads naturally, with real values restored, even though the external endpoint never touched them.

I held the token map in-process per request, never persisted with a real value attached outside the active call, and rebuilt it fresh from the synthetic roster each time. Running the same 40-query set through this version dropped exposure to 0% by construction. I also tracked round-trip fidelity, the percentage of responses where detokenization correctly restored every real value the client needed to see, and that held at or near 100% as well. The two numbers matter together. A tokenization proxy that hides everything but restores nothing correctly just trades one broken chatbot for another.

This shape isn't a novel technique at this point. Multiple sources describe an anonymize, infer, de-anonymize sandwich as the reference architecture for a PII privacy proxy wrapping every LLM call, and current LLM gateway products implement exactly this tokenize-before-call, detokenize-after-call pattern as standard infrastructure now. One source states the requirement in almost the words I'd use for PortfolioAssist's metric: the transformation has to ensure no identifiable sensitive data is exposed to the external service. I'd treated reversible tokenization as an unproven idea in earlier notes of my own. Building PortfolioAssist and checking it against what's actually shipping in production changed my mind. Multiple vendors ship this as standard infrastructure, and the academic framing matches what I built almost word for word.

Design considerations

The scope of what this pattern actually covers matters more than it looks like it should. I scoped PortfolioAssist's proxy to one boundary: the outbound call to the external model. It never touches anything else, and that's a deliberate, narrow claim rather than an oversight to apologize for.

There's a channel this scope doesn't reach, and I want to name it precisely because the shape of the miss is easy to overlook. A malicious tool, crafted with a name and description designed to get selected, can induce an agent to pass its own runtime context, its prompt, its trajectory, its available tool list, as the tool's own input arguments, which the tool then transmits somewhere outside the trust boundary entirely. That's the exfiltration channel ContextLeak, an August 2026 paper, demonstrates directly. It runs straight through a boundary this proxy was never built to cover: the arguments an agent hands to a tool it decided to invoke on its own, a channel completely separate from the model call this pattern protects. A tokenization proxy sitting in front of the model call does nothing about a tool call that never routes through it.

The practical calibration question is where to draw the boundary of what "the proxy" actually means in a given system. If an agent only ever talks to one external model and never invokes tools that could carry their own egress path, the model-call boundary is the whole system, and this pattern covers it completely. The moment an agent gains tool-calling ability, especially tools built or maintained outside your own control, the model-call boundary stops being the only boundary that matters, and a proxy scoped to it alone will look like complete protection while leaving a second, unguarded door standing open. I'd build the tokenization proxy the same way regardless, because it's the right fix for the boundary it targets. I just wouldn't let it stand in as the answer for a boundary it was never built to reach.