The Approval That Expires - Building a Vendor Gate on Evidence

The problem

I built a small governance agent to answer one question employees kept asking: can I use this third-party AI vendor on a client's documents. The obvious first version checks a single allow-list, a flat table of vendor names marked approved or not approved, updated whenever someone remembers to update it. An employee asks about a vendor, the agent looks the name up, and answers yes or no based entirely on whether that name is sitting on the list.

I seeded the evidence behind that list the way a real compliance team would have built it over time: twelve vendors, each with a pen-test date, a SOC 2 report date, a data-handling scan result, and an uptime record on file. Five of those twelve had evidence that was more than twelve months stale, a report that had expired, a scan that was never rerun, monitoring that had gone quiet, despite every one of them still sitting on the approved side of the list. I ran twenty approval requests across the twelve vendors, seven of which should have been denied if anyone were actually checking whether the underlying evidence still held. The allow-list version approved requests against stale evidence at better than a thirty percent rate. Nearly a third of the approvals I logged were really just a name match against a table nobody had touched recently.

What bothered me about that number was the mechanism sitting underneath it, once I looked past the specific vendors involved. The list answers "was this vendor approved," which is a question about the past. It never asks "is this vendor still what it was when we approved it," which is the only question that actually matters at the moment somebody wants to hand it a client's documents. A vendor's SOC 2 report from fourteen months ago tells you what an auditor observed fourteen months ago. It says nothing about this week.

The pattern

The fix replaces the lookup with a pipeline that runs at request time instead of at approval time. A fetch step pulls the vendor's current evidence record: the pen-test date, the SOC 2 report date, the data-handling scan result, the uptime status, whatever artifacts the policy cares about. An evaluation step checks each artifact against an explicit, written threshold: a SOC 2 report inside twelve months, a pen test inside twelve months, no failed data-handling scan on file. A decision step approves or denies and cites which threshold each denial failed.

Run the same twenty requests through that version and the stale-evidence approval rate drops to zero, because no request tied to expired evidence clears the gate regardless of what the list used to say. A vendor approved yesterday gets denied today the moment its pen test rolls past the twelve-month mark. A brand-new vendor with a fresh SOC 2 report and a clean scan gets approved on the strength of that evidence alone, without ever having spent a day on a legacy list. The list stops being the thing that grants access. It becomes, at most, a cache of a decision the evidence has to keep re-earning.

I didn't invent this to make an internal process feel more rigorous. It matches where SOC 2 auditing practice was already heading in 2026. Bitsight's guidance to auditors states plainly that a point-in-time vendor questionnaire no longer satisfies the standard: auditors now expect a continuous-monitoring export with timestamps proving a control held for the entire audit window, the kind of proof a single day's paperwork can't produce. That's the same claim stated as an audit requirement instead of an engineering one. An approval has to be re-earned against current evidence. It can't be granted once and assumed to hold forever. The same mechanism extends cleanly to a different kind of evidence in the MCP tool ecosystem: NSA's MCP Security Design paper and the Cloud Security Alliance both push toward hash-pinning and signing a tool's own definition, so a tool's identity is a cryptographic artifact that has to stay checkable and current, the same way a pen-test date does. And OWASP's AIUC-1 standard now crosswalks to the broader Agentic Applications Top 10, which means a vendor's registration and risk-rating evidence increasingly has to satisfy an insurance-underwriting rubric on top of an internal checklist. Evidence just keeps growing new categories. The mechanism for checking it stays the same.

Design considerations

The gate is only as good as two things neither the pipeline nor the thresholds can fully control: whether the compliance function chose the right thresholds, and whether the vendor's reported evidence is actually true. Twelve months is a defensible cutoff for a pen test, but it's a policy choice somebody has to own and defend, open to revision as practice changes. And the pipeline has no way to catch evidence that's current on paper but falsified, or a vendor that passes today's check cleanly and gets breached tomorrow, well before the next scheduled evaluation. Fresh evidence is a much better signal than a stale name on a list. It is still evidence, with all the limits evidence has.

There's a real difference between what this pattern answers and two adjacent questions that look similar from a distance. Checking a tool's own cryptographic signature is the identical mechanism applied one level down, at the individual tool rather than the vendor, and it's tempting to treat the two as the same problem solved twice. They're close enough that I read them side by side before trusting either one, because a vendor can be fully compliant while shipping one compromised tool, and a tool can carry a valid signature from a vendor whose broader posture has quietly lapsed. Vendor-level and tool-level evidence answer different halves of the same trust question, and collapsing them into one check loses whichever half you weren't looking at. Separately, a policy that routes traffic to whichever backend is currently the cheapest or fastest compliant option answers "which approved vendor handles this request." It never asks "is this vendor still approved." Evidence freshness feeds that routing decision, but the two questions stay distinct.

Where this pattern strains is speed. Building a real evidence pipeline, one that actually pulls a scan result or a monitoring export instead of trusting a self-reported PDF, takes integration work per vendor, and that work doesn't scale down gracefully for an urgent one-off request. If a team needs an answer on a brand-new vendor this afternoon for a time-sensitive client ask, an evidence pipeline built for twelve steady vendors won't have that vendor's feed wired up yet. The practical fallback in that case is a manual evidence pull done once, by a human, fast, which is exactly the workflow this pattern exists to retire for everything that isn't an emergency. Used as the everyday gate, it holds up well. Used to bless an unvetted vendor on a deadline, it's only as fast as the person doing the pulling.