A Benchmark Nobody Outside the Vendor Can Run

My five questions for vendor benchmarks end with reproduction: has anyone outside the vendor run the claim? Gemini 4 Argon's launch is a case where that question has an unusual answer. For the first stretch after announcement, only Google and a small vetted circle could run the claim.

Google announced Argon on September 30 and reported 77.9 percent on DeepSWE v1.1, a tie for first at 68 percent on CWE-bench, and 65.4 percent on Vals Finance Agent v2, along with wins on 13 of 18 published benchmarks against GPT-6 Astra. Access was limited. Vetted cyber defenders in a program called Fairwind came first, then paid API customers and AI Ultra subscribers, with no general availability date published. A model with that access pattern produces benchmark numbers that only the vendor, and a small circle it selected, can reproduce.

Bloomberg reported that some Google employees privately doubt Argon's real-world coding strength. CNBC asked within days whether Google's model could hold the frontier. Independent evaluators published results within hours of the announcement, which means some outside access existed, and those results were mixed against competing models. Google's figures may well hold up. Until someone outside the vendor reproduces them at scale, they rest on the vendor's own confirmation.

The gap exposes a hole in my original framework. The fifth question assumed that reproduction was possible and asked whether anyone had done it. Gated access makes reproduction impractical for most buyers for a period, so the right status for such a number is unreproducible for now, with an expiry date set by the vendor's rollout that nobody's diligence can move.

A number in that state belongs in a different column from a number that is merely unverified. An unverified claim could be checked tomorrow by anyone willing to run the test. An unreproducible claim waits on an access decision someone else controls. A client recommendation that cites it is borrowing against a date nobody has announced.

Reflection AI's Beam release shows a second variant. The company reports 77.2 on SWE-Bench Pro v2-Hard and 80.1 on Terminal-Bench v2.1, with weights promised for later in October. Until the weights ship, those figures sit in the same category, vendor-reported and not independently runnable. The distinction from Argon is that Beam's gate has a stated end date and a stated mechanism, a public Apache 2.0 release, which makes the waiting period easier to plan around.

Beam's number also illustrates a smaller trap. Beam reports Terminal-Bench v2.1 at 80.1. Anthropic's Claude Sonnet 5.5 reports 70.6 percent on Terminal-Bench 4.0. They share a name and different version numbers, and a reader skimming the two lines will compare them. Benchmark versions change task sets, scoring, and difficulty, so a score on one version says very little about a score on another. Nine and a half points of apparent gap might be nine and a half points of capability, or it might be entirely a difference between tests.

I am adding two lines to my own checklist. The first asks whether anyone outside the vendor can run the claim today, recorded with a date, and a claim that cannot be run gets labeled as such in any document that cites it, along with the vendor's stated plan for opening access. The second confirms the benchmark version before any two numbers get compared, and treats cross-version comparisons as void.

The practical effect on recommendations is modest and cumulative. A recommendation that cites Argon's figures needs a footnote saying they are vendor-reported on a gated model. One that cites Beam needs the weights-pending note. Both footnotes cost a line, and both protect the author from the situation where the number moves after the recommendation went out.

Launch-week benchmark tables will keep arriving as part of how frontier models are introduced. The part under my control is the label I put on each number before it goes into a client document, and the label has to say who can verify it today.

Sources: Google, "Gemini 4 Argon" (Sept 30, 2026); VentureBeat, "Google unveils Gemini 4 Argon" (Sept 30, 2026); CNBC, "Can Google's new model catch up at the frontier?" (Oct 2, 2026); Reflection AI, "Introducing Beam" (Oct 2026); AI Industry Digests (Oct 1-19, 2026). One listing placed Argon developer availability on Oct 13; I could not corroborate it.