Tiering Isn't One Decision, It's Five Ask a team running agents in production how they picked their model, and most give the same answer: the frontier model, because it's the best one on the market. That used to be a defensible position. Five pieces of research published in 2026 make it look like an
The Stopping Rule Your Agent Doesn't Have Every coding agent harness I've configured this year ships with the same buried setting: a repair loop with a round limit. Three tries. Five tries. Ten, if whoever set it up was feeling generous. Nobody measured that number into existence. It's the default that came with
The Sandbox Held Until the Model Decided It Didn't Need To On March 26, a misconfigured content management system at Anthropic exposed roughly 3,000 unpublished internal assets to the open internet. Nothing exotic broke it. No adversary found a hole. A data cache that should have been private wasn't, and among the files sitting in it were draft
The Plan Was Right When You Made It Hand a long-running coding agent a multi-file refactor and walk away. Come back in twenty minutes. The plan it built at minute one assumed a repo that, by minute five, two of your teammates had already changed. It is still executing that plan, confidently, against a codebase that
The Orchestration Layer Is the Reliability Lever Watch a client team debug a disappointing agentic pilot and you can predict the first thing they'll ask for: a better model. GPT to Claude, Claude to Gemini, whichever model is newest this quarter. It's the reflex, and it's usually answering the wrong question.
The Open-Weight Cost Curve Is an Architecture Story Name the last open-weight model that topped a coding leaderboard. Now try to remember which one still held that spot two weeks later. Kimi K2.5 gave way to K2.6, then K3. GLM-5 became 5.2, then 5.5. DeepSeek V4 landed somewhere in the middle of
The Model Didn't Change. The Default Did. Open Anthropic's April 23 postmortem and look for the sentence where they say the model got worse. It isn't there. The model never changed. Something upstream of it did, for thirty-four days, and it took Anthropic's own engineers most of a month to
The Interpretability Audit You Can't Run Yet A client asked me a version of this question last month: can you audit our agent for whether it's actually compliant, or just performing compliance because it knows it's being watched. That's a question about whether it behaves one way when it thinks the
The Gap Between Coding Benchmarks and Client-Ready Software Pull up the benchmark slide in any AI coding vendor's deck and ask one question before you read another number on it: would a senior engineer at your company actually merge this pull request? Not "did the tests pass." Would they merge it, on your codebase,
The Free Half of Speculative Decoding Ask most teams running speculative decoding on constrained hardware what they've quantized, and they'll tell you about the big model. Shrink the target model to INT4, keep the draft model at full precision, ship it. That's the instinct, and it leaves a second speedup
The Bottleneck Isn't the Fix, It's Finding What to Fix Ask a client team why their coding agent pilot is burning through budget on a modernization sprint and the answer is almost always the same: the model. Swap Sonnet for GPT, GPT for Gemini, whichever one had the best demo last quarter. I've sat in enough of these
The Benchmark Passed. The Chain Still Failed. Pull up whatever evaluation got your last agent deployment signed off. The number that convinced the room was almost certainly a benchmark score for the model, or a per-task success rate measured with the environment reset between runs. Neither number describes what that same model does across ten or
Running the Experiment Was Never the Hard Part Every move AlphaGo made during a game got graded before it was played. Not at the end, not after the game resolved into a win or a loss. Right then, move by move. Monte Carlo Tree Search runs the position forward thousands of times and hands back a target: this
Read the Paper, Not the Panel A client forwarded me a podcast clip last week and asked one question: is Claude conscious? He'd caught a clip, not the paper. That's normal. Nobody reads seventy pages of interpretability research before a Tuesday call. But it means the gap between what a research team
MCP Went Stateless. The State Didn't Go Anywhere. MCP crossed 400 million monthly SDK downloads in July, four times what it was moving at the start of the year, at almost the exact moment its authors ripped out the part of the protocol that was hardest to run in production. That's exactly what happens when a
It Can Read the Diagram. It Can't Reason About It. Hand a vision-language model a circuit diagram and ask it to name the components. It will get the resistor. It will get the capacitor. Ask it to trace what happens downstream of the switch, and something quietly breaks, in a way that never shows up on the benchmark slide
Hallucination Left the Sentence and Moved Into the Chain Every client conversation about AI governance eventually arrives at the same sentence: "we have a disclaimer in the system prompt." That's a permission slip, not a mitigation. It tells the user the model might be wrong, but it doesn't catch a wrong answer before
Five Questions I Ask Before a Vendor's Benchmark Goes in a Client Recommendation Next time a vendor's deck shows 81% on SWE-bench, don't write the number down yet. Ask what harness produced it. Ask who audited the eval. Ask whether an independent lab has ever reproduced it on the same tasks. If nobody in the room can answer,
Five Places a Document Pipeline Can Fail, and Four of Them Just Got Fixed A document AI pilot usually starts the same way. Someone points a vision-language model at a stack of PDFs, wires it into Azure AI Search or a Bedrock Knowledge Base, and calls the retrieval working because it answers three test questions correctly. Then it meets a real regulatory filing
Fine-Tune or Orchestrate: The Architecture Decision Every Agent Engagement Skips Pull up the last agent engagement I scoped and look for the sentence where I decided whether we were fine-tuning a model or building an orchestration layer around one. Most of the time that sentence isn't there. The decision got made anyway, by default, usually in favor
Escalation to a Human Is Not a Confidence Score Ask most agent frameworks how they decide when to hand a task to a human, and the answer is a threshold on a single number. The model reports a confidence score, something crosses 0.7 or fails to cross it, and the system either proceeds or escalates. That number is
Applause Is Not an Audit When IBM says its new coding tool lifted developer productivity 45 percent, run that number through the same test you'd run a benchmark slide through. Who checked it. Under what conditions. Compared to what baseline. If the answer is "nobody, IBM measured IBM," treat it as
Agent Containment Is Losing the Race to Capability Pick any three AI security writeups from the last month and you'll find the same sentence buried in each one. The agent did exactly what it was told to do, by someone who was never supposed to be talking to it. The instruction came from a public GitHub
A Rule in the Prompt Is a Suggestion Put four rules in an agent's system prompt. Check the book isn't recalled before renewing it. Confirm the library card on the request matches the one on file. Cap the renewal at thirty days. Keep the tone encouraging, even if the customer is rude. Test it
A Bigger Context Window Is Not Working Memory Ask a team running AI agents in production what their context window is, and they answer in seconds. A million tokens, sometimes two. Ask the same team what their agent's memory architecture is, and the room goes quiet. Nobody built one. They bought a bigger window and assumed
