The Tie Check - Refusing to Guess When the Evidence Splits Evenly The problem I built a diagnostic agent for a manufacturing line, meant to look at sensor logs off a stamping press and recommend a root cause when the line started kicking out defective parts. Feed it two hundred seeded incidents, and in forty-five of them I built the vibration-
The Ten-Run Rule - Telling a Real Regression From Ordinary Noise The Ten-Run Rule - Telling a Real Regression From Ordinary Noise The problem I had a refund-eligibility prompt running at production temperature, 0.7, deciding approve, deny, or escalate on fifteen seeded test cases, with a reason attached to each decision. I ran it through five scripted edits, some
The Suppression Test - Verifying What the Model Says Caused It I built a synthetic, non-clinical triage-style decision-support agent and ran it against forty invented patient-case vignettes, no real patient data anywhere in the system. For each case, the agent produced a recommendation and a verbalized chain-of-thought explaining it, something like "recommending urgent review
The Sprint That Was Sized for the Wrong Stage The problem An internal platform team I worked with had a code-review-comment agent that had been stable for months. It drafted PR review comments flagging style and lint issues, a single job, run enough times that everyone trusted its error rate. Leadership wanted it wired into the CI
The Sixty-Day Gap Between a Model Swap and Anyone Noticing The Sixty-Day Gap Between a Model Swap and Anyone Noticing The problem I built ActionAgent to execute internal operations, password resets, access-grant changes, config edits, under a risk-tiered oversight scheme: low-risk actions auto-approve, medium-risk actions get asynchronous human review, high-risk actions need synchronous
The Six-Suitcase Problem - Letting Failure Set Decomposition Depth The Six-Suitcase Problem - Letting Failure Set Decomposition Depth I ran a coding agent through ten tasks against a small seeded repository. Three were easy: rename a variable across two files. Four were moderate: add a validation function. Three were genuinely hard: refactor a multi-file transformation to support a
The Signal That Beats the Label by a Month The Signal That Beats the Label by a Month I built a telecom churn model once against eight thousand synthetic subscribers over a hundred and twenty simulated days, scoring on features like average monthly data usage and support tickets in the trailing thirty days. On day forty, a new discounted
The Shared ID - One Lookup Instead of Five Disconnected Logs I built a five-step code review pipeline to see how bad log correlation actually gets once an agent chain has a revision loop in it. A planner agent breaks down the task, a retriever pulls the relevant files, a code-generator writes the diff, a test-runner checks it,
The Self-Graded Node - Finding a Bad Step Through Arithmetic I ran a four-step marketing-copy pipeline against twenty synthetic campaign briefs: a brief-intake agent, an outline agent, a draft agent, and a compliance-check-and-polish agent, graded only at the end against a four-part rubric covering accuracy, tone, compliance, and brand fit. I planted a
The Second Reviewer Problem - Why Adding an Agent Usually Adds Nothing The problem I built a financial-report QA workflow around a synthetic quarterly report I seeded with fifteen planted errors: misstated figures, stale footnote references, the kind of mistake that actually shows up in a real filing when nobody's looking closely enough. An Analyst agent reviews the report
The Schema the Model Cannot Violate The Schema the Model Cannot Violate The problem I built a scheduling agent for a clinic that takes a request like "can I move my Tuesday checkup with Dr. Lee to Thursday afternoon" and turns it into a structured booking action: an action type, a date, a time,
The Schema You Add After the Test Fails The Schema You Add After the Test Fails The problem I built a meeting-notes tool that extracts action items from transcripts, and the first version I tried jumped straight to a fully structured output. A strict JSON schema, owner as a typed enum that had to match an exact
The Review Queue Sets the Speed Limit on Generation The problem I ran a marketing content-generation pipeline through a scripted fourteen-day trace where the team upgrades to a faster, cheaper generation model partway through and scales daily output from twenty drafts to two hundred. The human review team's capacity doesn't move an inch
The Retry Gate - Telling a Bug From a Blip Before You Try Again The Retry Gate - Telling a Bug From a Blip Before You Try Again The problem I built a small travel-booking agent to run sixty synthetic flight-booking requests through a mocked pricing-and-booking API, because I wanted to watch a specific failure happen under conditions I controlled instead
The Rate Is the Tell - A Kill Switch That Doesn't Ask Nicely The Rate Is the Tell - A Kill Switch That Doesn't Ask Nicely The problem I built a research agent that loops between an LLM call and a search tool, re-querying and re-summarizing a topic until it decides on its own that it's done. Into
The Pre-Call Sort - Routing Sensitive Queries Before They Ever Dispatch The Pre-Call Sort - Routing Sensitive Queries Before They Ever Dispatch The problem Picture a nurse at a clinic network opening the same chat window twice in one shift. The first question is harmless: what's the max daily dose of a common pediatric medication. The second names a
The Persona Frame - Replacing Forty-Seven Rules With One Voice The Persona Frame - Replacing Forty-Seven Rules With One Voice The problem A retail bank's complaint-response assistant had accumulated forty-seven rules over eighteen months. Never say "sorry for the inconvenience." Always mention the case number. If the complaint mentions a wire transfer, cite Reg
The Outdated Fix That Kept Beating Its Own Replacement The Outdated Fix That Kept Beating Its Own Replacement I built a customer-support triage agent once that routed incoming tickets to a suggested fix by searching a "knowledge graph" built from five hundred resolved support tickets. I put that word in quotes for a reason, and the
The Orchestration Prenup - A Framework Choice You Can Still Walk Away From The Orchestration Prenup - A Framework Choice You Can Still Walk Away From The problem I've watched the same six weeks play out on an internal ticket-triage feature more than once. Week one, the requirement is narrow: classify an incoming ticket and route it to a queue, one
The Obligation Nobody Wired In - Tagging Compliance at the Source Boundary The Obligation Nobody Wired In - Tagging Compliance at the Source Boundary The problem I built a legal-research assistant for a small healthcare provider that retrieved from three sources, each carrying a genuinely different compliance obligation. A case-law citation database required a citation whenever any of it got used.
The Ninety-Second Token - Minting Credentials That Die With the Action The problem I built an agent to handle the kind of SharePoint housekeeping nobody wants to do by hand: move finished marketing assets into an archive folder, rename files to match a naming convention, share a link with an outside agency contact once a campaign wraps. Routine work, low judgment
The Nightly Job That Reported Success on the Two Days It Shouldn't Have The Nightly Job That Reported Success on the Two Days It Shouldn't Have I built a retail demand-forecasting pipeline once that ingested daily point-of-sale transactions for fifty SKUs across ten stores, cleaned and deduped them into a silver layer, and aggregated them into gold feature
The Nightly Batch Job That Ate the Real-Time Queue The Nightly Batch Job That Ate the Real-Time Queue I ran a dynamic-pricing model for a ride-hailing scenario once through a single synchronous endpoint, and I asked that one endpoint to do two genuinely different jobs. One was real-time price quotes for individual ride requests, needing
The Marker That Has to Survive Generation - Keeping Every Claim Attached to Its Source The Marker That Has to Survive Generation - Keeping Every Claim Attached to Its Source The problem I built a market-research brief assistant for a small analyst team that pulled from three sources into a single answer: a licensed analyst-report database whose license terms required attribution whenever any of
The Ledger That Ends the Keep-or-Cut Argument The Ledger That Ends the Keep-or-Cut Argument The problem I built KBAssist to answer policy and pricing questions out of an internal knowledge base, and shipped it with a fact-level validity gate, because retrieval was measured citing expired facts, superseded pricing tiers, lapsed policy terms, at a
