The Wire Transfer and the Taxi Receipt - Tiering Autonomy Instead of Switching It The Wire Transfer and the Taxi Receipt - Tiering Autonomy Instead of Switching It The problem I built an expense-report automation agent to handle a seeded set of one hundred fifty reports, and I deliberately made that set span a genuinely wide range: a twelve-dollar taxi receipt sitting right
The Webcam Light Question Nobody Scripted For The Webcam Light Question Nobody Scripted For The problem I built an IT helpdesk chatbot around three defined workflows: password reset, VPN access, hardware request. Each one is a scripted, multi-turn conversation with slots to fill, name, employee ID, the specific system involved, the kind of structured path that
The View That Forgets SSN - Withholding Sensitive Columns at the Source The View That Forgets SSN - Withholding Sensitive Columns at the Source The problem I built an internal HR policy assistant for a fictional manufacturing company, one I called AskHR, and gave it a job most internal assistants end up doing sooner or later: answer general policy questions and, when someone
The Untagged Aside - Why Source Identity Has to Be Captured at Ingestion The Untagged Aside - Why Source Identity Has to Be Captured at Ingestion The problem I built a support copilot once, on purpose, so I could watch it fail. It pulled from three sources indexed into the same Chroma collection: CRM account notes, a ticketing system export, and a compliance handbook,
The Tone-Polish Tax - Gating a New Step by What It Actually Buys The Tone-Polish Tax - Gating a New Step by What It Actually Buys The problem I ran a customer-support reply-drafting pipeline as a stable, boring, two-step process: draft a reply, send it. Then I did what every team eventually does with a pipeline that's working
The Tokenize-and-Restore Sandwich - A Proxy That Never Sees the Real Value The Tokenize-and-Restore Sandwich - A Proxy That Never Sees the Real Value The problem I built a client-service chatbot for a fictional wealth-management firm, one I called PortfolioAssist, and pointed it at the kind of question relationship managers actually field: what's the balance on account
The Token Scoped to Exactly One Employee ID The problem I built an internal agent that lets employees check their own pay stubs, PTO balance, and direct deposit details, and update direct deposit on request. Call it HRBot. The API behind it calls a records service using a single standing service-account credential with broad read and write
The Time-of-Decision Log - Scoring Judgment Before the Outcome Arrives The problem I built an inventory-reorder agent that recommended reorder quantities across eighty retail SKUs, tracked over a twelve-week trace. In week eight, I built in a demand shock: an unforeseeable disruption that hit ten of those SKUs hard. The reorder decisions made on those ten SKUs back
The Tie Check - Refusing to Guess When the Evidence Splits Evenly The problem I built a diagnostic agent for a manufacturing line, meant to look at sensor logs off a stamping press and recommend a root cause when the line started kicking out defective parts. Feed it two hundred seeded incidents, and in forty-five of them I built the vibration-
The Ten-Run Rule - Telling a Real Regression From Ordinary Noise The Ten-Run Rule - Telling a Real Regression From Ordinary Noise The problem I had a refund-eligibility prompt running at production temperature, 0.7, deciding approve, deny, or escalate on fifteen seeded test cases, with a reason attached to each decision. I ran it through five scripted edits, some
The Suppression Test - Verifying What the Model Says Caused It I built a synthetic, non-clinical triage-style decision-support agent and ran it against forty invented patient-case vignettes, no real patient data anywhere in the system. For each case, the agent produced a recommendation and a verbalized chain-of-thought explaining it, something like "recommending urgent review
The Sprint That Was Sized for the Wrong Stage The problem An internal platform team I worked with had a code-review-comment agent that had been stable for months. It drafted PR review comments flagging style and lint issues, a single job, run enough times that everyone trusted its error rate. Leadership wanted it wired into the CI
The Sixty-Day Gap Between a Model Swap and Anyone Noticing The Sixty-Day Gap Between a Model Swap and Anyone Noticing The problem I built ActionAgent to execute internal operations, password resets, access-grant changes, config edits, under a risk-tiered oversight scheme: low-risk actions auto-approve, medium-risk actions get asynchronous human review, high-risk actions need synchronous
The Six-Suitcase Problem - Letting Failure Set Decomposition Depth The Six-Suitcase Problem - Letting Failure Set Decomposition Depth I ran a coding agent through ten tasks against a small seeded repository. Three were easy: rename a variable across two files. Four were moderate: add a validation function. Three were genuinely hard: refactor a multi-file transformation to support a
The Signal That Beats the Label by a Month The Signal That Beats the Label by a Month I built a telecom churn model once against eight thousand synthetic subscribers over a hundred and twenty simulated days, scoring on features like average monthly data usage and support tickets in the trailing thirty days. On day forty, a new discounted
The Shared ID - One Lookup Instead of Five Disconnected Logs I built a five-step code review pipeline to see how bad log correlation actually gets once an agent chain has a revision loop in it. A planner agent breaks down the task, a retriever pulls the relevant files, a code-generator writes the diff, a test-runner checks it,
The Self-Graded Node - Finding a Bad Step Through Arithmetic I ran a four-step marketing-copy pipeline against twenty synthetic campaign briefs: a brief-intake agent, an outline agent, a draft agent, and a compliance-check-and-polish agent, graded only at the end against a four-part rubric covering accuracy, tone, compliance, and brand fit. I planted a
The Second Reviewer Problem - Why Adding an Agent Usually Adds Nothing The problem I built a financial-report QA workflow around a synthetic quarterly report I seeded with fifteen planted errors: misstated figures, stale footnote references, the kind of mistake that actually shows up in a real filing when nobody's looking closely enough. An Analyst agent reviews the report
The Schema the Model Cannot Violate The Schema the Model Cannot Violate The problem I built a scheduling agent for a clinic that takes a request like "can I move my Tuesday checkup with Dr. Lee to Thursday afternoon" and turns it into a structured booking action: an action type, a date, a time,
The Schema You Add After the Test Fails The Schema You Add After the Test Fails The problem I built a meeting-notes tool that extracts action items from transcripts, and the first version I tried jumped straight to a fully structured output. A strict JSON schema, owner as a typed enum that had to match an exact
The Review Queue Sets the Speed Limit on Generation The problem I ran a marketing content-generation pipeline through a scripted fourteen-day trace where the team upgrades to a faster, cheaper generation model partway through and scales daily output from twenty drafts to two hundred. The human review team's capacity doesn't move an inch
The Retry Gate - Telling a Bug From a Blip Before You Try Again The Retry Gate - Telling a Bug From a Blip Before You Try Again The problem I built a small travel-booking agent to run sixty synthetic flight-booking requests through a mocked pricing-and-booking API, because I wanted to watch a specific failure happen under conditions I controlled instead
The Rate Is the Tell - A Kill Switch That Doesn't Ask Nicely The Rate Is the Tell - A Kill Switch That Doesn't Ask Nicely The problem I built a research agent that loops between an LLM call and a search tool, re-querying and re-summarizing a topic until it decides on its own that it's done. Into
The Pre-Call Sort - Routing Sensitive Queries Before They Ever Dispatch The Pre-Call Sort - Routing Sensitive Queries Before They Ever Dispatch The problem Picture a nurse at a clinic network opening the same chat window twice in one shift. The first question is harmless: what's the max daily dose of a common pediatric medication. The second names a
The Persona Frame - Replacing Forty-Seven Rules With One Voice The Persona Frame - Replacing Forty-Seven Rules With One Voice The problem A retail bank's complaint-response assistant had accumulated forty-seven rules over eighteen months. Never say "sorry for the inconvenience." Always mention the case number. If the complaint mentions a wire transfer, cite Reg
