The Elbow in the Curve - Where Extra Reasoning Stops Paying For Itself

The problem

I had an internal agent estimating engineering effort from a written requirements doc, the kind of task where the model has to read a paragraph of ambiguous scope and turn it into a number of days. Somewhere early in building it, someone set the reasoning budget to 8,000 thinking tokens, the largest tier the model offered, with the justification you hear on every team that's ever made this choice: to be safe. Nobody measured whether 8,000 tokens bought anything over a smaller number. It felt like the responsible setting, so it shipped as the responsible setting, and it stayed that way for months.

I finally ran the eval I should have run before ever picking a number. Fifty requirements-estimation cases with graded ground truth, the same fifty cases run at six different budget tiers: zero tokens, 256, 512, 1024, 2048, 4096, and the full 8,000. Accuracy at zero tokens was predictably weak. Accuracy climbed through the early tiers the way you'd expect. And then it stopped climbing, well before 4096, sitting essentially flat from there through 8000. Cost and latency did not stop climbing. They scaled straight through to 8000 exactly the way you'd expect a linear cost model to scale, with nothing about the model's own behavior suggesting it needed the extra room to think.

What that means in plain terms is that every production request through this agent was paying for roughly the last four thousand tokens of reasoning budget and getting nothing back for it. Not a marginal improvement, not a small one. Nothing measurable at all. The number had been chosen by intuition, "to be safe" standing in for a guess nobody had checked, and the guess turned out to be paying full price for reasoning that had already stopped buying anything three tiers earlier.

The pattern

The fix is the sweep itself, formalized as the thing you run before picking a production number instead of the thing you run once, out of curiosity, after the fact. Run the same eval set across a defined range of budget tiers. Plot accuracy against budget. Find the elbow: the first tier where the accuracy gain over the previous tier drops below a small threshold, one percent in my case. Pin production to the tier just above that elbow, not to the largest available number and not to whatever round figure feels conservative.

The calculation is mechanical once you have the sweep data, which is the whole point. You're not eyeballing a chart and picking a spot that looks like a plateau. You're computing, tier by tier, whether the marginal accuracy gain justifies the marginal cost, and stopping at the first tier where it doesn't. In my case that put production at a tier well below the fixed 8,000 the team had been running, with accuracy landing within two percent of the fixed-8000 number while cost and latency dropped by half or more. The accuracy I gave up was smaller than the accuracy variance you'd see from re-running the same eval twice.

What makes this different from just picking a smaller number and hoping is that the ceiling is read directly off measured behavior on your actual task, not borrowed from a blog post about someone else's task or a round number that sounded reasonable in a planning meeting. A different task, a different model, a different eval set will have a different elbow, sometimes earlier, sometimes later. The mechanism is the sweep and the elbow calculation. The specific number it produces is not the thing to remember from this.

Design considerations

The threshold you use to define the elbow is doing real work, and it's a genuine judgment call rather than something the data hands you automatically. I used one percent, meaning the first tier where the accuracy gain over the previous tier drops below one percentage point gets treated as the plateau. Set that threshold looser and you'll stop climbing earlier, saving more cost at the price of leaving some real accuracy on the table. Set it tighter and you'll chase small, possibly noisy gains further up the tier list, past the point where those gains still justify the cost. There's no universal right answer here either, and I'd calibrate it against how expensive a wrong answer actually is downstream. An estimation agent that's wrong by half a day here and there is a very different cost profile than a medical-triage agent where the last percentage point of accuracy is the point that matters most.

This pattern answers a narrower question than it might look like it answers, and the narrowness matters. It tells you how much reasoning budget to allocate to a single, roughly uniform task type. It says nothing about a workload made up of a mix of easy and hard requests. There, the right move is a cheap classifier routing each request to a tier matched to its actual difficulty, instead of one calibrated ceiling applied everywhere. Those are two different problems that happen to share a sweep-and-measure instinct. Confusing them gets you a single elbow-calibrated ceiling applied to a workload where most requests never needed anywhere near that ceiling in the first place, and a handful of genuinely hard requests that needed more than the ceiling gives them.

There's also a limit worth naming before someone runs into it the hard way. Research on reasoning models has found something sharper than a simple plateau in some cases: past a certain problem-complexity threshold, accuracy can fall off outright, not just stop improving, regardless of how much additional budget gets thrown at it. Apple's work on this, published under the title "The Illusion of Thinking," documents that collapse directly. That complicates the assumption underneath elbow-finding in a specific way. An elbow measured against one band of task difficulty isn't guaranteed to hold for a harder band of the same task type. If your production traffic drifts toward harder cases than the fifty you calibrated against, the elbow you measured on the easier distribution may not be the elbow that actually applies anymore, and the plateau you found could be sitting on the near side of a cliff you never tested.

A separate finding belongs alongside that one: chain-of-thought reasoning layered explicitly on top of a reasoning model's own internal process has been shown, in some cases, to degrade the model's adherence to output format and other explicit constraints, which is close to the opposite of what explicit step-by-step prompting classically bought you on non-reasoning models. The sweep-and-calibrate mechanism still holds up. What changes is the assumption sitting quietly underneath it, that more explicit reasoning structure is neutral at worst, which I'm no longer willing to take for granted. I still run the sweep. I just treat the number it hands me as a fact about this task, on this eval set, measured on this particular day, and nothing broader than that.