Two Hundred Tools, Five Shown at a Time

The problem

Every framework's tool-calling tutorial uses about five tools, bound into one system prompt, and it works every time. The test just never gets big enough to fail. Nobody ships five tools. Production systems connect dozens of MCP servers, each one written by a different team, each contributing its own batch of schemas, and the ordinary way to wire that up is to bind the whole catalog into context on every single turn, whether the current request needs three of those tools or all of them.

I wanted to see what that actually costs, not just assert it, so I built two versions of the same internal helpdesk copilot and ran them against the identical workload. The setup: a synthetic catalog of 200 actions, things like password resets, VPN provisioning, license assignment, and mailbox delegation, spread across 40 fake internal systems, plus 50 seeded employee tickets, each one tagged with the one tool that actually answers it. The flat version binds all 200 tool schemas into context on every ticket, no filtering, no exceptions. I swept catalog size at 10, 50, 100, and 200 tools and tracked two numbers: how often the model called the correct tool, and how many prompt tokens got spent on tool definitions before it even reached the ticket text.

The curve is exactly as ugly as you'd expect once you see it plotted instead of just describing it. Accuracy degrades as the catalog grows, because the model is choosing among dozens of near-duplicate schemas crowded into the same context window instead of a handful of genuinely distinct options. The token spend is the part that surprised me even though I'd read the industry numbers going in. Connecting four MCP servers can burn roughly 67,000 tokens before a user types a single prompt. One measured production session spent 143,000 of a 200,000-token window on tool schema alone, before any real work started. Cursor caps sessions at 40 tools specifically because of this, and Claude Code's own selection accuracy measurably degrades past 50 bound tools. None of that is a hypothetical ceiling. It's what happens when you take the five-tool demo and let it grow up.

The pattern

The fix leaves the model untouched and changes only what gets shown to it: keep the full tool catalog out of context by default, and load only a task-relevant subset, on demand, for the current turn. I built a second version of the same helpdesk copilot to test this directly: before the tool-calling step, a search_tools call embeds the ticket text and retrieves the top five candidate schemas from a vector index built over all 200 tool descriptions. Only those five get bound to the call that actually picks a tool. Same 200-tool catalog, same 50 tickets, same everything except that one step inserted ahead of the model's decision.

Run the identical sweep across catalog sizes 10 through 200 and the two lines split apart on the same chart. The flat version's accuracy falls as the catalog grows. The search version stays flat, because the model never actually reasons over the full catalog at any size. It reasons over whatever the search step handed it, and that stays a fixed, small number regardless of how large the underlying system gets. I set the bar at retrieving the ground-truth tool among the top five for at least 95% of tickets, which is the number that has to hold for this to actually work rather than just look good on a chart with two lines.

I didn't invent the retrieve-then-bind mechanism myself, and I'd rather say where it came from than let it read like an original discovery. RAG-MCP's own benchmark reports 43.13% tool-selection accuracy with retrieval against a 13.62% no-retrieval baseline, and that's the paper this whole approach traces back to. What I find more convincing than my own toy project or that paper is what happened afterward: Claude Code shipped the identical mechanism natively, starting at version 2.1.7, automatically switching to deferred loading once active tool descriptions cross 10% of the context budget, and reporting roughly an 85% token reduction in production use. A vendor arriving at the same fix independently, at scale, inside a tool millions of people use every day, tells me more than a benchmark number does.

Design considerations

The mechanism only works as well as the descriptions it's retrieving over. Feed it two tools that describe overlapping functions in similar language, and the search step can retrieve the wrong five, or hand back both of two redundant options and leave the model to guess between near-identical choices at the narrow step instead of the wide one. That's a curation problem sitting one layer upstream of retrieval, and no amount of tuning the search step fixes a catalog where three tools do the same thing under three different names. I think of retrieval and catalog hygiene as two separate jobs that happen to sit next to each other in the pipeline.

I also haven't settled whether a fixed top-five is even the right shape. Some retrieval research points toward a variable depth per query outperforming a fixed K, wider for an ambiguous request, narrower for a specific one, but I've only got two sources leaning that direction, not the three independent signals I'd want before changing the design. I'm running with five for now because it's measurable and it's working. Whether it's the actual ceiling is still an open question for me.

The place this doesn't fit is the same five-tool catalog the tutorials use. If your entire system genuinely has five or ten tools that a human could eyeball in one glance, adding a retrieval step ahead of tool selection is complexity with nothing to buy back. The whole argument for deferred disclosure is that a fixed retrieval cost stops scaling with catalog size while a bound-everything approach keeps getting worse. Below a certain catalog size, both approaches cost about the same, and the simpler one wins by default.

Catalog size is only one source of selection failure, and the edge of what this actually fixes matters as much as the result. Two independently authored tools can share a name and route a call to the wrong one, entirely apart from whether the catalog is large or small. A model can attend to the correct tool and still name the wrong one at the final decision step, a different failure sitting somewhere in the decoder's readout rather than in what it was shown. Retrieval narrows what the model sees, but it doesn't touch what happens once the model is looking at the right five and still picks wrong. I'd rather name that gap than let the flat-versus-search chart imply this solves tool selection in general. It solves the specific, measurable failure that comes from a catalog too big to fit in a model's attention at once, and that failure alone shows up often enough to earn fixing first.