The Closest Match - Blocking a Duplicate Skill Before It Ships

Picture a platform team running a DevOps agent with skills like restart-service, rotate-credential, and provision-vm. Different engineers write these in isolation from each other, and nothing at publish time checks whether a capability already exists under a different name. restart-postgres-service ends up sitting next to restart-database-service. rotate-api-key sits next to rotate-service-credential. Neither engineer was careless. Neither one could see the other's work from where they sat.

The problem

I ran a version of this with thirty skills seeded into a registry, eight of them deliberate near-duplicate pairs. In every pair I gave one skill something the other lacked: a missing flag, a narrower scope, logic that fell behind a service topology change nobody had gone back to update. That's the realistic case. A duplicate skill in production almost never looks like a copy-paste. It looks like two reasonable engineers solving the same problem six months apart, with the newer one aware of something the older one predates.

The part that actually does the damage is what happens at invocation time. Skill selection is a retrieval problem before it's anything else: embed the incoming task, cosine-match it against every published skill's description, invoke whichever wins. A duplicate pair splits that match. Sometimes the agent grabs the current, correct sibling. Sometimes it grabs the stale one, and it does so with the exact same confidence either way, because nothing in the match score tells it which skill actually reflects how the infrastructure looks today. Across sixty scripted task invocations against that seeded registry, I saw collisions on roughly a quarter of them.

A quarter isn't a rounding error. It means one in four operational tasks routed to a skill carrying a known gap, and the output looked completely normal on the way out. That's what makes this failure mode expensive to catch. A crashed skill gets noticed the same day. A skill that runs restart-database-service instead of restart-postgres-service, using logic that predates a schema change, executes cleanly, returns a clean status, and leaves the actual problem unresolved until someone traces a recurring incident back to which skill handled it.

The pattern

The fix runs earlier than review. It runs at the moment a skill is submitted, before it's ever inserted into the registry at all. A new submission gets its description and signature embedded, then compared by cosine similarity against every existing entry. Above a threshold of 0.85, publish is blocked, and the author is shown the closest match instead of a green light.

The threshold does specific work beyond just catching lookalikes. It's calibrated to favor keeping many narrow, single-purpose skills over one broad skill trying to cover several cases. A genuinely distinct narrow skill, restart-postgres-service scoped only to that one service, still clears the gate even when its description shares vocabulary with a dozen other restart skills. A true duplicate, one describing the same action with the same scope, doesn't. Replayed against the eight seeded pairs I built, the gate caught seven.

That seventh pair matters as much as the seven it caught. I let one duplicate through on purpose, because a similarity gate is only a probabilistic filter operating on how an author chose to describe their own skill. It offers no guarantee about what the skill actually does underneath. An author who writes a description different enough from the original, whether on purpose or by accident, can still get a functionally identical skill past a 0.85 threshold. Cosine similarity catches shared vocabulary. It has no way to catch shared behavior wrapped in different words.

That gap is where a separate strand of research picks up. SkillResolve-Bench, an arXiv paper measuring what it calls same-capability ambiguity in agent skill retrieval, studies exactly this: two skills that are legitimately distinct in how they describe themselves but functionally overlapping enough that an agent can't confidently choose between them at run time. That's a different failure from the one this gate closes. This gate stops a duplicate from being admitted to the registry in the first place. The retrieval-ambiguity problem shows up after admission, between skills that were never meant to be duplicates and technically aren't, and it needs a different fix running at a different point in the pipeline.

Design considerations

The threshold itself is the first thing I want to be honest about. 0.85 isn't a number I'd defend as universally correct. Set it too low and the gate starts blocking skills that are merely adjacent, penalizing an engineer for describing a narrow, legitimate capability in language that happens to overlap with something else. Set it too high and the gate stops catching anything but near-verbatim copies, which is close to not having a gate at all. The right number depends on how varied an organization's skill descriptions actually are and how much tolerance it has for occasionally sending an author back to revise a description that was fine all along.

The bigger limitation is structural rather than numerical. This gate operates entirely on the description an author chooses to write. It has no visibility into the skill's actual runtime behavior. Two skills can describe themselves in completely different terms and still do the identical thing under the hood, and nothing about an embedding comparison over natural-language descriptions will ever catch that. Closing that gap means comparing behavior instead of language: sampling actual outputs across a shared set of test inputs and looking for convergent results, which is a heavier and slower check than anything that can run at publish time without becoming a bottleneck.

I've also found the gate does its best work when it's paired with a habit rather than left to run alone. Showing an author the closest match at the moment of a blocked publish only helps if extending or versioning the existing skill is genuinely easier than starting over. If the path to modify someone else's skill is buried behind unfamiliar code or an unclear ownership model, an author blocked at 0.85 will sometimes just tweak the description until it clears the threshold, which defeats the entire point. The gate assumes a registry where extending what exists is genuinely the path of least resistance.