ResolveMesh: Evidence-First Agent Tool Discovery

ResolveMesh answers a narrow question: what should an AI agent do when its installed tools cannot finish the job?

The easy answer is a directory. The useful answer is a resolver that can understand the missing capability, rank compatible options across MCP, A2A, and REST, explain the ranking, and stay honest when there is not enough evidence.

I designed and built the product, ranking model, evaluation harness, discovery study, and public experience. That included the data contracts, normalization layer, deterministic scoring, benchmark cases, CI gate, and the guardrails that keep discovery read-only.

Discovery cannot become execution by accident. ResolveMesh may inspect public metadata and health signals, but it does not invoke a tool, ask for credentials, or claim a provider is trustworthy because a description sounds polished.

That constraint forced a useful separation:

• Discovery answers whether a capability appears relevant and compatible.
• Evidence records what is actually known about identity, health, and behavior.
• The requesting agent or human keeps control of connection and execution.

Every source is normalized into a capability record with explicit inputs, outputs, transport, authentication requirements, and evidence. The ranker then scores query-to-capability fit using deterministic features rather than a hidden model judgment.

A result includes its score breakdown. If a new tool lacks enough observations, the system returns INSUFFICIENT_DATA instead of manufacturing confidence. That choice makes the product less impressive in a demo and far more useful in a real agent workflow.

The resolution path is:

1. Translate the unfinished job into a capability request.
2. Normalize candidates from supported discovery surfaces.
3. Filter for protocol, permission, and input compatibility.
4. Rank the remaining candidates with explainable features.
5. Return the best options with evidence and uncertainty attached.
6. Record the outcome as a privacy-safe signal after the caller decides what to do.

I built a 150-intent benchmark that covers exact requests, paraphrases, ambiguous wording, and close-but-wrong capabilities. It runs in CI so changes to ranking cannot quietly improve one demo while degrading the rest of the catalog.

The public methodology reports these bounded results. The aggregate scores do not, by themselves, explain whether a failed case came from retrieval, ranking, or its expected label. That diagnosis requires inspecting the individual benchmark cases.

These are results from the case study's fixed evaluation set, not production success rates. Inspecting public metadata is also not evidence that any of the 823 discovered tools executed correctly.

The public product does not fill its production catalog with tools just because they were discoverable. Registry identity, provider attestations bound to exact tool definitions, non-invasive health checks, and manual review are separate evidence layers. Until that evidence clears, the honest catalog state can be empty.

That is not a growth-optimized choice. It is the right systems choice for infrastructure that agents may eventually rely on.

ResolveMesh is a small product with a systems-shaped problem. It required protocol normalization, ranking, evaluation, privacy boundaries, failure semantics, and a user experience that explains uncertainty without burying the answer.

The product and the evaluation answer different questions. One explains a discovery result; the other tests ranking against a fixed set of intents. Keeping that boundary visible is the kind of AI product work I care about: a useful decision, a testable method, and no more confidence than the evidence supports.