Tool Contracts: Why Agents Skip Your Tools

The eval run that started this document looked healthy at first glance. I was running gold-set evaluations against a production-grade agent for an enterprise proof-of-value: a curated set of tasks, each with an expected sequence of tool calls, replayed against the agent and scored automatically. The reasoning traces were genuinely good. The model decomposed problems correctly, planned sensibly, and wrote clear final answers. And it was skipping tools it was required to call on more than 20 percent of the cases. Not failing the calls. Not calling them with bad arguments. Just quietly answering from its own head as if the tool did not exist.

My first instinct was the usual one: swap the model, then rewrite the system prompt. Neither moved the number much. A more expensive model shaved a few points off the skip rate at 13 times the cost. Prompt surgery (“you MUST use the available tools”) produced the opposite failure, the agent started calling tools it did not need. What finally fixed it was boring: I rewrote the tool descriptions. I stated when each tool should be used, when it should not, and what it returned. I moved constraints out of prose and into the schema. The skip rate dropped to low single digits. Same model, same system prompt, same eval set.

That result repeated across every agent I evaluated afterward, and it changed how I think about tool-calling failures. The dominant cause is not model weakness. It is underspecified tool contracts: vague descriptions, overlapping tools, schemas that accept garbage, constraints that live only in a developer’s head. Teams write tool specs for developers who can read the source. The actual reader is a probabilistic model that sees exactly the JSON you registered and nothing else. This guide is what I now do about it, including the deterministic linter I built to make it enforceable in CI.

A taxonomy of tool-call failures

When I score eval transcripts, every tool-related failure lands in one of five buckets. The useful discipline is tracing each bucket back to the spec defect that caused it, because that tells you what to fix. Blaming “the model” tells you nothing actionable.

Failure mode	What it looks like in the transcript	Root cause in the spec	How to detect it in evals
Skip	No tool call where one was required; the agent answers from parametric knowledge or guesses	Description doesn’t say when the tool applies, or doesn’t establish that its data is live and authoritative	Golden trace expects tool X; transcript contains no call to X. Report skips per tool, not just overall
Wrong tool	The agent calls a plausible neighbor: search_documents instead of search_customer_accounts	Overlapping tools with near-identical descriptions and no disambiguation clause	Expected tool A, observed tool B. Cluster confusions in a matrix; overlapping pairs light up immediately
Malformed arguments	Call fails schema validation or the API returns 400; the agent retries blindly or gives up	Schema is too loose: untyped params, no required array, free-string fields where formats exist	Count schema-validation failures and 4xx responses per tool in traces
Hallucinated arguments	Arguments are syntactically valid but invented: an account ID that appears nowhere in the conversation	Spec is missing constraints and examples, so the model fills gaps with plausible-looking values	Diff argument values against values actually present in the conversation or prior tool results
Premature answer	The agent produces a confident final answer without calling the tools that ground it	Description doesn’t tell the model its knowledge is stale here (“account data changes daily; call this even if you think you know”)	Final answer present, required calls absent; check whether answer claims are traceable to tool output

Two things about this table earned their place through repetition. First, skip is the most common failure and the least visible one. A malformed call throws an error you can alert on. A skip produces a fluent, confident, wrong answer, and nothing in your logs flags it. You only see skips if your evals encode which tool calls were expected. Second, four of the five buckets are spec defects. Only wrong-tool selection under a genuinely well-separated toolset is plausibly a model limitation, and even there the spec is usually an accomplice.

Your tool description is a prompt. It is injected into the context window on every single turn. Most teams spend weeks iterating the system prompt and zero minutes on the fifteen prompts they shipped alongside it.

What a tool contract actually is

I use the word contract deliberately. A tool sits on the boundary between two very different systems: a deterministic implementation that does exactly what its code says, and a probabilistic caller that decides whether and how to invoke it based purely on the text and schema you registered. The model cannot read your source. It cannot ask a colleague. The spec is the entire interface, and it has six parts:

1. Name. The strongest single signal for tool selection. Models pattern-match on names before they read a word of description.
2. Description. When to use the tool, when not to, and what it returns. This is documentation where the reader is a model, which changes the writing rules.
3. Schema. The machine-checkable shape of the input: types, required fields, enums, formats, ranges.
4. Constraints. Everything the schema cannot express structurally, stated explicitly: ordering dependencies (“call search_customer_accounts first to obtain the account_id”), rate expectations, data freshness.
5. Examples. At least one concrete, valid invocation. Models generalize from examples far more reliably than from abstract descriptions.
6. Error semantics. What failure looks like and what the caller should do about it. An agent that knows STATEMENT_NOT_READY means “stop and tell the user” behaves differently from one staring at a stack trace.

The single highest-leverage habit is writing behavior-oriented descriptions instead of implementation descriptions. An implementation description says what the code does: “Queries the customer database via the accounts service.” A behavior description says what the caller should do: “Searches customer accounts by name, email, or account number. Use this when the user asks about a specific customer and you do not already have their account ID.” The first is accurate and useless to the model. The second reads like a routing rule, because that is exactly how the model consumes it. Every description I ship now contains the literal phrase “Use this when” and, for any tool with a plausible neighbor, “Do not use this for.”

The rules that measurably matter

These are the rules I apply when reviewing tool specs, ordered roughly by how much each one moved skip and error rates in my evals. None of this is exotic. All of it is regularly absent from production MCP servers.

One job per tool, and no overlapping neighbors

If two tools could plausibly handle the same request, the model will split its calls between them, and your evals will show a smeared confusion pattern rather than a clean failure. The worst offenders I see are pairs like get_customer / fetch_customer_details / lookup_client that grew organically from different teams. Merge them, or give each a sharply distinct boundary and cross-reference the other by name in the description (“For transaction history, use list_account_transactions instead”). A model deciding between tools is doing a classification task; your job is to make the classes separable.

Say when NOT to use it

Positive descriptions define a fuzzy region. Negative clauses define its edges. The description that cut skip rates in my opening example did not just say what the tool did; it said “Use this even if you believe you already know the answer; holdings data changes daily.” That one sentence attacks skip and premature-answer failures directly, because the model’s default assumption is that its parametric knowledge is adequate. Conversely, “Do not use this for X” clauses attack wrong-tool selection. Both directions matter.

Put constraints in the schema, not the prose

Prose constraints are suggestions. Schema constraints are checkable, and models respect them more reliably because they are structural. Every time you write “should be a valid date” in a description, you could have written a pattern or format in the schema. Here is a before and after from a real review, generalized:

{
  "name": "get_statement",
  "description": "Gets a statement. Date should be in the right format and not in the future.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "date": { "type": "string" },
      "acct": { "type": "string" }
    }
  }
}

Everything wrong here is invisible until an eval run: no required array (so the model may omit both fields), no format for date (I observed “last month”, “2026/05/01”, and epoch millis in traces), and acct gives the model nothing to ground on, which is where hallucinated account IDs come from. The rewrite:

{
  "name": "get_monthly_statement",
  "description": "Retrieves one closed monthly account statement as a PDF link. Use this after you have an account_id from search_customer_accounts. Do not use this for real-time balances; use get_account_balance. Returns error code STATEMENT_NOT_READY if the requested month is not yet closed; do not retry, tell the user when it will be available.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "account_id": {
        "type": "string",
        "pattern": "^ACC-[0-9]{8}$",
        "description": "Account identifier, e.g. ACC-00417233. Obtain via search_customer_accounts; never guess this value."
      },
      "month": {
        "type": "string",
        "pattern": "^[0-9]{4}-(0[1-9]|1[0-2])$",
        "description": "Statement month as YYYY-MM, e.g. 2026-05. Must be a fully closed month, not the current month."
      }
    },
    "required": ["account_id", "month"],
    "additionalProperties": false
  },
  "annotations": { "readOnlyHint": true }
}

Enums deserve special mention. Any parameter with a finite set of legal values should be an enum, full stop. A free string with “one of: active, dormant, closed” in the description will eventually receive “inactive”, and your backend will do whatever it does with that.

Be honest about required vs optional

Two failure directions. Marking a genuinely required field as optional invites calls that omit it, and the model then learns from the error message, wasting a turn. Marking an optional field as required forces the model to invent a value, which is a hallucination you created by decree. I see the second one constantly in specs generated from backend DTOs where every field was required for the database, not for the caller. The rule: required reflects what the tool needs to do useful work, and every optional field’s description says what happens when it is omitted (“Omit to search all statuses”).

Fail with codes, not prose

Error messages are part of the contract because the agent reads them and decides what to do next. A raw stack trace or a generic “internal error” gives the model two options: retry blindly or apologize. A deterministic error code with a machine-actionable hint gives it a branch to take. The pattern I use: a stable code field, a one-sentence human-readable message, and where applicable a directive (retry_after_seconds, “obtain a valid account_id first”). In evals, this converts a class of dead-end transcripts into recoveries, and it makes failure assertions in your test suite trivial to write.

Name for the caller, not the codebase

Verb-object, snake_case, no internal jargon. search_customer_accounts beats queryCustDB, cst_lkp_v2, and CustomerAccountSearchServiceEndpoint. Internal system names are especially costly: the model has no idea that “Falcon” is your pricing engine, so query_falcon might as well be unlabeled. The name should let a model with zero organizational context predict what the tool does. This sounds cosmetic. It is not; names are the first-pass filter in tool selection, and in confusion matrices I can watch badly named tools lose calls to better named neighbors that do the wrong thing.

Respect the length budget

Description length has a sweet spot. Under about ten words, the tool is underspecified and gets skipped or misused. Past a couple hundred words, the signal drowns: the when-to-use clause is buried in paragraph four and the model has fifteen other tool descriptions competing for attention in the same context window. In my reviews, the descriptions that perform best run three to six sentences: what it does, when to use it, when not to, what it returns, error behavior. If you need more than that, the tool is probably doing too many jobs (see rule one) or the extra material belongs in the schema.

Scoring specs deterministically

After enough of these reviews, I noticed I was applying the same checklist every time, and a checklist you apply by hand is a linter you have not written yet. So I built one: Tool Credit Score, part of Agent Contract Studio, open source and running live at agentic-contract-studio.vercel.app. You paste an MCP tool spec (or import it from GitHub, or discover it from a running MCP server), and it scores every tool 0 to 100 against a deterministic rubric, then auto-fixes what it can.

The rubric has five weighted categories, which map directly onto the failure taxonomy above:

Category	Weight	What it checks	Failure mode it targets
Description	25%	Present, 10+ words, starts with an action verb, under 200 words	Skip, premature answer
Parameters	25%	Schema is type: object; every param has a type and description; a required array exists and is honest	Malformed and hallucinated arguments
Best practices	20%	MCP annotations (readOnlyHint, destructiveHint, idempotentHint); additionalProperties: false; description mentions returns and errors	Wrong tool, dead-end error handling
Naming	15%	snake_case, verb prefix, under 64 characters	Wrong tool
Examples	15%	At least one valid example invocation	Malformed and hallucinated arguments

Scores roll up to a letter grade (90+ is an A, below 60 is an F), and the auto-fixer handles the mechanical repairs deterministically: renaming to snake_case with a verb prefix, adding missing parameter types, scaffolding examples and annotations. There is an optional LLM polish pass for description wording, but it is exactly that, optional, local-only, and it never touches parameter names, types, or schema structure. The core pipeline needs no model and no API key.

That last design decision is the point, and it is worth defending because “just have an LLM review the spec” is the obvious alternative. I chose deterministic linting for CI for four reasons. It is reproducible: the same spec gets the same score every run, so a score regression in a pull request means the spec changed, not the judge’s mood. It is fast and free, so you can run it on every commit without a token budget conversation. It is explainable: every deducted point maps to a named rule with a concrete fix, which is what you want in a code review comment. And it cannot be argued with, which matters more than it should; I have watched teams relitigate an LLM judge’s verdict for an hour, and nobody argues with a regex.

The honest caveat: a linter checks form, not meaning. Tool Credit Score will catch a missing required array every time; it cannot tell you that two of your tools semantically overlap, or that your when-to-use clause is factually wrong about your own system. Those defects only surface in evals (measured skip and confusion rates) or in a human review. The linter’s job is to make sure the cheap, mechanical defects never reach that stage, so the expensive review effort goes where it is actually needed.

A 90+ Tool Credit Score does not guarantee agents will use your tools well. A score under 60 nearly guarantees they won’t. Linting is a floor, not a ceiling.

Where MCP servers concentrate these failures

MCP made tools portable, which is genuinely useful, and it also made bad specs portable. Three patterns account for most of the low-scoring servers I have reviewed.

Generated-from-OpenAPI servers. Point a converter at an OpenAPI document and you get an MCP server in minutes: one tool per endpoint, parameter names lifted straight from the backend, descriptions that were written for developers reading API reference docs, if they were written at all. The result is a spec surface shaped like your database, not like your users’ tasks. An agent facing post_v2_accounts_search, get_accounts_by_id, and get_accounts_search_legacy will produce exactly the confusion matrix you would predict. Generated servers are a fine starting point; they are not a shippable contract. Budget a curation pass that merges endpoint-shaped tools into task-shaped ones.

Kitchen-sink servers. A server that wraps an entire SaaS product and exposes 80 tools because the API has 80 capabilities. Every one of those tool definitions is injected into the model’s context on every turn, so you pay for the whole surface in tokens whether or not this conversation needs it, and tool selection accuracy degrades as the candidate set grows, especially when the candidates share a vocabulary. My working budget: keep the tools actually exposed to an agent for a given task under about 20. That number is an observation from my evals, not a law, and better models keep nudging it upward. The direction of the effect has held for every model I have tested.

No progressive disclosure. The fix for the kitchen sink is not deleting capability, it is staging its exposure. Expose a small core set for the common path. For long-tail capability, use a discovery pattern: a search_tools-style tool that returns full contracts on demand, deferred schema loading where your client supports it, or dedicated subagents that each see only their slice of the toolset. MCP’s design supports this: clients choose which tools to surface, and server-side tool filtering by task context is straightforward to implement. The contract discipline still applies at every layer; a discovered tool with a vague description fails identically to a preloaded one, just one turn later.

MCP’s annotations block (readOnlyHint, destructiveHint, idempotentHint, openWorldHint) deserves more use than it gets. Beyond the safety value, annotations let a client apply blanket policy (auto-approve read-only tools, gate destructive ones behind confirmation) without parsing descriptions, and they are one of the cheapest score improvements in the linter.

Close the loop with evals

None of the rules above are worth much as opinions. They are worth a lot as measured deltas, and the measurement discipline is the same one I described in Evaluating Agentic Systems: a gold set of tasks with expected tool-call sequences, replayed against the agent, scored automatically. The specific loop for tool contracts:

1. Baseline. Run the gold set against the current specs. Record per-tool skip rate, wrong-tool confusions, and argument validation failures. Per-tool matters: an aggregate skip rate of 8 percent can hide one tool that is skipped half the time.
2. Fix the specs. Lint first (mechanical defects), then apply the semantic rules: boundaries, when-not-to-use clauses, schema constraints.
3. Re-run the same gold set. The model, the prompt, and the tasks have not changed, so the delta is attributable to the spec. This is the cleanest A/B you will ever get in agent engineering, and it is how I know description rewrites cut a 20-plus percent skip rate to low single digits in the engagement that started this guide.
4. Gate it in CI. Tool specs are code. Version them, diff them, and treat a description change as a behavior change, because it is one; it edits a prompt that runs on every turn. My pipeline fails a pull request if any tool’s Tool Credit Score drops below threshold, and flags any spec diff for an eval re-run before merge.

The same wealth-management engagement produced the other number I keep quoting: once the tool contracts were fixed, a mid-tier model matched the premium model’s accuracy on our gold set at roughly 1/13th the cost, with tool-skip rates measured rather than assumed. Bad specs do not just cause failures. They silently inflate your model bill, because the expensive model’s extra capability was being spent compensating for contracts the cheap model could have executed if they had been written properly.

What I would still do differently: my gold sets are replayed conversations, not live users, and real users phrase requests in ways that stress tool selection harder than curated tasks do. Skip rates in production sit above eval skip rates for every agent I have shipped. Treat the eval number as a lower bound and keep sampling production traces against the same taxonomy.

The one-paragraph version

Most tool-call failures in production agents are spec defects, not model defects: skips, wrong-tool selections, malformed arguments, hallucinated arguments, and premature answers all trace back to contracts written for developers instead of for the model that actually reads them. Fix the contract: one job per tool with no overlapping neighbors, behavior-oriented descriptions with explicit when-to-use and when-not-to-use clauses, constraints expressed as enums, formats, and ranges in the schema rather than prose, an honest required array, at least one example, deterministic error codes an agent can branch on, and verb-object names free of internal jargon. Lint all of that deterministically in CI (my open-source Tool Credit Score does it, 0 to 100 with auto-fix) because reproducible scores beat LLM-judged reviews for gating, then prove the semantic fixes with evals by measuring per-tool skip rates before and after. In my enterprise eval work, that loop cut a 20-plus percent tool-skip rate to low single digits without touching the model or the system prompt, and it is the reason a mid-tier model matched a premium one at roughly 1/13th the cost.

References

• Model Context Protocol specification, including tool definitions and annotations
• Anthropic: tool use documentation, the practical reference for how models consume tool specs
• Anthropic engineering: writing effective tools for agents
• Agent Contract Studio (Tool Credit Score), the deterministic MCP spec linter described above, live at agentic-contract-studio.vercel.app
• Evaluating Agentic Systems, my field guide to the eval harness this document’s measurements come from