In early 2026 I was running a wealth-management proof of value at a major North American bank. The agent under review was not a chatbot. It lived inside a governed workflow: it received a case, called tools (document retrieval, screening checks, account lookups), and produced a structured output that downstream process steps consumed without a human reading it first. Near the end of a working session, someone on the client side asked the question that every enterprise eventually asks: "Why this model? And what will it cost us per case?"

I did not have a defensible answer, and neither did the market. Everything published about model quality at the time was built for the chatbot era: MMLU scores, arena leaderboards, vibes-based bake-offs where someone pastes the same prompt into three models and picks a favorite. None of it says anything about whether this agent, with these tools, inside this process, on these cases, produces correct fields at a predictable price. A leaderboard Elo cannot tell you whether the model will silently skip a sanctions check when the case looks routine.

So I built an evaluation harness for it. At the time, "agent evals" was not a category anyone was asking for; there was no off-the-shelf tooling for evaluating a process-embedded agent, so I had to decide from scratch what was even worth measuring. I landed on four axes: cost, quality, tool reliability, and temperature sensitivity. Since then the same four axes have run across many enterprise engagements, and the question I could not answer in that room now comes up in nearly every first meeting. The market caught up to the question. This guide is the method.

![The four-axis eval as a two-by-two grid: cost per case, per-field accuracy on a gold set, tool reliability with measured skip rates, and temperature sensitivity](/diagrams/eval-four-axes.svg)

## Why benchmark evals answer the wrong question

A process-embedded agent differs from a chatbot in four ways that break every chatbot-era evaluation technique.

**It is multi-turn by construction.** One process instance is a loop: the model plans, calls a tool, receives the result, plans again. A single case might involve two model calls or fourteen. Any metric computed on "one prompt, one completion" measures a thing that never happens in production.

**It calls tools, and the tool calls are the work.** The final answer is often the least interesting part of the transcript. Whether the agent called the beneficial-ownership lookup before rendering a verdict matters more than how the verdict is phrased. An eval that only inspects final text is blind to the majority of failure modes.

**Its output is structured and machine-consumed.** The agent emits JSON that downstream steps route on. There is no human in the loop to notice that a field is subtly wrong. "Sounds right" is not a grade; either `risk_rating` matches the gold label or it does not.

**Its cost is a property of the whole loop, not the token price list.** Every tool round trip re-sends the growing conversation context. Retries on malformed output are model calls too. Two models with similar per-token prices can differ by multiples in measured cost per case, because one of them takes more iterations to finish.

| Dimension | Benchmark evals (chatbot era) | Process evals (agent era) |
|---|---|---|
| Unit of measurement | One prompt, one completion | One process instance, end to end |
| Interaction shape | Single-turn Q&A | Multi-turn loop with tool calls |
| Output judged | Free text, human preference | Structured fields, graded per field |
| Cost model | Price per million tokens | Measured cost per case, including retries and tool round trips |
| Failure modes visible | Wrong or unhelpful answer | Wrong field, skipped tool, wrong tool, malformed or hallucinated arguments, premature answer |
| What "good" means | Leaderboard position | The process completed correctly at an acceptable, predictable cost |

The rest of this guide takes the four axes one at a time. Each one is a set of concrete things to log and metrics to compute. Together they answer the two questions from that room: which model, and at what cost per case.

## Axis 1: Cost per process instance

The unit that matters is not the token. It is the case: one loan application, one KYC review, one claim. That is the unit the business prices, staffs, and budgets against, so it is the unit your eval must report in.

Measured cost per instance is the sum of every model call the instance triggered:

- The initial planning call.
- One call per tool round trip, each re-sending the accumulated context. In a naive agent loop, input tokens grow with every iteration, so tool-heavy cases cost disproportionately more than the first call suggests.
- Every retry. If the model emits output that fails schema validation and the orchestration layer re-prompts it, that retry is real money and belongs in the number.
- Any auxiliary calls the pattern requires: output validators, guardrail checks, summarization of oversized tool results.

To compute it, log per model call: model identifier, input tokens, output tokens, latency, and finish reason. Log per instance: iteration count and retry count. Multiply tokens by the provider's current prices and sum. Report the median and the p95, because agentic cost distributions have long tails; the case that triggers three retries and twelve tool calls is the one that shows up in the invoice.

One derived metric has earned its place in every engagement since: **cost per correct case**, which is cost per instance divided by the fraction of instances the model got fully right. A cheap model that is wrong a third of the time is not cheap, because someone pays for the rework.

The finding that made this axis famous inside my own work: on a KYC-style task at that first engagement, a mid-tier model matched the premium model's accuracy at roughly 1/13th the cost per case. Not 20 percent cheaper. Thirteen times cheaper, with no measured quality loss on that task. The reason, in hindsight, is unexciting: the task was extraction and rule-following against well-specified tools, not open-ended judgment. Mid-tier models are good at that now. But nobody in the room would have bet on that ratio beforehand, and without a harness nobody could have proven it. Per-token price lists predicted a gap; only measured cost per case sized it.

![Cost versus quality scatter: a mid-tier model matches the premium model's accuracy on the gold set at one thirteenth the cost per case, inside a shaded right-sizing zone](/diagrams/eval-cost-quality.svg)

The pitfall on this axis is comparing price lists instead of measuring. A model that costs half as much per token but averages two extra loop iterations and one retry per case can come out more expensive. I have watched that exact inversion happen. Measure the loop, not the list.

## Axis 2: Quality, graded per field

Quality for a process-embedded agent means: does the structured output match what a competent expert would have produced for this case? That requires two things most teams skip: a gold set built from real cases, and per-field grading.

**Building the gold set.** My rule from the field: 10 to 30 scenarios drafted from real cases beats 1,000 synthetic ones. Synthetic case generators (usually another LLM) reproduce the same blind spots as the model under test; they generate clean, well-formed, statistically average cases. Real cases carry the awkwardness that actually breaks agents: the hyphenated name that defeats a matching heuristic, the client with dual citizenship, the document that is missing a page, the entity whose registered address differs from its operating address. The procedure I use:

1. Sit with the people who do the process today and pull 10 to 30 real cases, spanning routine, edge, and known-hard.
2. Anonymize them structurally: preserve the shape of the difficulty (the hyphenation, the missing document) while replacing every identifying value.
3. For each scenario, have the expert write down the correct value for every output field, plus the tools a correct execution must call. That second annotation feeds Axis 3.
4. Get a second expert to review the labels. Where the two disagree, the field is ambiguous; either tighten its definition or exclude it from grading. Ambiguous gold labels are how evals lie to you.

**Grading per field, not per vibe.** Because the output is structured, most grading can be deterministic: exact match for enums and identifiers, normalized match for names and dates, numeric tolerance for amounts. Reserve LLM-as-judge for the free-text rationale fields, and treat those scores as directional, not authoritative (more on judge bias under limitations). Compute:

- Per-field accuracy across scenarios, so you can see that `entity_type` is fine but `source_of_wealth` is weak.
- All-fields-correct rate per scenario, the number that maps to "cases needing no rework."
- Critical-field accuracy as its own headline. In a compliance process, getting `pep_flag` wrong is not the same as getting a formatting nuance wrong. Weight accordingly and say so in the report.

The pitfall on this axis is grading leniency creep. When a model scores poorly, the temptation is to soften the matcher ("close enough") rather than accept the score. Freeze the grading rules before you run the bake-off, and change them only with the same discipline you would apply to changing a test suite.

## Axis 3: Tool reliability, the axis nobody measures

This is the axis I have never seen in anyone else's eval, and it is the one that has changed the most decisions. Chatbot metrics have no concept of it, because chatbots have no tools to skip.

A process-embedded agent is given tools with an implicit contract: for this class of case, you call the screening check, then the ownership lookup, then you answer. The model can breach that contract in ways that produce no error, no exception, and no visible artifact in the final output. The worst of these is the **tool skip**: the agent silently not calling a tool it should have called, then producing an answer that reads as complete.

Skip rate is invisible in demos and fatal in production. In a demo, the happy-path case gets shown, the output looks plausible, and everyone nods. In production, "the agent did not run the sanctions check on 4 percent of cases" is a regulatory finding, not a quality metric. The only defense is to measure it, which is why the gold set carries expected-tool-call annotations. My standing line in readouts: skip rates measured, not assumed.

> If your harness only records what the agent said, you are evaluating a text generator standing near some tools. Record what it did, and what it failed to do.

From the transcripts (which you are already logging for the cost axis), compute per model:

- **Tool recall**: of the tool calls a correct execution requires, what fraction did the agent make?
- **Skip rate per tool**: which specific tools get skipped, and on which scenario classes? Skips cluster; a tool that gets skipped is usually one whose description does not make its necessity obvious.
- **Wrong-tool rate**: the agent called something, but not the right something.
- **Malformed-argument rate**: calls that failed schema validation.
- **Argument accuracy**: calls that passed validation but carried wrong values.

The full taxonomy of tool failures, in the order I worry about them:

#### Tool skip

The agent should have called a tool and did not, and answered anyway. No error is raised anywhere. Detection requires expected-call annotations; nothing in the runtime will tell you. This is the failure mode that justifies the entire axis.

#### Premature answer

A systemic variant of the skip: the agent starts the required sequence, then decides mid-loop that it knows enough and renders a verdict with the sequence incomplete. Common on cases that look routine, which is exactly when reviewers are not looking. Detect by comparing the executed call sequence against the required set, not just checking that each tool appears somewhere.

#### Hallucinated arguments

The call is schema-valid and the runtime accepts it, but an argument value was invented rather than taken from the case: a fabricated account identifier, a date the documents do not contain. This one passes every automatic gate and poisons downstream data. Detection needs argument-level gold values or an expert spot-check; budget for it.

#### Wrong tool selection

The agent calls the account-history tool when the case needed the screening tool. Usually a symptom of overlapping tool descriptions rather than model weakness. Detectable automatically against the annotations.

#### Malformed arguments

The call fails schema validation. Loud, cheap to detect, and mostly a retry-cost problem rather than a correctness problem, provided your orchestration layer actually validates and retries. The models that malform arguments frequently also show up as expensive on Axis 1, which is how the axes cross-check each other.

Here is the part that makes this axis actionable rather than fatalistic: in my engagements, a large share of tool-reliability failures traced back to the tool specifications, not the model. Vague descriptions, overlapping names, undocumented enums, parameters whose purpose is only clear to the person who wrote the API. Fixing the specs moved skip and wrong-tool rates more than switching models did. I wrote up the spec-side discipline separately in [the tool contracts guide](/guides/tool-contracts/); the eval harness is how you find out which contracts are failing, and the contracts are how you fix it.

## Axis 4: Temperature sensitivity

I added this axis expecting it to be boring. Run the grid at a few temperatures, confirm that temperature zero reduces output variance, write one slide, move on. That is not what the data said.

At temperature zero, the models I tested did not merely become more consistent. They became more thorough with tools: higher tool recall, lower skip rates, on the same scenarios with the same prompts. Temperature was changing what the agent did, not just how repeatably it did it. My working explanation is mechanical rather than mysterious: at each decision point the model weighs "call a tool" against "answer from what I have," and sampling at higher temperature occasionally takes the lower-probability path, which for a well-prompted agent is the lazier one. At temperature zero the greedy path more often includes the tool call. I hold that explanation loosely; the measurements I hold firmly.

Why this axis exists at all: regulated customers ask for temperature zero by name, and vendors routinely present it as a safety control. Before you repeat that claim in a bank, you should know what temperature zero actually does to your specific agent, because "it changes which tools get called" is a materially different statement than "it reduces randomness."

> Do not tell a regulated customer that temperature zero is a safety control until you have measured what it changes. In my runs it changed which tools got called.

Method: run the full gold set at each candidate temperature (I use 0, a low value like 0.2 to 0.3, and the provider default), with each scenario repeated several times per setting so you can separate variance from level. Report the other three axes broken out by temperature: accuracy, tool recall, skip rate, and cost per case, plus run-to-run disagreement as the variance measure. Two caveats from the field: some reasoning-class models ignore or reject the temperature parameter entirely, so verify the setting is actually being applied; and temperature findings are per model and per provider, so re-measure when either changes rather than porting the conclusion.

## What the numbers do to your architecture: model routing

The four axes are not a report card to file. They are routing inputs.

Plot each candidate model as a point: measured cost per case on one axis, all-fields-correct rate on the other. The points trace a cost-of-quality curve, and the shape of that curve is the decision. Three shapes recur:

- **Flat curve.** The mid-tier model sits at the same quality as the premium model at a fraction of the cost. This was the KYC-style finding: same accuracy, roughly 1/13th the cost. Decision: run the mid-tier model, keep the premium model as an escalation target, and bank the difference. Flat curves are common on extraction and rule-following tasks with good tool contracts.
- **Cliff curve.** Quality drops sharply below a certain model tier. Common on tasks requiring genuine multi-step judgment or synthesis across many documents. Decision: pay for the tier above the cliff; do not let procurement talk you below it.
- **Paying-for-nothing curve.** The premium model is more expensive and no better, sometimes worse on tool recall (bigger models are not uniformly better agents; measure, do not assume). Decision: obvious.

The deeper consequence is per-step routing. A process-embedded agent is not one model decision; it is one decision per step, because the orchestration layer (Camunda in my builds, but Temporal, LangGraph, or a hand-rolled Node.js loop give you the same seam) binds a model to each task independently. That enables the multi-model philosophy I now bring to every engagement:

- **Right-size each step.** Document extraction and classification steps go to the mid-tier model that the eval showed is sufficient. The one step that requires weighing contradictory evidence gets the premium model. Run the harness per step, not just per process.
- **Route to premium on signal, not by default.** Low model-reported confidence, schema validation failure, or a gold-set-identified hard case class can route an instance to the premium model, or to a human. The escalation decision lives in the process definition, where it is visible and auditable, not buried in a prompt.
- **The cheapest model is no model.** If a step's eval-measured behavior is reproducible by a rules engine or a lookup, take the LLM out of that step entirely. The eval is often how you discover this: when a model scores 100 percent on a step across every tier, that step probably did not need a model.

Cost per case at the process level then becomes a weighted sum over steps, and you can answer the room's second question ("what will it cost per case?") with a number that has error bars instead of a shrug.

## Running evals continuously

The first run of the harness answers "which model." Every subsequent run answers a more important question: "did we just break it?"

A process-embedded agent has at least five change surfaces, and every one of them can silently move the four axes:

1. **Prompt edits.** The innocent wording tweak that drops tool recall by ten points. I have seen a one-sentence prompt "clarification" double a skip rate.
2. **Tool spec changes.** Renaming a parameter, adding a tool, tightening a description. Any of these changes what the model selects and how it argues with the schema.
3. **Model version bumps.** Deliberate upgrades. Never assume the new version dominates the old on all four axes; check tool recall in particular.
4. **Provider drift.** The same model alias behaving differently after a provider-side update you were not told about. Pin exact model versions where the provider allows it; where it does not, a scheduled rerun of the harness is your drift detector.
5. **Configuration changes.** Temperature, token limits, retry policies. Axis 4 exists because these are not neutral.

So treat the harness as a regression suite. Version the gold set in the same repository as the prompts and tool specs. Stamp every eval run with a config hash covering prompt version, tool spec version, model identifier, and temperature, so any score is reproducible and any regression is bisectable. Gate changes the same way you gate code: a prompt edit that tanks tool recall should fail review before it reaches production, and the eval report is the review artifact.

This is affordable because the gold set is small. Twenty scenarios, three repetitions, one model: that is a coffee-break run, cheap enough to attach to every meaningful change rather than saving for quarterly bake-offs. The 10-to-30-scenario discipline from Axis 2 is not just about label quality; it is what makes continuous evaluation economically boring, which is what makes it actually happen.

## What this method does not solve

The honesty section, because a method you trust past its limits is worse than no method.

**Gold sets go stale.** The process changes, document formats change, regulation changes, and the gold set quietly stops representing production. A gold set is a snapshot of the process at drafting time. Review it whenever the process itself is reviewed, and retire scenarios that no longer occur. A harness scoring green against last year's cases is a false comfort.

**Small n gives you rankings, not certifications.** Ten to thirty scenarios will reliably rank models and catch large regressions. It will not support "94.2 percent accurate" in an audit document; the confidence interval on 20 cases is wide, and pretending otherwise is malpractice. Use the harness for engineering decisions. If a compliance function needs a certified accuracy number, that is a larger, separately governed sampling exercise.

**LLM-as-judge carries known biases.** Verbosity bias, position bias, and self-preference (a model rating its own family's output favorably) are all documented. This is why the method pushes grading toward deterministic per-field checks, which structured outputs make possible. Where a judge is unavoidable (free-text rationales), use a model from a different family than any candidate under test, a fixed rubric, and periodic human agreement checks. And keep judge scores out of the headline numbers.

**Evals overfit like everything else.** If you tune prompts against the gold set for weeks, you are teaching to the test, and the score stops predicting production. Hold out a few scenarios that never inform prompt changes, and rotate fresh real cases in as they arrive. When the held-out score and the development score diverge, believe the held-out score.

**Four axes are not all axes.** This method says nothing about adversarial robustness, prompt injection through tool results, long-horizon memory, or the quality of human-in-the-loop handoffs. Those need their own evaluations. Four axes were the ones that answered the questions enterprises actually asked me; they are a floor, not a ceiling.

## The one-paragraph version

Chatbot-era evals cannot tell you which model should run an agent embedded in a governed process, so measure the agent the way the business will experience it: build a gold set of 10 to 30 real anonymized cases with per-field expected outputs and expected tool calls, then score every candidate model on four axes: cost per process instance (every loop iteration, retry, and tool round trip, reported per case, where a mid-tier model once matched a premium model at roughly 1/13th the cost), per-field accuracy against the gold labels (deterministic matching, not vibes), tool reliability (skip rates, wrong-tool selection, and malformed or hallucinated arguments, measured rather than assumed, because a silently skipped compliance check is invisible in a demo and fatal in production), and temperature sensitivity (temperature zero changed tool thoroughness in my runs, not just variance, so measure before calling it a safety control). Feed the resulting cost-of-quality curves into per-step model routing, rerun the harness as a regression suite on every prompt, tool, and model change, and hold the results loosely where the method is weak: small gold sets rank models but do not certify them, and gold sets go stale.

## References

- Anthropic, "Create strong empirical evaluations": [docs.anthropic.com/en/docs/build-with-claude/develop-tests](https://docs.anthropic.com/en/docs/build-with-claude/develop-tests)
- Anthropic, "Building effective agents": [anthropic.com/engineering/building-effective-agents](https://www.anthropic.com/engineering/building-effective-agents)
- OpenAI, "Evals" guide: [platform.openai.com/docs/guides/evals](https://platform.openai.com/docs/guides/evals)
- Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (the canonical treatment of judge biases): [arxiv.org/abs/2306.05685](https://arxiv.org/abs/2306.05685)
- Yao et al., "τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains": [arxiv.org/abs/2406.12045](https://arxiv.org/abs/2406.12045)
- Berkeley Function-Calling Leaderboard, for public tool-calling comparisons (useful context, not a substitute for your own gold set): [gorilla.cs.berkeley.edu/leaderboard.html](https://gorilla.cs.berkeley.edu/leaderboard.html)
- Hamel Husain, "Your AI Product Needs Evals": [hamel.dev/blog/posts/evals](https://hamel.dev/blog/posts/evals/)
- Companion guide on the spec side of tool reliability: [Tool Contracts](/guides/tool-contracts/)
- This method was extracted from production proof-of-value work with large financial institutions; the engagements are generalized and no customer eval code is public.