Evaluate What Your Agent Does, Not What It Says: A Guide to Tool-Call Evals

The most expensive bug I've shipped in an agent never threw an error. The agent quoted a customer a price, warmly and in perfect prose. No pricing tool had been called; the number was simply plausible. The transcript read beautifully, and my AI agent evals (at the time, an LLM judge scoring the final reply for helpfulness and tone) gave it a high score.

That's the failure this post is about. An agent can describe an action perfectly and never perform it. If your evals only read what the agent says, they will reward exactly that behavior. Here's how I restructured agent evaluation around what the agent does: which tools it called, which ones it avoided, and whether every fact in its reply traces back to a tool result.

The Failure Mode: Perfect Descriptions, Zero Execution

Text-only evaluation made sense when LLM apps were chatbots. The output was the product, so you graded the output. Agents broke that assumption. The product of an agent is a set of side effects: a row written, an email sent, a calendar slot taken. The text is just the narration.

The narration and the side effects drift apart in predictable ways:

• Phantom actions. "I've saved your details and someone will reach out." No create_lead call in the trace.
• Invented facts. "That model is $1,240." The price never came from a tool; the model pattern-matched a plausible number.
• Redundant permission questions. "Shall I check availability for you?", asked after the availability tool already ran and returned results.
• Wrong-tool actions. The agent books a test ride when the user only asked whether one was possible.

None of these show up in a transcript skim. Each looks fine in isolation. And an LLM-as-judge that only sees the final message can't catch them either, because the judge has no way to know what should have happened underneath.

Read the trace, not the transcript. That's the whole thesis. The rest of this post is how to make it mechanical.

Three Assertion Types for AI Agent Evals

I now write every eval case around three questions. Each maps to an assertion type that runs against the trace: the ordered list of tool calls, their arguments, and their results, not just the reply.

1. Expected tools. Did the agent call the tools this turn requires? If the user says "book me a test ride Saturday at 10," a booking call must appear in the trace, with the right date.
2. Forbidden tools. Did the agent avoid the tools this turn must not trigger? A user asking "do you do test rides?" should get an answer, not a booking. Forbidden-tool assertions catch over-eager agents, which are often worse than lazy ones.
3. Grounded facts. Is every checkable fact in the reply (price, stock, name, date) present in some tool result from this conversation? If the reply says "$1,240" and no tool returned 1,240, the case fails.

Here's how the three compare to what most teams run today:

Eval type	What it checks	Catches	Misses
Text-only (LLM judge on the reply)	Tone, helpfulness, relevance of the final message	Rude or off-topic replies, obvious refusals	Phantom actions, invented prices, wrong-tool side effects
Tool-call eval	Expected and forbidden tools in the trace, plus key arguments	Actions not taken, actions taken that shouldn't be, wrong arguments	Correct actions narrated with made-up details
Grounded-fact eval	Every checkable fact in the reply appears in a tool result	Hallucinated prices, names, availability, dates	Whether the right action happened at all

You need the last two together. Tool-call evals prove the agent acted. Grounded-fact evals prove it didn't embellish. The text-only judge can stay for tone, but it shouldn't be the gate.

The flow per test case looks like this:

If you already trace with LangSmith or Langfuse, you have the raw material. Both record tool spans with inputs and outputs, and the OpenTelemetry GenAI semantic conventions give you a vendor-neutral shape for the same data. The eval just has to read spans instead of only reading the last message.

Building the Eval Set From Real Conversations

My first eval sets were invented prompts. I sat down, imagined what users would ask, and wrote cases. They were clean, grammatical, and almost useless. Real users don't write "I would like to schedule a test ride for the touring bike on Saturday." They write "sat morning? the blue one."

Now I build eval sets from real conversations, anonymized before they ever touch the repo:

1. Pull conversations where something went wrong: a human had to step in, a user repeated themselves, a complaint came in. These are your highest-value cases.
2. Strip personal data: names, phone numbers, emails, addresses. Replace with obviously fake stand-ins.
3. Cut each conversation at the decision turn: the history up to the moment the agent had to act, plus the user message that triggered it.
4. Write the assertions by hand. This is the part that can't be automated: a human decides what should have happened.

Each case ends up as a small, readable record. Using a made-up bike shop as the domain:

Field	Example
Input	"sat morning? the blue one" (history: user asked about the touring bike in blue)
Expected tools	check_availability with date = next Saturday, product = touring bike, color = blue
Forbidden tools	book_test_ride (user hasn't confirmed a slot yet)
Facts to ground	Any time slots mentioned; any price mentioned
Notes	Agent should offer slots, not book one

That table is the spec for the behavior. If you work spec-first, this will feel familiar. It's the same move I describe in What Is Spec-Driven Development?: write the acceptance criteria before the implementation, and make them checkable. For multi-turn scenarios with a simulated user, I go deeper in How to Build an Eval Gym for Conversational AI Agents.

Oracles: Checking Facts Against Your Source of Truth

Grounded-fact assertions need something to check against. That's an oracle: a function that knows the true value of a fact independently of the agent.

The simplest oracle is the trace itself: "did this number appear in any tool result?" That catches pure invention. But it misses a subtler failure: the agent called the price tool for the wrong product and quoted that price faithfully. Grounded in a tool result, still wrong.

So for the facts that matter most, I query the source of truth directly:

• Prices: look up the product in the database and compare against the number in the reply, with an explicit tolerance for currency formatting.
• Names: product and staff names in the reply must exist in the reference table. Near-misses ("Touring 500" vs "Tour 500") fail.
• Availability: any slot the agent offers must be open in the scheduling system at eval time, or in a frozen fixture if you run against snapshots.

Extracting facts from free text is the fiddly part. I use a cheap structured-output call to pull claims out of the reply into a typed shape ({ type: "price", product: "...", value: ... }), then compare deterministically. The LLM extracts; plain code judges. Keep the judge out of the judging whenever a comparison can be exact.

Output Guardrails: Pruning Claims No Tool Backed

Evals tell you about failures after the fact. In production, you also want the agent to stop making unbacked claims before the user sees them. That's an output guardrail, and it uses the same logic as the grounded-fact eval, just inline.

Two rules do most of the work:

• No confirmation without a backing call. If the reply says "booked," "saved," "sent," or "confirmed," there must be a successful call to the matching tool in this turn. Otherwise the sentence is removed or replaced with an honest version ("I can book that for you. Want me to go ahead?").
• No permission question for something already done. If the availability tool ran, "Shall I check availability?" is noise that makes the agent look confused. Drop it.

The trade-off is real: a guardrail that rewrites text can produce stilted replies, and claim extraction adds latency. I keep the rule set small and deterministic, and I log every prune. A spike in pruning is an early signal that a prompt change broke something. This is one layer of a broader design I cover in Separate Talking From Acting: A Guardrail Architecture for Production AI Agents.

Fail-Closed Search: Why "Nothing Found" Invites Hallucination

This one surprised me. Some of the worst invented facts didn't come from the model ignoring tools. They came from a tool returning nothing.

When a search tool returns an empty result, the model is left holding a user question and no data. Models hate empty hands. They fill the gap with something plausible: a product that sounds right, a price that fits the range. The tool technically worked, and the hallucination happened one step later.

The fix is to make search fail closed: never return a bare "not found." Return something the model can ground on instead:

Search returns	What the model tends to do
[] or "not found"	Improvises an answer from priors
A short list of nearest matches with IDs	Offers real alternatives ("Did you mean…?")
An index of available categories	Asks a clarifying question grounded in real options
An explicit { match: none, suggestion: "ask the user to clarify" }	Asks instead of inventing

Fuzzy matching on the query plus a fallback list of real options changed the failure from "confident fiction" to "useful clarifying question." It also makes the grounded-fact eval more meaningful, since there's always real data in the trace to check against.

Measure First, Then Add Tools

Here's the order of work I wish I'd followed from day one:

1. Instrument tracing before anything else. If you can't see tool calls, you can't evaluate them.
2. Write evals for the tools you already have. Expected, forbidden, grounded. Get a baseline score, even an ugly one.
3. Fix the cheapest failures: usually tool descriptions, empty-result handling, and missing guardrails.
4. Only then add a new tool, with its eval cases written in the same change.

The temptation runs the other way. A new tool is a visible feature; an eval suite is invisible plumbing. But every tool you add without evals is one more thing you can't prove works, and tools interact. Adding a create_lead tool can make the agent call it in conversations where it used to just answer a question. Without forbidden-tool assertions on the old cases, you won't notice until a sales rep asks why the CRM is full of junk.

Tool design matters here too. In my public mcp-deep-dive lab I spent a lot of time on how tool names and descriptions steer selection; the same lessons apply whether the tools are exposed over MCP or wired directly into a LangGraph graph. A vague description shows up in your evals as a wrong-tool failure long before it shows up in a bug report.

Wiring Evals Into CI and Prompt Changes

Evals that run when someone remembers are evals that don't run. I treat them like tests with one difference: they're probabilistic, so a single run proves less.

• Run the suite on every prompt, tool-description, or model change. Prompts are code. A one-word edit to a system prompt can flip tool selection across dozens of cases.
• Run each case several times and score the pass rate, not a single pass/fail. A case that passes four times out of five is a flaky behavior, and you want to know.
• Gate on regressions, not absolute scores. Compare against the main branch's baseline. Say the suite drops 1.5 points on forbidden-tool assertions: that's a signal worth blocking on even if the absolute number still looks fine.
• Split hard assertions from soft scores. A forbidden destructive tool being called is a hard fail. A tone score dipping is a soft signal for a human to review.
• Store traces from failing runs as artifacts, so reviewers see what the agent did, not just that it failed.

If you're already doing spec-driven work with Claude Code agents and skills, the eval cases become part of the acceptance criteria, and a coding agent can run the suite as its verification step before it claims a change is done.

Checklist: 10 Signs Your Agent Evals Only Test Prose

If more than two of these are true, your evals are grading the narration:

• Your eval cases have no field for expected tool calls.
• Nothing asserts that a tool was not called.
• The only scorer is an LLM judge that sees the final reply.
• Prices, names, or dates in replies are never compared to a database.
• Your eval prompts were all written by the team, not taken from real conversations.
• Empty search results return [] or "not found" with no alternatives.
• Nothing in production stops a "confirmed!" that no tool backed.
• Prompt and tool-description changes ship without an eval run.
• Each case runs once, and a single pass counts as passing.
• When a case fails, you see the reply but not the trace.

How to start this week

1. Turn on tool-call tracing (LangSmith, Langfuse, or OpenTelemetry spans).
2. Pick ten real conversations where something went wrong. Anonymize them.
3. For each, write expected tools, forbidden tools, and facts to ground.
4. Write one oracle for the fact that costs you most when it's wrong, usually price.
5. Add a single guardrail: no confirmation sentence without a successful backing call.
6. Run the suite on your next prompt change and compare against the baseline.

That's a day or two of work, and it'll tell you more about your agent than a month of reading transcripts.

The Takeaway

An agent's reply is a claim. The trace is the evidence. Good AI agent evals check the evidence: the right tools called, the wrong ones avoided, every fact traceable to a result. Pair that with output guardrails that prune unbacked claims and search that never returns empty-handed, and the "perfectly described, never executed" failure gets hard to ship.

Build the measurement before you add the next capability. If you want to push further, read about red-teaming agents with prompt injection tests next. The same trace-based assertions work for security, too.

---

Building evals for an agent that takes real actions? Reach out. I'm always interested in how other engineers prove their agents work.