How to Build an Eval Gym for Conversational AI Agents

For a long time, my idea of testing a chat agent was to open the chat and talk to it until it felt right. I'd try five or six messages, nod at the answers, change a line in the system prompt, try the same five or six messages again, and ship. It worked fine until the week I changed the instructions, swapped a tool and bumped the model version all at once. Then I couldn't tell which change had made the agent worse, or whether it had got worse at all.

That week is why I now treat AI agent evaluation as its own piece of infrastructure. I call it an eval gym: a place where the agent works out every time something changes, on a fixed set of exercises, with scores I can compare from one run to the next. This post covers how I structure one for conversational agents. It has four layers, a scoring approach that combines rules with an LLM judge, the statistics that keep me honest, and a habit that keeps the whole thing relevant.

Why Vibe-Checking Fails as AI Agent Evaluation

Manual spot checks hold up as long as only one thing changes at a time. A conversational agent in production has at least three moving parts:

• Instructions: the system prompt, policies and few-shot examples.
• Tools: the functions the agent can call, their schemas and what they return.
• Model: the provider, the version, the temperature and the reasoning settings.

When all three move in the same sprint, a human chatting with the bot runs into two problems. You can't cover the space, because a real conversation has memory, ambiguity and tone, and five messages don't exercise any of that. And you can't compare runs, because LLM output is stochastic, so an answer that looked better might just have been a luckier sample.

The second problem is the one that catches experienced engineers out. We're used to tests that pass or fail the same way every time. Agent behaviour is a distribution, and evaluating it means measuring the distribution.

The Four Layers of an Eval Gym

I split the gym into four layers, ordered from cheapest and fastest to most expensive and most realistic. Each one catches a different kind of failure.

Layer	What it tests	Typical cost / speed	Catches
Classifier	Intent detection, routing, language detection	Very cheap, seconds	Misrouted requests, broken labels after a prompt edit
Policy	A single agent reply against explicit rules	Cheap, under a minute	Forbidden claims, missing disclaimers, wrong format
Multi-turn	Scripted conversations of 3–10 turns	Moderate, a few minutes	Lost memory, bad handoffs, tool misuse mid-conversation
Simulation	A full conversation driven by a simulated user	Expensive, the slowest	Emergent failures nobody scripted

The base of the pyramid runs on every commit. The top runs before a release, or whenever I touch something structural. If your agent follows the pattern I describe in LLMs at the edges with a deterministic core, the classifier layer matters even more, because the classifier decides which deterministic branch the conversation takes.

Scoring That Holds Up: Rules First, Judge Second

The biggest early mistake I made was sending everything to an LLM judge. Judges are flexible, but they're also slow, they cost money, and they add noise. My rule now is that if a check can be deterministic, it is.

Scoring method	Use it when	Example rule
Deterministic rule	The property can be checked mechanically	"The reply never contains a price unless the get_price tool was called in this turn"
LLM judge	The property depends on meaning or tone	"The reply acknowledges the customer's frustration before offering a solution"
Both	A cheap rule can flag violations but can't confirm compliance	"Refuses out-of-scope requests": a regex catches obvious leaks, and the judge checks that the refusal is polite and offers an alternative

The "both" row is where most of the value is. Take a refusal policy. A rule can catch the obvious failure (the agent answered a medical question it should have declined) in milliseconds with no variance. But a rule can't tell a curt "I can't help with that" from a helpful redirect, so the judge scores that part. When I combine them, the rule acts as a hard gate (fail means fail, and the judge never runs) and the judge adds a graded score on top.

A few things make judges more trustworthy:

1. One criterion per judge call. A judge asked to score "helpfulness, accuracy and tone" will blur them together.
2. Binary or 3-point scales. "Did it do X: yes / partially / no" is far more stable than a 1–10 score.
3. Ask for the reason before the verdict. The reasoning makes the verdict auditable when you disagree with it.
4. Calibrate against yourself. Label 20–30 transcripts by hand and check how often the judge agrees with you before trusting it on thousands.

Tool calls deserve their own treatment, because a lot of agent failures happen in what the agent does, not what it says. I go deeper on that in tool-call evals.

Signal vs Noise: N-Run Averaging and Standard Deviation

This lesson changed how much I trust my own numbers.

Run the same suite twice with no code changes and you'll get two different scores. The agent samples differently and so does the judge. If you run each suite once, a prompt change that does nothing can look like a two-point win, and you'll ship it, and the next week you'll "regress" by the same amount for no reason.

So every suite in the gym runs N times (I default to 5) and reports the mean and the standard deviation. Here's a made-up example of a before/after for a prompt change:

Run	Before (pass %)	After (pass %)
1	84.0	86.0
2	86.5	83.5
3	83.0	85.5
4	85.5	87.0
5	86.0	83.0
Mean	85.0	85.0
Std dev	~1.5	~1.7

If I'd only looked at run 1, I'd have called the change a 2-point improvement. Over five runs, the means are identical. The general rule I follow is that a delta smaller than the noise isn't a result. If the change in the mean is about the size of one standard deviation (say a +1 point change against 1.5 points of noise), I treat it as no change and either run more iterations or look for a sharper eval.

Two practical notes:

• Pin what you can. Use temperature 0 for the judge, a fixed judge model version and fixed seeds where the provider supports them. Variance in the agent itself is real and you want to measure it. Variance in the judge is just measurement error.
• Report per-case flakiness, not just the aggregate. A case that passes 3 out of 5 times is telling you something. Usually the instruction is ambiguous, and that's worth fixing on its own.

Multi-Turn Coverage by Dimension, Not by Happy Path

My first multi-turn suite was a handful of happy-path conversations: the customer asks, the agent answers, the customer buys. They all passed, and none of them told me anything.

The change that worked was to organise multi-turn cases by dimension, meaning the capability under test, rather than by user journey. Every dimension gets several cases, and a new case has to say which dimension it covers.

Checklist: 10 multi-turn dimensions to cover

• Memory: the agent uses a fact the user gave three turns ago without asking again.
• Handoff: it recognises when to escalate to a human and does it cleanly (see human handoff design).
• Refusal: it declines out-of-scope or unsafe requests and offers a useful alternative.
• Tool use: it calls the right tool with the right arguments, and doesn't call tools it doesn't need.
• Tone: it stays consistent under pressure, including with rude or impatient users.
• Recovery: it handles a tool error, a timeout or its own mistake without inventing an answer.
• Audience: it adapts to who's asking (a first-time visitor or a returning customer, a beginner or an expert).
• Ambiguity: it asks a clarifying question instead of guessing when the request is underspecified.
• Topic switching: it follows the user to a new subject and back without mixing up their context.
• Adversarial input: it resists prompt injection and attempts to extract its instructions (more in red-teaming your LLM agent).

Each case is a small fixture. Mine look roughly like this. It's a generic shape, not tied to any framework:

# scenario fixture (illustrative shape)
id: store-assistant-memory-03
dimension: memory
persona: returning customer, terse, already gave shoe size in turn 1
turns:
  - user: "Hi, I wear a 42. Looking for trail shoes."
  - user: "Something waterproof."
  - user: "Ok, which of those come in my size?"
expect:
  rules:
    - no_question_about: shoe_size
    - tool_called: search_catalog
    - tool_arg_equals: { size: 42 }
  judge:
    - "The final reply only lists items available in the size the user gave earlier."

The expect block mixes rules and judge criteria, as in the scoring table above. A fixture with no rules at all is a red flag: it usually means I haven't thought hard enough about what "correct" looks like.

Simulated Customers: Personas for the User Side

Scripted multi-turn cases have a blind spot: the user side never adapts. Real users react to what the agent says. They get confused, they push back, they wander off. The top layer of the gym handles that with simulation. One LLM plays the customer, the real agent runtime answers (with real tools, or tools stubbed at the network boundary), and a third model judges the finished transcript.

The simulated customer is only as good as its persona. I write personas as short behavioural briefs with a goal and a stopping condition. For a generic store assistant, a starter set looks like this:

• The window-shopper. Browsing with no intent to buy today and asks broad questions. This tests whether the agent stays helpful without becoming pushy.
• The impatient buyer. Knows exactly what they want and gets annoyed by clarifying questions. This tests whether the agent can be brief and still get the details right.
• The off-topic one. Keeps drifting to unrelated subjects and occasionally asks for something the agent shouldn't do. This tests scope control and the quality of refusals.
• The confused returner. Gives contradictory information across turns. This tests memory and clarification.

Three things I learned the hard way:

1. Give the simulator an exit. Without a "you've reached your goal, end the conversation" or "you've given up" condition, simulated customers go on forever, politely.
2. Keep the simulator dumber than the agent. A simulator running on a stronger model than the agent tends to solve the task for it ("Maybe you could check the catalog tool?").
3. Log the tool calls next to the transcript. The judge needs to see what the agent did, not only what it said, or it will reward confident hallucinations.

Simulation is the most expensive layer, so I run a few personas across a few goals, N times each, and save the big matrix for release candidates.

Classifier Confidence You Can Trust: Logprobs, Not Self-Report

Many conversational agents start with a classification step, such as intent, language or "is this in scope?". It's tempting to ask the model to output a confidence score alongside the label. Don't. A model's stated confidence ("confidence: 0.92") is just more generated text, and it's poorly calibrated.

Where your provider exposes token log probabilities, use those. If the classifier answers with a single label token, the probability of that token is a far better confidence signal than anything the model says about itself. The Vercel AI SDK surfaces logprobs for providers that support them through provider-specific options. Check your provider's docs, because not every model offers them.

In the gym, that gives me two extra classifier evals:

• Calibration: among the cases where the classifier is "90% confident", is it right about 90% of the time?
• Threshold behaviour: below the confidence threshold, does the system take the safe path (ask a clarifying question or route to a human) instead of guessing?

The second one ties directly into the guardrail design in separate talking from acting. Low confidence should never lead to an action.

From Incident to Fixture: The Habit That Keeps the Gym Honest

An eval gym that only contains cases you imagined will drift away from reality. The habit that fixes this is simple and non-negotiable for me: every production bug becomes a fixture before it becomes a fix.

The "fixture fails on current agent" check matters. If the new fixture passes before you've fixed anything, it doesn't reproduce the bug, and you're about to add a test that proves nothing. It's the same discipline as writing a failing test first, applied to stochastic systems. If you work spec-driven, the fixture also works as the acceptance criterion for the fix.

Over time, the incident-derived fixtures become the most valuable part of the gym, because they encode failures that actually happened rather than ones you guessed at.

Tooling: Use What's There

You don't need a platform to start. The pieces that matter are a dataset of fixtures, a runner that can repeat suites, and somewhere to store results so you can compare runs. Public options I've used or studied:

• Evalite: a lightweight, open-source TypeScript eval runner that fits well alongside a Vercel AI SDK agent.
• Langfuse and LangSmith: both have evaluation datasets, experiment comparison and trace-linked scores, which helps when you want to jump from a failed eval straight to the trace.
• Your own agent repo. My public langgraph-ts-agents and local-rag-chatbot labs are small enough to wire a first eval suite into. They're good practice before you do it on something that matters.

Whichever runner you pick, add the N-run wrapper yourself if it doesn't have one. It's the feature that matters most and the one tools most often leave out.

A Starter Checklist for Your First Week

If you're starting from zero, here's the order I'd follow:

1. Day 1: Collect 30 real or realistic conversations. Pull them from logs if you have them, or write them yourself if you don't. Tag each with the dimension it exercises.
2. Day 1: Write the classifier suite. Inputs with expected labels, scored by exact match. It's the cheapest layer and it pays off immediately.
3. Day 2: Write 5–10 policy rules as deterministic checks. Forbidden phrases, required disclaimers, "no price without a tool call" and so on.
4. Day 2: Add one LLM judge criterion. Pick the one rule you can't express deterministically. Calibrate it by hand-labelling 20 outputs.
5. Day 3: Wrap everything in an N-run harness. Five runs, report mean and standard deviation, and flag cases that pass only some of the time.
6. Day 3: Record a baseline. Run the gym on the current agent and save the numbers. Every future change gets compared to this.
7. Day 4: Write multi-turn fixtures for at least 5 dimensions. Memory, handoff, refusal, tool use and recovery are the best place to start.
8. Day 5: Build one simulated persona and one goal. Get the loop working end to end before adding more.
9. Ongoing: Turn every incident into a fixture. No exceptions.
10. Ongoing: Put the base layers in CI. Classifier and policy suites on every PR, and the full gym before releases.

Conclusion

An eval gym won't make your agent good. What it does is let you see whether a change made the agent better, worse or no different, and that's what lets you iterate quickly without breaking things. The four layers each catch a different class of failure. Rules-first scoring keeps the judge honest. N-run averaging stops you mistaking noise for progress. And the incident-to-fixture habit keeps the whole gym tied to what users actually do.

Start small: a classifier suite, a handful of rules and a five-run harness. Once you can trust the numbers, you'll be comfortable changing the instructions much more often than before.