The Forge
An eval is a test for your agent. You give it an input, run it, and apply grading logic to what comes out. That's the whole idea, and almost nobody building agents has one. The result is a familiar loop: you tweak a prompt, someone says "it feels worse," and you have no way to check except to guess and try again. You're flying blind, and your users are the ones who find the regression.
We've built the runtime guardrails already. Issue #13 gave a Managed Agent a grader that rereads its own work against a rubric and revises until it passes. That's a live check on a single run. An eval suite is a different tool for a different moment: it runs your agent against a bank of saved cases before you ship, so you catch the regression in development instead of in production. One grades the work as it happens. The other tells you whether your last change made the agent better or worse across a hundred scenarios at once.
The common take is that evals are heavy infrastructure you build later, once you're at scale, so early on you skip them. That's backwards, and it's the mistake that costs teams the most. Anthropic's own guidance is blunt about it: 20 to 50 simple tasks drawn from real failures is a great start, and evals get harder to build the longer you wait. Skip them now and later you're reverse-engineering success criteria from a live system, trying to remember what "good" was supposed to mean. Write them early and each product requirement turns straight into a test case.
There's a second payoff that shows up later. When a new model drops, and this month it's Claude Sonnet 5, the teams with an eval suite run it and know in an afternoon whether to upgrade. The teams without one spend weeks testing by hand while the first group already shipped.
The short version: an eval suite turns "the agent feels worse" into a number, and 20 tasks you write tonight beat the perfect suite you never get around to building.
The Blueprint
Three pieces: a balanced task set, the graders that score it, and a runner that gives you a pass rate. Here's a harness small enough to read in one sitting. Plug your own agent into run_agent and it works.
Step 1: write tasks, and balance them. A task is an input plus a success criterion. The part everyone gets wrong is testing only the cases where a behavior should happen. Test both sides, or you get one-sided optimization. Anthropic hit this building web search for Claude: if you only check that the agent searches when it should, you train an agent that searches for everything. So you pair "should search" with "should answer from memory."
import anthropic
client = anthropic.Anthropic()
# A balanced task set. Test where a behavior SHOULD happen and where it SHOULDN'T.
# Pull these from real failures. Aim for 20 to 50 to start.
TASKS = [
{"id": "weather_search", "input": "What's the weather in Denver right now?", "should_search": True},
{"id": "founder_no_search", "input": "Who founded Apple?", "should_search": False},
{"id": "news_search", "input": "What shipped in the MCP spec this week?", "should_search": True},
{"id": "math_no_search", "input": "What's 18% of 240?", "should_search": False},
]Step 2: pick graders. The rule is deterministic where you can, an LLM judge where you can't. A deterministic grader is fast, cheap, and objective: did the agent use search exactly when it should have? An LLM judge handles the nuance a string match can't, like whether the answer is actually any good. Two things that matter here. Grade what the agent produced, not the exact path it took, because agents find valid routes you didn't script and rigid step checks punish that. And give the judge a way out, an "Unknown" option, so it doesn't hallucinate a verdict when it can't tell.
# Deterministic grader: did it search exactly when it should have?
def grade_search(task, result) -> bool:
used = "web_search" in result["tools_used"]
return used == task["should_search"]
# LLM-judge grader: is the answer actually correct and complete?
def grade_quality(task, result) -> bool:
verdict = client.messages.create(
model="claude-sonnet-5",
max_tokens=8,
messages=[{"role": "user", "content": (
f"Question: {task['input']}\n"
f"Answer: {result['answer']}\n\n"
"Is the answer correct and complete? Reply with one word: "
"PASS, FAIL, or UNKNOWN."
)}],
).content[0].text.strip().upper()
return verdict == "PASS"Step 3: run it and count. Your agent's output varies run to run, so one trial per task tells you almost nothing. Run each task a few times and report the rate. Here run_agent is your real harness, the same one you ship, returning the answer and the tools it actually called.
def run_agent(user_input: str) -> dict:
# Your production agent goes here. It must match what real users hit.
# Return the final answer and the tool names it called.
...
return {"answer": "...", "tools_used": ["web_search"]}
def run_suite(trials: int = 3):
total_pass, total = 0, 0
for task in TASKS:
passes = sum(
grade_search(task, r) and grade_quality(task, r)
for r in (run_agent(task["input"]) for _ in range(trials))
)
print(f"{task['id']:20} {passes}/{trials}")
total_pass, total = total_pass + passes, total + trials
print(f"\npass@1 rate: {total_pass / total:.0%}")
run_suite()That's the whole harness. Four tasks, two graders, a runner. It prints a per-task score and an overall pass rate you can watch move every time you change a prompt or a model. Grow the task list from there, one real failure at a time.
The Anvil
Now the part that separates a suite you trust from a number that lies to you.
A one-sided task set optimizes one side. This is the failure worth repeating, because it's the one that quietly wrecks an agent. If every task tests where the behavior should fire and none test where it shouldn't, a passing score just means the agent does that thing eagerly, not correctly. The fix is in the task set, not the grader: for every "should search," write a "should not," and the same for every tool, escalation, or action your agent can take.
Grading the path instead of the result. There's a strong pull to assert that the agent called these exact tools in this exact order. Resist it. Agents regularly find valid solutions the eval author didn't anticipate, and a rigid trajectory check fails them for being creative. Grade the outcome, did it end up searching or not, did the refund get processed, not the specific sequence of steps it took to get there.
Reading a 0% as a dumb agent. When a task fails every single trial with a capable model, the problem is almost always the task, not the agent. Anthropic saw a benchmark score jump from 42% to 95% after someone fixed rigid grading that rejected "96.12" when it wanted "96.124991," plus a handful of ambiguous specs. A task where two experts couldn't independently agree on pass or fail is a broken task. Write a reference solution that you know should pass, and if the grader fails it, fix the grader.
Trusting the score without reading transcripts. The number is not the point. The transcript is. When a task fails, only the transcript tells you whether the agent made a real mistake or your grader rejected a good answer. Read them, especially early. A score you haven't traced back to actual behavior is confidence you haven't earned.
The rule of thumb: capability evals start with a low pass rate and give you a hill to climb, and regression evals sit near 100% and act as an alarm when something breaks. If your whole suite already passes, it can only catch backsliding, so add harder tasks for signal. If your whole suite fails, your tasks are probably broken, not your agent. A healthy suite has some of each, and you read the transcripts either way.
Sparks
A few more things worth your attention this week:
- Claude Sonnet 5 shipped, billed as the most agentic Sonnet yet, with introductory pricing of $2 per million input tokens and $10 per million output through August 31. This is exactly the moment an eval suite earns its keep: point it at the new model and you'll know by tonight whether the upgrade helps your agent or quietly regresses it.
- Claude Cowork expanded past the desktop to web and mobile in beta, with sessions that move between devices and tasks that keep running in the cloud while your machine is off. The autonomy keeps climbing, which is one more reason to have tests in front of it.
- Claude Code 2.1.208 landed, adding a screen-reader mode claude --ax-screen-reader), vim insert-mode remaps, and fixes for memory leaks in long sessions. Small, but the long-session memory fix matters if you run big agent jobs locally.
- When a single script stops scaling, the eval frameworks worth a look are Harbor (containerized trials, ships Terminal-Bench 2.0), Braintrust (offline evals plus production monitoring), and Arize Phoenix (open-source tracing and evals). All of them are only as good as the tasks you feed them, so start with the script.
- An eval tells you the agent got the right answer. It doesn't show you the messy path it took to get there. Seeing inside a live run, every reasoning step and tool call as one readable trace, is its own technique, and we may build one in a future issue.
The Smith's Take
Teams without evals live in a reactive loop: fix one failure, cause another, and never quite know if today's change helped or hurt. Teams with evals get the opposite, where every failure becomes a test case, every test case blocks a future regression, and "the agent feels worse" becomes a number on a dashboard you can actually act on. The cost is real and it's upfront. The payoff compounds, and it's easy to skip precisely because you pay first and collect later.
You don't need a framework or a hundred tasks to start. Take the last three times your agent embarrassed you, the answer it botched, the search it should have run, the tool it fired when it shouldn't have, and write each one as a task with a single pass-or-fail check. Run them tonight with the harness above. That's your suite. It'll be small and a little ugly, and it will still tell you more about your next change than any amount of staring at the prompt.
Build agents that actually work.