AI Agents
Evaluating an agent before shipping
A 100-example eval set beats a demo, every time. Here’s how we build one.
Published by CloudFixies · 09 December 2025
The eval set
Fifty real user requests. Twenty edge cases (missing data, permission denied, ambiguous phrasing). Twenty adversarial prompts. Ten cases where the correct answer is "I can’t help with that."
The score
Not a single number. A dashboard: correctness, tool-call accuracy, refusal rate on adversarial, latency p95. Regress on every one before shipping.
◆ Takeaway
A hundred labelled examples, four scores. Ship when all four hold.