WTF are evals? How to test agents and the software they use

Agents now act as users of software, performing tasks on behalf of people, which requires new testing methods. Evals provide a way to measure how effectively agents complete these tasks by simulating user workflows and grading outcomes. Unlike traditional software testing, evals focus on whether the agent’s actions align with the user’s intent, such as ordering an iced latte instead of a hot one. This approach addresses the variability of LLM outputs, which can produce different results even when the underlying task appears successful in logs.

Evals categorize tests by the number of LLM calls required: deterministic tests (0 calls), one-turn evals (1 call), and end-to-end (E2E) evals (2+ calls). E2E evals are particularly useful for assessing multi-step tasks, such as ordering coffee, where individual steps may seem correct but the final outcome is wrong. Frameworks like Harbor, an open-source tool from Terminal-Bench creators, automate the process by running agents in containers, recording their actions, and verifying results against predefined criteria.

Building comprehensive E2E evals for all possible agent behaviors is impractical due to the vast surface area of potential LLM interactions. Instead, Vercel recommends starting with a single high-impact outcome users care about, such as ensuring agents order the correct drink or meet budget constraints. This targeted approach helps identify critical failures without requiring exhaustive testing coverage.

Vercel highlights real-world examples of agent failures, such as ordering a hot latte instead of an iced one or miscalculating totals including tax and tip. Observability data shows most agent traffic comes from coding tools like Claude Code and Codex, while customer support tickets reveal common issues like incorrect drink orders or missed pickup times. These insights guide the development of focused evals to address the most frequent and impactful problems.