The main public AI agent benchmarks are τ-bench (tau-bench), WebArena, GAIA, SWE-bench, AgentBench and OSWorld. Each measures whether an AI agent can finish a defined task in a controlled environment: a customer-service conversation, a working website, a research question, a code repository or a desktop computer. None measures whether an agent secured good commercial terms for the party it represents, which is the core of media buying.

TL;DR: General agent benchmarks score task completion against a known end state, and they have exposed real weaknesses such as inconsistency across repeated attempts. Media buying adds a counterparty with its own interests, a budget and contract terms that can be weak even when the task counts as a success. A media-buying benchmark has to score the deal itself, not only whether a deal was made.

What is an AI agent benchmark?

An AI agent benchmark is a fixed set of tasks, an environment the agent acts in, and a scoring rule that decides whether each attempt succeeded. Unlike a question-and-answer test, it lets the system take actions such as calling tools, browsing or editing files before the result is checked. The score reflects the whole system, including the prompts and software around the model.

Most agent benchmarks check the end state rather than the reasoning. τ-bench compares the database state at the end of a conversation with the annotated goal state, and SWE-bench reports the percentage of task instances resolved. That makes scoring reproducible, but anything outside the end state goes unmeasured.

What are the main AI agent benchmarks and what do they measure?

The six benchmarks below are among the most cited public tests of agent behaviour, each described from its own paper or site. Headline figures come from the original papers, so they describe the agents tested at publication rather than current systems.

Benchmark	Lead authors	What the agent does	How success is scored
τ-bench	Yao et al., 2024	Serves a simulated customer under airline or retail policy rules	Final database state against a goal state, plus pass^k
WebArena	Zhou et al., 2023	Completes tasks on working websites	Functional correctness of the result
GAIA	Mialon et al., 2023	Answers questions needing browsing and tool use	Match with a ground-truth answer
SWE-bench	Jimenez et al., 2023	Edits a Python codebase to resolve a GitHub issue	Percentage of issues resolved
AgentBench	Liu et al., 2023	Acts in eight code, game and web environments	Success rate and per-environment metrics
OSWorld	Xie et al., 2024	Operates real desktop and web apps	An execution-based script per task

1. τ-bench: rules, users and reliability

τ-bench (Yao et al., 2024) puts an agent in a conversation with a user simulated by a language model and gives it domain-specific API tools and policy guidelines. Its authors argued that "existing benchmarks do not test language agents on their interaction with human users or ability to follow domain-specific rules". The paper introduced pass^k, which asks whether an agent succeeds on the same task across k trials, and found leading function-calling agents succeeded on fewer than 50% of tasks, with pass^8 below 25% in retail.

The project's repository now points users to τ³-bench, which adds a banking domain and voice evaluation. The limit for advertising is that the simulated user wants what the agent is there to deliver, so nobody is bargaining against it.

2. WebArena: tasks on working websites

WebArena (Zhou et al., 2023) is a self-hosted environment of functional websites across e-commerce, social forums, collaborative software development and content management. The paper released 812 long-horizon tasks, each scored on the functional correctness of the result. The best agent its authors tested reached an end-to-end success rate of 14.41%, against 78.24% for humans.

WebArena tests whether an agent can operate an interface to reach a goal. It does not ask whether the goal was a good one.

3. GAIA: questions that are easy for people

GAIA (Mialon et al., 2023) is built from real-world questions that need reasoning, multimodal handling, web browsing and tool use. Its authors describe them as "conceptually simple for humans yet challenging for most advanced AIs", and reported human respondents at 92% against 15% for a plugin-equipped language model. The set holds 466 questions across three difficulty levels.

GAIA rewards one correct answer. Media buying rarely has one, because several portfolios can satisfy the same brief with different trade-offs.

4. SWE-bench: real issues in real codebases

SWE-bench (Jimenez et al., 2023) contains 2,294 problems drawn from real GitHub issues and pull requests across 12 Python repositories. An agent must edit the codebase to resolve each issue, often with coordinated changes across several files. On release, the best model tested resolved 1.96% of issues.

The SWE-bench leaderboards now include SWE-bench Verified, a human-filtered subset of 500 instances. The tasks have an objective check but no counterparty and no budget.

5. AgentBench: breadth across eight environments

AgentBench (Liu et al., 2023) tests language models as agents in eight environments grouped as code (operating system, database, knowledge graph), games (digital card game, lateral thinking puzzles, house-holding) and web (shopping and browsing). The current paper examines 29 API-based and open-source models. Its authors name poor long-term reasoning, decision-making and instruction following as the main obstacles to usable agents.

That breadth helps compare general agent ability, but no single environment resembles a negotiated commercial deal.

6. OSWorld: a real computer

OSWorld (Xie et al., 2024) gives multimodal agents a real computer across Ubuntu, Windows and macOS, with 369 tasks spanning web and desktop apps, file operations and multi-application workflows. Each task has its own initial-state setup and execution-based evaluation script. The authors reported humans completing more than 72.36% of tasks, against 12.24% for the best model tested.

What do general agent benchmarks leave out?

The first gap is the scoring itself. Kapoor et al. (2024), in "AI Agents That Matter", found "a narrow focus on accuracy without attention to other metrics" across agent benchmarks and argued for measuring cost alongside accuracy. They also found many benchmarks have inadequate holdout sets, which lets agents take shortcuts and overfit.

The second gap is the counterparty. In τ-bench the simulated user and the agent want the same result, and none of the other five asks the agent to negotiate terms. Media buying is a two-sided trade, so an agent can finish the task and still give away value.

The third gap is the contract. A resolved issue is finished, but a media deal carries terms such as make-goods and viewability floors that decide how well protected the buyer is after it closes.

Why does media buying need its own benchmark?

Media buying needs its own benchmark because a completed buy and a good buy are different outcomes, and a general benchmark only sees the first. A buying agent must read a brief, negotiate with sellers whose interests differ from its own, stay within budget, reach the required audience and secure protective terms. Each step can fail on its own.

Concourse Bench v1, a media-buying benchmark published by Alkimi, is one worked example. It gave eight model-driven buyers, built on OpenAI and Anthropic models, one fictional brief and budget, and had each negotiate with three sellers run by fixed software policies across four buying situations, three attempts per situation. The buyer harness, the software that supplies the brief, checks each action and records the deals, was evaluated with the model as one system.

The results show why completion alone misleads. 55 of 96 registered attempts ended in a completed buy, but across the 110 contracts signed in those buys, 95 promised a viewability floor below 70%. A pass or fail score would have counted all 55 as successes without showing how weakly many protected the buyer.

What should a media-buying agent benchmark measure?

A media-buying benchmark should score the deal on separate measures, because a single number hides the trade-offs a buyer cares about. This checklist draws on the gaps above.

Completion against the brief: a portfolio that fits the budget and meets every requirement, not just signed contracts.
Contractual protection: make-goods and viewability floors, scored apart from completion.
Reliability: the same brief run several times, in the spirit of pass^k.
Separate failure modes: negotiation, portfolio assembly and technical interruptions reported apart.
A human baseline: the same briefs given to experienced buyers. Concourse lists this as planned but not yet measured, so its completion rate is recorded rather than graded.
Held-out briefs: tasks the developers have not seen, to limit overfitting.

Teams preparing an agent for a specific buying workflow can request a scoping conversation through the Concourse form. It starts with the brief, the constraints and what a good outcome means, rather than with a run.

What comes next for media-buying agent benchmarks?

The next step is studying both sides of the trade. Concourse has published a versioned roadmap: v2 tests buying quality against fixed sellers, v2.1 is a seller-side benchmark, and v2.2 will study model-driven buyers and sellers together.

Alkimi, which publishes Concourse, also runs a Gym where buyers and sellers build their own agents, set the rules those agents work to and spar against counterparty agents before any real money moves. Teams that want to see how their buying agent handles a negotiated brief can start with a scoping conversation.

Frequently asked questions

What are the main AI agent benchmarks?

The most cited public benchmarks include τ-bench, WebArena, GAIA, SWE-bench, AgentBench and OSWorld. Each tests whether an agent can complete defined tasks in a controlled environment.

What does pass^k mean?

pass^k is a reliability metric introduced with τ-bench that asks whether an agent succeeds on the same task across k independent trials. An agent that succeeds only some of the time scores far lower on pass^8 than on a single run.

Can a general benchmark predict how well an agent will buy media?

Only partly, because general benchmarks test completion without a counterparty, a budget or contract terms. An agent can complete a buy and still accept weak protections.

Do agent benchmarks test the model or the whole system?

Most test the whole system: the model plus its prompts, tools and surrounding software. A score therefore describes that combination, not the model alone.