Several published benchmarks now test AI agents on advertising work, but each is run by a company or research team with its own scope, and none of those reviewed here is an industry-accredited standard for agents that negotiate and buy media. The useful test for a buyer is whether a benchmark publishes its method, reports every failed run and measures the quality of the deal as well as its completion.

TL;DR: General agent benchmarks such as Sierra's τ-bench and Andon Labs' Vending-Bench show that reliability over repeated and long runs is the hard part for AI agents. Advertising-specific benchmarks now cover account analytics and paid-media decisions, and Concourse Bench covers negotiated buying through to a committed portfolio. No single benchmark settles whether an agent is ready to spend real money, so buyers should read each one for what it tests and what it leaves out.

Is there an independent benchmark for AI media-buying agents?

Several benchmarks for advertising agents are published openly, with tasks, methods and results that anyone can read. Independence is a separate question: several of the best-known agent benchmarks are built by companies that also sell AI products, which is common in AI evaluation.

A buyer should therefore judge a benchmark on three things. Is the method published, are failures reported alongside successes, and can outsiders inspect or rerun the test?

What benchmarks exist for AI agents in advertising?

What is a benchmark for an AI buying agent?

A benchmark is a fixed, repeatable test: the same tasks, rules and scoring applied to every agent so that results can be compared. For a buying agent, the task is usually a brief, a budget and a set of sellers or platform data, and the score records what the agent did and agreed. The fixed setup is what separates a benchmark from a demo.

Why can't general AI benchmarks answer the media-buying question?

General agent benchmarks test transferable skills such as tool use, rule-following and staying coherent, not the commercial judgement a media brief demands. Sierra's τ-bench, for example, covers retail and airline customer service tasks. A model that scores well there has shown it can follow policy through a conversation, not that it can secure sensible contract terms from three sellers.

Which advertising-specific benchmarks have been published?

AdsBench, published by Soku AI in August 2026, gives fifteen models the same six tasks on one frozen Google Ads account, with up to 30 runs each. AD-Bench, an academic benchmark, is built from real marketing-analysis requests on a production advertising platform. MediaBuyerBench is an open benchmark that scores what it calls source-grounded decision safety on static paid-media cases.

All three focus on analysis and recommendations inside platform data rather than on negotiating and committing a buy. On AD-Bench, the best-scoring model answered 76.9% of requests correctly at the first attempt overall, falling to 61.4% on the hardest tier.

Is there a benchmark that tests negotiation and buying?

Concourse Bench, a versioned benchmark published by Alkimi, tests whether a model-driven buyer can negotiate with three sellers, stay within one budget and commit a portfolio that meets the brief. In v1, eight models were tested across four buying situations, with three registered attempts in each. The sellers were fixed software policies rather than language models, which the benchmark states openly.

In Concourse Bench v1, 55 of 96 registered attempts ended in a completed buy. Across the 110 contracts signed in those completed buys, 95 promised a viewability floor below 70%.

Is a live pilot the same as a benchmark?

No: a pilot shows that one agent worked in one campaign, while a benchmark compares agents under identical conditions. Butler/Till's first agentic buying test with PubMatic, reported by Digiday, ran in two flights from December 2025 into January and was independently audited by verification firm Jounce.

Gina Whelehan of Butler/Till described the aim as a proof of concept: "Does this actually work? Can we effectively get a campaign in market, and it runs, and it scales?" Pilots and benchmarks answer different questions, and both are useful.

What should a benchmark for a buying agent measure?

What is the difference between completing a buy and doing a good one?

Completion records that the agent finished the workflow, while quality records whether the terms it agreed protect the buyer. The Concourse Bench FAQ puts it directly: "Completion and contractual protection are separate measures." A benchmark that reports only a pass rate hides that gap, which is the argument made in Beyond completion: the four measures that make AI media buying evaluable.

Why does reliability across repeated runs matter?

An agent that succeeds once may fail on the next identical attempt, and a buyer lives with every attempt. Sierra's τ-bench introduced a pass^k metric that measures success across k repeated runs, and in its early results Sierra reported that even state-of-the-art agents succeeded in fewer than 50% of tasks and only about 25% when repeating the same task eight times. Repeated attempts on the same brief should be standard in any buying benchmark.

Should a benchmark test the model or the whole agent?

The whole agent, because a model never buys media on its own. Concourse Bench evaluates the model together with its buyer harness, the software that supplies the brief, checks every action and records the deals. The case for treating them as one system is set out in Why the buyer harness matters as much as the model.

Why do the sellers in a benchmark matter?

A buying agent's result depends on who it negotiates against. Fixed software sellers make runs comparable, while sellers that bargain hard or behave badly test resilience. Andon Labs built that second condition into Vending-Bench 2, where suppliers may be adversarial and negotiation is described as key to success.

How long should a benchmark run last?

Long enough to expose the failures that only appear over time. In the original Vending-Bench paper, every model tested had runs that broke down, through misreading delivery schedules, forgetting orders or falling into unproductive loops, and the authors found no clear link between those failures and a full context window. Media campaigns run for weeks, so a benchmark that tests a single negotiation leaves the in-flight phase untested.

How should failed and interrupted runs be reported?

Every registered attempt should stay in the denominator, with technical interruptions shown separately rather than quietly removed. Concourse Bench reports registered and assessed attempts side by side for this reason. A completion rate calculated only on the runs that finished cleanly will always flatter the agent.

How should buyers read and use benchmark results?

Does a benchmark need a human baseline?

Yes, because a completion rate means little without knowing how a human buyer would do on the same brief. Concourse Bench says in its own FAQ that a human comparison is not yet available, is planned, and is the next thing needed before completion rates can be read as good or bad rather than simply recorded.

Can a benchmark go out of date?

Yes, in two ways. Models change quickly, so a result describes a specific model version at a specific time, and live platform data shifts: the AD-Bench authors built a dynamic ground-truth pipeline because fixed answers "quickly become obsolete as data and platform rules evolve".

Versioning helps. Concourse treats v2 as a new benchmark with different tasks and rules rather than a rerun of v1.

Who should run the benchmark: the vendor, the buyer or a third party?

Each has a role. Vendors often know the task best, buyers can test agents against their own briefs, and outside parties can verify outcomes, as Jounce did in the Butler/Till test. The NIST AI Risk Management Framework, released in January 2023 for voluntary use, is intended to help organisations build trustworthiness into the design, development, use and evaluation of AI systems.

How can a buying team benchmark its own agent?

Start with the brief, the constraints and what a good outcome means, then run the agent repeatedly against the same sellers and record every attempt. Teams that want an outside evaluation can request a Concourse scoping conversation, which starts with the brief rather than a run. Comparing a custom agent with general-purpose models is covered in Is your AI buying agent better than GPT or Claude?

Where does this leave a buyer choosing an agent?

Treat any single benchmark as evidence about one setup, and look for repeated runs, published failures and a measure of deal quality before trusting the headline. Alkimi publishes Concourse Bench and scopes evaluations for teams automating a buying workflow, and the scoping form asks about the brief, the constraints and what a good outcome looks like. Its Gym lets buyers and sellers build their own agents, set the rules they work to and spar against counterparty agents before any real money moves.

Frequently asked questions

Can sell-side agents be benchmarked too?

Yes, and the same principles apply: fixed buyers, repeated runs and measures of what the seller agreed. The Concourse roadmap includes a seller-side benchmark, v2.1, and no measured seller-model results have been published yet.

Do AAMP and AdCP help with benchmarking?

IAB Tech Lab's Agentic Advertising Management Protocols (AAMP) include open-source buyer and seller agent reference implementations, and the Ad Context Protocol (AdCP), published by AgenticAdvertising.org, is an open protocol for agents to discover, plan, buy, sell and measure media. Neither is a benchmark, but shared protocols make it easier to run the same test against agents built by different teams.

What should a buyer ask before trusting a benchmark result?

Ask who built it, whether the method and every failed run are published, and whether it measured deal quality as well as completion. A practical sequence for pre-launch testing is in How to test an AI media buyer before you deploy one.

Should a benchmark score decide how much autonomy an agent gets?

It should inform the decision rather than make it. A sensible path moves an agent through stages of earned autonomy, from observe and recommend to draft, human-approved action and finally bounded automatic action, with evidence required at each step.