AI media buyers cannot yet be ranked against human media buyers with any confidence, because the benchmark data published so far measures agents on their own. A completion rate or a set of signed contract terms only becomes good or bad once people have done the same job under the same rules. Until that human baseline exists, a claim that agents buy better, worse or faster than traders is an assumption rather than a finding.

TL;DR: A benchmark for AI media buyers needs a human baseline because a raw score has no reference point. A useful baseline puts traders through the same brief, budget, sellers and permitted actions as the agent, then grades both on completion, contractual protection and time. It must also say which humans were measured, because research on AI evaluation shows that the choice of comparator changes the answer.

How do AI media buyers compare with human media buyers today?

The honest answer is that the comparison has not been run on equal terms. The measurements that exist describe what an agent did, without saying whether a person would have done better or worse.

Concourse Bench v1, the benchmark for AI media-buying agents that Alkimi publishes, is a concrete example. In its first version, 55 of 96 registered runs completed a buy, and 95 of the 110 contracts signed in those completed buys promised a viewability floor below 70%. Asked whether there is a human baseline to compare against, the Concourse FAQ answers: "Not yet. It's planned, and it's the next thing we need to make these numbers meaningful."

That candour is the right starting point. Whether agents will replace buyers is a separate question, covered in Will AI Agents Replace Media Buyers? The Evidence-Based Answer. This explainer deals with the narrower measurement problem underneath it.

Why does an AI benchmark score need a human reference point?

A benchmark score is a count, and a count has no direction until something sits beside it. Whether 55 completions from 96 attempts is impressive depends on how often a competent trader would finish the same brief with the same budget and sellers.

AI research has learned this the hard way. When the GAIA benchmark for general AI assistants was published in November 2023, its authors reported that human respondents scored 92% against 15% for GPT-4 equipped with plugins, on questions they described as "conceptually simple for humans yet challenging for most advanced AIs". Without the human column, the 15% would have read as a low score on a hard test rather than a large gap on an easy one.

METR uses people as the yardstick in a different way, measuring tasks by how long they take humans with the right expertise. In its March 2025 analysis, models of the time had almost 100% success on tasks taking humans under four minutes but succeeded less than 10% of the time on tasks taking more than around four hours. The human time is what turns a pass rate into something a manager can plan around.

Media buying has the same problem with a commercial edge. A promised viewability floor below 70% might be a weak outcome or a normal one for that inventory, and only a human run on the same brief can say which.

What should a human baseline for AI media buying measure?

A human baseline should measure the same things as the agent run, on the same tasks, so that the only variable is who does the buying. A review of 115 human baselines in AI model evaluations by Kevin Wei and colleagues, accepted to ICML 2025 as a position paper, recommends using the same test set for humans and AI and comparing results only on that shared set.

For media buying, that means holding four things constant between the trader and the agent: the brief, the budget, the counterparty sellers and the permitted actions. The measures are then whether the portfolio was completed within budget and brief, what protection the contracts secured (make-goods, viewability floors and similar terms), and how long the job took. Beyond completion: the four measures that make AI media buying evaluable sets out why completion alone hides contract quality.

Time needs particular care. Concourse reports elapsed minutes per brief but labels them "Wall time, not human effort" and states that "No human comparison was measured". A fair baseline records human time the same way, from receiving the brief to the last signed contract, rather than setting an agent's run time against a person's working day.

Which humans should an AI media buyer be compared with?

The comparator has to be named, because "human" is not a single level of skill. The Wei review asks evaluators to specify which subset of humans a baseline is meant to represent, and accepts convenience samples for expert baselines provided the eligibility criteria are clearly defined. METR makes the same point about its own figures: "Our estimate of the length of tasks that an agent can complete depends on methodological choices like the tasks used and the humans whose performance is measured."

In media buying, a junior trader on a first CTV brief and a senior negotiator with long publisher relationships would set very different bars. OpenAI's GDPval benchmark shows one way to handle expertise, building its tasks from the work of industry professionals with an average of 14 years of experience across 44 occupations. A media-buying baseline would probably need at least two tiers, so that a buyer can see whether an agent sits nearer the new hire or the veteran.

Why is a human baseline harder to build than it looks?

People disagree with each other, and a baseline has to show that spread rather than hide it. In GDPval's blinded pairwise comparisons, the authors report human expert inter-rater agreement of 71%, so even the grading of good work carries noise.

Briefs add a second difficulty. The GDPval authors found that model performance was worse on under-specified prompts, noting that "the models struggled to figure out context", a familiar condition for anyone who has received a one-line media brief. A trader would pick up the phone to clarify, so the baseline has to decide whether clarification is allowed and give the agent the same option.

The third difficulty is documentation. The Wei review treats the publication of tasks, metrics and experimental materials as a stage of the baseline in its own right, because a comparison nobody can inspect cannot be reproduced. Teams already running a buying agent do not have to wait for a published baseline to test it against their own briefs: Concourse takes scoping requests that start with the brief, the constraints and what a good outcome means.

How should buyers read AI versus human claims until then?

Any claim that an agent beats, matches or saves a human is a methods question first. Ask who the human comparator was, whether they worked the same brief with the same budget and sellers, who graded the results and whether the grading was blind.

Live demonstrations are valuable, and they are a different thing from a baseline. Digiday's Tim Peterson, for example, built an interactive game in which Gemini-powered AI agents represented buyers and sellers and asked Wpromote's Skyler McGill and Butler/Till's Ryan Lammela to test it, which shows how a negotiation between people and agents feels from the inside. A baseline adds repetition, controlled conditions and a published scoring rule.

The same discipline works inside a single team. A trader and an agent given the same historical brief, budget and permitted actions produce two sets of signed terms that can be compared line by line, and How to test an AI media buyer before you deploy one covers the wider pre-launch checks.

What does a human baseline change for autonomy decisions?

A human baseline gives teams evidence for deciding how much autonomy an agent should hold. Earned autonomy moves an agent through five stages (Observe, Recommend, Draft, Human-approved action and Bounded automatic action), and each step up should rest on proof that the agent's work holds up against a person doing the same job. What are the five stages of earned autonomy in AI media buying? explains how teams move between them.

Without a human comparison, a team promoting an agent from Draft to Human-approved action is judging it against an imagined trader. With one, the decision rests on the same brief, the same sellers and the same contract terms.

Where does Concourse go from here?

Alkimi built Concourse to make differences between AI buyers visible, and its published roadmap moves from buyer completion to buying quality, a seller-side benchmark and buyers and sellers working together. Concourse V2 is currently running, and the human baseline is still planned rather than measured, so no human comparison result exists yet. Teams that want their own buying agent evaluated against a defined brief can request a scoping conversation.

Frequently asked questions

Do AI media buyers perform better than human media buyers?

The benchmark data published so far measures agents on their own, so there is no like-for-like answer yet. Concourse, for example, lists its human baseline as planned rather than measured.

What is a human baseline in an AI benchmark?

A human baseline is a set of results from people doing the same tasks under the same conditions as the AI system. It gives the AI scores a reference point, so a completion rate or contract outcome can be read as strong or weak.

Why is completion rate not enough to judge an AI media buyer?

An agent can complete a buy and still sign weak contract terms, such as a low viewability floor or thin make-goods. Completion, contractual protection and time need to be measured separately and compared with a person doing the same brief.

Which human should an AI buying agent be compared with?

The baseline should name its comparator, such as a junior trader or a senior negotiator, because the result changes with the level of expertise. Using two tiers shows where an agent sits on that range rather than against one undefined human.

Can a team run its own human baseline?

Yes, at small scale, by giving a trader and an agent the same historical brief, budget and permitted actions and comparing the signed terms. The results should be read as indicative, because a small sample cannot show how much performance varies between people.