You cannot prove an AI buying agent is better than GPT or Claude by naming the model inside it. You prove it by running your agent and a frontier model through the same media brief, against the same sellers, several times over, and scoring completion and contract quality separately. Anything less is a demo, not evidence.

TL;DR: In Concourse Bench v1, the top and bottom of the results table each included models from both providers: one OpenAI model and one Anthropic model completed 12 of 12 buys, while another model from each completed none. "Better than Claude" is therefore not a single bar. A credible claim needs a standardised brief, fixed sellers, repeated attempts and a scorecard that measures protection as well as completion.

Why is "better than GPT or Claude" the wrong question?

GPT and Claude are families of models, not single benchmarks. In Concourse Bench v1, eight model-driven buyers, four from OpenAI and four from Anthropic, each ran the same fictional media-buying brief across four buying situations with three attempts in each.

The completion results split within each provider, not between them. The OpenAI model labelled Sol completed 12 of 12 attempts and the one labelled Luna completed none; the Anthropic model labelled Fable completed 12 of 12 and the one labelled Haiku completed none.

So an agent builder who says "our agent beats Claude" has to say which Claude, under which conditions and on which measure. These figures come from one benchmark under one brief and describe how each system performed in that controlled market, not a general ranking of providers.

Concourse, the benchmark Alkimi built for AI media-buying agents, is the worked example used throughout this guide because its method and results are published in full.

Are you comparing the model or the agent?

An AI buying agent is a model plus a harness: the software that supplies the brief, checks every action and records the deals. Concourse evaluates the two as one system, not the model in isolation, because that is how buying agents are deployed.

That framing matters for anyone who has built a custom agent. Most custom agents run on a frontier model underneath, so the real claim is rarely "our model beats GPT". It is "our harness gets more out of the model than a standard setup does".

The cleanest test of that claim runs the same underlying model twice, once inside your harness and once inside a reference harness, on the same brief. Concourse's published roadmap includes a later controlled experiment designed to isolate the effect of harness changes.

What does a fair head-to-head test need?

Four design choices separate a test from a demo. Concourse Bench v1 used all four, and its method is public.

Fixed sellers. In v1, Seller A, Seller B and Seller C each ran on a fixed software policy, so a better result could not come from finding a softer counterparty. If sellers react differently to different buyers, the comparison breaks.
More than one buying situation. v1 used four: a reference brief, a compressed budget, a wider inventory pool and sellers holding firm. An agent that only completes the easy version has a narrow range.
Repeated fresh attempts. v1 registered three attempts per model in each situation, twelve per model in total. One clean run proves very little about reliability.
Interruptions reported, not hidden. Five of v1's 96 attempts could not be assessed because of technical interruptions, and the published results keep them visible in the denominator. Dropping failed runs flatters every agent and tells a client nothing.

What should the scorecard measure?

Completion is the entry ticket, not the score. Record six measures for every attempt, for your agent and for the frontier model alike:

Completion: did the agent commit a full portfolio that satisfied the completion rule?
Budget discipline: did the final commitment stay within the budget?
Brief fit: did it meet every stated requirement, including the audience reach target?
Protection secured: the make-good terms and promised viewability floor written into each contract.
Failure behaviour: when the agent could not finish, did it stop cleanly or close on weak terms?
Cost and time: model API spend and elapsed time, always stated for a named population of runs.

Concourse reports how a run can fall short along these same lines. A model can negotiate valid individual contracts and still miss the budget, miss a requirement in the brief or fail to reach enough of the required audience.

Keeping those outcomes apart stops negotiation skill, portfolio assembly and execution reliability collapsing into one pass or fail number. It also shows a client exactly where an agent is strong and where it needs a human.

Why is completing the buy not enough?

An agent can close every deal and still leave the buyer exposed. Across v1 as a whole, 55 of 96 registered attempts completed a buy, and of the 110 contracts signed inside those completed buys, 95 promised a viewability floor below 70%.

That is the gap between finishing the workflow and doing a good job. Concourse shows that two portfolios can commit identical cash and still carry different make-good terms and viewability floors, and a completion score alone rates them the same.

For an agent builder, protection is the most persuasive place to win. A harness that completes reliably while holding out for stronger contractual terms gives a client something a completion rate cannot.

Why won't your own test convince a client?

You set the sellers, the brief and the pass mark. Even a careful internal test reads as marking your own homework, and a client has no way to reproduce it.

An independent benchmark changes three things. The sellers are fixed by someone else, the method is published, and your agent's results sit beside frontier models run under the same rules.

There is also no human baseline yet. Concourse says so in its FAQ: a human comparison is planned, and it is needed before completion rates can be read as good or bad rather than simply recorded.

How do you get your agent benchmarked?

Start with the workflow, not the run. The Concourse scoping form asks which buying workflow is being automated, and the first conversation covers the brief, the constraints and what a good outcome means before anything is run.

Prepare four things before that conversation:

The workflow: which part of buying the agent owns, from brief intake to signed commitment.
A representative brief: the budget, channel, audience and quality floors of a real campaign, anonymised if needed.
The constraints: what the agent may and may not do when it cannot meet the brief.
The success definition: what a good outcome looks like to the client, in terms that can be measured.

Concourse v2, a stricter test of buying quality, is running now. Its results will be published separately, because v2 uses different tasks and rules and is a new benchmark rather than a rerun of v1.

Frequently asked questions

Can I benchmark my AI buying agent against GPT or Claude?

Yes, provided both run the same brief against the same fixed sellers under the same rules. Concourse Bench v1 tested four OpenAI and four Anthropic models this way, and the scope for evaluating other agents is agreed through the Concourse form.

Which AI model is best at media buying?

No single answer holds across conditions. In v1, two models completed every attempt and two completed none, and both of those pairs included one model from each provider.

How many test runs are enough?

Enough to see a pattern across several buying situations. v1 used three attempts in each of four situations, twelve per model, because a single successful run says little about reliability.

Does a completed buy mean the agent did a good job?

No. In v1, 95 of the 110 contracts in completed buys promised viewability floors below 70%, so protection has to be scored on its own.

Were the v1 sellers AI agents?

No. They were fixed software policies, and model-driven sellers belong to the planned v2.1 seller-side study.

Do API costs show how efficient an agent is?

They show one part of it. v1 recorded model API expense and elapsed time per run, which are not full operating costs or measured savings against human work.

What is the next step for agent builders?

A claim that an agent beats GPT or Claude is only as strong as the test behind it. Fixed sellers, several buying situations, repeated attempts and a scorecard that values protection as much as completion turn a sales line into evidence a client can check.

Alkimi built Concourse to make those differences visible, first for buying and in later versions for selling. To find out how your agent would handle these briefs, request a scoping conversation.

Is Your AI Buying Agent Better Than GPT or Claude? How to Prove It