Before deploying an AI media buyer, run it through a standardised brief under controlled conditions and measure both completion rate and contract quality. The Concourse benchmark gives buyers a reference methodology; what it measures is the right starting point for any internal evaluation.

At a glance

Standardised brief: a media brief with fixed parameters (budget, channel, audience, quality floors) that every agent runs identically so results are comparable across models.

Completion rate: the percentage of runs in which an agent assembled a full, signed, budget-compliant portfolio. A pass/fail metric that tells you whether the agent can function, not whether it performs well.

Contract quality: the terms accepted on completed buys. Viewability floors, brand safety commitments, CPM rates. This is what determines value, not completion alone.

Failure mode: how an agent behaves when it cannot complete the task within constraints. Graceful failure (escalate, flag, stop) is preferable to silent failure (accept bad terms and close anyway).

Why testing before deployment matters

The Concourse v1 benchmark ran eight AI buyers through 96 identical media-buying runs. Sol and Fable completed all 12 of their runs. Luna and Haiku completed none. The variation was not visible from the model names or marketing materials.

Deploying an untested AI buyer into a live campaign carries the same risks as deploying untested software to production. You need to know it works before it handles real budget.

Testing also surfaces failure modes. An agent that cannot complete a brief within constraints needs a clear stop condition. If it does not have one, it will find a way to close that you did not sanction.

Step 1: define a standardised brief

Write a brief with a fixed budget, a named channel, a target audience, and explicit quality floors (viewability, brand safety, CPM ceiling). Keep it representative of a real buy but simple enough to run 10 to 20 times.

The brief must specify what the agent should do when it cannot meet the constraints. Can it go over budget by 5%? Can it substitute a different format? Can it sign a contract below the viewability floor? Decide in advance.

Run the same brief multiple times. An agent that completes 8 of 10 runs tells you more than one that completed 1 of 1.

Step 2: measure completion rate and contract quality separately

Record whether the agent completed each run and the terms it accepted when it did. Log viewability floors, CPM rates, and any constraint violations on a per-contract basis.

Concourse v1 found that 95 of 110 signed contracts accepted viewability floors below 70%, even though the agents technically completed the task. Contract quality and completion rate are independent variables.

An agent that completes every run but consistently accepts contracts below your quality floors is not deployment-ready. An agent that completes 8 of 10 runs with strong terms on each may be preferable.

Step 3: test failure modes deliberately

Set an impossible brief: a budget that cannot assemble a compliant portfolio, or quality floors that no available inventory meets. Watch what the agent does. Does it fail gracefully and escalate, or does it close something anyway?

Silent failure is the most dangerous mode. An agent that accepts a bad contract because no good ones were available will look like a successful completion in your logs. You only discover the problem when the campaign runs.

A human approval step at the deal level catches both silent failures and edge cases. Review contracts before activation, not after.

Step 4: compare models on the same brief

If you are evaluating multiple AI buyers, run each on the same standardised brief. The Concourse methodology does this across eight models. The results show meaningful variation that would not be apparent from reading each model's documentation.

Compare completion rate, average viewability floor accepted, CPM deviation from ceiling, and failure mode across models. Weight contract quality at least as heavily as completion rate.

Frequently asked questions

How many test runs do I need to evaluate an AI media buyer? Run the same brief at least 10 times per model. Single-run results are not reliable. Concourse v1 ran 12 identical runs per model and still found meaningful variance.

What should I measure beyond completion rate? Measure viewability floors accepted, CPM rates relative to your ceiling, any constraint violations, and failure mode when the brief cannot be met. Contract quality data is at least as important as completion rate.

What does the Concourse benchmark measure? Concourse measures completion rate and contract quality across eight AI buyers running 96 identical media-buying runs. It is the most structured public evaluation of AI buyer performance available.

Can I test an AI buyer without building a benchmark from scratch? You can use a simplified version of the Concourse methodology: one brief, fixed constraints, run 10 times, log completion rate and contract terms on each run. You do not need a full multi-model benchmark to make an informed deployment decision.

What is a safe way to deploy an AI buyer after testing? Start with human approval on every contract. Allow the agent to draft deals; require a human to review and activate them. Expand autonomy only after you have evidence that the agent's contracts consistently meet your quality standards.

How to test an AI media buyer before you deploy one