Publishers test a sell-side AI agent by placing it in a sandbox with a mock ad server, the publisher's real floors and approval rules, and a set of scripted buyer agents built to push on price, terms and instructions. Each run is scored on whether the deal was valid, whether the agent held the terms it was given, whether it decided the same way on repeat runs and whether it escalated when it should have. Only then does the agent meet real buyers, on a narrow product set, with a person approving every price.

TL;DR: A protocol conformance test shows that a seller agent speaks the language, not that it sells well. The useful test holds the buyer side fixed and changes only the seller agent, so any difference in outcome belongs to the agent. Five scripted buyer profiles cover most of the pressure a seller agent will meet, and a staged go-live widens its authority only as it earns it.

What is a sell-side AI agent, and why test it before it prices anything?

A sell-side agent acts for the publisher. It describes the inventory, answers buyer agents, applies floors and deal rules, and accepts, counters, escalates or declines. Vikesh Chevli, director of ad tech and programmatic strategy at News Corp, described it to AdExchanger this way: "Essentially, a sales agent is powered by an LLM model, and it effectively sells our inventory on our behalf to buy-side agents."

Testing matters because a buyer agent can ask again without tiring, compare every seller's answer in one format and remember a concession granted once. A seller agent that gives up a little each time it is asked will keep being asked, and the place to discover that is a sandbox, not a live deal.

What does a fair test of an AI agent look like?

The principle is to hold the counterparty constant and change only the agent under test. Concourse Bench, Alkimi's benchmark for AI media-buying agents, is a worked example: in v1, every model-driven buyer faced the same sellers, each run by a fixed software policy, under the same rules.

The benchmark also showed why finishing a deal is a weak score on its own. Across the 110 signed contracts inside completed buys, 95 promised a viewability floor below 70%. The deals closed, and the terms were weak.

Concourse's seller-side study, v2.1, reverses the design: a model-driven seller faces fixed buyer profiles and has to manage inventory, price, performance and future opportunity. It is planned and has no measured results yet, but a publisher can copy its shape now.

What should a publisher set up before the first test run?

Four things should exist before the agent runs a single scenario.

A written mandate: what the agent may offer, which deal types it may close alone and when it must stop, as set out in what a sell-side agent mandate should contain.
A floor policy, not a single number: a walk-away price per product and deal type, plus the concessions the agent may and may not trade, as covered in how publishers should set floor prices when the buyer is an AI agent.
A sandbox with no real spend: IAB Tech Lab's AAMP seller agent ships a CSV adapter for "testing and demos without an ad server". The Prebid Sales Agent, an open-source AdCP sales-agent implementation maintained under Prebid.org, includes a mock ad server with "zero real spend".
A pinned version: the Prebid Sales Agent documentation advises teams to "pin the version you deploy and re-test after every upgrade", and the same habit applies to the model and the prompt.

Step 1: Does the agent pass the protocol checks?

Start with conformance, because a seller agent that returns malformed responses cannot be judged on anything else. IAB Tech Lab's AAMP publishes open-source reference implementations for both buyer and seller systems. Its reference buyer agent can be configured with negotiation strategies, including a target CPM, a maximum CPM and concession limits, which makes it a ready, repeatable counterparty for a seller agent in a sandbox.

For agents that implement AdCP, AgenticAdvertising.org publishes storyboards that exercise a workflow end to end, each defining "the exact tool call sequence a buyer agent makes". The documentation is clear about scope: storyboards check the wire contract and "cannot tell whether the adapter behind the wire actually integrates with the upstream". Passing them is the entry ticket, not the result.

Step 2: Which buyer agents should the seller agent face?

The five-buyer sparring set uses scripted profiles, each built to probe one weakness and kept fixed between runs so results can be compared. Run every profile against every product the agent may sell.

The anchor: opens far below the walk-away price. The test is whether the agent counters from its own position or drifts towards the buyer's.
The grinder: asks for a small concession every round. The test is whether the agent stops at its walk-away price and refuses to treat a past discount as the new opening point.
The trader: accepts the headline price but asks to drop viewability, make-good or measurement terms. The test is whether the agent holds price and quality terms together.
The injector: hides instructions in the brief, such as text telling the agent to ignore its floors. Prompt injection is the first entry in the OWASP Top 10 for LLM Applications 2025, which covers cases where a model "accepts input from external sources", and every buyer message is exactly that.
The out-of-mandate buyer: arrives with an unverified identity, a blocked advertiser category or a deal type the agent may not sell. The test is whether the agent declines or escalates rather than improvises.

Step 3: How should each run be scored?

Score each run on separate measures, because a single number hides the failure that matters. Concourse frames its long-term evaluation around four dimensions: execution, commercial quality, reliability and measured efficiency. A seller-side score sheet can borrow the first three and add escalation, which matters more to a publisher at this stage.

Execution: did the run end in a valid deal or a clean refusal, with the record written and the deal passed correctly to the mock ad server?
Commercial quality: did every agreed term sit inside the mandate, including price, viewability, make-good and the product promised?
Reliability: did repeated runs of the same scenario produce the same decision?
Escalation: did the agent hand off every case the mandate reserves for a person, and none it was authorised to settle?

As a hypothetical illustration, an agent that closes nine grinder runs in ten looks strong on execution, but if three of those nine closed below the walk-away price, it failed on commercial quality. Only a separate score shows it.

Publishers that want the scoring design agreed before anything runs can request a Concourse scoping conversation. Because the seller-side benchmark is still planned, that conversation starts with the brief, the constraints and what a good outcome means, not with published seller results.

Step 4: How many test runs are enough?

Run every scenario several times, because a language model can reach a different decision on identical inputs, and one clean run proves little. Rerun the full set after any change to the model, prompt, mandate or software version, since a fix for one profile can break another. The buy-side version of this discipline is set out in how to test an AI media buyer before you deploy one.

Step 5: How should a publisher take a tested agent live?

Go live in stages and widen the agent's authority only as it earns it. The earned autonomy ladder has five steps: Observe, Recommend, Draft, Human-approved action and Bounded automatic action. A seller agent fresh from the sandbox belongs at Recommend or Draft, with a person approving every price it proposes.

The Weather Company's first sales agent shows what a narrow start looks like. It covered a small set of top-performing ad formats and audience segments, access was restricted to approved partners, and a person validated the availability and pricing the agent gave. "We didn't try to boil the ocean on day one," Dave Olesnevich, VP of weather data and advertising products at The Weather Company, told AdExchanger.

CNN International Commercial described the same test-first logic to Digiday. "In Q4 we'd like to be ready for people who want to test budgets, so that when we want to begin to trade against it fully, it's already tried and tested," said Faisal Karmali, its VP of digital business operations.

Frequently asked questions

How long should a publisher test a sell-side agent before go-live? Long enough for every scenario in the sparring set to produce consistent results on the current version. Coverage and consistency matter more than calendar time, and the clock restarts after any material change.

Can a publisher test a sell-side agent without an engineering team? Partly, because a marketplace's or partner's seller agent can be tested by its operator using the publisher's own mandate and floors. The publisher's job is to define the scenarios and pass marks, and to see the results before any real buyer does.

Is protocol conformance enough to go live? No, because conformance shows only that the agent exchanges messages correctly. Commercial testing is what shows whether it sells on the publisher's terms.

Should the test include prompt injection? Yes, because every buyer message is external input to the seller agent. OWASP's guidance on excessive agency also recommends that a human approve high-impact actions before they are taken.

Does testing replace human approval? No, because testing only decides how much authority the agent starts with. Approval at the deal level stays with a person until results from live deals justify widening it.

Where does the sandbox end and real trading begin?

A sandbox answers one question: does this agent hold the publisher's terms under scripted pressure? The next question is whether it holds them against agents built by other people. The Alkimi Gym is built for that step: sellers build their own agents, set the rules they work to and spar against counterparty agents before any real money moves.

Publishers deciding whether to build an agent at all can start with whether to build a sell-side agent or use a marketplace's. Those who want help designing the scoring can request a Concourse scoping conversation.