Incrementality is the share of outcomes that would not have happened without the advertising, and it is measured the same way whether a person or an AI agent buys the media: by comparing a group that was exposed with a comparable group that was not. Randomised holdouts and geo tests remain the most defensible methods; attribution models, which divide credit among conversions that may have happened anyway, do not answer the question. What changes is timing. An agent reallocates budget continuously, so the test design has to be agreed and written into its mandate before it spends, or it will optimise across the very boundaries the test relies on.

TL;DR. Attribution shows which ads were near a conversion; incrementality shows which conversions the ads caused. Holdout tests randomise people or households into exposed and unexposed groups, while geo tests do the same with regions and depend less on identity. An agent must be told which cells it may buy in, which it must leave alone and which decisions it may not make during the test. Spend can only be assigned to a test cell reliably if each deal's record states its targeting, flight and cell, and buyer and seller hold the same version of it.

Why is attribution not the same as incrementality?

Attribution assigns credit for conversions that were recorded. It cannot say how many of those conversions would have occurred with no advertising at all. A customer who searched for the brand, visited the site and was then shown a retargeting ad will often be credited to that ad, even though the purchase was already likely.

The gap matters more when an agent buys. An agent told to maximise attributed conversions will learn, correctly, that audiences already close to buying produce the most credited conversions per pound, and it will move budget toward them. Reported performance rises while the number of customers the advertising actually added stays flat or falls. Incrementality testing is the check on that drift.

Which test designs work when an agent buys the media?

Three designs are in common use, and each suits a different situation.

User or household holdouts withhold advertising at random from part of an eligible audience and compare outcomes with the exposed group. They are precise when the advertiser can reliably identify and suppress the holdout, which depends on identity coverage in the channel.

Geo tests split a market into comparable regions, run advertising in treatment regions, withhold it in control regions, then compare sales or sign-ups by region. Because they do not rely on individual identity, they suit channels where user-level suppression is unreliable. They need enough regions, enough volume per region and a pre-test period showing treatment and control moved together.

Time-based switchbacks turn activity on and off in alternating periods. They are cheap to run but sensitive to seasonality and to effects that carry over from one period into the next.

All three are compatible with agent buying. None is compatible with an agent that has not been told the test exists.

Why must the measurement design be agreed before the agent starts?

An agent acts on the objective and limits it is given. If the mandate does not mention the test, the agent may buy in control regions because inventory there is cheap, or pull budget from a treatment cell that looks weak in week one. Either breaks the comparison.

The measurement design therefore belongs in the mandate, agreed by the marketer, the measurement lead and finance before the first deal. It should state the test cells and how they were assigned, the regions or audiences the agent must exclude, the primary outcome metric, the measurement window, the minimum spend per treatment cell, and the decisions the agent may not make during the test, such as moving budget between cells. The wider document is covered in how to write a media buying mandate.

Why does the deal record matter for assigning spend to a test cell?

An incrementality result is a ratio: the extra outcomes in the treatment cell divided by the spend that produced them. If the spend side of the ratio is wrong, the result is wrong, however clean the outcome data.

Spend reaches test cells through deals. Each deal the agent agrees should carry its geographic targeting, flight dates, price and test-cell label, with delivery reported against the same deal identifier. Problems start when the buyer's and the seller's records of a deal differ. If the seller's version lacks a regional exclusion the buyer's agent agreed, impressions can serve in a control region, and neither side notices until the results look odd. A record of terms written at the moment of agreement and held identically by both parties lets the analyst confirm which spend belongs to which cell before reading the result.

What does a hypothetical geo test design look like?

The design below is illustrative, for a fictional UK retailer, and is not drawn from any real campaign.

Question: do agent-bought display deals add online sales beyond what existing activity produces?
Cells: twenty regions matched on past sales, randomly split into ten treatment and ten control.
Pre-test check: four weeks of historical sales confirming the two groups moved together.
Budget: a hypothetical £50,000 over six weeks, spent only in treatment regions.
Agent constraints: no buying in control regions, no budget moves between cells and no creative changes during the window.
Outcome: online sales by region, with a two-week post-period to capture delayed purchases.
Spend check: every deal tagged with its cell, and buyer and seller records reconciled before analysis.
Decision rule: written in advance, stating which results lead to scaling, changing or stopping the activity.

What are the common pitfalls?

Most failed incrementality tests fail on design rather than statistics. The usual causes are contamination, where advertising leaks into control cells through broad targeting or national inventory; a test too short or too small to separate the effect from normal variation; changing course on interim results; changing several things at once; and seasonal events that hit treatment and control regions differently.

Two pitfalls are specific to agent buying. The first is letting the agent keep optimising toward attributed conversions inside the test, which steers spend toward people who would have bought anyway. The second is analysing results before confirming that every pound sits in the cell the design says it does.

That second check is where a shared record earns its place. Alkimi, an agentic advertising marketplace whose agent is listed in the IAB Tech Lab Agent Registry, has buyer and seller agents record agreed terms on a DealSheet that each side can retrieve independently, with people retaining approval rights over each deal.

Frequently asked questions

What is incrementality in advertising?

Incrementality is the difference in outcomes between people or regions exposed to advertising and comparable ones that were not. It measures what the advertising caused, rather than what happened near it.

Is attribution enough to judge an AI buying agent?

No. Attribution credits recorded conversions, including many that would have happened anyway, so an agent optimised on it can look better on paper while adding fewer customers.

Should advertisers use a holdout or a geo test for agent-bought media?

Use a holdout where the channel allows reliable identification and suppression of the control group. Use a geo test where it does not, provided there are enough comparable regions and enough volume in each.

Can the AI agent design and run the incrementality test itself?

The agent can follow the buying constraints, but the design, cell assignment and analysis should sit with people who do not answer to the agent's objective. The system being measured should not also mark its own work.

How long should an incrementality test run?

There is no standard length. The window should cover the typical purchase cycle, include a post-period for delayed conversions and avoid seasonal peaks unless the test is about them.

How Should Advertisers Measure Incrementality When an AI Agent Buys the Media?