iROAS Is Not a Number, It's a Negotiation

In March 2026, researchers from Albertsons Media Collective, Ovative Group, and the Kellogg School of Management took 42 real onsite display campaigns and re-measured each one under 54 defensible methodology combinations — without changing anything about the campaigns themselves. Within a single campaign, the gap between the highest and lowest incremental return on ad spend averaged 6.5x (median 2.5x). And 83% of campaigns flipped from positive to negative depending on which methodology was chosen (Retail Media iROAS Demystified, PDF; landing page).

Read that again. Not “results varied.” The same campaign, the same spend, the same shoppers — profitable under one defensible recipe, money-losing under another. The paper’s own framing: “observed performance is often as influenced by methodological decisions as by underlying customer behavior.”

One disclosure before anything else: this is vendor-and-network-sourced research. Ovative sells measurement services; Albertsons Media Collective sells the media being measured; two Kellogg professors co-authored. But note the direction of the finding — a retail media network publishing evidence that its category’s headline metric is 83% sign-unstable is testifying against its own interest, which is precisely what makes the numbers worth taking seriously.

A negotiated artifact, not a measurement

Every definitional post on the first page of Google agrees on the formula: iROAS = incremental revenue ÷ ad spend. The formula is not the problem. The entire dispute lives inside the numerator — what counts as incremental — and, less discussed, inside the denominator’s scope: whose spend is charged against which audience’s incremental sales.

The Ovative–Albertsons team varied just four choices to generate their 54 recipes:

1. Filtering — restrict test and control to past brand buyers, past category buyers, or nobody
2. Matching approach — clustering versus propensity score matching (PSM)
3. Matching features — which shopper attributes the match is built on
4. Incremental revenue calculation — observed test-vs-control difference with a t-test, or a modeled counterfactual (Bayesian structural time series, the method behind Google’s CausalImpact)

Each choice is defensible in isolation. Each one moves the number — sometimes by multiples, sometimes across zero. A few of the individual swings, all from the same paper:

• Applying a brand-purchase filter moved aggregate iROAS from $2.27 to $0.22, shrank the sample 83%, and made match quality 35% worse. A category filter left the number essentially untouched ($2.27 to $2.20).
• Clustering versus PSM: $1.80 versus $0.73 in aggregate — and the method with roughly 12x worse covariate balance is the one reporting the number 2.5x higher.
• Removing one matching feature (historical brand sales) flipped the PSM estimate from $1.23 to −$0.14. A sign flip, from one toggle.
• Observed difference versus modeled counterfactual: $1.56 versus $0.97 on average — roughly 90% divergence — because the observed method absorbs trend and seasonality into “lift.”

Before any of that, the paper identifies five scope choices — unit of analysis, exposure measured, audience included, outcome window, spend and revenue definition — that determine what question the iROAS answers in the first place. Its verdict: “Two results can both be labeled iROAS and still answer meaningfully different business questions.”

Now consider who chooses. In most retail media relationships, the network — the seller — selects the filter, the matching method, the features, and the calculation. The buyer receives a single number with two decimal places and a currency sign. That number is not a measurement of the world. It is the output of a negotiation the buyer never attended, conducted entirely by the counterparty, whose revenue depends on the result.

This is why the standards context matters. The IAB and IAB Europe incrementality guidelines (November 2025) — decoded in detail in the iROAS decode — set out three requirements a model must satisfy to “credibly claim causality”: a credible counterfactual, controlled bias, and separation of signal from noise. The same document files platform-reported incrementality in its weakest evidence tier — “hybrid proxies” — alongside new-to-brand percentages and simple multi-touch attribution, and states that hybrid proxies are not suited for comparing performance across networks. The standards body, whose contributors include Criteo, Walmart, Instacart, Albertsons, Meta, and Google, has already told you how much to trust an unaudited network number. The guidelines’ words for statistically insignificant lift: “not reliable for decision making.”

So the question is not whether to trust iROAS. It is whether you, the buyer, get to sit at the negotiation. What follows is the protocol for taking your seat.

The audit protocol

Eight questions. Ask them of every network, every measurement partner, every dashboard. For each: the question, the failure mode it catches, and the IAB principle it enforces. None requires a statistics degree — only the willingness to keep asking until you get a specific answer.

1. What question does this iROAS answer?

Ask for the five scope choices in writing: unit of analysis, exposure measured (single campaign, always-on presence, or audience strategy), audience included and any pre-filtering, outcome window, and the exact spend and revenue definitions in the ratio.

Failure mode caught: cross-network comparison of incomparable numbers. A 14-day, sponsored-product, brand-halo-included iROAS from one network placed side by side with a 30-day, display-only, item-level iROAS from another is not a comparison; it is two answers to two different questions wearing the same label. The IAB use-case grid explicitly rules hybrid proxies “not suited” for cross-network comparison.

IAB principle enforced: a credible counterfactual starts with a precise definition of the intervention — you cannot model “what would have happened without the campaign” until “the campaign” has an exact boundary.

2. What is the counterfactual, and who was in it?

A real holdout audience? Matched non-exposed shoppers? A synthetic control modeled from pre-period data? Or no counterfactual at all — a baseline assumption?

Failure mode caught: correlation dressed as causation. The current top-ranking definitional post for “what is iROAS” (Skai) walks through an example where a $4.50 ROAS becomes a $3.15 iROAS by assuming 30% of conversions were organic. An assumed haircut is not a counterfactual; it is a negotiated discount rate presented as measurement — exactly the practice the IAB taxonomy grades weakest.

IAB principle enforced: requirement one, verbatim: “Without a credible counterfactual, measured differences may reflect correlation rather than true campaign impact. The rigor of this initial design is the primary factor determining the causal strength of the entire model.”

3. How was the audience filtered before measurement?

Past brand buyers, past category buyers, or unfiltered — and what happened to sample size and match quality after filtering?

Failure mode caught: the brand-filter collapse. In the Ovative–Albertsons data, filtering to past brand buyers took iROAS from $2.27 to $0.22 while destroying 83% of the sample and degrading match quality 35%. The filter choice alone spans a 10x range — and a network that wants a bigger number simply filters less, or differently. It also catches the quieter denominator artifact: filtering shrinks the audience whose incremental sales count in the numerator while the full campaign spend stays in the denominator.

IAB principle enforced: controlling for selection bias — requirement two’s core concern. Who ends up in test versus control is a bias decision, not a technical detail.

4. Which matching method — and what is the covariate balance?

If exposed shoppers were matched to non-exposed shoppers, ask for the method (clustering, PSM, nearest-neighbor) and for balance diagnostics — standardized mean differences between test and control.

Failure mode caught: the weak-match premium. Clustering versus PSM produced $1.80 versus $0.73 in aggregate, and the clustering approach had roughly 12x worse covariate balance. Worse matching means the “control” group differs from the exposed group in ways that inflate apparent lift. If a vendor reports the higher number and cannot produce balance statistics, you are likely being shown the weak-match premium as performance.

IAB principle enforced: bias control with explicit diagnostics — the guidelines call for stated identification assumptions and tests, not method names.

5. Which features was the match built on?

Specifically: is historical brand purchasing included as a matching feature?

Failure mode caught: the one-toggle sign flip. Dropping historical brand sales from the PSM feature set moved the estimate from $1.23 to −$0.14. If matching features are undisclosed, the sign of your iROAS is undisclosed too.

IAB principle enforced: omitted variable bias — requirement two names unmeasured confounders (seasonality, pricing, promotions, concurrent campaigns) as a primary threat. The strongest single confounder in retail is whether the shopper already bought the brand.

6. How is incremental revenue computed — observed difference or modeled counterfactual?

A t-tested observed gap between test and control, or a time-series counterfactual (BSTS / CausalImpact) that models what control-group sales would have done given trend and seasonality?

Failure mode caught: seasonality laundered into lift. Observed methods averaged $1.56 against $0.97 for the modeled counterfactual — about 90% divergence — because campaigns often coincide with promotions, holidays, and category upswings that the observed method credits to the media.

IAB principle enforced: requirement two again — seasonality, pricing, and promotional effects are the first confounders on the IAB’s list.

7. What is the confidence interval — and does it exclude zero?

Not the point estimate. The interval, the significance test, and ideally falsification checks (does the method find “lift” in a placebo period where no campaign ran?).

Failure mode caught: noise sold as signal. Recall the headline: 83% of campaigns crossed zero across methodology choices. A $1.40 iROAS whose interval spans −$0.60 to $3.40 is not a $1.40 iROAS; it is a shrug with decimal places. The IAB guidelines are blunt: a statistically insignificant lift estimate “is not reliable for decision making.”

IAB principle enforced: requirement three, separation of signal from noise — confidence intervals excluding zero, bootstrapping, falsification tests, sensitivity analyses, and repeated measurement. “Incrementality is not a one-and-done test but an ongoing process.”

Two follow-ups the interval question implies, and both belong in writing before launch: what minimum detectable effect was this test powered to find? — a test that could never have detected a plausible lift was theater from day one — and how many times was this result examined before it was reported? Every undisclosed interim peek inflates the false-positive rate, so an interval that excludes zero after multiple looks does not mean what it appears to mean. Randomized assignment is what makes a holdout an experiment; power declared at plan time and a pre-committed look schedule are what make its interval trustworthy.

8. Where does this method sit in the IAB evidence hierarchy — and who graded it?

Ask the network to place its own methodology on the four-tier scale: experiment-based (strong), model-based counterfactual (strong to moderate), econometric (moderate to weak), hybrid proxy (weak). Then ask whether any independent party has validated it.

Failure mode caught: self-graded homework. Networks and their measurement arms sit on the standards bodies that write the rules — Albertsons Media Collective’s VP of Measurement sits on the IAB Commerce Board that produced the guidelines. That is not a scandal; multi-party consensus is the closest thing to neutral ground this industry has. But it means the grading scale exists and the network knows exactly where its product falls on it. The guidelines themselves say platform lift estimates need external ground-truth experiments “to avoid false confirmation.” Epsilon’s own measurement guidance — a media seller — concedes the bar: concurrent user-level holdouts, statistically powered tests, independent accreditation.

IAB principle enforced: all three at once. The taxonomy is the three requirements applied to method families.

The protocol in the wild

Two live examples show why the questions bite.

In April 2026, Albertsons Media Collective — co-author of the variance research — launched onsite incrementality measurement, explicitly citing the 6.5x finding as motivation (official release). The launch case study reports a $7.45 iROAS for S. Martinelli & Co., with 65% new-to-brand and 33% sales lift. All three numbers are network-reported, and the press release does not disclose which methodology family produced them. The protocol’s first response: $7.45 by which of the 54 recipes? A network that co-authored the paper proving the recipe determines the answer owes the recipe alongside the number.

Amazon’s multi-touch attribution, announced at unBoxed and rolled out through 2025, is calibrated by what Amazon describes as hundreds of thousands of randomized controlled trials (methodology paper). That is real experimental machinery — and question two still applies, because the RCTs are Amazon’s internal calibration set, not a per-campaign counterfactual the buyer can inspect. The output distributes credit across Amazon touchpoints; it is attribution tuned by experiments, not an auditable campaign-level lift estimate. Strong engineering, wrong tier for the cross-network comparison question — which buyers should route through the broader retail and commerce media measurement standards instead.

What a good-faith answer looks like

The protocol is not adversarial theater. Networks can pass it, and the research points to exactly how — the Ovative–Albertsons paper itself proposes five transparency standards for retail media networks and ships a ready-made advertiser question guide in its appendix. A network answering in good faith does five things:

1. States the question its iROAS answers — the five scope choices, in the reporting, not in a methodology PDF three clicks away.
2. Names the counterfactual — holdout, matched, or modeled — and discloses filter, matching method, and feature set, including whether historical brand purchasing is in the match.
3. Ships the diagnostics with the estimate — covariate balance, confidence intervals, and what a placebo test found. A number that arrives without its interval is marketing.
4. Self-locates on the IAB tier scale, unprompted — “this is a model-based counterfactual, strong-to-moderate causal evidence, and here is what would upgrade it” is a sentence a confident measurement team can say out loud.
5. Offers the upgrade path — a real holdout on your next flight, or third-party validation against an external experiment. The IAB’s phrase is the right operating posture: an ongoing process, not a one-and-done test.

Any network that clears those five has stopped selling you a number and started selling you a measurement. Most of what ranks for “what is iROAS” today is written by companies with a recipe to sell — an attribution platform, an incrementality module, an experiments product. That is not disqualifying, but it is why the definitional content never mentions that the definition is the negotiation.

The 6.5x variance is not a flaw to be engineered away. It is the honest size of the methodological freedom that exists inside a single innocuous-looking metric. The only question is who exercises that freedom — the party selling the media, or the party paying for it. Ask the eight questions. Take your seat.