How control groups work in retail media tests
A valid control group follows three rules.
- Assign at random. MRC standards highly recommend randomized controlled trials, because random assignment helps make the groups "statistically identical in all respects except for the treatment".
- Use the same eligibility rules. Placing a shopper in the test group only gives them a chance to see the ad, which IAB/MRC call intent-to-treat. The control group should be selected by the same targeting and bidding logic as the exposed group. Shoppers who would never have seen the ad dilute the result.
- Count total sales. The control group has no impressions or clicks from the brand, so its attributed sales are zero by design and an attributed-only comparison measures the two groups on different terms. Compare all purchases of the advertised products in both groups, per shopper when the groups differ in size.
Types of control groups: holdouts, ghost ads, matched markets and synthetic controls
IAB rates holdouts, ghost ads and matched markets as experiment-based methods of strong causal strength, and synthetic controls as model-based.
| Type | How the control group is formed | Main trade-off |
|---|---|---|
| Randomized user-level holdout test | Shoppers are randomly withheld from a brand's ads | Withheld shoppers cost sales during the test |
| Ghost ads (ghost bidding) | The auction still runs for control shoppers; when the brand's ad would have won, it is withheld and logged | The network must simulate its auction to find those shoppers |
| Matched markets (geo holdout) | Stores or regions are paired on type, assortment and sales, and one side gets the campaign | Price cuts, displays and coupons must be modeled out |
| Synthetic control | Unexposed shoppers are matched on purchase and browsing history, for example by propensity score | Only as good as the matching variables |
Ghost ads were popularized by Google researchers and published in the Journal of Marketing Research in December 2017, and several advertising platforms now offer the method. IAB/MRC list public service announcement (PSA) methods alongside ghost ads as ways to run a randomized test.
Example: Instacart's holdout groups
In a June 2023 engineering post, Instacart described splitting shoppers at random, for each advertiser, into an ads group and a holdout group. Filtering ads after the auction would have left ad slots underused and forced pricing adjustments, so the holdout filter was placed before the auction picked a winner. Some holdout impressions would have lost that auction anyway, and counting them diluted the result. A ghost ad simulation fixed this by identifying which holdout impressions would have won. That let Instacart compare shoppers who saw the ad with shoppers who would have seen it, and estimate incremental sales, lift and incremental sales per ad dollar from the difference.
Kroger Precision Marketing also compares exposed and unexposed groups, matching media exposure to loyalty households' in-store and ecommerce purchases.
Common control group mistakes
- Contamination. Control shoppers see the ad on a platform the test does not measure.
- Diluted groups. Comparing everyone assigned to the test, including shoppers who never had a chance to see the ad, blurs the effect.
- Too small to read. IAB warns that small samples can miss a real effect and asks for confidence intervals that exclude zero. See statistical significance.
- Unmodeled store differences. Matched stores still differ in price reductions, displays and digital coupons.
- Before-and-after reads. Pausing and restarting spend to compare periods, an approach some agencies use, leaves no simultaneous control group, so seasonality and promotions can blur the read.
- One test treated as final. IAB calls incrementality an ongoing process, because competition and shopper behavior keep moving it.
Why it matters for brands and sellers
A lift number is only as good as its control group. Before acting on one, ask the network:
- How were shoppers assigned to the two groups?
- Did both groups use the same targeting and bidding logic?
- Were total sales compared, or only attributed sales?
- How large was the control group?
IAB/MRC expect networks to disclose methodology, test duration and pre-campaign hold-out periods. Holdouts also cost money: brands and retailers are reluctant to leave sales on the table, which is one reason IAB/MRC give for synthetic controls. Read each result at the scope it was measured, since IAB rates experiments rarely practical across networks. Teams judging retail media effectiveness on Instacart Ads or elsewhere can start with these questions.
Related terms
- Incrementality: the effect a control group makes measurable.
- Incrementality testing: designing and running the whole test.
- Lift studies: studies that report a campaign's lift.
- A/B testing: comparing two ad versions with each other.
- Statistical significance: whether a measured difference is real.
- Closed-loop attribution: what networks report without a control group.
Frequently asked questions
What is a control group in marketing?
In retail media, a control group is a set of shoppers, households or stores kept from seeing a campaign and chosen by the same rules as the shoppers who see it. Their purchases show what would have sold anyway, and the gap between the groups is the campaign's incremental effect.
What is the difference between a test group and a control group?
The test group, also called the exposed or treatment group, is eligible to see the ads. The control group is selected by the same targeting and bidding logic and has the ads withheld. Comparing total purchases per shopper in the two groups isolates what the ads added.
What is the difference between a control group and a holdout group?
In retail media measurement the terms are used interchangeably. Instacart calls the control group in its advertiser experiments a holdout group, and IAB lists holdouts among experiment-based methods. IAB/MRC also use "hold-out" for pre-campaign periods, a time-based baseline.
What are ghost ads?
Ghost ads, or ghost bidding, build a control group inside an ad auction. For control shoppers the auction still runs, and when the brand's ad would have won it is withheld and logged. Those shoppers are then compared with shoppers who saw the ad.
How large should a control group be?
A control group should be large enough to read the lift. IAB treats a lift inside the noise floor as unreliable and asks for confidence intervals that exclude zero. IAB/MRC give matching-ratio examples: 4:1 when treatment far outnumbers control, and 1:19 when treatment is a 5% slice of a geo-randomized test.
