# Outbound Email Holdout Groups: Measure Incremental Lift

> Learn how to run an outbound email holdout group: randomise accounts, suppress a control, size the test and calculate incremental lift to prove outbound caused meetings.

Canonical: https://revnu.com/blog/outbound-email-holdout-groups-measure-incremental-lift
Published November 22, 2026 by George Jefferson, Co-founder and CEO, Revnu

# Outbound Email Holdout Groups: Measure Incremental Lift

To measure incremental lift from outbound email, randomly split your target accounts into an emailed group and a suppressed holdout group, run both over the same fixed window, and compare how many in each group booked a meeting. The gap between the two rates is the lift outbound caused. Everything else in your dashboard (replies, opens, attributed calls) only tells you what happened to people you emailed, not what would have happened anyway.

That difference matters more than most small B2B teams admit. Some of your best-fit accounts were already looking, already on your site, already talking to a friend who mentioned you. A holdout group is how you find out how many of those meetings outbound really created. This guide walks through a version a small team can run without a data scientist: account-level randomisation, one pre-set outcome, a power-checked sample size and intent-to-treat analysis.

## What an outbound email holdout group is (and what it isn't)

An outbound email holdout group is a randomly chosen slice of your eligible target accounts that you deliberately do not email, kept as a control so you can compare their outcomes against the accounts you did email. Same list, same period, same sales team. The only systematic difference is whether outbound reached them.

What it isn't: a list of people who didn't reply, a set of contacts who bounced, or accounts you skipped because they looked like poor fits. Those groups are selected by behaviour or by your judgement, so they differ from the emailed group before you send anything. Randomness is the whole point. If you choose the holdout by hand, you will tend to hold out the accounts you think are weak, and the test will flatter your campaign.

A holdout is also not a pause button on your pipeline. Holdout accounts can still find you through search, referrals and inbound. That is exactly what you want to measure: the baseline rate of meetings that happen without outbound.

## Why attribution can't tell you incremental lift

Attribution answers "which touch got credit for this meeting?" while incrementality answers "would this meeting have happened without the email?" Those are different questions, and only the second one tells you whether outbound is worth the money.

When a prospect replies to your email and books a call, attribution credits outbound. But if that prospect had already visited your pricing page twice and your founder had been in their LinkedIn feed for a month, the email may simply have been the last click before a decision that was already made. Attribution cannot see the counterfactual: the world where you never sent it.

The bias runs in one direction. Outbound targets accounts that match your ICP, and ICP-matching accounts convert at a higher rate than random ones even with no email at all. So an attributed-meetings report will always look good. That's why a healthy-looking dashboard and a campaign that adds nothing can coexist.

The rule: use attribution to diagnose which message or segment performs, and use a holdout to decide whether the channel is worth running. Don't swap one for the other.

## How to design the test: randomise accounts, suppress the holdout, pick one outcome

A sound outbound email incrementality test design has three decisions, all made before you send a single email: who gets randomised, how the control stays clean, and what counts as success.

### Randomise at the account level, not the contact level

In B2B, several people at one company can see your outreach. If you randomise contacts, one colleague lands in the emailed group and another in the holdout, and the holdout person hears about you in the next meeting. Randomise by account instead: every contact at a company gets the same assignment. This is the standard approach for B2B and it limits contamination between arms.

This is also the practical answer to geo holdout versus user-level holdout for B2B outbound. A geo holdout (suppressing whole regions) suits consumer or local campaigns where people cluster by place. For B2B outbound, account-level assignment is cleaner. Your buyers sit inside companies, not postcodes, and a geo split mixes in market differences you can't control. Use geography only if you genuinely sell region by region.

### Lock the list and suppress the holdout

Build the full eligible list first, then randomise it. Use a random number or a shuffled spreadsheet column, not a judgement call. A 10 to 20 percent holdout is a practical starting point. Then add the holdout accounts to a suppression list in every sending tool so they get no outbound email, and make sure no one on the team contacts them by other outbound routes for the test window. Keep the audience fixed once the test starts.

### Pick one outcome and a window

Choose a single primary outcome before launch: meetings booked, qualified opportunities, or pipeline created. Meetings are the usual choice for a small team because they happen fast and in volume. Fix the measurement window too, long enough for the outcome to occur. Replies resolve in days, meetings in weeks, pipeline in a sales cycle. Write the outcome and window down. Changing them after you see data is how people talk themselves into a lift that isn't there.

## How big the holdout needs to be: sample size and minimum detectable effect

The holdout must be large enough to detect the smallest lift you would act on, and that depends on your baseline rate, not on a fixed percentage. A holdout that is too small cannot tell a real effect from noise, so a flat result tells you nothing.

Four inputs drive sample size for an email holdout test:

- **Baseline rate:** how often accounts book a meeting without outbound.
- **Minimum detectable effect (MDE):** the smallest lift worth acting on.
- **Significance level:** commonly 5 percent.
- **Power:** commonly 80 percent, the chance of catching a real effect of that size.

Plug them into a two-proportion power calculator and read off the sample needed per arm. As a worked example, assuming a two-sided test at 5 percent significance and 80 percent power, detecting a rise from 2 percent to 4 percent needs roughly 1,140 accounts per arm, about 2,280 in total.

That number surprises small teams. Smaller expected lifts and rarer outcomes need far larger samples. If your list is a few hundred accounts, you cannot detect a one-point lift in meeting rate, however carefully you run it. You have three options. Choose a bigger MDE, because only a large effect justifies the spend anyway. Run the test longer or across several cycles and pool comparable periods. Or move to a more frequent outcome such as positive replies, knowing it is a weaker proxy for pipeline.

A statistically non-significant result from an underpowered test is inconclusive. It is not proof that outbound does nothing. Say so in the readout.

## Calculate incremental lift: absolute lift, relative lift and extra meetings

The incremental lift calculation takes three numbers from the same window: the outcome rate in the emailed group, the rate in the holdout, and the number of accounts you emailed.

- **Absolute lift** = emailed rate minus holdout rate
- **Relative lift** = (emailed rate minus holdout rate) divided by holdout rate
- **Incremental outcomes** = absolute lift times the number of emailed accounts

Example: meetings occur for 6 percent of emailed accounts and 3 percent of holdout accounts. Absolute lift is 3 percentage points. Relative lift is 100 percent, meaning outbound doubled the meeting rate. If you emailed 1,000 accounts, outbound added about 30 meetings. The other 30 would have come anyway.

That last line is the one to take to your budget conversation. Divide your campaign cost by the 30 incremental meetings, not the 60 attributed ones, and you get the real cost per meeting outbound created. To measure incremental pipeline from outbound email, run the same arithmetic on pipeline value per account instead of meeting counts.

Report the uncertainty, not just the point estimate. A 3-point lift with an interval that runs from 0.5 to 5.5 points is a different story from one that runs from 2.8 to 3.2.

Analyse by assignment, never by outcome. This is intent-to-treat: every account counts in the group it was assigned to, whether its emails were delivered, opened or bounced. Dropping bounces or comparing only openers selects on behaviour that happened after randomisation, and that breaks the comparison you worked to build.

## Mistakes that ruin a holdout (contamination, openers-only analysis, peeking, underpowered tests)

Four mistakes account for most holdouts that mislead.

**Contamination.** Holdout accounts get emailed through a side channel: a rep adds them to another sequence, a founder DMs them, an agency runs a parallel campaign. Even a handful of leaks pulls the holdout rate toward the emailed rate and shrinks your measured lift. Suppress the holdout everywhere and tell everyone who touches outbound.

**Openers-only analysis.** Comparing accounts that opened your email against the holdout looks natural and is badly biased. Openers are the more engaged accounts to begin with. You are measuring who opens, not what the email did. Stick to intent-to-treat.

**Peeking.** Checking results daily and stopping the moment the emailed arm leads inflates false positives. Set the window and stopping rule in advance and read the result once.

**Underpowered tests.** A 5 percent holdout on a 400-account list cannot detect anything short of a huge effect. Run the power check first. If the numbers don't work, extend the test or pool cycles instead of trusting a noisy read.

Two smaller ones: splitting a buying committee across arms by randomising contacts, and changing the audience mid-test. Both quietly turn a clean comparison into a story you tell yourself.

## Run it every cycle: how Revnu builds the holdout into outbound and checks back

A holdout is worth far more as a habit than as a one-off, because baseline rates drift, lists change and one result is one sample. The catch for a small team is that a clean holdout is fiddly: randomise, suppress, track assignment, wait out the window, run the maths, then do it again next month.

Revnu is an agent that runs growth work for a business across channels including outbound, then measures what worked and changes what happens next. For outbound, the pieces of this process line up with how it already operates. It researches target companies and the right people, keeps them in a Leads CRM, runs sequences from your Gmail or from managed inboxes, and supports suppression so a blocked domain also stops follow-ups already scheduled. A clean holdout depends on exactly that suppression behaviour. Revnu also schedules check-backs and reports results in Slack or iMessage, so the readout at the end of the window arrives without you chasing it.

The practical split: you decide the one outcome and the minimum lift worth acting on, and Revnu works the list and checks back when the window closes. Pair the result with the success criteria you set up front, as covered in [Outbound Pilot Success Criteria: When to Scale](https://revnu.com/blog/outbound-pilot-success-criteria-when-to-scale), and use a regular readout like the one described in [Weekly Marketing Report: What Your AI Agent Should Explain](https://revnu.com/blog/weekly-marketing-report-what-your-ai-agent-should-explain) so the incremental number sits next to the attributed one, where you can see the gap.

## Hold out 10 to 20 percent of accounts on your next outbound cycle

Attribution will always tell you outbound is working, because it only counts the people you already emailed. A holdout tells you how many meetings you actually bought. Before your next send, randomise accounts, suppress the holdout everywhere, write down one outcome and a window, and check the sample size against your baseline. If you'd rather not run the mechanics yourself, start with Revnu at revnu.com. Pro is from $99 a month and self-serve, and Revnu works the list and checks back with the result.

## Frequently asked questions

### What is a holdout group?

A holdout group is a randomly selected set of accounts or people deliberately left out of a campaign so you can compare their outcomes with those who received it. Because assignment is random, the difference between the two groups estimates the campaign's true incremental effect, not just what happened to people you reached.

### How big should an outbound email holdout group be?

A 10 to 20 percent account holdout is a practical starting point, but size it by power analysis, not a fixed percentage. Set your baseline meeting rate, the smallest lift worth detecting, 5 percent significance and 80 percent power, then calculate the sample per arm. Smaller lifts and rarer outcomes need larger samples.

### What is the difference between attribution and incrementality for outbound email?

Attribution assigns credit for a meeting to a touch, such as an email. Incrementality asks whether the meeting would have happened without that email. Attribution can't see the counterfactual, so it overstates outbound's impact on accounts that were already likely to buy. Only a holdout group measures what outbound actually added.

### How do you calculate incremental lift for email?

Subtract the holdout's outcome rate from the emailed group's rate to get absolute lift. Divide that by the holdout rate for relative lift. Multiply absolute lift by the number of emailed accounts to get incremental outcomes. For example, 6 percent versus 3 percent across 1,000 accounts means about 30 extra meetings.

### Should I use a geo holdout or a user-level holdout for B2B outbound?

For B2B outbound, randomise at the account level. Buyers sit inside companies, and colleagues share information, so splitting contacts leaks the campaign into the control. Geo holdouts suit local or consumer campaigns where people cluster by place. Revnu supports suppression so a blocked domain stops follow-ups, which helps keep a holdout clean.
