On this page
- What an outbound email holdout group is (and what it isn't)
- Why attribution can't tell you incremental lift
- How to design the test: randomise accounts, suppress the holdout, pick one outcome
- How big the holdout needs to be: sample size and minimum detectable effect
- Calculate incremental lift: absolute lift, relative lift and extra meetings
- Mistakes that ruin a holdout (contamination, openers-only analysis, peeking, underpowered tests)
- Run it every cycle: how Revnu builds the holdout into outbound and checks back
- Hold out 10 to 20 percent of accounts on your next outbound cycle
To measure incremental lift from outbound email, randomly split your target accounts into an emailed group and a suppressed holdout group, run both over the same fixed window, and compare how many in each group booked a meeting. The gap between the two rates is the lift outbound caused. Everything else in your dashboard (replies, opens, attributed calls) only tells you what happened to people you emailed, not what would have happened anyway.
That difference matters more than most small B2B teams admit. Some of your best-fit accounts were already looking, already on your site, already talking to a friend who mentioned you. A holdout group is how you find out how many of those meetings outbound really created. This guide walks through a version a small team can run without a data scientist: account-level randomisation, one pre-set outcome, a power-checked sample size and intent-to-treat analysis.
What an outbound email holdout group is (and what it isn't)
An outbound email holdout group is a randomly chosen slice of your eligible target accounts that you deliberately do not email, kept as a control so you can compare their outcomes against the accounts you did email. Same list, same period, same sales team. The only systematic difference is whether outbound reached them.
What it isn't: a list of people who didn't reply, a set of contacts who bounced, or accounts you skipped because they looked like poor fits. Those groups are selected by behaviour or by your judgement, so they differ from the emailed group before you send anything. Randomness is the whole point. If you choose the holdout by hand, you will tend to hold out the accounts you think are weak, and the test will flatter your campaign.
A holdout is also not a pause button on your pipeline. Holdout accounts can still find you through search, referrals and inbound. That is exactly what you want to measure: the baseline rate of meetings that happen without outbound.
Why attribution can't tell you incremental lift
Attribution answers "which touch got credit for this meeting?" while incrementality answers "would this meeting have happened without the email?" Those are different questions, and only the second one tells you whether outbound is worth the money.
When a prospect replies to your email and books a call, attribution credits outbound. But if that prospect had already visited your pricing page twice and your founder had been in their LinkedIn feed for a month, the email may simply have been the last click before a decision that was already made. Attribution cannot see the counterfactual: the world where you never sent it.
The bias runs in one direction. Outbound targets accounts that match your ICP, and ICP-matching accounts convert at a higher rate than random ones even with no email at all. So an attributed-meetings report will always look good. That's why a healthy-looking dashboard and a campaign that adds nothing can coexist.
The rule: use attribution to diagnose which message or segment performs, and use a holdout to decide whether the channel is worth running. Don't swap one for the other.
How to design the test: randomise accounts, suppress the holdout, pick one outcome
A sound outbound email incrementality test design has three decisions, all made before you send a single email: who gets randomised, how the control stays clean, and what counts as success.
Randomise at the account level, not the contact level
In B2B, several people at one company can see your outreach. If you randomise contacts, one colleague lands in the emailed group and another in the holdout, and the holdout person hears about you in the next meeting. Randomise by account instead: every contact at a company gets the same assignment. This is the standard approach for B2B and it limits contamination between arms.
This is also the practical answer to geo holdout versus user-level holdout for B2B outbound. A geo holdout (suppressing whole regions) suits consumer or local campaigns where people cluster by place. For B2B outbound, account-level assignment is cleaner. Your buyers sit inside companies, not postcodes, and a geo split mixes in market differences you can't control. Use geography only if you genuinely sell region by region.
Lock the list and suppress the holdout
Build the full eligible list first, then randomise it. Use a random number or a shuffled spreadsheet column, not a judgement call. A 10 to 20 percent holdout is a practical starting point. Then add the holdout accounts to a suppression list in every sending tool so they get no outbound email, and make sure no one on the team contacts them by other outbound routes for the test window. Keep the audience fixed once the test starts.
Pick one outcome and a window
Choose a single primary outcome before launch: meetings booked, qualified opportunities, or pipeline created. Meetings are the usual choice for a small team because they happen fast and in volume. Fix the measurement window too, long enough for the outcome to occur. Replies resolve in days, meetings in weeks, pipeline in a sales cycle. Write the outcome and window down. Changing them after you see data is how people talk themselves into a lift that isn't there.
How big the holdout needs to be: sample size and minimum detectable effect
The holdout must be large enough to detect the smallest lift you would act on, and that depends on your baseline rate, not on a fixed percentage. A holdout that is too small cannot tell a real effect from noise, so a flat result tells you nothing.
Four inputs drive sample size for an email holdout test:
Baseline rate: how often accounts book a meeting without outbound.
Minimum detectable effect (MDE): the smallest lift worth acting on.
Significance level: commonly 5 percent.
Power: commonly 80 percent, the chance of catching a real effect of that size.
Plug them into a two-proportion power calculator and read off the sample needed per arm. As a worked example, assuming a two-sided test at 5 percent significance and 80 percent power, detecting a rise from 2 percent to 4 percent needs roughly 1,140 accounts per arm, about 2,280 in total.
That number surprises small teams. Smaller expected lifts and rarer outcomes need far larger samples. If your list is a few hundred accounts, you cannot detect a one-point lift in meeting rate, however carefully you run it. You have three options. Choose a bigger MDE, because only a large effect justifies the spend anyway. Run the test longer or across several cycles and pool comparable periods. Or move to a more frequent outcome such as positive replies, knowing it is a weaker proxy for pipeline.
A statistically non-significant result from an underpowered test is inconclusive. It is not proof that outbound does nothing. Say so in the readout.
Calculate incremental lift: absolute lift, relative lift and extra meetings
The incremental lift calculation takes three numbers from the same window: the outcome rate in the emailed group, the rate in the holdout, and the number of accounts you emailed.
Absolute lift = emailed rate minus holdout rate
Relative lift = (emailed rate minus holdout rate) divided by holdout rate
Incremental outcomes = absolute lift times the number of emailed accounts
Example: meetings occur for 6 percent of emailed accounts and 3 percent of holdout accounts. Absolute lift is 3 percentage points. Relative lift is 100 percent, meaning outbound doubled the meeting rate. If you emailed 1,000 accounts, outbound added about 30 meetings. The other 30 would have come anyway.
That last line is the one to take to your budget conversation. Divide your campaign cost by the 30 incremental meetings, not the 60 attributed ones, and you get the real cost per meeting outbound created. To measure incremental pipeline from outbound email, run the same arithmetic on pipeline value per account instead of meeting counts.
Report the uncertainty, not just the point estimate. A 3-point lift with an interval that runs from 0.5 to 5.5 points is a different story from one that runs from 2.8 to 3.2.
Analyse by assignment, never by outcome. This is intent-to-treat: every account counts in the group it was assigned to, whether its emails were delivered, opened or bounced. Dropping bounces or comparing only openers selects on behaviour that happened after randomisation, and that breaks the comparison you worked to build.
Mistakes that ruin a holdout (contamination, openers-only analysis, peeking, underpowered tests)
Four mistakes account for most holdouts that mislead.
Contamination. Holdout accounts get emailed through a side channel: a rep adds them to another sequence, a founder DMs them, an agency runs a parallel campaign. Even a handful of leaks pulls the holdout rate toward the emailed rate and shrinks your measured lift. Suppress the holdout everywhere and tell everyone who touches outbound.
Openers-only analysis. Comparing accounts that opened your email against the holdout looks natural and is badly biased. Openers are the more engaged accounts to begin with. You are measuring who opens, not what the email did. Stick to intent-to-treat.
Peeking. Checking results daily and stopping the moment the emailed arm leads inflates false positives. Set the window and stopping rule in advance and read the result once.
Underpowered tests. A 5 percent holdout on a 400-account list cannot detect anything short of a huge effect. Run the power check first. If the numbers don't work, extend the test or pool cycles instead of trusting a noisy read.
Two smaller ones: splitting a buying committee across arms by randomising contacts, and changing the audience mid-test. Both quietly turn a clean comparison into a story you tell yourself.
Run it every cycle: how Revnu builds the holdout into outbound and checks back
A holdout is worth far more as a habit than as a one-off, because baseline rates drift, lists change and one result is one sample. The catch for a small team is that a clean holdout is fiddly: randomise, suppress, track assignment, wait out the window, run the maths, then do it again next month.
Revnu is an agent that runs growth work for a business across channels including outbound, then measures what worked and changes what happens next. For outbound, the pieces of this process line up with how it already operates. It researches target companies and the right people, keeps them in a Leads CRM, runs sequences from your Gmail or from managed inboxes, and supports suppression so a blocked domain also stops follow-ups already scheduled. A clean holdout depends on exactly that suppression behaviour. Revnu also schedules check-backs and reports results in Slack or iMessage, so the readout at the end of the window arrives without you chasing it.
The practical split: you decide the one outcome and the minimum lift worth acting on, and Revnu works the list and checks back when the window closes. Pair the result with the success criteria you set up front, as covered in Outbound Pilot Success Criteria: When to Scale, and use a regular readout like the one described in Weekly Marketing Report: What Your AI Agent Should Explain so the incremental number sits next to the attributed one, where you can see the gap.
Hold out 10 to 20 percent of accounts on your next outbound cycle
Attribution will always tell you outbound is working, because it only counts the people you already emailed. A holdout tells you how many meetings you actually bought. Before your next send, randomise accounts, suppress the holdout everywhere, write down one outcome and a window, and check the sample size against your baseline. If you'd rather not run the mechanics yourself, start with Revnu at revnu.com. Pro is $99 a month and self-serve, and Revnu works the list and checks back with the result.
Written by
Co-founder and CEO, Revnu
