Learn · Measurement

Running your first holdout test

A holdout test keeps a small random group of members on your current programme while the rest experience a planned change. Compare the two groups over the same period. The difference in points earned, redemptions or spend shows the change's true effect, separating it from normal swings in member behaviour.

How it works

Identify one change you want to test, such as a new earning rate, a bonus offer or a reminder email. Randomly split eligible members into two groups. The holdout group continues with the existing programme and experiences no part of the change. The treated group receives the new rule or communication. Random assignment means the groups are similar apart from the change.

Run the test for a full business cycle: one month for a monthly earning period, or one campaign window for an offer. Measure the same metric in both groups, for example points earned per member, redemption rate or average spend. The difference between treated and holdout is the incremental effect of your change. Without a holdout, you might credit the change for a lift that would have happened anyway.

Worked example

The following is illustrative arithmetic. Imagine 10,000 eligible members split into 9,000 treated and 1,000 holdout. You test a double points month: treated members earn 2 points per dollar spent, control members earn 1 point per dollar. In the baseline month, both groups spend 40 dollars per member on average. During the test month, control members still spend 40 dollars and earn 40 points each. Treated members spend 44 dollars and earn 88 points each.

The treated group shows a 4 dollar, or 10 percent, lift in average spend. Across 9,000 treated members, that is 36,000 dollars of additional spend. Points issued also rise: each treated member earns 48 more points than a control member, so 9,000 members produce 432,000 extra points. If a point costs one cent to fund, the additional point cost is 4,320 dollars. You compare this cost with the 36,000 dollars of extra spend to judge whether the offer is worth rolling out.

What to do

First, define the change and the primary metric before you start. For example, you might test a 10 percent bonus on points earned for one month and measure average points earned per member. Second, split the eligible group randomly. Use a holdout of 5 to 10 percent of members, but never fewer than a few hundred, so the comparison is stable. Third, run the test for one full cycle and record the metric for both groups over the same dates. Fourth, subtract the holdout result from the treated result to get the incremental lift. Fifth, decide using both the lift and the cost: if the extra points or revenue justify the change, roll it out to all members; if not, keep the current programme and test something else.

Common mistakes

  • Picking the holdout group from a different region or tier than the treated group, which makes any lift meaningless.
  • Stopping the test after three days because early results look strong, before members have completed a normal purchase cycle.
  • Sending holdout members a related communication, such as a statement or app notification, that accidentally changes their behaviour.
  • Treating a zero difference as a failed test rather than evidence that the change added no incremental value.

Related