While Klaviyo has built-in support to set up A/B tests for email and SMS, there’s no equivalent for creating a holdout test. You can, however, create them easily—you just need to use operational flows and customer tags.
The effort is worth it. A/B tests may tell you whether a specific aspect of an email proves impactful, but a holdout test can tell you whether even sending an email was impactful.
Therein lies the value of a holdout test: They’ll tell you how exposure to a specific tactic will impact a customer over time, helping you shape your overall retention strategy.
Creating holdout test in Klaviyo requires three steps:
- Structuring your holdout test
- Setting up your test segments
- Running your test with segment filters
Here’s how to do it.
Structuring your holdout test
The first step in creating a holdout test is to design the test.
You’ll need to determine how many test segments you need for your holdout test. The number of variants you’ll need is dependent on the test you’re planning to run. For purposes of these instructions, we’ll assume three segments: a control audience and two variant audiences.
This type of segmentation is helpful, for instance, in understanding whether certain tactics—campaigns versus flows, flow timing, number of emails sent, etc—impact repeat purchase rates.
An example test might be testing whether dedicated replenishment flows and lapsed customer prevention flows deliver incremental revenue via higher repeat purchase rates or higher returning customer AOV.
Here’s an example of how that might look:
Once you’ve set your segments and defined which variant audience receives which variant tactics, it’s time to build the holdout test in Klaviyo.
Setting up your holdout test segmentation
To run a holdout test like the above in Klaviyo, we’re going to tag new customers into one of the variant groups or the control group using Klaviyo’s customer tags via an operational flow.
To set up that flow, we’ll first:
- Start a new flow from scratch
- Trigger the flow when someone has placed an order
- Filter the flow to apply only to customers who have placed one order
Once complete, we can begin randomly sampling customers into the Control, Variant 1, and Variant 2 groups. If we want to equally split customers among the three segments, we’ll need to do the following:
- Use the “random sample” choice on Klaviyo’s conditional split tool
- For the first conditional split, send 33% of traffic to the “Yes” branch and the remaining 66% of traffic to the “No” branch
- Create a second conditional split on the “No” branch, sending 50% of the remaining traffic to the “Yes” branch and the other 50% of the remaining traffic to the “No” branch
(Note: You’ll need to adjust the split volume to ensure each segment gets 33% of the volume.)
The final step is tagging each of the segments.
To do that, you’ll add an “Update Profile Property” action to the end of each conditional split, creating a new profile property and adding your tag naming convention.
It should look like this:
This test structure will apply to all new customers, meaning your holdout test will begin amassing traffic when you turn on the flow, but you won’t see results from this test until your Klaviyo filters (discussed in the next section) begin firing.
Using Klaviyo filters to run a holdout test
While Klaviyo is building your holdout test segments using the operational flow covered above, you’ll also need to use Klaviyo filters to complete the setup of your test.
If we’re using the test design example from above, those filters are used to include/exclude certain segments from flows.
For example:
- Control would be excluded from all flows being tested
- Variant 1 would be included in the replenishment flows being tested, but excluded from the lapsed customer prevention flow being tested
- Variant 2 would be included in all flows being tested
To set these filters, you use Klaviyo’s flow filter functionality on the flow trigger.
Because Klaviyo’s flow filters are inclusive by definition, you’ll want to add both Variant 1 and Variant 2 to the flow filter (make sure to use the “or” operator), but not the Control. It should look like this:
Repeat this process for the remaining flow/segment combinations.
Once done, your test is fully live.
Measuring a Holdout test in Klaviyo
If you want a cohorted view of your segments, measuring a holdout test in Klaviyo requires some spreadsheet work.
An easier, scrappier—though not necessarily ideal—way to understand impact is to create a single metric report from scratch in Klaviyo. You can use this report to look at the Placed Order value stored in Klaviyo and group by your “repeat-test-group” profile property.
Doing so will show you how many orders each segment has placed over your defined time frame, giving you the raw data needed to understand impact.
An easier way to improve your repeat purchase rate
Repeat boosts repeat purchase rates by automatically delivering personalized messaging at high-leverage Moments.