A/B testing emails is the fastest way to learn what resonates with your audience to drive more revenue per recipient, but those tests are often limited to campaigns—and often measured on direct response timelines. What about flows? And what about longer-term impact on customer LTV?
Building a comprehensive email testing program is an underutilized tactic that can have meaningful impact on metrics like repeat purchase rates and customer LTV—metrics that can better change the trajectory of a business, not just the efficiency of a channel. To do that, though, you need to invest in running holdout tests. Yet most testing guides for Klaviyo don’t cover this aspect. They focus on A/B tests for campaigns. So, we’re going to dive in.
In this post, we’ll cover how to build an email testing program Klaviyo that goes beyond traditional A/B tests.
The value of a comprehensive email testing program for Klaviyo
Building holdout tests into any email testing program is an underutilized tactic that can have meaningful impact on LTV-focused metrics, but are often overlooked because of the perceived complexity and the time it takes to develop learnings.
These learnings, though, can help you understand what an optimum lifecycle marketing program looks like, how many of your emails can be automated into Klaviyo flows, and which products you should be recommending to customers to improve cross-sell rates.
The result isn’t just potential revenue gains, it’s efficiency gains, too: If your email tests show your retention program can be optimized for timing, you can automate many of those touchpoints, reducing creative needs and resources.
Types of A/B tests to run in Klaviyo
With some ingenuity, you can test just about anything you’d like in Klaviyo. Klaviyo, though, makes it easy to A/B the following aspects of an email (or SMS message):
Email A/B tests in Klaviyo by goal
Email Open Rate Tests
To understand impact on email open rates, you can A/B test a number of variables outside of the email body, including:
- Subject line styles
- Preview text styles
- Sender name
- Send time
For these tests, you’ll want to use Klaviyo’s built-in A/B test functionality and limit the variables you’re testing per metric to understand the impact of each change.
Email Click-Through Rate Tests
To understand impact on click-through rates in your emails, you can A/B test a number of variables inside the email body, including:
- Main creative treatments (think UGC versus branded; lifestyle versus product; etc)
- Secondary offers (think cross sell, product recommendations, subscription/membership upsells)
- Value proposition themes (think product attributes versus solution value, including educational information)
- Design choices (think template structures, brand assets, etc)
Email Placed Order Rate Tests
To understand impact on placed order rates for your emails, you can A/B test a number of variables inside and outside the email body, including:
- Landing page experience (think homepage versus category page versus product page)
- Secondary offers (think cross sell, product recommendations, subscription/membership upsells)
In order to run these tests at the individual email level, you can use Klaviyo’s A/B testing tool to gain insights on the above (and other tests). If, however, you want to learn how changing these components impact customers over a longer period of time than a single email, you need to adjust your test structure to account for that. Enter the holdout test.
Creating a holdout test in Klaviyo
While a traditional A/B test tells you how a specific change within an email impacts performance of that particular email, a holdout test essentially lives a level above that: it can tell you whether sending that email is impactful at all.
Holdout tests are a specific type of email test that will tell you how exposure to a specific variable will impact a customer over time. This becomes valuable in setting your email strategy.
Klaviyo does not offer holdout test structures out of the box, but they are easy to set up. To do so, you start by:
- Tagging customers into variant and control groups
- Using customer tags to show/include variant treatments to variant groups and hide/exclude variant content from control group
- Reporting at the tag level on whether variant treatments impacted variant groups different than control groups
Need more help? Check out these comprehensive instructions (plus video) here.
This setup for a holdout test in Klaviyo is simple, but effective. And once it’s built, it is flexible as a template to help you begin testing a variety of dynamics in your email program.
One note, however: Much in the same way A/B testing requires certainly levels of email volume to understand impact, holdout tests require the same. And if you’re running them over time (on a longitudinal basis), you’ll likely need more time than you’re accustomed to.
Types of holdout tests to run in Klaviyo
Some of the same tests you can run as straightforward A/B tests can also be run as holdout tests to measure other variables that may not be as easily captured in one of the A/B tests outlined above.
For instance, you may find through particular A/B tests that a certain subject line styles positively improve open rates. This, of course, will have a positive impact on your email deliverability. But does prolonged exposure to a specific email subject line style negate the initial impact you see within an A/B test? In other words, might customers tire of that winning subject line style and, if so, how quickly does that happen?
Answering this question is not possible in a straightforward A/B test. But neither is looking at impact on things like repeat purchase rate, cross-selling rate, or LTV. This is why holdout tests are valuable.
Let’s look at some possible holdout tests to run in Klaviyo:
Holdout tests in Klaviyo by goal
Repeat Purchase Rate Tests
One of the keys to any business is converting one-time buyers into repeat customers. To understand your email program’s impact on repeat purchase rates, you can use holdout tests to learn:
- How many incremental repeat sales sending emails generates
- What the optimum timing is for sending emails that drive a second purchase
- Which products or product attributes most effectively drive repeat purchases
Cross-Sell Rate Tests
From a merchandising perspective, a tried-and-true way to improve repeat purchase rates and purchase frequency is to offer more products, so your customers have more to buy. But how do you know whether your email program is effective at cross-selling those additional products? You can use holdout tests to learn:
- Whether emails dedicated to specific products improve adoption of said products
- Which products—and which product combinations—are more effective at adding to cross-selling efforts
- When it is the best time to push specific products to drive cross-selling initiatives
Measuring holdout test performance for your email program
For many tests, you’ll be able to pull your data directly from Klaviyo to measure holdout test performance. A little Excel (or Google Sheets) knowledge will get you a long way.
For more complicated tests, you can use some of the newer business intelligence and attribution tools on the market, so long as they give you access to the datapoints you need. The most critical data points are the ones you structured the test around: the tag denoting whether a customer falls into the variant or control group and data you identified as needing to prove or disprove your test hypothesis.
An easier way to improve your repeat purchase rate
Repeat boosts repeat purchase rates by automatically delivering personalized messaging at high-leverage Moments. Learn more here.
