How to run A/B Tests in eCommerce Email Marketing

The majority of A/B testing in email marketing is a waste of time.

There, I said it.

It’s not that marketers don’t have good intentions when it comes to testing. It’s just that their energy is often misplaced.

Since starting our agency, I’ve seen the backend of over 100 different eCommerce brands CRM systems.

There’s been a huge variety of testing deployed across these brands, from both incomprehensibly complex tests to little or no testing, and everything in between.

There’s a pattern I’ve noticed here:

Test too much and you overwhelm yourself and struggle to harvest meaningful findings.

Test too little and you leave money on the table.

Unfortunately, however, no matter which end of the spectrum it’s fallen, the majority of A/B testing I’ve seen has been largely ineffective.

Why is that?

Simple: marketers are testing the wrong things which don’t move the needle.
‍

Not all A/B tests are created equal

First, let me clarify that I’m not against scientific testing, as that would be stupid.

Executed correctly, A/B testing and experimentation as a business practice is essential to fostering a results-driven culture that removes bias.

It can be an incredibly data-driven, enlightening experience to establish a culture of successful testing in any organisation.

This is why it’s part of Phase 3 in our Email Marketing Bible framework we follow with all our clients at Magnet Monster.

But you need to focus on what actually moves the needle most efficiently.

A CTA in a customer winback flow or a headline in a Sunset Flow, for example, is usually a waste of time and resources. They have no defined metric (or at worst, vanity metrics) for determining success, and seldom move the needle in practical ways that translate to business success.

When this happens, it can lead to a disillusioned state that A/B testing is a waste of time.

This can discourage your team and lead to an anti-scientific culture, or one that doesn’t value its importance.

DON'T MISS: The most powerful A/B test I’ve ever ran opened my eyes to discounting damage‍

Email Marketing Rules for A/B Testing

There are 3 primary criteria I look for when deciding where to run an A/B test with a brand I’m working with. They are:

Sufficient traffic (sample size) to have a significant effect and repeatable results
A clearly identified metric choice for making the decision (i.e. clicks VS conversions)
Long enough time for the experiment to run its course and result in statistical significance

There are other factors that should be taken into consideration when devising a testing protocol, but for the sake of simplicity, these are the core 3 I generally use to set up tests.

Test the Big 3

When you whittle down the number of tests at your exposure to meet the aforementioned criteria, what you’re generally looking to do is set up effective tests that increase revenue for the brand you’re working with.

This usually leaves you with 3 core elements to run frequent tests on:

Signup Forms
Welcome Flow
Abandoned Cart Flow

The overwhelming majority of tests we set up for our clients are focused on these 3, simply because they drive the biggest change in the business.

To illustrate my point: consider the average conversion rate of a Welcome Flow (15%) when compared to a Winback Flow (3% if you’re lucky). It is just far more logical to iterate on the Welcome Flow more frequently as the results are tangibly so much greater to the business than customers who are about to churn.

These elements of the CRM system just have a much more vast sample size to work with, consistent traffic and higher conversion rates, making incremental improvements much more profound when working on them.

There are two additional elements I will test with a moderate frequency once a consistent protocol has been set up for the above. They are:

Post-purchase upsells
Campaigns

More on these in another article though.

1. Signup Form Tests

Testing signup forms should be the first port of call for any eCommerce brand. They can provide masses of data as well as be iterated on rapidly leading to statistical significance. This gives signup forms huge leverage in your business to drive actionable insights and results.

It is very important when testing signup forms to realise you’re not solely tracking conversion rates on the forms themselves. The primary metric you’re looking for is follow through to conversion once the code has been harvested (assuming you’re using this as an incentive), as well as whether you can generate additional zero-party data that can help with future forecasting (more on this later).

Here are some good starting points on tests you can run on signup forms that can rapidly be iterated on:

Offer on form: the offer you serve on a pop-up will have the biggest impact on signup rates and conversion rates across the board. Try testing discount % VS monetary value (i.e. $10 off first purchase), lead magnets, loyalty club incentives and sweepstakes to find out what will drive the most sign-ups but crucially, what also generates conversions within a designated time frame (as these will typically hold more value to the brand)
Behaviour of pop-up: timing, page scroll, exit-intent and pop-up type (full-screen takeover, slide-in form, etc) are all variables that can be used to rapidly gather data on a weekly basis for most stores
Zero-party data collection: how many fields and data points will the customer provide before a tangible drop-off in performance and how can this be used to help shape the customer journey?

If the store you’re working on has enough traffic, you could arguably justify investing over 80% of your efforts alone into testing signup forms based on the aforementioned recommendations.
‍

2. Welcome Flow Tests

From a flow perspective, nothing converts as well as the Welcome Flow in your email marketing setup. It’s also probably the most important as it onboards newly acquired customers and shares key information about your value proposition and brand story.

Here are some tests that can be leveraged to improve the impact of your Welcome Flow:

Headlines: important for increasing open rates and getting eyeballs on the emails (NOTE: you’re looking for follow-through to conversions here, not just open rates alone)
The cadence of the emails and how they subsequently drive clicks (and again, conversions)
Timing: can a more aggressive follow-up sequence (daily) be used to drive more conversions than a less aggressive nurture sequence? Experiment with the length and frequency of messaging in the flow to determine what converts best within the first 30 days
Use of Zero-party data in the flows: derived from the pop-up; can we help improve conversions by leveraging the data we’ve acquired in intelligent ways? There are innumerable permutations to test here if you’re creative enough
‍

3. Abandoned Cart Flow Tests

This is a critical element to test as the customers at this stage of the funnel have obviously shown intent to purchase. A lot of energy should go into optimising the abandoned cart flow for this reason.

I would recommend testing:

Offer VS no offer - can be a great test to see whether we’re negatively impacting contribution margin or improving profitability
Timing can be tested to see how it impacts conversion rate
Email VS SMS to monitor the impact on conversion rate
Deploying a conversational approach in sections of the funnel
Headlines can be tested to see their impact on open rates (and subsequent conversion rates)
Email design (use of hero image VS no hero image; plain text VS HTML)

DON'T MISS: How to build an email marketing strategy for a 6, 7, 8 & 9-figure store

Conclusion: Test what matters

As you can see from the above, there are innumerable and often timely tests you can run across multiple areas of your email marketing setup that alone can consume a lot of resources just on those 3 areas.

Once a consistent testing protocol has been established for these 3, only then should you proceed to experiment on different areas of the funnel.

The only exception to this rule would be if your business stocks hundreds - if not thousands - of SKUs and sends out regular campaigns, but even then, testing conditions are likely to be far less stable and erratic which can lead to skewed results.

For all intents and purposes, testing signup forms, the welcome flow and abandoned cart flow, will keep you busy and productive enough to drive tangible business results in your email marketing strategy that provide actionable insights that significantly improve the business.

‍

I cover this more extensively in Monster Email Marketing where I spill the beans on the exact strategies I've used to win deals with 8-9 figure DTC brands and continuously drive an additional 15% revenue from every email campaign you send.

Enjoyed this article on eCommerce email marketing? Follow me on Twitter or LinkedIn, and don't forget to join 2,000+ hungry D2C enthusiasts who lap up our weekly insider insights on eCommerce email marketing.