The importance of testing
When it comes to email, marketers know that the effectiveness of messaging and content is nowhere near an exact science. There are so many ever-changing variables: design trends, your company’s creative strategy, and even how people receive their emails. So, with all of these consistently moving parts, marketers are also keen to test any number of variables of content within their emails to know what’s working. These tests allow you to hone in on what your subscribers are looking for and can lead to an overall higher performance from your email strategy.
When to test
One of the best times to look at testing is when you are analyzing a regularly-sent campaign, such as your recurring newsletters. Because these campaigns have rich historical data, you can more easily determine the success of the tests you run. When you notice these campaigns have plateauing open or click rates, come up with a testing strategy to see what improvements lead to higher engagement, and a better understanding of your audience.
What to test
Once you've decided it's time to test one or more of your email campaigns, the next step is determining how to test. Delivra's A/B testing makes it easy to set up a subject line test or design test. If you follow the same structure with your subject line, such as "Monthly Newsletter: May 2020," try mixing it up with emojis or use different punctuation. For design tests, look to improve your click rates, or click-to-open rates by testing plain text emails vs. designed templates, layout changes, or adding images. Testing these changes can indicate how your audience consumes your content.
How to test
Once you've decided what part of your email to test, the next step is determining your test group. There are generally two recommended strategies: Splitting your audience in half, and sending your variant (the new version) to one group and the control (what you typically send) to the other. Or, take a small portion of your audience to test each version and send the winning version to the majority of your audience. There are advantages and disadvantages to both testing strategies. An A/B test (50/50 split) is easier to set up and run because you don't have to factor in wait times to deliver the winning version. However, running a small test before reaching your full audience gives a majority of your audience the winning version. If you plan to test a portion your audience to determine a winning version, keep in mind your test audience needs to be large enough to clearly determine a winner and you also should give that audience enough time to interact with your campaign (you should let the test run for several hours, if not a complete 24 hours).
Best Practices
What are some email testing best practices to keep in mind? Here are six A/B testing best practices to keep your process more focused, efficient, and productive.
- Have an end goal. Don’t just form a hypothesis. Identify your end goal, too. If your typical open rate (before it started to decline) was 25%, do you aim for that or a little higher? Name the number and stick to it. A/B testing allows you to learn more about email marketing and your subscribers during the process. Its purpose is not solely to increase stats.
- Conduct simultaneous tests. Run both variations of an A/B test at the same time. If you do another round of split testing to improve on your previous “winner,” run that test under similar conditions, too. Not doing so can lead to tests having wildly inaccurate results.
- Keep a control version. A/B testing works best if one of the variations is your original email campaign. This is your real and reliable baseline. If you want to test more than two variations, conduct separate A/B tests for each. This will make confounding variables easier to spot.
- Act on statistically significant results. Perform email testing on the largest subset of your email list as you can. This will lessen the probability of random chance or error and increase the accuracy of your results.
- Test continuously and test often. This doesn’t cancel out having an end goal. When you reach your goal, implement the change proven effective by testing. However, think of new hypotheses when you have a moment to spare. Running A/B tests in the background all the time with different variables changed can lead to highly optimized email campaigns months down the line
- Trust the data. Don’t ignore the results when they come in and don’t choose your gut over empirical data. Don’t wait until your email list evolves and your subscriber behavior and demographic changes completely before getting around to your action plan.
Wrap Up
If A/B testing is a new strategy for you, it's best to start small. Choose one campaign and optimize it over a long period of time. When one test becomes conclusive, look at the next element to test in that same email. Once you have a testing process in place that has shown results, you can more easily replicate the process to your other campaigns. Learn about Delivra's A/B Test functionality.
