Email marketing A/B testing experiments

In marketing, it's important to experiment with different variables to determine what works and what doesn't, and this rule applies equally to email marketing.

This process of experimenting in email marketing is usually done by creating two or more versions of a single email, with one variable changed, and sending them to a small portion of the subscriber list to see which performs better. By systematically testing elements like subject lines, calls to action (CTAs), and send times, businesses can move beyond guesswork and make data-driven decisions that improve engagement and drive better results.

The Role of Email Marketing Experiments

The main goal of A/B testing in email marketing is to understand how subscribers act and what they prefer by performing experiments. This helps improve email campaigns to achieve goals like higher open rates, more clicks, and more conversions.

Key roles include:

• Better understanding of your audience: Experiments reveal what your audience prefers, from the subject line style to the email design, helping you target and personalise more effectively.
• Higher engagement and conversion rates: By identifying the most effective parts of an email, you can encourage more subscribers to interact with your content, boosting clicks and conversions.
• Increased ROI: A/B testing is an affordable way to improve campaign performance without extra spending. Small, data-backed changes can significantly increase your return on investment.
• Data-driven strategy: Results from email experiments provide important data to guide your long-term marketing strategy, enabling you to improve your approach over time for better outcomes.

Email marketing experiments to try

There are probably hundreds of things you can change and experiment with, but when you are just starting, it's better to stick with primary elements like subject lines, CTAs, etc. Let's explore how you can test each element and get the best results.

Subject lines

A subject line is one of the first things a user sees when your email arrives in their inbox, so it's a good idea to test different versions. You can experiment with two completely different subject lines or simply make minor adjustments.

Here are some ideas for subject line experiments:

1. Test different wording (Variant A: How to use AI in email marketing, Variant B: Is AI really useful in Email Marketing?)
2. Test with and without Emoji (Variant A: How to use AI in email marketing, Variant B: 🤖 How to use AI in email marketing)
3. Test generic vs personalised wording (Variant A: How to use AI in email marketing, Variant B: Hey {{FirstName}}, here’s how you can use AI in email marketing)
4. Test using numbers (Variant A: How to use AI in email marketing, Variant B: 10+ ways to use AI in email marketing)

These are just a few options, but you can test many other elements in a subject line, such as length and tone.

Preview text

The preview text is a short summary that appears after the subject line in your email inbox.

Most mobile and desktop email clients show this text to give you an idea of what's inside an email before you open it. Though not mandatory, it can easily be added while creating your campaign on EmailOctopus.

While it's not as noticeable as the subject line, the preview text still impacts the open rate because it can highlight a longer message, influencing whether someone opens the email or not. After you've tested the subject line, focus on optimising the preview text for better open rates.

Here are some ideas for experimenting with preview text:

1. Try different wording
2. Test with and without emojis
3. Compare text with a call to action (Example A: 20% Off Sale Today! Enjoy Christmas discount on your favourite snack today. Grab it while stock lasts. Example B: Enjoy Christmas discount on your favourite snack today. Grab it while stock lasts)

Call-to-action (CTA)

CTA usually refers to the main/primary button seen in marketing emails. As it’s the primary touchpoint when it comes to conversion, it’s a good measure to test its variations as well.

Here are some CTA experiment ideas:

1. Testing different colours
2. Testing different text (e.g., Variant A: Buy Now, Variant B: Save 25% Now)
3. Testing CTA button size
4. Testing CTA button shape (e.g., rectangle, rounded, rounded rectangle)

Sender name

Just like the subject line and preview text, the sender's name is also visible even before you open the email.

Unlike other parts, you don't have many options to experiment with the sender name. You can choose either the company name (e.g., EmailOctopus) or a person’s name (e.g., Tom from EmailOctopus). Besides the sender name, you can also change your sender profile image. To test the profile image, you’ll need to use a different sender address (e.g., newsletter@emailoctopus.com and tom@emailoctopus.com) for each variant since the profile image is linked to the email address.

If you're unsure how to change the sender image, here’s a quick guide to help you:

https://help.emailoctopus.com/article/195-setting-or-updating-a-profile-image

Send time

The time you send an email can also affect open rates and conversions. Some emails work great on weekends and some on weekdays; similarly, there are other time factors that can have a huge impact.

Try these ideas for testing send times:

1. Send on weekdays versus weekends
2. Send in the morning, afternoon, or evening
3. Send during working hours versus non-working hours

Note: If your list includes contacts from various regions, consider their time zones when testing send times. Ignoring subscribers’ time zones will lead to incorrect data.

Design element testing

You can also test different design elements in your email template to see if any changes improve your conversion rates, as design is also a key factor in conversion.

Here are some design element ideas to experiment with:

1. Trying different headline sizes
2. Changing the email content font
3. Adjusting the image layout
4. Experimenting with colours and contrast

Design offers endless possibilities, so do your research to find what might work best and then test it.

Best Practices

Now that you know what elements you can test in an email, let's look at the best practices for conducting these experiments. This will help you achieve better results and ensure your data is accurate.

Create & randomise segments

The first important step is to create random segments of your email list for the experiments. Randomness improves the accuracy of your results by eliminating bias that can arise when you segment by specific features like gender or country.

Here’s a quick guide on how to create random segments in EmailOctopus:

We generally suggest a simple way to create two segments of your email list. Start by using the first letter of each email address to sort them. Place emails starting with A, C, E, G, I, and so on, into one group until you cover about half of your list. The rest should go into the other group.

Small lists might not be reliable

While this blog post may excite and motivate you to perform similar experiments, note that the accuracy of these experiments depends on your list size, too. If your audience is still relatively small, leave these experiments for the future.

Most marketers recommend a minimum sample size of 1,000 contacts (500 per variant) to achieve statistical significance. Even though smaller lists can start with 100 contacts per group, larger lists provide more reliable data for analysing open rates, clicks, and conversions.

Documenting results

Recording A/B testing results accurately is crucial for improving future marketing emails. Make sure your document or sheet tracks key metrics like open rates, click-through rates, conversion rates, bounce rates, unsubscribe rates, and other important custom KPIs to assess each variant's effectiveness.

If you're not a marketer, you can share this sheet with ChatGPT or Gemini to interact with the data conversationally and understand it easily.

Test one variable at a time

When performing these tests, always focus on testing just one element or variable at a time. This ensures clarity about which change led to which outcome. Testing multiple variables at once can cause confusion and make it difficult to identify what caused the difference in results.

Test multiple times

It’s also highly recommended to run your A/B test experiments multiple times (2-3 times) to make sure your results are consistent. If the results are consistent, you can confidently conclude which option is best. However, if the outcomes vary, it suggests both versions performed similarly, and there is no clear winner.

Conclusion

Email marketing experiments help you make decisions based on data rather than guesses. By testing things like subject lines, CTAs, sender names, and send times, you can learn more about what your audience likes and how they behave.

To get the best results, follow these tips: use random segments, make sure your list is big enough for valid results, test one thing at a time, and repeat tests to confirm findings. This process of testing and refining will improve your campaigns now and help you create a better email marketing strategy for ongoing success.