With an A/B split test, you can test different versions of your campaign against each other and find out what works best with your recipients. You can split test by subject line, content, sender and send time.
In this guide, you’ll learn what an A/B split test is and how to set it up in Heyloyalty.
What is an A/B split test?
In an A/B split test, you compare two different versions of the same campaign by splitting the traffic equally between them. A small portion of your recipients gets version A, another small portion gets version B, and the version that performs best is automatically or manually sent out to the remaining recipients.
That way, you make your decisions based on data – not gut feelings.
Note: We recommend that you only test one thing at a time. If you test both content and subject line, you won’t know which factor made the difference.
Step 1: Enable A/B split testing
When you create your campaign, in Step 1, select AB split test campaign under Broadcast type.
Also choose a List and an internal campaign name.
Step 2: Set up A/B split testing
Under A/B split test settings, you decide how many of your recipients should be included in the actual test. You can choose either a percentage or a specific number of recipients.
For example, if you enter 10% in the field:
- 5% receive version A
- 5% receive version B
The remaining 90% receive the winning version.
You also need to consider the following settings:
- Winner criteria: Choose whether to determine the winner based on the highest open rate or the most clicks.
- Send winner: Choose whether to send the winning version automatically or manually.
- Perform AB split test on: Select the test parameter. You can test on subject line, content, sender or send time. You can select multiple test parameters at the same time.
- Subject line: The subject the recipient sees in their email client. If you are testing on subject line, you need to enter both an A and a B variant.
- Preheader: The text that appears below the subject line in the email client. You cannot do A/B testing on the preheader.
- Sender: The sender the recipient sees in their email client. If you test on sender, you need to choose a sender for both the A and B variant.
- Broadcast time: Select when to broadcast the campaign. If you are testing on broadcast time, you need to select a time for both the A and B variant.
- Winner send time: If you choose to send the winner automatically, you need to specify when to send the winning version.
Note: Remember to leave time between broadcasts. If you test on broadcast time, the A, B and winning variants cannot be sent at the same time.
Once the settings are in place, scroll to the top and click Next to set up the campaign.
Test content – create an A and a B variant
If you choose to test on Content, you need to create two versions of the campaign in Step 2. If you are not testing on content, you can skip this section.
Versions A and B are created in the same way as a regular campaign. When you get to Step 2, variant A is shown first. To edit variant B, switch the variant here:
If the content is almost identical in the two variants, you can copy the content from variant A to variant B. To do this, go to Variant B and click Copy variant A to variant B.
Send the winning variant manually
If you choose to send the winner manually, you must select the winning version yourself after the campaign has been sent. To do this, go to: Campaigns → click on the three dots next to the A/B split test → See test details
Here you choose the winner by clicking on either Variant A or Variant B.
Note: The winner information for your campaign is only saved for one week. If you want to choose the winning email manually, you must do so within a week of the split test ending.
Find the statistics
To see the general statistics for the entire A/B split test, such as how many recipients the campaign was sent to, open rate and click rate, click the three dots next to the campaign and select Statistics.
To see test details for the A and B variants that were sent out, click on the three dots next to the campaign and select View test details. Here you can see how the two versions performed and which version won.
You are now ready to test
You have now learned how to set up an A/B split test in Heyloyalty, choose the test parameter and find the statistics afterwards. For example, start by testing two different subject lines on your next newsletter and you’ll quickly gain insight into what resonates best with your recipients.
The more you test, the sharper your campaigns will be. Have fun with your next split test!