Subject line split testing is a way of working out which of two, or three, subject lines is the most effective at encouraging people to open your email campaign. It’s also a great way to learn what works and what doesn’t, so that you can continuously move toward improving your conversion rates. As email marketing specialists, we are always recommending that our users test as much as possible, and you can also conduct a split test on an entire campaign.
4 great reasons to split test your subject lines
- To see what kind of promotion works best, for example, 20% off versus free shipping.
- To see if a generic subject line gets more opens than a longer subject line that’s more specific.
- Add a personalization to identical subject lines to see which gets a better response.
- Test two completely different topics as the subject line, to see what content is of most interest to your contacts.
Subject line split test overview
In a subject line split test, you’ll write two or three versions of a subject line for one campaign. You will also have the option to Enter preheader text, which is what will display for most recipients in their inbox preview. When the test starts, the mailing will be sent to 20% of the recipients selected. For example, if the total number of recipients is 5,000 contacts, the randomly selected test group will be 1,000 contacts. However, in order for the split test to function, you must send it to at least 30 recipients.
There is also a small box that is checked by default, this is set to automatically send the winning variation to the remaining 80% of your recipients when the test completes. You can uncheck this option, but we recommend leaving it checked because if you forget to manually choose a winner after the test has ended, the campaign will not be sent to the remaining recipients that you have selected. A subject line split test period runs for four hours, during which time opens are tracked to determine the winning subject line.
How to set up a subject line split test
The option to set up a subject line test is provided after you’ve created a campaign. So you’ll either need to create a new campaign or select one from your library and open it up in order to add the subject lines you want to test.
1. At the top right of the campaign editor, click the Review & send button.
2. You’ll be brought to a new page to enter the details of your subject line and to select the groups you would like to receive the campaign. Below the subject line field, click on the Switch to split test button and you’ll immediately see three options for the subject line instead of just one.
3. Enter two or three subject line variations. Refer to the ‘4 great reasons to split test your subject lines’ section above for ideas.
4. (Optional) Check or uncheck the ‘Autosend the winning variation’ box to automatically send the most popular subject line when the test period ends. (We recommend leaving it checked.)
5. (Optional) Connect Google Analytics to the campaign by clicking on the Add Google Analytics button on the left side and follow the prompts.
6. Review all of your options and then either click the Start split test now button to begin testing immediately or click Schedule split test to start later and follow the prompts.
How to manually select a winner and check the status of your split test
While the test is running, you can check to see which subject line is winning, and optionally select the winner yourself. To check on progress while a test is running, open the Response section of your account. Your ‘Mailings’ page will open on the ‘In progress’ tab. Click on the name of the mailing to view the test results page.
From here you have the option to end the test early, for example, if there’s a clear front runner, or a need to get the email sent earlier than planned. Determine the winning option at any time during the test, simply click End test & send this variation next to the subject line you want to send to the remaining recipients.
Frequently Asked Questions about subject line split tests
Here are some answers to questions we’re commonly asked about subject line split testing.
Why does my winning subject line have less opens than the other subject lines?
After the winning version has been sent, opens continue to be recorded for the 20% of recipients in the test group. Sometimes this can result in another variation of the subject line outperforming the version that was in the lead when the test period ended.
How is the winner chosen if my subject line split test is 50/50?
In the case of a true tie, the winner is chosen based on the highest unique open rate for the variation at the end of the testing phase. However, response data continues to track for losing variations beyond the end of the testing phase.
Can I select which of my contacts will receive the subject line split test?
No, the test group will be randomly selected to send to 20% of your chosen recipients.
Can I cancel a split test after the test has started?
No, only the account administer has the access to cancel a split test after it’s started. You can, however, end the split test at any time during the 4-hour test period by manually selecting a winner.
Can I view my split test results on the response summary email?
No, you can’t because split tests are not included in the response summary email that gets sent 24 hours after you send the campaigns, but you can see your results at any time by going to the Response section of your account.
Can I edit a split test if it’s scheduled to send later?
Yes, you can, please refer to our instructions for editing a scheduled mailing. Just be aware that canceling a scheduled mailing reverts it to a saved draft, and this will deactivate your split test settings.
To re-add your split test settings, on the ‘Review & Send’ page you’ll need to click on the ‘Switch to split test’ button again. Variation A will be preserved, but you’ll need to re-enter the other variations of your subject line.
Will new contacts that are added after the test has started be sent the campaign?
No. Any contacts added to your recipient groups or segments while the test is in progress will have to be targeted separately.