Subject line split testing is a way of working out which of two, or three, subject lines is the most effective at encouraging people to open your email campaign. It’s also a great way to learn what works and what doesn’t, so that you can continuously move toward improving your conversion rates. You can also use a content split test to test entire campaigns.

Who can create a subject line split test?

Depending on your account type, certain users may not be able to create content split tests. Users in the Editor, Author, and Response only roles cannot create content split tests. Users in the Parent, Manager, Administrator, and Full roles can create content split tests.

Users must have permission to send campaigns in order to create subject line split tests. Users who cannot send campaigns also cannot create subject line split tests.

How to create a subject line split test

In order to conduct a subject line split test, you’ll need to create a campaign. Once you’ve done that and you’re ready to send, follow the instructions below.

Split tests cannot be sent to subscriptions, only to groups and segments. If you would like to send a split test to a subscription, you will need to create a segment based on that subscription and then send the split test to that segment instead.

1. Click on the Review button or the Review & send button, depending on which editor you are using.
2. On the next screen, click on the Switch to split test button.
   
3. Enter two or three subject line variations.
   
4. (Optional) Check or uncheck the Autosend the winning variation box to automatically send the most popular subject line when the test period ends. We recommend leaving it checked.
5. (Optional) Connect Google Analytics to the campaign by clicking on the Add Google Analytics button on the left side and following the prompts.
6. Select your audience for the campaign and double check your sender name, sender email, and reply-to email.
7. Review all of your choices and click on the Start split test now button to begin testing immediately or click on Schedule split test to start later and follow the prompts.

When the test starts, the mailing will be sent to 20% of the recipients selected. For example, if the total number of recipients is 5,000 contacts, the randomly selected test group will be 1,000 contacts. However, in order for the split test to function, you must send it to at least 30 recipients. Subject line split tests run for four hours, during which time opens are tracked to determine the winning subject line.

There is also a small box that is checked by default, this is set to automatically send the winning variation to the remaining 80% of your recipients when the test completes. You can uncheck this option, but we recommend leaving it checked because if you forget to manually choose a winner after the test has ended, the campaign will not be sent to the remaining recipients that you have selected.

When our system automatically sends the winning campaign, it will identify the winner based on the response data from the four hour test period. It is possible that the response data will change after more time elapses so that it looks like the “losing” campaign was sent instead of the winning campaign. When this occurs, it is because the “losing” campaign had more engagement during the four hours test period and so was actually the winning campaign at the end of the test period.

How to manually select a winner and check the status of your split test

While the test is running, you can check to see which subject line is winning, and optionally select the winner yourself. To check on progress while a test is running, open the Response section of your account. Your Mailings page will open on the In progress tab, where you can click on the name of the mailing to view the test results page.

From here you have the option to end the test early, for example, if there’s a clear front runner, or if you need to get the email sent earlier than planned. Determine the winning option at any time during the test, simply click End test & send this variation next to the subject line you want to send to the remaining recipients.

Frequently asked questions about subject line split tests

Can I send a split test to a subscription?

No, split tests cannot be sent to subscriptions, only to groups and segments. If you would like to send a split test to a subscription, you will need to create a segment based on that subscription and then send the split test to that segment instead.

Why does my winning subject line have fewer opens than the other subject lines?

After the winning version has been sent, opens continue to be recorded for the 20% of recipients in the test group. Sometimes this can result in another variation of the subject line outperforming the version that was in the lead when the test period ended.

How is the winner chosen if my subject line split test is 50/50?

In the case of a true tie, the winner is chosen based on the highest unique open rate for the variation at the end of the testing phase. However, response data continues to track for losing variations beyond the end of the testing phase.

Can I select which of my contacts will receive the subject line split test?

No, the test group will be randomly selected to send to 20% of your chosen recipients.

Can I cancel a split test after the test has started?

No, subject line split tests cannot be canceled once they’ve started sending. You can, however, end the split test at any time during the 4-hour test period by manually selecting a winner.

Can I view my split test results in the response summary email?

No, you can’t because split tests are not included in the response summary email that gets sent 24 hours after you send the campaigns, but you can see your results at any time by going to the Response section of your account.

Can I edit a split test if it’s scheduled to send later?

Yes, you can, please refer to our instructions for editing a scheduled mailing. Just be aware that canceling a scheduled mailing reverts it to a saved draft, and this will deactivate your split test settings.

To re-add your split test settings, on the Review & send page you’ll need to click on the Switch to split test button again. Variation A will be preserved, but you’ll need to re-enter the other variations of your subject line.

Will new contacts that are added after the test has started be sent the campaign?

No. Any contacts added to your recipient groups or segments while the test is in progress will have to be targeted separately.

Additionally, if you have scheduled the split test in advance, any contacts who were added to the groups / segments that make up the audience of the mailing after the mailing was scheduled will not receive it. This is a result of the way split tests are queued for sending and cannot be changed.