# Sequence A/B tests Use an A/B test to determine which version of a message has the best impact on a Sequence's conversions or engagement. ## About A/B tests for Sequences

Create a variant for any message in a [Sequence](https://www.airship.com/docs/reference/glossary/#sequence). The variant is a duplicate of the original message that you can then edit, changing its content, delivery settings, or [Channel Coordination](https://www.airship.com/docs/reference/glossary/#channel_coordination) settings. Audience allocation is set to 50% for each variant by default, but you can change the percentages. After starting the test, you will wait till the Confidence level meets or exceeds 95% and then select the winning message. The Sequence is then republished with the winning message. Audience members who receive the variant message are randomly selected on entry to the Sequence.

Related events and conversions are recorded for both audiences, providing data you can use to evaluate Sequence performance based on your selected metric.

When running A/B tests and a [Holdout Experiment](https://www.airship.com/docs/reference/glossary/#holdout_experiment) simultaneously, Airship prevents holdout group users from being included in the A/B tests. This eliminates potentially skewed data in cases where there are overlapping experimentation audiences. It also ensures that the most critical experiments maintain integrity.

To prepare for your tests, see About A/B testing.

> **Tip:** You can run Sequence A/B tests and [control groups](https://www.airship.com/docs/guides/experimentation/control-groups/) concurrently. ## Creating a Sequence A/B test > **Note:** * You must start the Sequence before you can create an A/B test. > * You cannot start an A/B test for a Sequence that has unpublished changes. 1. Go to **Messages**, then **Messages Overview**, and select the pencil icon ( ) or report icon ( ) for a Sequence. 1. Select **Experiments** in the left sidebar, then **Create an A/B test**. 1. Configure settings, then select **Save and continue**:
Setting Description
Name and description Text that describe the purpose of the test
Primary metric The metric to use for reporting experiment performance:
  • Engagement does not require additional setup.
  • Sequence conversion requires a conversion event as the Sequence's Outcome.
Variant The message to duplicate for editing
Variant allocation The percentage of the Sequence audience who will receive the variant
1. Select **Create variant**, then edit the Content or Delivery steps, or edit the [Channel Coordination](https://www.airship.com/docs/reference/glossary/#channel_coordination) setting. For Channel Coordination, select the gear icon (), make a new selection, then select **Save & continue**. > **Tip:** * A test with a single variable is measurable. When you make multiple changes in the variant, you will not know which change had an effect. > > * If your test's primary metric is Sequence conversions, consider editing any part of the message. For example, for a push notification, you could edit the title OR timing OR change your Channel Coordination selection. > > * If your test's primary metric is Engagement, focus on what users can experience before they interact with the message. For example, for an email, you would change the subject line only. 1. Select the **Review** step and review the device preview and message summary. Select the arrows to page through the various previews. The channel and display type dynamically update in the dropdown menu above. You can also select a preview directly from the dropdown menu. If you would like to make further changes, return to Review after you finish editing. Select **Send Test** to send a test message to verify its appearance and behavior on each configured channel. The message is sent to your selected recipients immediately, and it appears as a test in [Messages Overview](https://www.airship.com/docs/reference/glossary/#messages_overview). Follow the same steps as in the [Review step for a Sequence message](https://www.airship.com/docs/guides/messaging/messages/sequences/create/add-messages/#review). 1. Select **Save & continue**, and you will then see the original and variant on the A/B test summary screen. 1. Select **Start A/B test** to make the variant available to your audience or select **Exit** to save the test without starting it. To start a saved A/B test: 1. Go to **Messages**, then **Messages Overview**, and then select the pencil icon ( ) or the report icon ( ) for a Sequence to go to the [Sequence Manager](https://www.airship.com/docs/reference/glossary/#sequence_manager) or [Performance Report](https://www.airship.com/docs/reference/glossary/#sequence_performance). 1. Select **Experiments** in the left sidebar, then **View detail**. 1. Select **Start A/B test**. While the test is running: * You cannot edit the Sequence settings or messages. * On the Manage and Performance screens, the message with the variant is labeled with a flask icon (). * On the Performance screen, statistics are the aggregate of the variant and the original message. ## Selecting the winning variant After starting the A/B test, compare the performance of the original message and variant and determine which (or if either) message is having the expected impact on engagement or conversion.

You may want to end an A/B test early if you see a significant drop in conversions or engagement. If the drop is not significant or if it is observed early on in the test period, you may want to let the test continue, as the rate may correct itself. Another reason to end a test early is if you notice an error in your content. To end a test early, select a winner. This effectively cancels the test.

See also [Implementing A/B tests, outcomes, and compliance](https://www.airship.com/docs/guides/experimentation/a-b-tests/about/#implementing-ab-tests-outcomes-and-compliance) in *About A/B testing*. 1. Go to **Messages**, then **Messages Overview**, and then select the pencil icon ( ) or the report icon ( ) for a Sequence to go to the [Sequence Manager](https://www.airship.com/docs/reference/glossary/#sequence_manager) or [Performance Report](https://www.airship.com/docs/reference/glossary/#sequence_performance). 1. (From the Manage screen) Select the flask icon () for the message with the variant. 1. (From the Performance report) Select **Experiments** in the left sidebar to view basic report statistics. Then select **View results** to access the test summary. 1. In the test summary, compare the metrics for the original message and the variant: | Data | Description | | --- | --- | | **Sample size** | The number of users selected to receive the variant. The threshold is 10,000 users. | | **Lift** | The percent increase or decrease of your primary metric for users who have received the variant. Presented after seven days or when the sample size of 10,000 users is reached. | | **Confidence** | The probability that the same results would be obtained if the test were repeated. Presented after seven days or when the sample size of 10,000 users is reached. | Additional statistics are displayed for the original message and variant based on the test's primary metric. If your primary metric is Engagement, select a message type to view statistics for that type only. 1. After Confidence is at least 95%, select a winner. The Sequence will be republished with the winner, and the A/B test ends. First select **Select a winner**, then **Select original** or **Select variant**, then confirm your choice. > **Note:** The winning message is saved as configured. You cannot select the content or settings per channel or message type. ## Viewing A/B test history After selecting a winning message, the A/B test is added to the list of past experiments. 1. Go to **Messages**, then **Messages Overview**, and then select the pencil icon ( ) or the report icon ( ) for a Sequence to go to the [Sequence Manager](https://www.airship.com/docs/reference/glossary/#sequence_manager) or [Performance Report](https://www.airship.com/docs/reference/glossary/#sequence_performance). 1. Select **Experiments** in the left sidebar. 1. Select **Past Experiments**. Each A/B test is listed by name along with its end date. Select the report icon ( ) for an A/B test to open its summary.