# Message A/B tests

Experiment with up to 26 message variations to determine audience engagement.

See also [Scene A/B tests](https://www.airship.com/docs/guides/experimentation/a-b-tests/scenes/) and [Sequence A/B tests
](https://www.airship.com/docs/guides/experimentation/a-b-tests/sequences/).

## About A/B tests for messages

<p>Create variants of message content by duplicating the initial variant or from scratch. Each variant returns analytic data to help you determine the most effective way to engage your audience. Audience allocation is automatically evenly distributed between variants and a control group, but you can change the percentages.</p>

A/B tests for messages support these channels and message types:

* App — Push notifications, in-app messages, and Message Center
* Web
* Email
* SMS
* Open channel

Set up the test in two steps:

1. **Create two or more message variants** — Just like in the [Message composer](https://www.airship.com/docs/guides/messaging/messages/create/), for each variant, select channels, configure content for each channel, and set up delivery.

1. **Allocate an audience** — You can designate all users as eligible for the test or target specific users. Of that group, set the percentage that will participate in the test, distributed among variants and a control group. Audience members are randomly selected.

   The control group is the portion of your audience that won't receive any messages but will be included in the performance report for comparison. If you do not want to use a control group, you can set its percentage to 0% when [setting the test audience](#set-the-test-audience). The remaining audience, if you've allocated less than 100% between variants and control, won't receive any messages and won't be included in the report.

   Setting a lower audience percentage lets you test with a smaller group so you can effectively manage feedback and interpret results. By default, the overall audience percentage is divided evenly between variants and a control group, but you can set your own values.

After creating variants and setting the audience, you can start the test and review its results.

<p>When running A/B tests and a [Holdout Experiment](https://www.airship.com/docs/reference/glossary/#holdout_experiment) simultaneously, Airship prevents holdout group users from being included in the A/B tests. This eliminates potentially skewed data in cases where there are overlapping experimentation audiences. It also ensures that the most critical experiments maintain integrity.</p>

<p>To prepare for your tests, see <a href="https://www.airship.com/docs/guides/experimentation/a-b-tests/about/">About A/B testing</a>.</p>

## Create a message A/B test

First, select the **Create** dropdown menu (
), then **A/B Test**. Or you can start from your list of all A/B tests by going to **Experiments**, then **Message A/B Tests**, and then selecting **Create A/B Test**.

Next, select the test name and change it to something descriptive, then select the check mark to save it.

To finish setting up your test, you must add message variants and determine the audience. You can configure them in any order.

> **Tip:** You can also create a test from the [Message composer](https://www.airship.com/docs/guides/messaging/messages/create/). In the Review step, to use the message as an A/B test variant, select **Create A/B Test**. This creates a new A/B test with your message as the first variant.


### Add message variants

You can add up to 26 variants to an A/B test:

1. Select **Add variant**. After completing a step, select the next step in the header to move on.

1. For **Channels**:

   <p>First, select a [Channel Coordination](https://www.airship.com/docs/reference/glossary/#channel_coordination) strategy:</p>
   <ul>
   <li><strong>Fan Out</strong> targets a Named User on all the channels they are opted in to, maximizing the chances they receive your message.</li>
   <li><strong>Last Active</strong> targets a Named User on the opted-in channel they used most recently.</li>
   <li><strong>Priority Channel</strong> targets a Named User on the first channel they are opted in to, in the priority order you set.</li>
   </ul>
   <p>Then, enable the channel types to include in your audience. For Mobile Apps, also select from the available platforms. For Priority Channel, also drag the channel types into priority order.</p>

   > **Note:** For projects using the [channel-level segmentation system](https://www.airship.com/docs/guides/audience/segmentation/segmentation/#channel-level-segmentation), instead of Channel Coordination, enable the channels you want to send the message to.


   <p>Use <strong>Channel conditions</strong> to filter which channels are included in the audience. A channel must meet the conditions to remain in the audience.</p>
   <p>For example, if your audience includes users with app, email, and SMS channels, and you set a channel condition requiring membership in an email Subscription List:</p>
   <ul>
   <li>Only email channels that meet that condition would remain in the audience.</li>
   <li>All app and SMS channels would be excluded.</li>
   </ul>
   <p>To set channel conditions, use the same process as when building a [Segment](https://www.airship.com/docs/reference/glossary/#segment). You can use the following data in your conditions:</p>
   <ul>
   <li>[Autogroup](https://www.airship.com/docs/reference/glossary/#autogroup)</li>
   <li>[Channel ID](https://www.airship.com/docs/reference/glossary/#channel_id)</li>
   <li>[Device Properties](https://www.airship.com/docs/reference/glossary/#device_properties)</li>
   <li>[Events](https://www.airship.com/docs/reference/glossary/#events)</li>
   <li>[Lifecycle List](https://www.airship.com/docs/reference/glossary/#lifecycle_list)</li>
   <li>[Predicted to Churn status](https://www.airship.com/docs/reference/glossary/#predicted_to_churn)</li>
   <li>[Subscription List](https://www.airship.com/docs/reference/glossary/#subscription_list)</li>
   <li>[Tag](https://www.airship.com/docs/reference/glossary/#tag) in the <code>device</code> [Tag Group](https://www.airship.com/docs/reference/glossary/#tag_group) — See <a href="https://www.airship.com/docs/guides/audience/tags/#device-tags">Primary device tags</a>.</li>
   <li>[Uploaded (Static) List](https://www.airship.com/docs/reference/glossary/#uploaded_list)</li>
   </ul>
   <p>Selected Lifecycle, Subscription, and Uploaded Lists must contain Channel IDs or Named Users as the identifier, not a mix of the two.</p>

   > **Note:** Setting channel conditions is not supported for projects using the [channel-level segmentation system](https://www.airship.com/docs/guides/audience/segmentation/segmentation/#channel-level-segmentation).


   Under **Localization**, enable the option if you want to provide different content to app and web users depending on their language and country.

1. For **Content**, configure the message content per enabled channel. See the [Content documentation](https://www.airship.com/docs/guides/messaging/messages/content/) per message type, [Content options](https://www.airship.com/docs/guides/messaging/in-app-experiences/configuration/optional-features/), and [Localization](https://www.airship.com/docs/guides/messaging/messages/localization/).
   
1. For **Delivery**, configure the message delivery timing and options. See [Message delivery](https://www.airship.com/docs/guides/messaging/messages/delivery/delivery/).

1. In the **Review** step, review the device preview and message summary:

   * Use the arrows to page through the various previews. The channel and display type dynamically update in the dropdown menu above. You can also select a preview directly from the menu.
   * If you want to make changes, select the associated step in the header, make your changes, then return to Review.
   * Select **Send Test** to send a test message to verify its appearance and behavior on each configured channel. The message is sent to your selected recipients immediately, and it appears as a test in [Messages Overview](https://www.airship.com/docs/reference/glossary/#messages_overview). Follow the same steps as in the [Review step for the Message composer](https://www.airship.com/docs/guides/messaging/messages/create/#message-review).

   When your review is complete, select **Save Variant**.

To add another variant from scratch, select **Add variant**. To duplicate an existing variant, select the more menu icon () at the end of a row and select **Copy to variant**.

### Set the test audience

After creating an A/B test, select **Audience** and then set up your test audience:

1. Choose and configure users:

   | Option | Description | Steps |
   | --- | --- | --- |
   | **All Users** | This option makes the test available to a percentage of your total audience. | n/a |
   | **Target Specific Users** | This option makes the test available to a percentage of users who meet specified conditions. | Select and configure one or more conditions. Use the same process as when building a [Segment](https://www.airship.com/docs/reference/glossary/#segment). |
1. Under **Audience allocation**, limit the selected audience to your specified percentage.
1. (Optional) To override the default variant distribution, enable **Allow uneven allocations**. Then, under **Variant allocation**, edit the percentage for each variant and the control group.
   > **Note:** If you later add more variants, also update your variant allocation settings.

1. Select **Save**.

### Start an A/B test

Once you've created your message variants and set the audience for your test, select **Start A/B test** and confirm. Each variant will send according to its delivery settings.

## View test results

After starting an A/B test, discover which variant performed best. Use the test- and message-level reports to determine the quality of each variant and strategies for increasing engagement. See also [Implementing A/B tests, outcomes, and compliance](https://www.airship.com/docs/guides/experimentation/a-b-tests/about/#implementing-ab-tests-outcomes-and-compliance) in *About A/B testing*.

To access test results, go to **Experiments**, then **Message A/B Tests**, select the more menu icon () for a test in the list, then **View results**. You can also select the name of a test from the list and then go to **Results**.

* A **Performance** section for each channel contains statistical data for each variant per channel and the control group, if any. Select a variant name to open its [message report](https://www.airship.com/docs/guides/reports/message/).
* **Message Detail** contains the same information and preview options shown in the Review step when creating each variant and in a variant's message report.

To export test data, select **Download CSV**, then **Performance Data**. If your test included [Custom Events](https://www.airship.com/docs/reference/glossary/#custom_event), you will also have the option to download **Variant Event Data**, which is a report of event conversions and associated values, broken out by variant and control group, if any.

> **Note:** Engagement data is sent to Airship as soon as it becomes available. Data may be delayed due to connectivity issues with a user's carrier, wifi, power, etc. Wait at least 12 to 24 hours before acting on test data to allow for potential lags.


## RTDS events

Messages used as variants in an A/B test include experiment information in [Real-Time Data Streaming](https://www.airship.com/docs/reference/glossary/#rtds) events.

The [Send event](https://www.airship.com/docs/developer/rest-api/connect/schemas/events/#send) includes an `experiments` object with the test details, including `experiment_id`, `type`, and `variant_id`. The `experiment_id` also appears in the `body` object.

The [Control event](https://www.airship.com/docs/developer/rest-api/connect/schemas/events/#control) includes an `experiment_id` at the top level and also in the `body` object.

## Managing A/B tests

Go to **Experiments**, then **Message A/B Tests** to view and manage your Message A/B Tests. Each test is listed by name with its status and the date it was last modified. Your last modified test is listed first. You can search for tests by name.

Options from the tests list:

| Option | Description | Steps |
| --- | --- | --- |
| **View** | Open the test to access its message variants, audience configuration, and results. | Select a test's name. Or select its more menu icon () and then **View test**. |
| **Duplicate** | Make a draft copy of a test with its message variants and audience configuration. | Select a test's more menu icon () and then **Duplicate**. |
| **View results** | Open the test's performance reports. | Select a test's more menu icon () and then **View results**. See [View test results](#view-test-results). |

### Editing message variants and audience

You can edit variants and audience settings for any test that has not yet been started. After opening a test from the message A/B Tests list, select the more menu icon () for a variant and select an option:

| Option | Description |
| --- | --- |
| **Edit** | Modify the variant's channels, content, or delivery settings. |
| **Duplicate** | Create a copy of the variant as a starting point for a new variant. |
| **Delete** | Remove the variant from the test. |

To modify the test audience, select **Audience** and adjust targeting or allocation settings. See [Set the test audience](#set-the-test-audience) for configuration details.