# Legacy message A/B tests Experiment with up to 26 message variations to determine audience engagement. > **Important:** This page is for the **legacy** message A/B tests. In our [current Message A/B tests](https://www.airship.com/docs/guides/experimentation/a-b-tests/messages/), the architecture allows multiple messages to be grouped as variants within a single test. You can define the test audience and allocation at the test level, separately from creating message variants. Delivery is also at the variant level. This flexible structure enables testing any part of a message, such as content, send time, delivery channels, and more. ## About A/B tests for messages Create variants of message content by duplicating the initial variant or from scratch. Each variant returns analytic data to help you determine the most effective way to engage your audience. You can retain a control group or send to 100% of your selected audience. Legacy A/B tests for messages support these channels and message types: * App — Push notifications and in-app messages * Web * Email * SMS * Open channel

When running A/B tests and a [Holdout Experiment](https://www.airship.com/docs/reference/glossary/#holdout_experiment) simultaneously, Airship prevents holdout group users from being included in the A/B tests. This eliminates potentially skewed data in cases where there are overlapping experimentation audiences. It also ensures that the most critical experiments maintain integrity.

To prepare for your tests, see About A/B testing.

### Audience groups in the API When you set up A/B Tests using the [`/experiments` API](https://www.airship.com/docs/developer/rest-api/ua/operations/a-b-tests/), your `audience` is split across the variants in your message by `weight` properties. You can also set a `control` group. The `control` group is a decimal (float) between 0 and 1 representing the portion of your audience who will not get a message. The remainder of your audience (after the control group is subtracted) receives messages according to their `weight`. If you don't set `weight` properties, Airship splits your audience evenly across your variants. Airship adds the `weight` properties in your payload and divides the total by an individual weight to determine the proportion of the audience that receives each variant. For example, if you set weights of 10, 10, and 5 for your variants, Airship splits your audience proportionally into subsets of 40%, 40%, and 20%: | Variant | Weight | Audience percentage | | --- | --- | --- | | A | 10 | 40% | | B | 10 | 40% | | C | 5 | 20% | **Example experiment with control and weights** ```json { "name": "Experiment 1", "audience": "all", "control": 0.2, "device_types": "all", "variants": [ { "push": { "notification": { "alert": "You're in a cool group" } }, "weight": 20 }, { "push": { "notification": { "alert": "You're in the coolest group" } }, "weight": 40 } ] } ``` ## Create a message A/B test The following steps walk you through creating a legacy message A/B test in the dashboard. For the API, see [A/B Tests](https://www.airship.com/docs/developer/rest-api/ua/operations/a-b-tests/) in the API reference. To get started, access the legacy A/B Test composer: 1. Select **Create** in the sidebar. 1. Next to **Build from scratch**, select **View all**. 1. Select **A/B Test — Legacy**. Each configuration step is labeled in the center of the header: ![The Audience step in the A/B Test composer](https://www.airship.com/docs/images/composer-progress-a-b_hu_ab1cb088f0c84181.webp) *The Audience step in the A/B Test composer* After completing a step, select the next one to move on. Select the gear icon () to [change the test name](https://www.airship.com/docs/guides/messaging/manage/edit/#message-names) or [flag it as a test](https://www.airship.com/docs/guides/messaging/manage/flag-as-test/). 1. In the Audience step, enter a descriptive name for the test, then enable channels and select which users should receive the test. User groups: | Option | Description | Steps | | --- | --- | --- | | **All Users** | Your entire audience for the selected channels | n/a | | **Target Specific Users** | Audience members in a group you define | Use the same procedure as when building a [Segment](https://www.airship.com/docs/reference/glossary/#segment) | | **Test Users** | Members of a [Test Group](https://www.airship.com/docs/reference/glossary/#preview_test_groups) | Select a Test Group. | 1. Select the **Variants** step, then select the number of variants you want to create and set the percentage of your target audience to test. By default we send your test to 80% of your target audience, keeping a control group of 20%. You can change the number of variants later. ![The Variants step in the A/B Test composer](https://www.airship.com/docs/images/abtest-variants_hu_beea0f5c013337a7.webp) *The Variants step in the A/B Test composer* 1. Select the **Content** step, then enter a name for variant A and configure the message content per enabled channel. See [Content by channel](https://www.airship.com/docs/guides/messaging/messages/content/). For additional variants, select a lettered tab, choose whether to copy content from an existing variant or start with a blank message, and complete message configuration. Select the plus icon ( ) or remove icon ( ) to add or remove variants. You cannot remove the last remaining variant. 1. Select the **Delivery** step, then set up delivery timing:

Option	Description	Steps
Send Now	Send the message immediately after review.	n/a
Schedule¹	Send the message on a specific date and time in a specific time zone or in each user's time zone. For delivery by time zone, a push notification scheduled for 9 a.m. will arrive for people on the east coast at 9 a.m. Eastern Time, in the midwest an hour later at 9 a.m. Central Time, then on the west coast two hours after that, at 9 a.m. Pacific Time.	Enter a date in YYYY-MM-DD format and select the time, then select a time zone or check the box for Delivery By Time Zone.
Optimize²	Send the message on a specific date and at each user's [Optimal Send Time](https://www.airship.com/docs/reference/glossary/#optimal_send_time). iOS, Android, and Fire OS only. Airship recommends scheduling your message at least three days in advance due to the combination of time zones and optimal times. You can reduce the lead time if your audience is more localized, e.g., only in the United States or in a certain European region.	Enter a date in YYYY-MM-DD format.

Option

Description

Steps

Send Now

Send the message immediately after review.

n/a

Schedule¹

Send the message on a specific date and time in a specific time zone or in each user's time zone. For delivery by time zone, a push notification scheduled for 9 a.m. will arrive for people on the east coast at 9 a.m. Eastern Time, in the midwest an hour later at 9 a.m. Central Time, then on the west coast two hours after that, at 9 a.m. Pacific Time.

Enter a date in YYYY-MM-DD format and select the time, then select a time zone or check the box for Delivery By Time Zone.

Optimize²

Send the message on a specific date and at each user's [Optimal Send Time](https://www.airship.com/docs/reference/glossary/#optimal_send_time). iOS, Android, and Fire OS only.

Airship recommends scheduling your message at least three days in advance due to the combination of time zones and optimal times. You can reduce the lead time if your audience is more localized, e.g., only in the United States or in a certain European region.

Enter a date in YYYY-MM-DD format.

^{1. Messages are only delivered by time zone to channels that have a time zone set. App and Web channels have their time zone set automatically by the SDK. Email, SMS, and Open channels will only have a time zone if set through the Channel Registration API. To do so, enter a value for the "timezone" key in the request body. See user registration information for Email, SMS, and Open channels. The API equivalent of Delivery By Time Zone is Push to Local Time.}
^{2. When your audience includes users without an optimal send time tag, those users will be dropped from delivery and will not receive the message. Since optimal send time is determined from user behavior over time, new users might not have an optimal send time determined for the first week or two after channel registration.} After selecting timing, configure: | Section | Description | Steps | | --- | --- | --- | | **Purpose¹** | Set or verify the [Message Purpose](https://www.airship.com/docs/reference/glossary/#message_purpose). This option only appears if Message Purpose is enabled for the project. | Select **Commercial** or **Transactional**. | | **Options** | Various options are available depending on the message types, channels, and platforms selected for your test. | See [Message delivery options](https://www.airship.com/docs/guides/messaging/messages/delivery/delivery-options/). | | **External data feed options** | If your message includes [External Data Feeds](https://www.airship.com/docs/reference/glossary/#external_feed), you must determine how the message is handled if the feed fails. Additionally, the default value is displayed for each send time variable in the feed URL. You can enter new values to override the default value for this message only. | For **Failure behavior**, select **Abort sending the message** or **Send message without this data**. For any **Default value for <var>**, enter a new value. | | **Ban List** | If your project has a [Ban List](https://www.airship.com/docs/reference/glossary/#ban_list) enabled and its request URL includes send time variables, you can override their default values for this message only. This setting does not appear if [Bypass Ban List](https://www.airship.com/docs/guides/messaging/messages/delivery/delivery-options/#bypass-ban-list) is also enabled. | For any **Default value for \\<variable>**, enter a new value. | ^{1. When Message Purpose is enabled and email and at least one other channel are selected for a message, Purpose is disabled in the Delivery step. Instead, set the purpose in the email's [Sender Information](https://www.airship.com/docs/guides/messaging/messages/content/email/email/#creating-content): In Content step, select the Email tab, select Edit
for Sender Information, and enable Transactional or leave it disabled if the message contains commercial content only. The commercial/transactional designation set in the email Sender Information will apply to all channels selected for the message.} 1. Select the **Review** step, then review the device preview and message summary. You can select a variant in the Content section or above the preview. Select the arrows to page through the various previews. The channel and display type dynamically update in the dropdown menu above. You can also select a preview directly from the menu. If you want to make changes, select the associated step in the header, make your changes, then return to Review. You can send a test message for each variant to verify its appearance and behavior on each configured channel. The message is sent to your selected recipients immediately, and it appears as a test in [Messages Overview](https://www.airship.com/docs/reference/glossary/#messages_overview). First, select **Send Test** and a variant. Then, complete the following:

Under Test audience, enter at least one [Named User](https://www.airship.com/docs/reference/glossary/#named_user) or [Test Group](https://www.airship.com/docs/reference/glossary/#preview_test_groups) and select from the results. If your message includes email, you can also search for email addresses. If no matches appear for an address, you can select Create channel for <address>, and the channel will be registered for your project and opted in to transactional messaging.

Users in an active [Holdout Experiment](https://www.airship.com/docs/reference/glossary/#holdout_experiment) will not receive a test message. You can view a user’s current holdout group status and history when viewing their channel details in Contact Management.

(If your message contains [Handlebars](https://www.airship.com/docs/reference/glossary/#handlebars)) Under Personalization, select and configure a personalization data source:

Data source	Description	Steps
Test message recipient	The message will be personalized using information associated with each test audience member.	n/a
Preview Data tool	The message will be personalized using the data currently entered in the Preview Data tool. The same values will apply to all test message recipients. You can also manually edit the JSON.	(Optional) Edit the JSON data.

Select Send.

When your review is complete, select **Send Message** or **Schedule Message**. ## View test reports After sending an A/B test, discover which variant performed best. Use the test- and message-level reports to determine the quality of each variant and strategies for increasing engagement. See also [Implementing A/B tests, outcomes, and compliance](https://www.airship.com/docs/guides/experimentation/a-b-tests/about/#implementing-ab-tests-outcomes-and-compliance) in *About A/B testing*. Go to **Messages**, then **Messages Overview**, and then select the report icon ( ) for an A/B test. Report sections: | Section | Description | | --- | --- | | **Header** | Displays the test name and its send date, time, and time zone | | **Performance** | Contains statistical data for each variant per channel and the control group, if any, and a link to a [message report](https://www.airship.com/docs/guides/reports/message/) for each variant | | **Message Detail** | Contains the same information and preview options shown in [the Review step when creating the A/B test](#create-a-message-ab-test) and in the [Message Detail section of a message report](https://www.airship.com/docs/guides/reports/message/#message-detail) | To export test data, select **Download CSV**, then **Performance Data**. If your test included [Custom Events](https://www.airship.com/docs/reference/glossary/#custom_event), you will also have the option to download **Variant Event Data**, which is a report of event conversions and associated values, broken out by variant or control group. > **Note:** Engagement data is sent to Airship as soon as it becomes available. Data may be delayed due to connectivity issues with a user's carrier, wifi, power, etc. Wait at least 12 to 24 hours before acting on test data to allow for potential lags.