Message A/B tests

About A/B tests for messages

Create variants of message content by duplicating the initial variant or by starting from scratch. Each variant returns analytic data to help you determine the most effective way to engage your audience. Message A/B tests can include a control group, and you can adjust audience allocation across the message variants and control group.

Set up the test in two steps:

1. Create two or more message variants — Just like in the Message composer, for each variant, select channels, configure content for each channel, and set up delivery.

2. Allocate an audience — You can designate all users as eligible for the test or target specific users. Of that group, set the percentage that will participate in the test. Audience members are randomly selected.

   You can also include a control group, which is the portion of your audience that doesn’t receive messages. It’s disabled by default, and you can enable it when setting the test audience. When enabled, the control group appears alongside variants in the By Channel view of your test results.

   The overall audience percentage is automatically divided evenly between variants and the control group, but you can set your own values.

After creating variants and setting the audience, you can start the test and review its results.

To have Airship optimize your experiment in real time and maximize conversions, use an Intelligent Rollout instead.

When running a message experiment and a Holdout ExperimentMeasures the effects of excluding a group of audience members from all messages or messages with specific Campaign Categories. You can compare the performance of the two audience groups in reports for selected goal events. simultaneously, Airship prevents holdout group users from being included in the message experiment. This eliminates potentially skewed data in cases where there are overlapping experimentation audiences. It also ensures that the most critical experiments maintain integrity.

To prepare for your tests, see About A/B testing.

Supported channels and measuring engagement

These channels and message types are supported for message A/B tests:

• App — Push notifications, in-app messages, and Message Center
• Web
• Email
• SMS
• Open channel

Airship uses the following engagement signals to determine the top-performing variant:

• Push — Direct clicks on a push message
• Message Center — Message reads
• Email — Clicks on any link in the email, excluding unsubscribe links
• SMS — Clicks on a link in the message

Create a message A/B test

First, select the Create dropdown menu (), then A/B Test. Or you can start from your list of all message experiments by going to Experiments, then Message Experiments, selecting Add experiment, and then A/B Test.

Next, select the test name and change it to something descriptive, then select the check mark to save it.

To finish setting up your test, you must add message variants and determine the audience. You can configure them in any order.

Add message variants

You can add up to 26 variants to an A/B test:

1. Select Add variant. After completing a step, select the next step in the header to move on.

2. For Channels:

   First, select a Channel CoordinationA group of strategies for targeting users where they are most likely to receive your message. strategy:

   • Fan Out targets a Named User on all the channels they are opted in to, maximizing the chances they receive your message.
   • Last Active targets a Named User on the opted-in channel they used most recently.
   • Priority Channel targets a Named User on the first channel they are opted in to, in the priority order you set.

   Then, enable the channel types to include in your audience. For Mobile Apps, also select from the available platforms. For Priority Channel, also drag the channel types into priority order.

   Note

   For projects using the channel-level segmentation system, instead of Channel Coordination, enable the channels you want to send the message to.

   Use Channel conditions to filter which channels are included in the audience. A channel must meet the conditions to remain in the audience.

   For example, if your audience includes users with app, email, and SMS channels, and you set a channel condition requiring membership in an email Subscription List:

   • Only email channels that meet that condition would remain in the audience.
   • All app and SMS channels would be excluded.

   To set channel conditions, use the same process as when building a SegmentA reusable audience group you create by selecting unique or shared user data.. You can use the following data in your conditions:

   • AutogroupA default tag group. When devices are registered in Airship, they are automatically assigned a tag within the autogroup tag group; the tag is a numeric value of 1 to 100. Some uses for autogroup tags are creating a control group, and to randomly Segment your audience. Autogroup must be enabled for your account — contact Support or your account manager.
   • Channel IDAn Airship-specific unique identifier used to address a channel instance, e.g., a smartphone, web browser, email address.
   • Device PropertiesMetadata representing the default attributes and property tags of a device, such as language and time zone settings, OS and browser versions, and notification opt-in status. Device properties are used for audience segmentation. The data used for the tags and attributes is collected automatically from the Airship SDKs, and are updated daily.
   • EventsEvents represent activities in the Airship system, as well as user behavior in your app or website. You can also send external activities (e.g., POS transactions) into the Airship system as events. Events power Airship data products and can be used in audience segments, as Automation triggers, and for partner integrations.
   • Lifecycle ListAutomatically generated audience lists that capture app open, uninstall, notification, and dormancy information within the past 24 hours, one week, or 30 days.
   • Predicted to Churn statusAnalyzes your audience for users that exhibit behaviors indicating they are likely to become inactive, and tags the users as High, Medium, or Low Risk.
   • Subscription ListAn audience list of users who are opted in to messaging about a specific topic. Users can manage their opt-in status per list using a Preference Center.
   • TagMetadata that you can associate with channels or Named Users for audience segmentation. Generally, they are descriptive terms indicating user preferences or other categorizations, e.g., wine_enthusiast or weather_alerts_los_angeles. Tags are case-sensitive. in the device Tag GroupAn array of tags that you can associate with both channels and Named Users. — See Primary device tags.
   • Uploaded (Static) ListA reusable audience list that you create. Uploaded lists are static and updatable. In the API, they are referred to as Static Lists.

   Selected Lifecycle, Subscription, and Uploaded Lists must contain Channel IDs or Named Users as the identifier, not a mix of the two.

   Note

   Setting channel conditions is not supported for projects using the channel-level segmentation system.

   Under Localization, enable the option if you want to provide different content to app and web users depending on their language and country.

3. For Content, configure the message content per enabled channel. See the Content documentation per message type, Content options, and Localization.

4. For Delivery, configure the message delivery timing and options. See Message delivery.

5. In the Review step, review the device preview and message summary:

   • Use the arrows to page through the various previews. The channel and display type dynamically update in the dropdown menu above. You can also select a preview directly from the menu.
   • If you want to make changes, select the associated step in the header, make your changes, then return to Review.
   • Select Send Test to send a test message to verify its appearance and behavior on each configured channel. The message is sent to your selected recipients immediately, and it appears as a test in Messages OverviewA view of all your project’s messages, with options for editing their settings, content, status, and more.. Follow the same steps as in the Review step for the Message composer.

   When your review is complete, select Save Variant.

To add another variant from scratch, select Add variant. To duplicate an existing variant, select the more menu icon () at the end of a row and select Copy to variant.

Set the test audience

After creating an A/B test, select Audience and then set up your test audience:

1. Choose and configure users:

   Option	Description	Steps
   All Users	This option makes the test available to a percentage of your total audience.	n/a
   Target Specific Users	This option makes the test available to a percentage of users who meet specified conditions.	Select and configure one or more conditions. Use the same process as when building a SegmentA reusable audience group you create by selecting unique or shared user data..

2. (Optional) Under Audience allocation, limit the selected audience to your specified percentage.
3. (Optional) Enable Control group.
4. (Optional) To override the default variant distribution, enable Allow uneven allocations and then edit the percentage for each variant and the control group.
   Note

   If you later add more variants, also update your variant allocation settings.

5. Select Save.

Start an A/B test

Once you’ve created your message variants and set the audience for your test, select Start and confirm. Airship then sends each variant according to its delivery settings.

After the test starts, an Overview section shows the test status, audience summary, and the Probability of Being the Best Variant chart from the Performance view.

View results

After starting an A/B test, discover which variant performed best and use the test- and message-level reports to determine the quality of each variant and strategies for increasing engagement.

To access test results, go to Experiments, then Message Experiments, select the more menu icon () for a test in the list, then View results. You can also select the name of a test from the list and then go to Results.

See also Implementing A/B tests, outcomes, and compliance in About A/B testing.

Performance

The Performance view is the initial state of test results. Its first component is a summary explaining the winning variant or if the results were inconclusive.

Airship declares a winning variant only after all of the following are true:

• All variants in the test have been sent.
• At least 90 minutes have passed since the last variant send.
• The test reached at least 1,000 participants.
• The winning variant’s Probability to Be Best (PTBB) is 95% or higher.
• The winning variant’s expected loss is less than 5%.

Following the summary, a table lists this data for each variant:

Select a variant name to open its message report.

Below the table, the following charts visualize variant performance:

• Probability of Being the Best Variant — This chart shows the probability that each variant is the top performer in your experiment. Taller bars indicate a higher likelihood of success, and a variant typically needs to reach 95% or higher to be declared the winner. If the bars are roughly the same height, the variants are performing similarly.

• Conversion Lift Compared to Others — This chart shows the potential uplift of each variant compared to the top performer, helping you distinguish meaningful improvements from minor fluctuations.

  • 0% uplift line — This line represents no difference between variants. Curves to the right indicate improvement. Curves to the left indicate decline.
  • Distribution spread — A narrow, tall curve indicates high confidence, while a wider curve suggests more uncertainty.
  • Bulk mass — If the majority of a variant’s curve lies above zero, the variant is highly likely to outperform the others.

• Likelihood of Variant Success — This chart shows the estimated conversion rate for each variant as a curve.

  • Height — The higher the peak, the more likely that conversion rate represents the variant’s true performance.
  • Width — A narrow curve indicates high confidence, while a wider curve suggests more uncertainty.
  • Overlap — Significant overlap means the variants are performing similarly. Minimal overlap indicates a clearer performance gap between them.

By channel

Select By Channel to see how variant performance differed across each channel in the test. Select a variant name to open its message report.

Export results

Export options for experiment results vary by view:

• In the Performance view, select Download.
• In the By Channel view, select Download Results, then Performance Data. If your experiment included Custom EventsEvents that indicate that a user performed a predefined action, such as adding an item to a shopping cart, viewing a screen, or clicking an Unsubscribe button. Custom Events can trigger automation, including Sequences and Scenes. You can code them into your app or website, or send them to Airship from an external source using the Custom Event API. Custom Events contain properties that you can use to personalize messages., you will also have the Variant Event Data option, which is a report of event conversions and associated values, broken out by variant.

RTDS events

Messages used as variants in an A/B test include experiment information in Real-Time Data StreamingA service that delivers user-level events in real time to your backend or third-party systems using the Data Streaming API. events.

The Send event includes an experiments object with the test details, including experiment_id, type, and variant_id. The experiment_id also appears in the body object.

The Control event includes an experiment_id at the top level and also in the body object.

Managing A/B tests

Go to Experiments, then Message Experiments to view and manage your message A/B tests. You can filter the list by experiment type and archive status. Each experiment is listed by name with its status and the date it was last modified. Your last modified experiment is listed first, and you can search by experiment name.

You can perform the following actions from the list:

Test and variant statuses

In your list of all message experiments, A/B tests display the following statuses:

Within a test, the Variants list displays the following statuses:

Editing message variants and audience

You can edit variants and audience settings for any test with Draft status. After opening a test from the Message Experiments list, select the more menu icon () for a variant and select an option:

To modify the test audience, select Audience and adjust targeting or allocation settings. See Set the test audience for configuration details.

For tests with Started status, you can manage variants configured for recurring delivery from the more menu icon ():