# Intelligent Rollouts

Maximize conversions by automatically optimizing message campaign performance in real time. {{< badge "axp" >}}

## About Intelligent Rollouts

Intelligent Rollouts identify and distribute your best-performing message variant automatically using real-time engagement data. This helps maximize conversions while the campaign is active, eliminates the manual effort of A/B testing, and minimizes exposure to less effective variants. They are powered by reinforcement learning ([multi-armed bandit](https://en.wikipedia.org/wiki/Multi-armed_bandit)) to dynamically optimize for the winning variant.

They are great for time-sensitive campaigns:

* **Optimize flash sales** — Test offers during a 12-hour sale, such as "20% Off" against another promotion, and let Airship shift more of the remaining audience to the offer driving more clicks.
* **Tune newsletter subject lines** — Send a weekly newsletter over a 24-hour window, compare subject lines, and automatically deliver the better-performing version to more subscribers.
* **Maximize holiday campaign engagement** — Compare messages such as "Gifts for Her" and "Gifts for Him," then increase delivery to the variant performing best with your general audience.

<p>When running a message experiment and a [Holdout Experiment](https://www.airship.com/docs/reference/glossary/#holdout_experiment) simultaneously, Airship prevents holdout group users from being included in the message experiment. This eliminates potentially skewed data in cases where there are overlapping experimentation audiences. It also ensures that the most critical experiments maintain integrity.</p>

### Supported channels and measuring engagement

These channels and message types are supported for Intelligent Rollouts:

* App — Push notifications, in-app messages, and Message Center
* Web
* Email
* SMS
* Open channel

Airship uses the following engagement signals to determine the top-performing variant:

* **Push** — Direct clicks on the push message
* **Email** — Clicks on any link in the email, excluding unsubscribe links
* **SMS** — Clicks on the link in the message — Links are required for optimization.
* **Message Center** — Message reads

### Workflow

Set up an Intelligent Rollout in three steps:

1. **Create two or more message variants** — Just like in the [Message composer](https://www.airship.com/docs/guides/messaging/messages/create/), for each variant, select channels, configure content for each channel, and set up delivery.

1. **Allocate an audience** — You can designate all users as eligible for the experiment or target specific users. To limit your audience, set the percentage that can participate.

1. **Schedule timing** — Set a send window between 6 and 24 hours, then choose whether to start immediately or at a specific date and time. The window gives Airship time to optimize delivery while the campaign is active.

After setup, you can start the experiment and review its results.

## Create an Intelligent Rollout

First, select the **Create** dropdown menu (
), then **Intelligent Rollout**. Or you can start from your list of all message experiments by going to **Experiments**, then **Message Experiments**, selecting **Add experiment**, and then selecting the same option.

Next, select the experiment name and change it to something descriptive, then select the check mark to save it.

To finish setup, add message variants, determine the audience, and configure the schedule. You can configure them in any order.

### Add message variants

You can add up to 26 variants:

1. Select **Add variant**. After completing a step, select the next step in the header to move on.

1. For **Channels**:

   <p>First, select a [Channel Coordination](https://www.airship.com/docs/reference/glossary/#channel_coordination) strategy:</p>
   <ul>
   <li><strong>Fan Out</strong> targets a Named User on all the channels they are opted in to, maximizing the chances they receive your message.</li>
   <li><strong>Last Active</strong> targets a Named User on the opted-in channel they used most recently.</li>
   <li><strong>Priority Channel</strong> targets a Named User on the first channel they are opted in to, in the priority order you set.</li>
   </ul>
   <p>Then, enable the channel types to include in your audience. For Mobile Apps, also select from the available platforms. For Priority Channel, also drag the channel types into priority order.</p>

   > **Note:** For projects using the [channel-level segmentation system](https://www.airship.com/docs/guides/audience/segmentation/segmentation/#channel-level-segmentation), instead of Channel Coordination, enable the channels you want to send the message to.


   <p>Use <strong>Channel conditions</strong> to filter which channels are included in the audience. A channel must meet the conditions to remain in the audience.</p>
   <p>For example, if your audience includes users with app, email, and SMS channels, and you set a channel condition requiring membership in an email Subscription List:</p>
   <ul>
   <li>Only email channels that meet that condition would remain in the audience.</li>
   <li>All app and SMS channels would be excluded.</li>
   </ul>
   <p>To set channel conditions, use the same process as when building a [Segment](https://www.airship.com/docs/reference/glossary/#segment). You can use the following data in your conditions:</p>
   <ul>
   <li>[Autogroup](https://www.airship.com/docs/reference/glossary/#autogroup)</li>
   <li>[Channel ID](https://www.airship.com/docs/reference/glossary/#channel_id)</li>
   <li>[Device Properties](https://www.airship.com/docs/reference/glossary/#device_properties)</li>
   <li>[Events](https://www.airship.com/docs/reference/glossary/#events)</li>
   <li>[Lifecycle List](https://www.airship.com/docs/reference/glossary/#lifecycle_list)</li>
   <li>[Predicted to Churn status](https://www.airship.com/docs/reference/glossary/#predicted_to_churn)</li>
   <li>[Subscription List](https://www.airship.com/docs/reference/glossary/#subscription_list)</li>
   <li>[Tag](https://www.airship.com/docs/reference/glossary/#tag) in the <code>device</code> [Tag Group](https://www.airship.com/docs/reference/glossary/#tag_group) — See <a href="https://www.airship.com/docs/guides/audience/tags/#device-tags">Primary device tags</a>.</li>
   <li>[Uploaded (Static) List](https://www.airship.com/docs/reference/glossary/#uploaded_list)</li>
   </ul>
   <p>Selected Lifecycle, Subscription, and Uploaded Lists must contain Channel IDs or Named Users as the identifier, not a mix of the two.</p>

   > **Note:** Setting channel conditions is not supported for projects using the [channel-level segmentation system](https://www.airship.com/docs/guides/audience/segmentation/segmentation/#channel-level-segmentation).


   Under **Localization**, enable the option if you want to provide different content to app and web users depending on their language and country.

1. For **Content**, configure the message content per enabled channel. See the [Content documentation](https://www.airship.com/docs/guides/messaging/messages/content/) per message type, [Content options](https://www.airship.com/docs/guides/messaging/in-app-experiences/configuration/optional-features/), and [Localization](https://www.airship.com/docs/guides/messaging/messages/localization/).

1. For **Delivery**, configure the options. See [Message delivery](https://www.airship.com/docs/guides/messaging/messages/delivery/delivery/).

1. In the **Review** step, review the device preview and message summary:

   * Use the arrows to page through the various previews. The channel and display type dynamically update in the dropdown menu above. You can also select a preview directly from the menu.
   * If you want to make changes, select the associated step in the header, make your changes, then return to Review.
   * Select **Send Test** to send a test message to verify its appearance and behavior on each configured channel. The message is sent to your selected recipients immediately, and it appears as a test in [Messages Overview](https://www.airship.com/docs/reference/glossary/#messages_overview). Follow the same steps as in the [Review step for the Message composer](https://www.airship.com/docs/guides/messaging/messages/create/#message-review).

   When your review is complete, select **Save Variant**.

To add another variant from scratch, select **Add variant**. To duplicate an existing variant, select the more menu icon () at the end of a row and select **Copy to variant**.

### Set the audience

After creating an experiment, select **Audience** and then set it up:

1. Choose and configure users:

   | Option | Description | Steps |
   | --- | --- | --- |
   | **All Users** | This option makes the experiment available to a percentage of your total audience. | n/a |
   | **Target Specific Users** | This option makes the experiment available to a percentage of users who meet specified conditions. | Select and configure one or more conditions. Use the same process as when building a [Segment](https://www.airship.com/docs/reference/glossary/#segment). |
1. (Optional) Under **Total audience allocation**, limit the selected audience to your specified percentage.
1. Select **Save**.

### Set the schedule

Select **Schedule** to configure the send window and timing:

1. Set a send window between 6 and 24 hours. Longer windows give Airship more time to learn and optimize delivery. Choose a shorter window when your message is time-sensitive.
1. Choose whether to start immediately or at a specific date and time.
1. Select **Save**.

### Start the experiment

Once you've completed the setup, select **Start** and confirm. Airship then distributes variants automatically during the send window according to live engagement.

## View results

After starting the experiment, see how it performed. Use experiment- and message-level reports to evaluate engagement, variant distribution during the experiment window, and strategies for improving future campaigns.

To access results, go to **Experiments**, then **Message Experiments**, select the more menu icon () for an experiment in the list, then **View results**. You can also select its name from the list and then go to **Results**.

* A summary describes what happened during the experiment.
* A **Performance** section for each channel contains statistical data for each variant per channel, including variant distribution during the experiment window, conversions, and Probability to Be Best (PTBB), which indicates Airship's confidence in the current top-performing variant. Select a variant name to open its [message report](https://www.airship.com/docs/guides/reports/message/).

<p>To export data:</p>
<ul>
<li>In the Performance view, select <strong>Download</strong>.</li>
<li>In the By Channel view, select <strong>Download Results</strong>, then <strong>Performance Data</strong>. If your experiment included [Custom Events](https://www.airship.com/docs/reference/glossary/#custom_event), you will also have the <strong>Variant Event Data</strong> option, which is a report of event conversions and associated values, broken out by variant.</li>
</ul>

> **Note:** Engagement data is sent to Airship as soon as it becomes available. Data may be delayed due to connectivity issues with a user's carrier, Wi-Fi, power, etc. Wait at least 12 to 24 hours before acting on the data to allow for potential lags.

## Real-Time Data Streaming events

Messages used as variants include experiment information in [Real-Time Data Streaming](https://www.airship.com/docs/reference/glossary/#rtds) events.

The [Send event](https://www.airship.com/docs/developer/rest-api/connect/schemas/events/#send) includes an `experiments` object with the experiment details, including `experiment_id`, `type`, and `variant_id`. The `experiment_id` also appears in the `body` object.

## Managing Intelligent Rollouts

<p>Go to <strong>Experiments</strong>, then <strong>Message Experiments</strong> to view and manage your message A/B tests. You can filter the list by experiment type and archive status. Each experiment is listed by name with its status and the date it was last modified. Your last modified experiment is listed first, and you can search by experiment name.</p>

You can perform the following actions from the list:

| Option | Description | Steps |
| --- | --- | --- |
| **View** | Open the experiment to access its message variants, audience configuration, schedule, and results. | Select its name. |
| **Duplicate** | Make a draft copy with its message variants, audience configuration, and schedule. | Select its more menu icon () and then **Duplicate**. |
| **View results** | Open the performance reports. | Select its more menu icon () and then **View results**. See [View results](#view-results). |

### Editing message variants, audience, and schedule

You can edit variants, audience settings, and schedule settings for any experiment that has not yet been started. After opening it from the Message Experiments list, select the more menu icon () for a variant and select an option:

| Option | Description |
| --- | --- |
| **Edit** | Modify the variant's channels, content, or delivery settings. |
| **Duplicate** | Create a copy of the variant as a starting point for a new variant. |
| **Delete** | Remove the variant from the experiment. |

To modify the audience, select **Audience** and adjust targeting or allocation settings. See [Set the audience](#set-the-audience) for configuration details.

To modify the schedule, select **Schedule** and adjust the send window or timing.