Intelligent Rollouts

About Intelligent Rollouts

Intelligent Rollouts identify and distribute your best-performing message variant automatically using real-time engagement data. This helps maximize conversions while the campaign is active, eliminates the manual effort of A/B testing, and minimizes exposure to less effective variants. They are powered by reinforcement learning (multi-armed bandit ) to dynamically optimize for the winning variant.

They are great for time-sensitive campaigns:

• Optimize flash sales — Test offers during a 12-hour sale, such as “20% Off” against another promotion, and let Airship shift more of the remaining audience to the offer driving more clicks.
• Tune newsletter subject lines — Send a weekly newsletter over a 24-hour window, compare subject lines, and automatically deliver the better-performing version to more subscribers.
• Maximize holiday campaign engagement — Compare messages such as “Gifts for Her” and “Gifts for Him,” then increase delivery to the variant performing best with your general audience.

When running a message experiment and a Holdout ExperimentMeasures the effects of excluding a group of audience members from all messages or messages with specific campaign categories. You can compare the performance of the two audience groups in reports for selected goal events. simultaneously, Airship prevents holdout group users from being included in the message experiment. This eliminates potentially skewed data in cases where there are overlapping experimentation audiences. It also ensures that the most critical experiments maintain integrity.

Supported channels and measuring engagement

These channels and message types are supported for Intelligent Rollouts:

• App — Push notifications, in-app messages, and Message Center
• Web
• Email
• SMS
• Open channel

Airship uses the following engagement signals to determine the top-performing variant:

• Push — Direct clicks on a push message
• Message Center — Message reads
• Email — Clicks on any link in the email, excluding unsubscribe links
• SMS — Clicks on a link in the message

Because Airship uses email and SMS link clicks to measure engagement, links are required for those message types to drive optimization.

Create an Intelligent Rollout

First, select the Create dropdown menu (), then Intelligent Rollout. Or you can start from your list of all message experiments by going to Experiments, then Message Experiments, selecting Add experiment, and then selecting the same option.

Next, select the experiment name and change it to something descriptive, then select the check mark to save it.

To finish setup, add message variants, determine the audience, and configure the schedule. You can configure them in any order.

Add message variants

You can add up to 26 variants:

1. Select Add variant. After completing a step, select the next step in the header to move on.

2. For Channels:

   First, select a Channel CoordinationA group of strategies for targeting users where they are most likely to receive your message. strategy:

   • Fan Out targets a named user on all the channels they are opted in to, maximizing the chances they receive your message.
   • Last Active targets a named user on the opted-in channel they used most recently.
   • Priority Channel targets a named user on the first channel they are opted in to, in the priority order you set.

   Then, enable the channel types to include in your audience. For Mobile Apps, also select from the available platforms. For Priority Channel, also drag the channel types into priority order.

   Note

   For projects using the channel-level segmentation system, instead of Channel Coordination, enable the channels you want to send the message to.

   Use Channel conditions to filter which channels are included in the audience. A channel must meet the conditions to remain in the audience.

   For example, if your audience includes users with app, email, and SMS channels, and you set a channel condition requiring membership in an email subscription list:

   • Only email channels that meet that condition would remain in the audience.
   • All app and SMS channels would be excluded.

   To set channel conditions, use the same process as when building a segmentA reusable audience group you create by selecting unique or shared user data.. You can use the following data in your conditions:

   • AutogroupA default tag group. When devices are registered in Airship, they are automatically assigned a tag within the autogroup tag group. The tag is a numeric value of 1 to 100. Some uses for autogroup tags are creating a control group, and to randomly segment your audience. Autogroup must be enabled for your account.
   • ChannelA device or address registered with Airship to receive messages, such as a mobile app, web browser, email address, or SMS number. It stores opt-in status, device-specific information, and metadata used for targeting. Each channel has a unique channel ID. ID
   • Device propertiesInformation about a channel, such as language and time zone settings, OS and browser versions, and notification opt-in status. Values come from the user’s device when an SDK is present, or are set via the API for channels without one. Each property is automatically made available as an attribute, a tag, or both for audience segmentation.
   • EventsA record of an action in your app, on your website, in the Airship system, or in an external system. Examples are a message send, an app open, or a purchase transaction.
   • Lifecycle listsAn audience list Airship automatically populates based on user activity in your app, such as app opens, uninstalls, notifications received, and dormancy.
   • Predicted to Churn statusAnalyzes your audience for users that exhibit behaviors indicating they are likely to become inactive, and tags the users as High, Medium, or Low Risk.
   • Subscription listsAn audience list users opt in to for a specific messaging topic. Users can manage their opt-in status using a Preference Center.
   • TagsMetadata that you can associate with a channel or named user for audience segmentation. Generally, a tag is a descriptive term indicating a user preference or other categorization, such as wine_enthusiast or weather_alerts_los_angeles. Tags are case-sensitive. in the device tag groupAn array of tags that you can associate with both channels and named users. — See Primary device tags.
   • Uploaded listsAn audience list you populate by uploading a CSV of channel or named user IDs. In the API, uploaded lists are called static lists.

   Selected lifecycle, subscription, and uploaded lists must contain channel IDs or named users as the identifier, not a mix of the two.

   Note

   Setting channel conditions is not supported for projects using the channel-level segmentation system.

   Under Localization, enable the option if you want to provide different content to app and web users depending on their language and country.

3. For Content, configure the message content per enabled channel. See the Content documentation per message type, Content options, and Localization.

4. For Delivery, configure the options. See Message delivery.

5. In the Review step, review the device preview and message summary:

   • Use the arrows to page through the various previews. The channel and display type dynamically update in the dropdown menu above. You can also select a preview directly from the menu.
   • If you want to make changes, select the associated step in the header, make your changes, then return to Review.
   • Select Send Test to send a test message to verify its appearance and behavior on each configured channel. The message is sent to your selected recipients immediately, and it appears as a test in Messages Overview. Follow the same steps as in the Review step for the Message composer.

   When your review is complete, select Save Variant.

To add another variant from scratch, select Add variant. To duplicate an existing variant, select the more menu icon () at the end of a row and select Copy to variant.

Set the audience

After creating an experiment, select Audience and then set it up:

1. Choose and configure users:

   Option	Description	Steps
   All Users	This option makes the experiment available to a percentage of your total audience.	n/a
   Target Specific Users	This option makes the experiment available to a percentage of users who meet specified conditions.	Select and configure one or more conditions. Use the same process as when building a segmentA reusable audience group you create by selecting unique or shared user data..

2. (Optional) Under Total audience allocation, limit the selected audience to your specified percentage.
3. Select Save.

Set the schedule

Select Schedule to configure the send window and timing:

1. Set a send window between 6 and 24 hours. Longer windows give Airship more time to learn and optimize delivery. Choose a shorter window when your message is time-sensitive.
2. Choose whether to start immediately or at a specific date and time.
3. Select Save.

Start the experiment

Once you’ve completed the setup, select Start and confirm. Airship then distributes variants automatically during the send window according to live engagement.

Overview

After you start an Intelligent Rollout, the Overview section appears so you can monitor its progress.

The Overview shows the rollout’s schedule and audience, the same summary, Variant Allocation Over Time chart, and conversion metrics as in Results.

View results

After starting the rollout, see how it performed. Use experiment- and message-level reports to evaluate engagement, variant distribution during the send window, and strategies for improving future campaigns.

Performance

To access the full results, go to Experiments, then Message Experiments, select the more menu icon () for an experiment in the list, then View results. You can also select the name of an experiment from the list and then go to Results.

The Performance view is the initial state of the results. Its first component is a summary explaining what happened, including whether Airship identified a winner or the data was inconclusive. For completed rollouts, final results reflect the total optimized distribution across all variants.

Airship declares a winning variant only after all of the following are true:

• All variants in the rollout have been sent.
• The rollout reached at least 1,000 participants in total across all variants.
• The winning variant’s Probability to Be Best (PTBB) is 95% or higher.
• The winning variant’s expected loss is less than 5%.

Two estimates compare the rollout to a standard even-split A/B test:

• Conversion lift — This chart estimates the lift achieved relative to a standard even split, shown as a central estimate and range. A narrower range indicates higher confidence in the estimate.

  • Central estimate — The bold percentage is the most likely conversion lift.
  • Estimated range — The values below the percentage show the estimated range of the lift. When the entire range stays to the right of 0%, you can be confident the lift is positive. Your confidence level corresponds to how much of the range sits to the right of 0%.
  • Scale — The horizontal scale plots the estimate relative to 0%. A position to the right of 0% means the rollout outperformed a standard even split, a position to the left means it underperformed, and a position at 0% means parity.

• Conversions gained — This metric shows the estimated number of additional conversions produced compared with a standard even-split A/B test over the same period. A negative value means a standard even split would have produced more.

See Baseline comparison for how Airship calculates these estimates.

A table lists the following data for each variant:

Select a variant name to open its message report.

Below the table, the following charts visualize variant performance:

• Variant Allocation Over Time — This chart shows the total number of contactsAny user in your project. Contacts are identified as either an anonymous contact or a named user. Airship can set targeting data on these identifiers, which are also used to map devices and channels to a specific user. reached throughout the experiment.
  • Stacked layers — Each color represents a variant. The height of a layer shows how many participants have received that variant at that point in the experiment.
  • Optimization window — This is the period when Airship distributes sends and actively learns. In the early stages, traffic is distributed more evenly to gather data. As engagement data comes in, Airship identifies a leader and begins to “tilt” delivery to favor the variant with higher engagement.
  • Shifting proportions — A layer that becomes noticeably thicker over time indicates that Airship has recognized it as a top performer and is automatically maximizing its reach to improve campaign results.
  • Final result — The far-right side of the chart reflects the total optimized distribution across the entire audience.

• Probability of Being the Best Variant — This chart shows the probability that each variant is the top performer in your experiment. Taller bars indicate a higher likelihood of success, and a variant typically needs to reach 95% or higher to be declared the winner. If the bars are roughly the same height, the variants are performing similarly.

• Conversion Lift Compared to Others — This chart shows the potential uplift of each variant compared to the top performer, helping you distinguish meaningful improvements from minor fluctuations.

  • 0% uplift line — This line represents no difference between variants. Curves to the right indicate improvement. Curves to the left indicate decline.
  • Distribution spread — A narrow, tall curve indicates high confidence, while a wider curve suggests more uncertainty.
  • Bulk mass — If the majority of a variant’s curve lies above zero, the variant is highly likely to outperform the others.

• Likelihood of Variant Success — This chart shows the estimated conversion rate for each variant as a curve.

  • Height — The higher the peak, the more likely that conversion rate represents the variant’s true performance.
  • Width — A narrow curve indicates high confidence, while a wider curve suggests more uncertainty.
  • Overlap — Significant overlap means the variants are performing similarly. Minimal overlap indicates a clearer performance gap between them.

By channel

Select By Channel to see how variant performance differed across each channel in the experiment. Select a variant name to open its message report.

Export results

Export options for experiment results vary by view:

• In the Performance view, select Download.
• In the By Channel view, select Download Results, then Performance Data. If your experiment included custom eventsAn event you define to track an action in your app, on your website, or in an external system. Examples are a sign-up, a video watch, or a wish list addition., you will also have the Variant Event Data option, which is a report of event conversions and associated values, broken out by variant.

Statistical methods

Airship uses two methods to analyze Intelligent Rollout results: Bayesian inference to rank variants and a simulated baseline to estimate the lift the rollout produced over a standard even-split A/B test.

Variant analysis

Airship analyzes Intelligent Rollout results using Bayesian statistics , measuring confidence in each variant’s success while accounting for uncertainty in the data. Rather than relying on a fixed confidence threshold only after the experiment ends, Bayesian methods allow for continuously updating the understanding of variant performance as engagement data comes in.

Airship estimates probability distributions for each variant’s performance. These distributions help calculate how likely each variant is to be the best. A Beta(1,1) prior  is used to create the distributions, starting with a neutral assumption and letting the data drive the results.

Instead of only comparing variants to a single control, Airship evaluates each variant against all other variants. This gives a more complete picture of which variant performs best. For Intelligent Rollouts, these methods also inform how Airship shifts traffic to stronger-performing variants while the experiment is active.

Benefits of using Bayesian methods:

• Transparent decision-making — You can see whether a variant is performing better than others and the confidence in that result.
• More than just statistical significance — Instead of a pass/fail outcome, Bayesian methods give you probability-based confidence in the results.
• Flexibility — You can decide how much certainty you need before applying what you learn from a rollout to future campaigns.

Baseline comparison

The Conversion lift and Conversions gained metrics in the Overview compare your Intelligent Rollout against a counterfactual simulated baseline: an estimate of what you would have achieved had you run a standard, even-split A/B test over the same period. The baseline is not a second experiment running in parallel. It is a simulation derived entirely from engagement data collected during your actual rollout.

This “what-if” comparison answers a single question: did the rollout outperform the alternative of distributing traffic equally across all variants from start to finish?

As the rollout runs, Airship shifts traffic toward better-performing variants. The baseline models what your overall conversion rate would have looked like if every variant had received equal traffic throughout the same window. Because the baseline is a simulation rather than a controlled test, the value shown is the expected, or most likely, estimate. The true result of a hypothetical even-split test could fall slightly above or below it.

When a rollout concludes with fewer than 1,000 total participants across all variants, the sample size is too small to produce a reliable baseline comparison, and Conversion lift and Conversions gained are not shown.

Real-Time Data Streaming events

Messages used as variants include experiment information in Real-Time Data StreamingA service that delivers user-level events in real time to your backend or third-party systems using the Data Streaming API. events.

The Send event includes an experiments object with the experiment details, including experiment_id, type, and variant_id. The experiment_id also appears in the body object.

Managing Intelligent Rollouts

Go to Experiments, then Message Experiments to view and manage your message A/B tests. You can filter the list by experiment type and archive status. Each experiment is listed by name with its status and the date it was last modified. Your last modified experiment is listed first, and you can search by experiment name.

You can perform the following actions from the list:

Editing message variants, audience, and schedule

You can edit variants, audience settings, and schedule settings for any experiment that has not yet been started. After opening it from the Message Experiments list, select the more menu icon () for a variant and select an option:

To modify the audience, select Audience and adjust targeting or allocation settings. See Set the audience for configuration details.

To modify the schedule, select Schedule and adjust the send window or timing.