# Sequence Control Groups Control groups are tools for measuring campaign efficacy and managing controlled rollouts for Sequences. A control group is a percentage of an audience that is excluded from receiving messages. Audience members in the control group are randomly selected on entry to the Sequence, and they continue the [Sequence](https://www.airship.com/docs/reference/glossary/#sequence) just like active audience members, only without receiving any of its messages. Related events and conversions are recorded for both audiences, providing data you can use to evaluate Sequence performance. Using control groups in this way can help you: * Determine the attribution of conversions to, as well as the potentially negative impacts of, various marketing efforts. * Evaluate the impact your messaging has on driving conversion goals, KPIs, or uninstalls/opt-outs, to make informed decisions on how to best evolve your marketing strategy. * Preview and set the pace for a controlled rollout. When running a Sequence control group and a [Holdout Experiment](https://www.airship.com/docs/reference/glossary/#holdout_experiment) simultaneously, Airship prevents holdout group users from being included in the Sequence control group. This eliminates potentially skewed data in cases where there are overlapping experimentation audiences. It also ensures that the most critical experiments maintain integrity. > **Tip:** * You can run control groups and [A/B tests](https://www.airship.com/docs/guides/experimentation/a-b-tests/sequences/) for a Sequence concurrently. > * You can also create rollouts using [Feature Flags](https://www.airship.com/docs/guides/experimentation/feature-flags/) and [Scenes](https://www.airship.com/docs/guides/features/messaging/scenes/rollouts/). ## Creating a control group As a best practice, you should create a control group before starting a Sequence, but you can create a control group at any time. From the Sequence [Manage](https://www.airship.com/docs/reference/glossary/#sequence_manager) or [Performance](https://www.airship.com/docs/reference/glossary/#sequence_performance) screen: 1. Select **Experiments** in the leftside drawer. 1. Select **Create a control group** and enter the percentage of users to exclude from messaging. 1. Select **Save**. After saving, the Experiments drawer will update depending on Sequence configuration and status: * For Sequences **without a conversion event**, you will see **Control group allocation** and options to remove the control group and adjust the allocation. * For Sequences **with a conversion event**, you will see only **Control group allocation** until after you start the Sequence. After starting, you will instead see **Control group performance** options to remove the control group, adjust the allocation, and set a baseline. You will also see this data: | Data | Description | | --- | --- | | **Allocation** | The current control group percentage. | | **Sample size** | The number of users in the control group. Updated upon hard refresh and when allocation is changed. | | **Conversions** | The number of users who exited the Sequence by a conversion event. Updated upon hard refresh. | | **Conversion rate** | The number of conversions divided by the sample size. Updated upon hard refresh. | ## Comparing Performance reporting The Performance report shows audience behavior compared to the Sequence’s goal. While a control group is enabled, review the performance of the active group compared to the control group and determine if messaging is having the expected impact on the Sequence's conversion events. The report has the same layout as the [Sequence Manager](https://www.airship.com/docs/reference/glossary/#sequence_manager) and is available after you start the Sequence. For full documentation, see [Sequence Performance](https://www.airship.com/docs/guides/messaging/messages/sequences/performance/). 1. Go to **Messages**, then **Messages Overview**. 1. Select the report icon ( ) for a Sequence. **Performance data** contains statistical data for the Sequence and each message. * Select a time frame to update the viewable data. * Select ** Report** to open an individual [message report](https://www.airship.com/docs/guides/reports/message/). * Select *Active audience* / *Control group* to change the viewable data. ## Managing a controlled rollout Use a controlled rollout to increase the availability of a Sequence over time. This can be helpful for Sequences that are considered high-risk, such as promoting a new feature launch or product line. If you want to preview the performance before deployment, set the initial control group allocation to 100%, and [compare performance data](#comparing-performance-reporting) before reducing the group size. 1. [Create a Sequence](https://www.airship.com/docs/guides/messaging/messages/sequences/create/create/), but **do not start it**. 1. Go to the [Manage](https://www.airship.com/docs/reference/glossary/#sequence_manager) or [Performance](https://www.airship.com/docs/reference/glossary/#sequence_performance) screen. 1. Select **Experiments** in the leftside drawer. 1. Select **Create a control group**, and enter the percentage of users to exclude from messaging. 1. Select **Save**. 1. Select ** Start**. When you are ready to increase the availability of the Sequence: 1. Go to the [Manage](https://www.airship.com/docs/reference/glossary/#sequence_manager) or [Performance](https://www.airship.com/docs/reference/glossary/#sequence_performance) screen. 1. Select **Experiments** in the leftside drawer. 1. Select **Adjust allocation** and enter a lower percentage. 1. Select **Save**. Continue to reduce the control group size over time, and select **Remove control group** when you want to make the Sequence available to your entire audience. ## Measuring campaign lift *Lift* attempts to determine the efficacy of a Sequence by comparing the conversion rate of an active audience with that of a control group who received no messages. This can give you an idea of how your audience performs with and without your marketing efforts. Key terms: > Baseline > : The *baseline* is the benchmark conversion rate of a Sequence's control group. It represents the conversion rate you would expect to see without messaging, and it is used to calculate the lift rate. > Lift rate > : The *lift rate* is the percent increase or decrease in the active audience conversion rate against the baseline. > Conversion trend > : The *conversion trend* is the difference between the baseline and current conversion rates for a selected time frame. For example, a hotel chain may want to measure the impact of sending their audience a discount code for $150 off reservations for any stay of 2+ nights, using a Sequence conversion goal of hotel bookings. After creating the control group, they set a baseline when their desired sample size has been met. At the end of the 90-day campaign they find that the campaign lift rate is +86%: * *Baseline: 7%* — The booking rate they would expect to see without messaging. * *Conversion rate: 13%* — The conversion rate of the active audience. * *Conversion trend: +6%* — The difference between the baseline conversion rate and the campaign-end conversion rate. * *Lift rate: +86%* — For users who received the discount code, hotel bookings increased 86% over the baseline. As your campaign runs and at its end, determine the impact of your messaging by looking at the lift rate, conversion trend, and other factors specific to the campaign. Keep in mind that: 1. The baseline is intended to represent the behavior of users without marketing influence (the messages in your Sequence), however, they may be exposed to your marketing through other channels. 1. The lift rate is neutral data — even though a lift rate may be high, if the campaign cost to achieve your Sequence goal is also high, you may find the ROI is too low to continue the campaign. > **Note:** Airship does not provide historical reporting data for control groups. If you intend to compare the performance of a Sequence based on multiple control groups and baselines, you will need to record the data yourself. To get started, set up a Sequence and its control group: 1. [Create a Sequence](https://www.airship.com/docs/guides/messaging/messages/sequences/create/create/) that has a conversion event, but **do not start it**. 1. Go to the [Manage](https://www.airship.com/docs/reference/glossary/#sequence_manager) or [Performance](https://www.airship.com/docs/reference/glossary/#sequence_performance) screen. 1. Select **Experiments** in the leftside drawer. 1. Select **Create a control group** and enter the percentage of users to exclude from messaging. 1. Select **Save**. 1. Select ** Start**. Next you will establish a baseline for reporting. You decide when to set the baseline, by sample size or time — either wait till the number of users in the control group is representative of your Sequence's goal, or at least seven days or three times the Sequence length. When you set the baseline, the control group allocation is set to 0%, making the Sequence available to your entire audience, and Airship starts generating reporting data using the baseline. 1. Select **Experiments** in the leftside drawer. 1. Select **Set baseline** and confirm. After setting a baseline, **Sequence lift** is added to the Experiments drawer, which displays the lift rate, the baseline rate, and the date the baseline was set. You can view the lift rate and conversion trend in the [Journey Map](https://www.airship.com/docs/reference/glossary/#journey_map): 1. Go to **Journeys**. 1. Select the Sequence from the sidebar or in the map. The card in the map view displays the conversion trend next to the conversion rate. It appears in green for a positive trend and red for a negative trend. Select the trend to see the lift rate, baseline, and the date the baseline was set. Now you can let the campaign run to completion. As a best practice, do not edit a Sequence while it has a control group enabled. ## Creating a new control group When you need to change a Sequence's content and settings, you will likely also want to cancel your current control group and create a new one. For instance, you might update a Sequence for seasonality, such as changing from winter to spring campaigns. Creating a new control group follows the [same procedure as creating the first control group](#creating-a-control-group). However, the lift rate is always based on the latest set baseline. Even if you create a new control group, the baseline is not affected until you set a new baseline.