# Holdout Experiments {{< glossary_definition "holdout_experiment" >}} ## About Holdout Experiments When creating a Holdout Experiment, you first define its purpose, hypothesis, and a definition of success. This step serves as a guideline for designing an effective experiment, since your answers can influence what messages you choose include or exclude, what to measure, and the duration. The information also serves as a reference when evaluating reports. Holdout Experiments can be open-ended or time-bound, starting immediately or at a scheduled time and date. Only one experiment can be active at any time, and you cannot create or schedule an experiment while another is active. When you create your first experiment, [Message Purpose](https://www.airship.com/docs/reference/glossary/#message_purpose) is automatically enabled for your project, and you must then select a purpose when creating any message in the dashboard. ### Holdout group An experiment's holdout group is the percentage of your total audience that is excluded from messaging. The remaining audience is the treatment group. * **Selection** — Audience members in a holdout group are randomly selected. * **Application** — Holdout groups are applied at the user level. No messages will send from your project to a [Channel](https://www.airship.com/docs/reference/glossary/#channel_dev) associated with a [Named User](https://www.airship.com/docs/reference/glossary/#named_user) in an active holdout group until your experiment ends. Depending on when your experiment starts, Holdout Experiments might be applied partially to scheduled messages. Airship prevents users in holdout groups from being included in [A/B tests](https://www.airship.com/docs/guides/experimentation/a-b-tests/) or [Sequence control groups](https://www.airship.com/docs/guides/experimentation/control-groups/). This eliminates potentially skewed data in cases where there are overlapping experimentation audiences. It also ensures that the most critical experiments maintain integrity, even when other experiments run simultaneously. You can view a user's current holdout group status and history when [viewing their channel details in Contact Management](https://www.airship.com/docs/guides/audience/contact-management/#viewing-channel-details). ### Message exclusion and allowances You can exclude all messages from holdout group members or only messages with specific [Campaign Categories](https://www.airship.com/docs/reference/glossary/#campaign_categories). For example, retailers could exclude `purchase_journey` campaigns to learn how their onboarding, abandoned cart, product rating requests, and other purchase-related messages impact conversion rates. If your experiment excludes sending all messages, you can set exceptions that allow sending: * **Transactional messages** — When sending messages from the API, you can bypass this allowance. See IMPORTANT note in step 5 in [Creating Holdout Experiments](#creating-holdout-experiments). * **Messages with specific Campaign Categories** — This flexibility helps ensure your business-critical or other required messages still reach your intended audience. ### Goals and reports Goals are the events you want to measure in your experiment. You can select from project-level [Goals](https://www.airship.com/docs/reference/glossary/#goals) or create new ones for that experiment only. Maximum 10 goals per experiment. You can create Goals based on [Custom or Predefined Events](https://www.airship.com/docs/guides/audience/events/events/#event-types) or for a number of Default Events. For the list of Default Events, see [Goals](https://www.airship.com/docs/guides/reports/goals/). As an experiment runs, reports for each Goal show the performance of the holdout and treatment groups. Holdout Experiments generate the same reports as project-level Goals. After the experiment ends, or after a period of time relevant to the purpose of the experiment ends, evaluate the reports to determine the impact your messaging has on driving conversion goals or KPIs. If there is no significant difference between holdout and treatment group performance, you may want to consider your campaigns and experiments for areas of improvement. Even with significant differences, this data can help you make informed decisions on how to best evolve your marketing strategy. ### Data normalization Data in Holdout Experiment reports is normalized to make it easier to compare the effect of your campaigns on your Goals without having to compare vastly different audience group sizes. For example, instead of comparing the actual numbers of 10% control and 90% treatment groups, we down-sample the larger group to compare an equal, random amount of users in each group. If there were 1,000 total users in an audience, 100 being in the control and 900 in the treatment, we would compare the 100 in the control with a random 100 users in the treatment group. We would then look at the users in those groups with at least one Goal event and show those in the report. ### Workflow The following is the general workflow for Holdout Experiments. 1. [Handle Goals prerequisites](#adding-events-and-creating-goals) — If the events or project-level Goals you want to use for your experiment don't already exist in your project, you'll need to add them. 1. [Create the experiment](#creating-holdout-experiments) — Define the experiment and set the holdout group percentage and options, Goals, and duration. 1. [View reports and evaluate](#viewing-reports) ## Adding events and creating Goals

You must add Custom and Predefined Events to your project before you can select them for Goals. You do not need to add Default Events to your project before selecting them for Goals.

If you want to use project-level Goals in an experiment, you must first create them in your project settings. See [Goals](https://www.airship.com/docs/guides/reports/goals/). Otherwise, you can create Goals as you create experiments. ## Creating Holdout Experiments 1. Go to **Experiments**, then **Holdout Experiments**. 1. Select **Create Holdout Experiment**. After completing each step, select **Next** to move on. 1. Define the experiment. All fields are required: Name, Purpose, Hypothesis, and Definition of success. 1. Search for and select Goals, or create one. You can add up to 10 Goals.

To create a Goal, enter a Goal name in the search field, then select Create Goal and configure fields:

Field Description Steps
Goal name Used for identification within the experiment Enter text.
Description Additional information about the Goal Enter text.
Event The event you want to measure in the experiment Search for and select an event. If the event does not have a category assigned, select from the list or select Custom category and enter a category name.
After configuring fields, select **Create Goal**. 1. Set up the holdout group and message options:
Setting Description Steps
Holdout group allocation The percentage of your audience to exclude from messaging. Set a percentage.
Messages to withhold Determines which messages to withhold from members of the holdout group. All messages withholds all messages sent from your project. Specific campaigns withholds messages with specific Campaign Categories.

You cannot select Specific campaigns if allowances for Campaign Categories and/or Transactional messages are selected.

 Select All messages or Specific campaigns. For Specific campaigns, enter a Campaign Category and then select its name below the entry field. Repeat for additional categories.
Allowances: Campaign Categories Allows sending messages with specific Campaign Categories to members of the holdout group. Cannot be selected if Withhold by Campaign Category is selected. Enter a category and then select its name below the entry field. Repeat for additional categories.
Allowances: Transactional messages Allows sending transactional messages to members of the holdout group. Cannot be selected if Withhold by Campaign Category is selected. For more information, see Commercial vs. Transactional Email. The information also applies to other message types. No configuration is required.
> **Important:** If your experiment does not allow sending transactional messages to the holdout group, you can use the [`bypass_holdout_groups` boolean](https://www.airship.com/docs/developer/rest-api/ua/schemas/push/#pushoptions) to send them anyway. > > When set to `true`, the message, whether commercial or transactional, will be sent to holdout group members who are part of the message audience. This option is available for messages sent using the API only. 1. Set the duration. For specific times and dates, also specify the time zone. The UTC conversion displays below the settings and updates as you make changes. 1. Select **Save**. ## Viewing reports Go to **Experiments**, then **Holdout Experiments**. Information about the most recent experiment appears next to the report for the first Goal added when setting up the experiment. Ended experiments are listed under **Past Holdout Experiments** with their start and end dates and times. Select a date a column header to sort. You can search for experiments by name. Select the report icon ( ) for an experiment. ![Holdout Experiment reporting](https://www.airship.com/docs/images/holdout-exp_hu_a09e4a84d2462b37.webp) *Holdout Experiment reporting* ### Performance The **Performance** section contains the following reports per Goal:
Report name Description
Goal The number of times the event occurred per day and the 7-day average.
Channels per goal The number of [Channels](https://www.airship.com/docs/reference/glossary/#channel_engage) that performed the event at least one time. You can filter by “greater than or equal to” and “is between” and enter values.
Goal frequency per channel The frequency of event occurrence per [Channel](https://www.airship.com/docs/reference/glossary/#channel_engage). Data points displayed: 50th (median), 75th, and 99th percentiles.
Goals per platform The percentage of events that occurred per platform. Only appears if multiple platforms are configured for the project.
The default view is the last 30 days of data. You can select a new time frame, and the reports will reload with the data for that period. For reports for multiple platforms, you can filter by one or more platforms. To export data, hover over a report, then select the gear icon () and select **Download**. You can select from various output and other formatting options. ### Experiment detail The **Experiment Detail** section of an experiment report contains a summary of the experiment's settings and displays the number of users in the holdout and treatment groups. Select a count to access a list of the users' [Channel IDs](https://www.airship.com/docs/reference/glossary/#channel_id) and [Named User IDs](https://www.airship.com/docs/reference/glossary/#named_user), and then select **Download** to export the list. ## Managing Holdout Experiments You can manage your most recent experiment. Go to **Experiments**, then **Holdout Experiments**. Information about the most recent experiment appears next to the report for the first Goal added when setting up the experiment. Experiment management options: | Option | Description | Steps | | --- | --- | --- | | **Edit** | You can change the settings of an Active or Scheduled experiment. For Active experiments, you cannot change the holdout group allocation or transactional message status. | | | **Delete a Scheduled experiment** | Removes the experiment from your project. Deleted experiments cannot be recovered. | Select the trash can icon ( ). | | **Start a Scheduled experiment** | Changes the start date and time of the experiment to the current date and time. | Select **Start now**. | | **End an Active experiment** | Changes the end date and time of the experiment to the current date and time. You cannot restart an ended experiment. | Select **End now**. | ## Setting Campaign Categories You can set [Campaign Categories](#message-exclusion-and-allowances) per message. For the API, see the [Campaigns Object](https://www.airship.com/docs/developer/rest-api/ua/schemas/push/#campaignsobject) documentation. In the dashboard, these are the locations per composer: | Composer | Composer step | Documentation | | --- | --- | --- | | **Message, A/B Test, Automation, Sequence** | Delivery | [Campaign Categories](https://www.airship.com/docs/guides/messaging/messages/delivery/delivery-options/#campaign-categories) in _Message delivery options_ | | **In-App Automation, Scene** | Settings | [Campaign Categories](https://www.airship.com/docs/guides/messaging/in-app-experiences/configuration/optional-features/#campaign-categories) in _Set optional message features_ |