# Holdout Experiments

Use holdout experiments to compare the performance of audience members who receive messaging (the treatment group) with those who do not (the holdout group) for the duration of the experiment.

When you create a holdout experiment, you set up:

1. **Holdout group** — The percentage of your total audience that will be excluded from messaging
1. **Goals** — The events to measure in your experiment
1. **Duration** — The period when the experiment is active

As an experiment runs, reports for each goal show the performance of the holdout and treatment groups. After the experiment ends, or after a period of time relevant to the purpose of the experiment ends, you can evaluate the reports to determine the impact your messaging has on driving conversion goals or KPIs.

If there is no significant difference between holdout and treatment group performance, you may want to consider your campaigns and experiments for areas of improvement. Even with significant differences, this data can help you make informed decisions on how to best evolve your marketing strategy.

![Holdout experiment performance report](https://www.airship.com/docs/images/whats-new/holdout-exp_hu_a09e4a84d2462b37.webp)

*Holdout experiment performance report*

## Requirements and documentation

Minimum SDKs required for [In-App Automation](https://www.airship.com/docs/reference/glossary/#iaa), [Scenes](https://www.airship.com/docs/reference/glossary/#scene), and [Surveys](https://www.airship.com/docs/reference/glossary/#survey) only: iOS 17.2.0, Android 17.2.0.

Get all the details in our [Holdout Experiments](https://www.airship.com/docs/guides/experimentation/holdout-experiments/) docs and start experimenting! 🥽 🥼 🧪