# Real-Time Data Streaming {{< glossary_definition "rtds" >}} {{< badge "addon" >}} With Real-Time Data Streaming (RTDS), Airship is not only your digital engagement service *provider*, but your most valuable *source* of mobile engagement data. > **Note:** Real-Time Data Streaming is an add-on service. [Contact Airship Sales](https://www.airship.com/contact-us/) to enable the service for your account. ## About Real-Time Data Streaming Real-Time Data Streaming (RTDS) delivers an Airship project's events using the Data Streaming API. When you make a call to the API endpoint, it opens a stream of uninterrupted newline JSON, returning events as they occur. This stream remains open, continuously delivering events until you close your connection, so you can see how users interact with your messaging and app or website in real time. Airship sends the events you select during setup, along with relevant identifiers that associate them with a particular user and notification or group of notifications. If you need to monitor a subset of events or specific types of events, you can apply filter parameters when opening an event stream to determine the type and quantity of events returned. All data is available using a single endpoint, with separate base URLs for each data center: * North America: `https://connect.urbanairship.com/api/events` * Europe: `https://connect.asnapieu.com/api/events` See the [Real-Time Data Streaming API reference](https://www.airship.com/docs/developer/rest-api/connect/) and the [Data Formats](https://www.airship.com/docs/developer/rest-api/connect/schemas/) section for lists of all events. ### Compliance events for Email and SMS Compliance events represent operations where users opt in to or out of email and SMS notifications. They are effectively records of your users' consent to receive SMS or email notifications. You can open an event stream containing only these events for use in maintaining compliance records, proving compliance with data regulations, and ensuring that you respect your audience's right to privacy. Compliance events are available to all email and SMS customers through the `/api/events/general` endpoint. Each event is for a single email address or [MSISDN](https://www.airship.com/docs/reference/glossary/#msisdn). For more information, see the following in the RTDS API reference: * [Compliance Event Stream](https://www.airship.com/docs/developer/rest-api/connect/operations/compliance-event-stream/) * [Email compliance events](https://www.airship.com/docs/developer/rest-api/connect/schemas/email-compliance-events/) * [SMS compliance events](https://www.airship.com/docs/developer/rest-api/connect/schemas/sms-compliance-events/) ### Routing your data You can send your Airship data to another system using direct, webhook, and partner integrations: | Integration type | Description | Setup | | --- | --- | --- | | **Direct** | Send all events to your backend systems. You control when you pull data from the event stream and filter the data according to the needs of the system that consumes it. This integration type is designed for customers who are interested in building their own custom applications on top of the event stream. | [Configure an RTDS direct integration](#configure-an-rtds-direct-integration) | | **Webhook** | Send all or selected events to another system using a [webhook](https://en.wikipedia.org/wiki/Webhook), a custom HTTP callback that is generally triggered by an event or action initiated by a user, service, or application. You provide a URL, headers, and authorization info. When the event occurs, the originating application notifies the specified URL configured for the webhook. You can configure triggering sending events between systems, enabling real-time workflows. | [Configure an RTDS webhook integration](#configure-an-rtds-webhook-integration) | | **Partner** | Send all or selected events to an external service. | [Airship partner integrations](https://www.airship.com/docs/integrations/) | ## Connect to the test server If you don't have a project in the Airship system or just want to try a test connection, you can use our test server. The test server data is randomly generated and approximates a mobile user's potential lifecycle. While connecting to the test server is not required, it is a way to see what the event stream looks like. To connect to the test server, paste this code in your terminal window: ```bash curl -vv https://connect-testing.urbanairship.com/api/events/ \ --compressed \ -u "sample_connection:sample_connection" \ -H "Accept: application/vnd.urbanairship+x-ndjson; version=3;" \ -d "{}" ``` After a short time, you should see Airship events appear. ## Configure an RTDS direct integration Connect to the event stream from a terminal window on MacOS or a Linux environment. You will construct a cURL command that connects to the appropriate Airship server, passing in no parameters. The `POST` call will route your event stream data into your terminal window. First, create an access token and get your app key in the dashboard: 1. Next to your project name, select the dropdown menu ( ), then **Settings**. 1. Under **Project settings**, select **Real-Time Data Streaming**. 1. Under **Real-Time Data Streaming**, select **Direct Integration**. 1. Enter a user-friendly name and description. 1. Check the box if you'd like to send location events through this connection. 1. Select **Create Access Token**. 1. Copy the App Key and Access Token values and save them in a secure location. > **Warning:** You will not be able to view the Access Token after leaving this screen. 1. Select **Save and exit**. Next, construct a cURL command to access the Data Streaming API, replacing `` and `` with your actual values: **RTDS example request** ```bash curl -vv https://connect.urbanairship.com/api/events/ \ --compressed \ -H "Authorization: Bearer " \ -H "X-UA-Appkey: " \ -H "Accept: application/vnd.urbanairship+x-ndjson; version=3;" \ -d "{}" ``` You should now be connected and receiving events. There may be a perceived lag between when events are delivered via the Airship event stream and when they appear in your terminal. This is due to how cURL processes compressed events and does not reflect the actual real-time delivery speed of the events. ### Manage direct integrations After you set up a direct integration you can edit its name and description, create more tokens, and delete tokens. You can also delete the integration. In the dashboard: 1. Next to your project name, select the dropdown menu ( ), then **Settings**. 1. Under **Project settings**, select **Real-Time Data Streaming**. 1. Under **Enabled Integrations**, select the direct integration you want to edit. 1. Make your changes: | Option | Description | Steps | | --- | --- | --- | | **Edit the name or description** | The name and description appear in the list of enabled integrations. | Select **Edit**, enter new text, and then select **Save**. | | **Add a token** | You can create a maximum of 25 access tokens per integration. | Next to **Access Tokens**, select **Create Token**, enter a name to identify it, select **Create token**. Next, copy the App Key and Access Token values and save them in a secure location, and then select **Got it** to close the window. You will not be able to view the Access Token after closing the window. | | **Remove a token** | Any direct connection using the token will no longer function. | Under **Access Tokens**, select **Delete** for a token, and then confirm your choice. | | **Delete the integration** | The integration will be removed from your project and its access token will no longer be valid. | Select **Edit**, then **Delete**, and then confirm your choice. | ## Configure an RTDS webhook integration > **Warning:** Webhook integrations are not recommended for high volume applications. Processing time at the client can cause events to back up, resulting in a significantly delayed event stream. [Contact Airship Sales](https://www.airship.com/contact-us/) or [Support](https://support.airship.com/) for help scoping the right Real-Time Data Streaming integration to support your application. To connect to your project's event stream using a webhook integration, configure it in the dashboard: 1. Next to your project name, select the dropdown menu ( ), then **Settings**. 1. Under **Project settings**, select **Partner Integrations**. 1. Select **Webhook**. 1. Configure: | Section | Description | Steps | | --- | --- | --- | | **Name and description** | The name and description will appear in the list of enabled integrations. | Enter text. | | **Webhook request URL** | This is the endpoint URL you want your events POSTed to. The host must support HTTPS. Query parameters are supported. | Enter your URL. | | **Custom headers** | Optional. You can add key/value pairs for the custom header to be sent with the request URL. | Enter a key and value. Select **Add another** for more. | | **Batch size** | By default, a webhook integration sends a JSON array of 10 event objects. This helps prevent lag in higher volume applications. You can select a batch size of a single object or an array of 5, 10, 15, 20, 30, 50, or 100 objects. | Select a batch size. | | **Event types** | The selected events will be sent to your webhook. | Check the boxes to select event types. | 1. Select **Activate**. ### Manage webhook integrations After you set up a webhook integration you can edit its configuration or remove it. In the dashboard: 1. Next to your project name, select the dropdown menu ( ), then **Settings**. 1. Under **Project settings**, select **Partner Integrations**. 1. Select **Webhook**. 1. (To edit) Under **Outbound Integration**, select the pencil icon () for the webhook you want to edit, make your changes, and select **Update** in the last step. You can change the name, description, endpoint URL, custom headers, batch size, or which events to send. 1. (To remove) Under **Outbound Integration**, select the trash can icon (), and then confirm your choice. ## Example RTDS event This is an example of a Custom Event as it would appear in your event stream: ```http POST https://example.com HTTP/1.1 Content-Length: 799 Content-Type: application/json; charset=UTF-8 Custom-Header1: Value1 Custom-Header2: Value2 Authorization: Basic dXNhcj0wYXNzd29yZA== { "id":"7e7f5990-d277-4636-b22c-912cb2ca2ec9", "offset":"1245924", "occurred":"2016-12-08T21:27:01.125Z", "processed":"2016-12-08T21:27:03.000Z", "device":{ "ios_channel":"1819298f-065f-46f1-81a0-1fea91f65ce7", "attributes":{ "locale_variant":"", "app_version":"1.2.3", "device_model":"x86_64", "app_package_name":"com.company.app", "iana_timezone":"America/Los_Angeles", "push_opt_in":"false", "locale_country_code":"US", "device_os":"10.1", "locale_timezone":"-28800", "locale_language_code":"en", "location_enabled":"true", "background_push_enabled":"false", "ua_sdk_version":"8.0.1", "location_permission":"ALWAYS_ALLOWED" } }, "body":{ "name":"finished_game", "session_id":"e025bde1-f974-42fa-a925-aca7054218dc", "properties":{ "game_name":"snaps", "game_score":1000 } }, "type":"CUSTOM" } ``` ## RTDS FAQ The following are answers to frequently asked questions about Real-Time Data Streaming. Why do some events occur in the stream multiple times? : While we take measures to prevent them, duplicates can appear in the stream for various reasons. We provide an ID on every event, which you can use to de-duplicate if your use case requires it. For users of the PARTITION subset, note that partitioning is done by ID, ensuring duplicates will always be on the same stream partition. Why do events appear in the stream with `occurred_at` from the fairly distant past? : Devices that are unable to send events to Airship due to a lack of network connectivity will save generated events and dispatch them the next time the app is open and the device is connected to the internet. This can lead to events not reaching the Airship systems until long after they occur. We do not process or pass through any events older than 90 days. If you have stricter latency requirements, use the latency filter. Why do events appear in my stream with `occurred_at` in the future, or with `occurred_at` after `processed_at`? : Because device clocks can be unreliable, we sometimes receive events that a device claims occurred in the future. If an event's time is more than five minutes in the future, we mark it as having occurred at the current server time. Why do events come in with out of order `occurred_at` times? : Events may be out of time order for an app and for a single device. This can be due to the on-device latency mentioned above and because events are processed in parallel. Consequently, no guarantees are made about the order in which events appear in the stream. Why do events come in with out of order `processed_at` times? : As mentioned above, the systems that process events and prepare them for consumption via the Data Streaming API run in parallel, so we cannot guarantee the events in the resulting stream will be ordered. Why do some events seem to be missing, like no CLOSE event following an OPEN? : This can happen for multiple reasons: > * Events are batched on the device, but to limit the impact on device storage, they may be dropped if a device cannot communicate with Airship’s servers or has met internal storage limits. > * Events may arrive in different batches, with significant gaps between send times. While the Airship SDK will attempt to flush all buffered events on app close, it may not be able to do so. This could mean that the second event will not be received until the next time the app is opened during network connectivity. When a push goes to an uninstalled app instance, why do we get both a `SEND` and an `UNINSTALL` on iOS but only an `UNINSTALL` on Android? : Firebase Cloud Messaging (FCM, Google’s notification transport system) sends feedback about uninstalled devices synchronously in their API response. Apple Push Notification Service (APNs) sends feedback later via a separate channel. Because of this differing behavior, we can immediately discover that an Android user has uninstalled and not mark them as being sent to, but we cannot do the same for an iOS user. This is platform-dependent behavior, so it is subject to change. Why did my stream contain a `PUSH_BODY` for a push that went to 0 devices? : `PUSH_BODY` events are generated when a request is made to our push systems, from sending a message using the dashboard or the push API. At that point, we do not yet have the information to determine if anyone in the specified audience will be sent to. Why does my stream contain a `SEND` event for a device, but that device never received anything? : We emit a `SEND` event when the push platform (APNs or FCM) informs us that it has accepted the push request. That push platform might then fail or otherwise be unable to send the push to the specified device, but we will not receive feedback on this unless the reason that the device is unreachable is that the app has been uninstalled. Why do I get TLS handshake errors when I try to send a request to the Data Streaming API? : In order to protect your data, we support only the newest TLS version (1.2 and above) with a modern cipher:

* TLS_CHACHA20_POLY1305_SHA256 * TLS_AES_256_GCM_SHA384 * TLS_AES_128_GCM_SHA256 * TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 * TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 * TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 Some older HTTP libraries may not support this, but we have confirmed that the following work: * Java 1.6 or newer, but will require the JCE Unlimited Strength package appropriate for your java version from Oracle * Python 2.7.9 or newer * Python 3.4 or newer * Golang 1.5 or newer * NodeJS 0.10.33 or newer * Any HTTP library linked against OpenSSL 1.0.1f or newer (OpenSSL 1.0.1e-fips, which ships with Centos 6, also supports the needed ciphers) Why do I get an HTTP 402 error code when I try to issue a request to the Data Streaming API? : The number of simultaneous connections for a given connection point is restricted, and any connection attempt that exceeds that limit will receive an HTTP 402 error response. If this continues to occur after retrying for at least 30 seconds, please [contact Airship Support](https://support.airship.com/).