How to Integrate with DROID External Ingestion

Please reach out to #help-tcs, as we’re not quite ready for self-service.

Live vs Batch Ingestion

You will need to integrate with 2 separate pipelines to ingest your data into DROID: live and batch ingestion. Live ingestion is done at realtime whenever there is any change to any of your source data. Batch ingestion is done for all your source data in certain events where DROID need to repopulate your whole dataset e.g. DR, spin up new regions, updating transformers (the mapping between your source records and their keys in TCS).

Example

Let say you want to ingest the IP allow-lists of Atlassian tenants to DROID. Live ingestion is when some user creates or updates the IP allow-list of a tenant. Batch ingestion is when the DROID database is somehow corrupted (in an incident) and DROID needs to repopulate the IP allow-lists of all Atlassian tenants.

Live Ingestion

Please follow the Live Ingestion Integration guide here for a detailed step by step guide. We will need a few things from you to set up live ingestion:

• Entity type name(s)
• Streamhub AVI
• TCS key suffix(s)

(Optional) Data Transformation

If you require transformation of your source data, you will need to implement a custom transformer. Feel free to skip this if you simply want your source data to be passed through to TCS as-is.

(Optional) Default values returned for 404s

For 404 responses, where a particular key doesn’t exist, a default response body may be returned instead from TCS. If this was determined as a requirement for your use-case, the DROID team will go ahead and configure it for you.

Cache Size Configuration

The DROID team will work with you to determine the cache size for your data in TCS. This will be based on the size of your data and the expected access patterns detailed in your questionnaire. If you have special needs or further questions, please reach out to the DROID team for clarification.

Batch Ingestion

You will need to implement 2 endpoints for batch ingestion. Please follow the Batch Ingestion Integration guide here for a detailed step by step guide.

(Optional) Encrypting Your Data

If the data you're sending to DROID requires encryption, please follow the encryption guide here.

Granting access to consumers

Consumers can raise a pull-request adding their ASAP issuer to the access control list, for each of your record types. We ask that consumers discuss their requirements with the read-through providers either prior or on the PR.