Last updated Aug 13, 2024

Integrating with Live Ingestion

Sending your live data to DROID

Live ingestion is done via StreamHub (our platform for everything events at Atlassian) in a shoot-and-forget fashion. You are to publish events containing the data you wish to ingest to SH according to our contract. We will listen to those events, ingest them into DROID and distribute them to TCS and the sidecars accordingly.

Step 1: Review your data model and decide how many entity types will be needed

Entity types are how we organise source data coming into DROID. Think about it like tables in relational databases. Depend on your use case you may only need one entity type or you may need many. If you intend to do transformation on your data later make sure all data that you need in a transformer reside in a same entity type. We don’t support record stitching i.e. pulling data from multiple source entities to craft a transformer output.

Step 2: Decide on an AVI for each entity type and inform DROID team

For each entity type you need to choose an AVI (event type identifier) to publish new/updated source data to StreamHub. Your AVI should follow this pattern:

1
2
avi:<your-service-name>:external-ingestion:<entity-type>

When this is finalised, remember to inform the friendly Context - Integration squad about list of entity types and their AVIs.

Step 3: Integrate with StreamHub to publish events when your source data is changed

Follow StreamHub’s guide to publish appropriate events when necessary:

All events must use the schema published by us here. Schema validation is done by SH at the source.
You can use SH authorization to restrict who can produce and consume your events. We recommend you restrict the producing side to your service and the consuming side to DROID only (see Step 4 below).
If you need to encrypt any part of your data, have a look at [TCS-sidecar] Adding a new auto-decryptor

Event fields are documented in the schema but some deserve special mentions here:

Field	Description
`entityId`	A pair of `entityType` and `entityId` identify a unique entity within DROID. For dos and donts regarding `entityId`, please checkout Entity identifiers dos and donts
`version`	Extremely important to the correctness and efficient operation of TCS. Please set this to a positive number that reflects how up to date your data is. Incoming entity won’t be ingested if existing entity (of same `entityType` and `entityId`) has a greater version. 1 2 `e2.isNewerThan(e2) => e1.version > e2.version`
`deleted`	Set this to `true` if you want to delete this entity and all its keys from TCS
`content`	a JSON string of your source data. If `deleted` is set to `true` you can set this field to an empty JSON String `{}`

(Optionally) Step 4: Allow the DROID service access to your events

If SH authorization is used, you must allow DROID to be a consumer of your event. Our Micros’s serviceId is transformerservice.

Step 5: Check with DROID team to see if your events have been successfully ingested

Reach out to DROID team to check if they have received and ingested your events successfully. Meanwhile, you can also check StreamHub’s producer - consumer dashboard to see if your events have been delivered by StreamHub successfully.