Last updated Oct 24, 2022

Rate this page:

Ingestion Flow

An event is emitted by a provider (i.e. Cloud Provisioner, Bitbucket, Trello, etc). This event is expected to arrive at the (first) SQS Queue: rps-stg-east-admin-portfolio-tlt-streamhub-consumer-queue and the message will be stored in the (first) S3 Bucket: rps--stg-east--admin-portfolio--payload-store-cp.

When admin-portfolio service (APS) receives this initial message, it emits another message in a normalized format. This message is expected to arrive at the (second) SQS Queue: rps-stg-east-admin-portfolio-workspace-events-process-queue and the message will be stored in the (second) S3 Bucket: rps--admin-portfolio--event-store-raw.use1.staging.atl-paas.net.

When admin-portfolio service (APS) receives this second message, it fetches data from the provider (i.e. Cloud Provisioner/TCS, Bitbucket, Trello, etc). APS will use this data to persist to the elasticsearch database.

Cloud Provisioner provider is a special case in this flow. To retrofit cloud provisioner's PUSH model, the first S3 Bucket is used as a fallback if TCS does not return the latest data.

Rate this page: