What is External Ingestion?
For a comparison on DROID integration types between External Ingestion vs Read through, please see here.
Data providers send their data to TCS using StreamHub events (live) and Wrench (bulk). TCS ingest the data and write into DynamoDB, and TCS WebServers are used to lookup from DynamoDB.
TCS sends invalidation events when DynamoDB writes change data.
How does it work?
Your service to send events to StreamHub whenever a record is created or updated. The AVI, schema and throughput of the events must be agreed upon by both your team and Context team. The data is then transformed, ingested by TCS and stored in our DynamoDB backing store. On a cache miss, TCS will fetch the data from its own backing store.
Requirements
If you choose the external ingestion path your service must provide DROID with a mean to bootstrap the whole dataset. There are strict requirements regarding availability, reliability and scale of such method so that when necessary (e.g. in accidents) we can restore the whole dataset in a timely manner.
Best suited for...
| Data shape | Transformers supported |
| Record size | Max is determined as min of ( StreamHub event size limit, DynamoDB record size limit, SQS record size limit ) ~ 250KB as of Jan 5, 2024 |
| Dataset “liveness” / access pattern | suit use cases where most records may be used at some point |
| Frequency of change | |
| Traffic | |
| Producer Integration Tier | Can integrate with all service tiers. (ingestion freshness dependent on StreamHub supplier) |
Rate this page: