---

Data ingestion

Data ingestion is the entry point to CNS. The idea behind the ingestion layer is to independently identity business events that would eventually require a notification to be triggered. For example, a quote getting finalised would need the Open Quote email to be sent out, an order getting placed would require the Order Confirmation email to be sent out, etc. The aim is to shift the responsibility of identifying and sending the trigger for the notification (when to send, what to send, to whom to send) from multiple different client applications to CNS.

Architecture

Components and code flow

Business Event Processor

We receive different types of events from StreamHub such as Identity, Quotes, Order, etc. Each event has a dedicated Event processor to handle the pre-processing of the event payloads.

Below is a dummy event payload. Note the type and schema of the event. These are the avi and ari respectively which needs to be added to the data-ingestion code in their correponding yaml files so that its considered for processing. The corresponding event processor for the below mentioned event can be found here

1
2
{
    "type": "avi:ccp:accounts-receivable:update:ship-to-party",
    "schema": "ari:cloud:platform-services::streamhub-schema/ccp/accounts-receivable/ship_to_party_data_v1.json",
    "schemaAri": "ari:cloud:platform-services::streamhub-schema/ccp/accounts-receivable/ship_to_party_data_v1.json",
    "ingestionSource": "micros/shipToParty",
    "ingestionTime": "2024-07-26T05:34:07.285Z",
    "ingestionEnv": "stg-east",
    "ingestionIp": "10.176.146.12",
    "payload": {
        "id": "12345",
        "version": 1,
        "name": "Sample Name",
        "taxId": "TAX12345",
        "taxIds": [
            {
                "type": "VAT",
                "value": "VAT12345"
            }
        ],
        "postalAddress": {
            "line1": "123 Main St",
            "line2": "Apt 4B",
            "city": "Sample City",
            "state": "Sample State",
            "country": "Sample Country",
            "phone": "123-456-7890",
            "postcode": "12345"
        },
        "active": false,
        "createdAt": 1633036800000,
        "updatedAt": 1633123200000,
        "transactionAccount": {
            "id": "069b9214-3642-4d8f-b168-ee*********"
        },
        "additionalProperties": {
            "customField1": "Custom Value 1",
            "customField2": "Custom Value 2"
        }
    },
    "remote": false,
    "kplEvent": false,
    "dataClassifications": [
        "UGC/Primary",
        "UGC/PrimaryIdentifier",
        "PII/DirectRestricted"
    ],
    "producerServiceId": "soldToAddress",
    "eventId": "32268bcc-1c00-423f-bd1f-cee04*******"
}

Notification Message Generator

1. Determine Business Events

   The Notification Message Generator helps in identifying the business event associated with the payload received (for example, order confirmation, account modification, open quote event, etc).

2. Generate Notification Events

   Each business event may require notifications to be sent to multiple users. The relationship between Business Events and Notifications events are 1:many. For example, Order confirmation mail needs to be sent to both the Partners and ECCs (End Customer Contacts). In some cases, the same template is being used for different personas (conditional template rendering) while in other cases, different templates is chosen based on the user persona. To handle all such scenarios, the ingestion layer maintains the mapping of business events vs notification events as a configuration in notificationEventConfig. Hence, enabling/disabling a notification to a new user persona (for the same business event) will NOT require a code change in Ingestion.

3. Publish Notification Events

   If any business events require de-duplication (for eg, order items sent individually via StreamHub) then a FIFO queue is used for publishing such notification events else the notification is sent to PostOfficeService for publishing.

Message Context

The entire responsibility of fetching the data that needs to be rendered on the Email/Pdf lies with the orchestration layer. In order for the orchestration layer to fetch the appropriate data, there are some basic high-level key identifiers (ingested data) which needs to be passed on as part of the message event trigger (for example transaction account id, order number, template name, etc). This is termed as Message Context.

1. Attribute Mapper

   Lists down the attribute names needed per template.

2. Attribute Populator

   Populates the necessary template attributes with its value which are then added to the message context.

You can follow this sample PR for more understanding - data-ingetion sample PR