Summary

EPM is able to report on the posture of component instances that exist solely in the Federal boundary, despite the fact that various information about these instances is sensitive and not allowed to leave the boundary.

This is possible through an egress and obfuscation flow that brings specific pieces of information out of the boundary, available to query within the commercial environment where the EPM apps live.

This process utilizes common egress and data transfer solutions at Atlassian, and is mainly owned by the Cross Boundary team.

Instructions below will help EPM developers troubleshoot issues with this process, so that they can self-service common problems.

End-to-end Data Flow

Source datasets:

• CASE3K (aka CASE 3000) cloud_engineering_aws_ops.aws_case_data: This is a timeseries dataset containing a broad inventory of various AWS resources deployed by Atlassian in the federal boundary. Docs: https://hello.atlassian.net/wiki/spaces/CENG/pages/804237882/Service+CASE+3000
• AWS Accounts cloud_engineering_aws_ops.aws_account: This is a small dataset that lists the IDs and owners of each AWS account provisioned within the Federal boundary.

Egress Process:

The data is egressed using the Bulk Transfer Service, following the standard path for data egress.

If the data is failing to cross the boundary, issues could stem from any of the steps in the egress process. Pointers and nuances below:

• Access to the underlying data for the AWS inventory tables is restricted by default. To allow for access for bulk transfer, RegInd filed a ticket to Cloud Engineering to authorize specific services to access the underlying S3 buckets so that their automation can egress the data.
  • Ticket which also contains security review information: https://hello.jira.atlassian.cloud/browse/CENG-15555
• The RegInd Cross Boundary team has automation set up to utilize the Bulk Transfer Service that periodically reads this data and copies it out: bitbucket pipeline config
• There are bulk transfer transit policies in place to allow egress & obfuscation of this data:
  • CASE3K
  • AWS Accounts

Data Ingestion

Once the data crosses the transit boundary, it needs to be ingested somewhere to be useful.

Because EPM utilizes Socrates Databricks tables for its queries, the data needs to be ingested into Socrates tables.

Pointers:

• Ingestion flow definition
• Table definitions:
  • CASE3K (production.compliance_inventory_source.in_boundary_case_3k)
  • AWS Accounts (production.compliance_inventory_source.in_boundary_aws_accounts)

Note that these tables are distinct from the Commercial versions of the tables, so you will not find egressed federal data in the commercial versions of these tables at all. In the EPM codebase you will find references to the above federal table names.

This is a deliberate choice; the tables are meant for EPM and FinOps access only and are not for general CASE3K usage, hence the permissions are locked down to specific users and the EPM backend application(s).

If you are an EPM developer and need to view the content of these tables, reach out to the Cross Boundary team to get access.

Egressing New Fields

To egress a new field, you'll need to adjust the transit policy and the schema for the ingesting table.

If you are egressing a new property inside a field that is already on the table schema (such as tags on AWS resources), then only a transit policy change is needed.

Contacting the Cross Boundary Team

• Slack Channel: #help-cross-boundary