Adding a Component to the EPM Component Discovery Process

This document outlines the steps required to add a new component to the EPM Component Discovery process. Follow the instructions carefully to ensure successful integration.

Prerequisites

Before proceeding with the steps below, ensure the following prerequisites are met:

1. Component Definition: The component must be defined via the content management workflow.
2. Published State: The component must be in a published state.
3. Matching Slug Attribute: The bean definition for the component must include a slug attribute that matches the slug of the component defined in the content management workflow.

Step 1: Create SQL Discovery Queries

Use Socrates: Identify the source of data for the new component using Socrates. Socrates is used to extract data and gather the necessary information for the component.

Create SQL files to discover your component in different boundaries. Verify these results first in Socrates before implementing.

1. Create a SQL Query File: Write a SQL query that adheres to the required structure for EPM compliance. Save the query as a .sql file in the resources/compliance.assets.sql/ directory.

2. Mandatory Output Fields: Ensure your query includes the following mandatory fields, each serving a critical role in the component discovery process:

   • utilizingService (String): This field identifies the service that the component instance belongs to. This association is crucial because it determines which service the component instance rolls up to. Ensure this field is extracted accurately, as incorrect associations can lead to misalignment in compliance reporting.

   • resourceName (String): This represents the component's name and is used to determine uniqueness. Be cautious with how this name is generated:

     • Consistency is Key: If the resourceName changes between runs (e.g., due to dynamic name generation), it can result in loss of previous evaluations of any Component Instance Responsibilities (CIRs). Consistent naming ensures continuity in evaluations and historical data.

   • arn (String): The Amazon Resource Name (ARN) uniquely identifies the component and is critical for integration with assessment processes. Any changes or inaccuracies in ARN formatting can disrupt assessment integration, so ensure this field is correctly extracted.

Commercial Boundary Query

File: src/main/resources/compliance/assets/sql/vnext/aws-[component-name].sql

1
2
WITH unique_assets_stepfunction AS (
    SELECT
        arn,
        awsAccountId,
        COALESCE(
            get_json_object(tags, '$.micros_service_id'), 
            get_json_object(tags, '$.service_name'), 
            get_json_object(tags, '$.name')
        ) AS utilizingService,
        resourceId,
        resourcename,
        service,
        ROW_NUMBER() OVER (PARTITION BY ARN ORDER BY hour DESC) AS row_number
    FROM cloud_engineering_aws_ops.aws_case_data
    WHERE
        -- Time window filters (last 3 hours)
        YEAR >= date_format(CURRENT_TIMESTAMP - interval '3' HOUR, "yyyy")
        AND YEAR <= date_format(CURRENT_TIMESTAMP - interval '1' HOUR, "yyyy")
        -- ... additional time filters ...
        AND service = 'StepFunctions'  -- Your AWS service name
        AND resource = 'StateMachine'  -- Your resource type
        AND awsaccountid not in (select id from cloud_engineering_aws_ops.aws_account where platform = 'PaaS' and name ilike '%Prod%')
        AND (get_json_object(tags, '$.environment_type')= 'prod' OR get_json_object(tags, '$.environment') ILIKE '%Prod%')
)
SELECT arn, utilizingService, 'COMMERCIAL' as boundary, resourcename as resourceName, resourceId
FROM unique_assets_stepfunction
WHERE row_number = 1 AND utilizingService IS NOT NULL

FedRAMP Boundary Query

File: src/main/resources/compliance/assets/sql/vnext/fedramp-aws-[component-name].sql

1
2
WITH filtered_accounts AS (
    SELECT id
    FROM production.compliance_inventory_source.in_boundary_aws_accounts
    WHERE environment = 'Production'
      AND perimeter = 'fedramp-moderate'
),
unique_assets AS (
    SELECT
        arn,
        awsAccountId,
        resource,
        COALESCE(
            tags:micros_service_id,
            tags:service_name,
            tags:name
        ) AS utilizingService,
        ROW_NUMBER() OVER (PARTITION BY arn ORDER BY hour DESC) AS row_number
    FROM production.compliance_inventory_source.in_boundary_case_3k
    WHERE
        service = 'StepFunctions'  -- Your AWS service
        AND resource = 'StateMachine'  -- Your resource type
        AND awsaccountid NOT IN ('629627155375', '620464583330', '942306077086') -- Exclude Micros accounts
        AND hour BETWEEN CURRENT_DATE - INTERVAL 1 DAY + INTERVAL 3 HOUR
                    AND CURRENT_DATE - INTERVAL 1 DAY + INTERVAL 4 HOUR
)
SELECT
    arn,
    awsAccountId,
    'FEDRAMP' AS boundary,
    resource as resourceName,
    utilizingService
FROM unique_assets ua
JOIN filtered_accounts fa ON ua.awsAccountId = fa.id
WHERE ua.row_number = 1

Common Gotchas:

• Utilizing Service: Ensure that the service association is correct to avoid compliance discrepancies.
• Component Name: Maintain a stable naming convention to preserve historical data integrity.
• ARN Field: Verify the ARN format to ensure seamless assessment integration.

Step 2: Add Component to ComplianceAsset Enum

File: src/main/kotlin/io/atlassian/micros/responsibility/model/model/compliance/dashboard/ComplianceAsset.kt

Add your new component to the ComplianceAsset enum. This name will be mapped to the Component name created on UI.

1
2
enum class ComplianceAsset(val type: String) {
    // ... existing components ...
    AWS_STEP_FUNCTION("AWS_STEP_FUNCTION"),  // Add your new component here
}

Example: For AWS Step Functions, we added AWS_STEP_FUNCTION("AWS_STEP_FUNCTION")

Step 3: Add Assessment Asset Class

File: src/main/kotlin/io/atlassian/micros/responsibility/model/service/core/enums/AssessmentAssetClass.kt

Why needed: This enum maps component types to their assessment identifiers used in database queries and API calls. It provides the string values used to filter and sync assessment data for your specific component type.

Add the corresponding assessment asset class:

1
2
enum class AssessmentAssetClass(val value: String) {
    // ... existing classes ...
    AWS_STEP_FUNCTION("aws-step-function"),  // Add your new asset class
}

Note: Use kebab-case for the value (e.g., aws-step-function)

Step 4: Configure Component Discovery

File: src/main/kotlin/io/atlassian/micros/responsibility/model/config/ComponentDiscoveryConfig.kt

Why needed: This configuration class wires together your component type with its SQL discovery queries. It tells the system which queries to run and how to process the results for your component type. The configuration also specifies data sources and processing rules.

Add a configuration method for your component:

1
2
fun awsStepFunction(): ComponentConfiguration {
    return ComponentConfiguration(
        ComplianceAsset.AWS_STEP_FUNCTION,  // Your enum value
        dataSources = listOf(
            InventorySourceConfig(
                path = "compliance/assets/sql/vnext/fedramp-aws-stepfunction.sql",
                sourceType = SourceType.SOCRATES_V_NEXT,
            ),
            InventorySourceConfig(
                path = "compliance/assets/sql/vnext/aws-stepfunction.sql",
                sourceType = SourceType.SOCRATES_V_NEXT,
            ),
        ),
    )
}

Then add it to the componentConfigurationMap():

1
2
private fun componentConfigurationMap(): Map<String, ComponentConfiguration> {
    return mapOf(
        // ... existing configurations ...
        ComplianceAsset.AWS_STEP_FUNCTION.type to awsStepFunction(),  // Add your mapping
    )
}

Step 5: Update Assessment Upload Service

File: src/main/kotlin/io/atlassian/micros/responsibility/model/service/core/FastAssessmentResultUploadService.kt

Add assessment synchronization for your component:

In synchAssessments() method:

1
2
fun synchAssessments(): Map<String, Any> {
    // ... existing synch calls ...
    cirRepository.synchAssessmentsByClassTypeForARN(AssessmentAssetClass.AWS_STEP_FUNCTION.value)
    // ... rest of method
}

In synchAssessmentsForFulfillments() method:

1
2
fun synchAssessmentsForFulfillments(): MutableMap<String, Any> {
    // ... existing assessments ...
    val awsStepFunctionAssessments = cirfRepository.synchAssessmentsByClassTypeForARN(AssessmentAssetClass.AWS_STEP_FUNCTION.value)
    
    return mutableMapOf(
        // ... existing mappings ...
        "awsStepFunctionAssessmentsSynced" to awsStepFunctionAssessments,
    )
}

Step 6: Create Component via EPM UI

Once the code changes are deployed, the final step is creating the component through the EPM UI:

When adding a component from the UI, ensure that the slug name matches the one defined in the backend code. For example, if the backend slug is AWS_STEP_FUNCTION, create the component in the UI as 'Aws Step Function'. The system will automatically convert this to the correct slug format by adding underscores between words and capitalising all letters, resulting in AWS_STEP_FUNCTION.

For Staging Environment:

• Anyone can create components using the EPM UI
• Navigate to the component creation interface
• Select your new component type
• Configure as needed

For Production Environment:

• raise query in #help-enterprise-posture-management for adding it the UI
• Follow the same UI process as staging

Final Steps

• Test the Integration: Ensure that the component is correctly integrated and tested within the EPM Component Discovery process.
• Review Documentation: Update any relevant documentation to reflect the addition of the new component.

Getting Help

• Slack Channel: #help-enterprise-posture-management

Reference

• Reference PRs: PR #2174, PR #2188

By following these steps, you can successfully add a new component to the EPM Component Discovery process.