Validating the Correctness of Assessment Posture in EPM

This document outlines the steps required to validate the correctness of an assessment's posture as displayed in EPM. Follow these instructions to ensure accurate validation.

Prerequisites

Before proceeding, ensure the following prerequisites are met:

1. Successful EPM Workflows: The EPM workflows for the current date must have run successfully.
2. Published AR-Evaluator Results: The ar-evaluator must have successfully published results for the assessment.
3. Assessment as Responsibility: The assessment must be added as a responsibility under a requirement in EPM.

The validation script used in the following steps was created by the FedRAMP Control Monitoring Workstream of CAP during FY25Q4 and is located in this repository. More information can be found on this Confluence page.

Ensuring EPM Workflows Ran Successfully

1. Visit the EPM workflow orchestrator dashboard at go/rmsfx.
2. The Component Workflow Scheduler dashboard should show a green bar, indicating that all workflows succeeded.
3. If a red bar is displayed, consult the #help-enterprise-posture-management channel to check for known issues or to request support.
4. If the bar is red, scroll down to identify the failed workflow. You can proceed if the failure is unrelated to the Component Discovery Workflow or the Assessment Workflow.

Step 1: Set Up the Validation Script Locally

1. Clone the repository.
2. Create a Python virtual environment by running the following commands:

   1
   2
   python3 -m venv .venv
   source .venv/bin/activate
   

3. Install the required packages and set up the PYTHONPATH:

   1
   2
   pip install -r requirements.txt
   export PYTHONPATH='./src'
   

Step 2: Run the Validation Script

1. Fetch the Requirement ID for the requirement linked to your assessment.
   1. The ID is part of the EPM requirement URL. For example, in https://resp-model-ui-public.us-east-1.prod.public.atl-paas.net/content-management/requirements/4b1e7918-7d09-4191-b13d-555ece9862d3, the ID is 4b1e7918-7d09-4191-b13d-555ece9862d3.
2. Fetch the Component ID for the component linked to your assessment.
   1. This file lists all components currently in production: components-prod.json.
   2. If your component is not listed, visit EPM's GraphiQL interface: https://responsibility-model.prod.atl-paas.net/graphiql?path=/graphql.
      1. Use the following query to fetch all components:

         1
         2
         query GetAllComponents {
           components {
             id
             name
           }
         }
         

3. Fetch both the Micros and Non-micros counterparts of the component.
   1. Micros components are prefixed with Micros <RESOURCE_NAME>.
   2. Non-micros components are prefixed with AWS or just the <RESOURCE_NAME>.
4. Execute the script with the requirement and component IDs:

   1
   2
   python src/fedramp-assessments-validator/combined_validator.py <REQUIREMENT_ID> --component_id <COMPONENT_ID> <COMPONENT_ID>
   

Step 3: Analyze the Results

1. Analyze the output generated by the validation script.

The script may surface several types of issues, including but not limited to the following:

Inventory Mismatch

An inventory mismatch can occur if EPM displays a large number of component instances with an "Unknown" status. This often indicates that the component discovery query in EPM and the assessment query in ar-evaluator are not referencing the same time frame. The FedRAMP Control Monitoring Workstream established a standardized time frame for both processes, as detailed in this decision document.

Incomplete Workflows or Stale Data in EPM

Inaccurate or stale data can result from failed EPM workflows, which may not have successfully updated or ingested new data.

Untagged Resources

If ar-evaluator returns results for an instance that lacks a utilizingService tag, that instance will not appear in EPM. The component discovery process only ingests resources that can be mapped to a specific service. This issue typically points to a tagging problem, as most inventory is expected to be tagged with its corresponding service.