Loading Data on your local machine

In order to effectively develop new features in EPM. It is often helpful to have an understanding of the existing jobs, and be able to utilize them to load certain data. In this page, we provide an explanation of all the EPM jobs. Focusing on their purpose and how to call them.

Recommended Loading Sequence

Step 1: Import Data (Required First Step)

You MUST run the import/export workflow FIRST to load all the base data (frameworks, controls, requirements, responsibilities, components, services, apps, attestations, etc.) from production or staging.

See the Import/Export Job section below for details.

Step 2: Complete the Loading Process

After importing data, you have two options to complete the loading process:

Option A: Use the Orchestrator (Recommended - Fully Automated)

Invoke the Component Workflow Orchestrator which automatically runs all remaining workflows in the correct order, including user loading:

• Method: POST
• URL: /api/asyncJobs/componentWorkflows
• Body (Optional):

1
2
{
    "componentSlugs": ["COMPONENT_SLUG_1", "COMPONENT_SLUG_2"]
}

This orchestrator will handle the entire remaining workflow sequence for you, including loading users.

Option B: Run Individual Steps Manually (For More Control)

You can run the individual jobs listed below independently. This is useful when:

• You only need specific data loaded
• You want more control over the execution sequence
• You're debugging specific workflows

See the Fast Loading After Import-All section for the recommended manual sequence.

---

Import/Export Job (Loading Standard Data)

There's plenty of data that is unique to production and staging, and it's often helpful to be able to somehow get that data locally. The import/export job exists to help you load a whole lot of data that would otherwise require manual creation, saving you countless hours.

Here is a list of the data that can be exported:

• Frameworks (Standards)
• Controls
• Requirements
• Responsibilities
• Components
• Tags
• MCD (Manually contributed component instances)
• Attestations
• In Scope Standards
• Services (Component Instances of Component Type Service)
• Apps (Component Instances of Component Type App)
• Boundaries

All necessary junctions get exported as well.

Exporting

To export the data from a particular environment, run the following API:

• Method: GET
• URL: /api/admin/export-all

Note: The export API now returns a complete ImportDataRequest object that is ready to use. You can directly use the response as the request body for the import API without any modifications.

Individual Export Endpoints

You can also export specific data types individually:

• /api/admin/export-requirements - Requirements only
• /api/admin/export-components - Components only
• /api/admin/export-responsibilities - Responsibilities only
• /api/admin/export-services - Services only
• /api/admin/export-apps - Apps only
• /api/admin/export-attestations - Attestations only (excludes soft-deleted)
• /api/admin/export-mcds - MCDs only (excludes soft-deleted)
• /api/admin/export-tags - Tags only
• /api/admin/export-change-management-data - Change management data only

Importing

To import the data into a particular environment, use the response from the export API directly as your request body:

• Method: POST
• URL: /api/admin/import-all
• Body: Use the complete response from the export API

Optional: You can modify the configuration values in the export response if needed:

• Change "config" to "UPSERT" or "REPLACE_ALL"
• Change "withServicesAttestationsAndMCD" to true or false

Available Configurations

Additional Options

• withServicesAttestationsAndMCD:
  • If true, it will include attestations, services, and MCD.
  • If false, it will exclude attestations, services, and MCD.

Fast Loading After Import-All

Important: If you've completed the import-all workflow with withServicesAttestationsAndMCD: true, you do NOT need to run the following steps:

• Step 1: Loading Services
• Step 2: Loading Apps
• Step 5: Sync App In-Scope Standards
• Step 6: Sync Dry Run Standards (for services)

These are already included in the imported data.

⚠️ User Data Limitation: The import/export workflow does NOT support loading users. Users must be loaded separately.

To load data as fast as possible after import-all, manually invoke the remaining steps in sequential order:

1. Step 3: Load Component Instances (Inventory) - /api/workflow/load-asset-instances
2. Step 4: Load Users - /api/workflow/upload-user-relations ⚠️ Required - not included in import
3. Step 7: Sync Responsibilities - /api/workflow/sync-responsibilities-to-instances
4. Step 8: Upload Assessments - /api/workflow/upload-assessment-results
5. Step 9: Evaluate Component Adoption - /api/workflow/evaluate-component-adoption
6. Step 10: Refresh Views - /api/workflow/refresh-views ✅ Critical final step

---

Loading Services (Step 1 of Orchestrator)

You typically do not need to run this API if you've just completed the import/export sequence. However, if you did not load services and wish to do so deliberately, call the following API:

• Method: POST
• URL: /api/workflow/load-service-instances

Loading Apps (Step 2 of Orchestrator)

Load app instances from the app discovery source:

• Method: POST
• URL: /api/workflow/load-app-instances

Loading Component Instances (Inventory) (Step 3 of Orchestrator)

Run this job to obtain the latest inventory data for all components, excluding components of type service and app:

• Method: POST
• URL: /api/workflow/load-asset-instances
• Body:

1
2
{
    "componentSlugs": ["COMPONENT_SLUG_1", "COMPONENT_SLUG_2"]
}

Multi-threading Configuration: Component discovery can be run in multi-threaded mode for better performance. This is controlled by the feature flag epm_be_inventory_discovery_multi_threading_enabled and configured in application.yml:

1
2
threadPool:
  componentDiscovery:
    threadCount: 8        # Number of threads for parallel processing
    timeoutMinutes: 5     # Timeout for each thread

Loading Users (Step 4 of Orchestrator)

⚠️ Important: User loading is NOT supported via the import/export workflow (/api/admin/import-all). However, it IS automatically handled by the Component Workflow Orchestrator (/api/asyncJobs/componentWorkflows).

If you're running steps manually after using import/export, you must run this step:

• Method: POST
• URL: /api/workflow/upload-user-relations
• Query Parameters (Optional):
  • socratesPageSize (default: 10500)

Sync App In-Scope Standards (Step 5 of Orchestrator)

Synchronize in-scope standards for app instances:

• Method: POST
• URL: /api/workflow/sync-app-standards

Sync Dry Run Standards (Step 6 of Orchestrator)

Synchronize dry run standards for services:

• Method: POST
• URL: /api/workflow/sync-dry-run-standards

Synching Responsibilities (Step 7 of Orchestrator)

Run this job to instantiate CIRs/CIRFs (Component Instance Responsibilities). These entities capture default compliance posture records for responsibilities against instances. This job creates these junction records but does not evaluate their compliance state. As a result, you'll be able to see responsibilities/requirements for services that are in-scope in the service owner view.

• Method: POST
• URL: /api/workflow/sync-responsibilities-to-instances
• Query Parameters (Optional):
  • componentInstancePageSize

This job also syncs attestations automatically after syncing responsibilities.

Instance Assessment Upload/Evaluation (Step 8 of Orchestrator)

Prerequisite: Run the responsibility synchronization first.

After running the synchronization job, execute this job to upload the latest assessment result. This job inspects a table in Socrates maintained by Asset Readiness, uploading the latest assessment results into our database and then evaluating them.

• Method: POST
• URL: /api/workflow/upload-assessment-results
• Query Parameters (Optional):
  • socratesPageSize (default: 10000)

Multi-threading Configuration: Assessment upload can be run in multi-threaded mode for better performance. This is controlled by the feature flag epm_be_assessment_discovery_multi_threading_enabled and configured in application.yml:

1
2
threadPool:
  assessmentUpload:
    threadCount: 6         # Number of threads for parallel processing
    timeoutMinutes: 10     # Timeout for each thread

If your goal is to run only the evaluation without the upload, you can optionally use the following API:

• Method: POST
• URL: /api/workflow/synch-assessment-results

Component Adoption Evaluation (Step 9 of Orchestrator)

Prerequisite: Run the responsibility synchronization first.

All CIRs that belong to responsibilities of type adoption can be evaluated by inspecting the contextual state. This job evaluates and updates the state of those records to Compliant.

The process works as follows: For a given responsibility under a specific service, check if the service is using any of the components that must be adopted. If any such instances exist, the CIR is deemed "Compliant."

• Method: POST
• URL: /api/workflow/evaluate-component-adoption
• Query Parameters (Optional):
  • instancePageSize
  • persistBatchSize
  • paginate

Refresh Aggregated Views (Step 10 of Orchestrator - Final Step)

Critical: This step must be run after all data loading and evaluation jobs are complete. It refreshes the materialized views and aggregated data that power the UI and reporting.

• Method: POST
• URL: /api/workflow/refresh-views

This is the final step in the orchestrator workflow and ensures that all the data you've loaded is properly reflected in the application's views.

---

Housekeeping Workflows (Optional but Recommended)

The following workflows are part of the orchestrator but are primarily for maintenance. You can skip these during local development, but they're recommended for production-like environments:

Hard Delete Stale Instances

Permanently removes component instances that have been marked as stale for an extended period:

• Method: POST
• URL: /api/workflow/hard-delete-stale-instances

Hard Delete Stale Assessments

Permanently removes assessment records that are no longer relevant (controlled by feature flag epm_be_purge_stale_assessments_enabled):

• Method: POST
• URL: /api/workflow/hard-delete-stale-assessments

Mark Stale Instances

Marks component instances as stale if they haven't been seen in recent discovery runs (default: 24 hours):

• Method: POST
• URL: /api/workflow/mark-stale-instances

---

Additional Optional Workflows

Generate Report View Data

The Report view requires a CSV file, which can be generated by running the following job:

• Method: POST
• URL: /api/workflow/generate-requirement-report/bulk

Load Service In-Scope Standards

If you used the Export/Import sequence with the withServicesAttestationsAndMCD setting set to true, you should not need to run this job. However, if you would like to reload in-scope standards for services only, you may do so by executing the API below:

• Method: POST
• URL: /api/workflow/load-services-in-scope-standards

Note: The in-scope standards import handler now supports both SERVICE and APP component instances.