Lite Dynamic State Import/Export Runbook

Overview

The Lite Dynamic State Import/Export functionality provides a streamlined way to import and export component instance data derived from the ResponsiveResponsibilityAggregate materialized view. This system is designed to generate the minimum number of Component Instances (CIs) required to support the correct aggregated posture for UI display and analysis.

Purpose

This functionality analyzes aggregated objects and generates the minimum number of Component Instances (CIs) required to support the correct aggregated posture. For example, if you have a ResponsiveResponsibilityAggregate that represents 200 Component Instance Responsibilities (CIRs) with 190 passing and 10 failing, the system produces 1 CI, CIR, and Component Instance Responsibility Fulfillment (CIRF) for the failing state.

Use Cases

• UI Display: Prepare surface-level data for UI components that need aggregated posture information
• Bulk Operations: Efficiently handle large datasets (70k+ records) with optimized database operations
• Data Synchronization: Keep component instance data in sync with aggregated responsibility data
• Performance Optimization: Avoid the overhead of running the full orchestrator for surface-level analysis

Architecture

Data Flow

1. Export: Query ResponsiveResponsibilityAggregate materialized view
2. Transform: Convert aggregated data to ExportedLiteDynamicStateObject format
3. Import: Create minimal Component Instances, Responsibilities, and Fulfillments
4. Tagging: All imported records are tagged with createdBy: "lite-import-script"

Key Components

• Export Handler: LiteDynamicStateExportHandler
• Import Handler: LiteDynamicStateImportHandler
• Controllers: DataExportController, DataImportController
• Repositories: Optimized for bulk operations with 70k+ records

API Endpoints

Export Lite Dynamic State

Endpoint: POST /api/v1/admin/export-lite-dynamic-state

Request Body:

1
2
{
  "serviceNames": ["service-1", "service-2", "service-3"]
}

Response:

1
2
{
  "liteDynamicStateObjects": [
    {
      "responsibilitySlug": "security-control-001",
      "serviceName": "my-service",
      "status": "FAILING",
      "boundarySlug": "PRODUCTION",
      "componentSlug": "APP"
    }
  ]
}

Import Lite Dynamic State

Endpoint: POST /api/v1/admin/import-lite-dynamic-state

Request Body:

1
2
{
  "liteDynamicStateObjects": [
    {
      "responsibilitySlug": "security-control-001",
      "serviceName": "my-service",
      "status": "FAILING",
      "boundarySlug": "PRODUCTION",
      "componentSlug": "APP"
    }
  ],
  "config": "REPLACE_ALL"
}

Response:

1
2
{
  "componentInstancesInserted": 50000,
  "componentInstanceResponsibilitiesInserted": 50000,
  "componentInstanceResponsibilityFulfillmentsInserted": 50000,
  "totalRecordsProcessed": 50000,
  "componentInstancesDeleted": 45000,
  "componentInstanceResponsibilitiesDeleted": 45000,
  "componentInstanceResponsibilityFulfillmentsDeleted": 45000
}

Configuration Options

Import Configurations

INSERT Mode

• Purpose: Insert only new records that don't exist
• Behavior: Skips existing records, only adds new ones
• Use Case: Incremental updates, avoiding data loss

REPLACE_ALL Mode (Default)

• Purpose: Replace all existing script-created records
• Behavior:
  1. Deletes all existing records created by lite-import-script
  2. Imports new records
• Use Case: Full refresh, clean state management
• Performance: Optimized bulk delete operations for 70k+ records

Export Filtering

Service Name Filtering

• Optional: Filter exports by specific service names
• Default: Export all services if no filter provided
• Performance: Filtering happens at the database layer for efficiency

Data Model

ExportedLiteDynamicStateObject

1
2
{
  "responsibilitySlug": "string",     // Responsibility identifier
  "serviceName": "string",            // Service name
  "status": "PASSING|FAILING",        // Status
  "boundarySlug": "PRODUCTION|STAGING", // Boundary
  "componentSlug": "APP|SERVICE"      // Component type
}

Generated Component Instances

• Name: {componentSlug}-{serviceName}-{randomHash}
• Parent: Service component instance (if not SERVICE type)
• Boundary: Mapped from boundarySlug
• Component: Mapped from componentSlug
• Created By: lite-import-script

Performance Optimizations

Bulk Operations

• Database Level: Native SQL bulk delete operations
• Batch Processing: 500 records per batch for imports
• Memory Efficient: Avoids loading large datasets into memory

Query Optimization

• Entity Graphs: Eager loading of related entities
• Bulk Queries: findAllById instead of individual lookups
• N+1 Prevention: Bulk fetching of related entities

Methodology

Aggregation Analysis

The system analyzes ResponsiveResponsibilityAggregate records and generates the minimum number of Component Instances required to represent the aggregated posture:

1. Group by Service: Each service gets analyzed independently
2. Status Aggregation: Combines passing/failing states per responsibility
3. Minimal Representation: Creates only the necessary CIs for failing states
4. Efficiency: Reduces data volume while maintaining accuracy

Example Scenario

1
2
Input: ResponsiveResponsibilityAggregate
- Service: "my-service"
- Responsibility: "security-control-001" 
- Total CIRs: 200
- Passing: 190
- Failing: 10

Output: 1 Component Instance
- CI: "APP-my-service-abc123"
- CIR: Links to security-control-001
- CIRF: Status = FAILING

Prerequisites

Database Requirements

• Materialized View: ResponsiveResponsibilityAggregate must be up-to-date
• Refresh Views: Run REFRESH VIEWS before export operations
• Permissions: Database write access for import operations

System Requirements

• Memory: Sufficient heap for batch processing
• Database Connections: Pool size for bulk operations
• Transaction Timeout: Extended timeout for large imports

Step-by-Step Usage

1. Prepare Data

1
2
# Refresh the materialized view
curl -X POST "https://your-api/api/v1/admin/refresh-views"

2. Export Data

1
2
curl -X POST "https://your-api/api/v1/admin/export-lite-dynamic-state" \
  -H "Content-Type: application/json" \
  -d '{
    "serviceNames": ["service-1", "service-2"]
  }'

3. Import Data

1
2
curl -X POST "https://your-api/api/v1/admin/import-lite-dynamic-state" \
  -H "Content-Type: application/json" \
  -d '{
    "liteDynamicStateObjects": [...],
    "config": "REPLACE_ALL"
  }'

4. Verify Results

Check the response for insertion and deletion counts to verify the operation completed successfully.

Monitoring and Logging

Log Messages

1
2
INFO: Importing 50000 lite dynamic objects with config: REPLACE_ALL
INFO: Deleted 45000 ComponentInstanceResponsibilityFulfillment records
INFO: Deleted 45000 ComponentInstanceResponsibility records  
INFO: Deleted 45000 ComponentInstance records
INFO: Successfully imported 50000 component instances, 50000 responsibilities, 50000 fulfillments
INFO: Deletion summary - 45000 component instances, 45000 responsibilities, 45000 fulfillments deleted

Response Monitoring

• Insertion Counts: Track successful imports
• Deletion Counts: Monitor cleanup operations
• Error Handling: Failed records are logged with warnings

Limitations

Current Limitations

1. Assessment Results: Does not export assessment results today
2. Granular Detail: For super granular component instances, use the orchestrator instead
3. Real-time: Data is based on materialized view snapshots
4. Scope: Limited to aggregated posture representation

When to Use Orchestrator

• Detailed Analysis: Need granular component instance details
• Full Orchestration: Complete component instance lifecycle
• Complex Relationships: Multi-level component hierarchies

When to Use Lite Import

• UI Display: Surface-level aggregated data
• Bulk Operations: Large-scale data synchronization
• Performance: Fast aggregated posture analysis

Troubleshooting

Common Issues

Export Returns Empty Results

• Cause: Materialized view not refreshed
• Solution: Run REFRESH VIEWS command
• Prevention: Schedule regular view refreshes

Import Timeout

• Cause: Large dataset processing
• Solution: Increase transaction timeout
• Optimization: Use batch processing

Memory Issues

• Cause: Loading too many records at once
• Solution: Reduce batch size or increase heap
• Monitoring: Watch memory usage during operations

Error Codes

• 400: Invalid request format
• 500: Database operation failed
• Timeout: Transaction timeout exceeded

Best Practices

Performance

1. Refresh Views: Always refresh materialized views before export
2. Batch Size: Use default 500 record batches
3. Monitoring: Monitor response times and memory usage
4. Cleanup: Use REPLACE_ALL for clean state management

Data Quality

1. Validation: Verify service names exist before import
2. Consistency: Ensure boundary and component slugs are valid
3. Testing: Test with small datasets before large imports
4. Backup: Consider backing up data before REPLACE_ALL operations

Operational

1. Scheduling: Run during low-traffic periods
2. Monitoring: Set up alerts for failed operations
3. Documentation: Keep track of import/export operations
4. Maintenance: Regular cleanup of old script-created records

Related Documentation

• Materialized Views: ResponsiveResponsibilityAggregate documentation
• Component Instances: Component instance lifecycle management
• Orchestrator: Full component orchestration system
• API Reference: Complete API documentation

Support

For issues or questions:

1. Check logs for error messages
2. Verify materialized view freshness
3. Test with smaller datasets
4. Contact the platform team for assistance

---

Last Updated: [Current Date] Version: 1.0 Maintainer: Platform Team