Export File Specification

Overview

Use this reference to understand how the Simplified Exporter structures output files. It covers the two export modes — full table exports and incremental exports — along with chunking behaviour, the JSONL format, metadata headers, and standard fields. The exporter supports ActiveObjects (AO) tables, shared core tables, and ID mapping files.

File Structure

Export files use JSONL (Newline-Delimited JSON) format for efficient streaming and processing.

Metadata Header (First Row)

The first line of every export file contains metadata about the export:

Example Metadata line (formatted for readability)

1
2
{"table":"AO_6FF49D_DOMAIN_SCAN","columns":[
  {"name":"BLOCKED_DOMAINS_COUNT","type":"number","size":64,"nullable":true},
  {"name":"DISTINCT_DOMAINS_COUNT","type":"number","size":64,"nullable":true},
  {"name":"EXECUTION_TIME","type":"number","size":64,"nullable":true},
  {"name":"FINISHED_AT","type":"number","size":64,"nullable":true},
  {"name":"ID","type":"varchar","size":-1,"nullable":false},
  {"name":"LOOKUP_METHOD","type":"varchar","size":-1,"nullable":true},
  {"name":"STARTED_AT","type":"number","size":64,"nullable":true},
  {"name":"STATUS","type":"varchar","size":-1,"nullable":true},
  {"name":"DELETED","type":"boolean","size":1,"nullable":false},
  {"name":"TXN_SEQUENCE","type":"integer","size":19,"nullable":false}],
  "primaryKeys":["ID"],"rowCount":0,"TYPE":"METADATA"}

Data Records (Rows 2+)

Each subsequent line is a complete data record. There is no TYPE field for data records.

Example Data Record line (formatted for readability)

1
2
{"BLOCKED_DOMAINS_COUNT":0,"DISTINCT_DOMAINS_COUNT":1,"EXECUTION_TIME":3923,"FINISHED_AT":1773982903301,
  "ID":"821a09b3-8593-4c4b-a636-e483564f4d01","LOOKUP_METHOD":"UMS","STARTED_AT":1773982899378,
  "STATUS":"FINISHED","DELETED":false,"TXN_SEQUENCE":0}

Note: Exports include DELETED and TXN_SEQUENCE values for all data records.

Export Types

Files may be a full table export for an initial snapshot, or incremental changes after the snapshot. The file format is identical for both types of exports, the only difference is the incremental will make use of the DELETED and TXN_SEQUENCE columns to flag changes.

1. Full Table Export

Complete snapshot of all table data at export time.

• Contains: All table rows with complete column data including DELETED and TXN_SEQUENCE fields
• Standard Fields: DELETED: false and TXN_SEQUENCE: 0 for all records
• Use Case: Initial data migration, complete backups

2. Incremental Export

Captures only changes since last export using Change Data Capture.

• Contains: INSERT, UPDATE, DELETE operations from changes
• Standard Fields: DELETED indicates operation type, TXN_SEQUENCE tracks change order
• Use Case: Real-time synchronization, change tracking

When consuming incremental export files, the row with the highest TXN_SEQUENCE value for a key, should be retained. Each row is a complete data record for that key in the table. There is no need to retain data for earlier TXN_SEQUENCE rows for that key, or reconstruct changes.

File Chunking

Exports are automatically split into multiple files to manage file sizes:

Chunking Strategy

• Max File Size: 4 GB per file
• Trigger: Files are split when they would exceed the 4GB limit
• Metadata: Each chunk includes its own metadata header (first line)
• Reassembly: Chunks can be concatenated by skipping the metadata header from chunks 2+

Standard Fields

Every exported record includes two standard fields:

DELETED Field

• Type: boolean
• Purpose: Indicates if record represents a deletion
• Full Export: Always false
• Incremental Export: true for DELETE operations, false for INSERT/UPDATE
• Name is upper case.

TXN_SEQUENCE Field

• Type: integer
• Purpose: Tracks transaction order for change sequencing
• Full Export: Always 0
• Incremental Export: Increasing counter (1, 2, 7...)
• Name is upper case.

---