Skip to content

Pipeline Architecture & Data Flow

This document details the internal design and data flow of the Risk Bridge execution pipeline.

High-Level Pipeline Flow

Both simulated and user-data execution paths share a unified processing topology:

[Input Cohorts: target, source, reference]
                  │
                  ▼
[Preprocessing & Schema Canonicalization]
                  │
                  ▼
[Calibration Artifact Construction from Reference Data]
(Risk strata, external prevalence p_k*, discrete support enumeration)
                  │
                  ▼
[Sampling Stage]
(Propensity-Score Matching + Random Sampling)
                  │
                  ▼
[Model Fitting Stage]
(Unconstrained ML warm start -> Constrained cMLE via Solver Ladder)
                  │
                  ▼
[Evaluation & Diagnostic Computation]
(Discrimination AUC, Calibration CITL/slope/OE/Brier, Moment Residuals)
                  │
                  ▼
[Export Stage]
(Versioned CSV outputs + environment.json reproducibility sidecar)

The Orchestration Boundary

The pipeline enforces an explicit architectural partition:

\[ \text{Impure Setup} \longrightarrow \text{Typed State} \longrightarrow \text{Pure Row Builders} \longrightarrow \text{Impure Writes} \]
  1. Setup Phase: Reads CSV files, creates timestamped directories, validates user schemas, and initializes parallel workers.
  2. State Bundles: Pure dataclasses carry analysis samples and fitted model parameters without coupling to file formats.
  3. Pure Builders: Evaluates accuracy and calibration metrics without disk side-effects.
  4. Export Phase: Flushes tabular data conforming to CSV output contract schema 1.1.0.

Concurrency Model

Risk Bridge supports two levels of parallelism: - --n-jobs (Process-level): Parallelizes independent Monte Carlo iterations across multiple CPU cores using Python multiprocessing. - --path-jobs (Thread-level): Evaluates PSM and Random Sampling analysis paths concurrently within a single iteration using thread pools.