External Calibration Validation¶

This case study in cases/external_calibration_validation/ evaluates the empirical recovery of external calibration benchmarks under varying degrees of population shift and miscalibration.

Study Design¶

The protocol evaluates \(N_{\text{sim}} = 50\) Monte Carlo iterations. In each iteration: - A reference cohort provides empirical prevalence \(p_k^*\) across risk strata. - Unconstrained ML and constrained cMLE models are fitted on source data. - Calibration metrics (CITL, slope, O/E, Brier score) and parameter estimates are calculated on the target cohort.

Experimental Conditions¶

Matched Condition: The reference and target cohorts share common underlying event probability distributions. cMLE should recover near-zero CITL and slope near \(1.0\).
Degraded Condition (Negative Control): An artificial logit shift of \(+6.0\) is introduced to deliberately distort external calibration. This validates that the diagnostic gates detect miscalibration when assumptions fail.

Execution¶

Full Manuscript Profile (\(N_{\text{sim}} = 50\))¶

python -m cases.external_calibration_validation.run_suite \
  --profile full \
  --condition matched \
  --run-label manuscript_external_calibration

Smoke Test Profile (\(N_{\text{sim}} = 2\))¶

python -m cases.external_calibration_validation.run_suite \
  --profile smoke \
  --condition matched