Calibration Metrics & Validation¶

While discrimination (AUC) measures a model's ability to rank high-risk individuals above low-risk individuals, calibration measures whether the absolute numerical probabilities accurately reflect real-world event rates.

Risk Bridge exports four standard calibration metrics on the target population in final/calibration_metrics.csv.

1. Calibration-in-the-Large (CITL)¶

Calibration-in-the-large assesses whether overall predictions are systematically too high or too low.

We fit a univariate logistic regression model on the target population fixing the linear predictor slope to \(1\):

\[ \text{logit}\left(P(Y = 1 \mid \hat{p})\right) = a + \text{logit}(\hat{p}) \]

The estimated intercept \(\hat{a}\) is the Calibration-in-the-Large (CITL): - \(\hat{a} = 0\): Perfect overall calibration. - \(\hat{a} > 0\): The model systematically underestimates risk in the target population. - \(\hat{a} < 0\): The model systematically overestimates risk in the target population.

2. Calibration Slope¶

The calibration slope assesses the spread of predicted probabilities, detecting whether predictions are over-fitted (too extreme) or under-fitted.

We fit an unconstrained recalibration logistic regression model on the target population:

\[ \text{logit}\left(P(Y = 1 \mid \hat{p})\right) = a + b \cdot \text{logit}(\hat{p}) \]

The estimated coefficient \(\hat{b}\) is the Calibration Slope: - \(\hat{b} = 1.0\): Ideal calibration spread. - \(\hat{b} < 1.0\): Predictions are too extreme (over-fitted probabilities, overly optimistic at high ends and overly pessimistic at low ends). - \(\hat{b} > 1.0\): Predictions are too narrow/conservative.

3. Observed-to-Expected (O/E) Ratio¶

The Observed-to-Expected ratio compares the total count of observed binary events in the target cohort to the sum of predicted probabilities:

\[ \text{O/E} = \frac{\sum_{i=1}^N Y_i}{\sum_{i=1}^N \hat{p}_i} \]

\(\text{O/E} = 1.0\): Perfect balance between predicted and actual event counts.
\(\text{O/E} > 1.0\): More events occurred than expected (underestimation).
\(\text{O/E} < 1.0\): Fewer events occurred than expected (overestimation).

4. Brier Score¶

The Brier score measures overall mean squared error on probability forecasts:

\[ \text{Brier} = \frac{1}{N} \sum_{i=1}^N \left( Y_i - \hat{p}_i \right)^2 \]

A lower Brier score indicates superior accuracy. The Brier score can be decomposed into reliability (calibration error) and resolution.

5. Post-Fit Moment Residuals¶

In addition to scalar metrics, Risk Bridge reports the stratum-specific discrepancy in final/calibration_residuals.csv:

\[ r_k = \hat{P}(Y = 1 \mid X \in \mathcal{S}_k; \hat{\theta}) - p_k^* \]

For a well-calibrated cMLE fit, every residual \(r_k\) satisfies \(\lvert r_k \rvert \le \epsilon\).