calibration

Bundle of calibration, discrimination and decision measures.

Run the full calibration assessment and flag rare-event problems.

Area under the ROC curve, by the Mann-Whitney rank formula with ties.

Returns NaN when either class is empty.

Partition the Brier score into reliability, resolution and uncertainty.

``Brier = reliability - resolution + uncertainty``

frequency within a bin. Lower is better; zero means perfectly calibrated at this binning.

better; zero means the model separates nothing.

not of the model. At a base rate near zero it dominates, which is why a very low Brier score at a low base rate says almost nothing.

The partition is binning-dependent, and the residual returned as ``identity_gap`` reports how far the three terms are from reproducing the Brier score exactly.

Citation: Murphy, A. H. (1973). A new vector partition of the probability score. *Journal of Applied Meteorology*, 12(4), 595-600. Verified against the journal record; that article is where the reliability / resolution / uncertainty partition of the Brier score originates.

• reliability: mean squared gap between predicted probability and observed

• resolution: how far bin frequencies depart from the base rate. Higher is

• uncertainty: ``base_rate * (1 - base_rate)``, a property of the outcome,

Mean squared error of probability predictions.

Mean predicted probability against observed event rate.

The weakest calibration requirement: if these two disagree, the model is wrong about the overall level of risk regardless of how well it ranks.

Calibration intercept and slope from logistic recalibration.

The slope comes from regressing the outcome on ``logit(p)``. A slope below 1 means the predictions are too extreme (overfitted); above 1 means too compressed. The intercept is estimated with ``logit(p)`` as an offset, so it measures the remaining level shift once the slope is held at 1.

Perfect calibration is intercept 0 and slope 1.

Counts and derived rates when predictions above ``threshold`` trigger action.

``alerts_per_true_event`` is included explicitly because it is the number an operations manager experiences and the number that decides whether a system gets ignored.

Net benefit of the model, of treating all, and of treating none.

A model is worth deploying at a given threshold only where its net benefit exceeds both reference strategies. At low base rates that window is often empty, which is the point of computing it.

Expected and maximum calibration error over bins.

ECE is the sample-weighted mean absolute gap between predicted probability and observed frequency; MCE is the largest such gap. Both depend on the binning, which is reported alongside them.

Net benefit of acting on predictions above ``threshold``.

``NB = TP/n - (FP/n) * (t / (1 - t))``

The threshold encodes the exchange rate between a missed event and a needless intervention: acting at threshold ``t`` says one true positive is worth ``(1-t)/t`` false positives. Comparing net benefit against the treat-all and treat-none strategies is the decision-analytic evaluation that Steyerberg et al. (2010) put alongside discrimination and calibration.

Data for a reliability diagram: mean prediction against observed frequency.

Returns one row per bin with the bin bounds, count, mean predicted probability, observed frequency and a Wilson interval for the observed frequency. The interval is included because at low base rates a bin can easily contain zero events, and a plotted point with no uncertainty attached invites overreading.