calibration¶

Calibration metrics — reliability diagrams and Brier-score decomposition.

Brier(p, y) = mean((p - y)^2): = reliability − resolution + uncertainty (Murphy 1973)

where, for K equal-frequency bins:

reliability = sum_k (n_k/N) * (mean_p_k - mean_y_k)^2 (→ 0 when predicted probabilities match observed freqs)
resolution = sum_k (n_k/N) * (mean_y_k - mean_y)^2 (→ high when bins differ in their observed positive rate)
uncertainty = mean_y * (1 - mean_y) (→ property of the data alone, max 0.25 at p=0.5)

Smaller Brier is better; the decomposition tells you whether bad scores come from miscalibration (reliability) or low discrimination (low resolution).

class scitex_seizure_metrics.calibration.CalibrationReport(brier, reliability, resolution, uncertainty, expected_calibration_error, bin_centers, bin_observed, bin_counts)[source]¶

Bases: object

Container for calibration metrics + per-bin curve points.

Parameters:

brier (float)
reliability (float)
resolution (float)
uncertainty (float)
expected_calibration_error (float)
bin_centers (ndarray)
bin_observed (ndarray)
bin_counts (ndarray)

bin_centers: ndarray¶

bin_counts: ndarray¶

bin_observed: ndarray¶

brier: float¶

expected_calibration_error: float¶

reliability: float¶

resolution: float¶

uncertainty: float¶

scitex_seizure_metrics.calibration.calibration_report(y_true, y_proba, *, n_bins=10, strategy='uniform')[source]¶

Compute Brier decomposition + reliability table.

Parameters:

y_true – 1-D binary labels.
y_proba – 1-D continuous predictions in [0, 1].
n_bins (int) – number of probability bins for the reliability table.
strategy (str) – ‘uniform’ (equal-width) or ‘quantile’ (equal-frequency).

Returns:

CalibrationReport.

Return type:

CalibrationReport