ADR-0001: True negative for an alarm-based seizure-warning system — SOP-length interictal opportunities (2026-06-29)

Status: Accepted (codifies the convention implemented in src/scitex_seizure_metrics/_classification.py, shipped in v0.2.0).

Context

The detection (sample-based) regime has an unambiguous confusion matrix: every prediction window is one of {TP, FP, FN, TN}, so specificity, NPV and the rest fall out directly. The forecasting (alarm-based) regime does not. An alarm system emits a small number of discrete warnings over a long recording, and seizures are rare events. Three of the four cells are already well-defined and uncontested in the seizure-prediction literature (Mormann 2007; Snyder 2008; Schelter/Winterhalder 2006):

  • TP — a seizure that is caught: a seizure at t_s is a TP iff some alarm fires at t_a with t_a + SPH <= t_s <= t_a + SPH + SOP. This is _alarm.alarm_match’s seizure_caught.sum() — the package’s existing alarm-vs-seizure matching rule.

  • FN — a seizure that is not caught = n_seizures - TP.

  • FP — an alarm that catches no seizure.

The fourth cell, TN, has no canonical unit. There is no natural, agreed-upon “a correctly-quiet interictal moment” for an alarm system, because alarms are events, not a per-instant decision. Yet specificity and NPV — both of which reviewers routinely ask for, and which the detection regime reports — are undefined without one. v0.2.0 had to pick a convention to compute them, and that choice is genuinely convention- dependent: it must be documented, not buried.

Decision

A true negative is an interictal “prediction opportunity” of length SOP in which the system correctly stayed silent.

The interictal time — the same denominator the FP/hr rate already uses, with each seizure’s [t_s - SOP - SPH, t_s + SOP] window removed (Mormann tradition, AlarmPolicy.fp_denominator="interictal") — is partitioned into non-overlapping SOP-length opportunities:

n_opportunities = floor(interictal_seconds / SOP)
TN              = max(0, n_opportunities - FP)

i.e. every SOP-length interictal slot that did not contain a false alarm is one TN. From the four cells:

sensitivity (recall) = TP / (TP + FN)        [per-seizure]
specificity          = TN / (TN + FP)        [per-opportunity]
ppv (alarm precision) = TP / (TP + FP)
npv                  = TN / (TN + FN)
f1 (forecasting_f1)  = 2·TP / (2·TP + FP + FN)

Any ratio with a zero denominator returns NaN (fail-loud), never a silent 0, so “undefined” is never mistaken for “0”.

n_tn and n_opportunities are always reported alongside the scores (MetricsReport.n_tn / n_opportunities) so the TN denominator is never hidden. specificity and npv are read with their denominator; ppv and forecasting_f1 do not depend on the TN convention at all and are safe to compare across studies regardless.

Considered and rejected

The alternatives below are all defensible; the choice between them changes specificity/NPV by orders of magnitude on the same data, which is exactly why it needs an ADR.

Per-window TN (one TN per below-threshold prediction window). This is the detection-regime answer transplanted onto the alarm regime. Rejected for the alarm scores because it inflates TN by the number of windows per opportunity (SOP/cadence), pushing specificity to ≈ 1.000 for any system and making it uninformative. It is, however, still available — it is detection.evaluate’s specificity, which is why the package keeps both regimes side by side rather than forcing one number.

Per-hour TN (one TN per quiet interictal hour). Equivalent to this decision with the opportunity length fixed at 3600 s instead of SOP. Rejected as the default because the opportunity length would then be divorced from the alarm validity window: a 10-minute SOP and a 30-minute SOP would score identical specificity, hiding the very knob the policy exists to pin. (A caller who wants per-hour units can recover them by reading n_tn/n_opportunities and rescaling.)

Snyder 2008 / Schelter-Winterhalder 2006 “proportion of time under false warning” (time-based specificity). These report 1 − (time-in-warning / interictal-time) rather than a counted TN cell. Rejected as the cell definition because it does not yield an integer confusion matrix that PPV/NPV/F1 can share — but the package already reports the time-based quantity separately as MetricsReport.time_in_warning_frac, so no information is lost.

No TN at all (report only sensitivity, PPV, FP/hr). The honest minimal-assumption option, and what the package did before v0.2.0. Rejected because reviewers and cross-paper comparison routinely require specificity/NPV, and refusing to compute them pushes every user to re-implement the convention ad hoc — the exact fragmentation this package exists to prevent. Reporting them with a documented, visible denominator is more useful than omitting them.

Consequences

Positive.

  • specificity/NPV are now available in the alarm regime, so a method can be placed on any paper’s full confusion-matrix axis without re-running its pipeline.

  • The TN time unit (SOP) matches the alarm validity window and the FP/hr interictal denominator, so all alarm scores share one coherent time basis.

  • ppv and forecasting_f1 are convention-independent and directly comparable across studies.

  • Fail-loud NaN on empty denominators means a degenerate run (no seizures, no interictal time) never silently reports a misleading 0.

Negative / tradeoffs.

  • specificity and NPV scale with the chosen opportunity length (SOP). Two studies with different SOP are not directly comparable on specificity/NPV unless the SOP is held fixed. This is mitigated, not removed, by always reporting n_tn / n_opportunities: the reader can see the denominator and rescale. It is the unavoidable cost of giving an alarm system a TN cell at all.

  • Because TN dwarfs the other cells on a long interictal recording, specificity is almost always very high and NPV very high. They are therefore weak discriminators between methods — ppv, sensitivity, forecasting_f1, IoC and FP/hr remain the load-bearing scores. The confusion-matrix scores are provided for completeness and paper-axis matching, not as the headline metric.

  • TN = max(0, n_opportunities − FP) clips at 0: a system that fires more false alarms than there are opportunities (pathological / mis-policed) reports TN = 0 rather than a negative count. This is correct but means specificity = 0 in that regime rather than a more granular penalty.

Implementation

Element

Code

TN / opportunity definition

_classification.py::alarm_classification (module docstring carries the full convention)

Matching rule (TP/FP source)

_alarm.py::alarm_match

Interictal denominator

_alarm.py::interictal_seconds (reused for both FP/hr and the TN partition in forecasting.py::evaluate)

Fail-loud NaN

_classification.py::_safe_ratio

Reported denominators

report.py::MetricsReport.n_tn / n_opportunities

Wiring into reports

forecasting.py::evaluate (flows through evaluate_stream / sweep_thresholds / sweep_policies / to_dict / to_frame)

References

  • Mormann F et al. (2007). Seizure prediction: the long and winding road. Brain 130:314. doi:10.1093/brain/awl241 — false-prediction-rate / proportion-of-time conventions.

  • Snyder DE et al. (2008). The statistics of a practical seizure warning system. J Neural Eng 5:392.

  • Winterhalder M, Schelter B et al. (2003/2006). The seizure prediction characteristic. Epilepsy Behav — SPH/SOP validity-window framework.

  • Schulze-Bonhage A et al. (2020). Performance metrics for online seizure prediction. PMC7340210.

  • Andrade I, Teixeira C, Pinto M (2024). Sample- and alarm-based perspectives. Front Neurosci. doi:10.3389/fnins.2024.1417748.

  • Code: src/scitex_seizure_metrics/_classification.py, src/scitex_seizure_metrics/forecasting.py.

  • Conceptual docs page: docs/sphinx/math/alarm_confusion_matrix.md.