ADR-005: KL Divergence Threshold Calibration Method¶

Status: Accepted Date: 2025-12-26 Issue: #1 - GAP-DriftThreshold Deciders: Risk Team (Analytics), Data Science Related Gap: GAP-H1 (Drift Threshold Calibration)

Context¶

The Guardrail Framework uses KL divergence to detect distribution drift between the frozen baseline (training data distribution) and live production data. The current specification defines a provisional threshold τ = 0.5, but this value requires calibration based on observed data variability.

The Problem: - A threshold that is too sensitive (low τ) causes alert fatigue from false positives - A threshold that is too lenient (high τ) misses actual drift until model performance degrades - The provisional τ = 0.5 was chosen as a balanced starting point but lacks empirical grounding

Constraints: - Threshold must be calibrated before production launch (β-to-Prod milestone) - Calibration requires understanding of normal data variability - Must support multi-tier alerting (warning vs. critical)

Decision Drivers¶

1. Minimize false positives - Alert fatigue undermines trust in the system
2. Detect actual drift promptly - Before model performance degrades
3. Use statistically grounded methodology - Reproducible and auditable
4. Support tiered response - Different severity levels for different drift magnitudes
5. Enable recalibration - Threshold should be adjustable as baseline changes

Considered Options¶

Option 1: Static Fixed Threshold¶

Description: Keep τ = 0.5 as a fixed constant.

Verdict: Rejected - Does not account for actual data characteristics.

Option 2: Mean + k×σ Formula¶

Description: Set τ = μ + k×σ where μ and σ are computed from observed KL values, and k controls false positive rate.

Verdict: Considered - Good baseline method but may not handle heavy-tailed distributions.

Option 3: Percentile-Based Calibration (Selected)¶

Description: Set thresholds based on percentiles of observed KL values: - τ_warn = 90th percentile (warning level) - τ_crit = 99th percentile (critical level)

Verdict: Selected - Best balance of robustness and interpretability.

Option 4: Dynamic Adaptive Threshold¶

Description: Continuously adjust threshold based on model performance feedback loop.

Verdict: Deferred - Recommended for future enhancement after baseline approach proven.

Decision¶

We will use percentile-based calibration with two-tier thresholds:

τ_warn = P90(observed KL values)  → Warning threshold
τ_crit = P99(observed KL values)  → Critical threshold

Threshold Tiers¶

Calibration Procedure¶

1. Data Collection: Collect 30+ days of KL divergence values from shadow scoring
2. Validation: Ensure <5% missing values, no systematic gaps
3. Computation: Calculate P90 and P99 percentiles
4. Back-testing: Verify alert rate is acceptable (~10% warning, ~1% critical)
5. Documentation: Record rationale and statistical summary
6. Deployment: Update parameter freeze with new values

Expected Outcomes¶

Based on industry research (Evidently AI, Arthur.ai, Deepchecks):

Consequences¶

Positive¶

• Reduced false positives: Thresholds adapt to actual data variability
• Clear escalation path: Two-tier system prevents alert fatigue
• Auditable: Percentile-based method is transparent and reproducible
• Actionable alerts: Critical alerts are rare enough to demand attention

Negative¶

• Requires recalibration: When baseline distribution changes, thresholds must be recomputed
• Data dependency: Need 30+ days of representative data before production calibration
• Percentile choice: 90th/99th percentiles are reasonable defaults but may need adjustment

Risks¶

Validation¶

Pre-Deployment Validation¶

• [ ] Calibration script produces correct percentiles on test data
• [ ] Back-test shows acceptable alert rates
• [ ] τ_crit validated against provisional τ = 0.5
• [ ] Statistical rationale documented

Post-Deployment Validation¶

• [ ] False positive rate < 5% after 7 days
• [ ] No missed drift events (zero false negatives on known issues)
• [ ] Stakeholder feedback on alert actionability

Implementation¶

• Source: src/engine/drift.py - KL divergence computation
• Shadow mode: src/integration/pcw_decide.py - shadow_mode=True collects KL divergence observations from live proposals without enforcing decisions (Phase 1 complete)
• Tests: tests/integration/test_shadow_mode.py (44 tests), tests/test_engine.py (drift tests)
• Metrics: src/telemetry/prometheus_exporter.py - drift metrics, shadow evaluation counter

Related Documents¶

• Implementation Plan: /docs/implementation-plans/001-drift-threshold-calibration.md
• Interface Contract: schema/interface-contract.yaml §frozen_parameters
• Specification: spec/guardrails/Hardened_Quantitative_Guardrail_Framework_Specification.md §2.5

Changelog¶