Model Drift Monitoring

What Is Model Drift?

Model drift occurs when forecast accuracy degrades over time because the relationship between features and prices has changed. This can happen due to:

• Market structure changes: New regulations, plant closures, capacity additions
• Seasonal shifts: Unusual weather patterns, demand changes
• Data source changes: API schema updates, new indicator versions
• Feature degradation: A key feature losing predictive power

Detection Method

The monitoring system compares recent forecast accuracy against a baseline MAE established during backtesting:

drift_detected = (recent_mae > baseline_mae × 1.5)

The 1.5× threshold means drift is flagged when the rolling MAE exceeds 150% of the baseline. This allows normal performance fluctuation while catching genuine degradation.

Monitoring Windows

Recent vs Baseline Comparison

Baseline MAE:  Established during walk-forward backtest (e.g., 3.2 EUR/MWh)
Recent MAE:    Rolling 7-day MAE from live predictions

If recent_mae > 3.2 × 1.5 = 4.8 EUR/MWh → drift detected

Trend Detection

The system also checks for increasing error trends by comparing sub-windows:

recent_3d_mae = MAE over last 3 days
earlier_3d_mae = MAE over days 4-7

If recent_3d_mae > earlier_3d_mae × 1.3 → MAE trend: "increasing"

A 30% increase in the most recent 3-day window versus the earlier 3-day window suggests accelerating degradation.

Monitoring Dimensions

Drift can be global or localized to specific conditions:

Dimension	What to Watch
Overall MAE	Global performance degradation
Per horizon	D+1 accuracy stable but D+5 degraded → longer horizons affected first
Per hour	Hour 13–15 drift → solar generation feature issues
Per weekday	Monday drift → weekend-to-weekday transition issues
Per model	LightGBM drifting but HistGBM stable → model-specific issue

Response Actions

When drift is detected:

Level 1: Monitor (1.3×–1.5× baseline)

Action: Log warning, increase monitoring frequency
Status: "performance_warning"

Performance is degrading but still within operational limits. Continue serving predictions while investigating.

Level 2: Alert (≥ 1.5× baseline)

Action: Flag drift, recommend retraining
Status: "drift_detected"

Accuracy has degraded significantly. Retrain models with recent data to adapt to the new market regime.

Level 3: Critical (≥ 2.0× baseline)

Action: Consider model suspension
Status: "critical_degradation"

Forecasts may be less reliable than naive baselines. Investigate root cause (data issue? market shock?) before continuing to serve predictions.

Common Drift Patterns

Sudden Jump

Day 1-10: MAE = 3.5 EUR/MWh
Day 11:   MAE = 8.2 EUR/MWh  ← abrupt change

Usually indicates a data issue (API change, missing indicator) or a market event (plant outage, regulatory change). Check data pipeline logs first.

Gradual Increase

Month 1: MAE = 3.2 EUR/MWh
Month 2: MAE = 3.5 EUR/MWh
Month 3: MAE = 4.1 EUR/MWh
Month 4: MAE = 4.8 EUR/MWh  ← crosses 1.5× threshold

Typical of slowly changing market structure. Regular monthly retraining usually prevents this.

Seasonal Pattern

Summer: MAE = 3.0 EUR/MWh
Winter: MAE = 4.5 EUR/MWh (back every year)

Not true drift — the model naturally performs differently across seasons. The baseline should account for seasonal variation by including a full year of backtest data.

API Access

Monitoring data is available through the evaluation endpoints:

GET /api/v1/evaluation/summary?context=live

Returns current MAE, RMSE, comparison against baselines, and drift status indicators. The dashboard’s evaluation page visualizes these metrics with trend charts.