System Architecture

EPF is a cloud-hosted forecasting system that collects market data daily, generates machine learning price forecasts, and serves them through an API and interactive dashboard.

System Overview

Layer	Capability
Data Collection	Spanish grid data, weather forecasts, and commodity prices refreshed daily
Feature Engineering	150+ predictive features built per forecast horizon
Forecasting	LSTM-XGBoost hybrid (day-ahead); multi-model ensemble (strategic)
API	RESTful access to forecasts, actuals, and evaluation metrics
Dashboard	Interactive price charts, heatmaps, and model evaluation tools

Data Flow

Data Sources                Processing              Output
─────────────              ──────────              ──────

REE/ESIOS API ──┐
                ├── Data Collection ── Feature Engineering ── ML Models ──── API
Weather API ────┤       (daily)         (150+ features)    (LSTM+XGBoost) /api/v1/
                │
Commodity ──────┘
  (TTF, ETS)
                                                               │
                                                               ▼
                                                     ┌─────────────────┐
                                                     │  React Frontend │
                                                     │   (Dashboard)   │
                                                     └─────────────────┘

Collection Layer

REE/ESIOS: 16 electricity indicators (demand, generation, prices, interconnections) via the Spanish grid operator API
Open-Meteo: Weather forecasts from 5 population-weighted stations across Spain
Commodities: TTF natural gas (ICE) and ETS carbon prices (EMBER) with fallback chains

Processing Layer

Feature Engineering: Builds 90 tabular features including price lags, cyclical time encodings, rolling statistics, renewable mix ratios, weather interactions, and commodity dynamics
Training: Per-horizon-group models trained with walk-forward cross-validation and Optuna hyperparameter optimization
Prediction (Day-Ahead): A task-aligned LSTM encoder processes 7-day price sequences into 64-dimensional temporal embeddings, augmenting XGBoost’s tabular features (154 total). This v10.1 architecture eliminates the -12 EUR/MWh bias of the legacy ensemble (bias now -0.65)
Prediction (Strategic): Gradient boosting ensemble (HistGBT, LightGBM, XGBoost) with conformal calibration for confidence intervals

Serving Layer

API: RESTful access with 30+ endpoints covering prices, forecasts, indicators, evaluation, and model info
Data Store: All data stored in a single database — predictions, actuals, indicators, model metadata
Dashboard: Real-time visualization with interactive charts, heatmaps, and evaluation tools

Daily Operations

Two automated prediction runs per day, aligned with OMIE market publication:

Run	Time	Scope	Output
Day-Ahead	~10:00 UTC	D+1 (next 24h)	96 quarter-hour prices
Strategic	~15:00 UTC	D+2 to D+7 (6 days)	576 quarter-hour prices

Each run executes: data collection, feature engineering, prediction, naive benchmark generation, and database storage.