System Architecture & Tech Stack
ingest → store → ML → serveProduction ML on GCP · ES/PT/FR/DE day-ahead, hourly + 15-min, 7 days out
How it works
The system spans two environments and two PostgreSQL databases: a local one where models are built, and a production one (on the VM) where they run. Four processes operate across them at different cadences — separating them is the key to understanding the architecture:
Collectors pull prices, weather, generation, commodities and news on cron (hourly down to every 15 min) and land them in PostgreSQL. This never stops — it is the always-on supply of fresh features.
Runs locally, only when we run an experiment (the 1 GB VM cannot train). XGBoost trains per country on the local dev database’s history; the model artifact (a joblib) is uploaded to Cloud Storage. Manual and occasional — not part of the daily loop (dashed).
Every day Cloud Scheduler fires 8 Cloud Run jobs (4 countries × day-ahead + strategic). Each downloads its country model from Cloud Storage, reads the latest features from the production PostgreSQL, and writes the forecasts back to it.
When a visitor opens the dashboard, FastAPI reads the stored forecasts, Cloudflare caches them at the edge, and the React app renders them. No model runs here — it just serves what the daily jobs already computed.
The bridge between the two worlds is the Model Registry (Cloud Storage): training builds a model from the local database and ships it there; the daily jobs load it and write forecasts into the production database, which is all the dashboard ever reads. The model never runs when you open the page — you see what the scheduled jobs already computed.