-
Far beyond day-ahead with econometric models for electricity price forecasting
Authors:
Paul Ghelasi,
Florian Ziel
Abstract:
The surge in global energy prices during the recent energy crisis, which peaked in 2022, has intensified the need for mid-term to long-term forecasting for hedging and valuation purposes. This study analyzes the statistical predictability of power prices before, during, and after the energy crisis, using econometric models with an hourly resolution. To stabilize the model estimates, we define fund…
▽ More
The surge in global energy prices during the recent energy crisis, which peaked in 2022, has intensified the need for mid-term to long-term forecasting for hedging and valuation purposes. This study analyzes the statistical predictability of power prices before, during, and after the energy crisis, using econometric models with an hourly resolution. To stabilize the model estimates, we define fundamentally derived coefficient bounds. We provide an in-depth analysis of the unit root behavior of the power price series, showing that the long-term stochastic trend is explained by the prices of commodities used as fuels for power generation: gas, coal, oil, and emission allowances (EUA). However, as the forecasting horizon increases, spurious effects become extremely relevant, leading to highly significant but economically meaningless results. To mitigate these spurious effects, we propose the "current" model: estimating the current same-day relationship between power prices and their regressors and projecting this relationship into the future. This flexible and interpretable method is applied to hourly German day-ahead power prices for forecasting horizons up to one year ahead, utilizing a combination of regularized regression methods and generalized additive models.
△ Less
Submitted 1 June, 2024;
originally announced June 2024.
-
Efficient mid-term forecasting of hourly electricity load using generalized additive models
Authors:
Monika Zimmermann,
Florian Ziel
Abstract:
Accurate mid-term (weeks to one year) hourly electricity load forecasts are essential for strategic decision-making in power plant operation, ensuring supply security and grid stability, and energy trading. While numerous models effectively predict short-term (hours to a few days) hourly load, mid-term forecasting solutions remain scarce. In mid-term load forecasting, besides daily, weekly, and an…
▽ More
Accurate mid-term (weeks to one year) hourly electricity load forecasts are essential for strategic decision-making in power plant operation, ensuring supply security and grid stability, and energy trading. While numerous models effectively predict short-term (hours to a few days) hourly load, mid-term forecasting solutions remain scarce. In mid-term load forecasting, besides daily, weekly, and annual seasonal and autoregressive effects, capturing weather and holiday effects, as well as socio-economic non-stationarities in the data, poses significant modeling challenges. To address these challenges, we propose a novel forecasting method using Generalized Additive Models (GAMs) built from interpretable P-splines and enhanced with autoregressive post-processing. This model uses smoothed temperatures, Error-Trend-Seasonal (ETS) modeled non-stationary states, a nuanced representation of holiday effects with weekday variations, and seasonal information as input. The proposed model is evaluated on load data from 24 European countries. This analysis demonstrates that the model not only has significantly enhanced forecasting accuracy compared to state-of-the-art methods but also offers valuable insights into the influence of individual components on predicted load, given its full interpretability. Achieving performance akin to day-ahead TSO forecasts in fast computation times of a few seconds for several years of hourly data underscores the model's potential for practical application in the power system industry.
△ Less
Submitted 27 May, 2024;
originally announced May 2024.
-
Multivariate Simulation-based Forecasting for Intraday Power Markets: Modelling Cross-Product Price Effects
Authors:
Simon Hirsch,
Florian Ziel
Abstract:
Intraday electricity markets play an increasingly important role in balancing the intermittent generation of renewable energy resources, which creates a need for accurate probabilistic price forecasts. However, research to date has focused on univariate approaches, while in many European intraday electricity markets all delivery periods are traded in parallel. Thus, the dependency structure betwee…
▽ More
Intraday electricity markets play an increasingly important role in balancing the intermittent generation of renewable energy resources, which creates a need for accurate probabilistic price forecasts. However, research to date has focused on univariate approaches, while in many European intraday electricity markets all delivery periods are traded in parallel. Thus, the dependency structure between different traded products and the corresponding cross-product effects cannot be ignored. We aim to fill this gap in the literature by using copulas to model the high-dimensional intraday price return vector. We model the marginal distribution as a zero-inflated Johnson's $S_U$ distribution with location, scale and shape parameters that depend on market and fundamental data. The dependence structure is modelled using latent beta regression to account for the particular market structure of the intraday electricity market, such as overlap** but independent trading sessions for different delivery days. We allow the dependence parameter to be time-varying. We validate our approach in a simulation study for the German intraday electricity market and find that modelling the dependence structure improves the forecasting performance. Additionally, we shed light on the impact of the single intraday coupling (SIDC) on the trading activity and price distribution and interpret our results in light of the market efficiency hypothesis. The approach is directly applicable to other European electricity markets.
△ Less
Submitted 23 June, 2023;
originally announced June 2023.
-
Hierarchical forecasting for aggregated curves with an application to day-ahead electricity price auctions
Authors:
Paul Ghelasi,
Florian Ziel
Abstract:
Aggregated curves are common structures in economics and finance, and the most prominent examples are supply and demand curves. In this study, we exploit the fact that all aggregated curves have an intrinsic hierarchical structure, and thus hierarchical reconciliation methods can be used to improve the forecast accuracy. We provide an in-depth theory on how aggregated curves can be constructed or…
▽ More
Aggregated curves are common structures in economics and finance, and the most prominent examples are supply and demand curves. In this study, we exploit the fact that all aggregated curves have an intrinsic hierarchical structure, and thus hierarchical reconciliation methods can be used to improve the forecast accuracy. We provide an in-depth theory on how aggregated curves can be constructed or deconstructed, and conclude that these methods are equivalent under weak assumptions. We consider multiple reconciliation methods for aggregated curves, including previously established bottom-up, top-down, and linear optimal reconciliation approaches. We also present a new benchmark reconciliation method called 'aggregated-down' with similar complexity to bottom-up and top-down approaches, but it tends to provide better accuracy in this setup. We conducted an empirical forecasting study on the German day-ahead power auction market by predicting the demand and supply curves, where their equilibrium determines the electricity price for the next day. Our results demonstrate that hierarchical reconciliation methods can be used to improve the forecasting accuracy of aggregated curves.
△ Less
Submitted 25 May, 2023;
originally announced May 2023.
-
Multivariate Probabilistic CRPS Learning with an Application to Day-Ahead Electricity Prices
Authors:
Jonathan Berrisch,
Florian Ziel
Abstract:
This paper presents a new method for combining (or aggregating or ensembling) multivariate probabilistic forecasts, considering dependencies between quantiles and marginals through a smoothing procedure that allows for online learning. We discuss two smoothing methods: dimensionality reduction using Basis matrices and penalized smoothing. The new online learning algorithm generalizes the standard…
▽ More
This paper presents a new method for combining (or aggregating or ensembling) multivariate probabilistic forecasts, considering dependencies between quantiles and marginals through a smoothing procedure that allows for online learning. We discuss two smoothing methods: dimensionality reduction using Basis matrices and penalized smoothing. The new online learning algorithm generalizes the standard CRPS learning framework into multivariate dimensions. It is based on Bernstein Online Aggregation (BOA) and yields optimal asymptotic learning properties. The procedure uses horizontal aggregation, i.e., aggregation across quantiles. We provide an in-depth discussion on possible extensions of the algorithm and several nested cases related to the existing literature on online forecast combination. We apply the proposed methodology to forecasting day-ahead electricity prices, which are 24-dimensional distributional forecasts. The proposed method yields significant improvements over uniform combination in terms of continuous ranked probability score (CRPS). We discuss the temporal evolution of the weights and hyperparameters and present the results of reduced versions of the preferred model. A fast C++ implementation of the proposed algorithm is provided in the open-source R-Package profoc on CRAN.
△ Less
Submitted 6 February, 2024; v1 submitted 17 March, 2023;
originally announced March 2023.
-
Simulation-based Forecasting for Intraday Power Markets: Modelling Fundamental Drivers for Location, Shape and Scale of the Price Distribution
Authors:
Simon Hirsch,
Florian Ziel
Abstract:
During the last years, European intraday power markets have gained importance for balancing forecast errors due to the rising volumes of intermittent renewable generation. However, compared to day-ahead markets, the drivers for the intraday price process are still sparsely researched. In this paper, we propose a modelling strategy for the location, shape and scale parameters of the return distribu…
▽ More
During the last years, European intraday power markets have gained importance for balancing forecast errors due to the rising volumes of intermittent renewable generation. However, compared to day-ahead markets, the drivers for the intraday price process are still sparsely researched. In this paper, we propose a modelling strategy for the location, shape and scale parameters of the return distribution in intraday markets, based on fundamental variables. We consider wind and solar forecasts and their intraday updates, outages, price information and a novel measure for the shape of the merit-order, derived from spot auction curves as explanatory variables. We validate our modelling by simulating price paths and compare the probabilistic forecasting performance of our model to benchmark models in a forecasting study for the German market. The approach yields significant improvements in the forecasting performance, especially in the tails of the distribution. At the same time, we are able to derive the contribution of the driving variables. We find that, apart from the first lag of the price changes, none of our fundamental variables have explanatory power for the expected value of the intraday returns. This implies weak-form market efficiency as renewable forecast changes and outage information seems to be priced in by the market. We find that the volatility is driven by the merit-order regime, the time to delivery and the closure of cross-border order books. The tail of the distribution is mainly influenced by past price differences and trading activity. Our approach is directly transferable to other continuous intraday markets in Europe.
△ Less
Submitted 23 November, 2022;
originally announced November 2022.
-
Modeling Volatility and Dependence of European Carbon and Energy Prices
Authors:
Jonathan Berrisch,
Sven Pappert,
Florian Ziel,
Antonia Arsova
Abstract:
We study the prices of European Emission Allowances (EUA), whereby we analyze their uncertainty and dependencies on related energy prices (natural gas, coal, and oil). We propose a probabilistic multivariate conditional time series model with a VECM-Copula-GARCH structure which exploits key characteristics of the data. Data are normalized with respect to inflation and carbon emissions to allow for…
▽ More
We study the prices of European Emission Allowances (EUA), whereby we analyze their uncertainty and dependencies on related energy prices (natural gas, coal, and oil). We propose a probabilistic multivariate conditional time series model with a VECM-Copula-GARCH structure which exploits key characteristics of the data. Data are normalized with respect to inflation and carbon emissions to allow for proper cross-series evaluation. The forecasting performance is evaluated in an extensive rolling-window forecasting study, covering eight years out-of-sample. We discuss our findings for both levels- and log-transformed data, focusing on time-varying correlations, and in view of the Russian invasion of Ukraine.
△ Less
Submitted 10 February, 2023; v1 submitted 30 August, 2022;
originally announced August 2022.
-
Distributional neural networks for electricity price forecasting
Authors:
Grzegorz Marcjasz,
Michał Narajewski,
Rafał Weron,
Florian Ziel
Abstract:
We present a novel approach to probabilistic electricity price forecasting which utilizes distributional neural networks. The model structure is based on a deep neural network that contains a so-called probability layer. The network's output is a parametric distribution with 2 (normal) or 4 (Johnson's SU) parameters. In a forecasting study involving day-ahead electricity prices in the German marke…
▽ More
We present a novel approach to probabilistic electricity price forecasting which utilizes distributional neural networks. The model structure is based on a deep neural network that contains a so-called probability layer. The network's output is a parametric distribution with 2 (normal) or 4 (Johnson's SU) parameters. In a forecasting study involving day-ahead electricity prices in the German market, our approach significantly outperforms state-of-the-art benchmarks, including LASSO-estimated regressions and deep neural networks combined with Quantile Regression Averaging. The obtained results not only emphasize the importance of higher moments when modeling volatile electricity prices, but also -- given that probabilistic forecasting is the essence of risk management -- provide important implications for managing portfolios in the power sector.
△ Less
Submitted 10 December, 2022; v1 submitted 6 July, 2022;
originally announced July 2022.
-
High-Resolution Peak Demand Estimation Using Generalized Additive Models and Deep Neural Networks
Authors:
Jonathan Berrisch,
Michał Narajewski,
Florian Ziel
Abstract:
This paper covers predicting high-resolution electricity peak demand features given lower-resolution data. This is a relevant setup as it answers whether limited higher-resolution monitoring helps to estimate future high-resolution peak loads when the high-resolution data is no longer available. That question is particularly interesting for network operators considering replacing high-resolution m…
▽ More
This paper covers predicting high-resolution electricity peak demand features given lower-resolution data. This is a relevant setup as it answers whether limited higher-resolution monitoring helps to estimate future high-resolution peak loads when the high-resolution data is no longer available. That question is particularly interesting for network operators considering replacing high-resolution monitoring predictive models due to economic considerations. We propose models to predict half-hourly minima and maxima of high-resolution (every minute) electricity load data while model inputs are of a lower resolution (30 minutes). We combine predictions of generalized additive models (GAM) and deep artificial neural networks (DNN), which are popular in load forecasting. We extensively analyze the prediction models, including the input parameters' importance, focusing on load, weather, and seasonal effects. The proposed method won a data competition organized by Western Power Distribution, a British distribution network operator. In addition, we provide a rigorous evaluation study that goes beyond the competition frame to analyze the models' robustness. The results show that the proposed methods are superior to the competition benchmark concerning the out-of-sample root mean squared error (RMSE). This holds regarding the competition month and the supplementary evaluation study, which covers an additional eleven months. Overall, our proposed model combination reduces the out-of-sample RMSE by 57.4\% compared to the benchmark.
△ Less
Submitted 2 November, 2022; v1 submitted 7 March, 2022;
originally announced March 2022.
-
M5 Competition Uncertainty: Overdispersion, distributional forecasting, GAMLSS and beyond
Authors:
Florian Ziel
Abstract:
The M5 competition uncertainty track aims for probabilistic forecasting of sales of thousands of Walmart retail goods. We show that the M5 competition data faces strong overdispersion and sporadic demand, especially zero demand. We discuss resulting modeling issues concerning adequate probabilistic forecasting of such count data processes. Unfortunately, the majority of popular prediction methods…
▽ More
The M5 competition uncertainty track aims for probabilistic forecasting of sales of thousands of Walmart retail goods. We show that the M5 competition data faces strong overdispersion and sporadic demand, especially zero demand. We discuss resulting modeling issues concerning adequate probabilistic forecasting of such count data processes. Unfortunately, the majority of popular prediction methods used in the M5 competition (e.g. lightgbm and xgboost GBMs) fails to address the data characteristics due to the considered objective functions. The distributional forecasting provides a suitable modeling approach for to the overcome those problems. The GAMLSS framework allows flexible probabilistic forecasting using low dimensional distributions. We illustrate, how the GAMLSS approach can be applied for the M5 competition data by modeling the location and scale parameter of various distributions, e.g. the negative binomial distribution. Finally, we discuss software packages for distributional modeling and their drawback, like the R package gamlss with its package extensions, and (deep) distributional forecasting libraries such as TensorFlow Probability.
△ Less
Submitted 10 November, 2021; v1 submitted 14 July, 2021;
originally announced July 2021.
-
Smoothed Bernstein Online Aggregation for Day-Ahead Electricity Demand Forecasting
Authors:
Florian Ziel
Abstract:
We present a winning method of the IEEE DataPort Competition on Day-Ahead Electricity Demand Forecasting: Post-COVID Paradigm. The day-ahead load forecasting approach is based on online forecast combination of multiple point prediction models. It contains four steps: i) data cleaning and preprocessing, ii) a holiday adjustment procedure, iii) training of individual forecasting models, iv) forecast…
▽ More
We present a winning method of the IEEE DataPort Competition on Day-Ahead Electricity Demand Forecasting: Post-COVID Paradigm. The day-ahead load forecasting approach is based on online forecast combination of multiple point prediction models. It contains four steps: i) data cleaning and preprocessing, ii) a holiday adjustment procedure, iii) training of individual forecasting models, iv) forecast combination by smoothed Bernstein Online Aggregation (BOA). The approach is flexible and can quickly adopt to new energy system situations as they occurred during and after COVID-19 shutdowns. The pool of individual prediction models ranges from rather simple time series models to sophisticated models like generalized additive models (GAMs) and high-dimensional linear models estimated by lasso. They incorporate autoregressive, calendar and weather effects efficiently. All steps contain novel concepts that contribute to the excellent forecasting performance of the proposed method. This holds particularly for the holiday adjustment procedure and the fully adaptive smoothed BOA approach.
△ Less
Submitted 13 July, 2021;
originally announced July 2021.
-
Optimal bidding in hourly and quarter-hourly electricity price auctions: trading large volumes of power with market impact and transaction costs
Authors:
Michał Narajewski,
Florian Ziel
Abstract:
This paper addresses the question of how much to bid to maximize the profit when trading in two electricity markets: the hourly Day-Ahead Auction and the quarter-hourly Intraday Auction. For optimal coordinated bidding many price scenarios are examined, the own non-linear market impact is estimated by considering empirical supply and demand curves, and a number of trading strategies is used. Addit…
▽ More
This paper addresses the question of how much to bid to maximize the profit when trading in two electricity markets: the hourly Day-Ahead Auction and the quarter-hourly Intraday Auction. For optimal coordinated bidding many price scenarios are examined, the own non-linear market impact is estimated by considering empirical supply and demand curves, and a number of trading strategies is used. Additionally, we provide theoretical results for risk neutral agents. The application study is conducted using the German market data, but the presented methods can be easily utilized with other two consecutive auctions. This paper contributes to the existing literature by evaluating the costs of electricity trading, i.e. the price impact and the transaction costs. The empirical results for the German EPEX market show that it is far more profitable to minimize the price impact rather than maximize the arbitrage.
△ Less
Submitted 9 February, 2022; v1 submitted 29 April, 2021;
originally announced April 2021.
-
tsrobprep - an R package for robust preprocessing of time series data
Authors:
Michał Narajewski,
Jens Kley-Holsteg,
Florian Ziel
Abstract:
Data cleaning is a crucial part of every data analysis exercise. Yet, the currently available R packages do not provide fast and robust methods for cleaning and preparation of time series data. The open source package tsrobprep introduces efficient methods for handling missing values and outliers using model based approaches. For data imputation a probabilistic replacement model is proposed, which…
▽ More
Data cleaning is a crucial part of every data analysis exercise. Yet, the currently available R packages do not provide fast and robust methods for cleaning and preparation of time series data. The open source package tsrobprep introduces efficient methods for handling missing values and outliers using model based approaches. For data imputation a probabilistic replacement model is proposed, which may consist of autoregressive components and external inputs. For outlier detection a clustering algorithm based on finite mixture modelling is introduced, which considers time series properties in terms of the gradient and the underlying seasonality as features. The procedure allows to return a probability for each observation being outlying data as well as a specific cause for an outlier assignment in terms of the provided feature space. The methods work robust and are fully tunable. Moreover, by providing the auto_data_cleaning function the data preprocessing can be carried out in one cast, without comprehensive tuning and providing suitable results. The primary motivation of the package is the preprocessing of energy system data. We present application for electricity load, wind and solar power data.
△ Less
Submitted 11 October, 2021; v1 submitted 26 April, 2021;
originally announced April 2021.
-
CRPS Learning
Authors:
Jonathan Berrisch,
Florian Ziel
Abstract:
Combination and aggregation techniques can significantly improve forecast accuracy. This also holds for probabilistic forecasting methods where predictive distributions are combined. There are several time-varying and adaptive weighting schemes such as Bayesian model averaging (BMA). However, the quality of different forecasts may vary not only over time but also within the distribution. For examp…
▽ More
Combination and aggregation techniques can significantly improve forecast accuracy. This also holds for probabilistic forecasting methods where predictive distributions are combined. There are several time-varying and adaptive weighting schemes such as Bayesian model averaging (BMA). However, the quality of different forecasts may vary not only over time but also within the distribution. For example, some distribution forecasts may be more accurate in the center of the distributions, while others are better at predicting the tails. Therefore, we introduce a new weighting method that considers the differences in performance over time and within the distribution. We discuss pointwise combination algorithms based on aggregation across quantiles that optimize with respect to the continuous ranked probability score (CRPS). After analyzing the theoretical properties of pointwise CRPS learning, we discuss B- and P-Spline-based estimation techniques for batch and online learning, based on quantile regression and prediction with expert advice. We prove that the proposed fully adaptive Bernstein online aggregation (BOA) method for pointwise CRPS online learning has optimal convergence properties. They are confirmed in simulations and a probabilistic forecasting study for European emission allowance (EUA) prices.
△ Less
Submitted 19 November, 2021; v1 submitted 1 February, 2021;
originally announced February 2021.
-
Forecasting: theory and practice
Authors:
Fotios Petropoulos,
Daniele Apiletti,
Vassilios Assimakopoulos,
Mohamed Zied Babai,
Devon K. Barrow,
Souhaib Ben Taieb,
Christoph Bergmeir,
Ricardo J. Bessa,
Jakub Bijak,
John E. Boylan,
Jethro Browell,
Claudio Carnevale,
Jennifer L. Castle,
Pasquale Cirillo,
Michael P. Clements,
Clara Cordeiro,
Fernando Luiz Cyrino Oliveira,
Shari De Baets,
Alexander Dokumentov,
Joanne Ellison,
Piotr Fiszeder,
Philip Hans Franses,
David T. Frazier,
Michael Gilliland,
M. Sinan Gönül
, et al. (55 additional authors not shown)
Abstract:
Forecasting has always been at the forefront of decision making and planning. The uncertainty that surrounds the future is both exciting and challenging, with individuals and organisations seeking to minimise risks and maximise utilities. The large number of forecasting applications calls for a diverse set of forecasting methods to tackle real-life challenges. This article provides a non-systemati…
▽ More
Forecasting has always been at the forefront of decision making and planning. The uncertainty that surrounds the future is both exciting and challenging, with individuals and organisations seeking to minimise risks and maximise utilities. The large number of forecasting applications calls for a diverse set of forecasting methods to tackle real-life challenges. This article provides a non-systematic review of the theory and the practice of forecasting. We provide an overview of a wide range of theoretical, state-of-the-art models, methods, principles, and approaches to prepare, produce, organise, and evaluate forecasts. We then demonstrate how such theoretical concepts are applied in a variety of real-life contexts.
We do not claim that this review is an exhaustive list of methods and applications. However, we wish that our encyclopedic presentation will offer a point of reference for the rich work that has been undertaken over the last decades, with some key insights for the future of forecasting theory and practice. Given its encyclopedic nature, the intended mode of reading is non-linear. We offer cross-references to allow the readers to navigate through the various topics. We complement the theoretical concepts and applications covered by large lists of free or open-source software implementations and publicly-available databases.
△ Less
Submitted 5 January, 2022; v1 submitted 4 December, 2020;
originally announced December 2020.
-
On cointegration for modeling and forecasting wind power production
Authors:
Florian Ziel,
Antonia Arsova
Abstract:
This study evaluates the performance of cointegrated vector autoregressive (VAR) models for very short- and short-term wind power forecasting. Preliminary results for a German data set comprising six wind power production time series indicate that taking into account potential cointegrating relations between the individual series can improve forecasts at short-term time horizons.
This study evaluates the performance of cointegrated vector autoregressive (VAR) models for very short- and short-term wind power forecasting. Preliminary results for a German data set comprising six wind power production time series indicate that taking into account potential cointegrating relations between the individual series can improve forecasts at short-term time horizons.
△ Less
Submitted 15 October, 2020;
originally announced October 2020.
-
Distributional Modeling and Forecasting of Natural Gas Prices
Authors:
Jonathan Berrisch,
Florian Ziel
Abstract:
We examine the problem of modeling and forecasting European Day-Ahead and Month-Ahead natural gas prices. For this, we propose two distinct probabilistic models that can be utilized in risk- and portfolio management. We use daily pricing data ranging from 2011 to 2020. Extensive descriptive data analysis shows that both time series feature heavy tails, conditional heteroscedasticity, and show asym…
▽ More
We examine the problem of modeling and forecasting European Day-Ahead and Month-Ahead natural gas prices. For this, we propose two distinct probabilistic models that can be utilized in risk- and portfolio management. We use daily pricing data ranging from 2011 to 2020. Extensive descriptive data analysis shows that both time series feature heavy tails, conditional heteroscedasticity, and show asymmetric behavior in their differences. We propose state-space time series models under skewed, heavy-tailed distributions to capture all stylized facts of the data. They include the impact of autocorrelation, seasonality, risk premia, temperature, storage levels, the price of European Emission Allowances, and related fuel prices of oil, coal, and electricity. We provide rigorous model diagnostics and interpret all model components in detail. Additionally, we conduct a probabilistic forecasting study with significance tests and compare the predictive performance against literature benchmarks. The proposed Day-Ahead (Month-Ahead) model leads to a 13% (9%) reduction in out-of-sample continuous ranked probability score (CRPS) compared to the best performing benchmark model, mainly due to adequate modeling of the volatility and heavy tails.
△ Less
Submitted 6 August, 2021; v1 submitted 13 October, 2020;
originally announced October 2020.
-
The energy distance for ensemble and scenario reduction
Authors:
Florian Ziel
Abstract:
Scenario reduction techniques are widely applied for solving sophisticated dynamic and stochastic programs, especially in energy and power systems, but also used in probabilistic forecasting, clustering and estimating generative adversarial networks (GANs). We propose a new method for ensemble and scenario reduction based on the energy distance which is a special case of the maximum mean discrepan…
▽ More
Scenario reduction techniques are widely applied for solving sophisticated dynamic and stochastic programs, especially in energy and power systems, but also used in probabilistic forecasting, clustering and estimating generative adversarial networks (GANs). We propose a new method for ensemble and scenario reduction based on the energy distance which is a special case of the maximum mean discrepancy (MMD). We discuss the choice of energy distance in detail, especially in comparison to the popular Wasserstein distance which is dominating the scenario reduction literature. The energy distance is a metric between probability measures that allows for powerful tests for equality of arbitrary multivariate distributions or independence. Thanks to the latter, it is a suitable candidate for ensemble and scenario reduction problems. The theoretical properties and considered examples indicate clearly that the reduced scenario sets tend to exhibit better statistical properties for the energy distance than a corresponding reduction with respect to the Wasserstein distance. We show applications to a Bernoulli random walk and two real data based examples for electricity demand profiles and day-ahead electricity prices.
△ Less
Submitted 3 October, 2020; v1 submitted 29 May, 2020;
originally announced May 2020.
-
Probabilistic Multi-Step-Ahead Short-Term Water Demand Forecasting with Lasso
Authors:
Jens Kley-Holsteg,
Florian Ziel
Abstract:
Water demand is a highly important variable for operational control and decision making. Hence, the development of accurate forecasts is a valuable field of research to further improve the efficiency of water utilities. Focusing on probabilistic multi-step-ahead forecasting, a time series model is introduced, to capture typical autoregressive, calendar and seasonal effects, to account for time-var…
▽ More
Water demand is a highly important variable for operational control and decision making. Hence, the development of accurate forecasts is a valuable field of research to further improve the efficiency of water utilities. Focusing on probabilistic multi-step-ahead forecasting, a time series model is introduced, to capture typical autoregressive, calendar and seasonal effects, to account for time-varying variance, and to quantify the uncertainty and path-dependency of the water demand process. To deal with the high complexity of the water demand process a high-dimensional feature space is applied, which is efficiently tuned by an automatic shrinkage and selection operator (lasso). It allows to obtain an accurate, simple interpretable and fast computable forecasting model, which is well suited for real-time applications. The complete probabilistic forecasting framework allows not only for simulating the mean and the marginal properties, but also the correlation structure between hours within the forecasting horizon. For practitioners, complete probabilistic multi-step-ahead forecasts are of considerable relevance as they provide additional information about the expected aggregated or cumulative water demand, so that a statement can be made about the probability with which a water storage capacity can guarantee the supply over a certain period of time. This information allows to better control storage capacities and to better ensure the smooth operation of pumps. To appropriately evaluate the forecasting performance of the considered models, the energy score (ES) as a strictly proper multidimensional evaluation criterion, is introduced. The methodology is applied to the hourly water demand data of a German water supplier.
△ Less
Submitted 9 May, 2020;
originally announced May 2020.
-
Ensemble Forecasting for Intraday Electricity Prices: Simulating Trajectories
Authors:
Michał Narajewski,
Florian Ziel
Abstract:
Recent studies concerning the point electricity price forecasting have shown evidence that the hourly German Intraday Continuous Market is weak-form efficient. Therefore, we take a novel, advanced approach to the problem. A probabilistic forecasting of the hourly intraday electricity prices is performed by simulating trajectories in every trading window to receive a realistic ensemble to allow for…
▽ More
Recent studies concerning the point electricity price forecasting have shown evidence that the hourly German Intraday Continuous Market is weak-form efficient. Therefore, we take a novel, advanced approach to the problem. A probabilistic forecasting of the hourly intraday electricity prices is performed by simulating trajectories in every trading window to receive a realistic ensemble to allow for more efficient intraday trading and redispatch. A generalized additive model is fitted to the price differences with the assumption that they follow a zero-inflated distribution, precisely a mixture of the Dirac and the Student's t-distributions. Moreover, the mixing term is estimated using a high-dimensional logistic regression with lasso penalty. We model the expected value and volatility of the series using i.a. autoregressive and no-trade effects or load, wind and solar generation forecasts and accounting for the non-linearities in e.g. time to maturity. Both the in-sample characteristics and forecasting performance are analysed using a rolling window forecasting study. Multiple versions of the model are compared to several benchmark models and evaluated using probabilistic forecasting measures and significance tests. The study aims to forecast the price distribution in the German Intraday Continuous Market in the last 3 hours of trading, but the approach allows for application to other continuous markets, especially in Europe. The results prove superiority of the mixture model over the benchmarks gaining the most from the modelling of the volatility. They also indicate that the introduction of XBID reduced the market volatility.
△ Less
Submitted 29 August, 2020; v1 submitted 4 May, 2020;
originally announced May 2020.
-
Changes in electricity demand pattern in Europe due to COVID-19 shutdowns
Authors:
Michał Narajewski,
Florian Ziel
Abstract:
The article covers electricity demand shift effects due to COVID-19 shutdowns in various European countries. We utilize high-dimensional regression techniques to exploit the structural breaks in demand profiles due to the shutdowns. We discuss the findings with respect to coronavirus pandemic progress and regulatory measures of the considered countries.
The article covers electricity demand shift effects due to COVID-19 shutdowns in various European countries. We utilize high-dimensional regression techniques to exploit the structural breaks in demand profiles due to the shutdowns. We discuss the findings with respect to coronavirus pandemic progress and regulatory measures of the considered countries.
△ Less
Submitted 5 May, 2020; v1 submitted 29 April, 2020;
originally announced April 2020.
-
Multivariate Forecasting Evaluation: On Sensitive and Strictly Proper Scoring Rules
Authors:
Florian Ziel,
Kevin Berk
Abstract:
In recent years, probabilistic forecasting is an emerging topic, which is why there is a growing need of suitable methods for the evaluation of multivariate predictions. We analyze the sensitivity of the most common scoring rules, especially regarding quality of the forecasted dependency structures. Additionally, we propose scoring rules based on the copula, which uniquely describes the dependency…
▽ More
In recent years, probabilistic forecasting is an emerging topic, which is why there is a growing need of suitable methods for the evaluation of multivariate predictions. We analyze the sensitivity of the most common scoring rules, especially regarding quality of the forecasted dependency structures. Additionally, we propose scoring rules based on the copula, which uniquely describes the dependency structure for every probability distribution with continuous marginal distributions. Efficient estimation of the considered scoring rules and evaluation methods such as the Diebold-Mariano test are discussed. In detailed simulation studies, we compare the performance of the renowned scoring rules and the ones we propose. Besides extended synthetic studies based on recently published results we also consider a real data example. We find that the energy score, which is probably the most widely used multivariate scoring rule, performs comparably well in detecting forecast errors, also regarding dependencies. This contradicts other studies. The results also show that a proposed copula score provides very strong distinction between models with correct and incorrect dependency structure. We close with a comprehensive discussion on the proposed methodology.
△ Less
Submitted 16 October, 2019;
originally announced October 2019.
-
Conformal Prediction Interval Estimations with an Application to Day-Ahead and Intraday Power Markets
Authors:
Christopher Kath,
Florian Ziel
Abstract:
We discuss a concept denoted as Conformal Prediction (CP) in this paper. While initially stemming from the world of machine learning, it was never applied or analyzed in the context of short-term electricity price forecasting. Therefore, we elaborate the aspects that render Conformal Prediction worthwhile to know and explain why its simple yet very efficient idea has worked in other fields of appl…
▽ More
We discuss a concept denoted as Conformal Prediction (CP) in this paper. While initially stemming from the world of machine learning, it was never applied or analyzed in the context of short-term electricity price forecasting. Therefore, we elaborate the aspects that render Conformal Prediction worthwhile to know and explain why its simple yet very efficient idea has worked in other fields of application and why its characteristics are promising for short-term power applications as well. We compare its performance with different state-of-the-art electricity price forecasting models such as quantile regression averaging (QRA) in an empirical out-of-sample study for three short-term electricity time series. We combine Conformal Prediction with various underlying point forecast models to demonstrate its versatility and behavior under changing conditions. Our findings suggest that Conformal Prediction yields sharp and reliable prediction intervals in short-term power markets. We further inspect the effect each of Conformal Prediction's model components has and provide a path-based guideline on how to find the best CP model for each market.
△ Less
Submitted 17 September, 2020; v1 submitted 20 May, 2019;
originally announced May 2019.
-
Estimation and simulation of the transaction arrival process in intraday electricity markets
Authors:
Michał Narajewski,
Florian Ziel
Abstract:
We examine the novel problem of the estimation of transaction arrival processes in the intraday electricity markets. We model the inter-arrivals using multiple time-varying parametric densities based on the generalized F distribution estimated by maximum likelihood. We analyse both the in-sample characteristics and the probabilistic forecasting performance. In a rolling window forecasting study, w…
▽ More
We examine the novel problem of the estimation of transaction arrival processes in the intraday electricity markets. We model the inter-arrivals using multiple time-varying parametric densities based on the generalized F distribution estimated by maximum likelihood. We analyse both the in-sample characteristics and the probabilistic forecasting performance. In a rolling window forecasting study, we simulate many trajectories to evaluate the forecasts and gain significant insights into the model fit. The prediction accuracy is evaluated by a functional version of the MAE (mean absolute error), RMSE (root mean squared error) and CRPS (continuous ranked probability score) for the simulated count processes. This paper fills the gap in the literature regarding the intensity estimation of transaction arrivals and is a major contribution to the topic, yet leaves much of the field for further development. The study presented in this paper is conducted based on the German Intraday Continuous electricity market data, but this method can be easily applied to any other continuous intraday electricity market. For the German market, a specific generalized gamma distribution setup explains the overall behaviour significantly best, especially as the tail behaviour of the process is well covered.
△ Less
Submitted 2 December, 2019; v1 submitted 28 January, 2019;
originally announced January 2019.
-
Econometric modelling and forecasting of intraday electricity prices
Authors:
Michał Narajewski,
Florian Ziel
Abstract:
In the following paper, we analyse the ID$_3$-Price in the German Intraday Continuous electricity market using an econometric time series model. A multivariate approach is conducted for hourly and quarter-hourly products separately. We estimate the model using lasso and elastic net techniques and perform an out-of-sample, very short-term forecasting study. The model's performance is compared with…
▽ More
In the following paper, we analyse the ID$_3$-Price in the German Intraday Continuous electricity market using an econometric time series model. A multivariate approach is conducted for hourly and quarter-hourly products separately. We estimate the model using lasso and elastic net techniques and perform an out-of-sample, very short-term forecasting study. The model's performance is compared with benchmark models and is discussed in detail. Forecasting results provide new insights to the German Intraday Continuous electricity market regarding its efficiency and to the ID$_3$-Price behaviour.
△ Less
Submitted 23 September, 2019; v1 submitted 21 December, 2018;
originally announced December 2018.
-
The value of forecasts: Quantifying the economic gains of accurate quarter-hourly electricity price forecasts
Authors:
Christopher Kath,
Florian Ziel
Abstract:
We propose a multivariate elastic net regression forecast model for German quarter-hourly electricity spot markets. While the literature is diverse on day-ahead prediction approaches, both the intraday continuous and intraday call-auction prices have not been studied intensively with a clear focus on predictive power. Besides electricity price forecasting, we check for the impact of early day-ahea…
▽ More
We propose a multivariate elastic net regression forecast model for German quarter-hourly electricity spot markets. While the literature is diverse on day-ahead prediction approaches, both the intraday continuous and intraday call-auction prices have not been studied intensively with a clear focus on predictive power. Besides electricity price forecasting, we check for the impact of early day-ahead (DA) EXAA prices on intraday forecasts. Another novelty of this paper is the complementary discussion of economic benefits. A precise estimation is worthless if it cannot be utilized. We elaborate possible trading decisions based upon our forecasting scheme and analyze their monetary effects. We find that even simple electricity trading strategies can lead to substantial economic impact if combined with a decent forecasting technique.
△ Less
Submitted 21 November, 2018;
originally announced November 2018.
-
Quantile Regression for Qualifying Match of GEFCom2017 Probabilistic Load Forecasting
Authors:
Florian Ziel
Abstract:
We present a simple quantile regression-based forecasting method that was applied in a probabilistic load forecasting framework of the Global Energy Forecasting Competition 2017 (GEFCom2017). The hourly load data is log transformed and split into a long-term trend component and a remainder term. The key forecasting element is the quantile regression approach for the remainder term that takes into…
▽ More
We present a simple quantile regression-based forecasting method that was applied in a probabilistic load forecasting framework of the Global Energy Forecasting Competition 2017 (GEFCom2017). The hourly load data is log transformed and split into a long-term trend component and a remainder term. The key forecasting element is the quantile regression approach for the remainder term that takes into account weekly and annual seasonalities such as their interactions. Temperature information is only used to stabilize the forecast of the long-term trend component. Public holidays information is ignored. Still, the forecasting method placed second in the open data track and fourth in the definite data track with our forecasting method, which is remarkable given simplicity of the model. The method also outperforms the Vanilla benchmark consistently.
△ Less
Submitted 10 September, 2018;
originally announced September 2018.
-
Day-ahead electricity price forecasting with high-dimensional structures: Univariate vs. multivariate modeling frameworks
Authors:
Florian Ziel,
Rafal Weron
Abstract:
We conduct an extensive empirical study on short-term electricity price forecasting (EPF) to address the long-standing question if the optimal model structure for EPF is univariate or multivariate. We provide evidence that despite a minor edge in predictive performance overall, the multivariate modeling framework does not uniformly outperform the univariate one across all 12 considered datasets, s…
▽ More
We conduct an extensive empirical study on short-term electricity price forecasting (EPF) to address the long-standing question if the optimal model structure for EPF is univariate or multivariate. We provide evidence that despite a minor edge in predictive performance overall, the multivariate modeling framework does not uniformly outperform the univariate one across all 12 considered datasets, seasons of the year or hours of the day, and at times is outperformed by the latter. This is an indication that combining advanced structures or the corresponding forecasts from both modeling approaches can bring a further improvement in forecasting accuracy. We show that this indeed can be the case, even for a simple averaging scheme involving only two models. Finally, we also analyze variable selection for the best performing high-dimensional lasso-type models, thus provide guidelines to structuring better performing forecasting model designs.
△ Less
Submitted 17 May, 2018;
originally announced May 2018.
-
Short Term Load Forecasts of Low Voltage Demand and the Effects of Weather
Authors:
Stephen Haben,
Georgios Giasemidis,
Florian Ziel,
Siddharth Arora
Abstract:
Short term load forecasts will play a key role in the implementation of smart electricity grids. They are required to optimise a wide range of potential network solutions on the low voltage (LV) grid, including integrating low carbon technologies (such as photovoltaics) and utilising battery storage devices. Despite the need for accurate LV level load forecasts, previous studies have mostly focuse…
▽ More
Short term load forecasts will play a key role in the implementation of smart electricity grids. They are required to optimise a wide range of potential network solutions on the low voltage (LV) grid, including integrating low carbon technologies (such as photovoltaics) and utilising battery storage devices. Despite the need for accurate LV level load forecasts, previous studies have mostly focused on forecasting at the individual household or building level using data from smart meters. In this study we provide detailed analysis of a variety of methods in terms of both point and probabilistic forecasting accuracy using data from 100 real LV feeders. Moreover, we investigate the effect of temperature (both actual and forecasts) on the accuracy of load forecasts. We present some important results on the drivers of LV forecasting accuracy that are crucial for the management of LV networks, along with an empirical comparison of forecast measures.
△ Less
Submitted 6 April, 2018;
originally announced April 2018.
-
Probabilistic Mid- and Long-Term Electricity Price Forecasting
Authors:
Florian Ziel,
Rick Steinert
Abstract:
The liberalization of electricity markets and the development of renewable energy sources has led to new challenges for decision makers. These challenges are accompanied by an increasing uncertainty about future electricity price movements. The increasing amount of papers, which aim to model and predict electricity prices for a short period of time provided new opportunities for market participant…
▽ More
The liberalization of electricity markets and the development of renewable energy sources has led to new challenges for decision makers. These challenges are accompanied by an increasing uncertainty about future electricity price movements. The increasing amount of papers, which aim to model and predict electricity prices for a short period of time provided new opportunities for market participants. However, the electricity price literature seem to be very scarce on the issue of medium- to long-term price forecasting, which is mandatory for investment and political decisions. Our paper closes this gap by introducing a new approach to simulate electricity prices with hourly resolution for several months up to three years. Considering the uncertainty of future events we are able to provide probabilistic forecasts which are able to detect probabilities for price spikes even in the long-run. As market we decided to use the EPEX day-ahead electricity market for Germany and Austria. Our model extends the X-Model which mainly utilizes the sale and purchase curve for electricity day-ahead auctions. By applying our procedure we are able to give probabilities for the due to the EEG practical relevant event of six consecutive hours of negative prices. We find that using the supply and demand curve based model in the long-run yields realistic patterns for the time series of electricity prices and leads to promising results considering common error measures.
△ Less
Submitted 17 May, 2018; v1 submitted 31 March, 2017;
originally announced March 2017.
-
Forecasting wind power - Modeling periodic and non-linear effects under conditional heteroscedasticity
Authors:
Florian Ziel,
Carsten Croonenbroeck,
Daniel Ambach
Abstract:
In this article we present an approach that enables joint wind speed and wind power forecasts for a wind park. We combine a multivariate seasonal time varying threshold autoregressive moving average (TVARMA) model with a power threshold generalized autoregressive conditional heteroscedastic (power-TGARCH) model. The modeling framework incorporates diurnal and annual periodicity modeling by periodi…
▽ More
In this article we present an approach that enables joint wind speed and wind power forecasts for a wind park. We combine a multivariate seasonal time varying threshold autoregressive moving average (TVARMA) model with a power threshold generalized autoregressive conditional heteroscedastic (power-TGARCH) model. The modeling framework incorporates diurnal and annual periodicity modeling by periodic B-splines, conditional heteroscedasticity and a complex autoregressive structure with non-linear impacts. In contrast to usually time-consuming estimation approaches as likelihood estimation, we apply a high-dimensional shrinkage technique. We utilize an iteratively re-weighted least absolute shrinkage and selection operator (lasso) technique. It allows for conditional heteroscedasticity, provides fast computing times and guarantees a parsimonious and regularized specification, even though the parameter space may be vast. We are able to show that our approach provides accurate forecasts of wind power at a turbine-specific level for forecasting horizons of up to 48 h (short- to medium-term forecasts).
△ Less
Submitted 2 June, 2016;
originally announced June 2016.
-
Lasso estimation for GEFCom2014 probabilistic electric load forecasting
Authors:
Florian Ziel,
Bidong Liu
Abstract:
We present a methodology for probabilistic load forecasting that is based on lasso (least absolute shrinkage and selection operator) estimation. The model considered can be regarded as a bivariate time-varying threshold autoregressive(AR) process for the hourly electric load and temperature. The joint modeling approach incorporates the temperature effects directly, and reflects daily, weekly, and…
▽ More
We present a methodology for probabilistic load forecasting that is based on lasso (least absolute shrinkage and selection operator) estimation. The model considered can be regarded as a bivariate time-varying threshold autoregressive(AR) process for the hourly electric load and temperature. The joint modeling approach incorporates the temperature effects directly, and reflects daily, weekly, and annual seasonal patterns and public holiday effects. We provide two empirical studies, one based on the probabilistic load forecasting track of the Global Energy Forecasting Competition 2014 (GEFCom2014-L), and the other based on another recent probabilistic load forecasting competition that follows a setup similar to that of GEFCom2014-L. In both empirical case studies, the proposed methodology outperforms two multiple linear regression based benchmarks from among the top eight entries to GEFCom2014-L.
△ Less
Submitted 4 March, 2016;
originally announced March 2016.
-
Iteratively reweighted adaptive lasso for conditional heteroscedastic time series with applications to AR-ARCH type processes
Authors:
Florian Ziel
Abstract:
Shrinkage algorithms are of great importance in almost every area of statistics due to the increasing impact of big data. Especially time series analysis benefits from efficient and rapid estimation techniques such as the lasso. However, currently lasso type estimators for autoregressive time series models still focus on models with homoscedastic residuals. Therefore, an iteratively reweighted ada…
▽ More
Shrinkage algorithms are of great importance in almost every area of statistics due to the increasing impact of big data. Especially time series analysis benefits from efficient and rapid estimation techniques such as the lasso. However, currently lasso type estimators for autoregressive time series models still focus on models with homoscedastic residuals. Therefore, an iteratively reweighted adaptive lasso algorithm for the estimation of time series models under conditional heteroscedasticity is presented in a high-dimensional setting. The asymptotic behaviour of the resulting estimator is analysed. It is found that the proposed estimation procedure performs substantially better than its homoscedastic counterpart. A special case of the algorithm is suitable to compute the estimated multivariate AR-ARCH type models efficiently. Extensions to the model like periodic AR-ARCH, threshold AR-ARCH or ARMA-GARCH are discussed. Finally, different simulation results and applications to electricity market data and returns of metal prices are shown.
△ Less
Submitted 5 December, 2015; v1 submitted 23 February, 2015;
originally announced February 2015.
-
Forecasting day ahead electricity spot prices: The impact of the EXAA to other European electricity markets
Authors:
Florian Ziel,
Rick Steinert,
Sven Husmann
Abstract:
In our paper we analyze the relationship between the day-ahead electricity price of the Energy Exchange Austria (EXAA) and other day-ahead electricity prices in Europe. We focus on markets, which settle their prices after the EXAA, which enables traders to include the EXAA price into their calculations. For each market we employ econometric models to incorporate the EXAA price and compare them wit…
▽ More
In our paper we analyze the relationship between the day-ahead electricity price of the Energy Exchange Austria (EXAA) and other day-ahead electricity prices in Europe. We focus on markets, which settle their prices after the EXAA, which enables traders to include the EXAA price into their calculations. For each market we employ econometric models to incorporate the EXAA price and compare them with their counterparts without the price of the Austrian exchange. By employing a forecasting study, we find that electricity price models can be improved when EXAA prices are considered.
△ Less
Submitted 1 December, 2015; v1 submitted 5 January, 2015;
originally announced January 2015.
-
Efficient Modeling and Forecasting of the Electricity Spot Price
Authors:
Florian Ziel,
Rick Steinert,
Sven Husmann
Abstract:
The increasing importance of renewable energy, especially solar and wind power, has led to new forces in the formation of electricity prices. Hence, this paper introduces an econometric model for the hourly time series of electricity prices of the European Power Exchange (EPEX) which incorporates specific features like renewable energy. The model consists of several sophisticated and established a…
▽ More
The increasing importance of renewable energy, especially solar and wind power, has led to new forces in the formation of electricity prices. Hence, this paper introduces an econometric model for the hourly time series of electricity prices of the European Power Exchange (EPEX) which incorporates specific features like renewable energy. The model consists of several sophisticated and established approaches and can be regarded as a periodic VAR-TARCH with wind power, solar power, and load as influences on the time series. It is able to map the distinct and well-known features of electricity prices in Germany. An efficient iteratively reweighted lasso approach is used for the estimation. Moreover, it is shown that several existing models are outperformed by the procedure developed in this paper.
△ Less
Submitted 13 October, 2014; v1 submitted 27 February, 2014;
originally announced February 2014.
