-
Control Policy Correction Framework for Reinforcement Learning-based Energy Arbitrage Strategies
Authors:
Seyed Soroush Karimi Madahi,
Gargya Gokhale,
Marie-Sophie Verwee,
Bert Claessens,
Chris Develder
Abstract:
A continuous rise in the penetration of renewable energy sources, along with the use of the single imbalance pricing, provides a new opportunity for balance responsible parties to reduce their cost through energy arbitrage in the imbalance settlement mechanism. Model-free reinforcement learning (RL) methods are an appropriate choice for solving the energy arbitrage problem due to their outstanding…
▽ More
A continuous rise in the penetration of renewable energy sources, along with the use of the single imbalance pricing, provides a new opportunity for balance responsible parties to reduce their cost through energy arbitrage in the imbalance settlement mechanism. Model-free reinforcement learning (RL) methods are an appropriate choice for solving the energy arbitrage problem due to their outstanding performance in solving complex stochastic sequential problems. However, RL is rarely deployed in real-world applications since its learned policy does not necessarily guarantee safety during the execution phase. In this paper, we propose a new RL-based control framework for batteries to obtain a safe energy arbitrage strategy in the imbalance settlement mechanism. In our proposed control framework, the agent initially aims to optimize the arbitrage revenue. Subsequently, in the post-processing step, we correct (constrain) the learned policy following a knowledge distillation process based on properties that follow human intuition. Our post-processing step is a generic method and is not restricted to the energy arbitrage domain. We use the Belgian imbalance price of 2023 to evaluate the performance of our proposed framework. Furthermore, we deploy our proposed control framework on a real battery to show its capability in the real world.
△ Less
Submitted 30 April, 2024; v1 submitted 29 April, 2024;
originally announced April 2024.
-
Probabilistic forecasting of power system imbalance using neural network-based ensembles
Authors:
Jonas Van Gompel,
Bert Claessens,
Chris Develder
Abstract:
Kee** the balance between electricity generation and consumption is becoming increasingly challenging and costly, mainly due to the rising share of renewables, electric vehicles and heat pumps and electrification of industrial processes. Accurate imbalance forecasts, along with reliable uncertainty estimations, enable transmission system operators (TSOs) to dispatch appropriate reserve volumes,…
▽ More
Kee** the balance between electricity generation and consumption is becoming increasingly challenging and costly, mainly due to the rising share of renewables, electric vehicles and heat pumps and electrification of industrial processes. Accurate imbalance forecasts, along with reliable uncertainty estimations, enable transmission system operators (TSOs) to dispatch appropriate reserve volumes, reducing balancing costs. Further, market parties can use these probabilistic forecasts to design strategies that exploit asset flexibility to help balance the grid, generating revenue with known risks. Despite its importance, literature regarding system imbalance (SI) forecasting is limited. Further, existing methods do not focus on situations with high imbalance magnitude, which are crucial to forecast accurately for both TSOs and market parties. Hence, we propose an ensemble of C-VSNs, which are our adaptation of variable selection networks (VSNs). Each minute, our model predicts the imbalance of the current and upcoming two quarter-hours, along with uncertainty estimations on these forecasts. We evaluate our approach by forecasting the imbalance of Belgium, where high imbalance magnitude is defined as $|$SI$| > 500\,$MW (occurs 1.3% of the time in Belgium). For high imbalance magnitude situations, our model outperforms the state-of-the-art by 23.4% (in terms of continuous ranked probability score (CRPS), which evaluates probabilistic forecasts), while also attaining a 6.5% improvement in overall CRPS. Similar improvements are achieved in terms of root-mean-squared error. Additionally, we developed a fine-tuning methodology to effectively include new inputs with limited history in our model. This work was performed in collaboration with Elia (the Belgian TSO) to further improve their imbalance forecasts, demonstrating the relevance of our work.
△ Less
Submitted 24 April, 2024; v1 submitted 23 April, 2024;
originally announced April 2024.
-
HomeLabGym: A real-world testbed for home energy management systems
Authors:
Toon Van Puyvelde,
Marie-Sophie Verwee,
Gargya Gokhale,
Mehran Zareh Eshghdoust,
Chris Develder
Abstract:
Amid growing environmental concerns and resulting energy costs, there is a rising need for efficient Home Energy Management Systems (HEMS). Evaluating such innovative HEMS solutions typically relies on simulations that may not model the full complexity of a real-world scenario. On the other hand, real-world testing, while more accurate, is labor-intensive, particularly when dealing with diverse as…
▽ More
Amid growing environmental concerns and resulting energy costs, there is a rising need for efficient Home Energy Management Systems (HEMS). Evaluating such innovative HEMS solutions typically relies on simulations that may not model the full complexity of a real-world scenario. On the other hand, real-world testing, while more accurate, is labor-intensive, particularly when dealing with diverse assets, each using a distinct communication protocol or API. Centralizing and synchronizing the control of such a heterogeneous pool of assets thus poses a significant challenge. In this paper, we introduce HomeLabGym, a real-world testbed to ease such real-world evaluations of HEMS and flexible assets control in general, by adhering to the well-known OpenAI Gym paradigm. HomeLabGym allows researchers to prototype, deploy, and analyze HEMS controllers within the controlled test environment of a real-world house (the IDLab HomeLab), providing access to all its available sensors and smart appliances. The easy-to-use Python interface eliminates concerns about intricate communication protocols associated with sensors and appliances, streamlining the evaluation of various control strategies. We present an overview of HomeLabGym, and demonstrate its usefulness to researchers in a comparison between real-world and simulated environments in controlling a residential battery in response to real-time prices.
△ Less
Submitted 22 April, 2024;
originally announced April 2024.
-
Explainable Reinforcement Learning-based Home Energy Management Systems using Differentiable Decision Trees
Authors:
Gargya Gokhale,
Bert Claessens,
Chris Develder
Abstract:
With the ongoing energy transition, demand-side flexibility has become an important aspect of the modern power grid for providing grid support and allowing further integration of sustainable energy sources. Besides traditional sources, the residential sector is another major and largely untapped source of flexibility, driven by the increased adoption of solar PV, home batteries, and EVs. However,…
▽ More
With the ongoing energy transition, demand-side flexibility has become an important aspect of the modern power grid for providing grid support and allowing further integration of sustainable energy sources. Besides traditional sources, the residential sector is another major and largely untapped source of flexibility, driven by the increased adoption of solar PV, home batteries, and EVs. However, unlocking this residential flexibility is challenging as it requires a control framework that can effectively manage household energy consumption, and maintain user comfort while being readily scalable across different, diverse houses. We aim to address this challenging problem and introduce a reinforcement learning-based approach using differentiable decision trees. This approach integrates the scalability of data-driven reinforcement learning with the explainability of (differentiable) decision trees. This leads to a controller that can be easily adapted across different houses and provides a simple control policy that can be explained to end-users, further improving user acceptance. As a proof-of-concept, we analyze our method using a home energy management problem, comparing its performance with commercially available rule-based baseline and standard neural network-based RL controllers. Through this preliminary study, we show that the performance of our proposed method is comparable to standard RL-based controllers, outperforming baseline controllers by ~20% in terms of daily cost savings while being straightforward to explain.
△ Less
Submitted 18 March, 2024;
originally announced March 2024.
-
Distill2Explain: Differentiable decision trees for explainable reinforcement learning in energy application controllers
Authors:
Gargya Gokhale,
Seyed Soroush Karimi Madahi,
Bert Claessens,
Chris Develder
Abstract:
Demand-side flexibility is gaining importance as a crucial element in the energy transition process. Accounting for about 25% of final energy consumption globally, the residential sector is an important (potential) source of energy flexibility. However, unlocking this flexibility requires develo** a control framework that (1) easily scales across different houses, (2) is easy to maintain, and (3…
▽ More
Demand-side flexibility is gaining importance as a crucial element in the energy transition process. Accounting for about 25% of final energy consumption globally, the residential sector is an important (potential) source of energy flexibility. However, unlocking this flexibility requires develo** a control framework that (1) easily scales across different houses, (2) is easy to maintain, and (3) is simple to understand for end-users. A potential control framework for such a task is data-driven control, specifically model-free reinforcement learning (RL). Such RL-based controllers learn a good control policy by interacting with their environment, learning purely based on data and with minimal human intervention. Yet, they lack explainability, which hampers user acceptance. Moreover, limited hardware capabilities of residential assets forms a hurdle (e.g., using deep neural networks). To overcome both those challenges, we propose a novel method to obtain explainable RL policies by using differentiable decision trees. Using a policy distillation approach, we train these differentiable decision trees to mimic standard RL-based controllers, leading to a decision tree-based control policy that is data-driven and easy to explain. As a proof-of-concept, we examine the performance and explainability of our proposed approach in a battery-based home energy management system to reduce energy costs. For this use case, we show that our proposed approach can outperform baseline rule-based policies by about 20-25%, while providing simple, explainable control policies. We further compare these explainable policies with standard RL policies and examine the performance trade-offs associated with this increased explainability.
△ Less
Submitted 18 March, 2024;
originally announced March 2024.
-
Distributional Reinforcement Learning-based Energy Arbitrage Strategies in Imbalance Settlement Mechanism
Authors:
Seyed Soroush Karimi Madahi,
Bert Claessens,
Chris Develder
Abstract:
Growth in the penetration of renewable energy sources makes supply more uncertain and leads to an increase in the system imbalance. This trend, together with the single imbalance pricing, opens an opportunity for balance responsible parties (BRPs) to perform energy arbitrage in the imbalance settlement mechanism. To this end, we propose a battery control framework based on distributional reinforce…
▽ More
Growth in the penetration of renewable energy sources makes supply more uncertain and leads to an increase in the system imbalance. This trend, together with the single imbalance pricing, opens an opportunity for balance responsible parties (BRPs) to perform energy arbitrage in the imbalance settlement mechanism. To this end, we propose a battery control framework based on distributional reinforcement learning (DRL). Our proposed control framework takes a risk-sensitive perspective, allowing BRPs to adjust their risk preferences: we aim to optimize a weighted sum of the arbitrage profit and a risk measure while constraining the daily number of cycles for the battery. We assess the performance of our proposed control framework using the Belgian imbalance prices of 2022 and compare two state-of-the-art RL methods, deep Q learning and soft actor-critic. Results reveal that the distributional soft actor-critic method can outperform other methods. Moreover, we note that our fully risk-averse agent appropriately learns to hedge against the risk related to the unknown imbalance price by (dis)charging the battery only when the agent is more certain about the price.
△ Less
Submitted 23 December, 2023;
originally announced January 2024.
-
Demand response for residential building heating: Effective Monte Carlo Tree Search control based on physics-informed neural networks
Authors:
Fabio Pavirani,
Gargya Gokhale,
Bert Claessens,
Chris Develder
Abstract:
To reduce global carbon emissions and limit climate change, controlling energy consumption in buildings is an important piece of the puzzle. Here, we specifically focus on using a demand response (DR) algorithm to limit the energy consumption of a residential building's heating system while respecting user's thermal comfort. In that domain, Reinforcement learning (RL) methods have been shown to be…
▽ More
To reduce global carbon emissions and limit climate change, controlling energy consumption in buildings is an important piece of the puzzle. Here, we specifically focus on using a demand response (DR) algorithm to limit the energy consumption of a residential building's heating system while respecting user's thermal comfort. In that domain, Reinforcement learning (RL) methods have been shown to be quite effective. One such RL method is Monte Carlo Tree Search (MCTS), which has achieved impressive success in playing board games (go, chess). A particular advantage of MCTS is that its decision tree structure naturally allows to integrate exogenous constraints (e.g., by trimming branches that violate them), while conventional RL solutions need more elaborate techniques (e.g., indirectly by adding penalties in the cost/reward function, or through a backup controller that corrects constraint-violating actions). The main aim of this paper is to study the adoption of MCTS for building control, since this (to the best of our knowledge) has remained largely unexplored. A specific property of MCTS is that it needs a simulator component that can predict subsequent system states, based on actions taken. A straightforward data-driven solution is to use black-box neural networks (NNs). We will however extend a Physics-informed Neural Network (PiNN) model to deliver multi-timestep predictions, and show the benefit it offers in terms of lower prediction errors ($-$32\% MAE) as well as better MCTS performance ($-$4\% energy cost, $+$7\% thermal comfort) compared to a black-box NN. A second contribution will be to extend a vanilla MCTS version to adopt the ideas applied in AlphaZero (i.e., using learned prior and value functions and an action selection heuristic) to obtain lower computational costs while maintaining control performance.
△ Less
Submitted 21 May, 2024; v1 submitted 6 December, 2023;
originally announced December 2023.
-
Transfer Learning in Transformer-Based Demand Forecasting For Home Energy Management System
Authors:
Gargya Gokhale,
Jonas Van Gompel,
Bert Claessens,
Chris Develder
Abstract:
Increasingly, homeowners opt for photovoltaic (PV) systems and/or battery storage to minimize their energy bills and maximize renewable energy usage. This has spurred the development of advanced control algorithms that maximally achieve those goals. However, a common challenge faced while develo** such controllers is the unavailability of accurate forecasts of household power consumption, especi…
▽ More
Increasingly, homeowners opt for photovoltaic (PV) systems and/or battery storage to minimize their energy bills and maximize renewable energy usage. This has spurred the development of advanced control algorithms that maximally achieve those goals. However, a common challenge faced while develo** such controllers is the unavailability of accurate forecasts of household power consumption, especially for shorter time resolutions (15 minutes) and in a data-efficient manner. In this paper, we analyze how transfer learning can help by exploiting data from multiple households to improve a single house's load forecasting. Specifically, we train an advanced forecasting model (a temporal fusion transformer) using data from multiple different households, and then finetune this global model on a new household with limited data (i.e. only a few days). The obtained models are used for forecasting power consumption of the household for the next 24 hours~(day-ahead) at a time resolution of 15 minutes, with the intention of using these forecasts in advanced controllers such as Model Predictive Control. We show the benefit of this transfer learning setup versus solely using the individual new household's data, both in terms of (i) forecasting accuracy ($\sim$15\% MAE reduction) and (ii) control performance ($\sim$2\% energy cost reduction), using real-world household data.
△ Less
Submitted 29 October, 2023;
originally announced October 2023.
-
Real-World Implementation of Reinforcement Learning Based Energy Coordination for a Cluster of Households
Authors:
Gargya Gokhale,
Niels Tiben,
Marie-Sophie Verwee,
Manu Lahariya,
Bert Claessens,
Chris Develder
Abstract:
Given its substantial contribution of 40\% to global power consumption, the built environment has received increasing attention to serve as a source of flexibility to assist the modern power grid. In that respect, previous research mainly focused on energy management of individual buildings. In contrast, in this paper, we focus on aggregated control of a set of residential buildings, to provide gr…
▽ More
Given its substantial contribution of 40\% to global power consumption, the built environment has received increasing attention to serve as a source of flexibility to assist the modern power grid. In that respect, previous research mainly focused on energy management of individual buildings. In contrast, in this paper, we focus on aggregated control of a set of residential buildings, to provide grid supporting services, that eventually should include ancillary services. In particular, we present a real-life pilot study that studies the effectiveness of reinforcement-learning (RL) in coordinating the power consumption of 8 residential buildings to jointly track a target power signal. Our RL approach relies solely on observed data from individual households and does not require any explicit building models or simulators, making it practical to implement and easy to scale. We show the feasibility of our proposed RL-based coordination strategy in a real-world setting. In a 4-week case study, we demonstrate a hierarchical control system, relying on an RL-based ranking system to select which households to activate flex assets from, and a real-time PI control-based power dispatch mechanism to control the selected assets. Our results demonstrate satisfactory power tracking, and the effectiveness of the RL-based ranks which are learnt in a purely data-driven manner.
△ Less
Submitted 29 October, 2023;
originally announced October 2023.
-
PhysQ: A Physics Informed Reinforcement Learning Framework for Building Control
Authors:
Gargya Gokhale,
Bert Claessens,
Chris Develder
Abstract:
Large-scale integration of intermittent renewable energy sources calls for substantial demand side flexibility. Given that the built environment accounts for approximately 40% of total energy consumption in EU, unlocking its flexibility is a key step in the energy transition process. This paper focuses specifically on energy flexibility in residential buildings, leveraging their intrinsic thermal…
▽ More
Large-scale integration of intermittent renewable energy sources calls for substantial demand side flexibility. Given that the built environment accounts for approximately 40% of total energy consumption in EU, unlocking its flexibility is a key step in the energy transition process. This paper focuses specifically on energy flexibility in residential buildings, leveraging their intrinsic thermal mass. Building on recent developments in the field of data-driven control, we propose PhysQ. As a physics-informed reinforcement learning framework for building control, PhysQ forms a step in bridging the gap between conventional model-based control and data-intensive control based on reinforcement learning. Through our experiments, we show that the proposed PhysQ framework can learn high quality control policies that outperform a business-as-usual, as well as a rudimentary model predictive controller. Our experiments indicate cost savings of about 9% compared to a business-as-usual controller. Further, we show that PhysQ efficiently leverages prior physics knowledge to learn such policies using fewer training samples than conventional reinforcement learning approaches, making PhysQ a scalable alternative for use in residential buildings. Additionally, the PhysQ control policy utilizes building state representations that are intuitive and based on conventional building models, that leads to better interpretation of the learnt policy over other data-driven controllers.
△ Less
Submitted 21 November, 2022;
originally announced November 2022.
-
Physics Informed LSTM Network for Flexibility Identification in Evaporative Cooling Systems
Authors:
Manu Lahariya,
Farzaneh Karami,
Chris Develder,
Guillaume Crevecoeur
Abstract:
In energy intensive industrial systems, an evaporative cooling process may introduce operational flexibility. Such flexibility refers to a systems ability to deviate from its scheduled energy consumption. Identifying the flexibility, and therefore, designing control that ensures efficient and reliable operation presents a great challenge due to the inherently complex dynamics of industrial systems…
▽ More
In energy intensive industrial systems, an evaporative cooling process may introduce operational flexibility. Such flexibility refers to a systems ability to deviate from its scheduled energy consumption. Identifying the flexibility, and therefore, designing control that ensures efficient and reliable operation presents a great challenge due to the inherently complex dynamics of industrial systems. Recently, machine learning models have attracted attention for identifying flexibility, due to their ability to model complex nonlinear behavior. This research presents machine learning based methods that integrate system dynamics into the machine learning models (e.g., Neural Networks) for better adherence to physical constraints. We define and evaluate physics informed long-short term memory networks (PhyLSTM) and physics informed neural networks (PhyNN) for the identification of flexibility in the evaporative cooling process. These physics informed networks approximate the time-dependent relationship between control input and system response while enforcing the dynamics of the process in the neural network architecture. Our proposed PhyLSTM provides less than 2% system response estimation error, converges in less than half iterations compared to a baseline Neural Network (NN), and accurately estimates the defined flexibility metrics. We include a detailed analysis of the impact of training data size on the performance and optimization of our proposed models.
△ Less
Submitted 19 May, 2022;
originally announced May 2022.
-
Computationally efficient joint coordination of multiple electric vehicle charging points using reinforcement learning
Authors:
Manu Lahariya,
Nasrin Sadeghianpourhamami,
Chris Develder
Abstract:
A major challenge in todays power grid is to manage the increasing load from electric vehicle (EV) charging. Demand response (DR) solutions aim to exploit flexibility therein, i.e., the ability to shift EV charging in time and thus avoid excessive peaks or achieve better balancing. Whereas the majority of existing research works either focus on control strategies for a single EV charger, or use a…
▽ More
A major challenge in todays power grid is to manage the increasing load from electric vehicle (EV) charging. Demand response (DR) solutions aim to exploit flexibility therein, i.e., the ability to shift EV charging in time and thus avoid excessive peaks or achieve better balancing. Whereas the majority of existing research works either focus on control strategies for a single EV charger, or use a multi-step approach (e.g., a first high level aggregate control decision step, followed by individual EV control decisions), we rather propose a single-step solution that jointly coordinates multiple charging points at once. In this paper, we further refine an initial proposal using reinforcement learning (RL), specifically addressing computational challenges that would limit its deployment in practice. More precisely, we design a new Markov decision process (MDP) formulation of the EV charging coordination process, exhibiting only linear space and time complexity (as opposed to the earlier quadratic space complexity). We thus improve upon earlier state-of-the-art, demonstrating 30% reduction of training time in our case study using real-world EV charging session data. Yet, we do not sacrifice the resulting performance in meeting the DR objectives: our new RL solutions still improve the performance of charging demand coordination by 40-50% compared to a business-as-usual policy (that charges EV fully upon arrival) and 20-30% compared to a heuristic policy (that uniformly spreads individual EV charging over time).
△ Less
Submitted 26 March, 2022;
originally announced March 2022.
-
Physics Informed Neural Networks for Control Oriented Thermal Modeling of Buildings
Authors:
Gargya Gokhale,
Bert Claessens,
Chris Develder
Abstract:
This paper presents a data-driven modeling approach for develo** control-oriented thermal models of buildings. These models are developed with the objective of reducing energy consumption costs while controlling the indoor temperature of the building within required comfort limits. To combine the interpretability of white/gray box physics models and the expressive power of neural networks, we pr…
▽ More
This paper presents a data-driven modeling approach for develo** control-oriented thermal models of buildings. These models are developed with the objective of reducing energy consumption costs while controlling the indoor temperature of the building within required comfort limits. To combine the interpretability of white/gray box physics models and the expressive power of neural networks, we propose a physics informed neural network approach for this modeling task. Along with measured data and building parameters, we encode the neural networks with the underlying physics that governs the thermal behavior of these buildings. Thus, realizing a model that is guided by physics, aids in modeling the temporal evolution of room temperature and power consumption as well as the hidden state, i.e., the temperature of building thermal mass for subsequent time steps. The main research contributions of this work are: (1) we propose two variants of physics informed neural network architectures for the task of control-oriented thermal modeling of buildings, (2) we show that training these architectures is data-efficient, requiring less training data compared to conventional, non-physics informed neural networks, and (3) we show that these architectures achieve more accurate predictions than conventional neural networks for longer prediction horizons. We test the prediction performance of the proposed architectures using simulated and real-word data to demonstrate (2) and (3) and show that the proposed physics informed neural network architectures can be used for this control-oriented modeling problem.
△ Less
Submitted 21 March, 2022; v1 submitted 23 November, 2021;
originally announced November 2021.
