-
Control Policy Correction Framework for Reinforcement Learning-based Energy Arbitrage Strategies
Authors:
Seyed Soroush Karimi Madahi,
Gargya Gokhale,
Marie-Sophie Verwee,
Bert Claessens,
Chris Develder
Abstract:
A continuous rise in the penetration of renewable energy sources, along with the use of the single imbalance pricing, provides a new opportunity for balance responsible parties to reduce their cost through energy arbitrage in the imbalance settlement mechanism. Model-free reinforcement learning (RL) methods are an appropriate choice for solving the energy arbitrage problem due to their outstanding…
▽ More
A continuous rise in the penetration of renewable energy sources, along with the use of the single imbalance pricing, provides a new opportunity for balance responsible parties to reduce their cost through energy arbitrage in the imbalance settlement mechanism. Model-free reinforcement learning (RL) methods are an appropriate choice for solving the energy arbitrage problem due to their outstanding performance in solving complex stochastic sequential problems. However, RL is rarely deployed in real-world applications since its learned policy does not necessarily guarantee safety during the execution phase. In this paper, we propose a new RL-based control framework for batteries to obtain a safe energy arbitrage strategy in the imbalance settlement mechanism. In our proposed control framework, the agent initially aims to optimize the arbitrage revenue. Subsequently, in the post-processing step, we correct (constrain) the learned policy following a knowledge distillation process based on properties that follow human intuition. Our post-processing step is a generic method and is not restricted to the energy arbitrage domain. We use the Belgian imbalance price of 2023 to evaluate the performance of our proposed framework. Furthermore, we deploy our proposed control framework on a real battery to show its capability in the real world.
△ Less
Submitted 30 April, 2024; v1 submitted 29 April, 2024;
originally announced April 2024.
-
HomeLabGym: A real-world testbed for home energy management systems
Authors:
Toon Van Puyvelde,
Marie-Sophie Verwee,
Gargya Gokhale,
Mehran Zareh Eshghdoust,
Chris Develder
Abstract:
Amid growing environmental concerns and resulting energy costs, there is a rising need for efficient Home Energy Management Systems (HEMS). Evaluating such innovative HEMS solutions typically relies on simulations that may not model the full complexity of a real-world scenario. On the other hand, real-world testing, while more accurate, is labor-intensive, particularly when dealing with diverse as…
▽ More
Amid growing environmental concerns and resulting energy costs, there is a rising need for efficient Home Energy Management Systems (HEMS). Evaluating such innovative HEMS solutions typically relies on simulations that may not model the full complexity of a real-world scenario. On the other hand, real-world testing, while more accurate, is labor-intensive, particularly when dealing with diverse assets, each using a distinct communication protocol or API. Centralizing and synchronizing the control of such a heterogeneous pool of assets thus poses a significant challenge. In this paper, we introduce HomeLabGym, a real-world testbed to ease such real-world evaluations of HEMS and flexible assets control in general, by adhering to the well-known OpenAI Gym paradigm. HomeLabGym allows researchers to prototype, deploy, and analyze HEMS controllers within the controlled test environment of a real-world house (the IDLab HomeLab), providing access to all its available sensors and smart appliances. The easy-to-use Python interface eliminates concerns about intricate communication protocols associated with sensors and appliances, streamlining the evaluation of various control strategies. We present an overview of HomeLabGym, and demonstrate its usefulness to researchers in a comparison between real-world and simulated environments in controlling a residential battery in response to real-time prices.
△ Less
Submitted 22 April, 2024;
originally announced April 2024.
-
Real-World Implementation of Reinforcement Learning Based Energy Coordination for a Cluster of Households
Authors:
Gargya Gokhale,
Niels Tiben,
Marie-Sophie Verwee,
Manu Lahariya,
Bert Claessens,
Chris Develder
Abstract:
Given its substantial contribution of 40\% to global power consumption, the built environment has received increasing attention to serve as a source of flexibility to assist the modern power grid. In that respect, previous research mainly focused on energy management of individual buildings. In contrast, in this paper, we focus on aggregated control of a set of residential buildings, to provide gr…
▽ More
Given its substantial contribution of 40\% to global power consumption, the built environment has received increasing attention to serve as a source of flexibility to assist the modern power grid. In that respect, previous research mainly focused on energy management of individual buildings. In contrast, in this paper, we focus on aggregated control of a set of residential buildings, to provide grid supporting services, that eventually should include ancillary services. In particular, we present a real-life pilot study that studies the effectiveness of reinforcement-learning (RL) in coordinating the power consumption of 8 residential buildings to jointly track a target power signal. Our RL approach relies solely on observed data from individual households and does not require any explicit building models or simulators, making it practical to implement and easy to scale. We show the feasibility of our proposed RL-based coordination strategy in a real-world setting. In a 4-week case study, we demonstrate a hierarchical control system, relying on an RL-based ranking system to select which households to activate flex assets from, and a real-time PI control-based power dispatch mechanism to control the selected assets. Our results demonstrate satisfactory power tracking, and the effectiveness of the RL-based ranks which are learnt in a purely data-driven manner.
△ Less
Submitted 29 October, 2023;
originally announced October 2023.
