-
Optimal Control of District Cooling Energy Plant with Reinforcement Learning and MPC
Authors:
Zhong Guo,
Aditya Chaudhari,
Austin R. Coffman,
Prabir Barooah
Abstract:
We consider the problem of optimal control of district cooling energy plants (DCEPs) consisting of multiple chillers, a cooling tower, and a thermal energy storage (TES), in the presence of time-varying electricity price. A straightforward application of model predictive control (MPC) requires solving a challenging mixed-integer nonlinear program (MINLP) because of the on/off of chillers and the c…
▽ More
We consider the problem of optimal control of district cooling energy plants (DCEPs) consisting of multiple chillers, a cooling tower, and a thermal energy storage (TES), in the presence of time-varying electricity price. A straightforward application of model predictive control (MPC) requires solving a challenging mixed-integer nonlinear program (MINLP) because of the on/off of chillers and the complexity of the DCEP model. Reinforcement learning (RL) is an attractive alternative since its real-time control computation is much simpler. But designing an RL controller is challenging due to myriad design choices and computationally intensive training.
In this paper, we propose an RL controller and an MPC controller for minimizing the electricity cost of a DCEP, and compare them via simulations. The two controllers are designed to be comparable in terms of objective and information requirements. The RL controller uses a novel Q-learning algorithm that is based on least-squares policy iteration. We describe the design choices for the RL controller, including the choice of state space and basis functions, that are found to be effective. The proposed MPC controller does not need a mixed integer solver for implementation, but only a nonlinear program (NLP) solver. A rule-based baseline controller is also proposed to aid in comparison. Simulation results show that the proposed RL and MPC controllers achieve similar savings over the baseline controller, about 17%.
△ Less
Submitted 5 October, 2023;
originally announced October 2023.
-
Reinforcement Learning for Optimal Control of a District Cooling Energy Plant
Authors:
Zhong Guo,
Austin R. Coffman,
Prabir Barooah
Abstract:
District cooling energy plants (DCEPs) consisting of chillers, cooling towers, and thermal energy storage (TES) systems consume a considerable amount of electricity. Optimizing the scheduling of the TES and chillers to take advantage of time-varying electricity price is a challenging optimal control problem. The classical method, model predictive control (MPC), requires solving a high dimensional…
▽ More
District cooling energy plants (DCEPs) consisting of chillers, cooling towers, and thermal energy storage (TES) systems consume a considerable amount of electricity. Optimizing the scheduling of the TES and chillers to take advantage of time-varying electricity price is a challenging optimal control problem. The classical method, model predictive control (MPC), requires solving a high dimensional mixed-integer nonlinear program (MINLP) because of the on/off actuation of the chillers and charging/discharging of TES, which are computationally challenging. RL is an attractive alternative to MPC: the real time control computation is a low-dimensional optimization problem that can be easily solved. However, the performance of an RL controller depends on many design choices. In this paper, we propose a Q-learning based reinforcement learning (RL) controller for this problem. Numerical simulation results show that the proposed RL controller is able to reduce energy cost over a rule-based baseline controller by approximately 8%, comparable to savings reported in the literature with MPC for similar DCEPs. We describe the design choices in the RL controller, including basis functions, reward function sha**, and learning algorithm parameters. Compared to existing work on RL for DCEPs, the proposed controller is designed for continuous state and actions spaces.
△ Less
Submitted 14 March, 2022;
originally announced March 2022.
-
A model-free method for learning flexibility capacity of loads providing grid support
Authors:
Austin R. Coffman,
Prabir Barooah
Abstract:
Flexible loads are a resource for the Balancing Authority (BA) of the future to aid in the balance of power supply and demand. In order to be used as a resource, the BA must know the capacity of the flexible loads to vary their power demand over a baseline without violating consumers' quality of service (QoS). Existing work on capacity characterization is model-based: They need models relating pow…
▽ More
Flexible loads are a resource for the Balancing Authority (BA) of the future to aid in the balance of power supply and demand. In order to be used as a resource, the BA must know the capacity of the flexible loads to vary their power demand over a baseline without violating consumers' quality of service (QoS). Existing work on capacity characterization is model-based: They need models relating power consumption to variables that dictate QoS, such as temperature in case of an air conditioning system. However, in many cases the model parameters are not known or difficult to obtain. In this work, we pose a data driven capacity characterization method that does not require model information, it only needs access to a simulator. The capacity is characterized as the set of feasible spectral densities (SDs) of the demand deviation. The proposed method is an extension of our recent work on SD-based capacity characterization that was limited to linear time invariant (LTI) dynamics of loads. The method proposed here is applicable to nonlinear dynamics. Numerical evaluation of the method is provided, including a comparison with the model-based solution for the LTI case.
△ Less
Submitted 4 October, 2020;
originally announced October 2020.
-
Control oriented modeling of TCLs
Authors:
Austin R. Coffman,
Ana Bušić,
Prabir Barooah
Abstract:
Thermostatically controlled loads (TCLs) have the potential to be a valuable resource for the Balancing Authority (BA) of the future. Examples of TCLs include household appliances such as air conditioners, water heaters, and refrigerators. Since the rated power of each TCL is on the order of kilowatts, to provide meaningful service for the BA, it is necessary to control large collections of TCLs.…
▽ More
Thermostatically controlled loads (TCLs) have the potential to be a valuable resource for the Balancing Authority (BA) of the future. Examples of TCLs include household appliances such as air conditioners, water heaters, and refrigerators. Since the rated power of each TCL is on the order of kilowatts, to provide meaningful service for the BA, it is necessary to control large collections of TCLs. To perform design of a distributed coordination/control algorithm, the BA requires a control oriented model that describes the relevant dynamics of an ensemble. Works focusing on solely modeling the ensemble date back to the 1980's, while works focusing on control oriented modeling are more recent. In this work, we contribute to the control oriented modeling literature. We leverage techniques from computational fluid dynamics (CFD) to discretize a pair of Fokker-Planck equations derived in earlier work [1]. The discretized equations are shown to admit a certain factorization, which makes the developed model useful for control design. In particular, the effects of weather and control are shown to independently effect the system dynamics.
△ Less
Submitted 27 September, 2020;
originally announced September 2020.
-
Predictive resource allocation for flexible loads with local QoS
Authors:
Austin R. Coffman,
Matthew Hale,
Prabir Barooah
Abstract:
Loads that can vary their power consumption without violating their Quality of service (QoS), that is flexible loads, are an invaluable resource for grid operators. Utilizing flexible loads as a resource requires the grid operator to incorporate them into a resource allocation problem. Since flexible loads are often consumers, for concerns of privacy it is desirable for this problem to have a dist…
▽ More
Loads that can vary their power consumption without violating their Quality of service (QoS), that is flexible loads, are an invaluable resource for grid operators. Utilizing flexible loads as a resource requires the grid operator to incorporate them into a resource allocation problem. Since flexible loads are often consumers, for concerns of privacy it is desirable for this problem to have a distributed implementation. Technically, this distributed implementation manifests itself as a time varying convex optimization problem constrained by the QoS of each load. In the literature, a time invariant form of this problem without all of the necessary QoS metrics for the flexible loads is often considered. Moving to a more realistic setup introduces additional technical challenges, due to the problems' time-varying nature. In this work, we develop an algorithm to account for the challenges introduced when considering a time varying setup with appropriate QoS metrics.
△ Less
Submitted 29 June, 2020;
originally announced June 2020.
-
Characterizing capacity of flexible loads for providing grid support
Authors:
Austin R. Coffman,
Zhong Guo,
Prabir Barooah
Abstract:
Flexible loads are a resource for the Balancing Authority (BA) of the future to aid in the balance of supply and demand in the power grid. Consequently, it is of interest for a BA to know how much flexibility a collection of loads has, so to successfully incorporate flexible loads into grid level resource allocation. Loads' flexibility is limited by all their Quality of Service (QoS) requirements.…
▽ More
Flexible loads are a resource for the Balancing Authority (BA) of the future to aid in the balance of supply and demand in the power grid. Consequently, it is of interest for a BA to know how much flexibility a collection of loads has, so to successfully incorporate flexible loads into grid level resource allocation. Loads' flexibility is limited by all their Quality of Service (QoS) requirements. In this work we present a characterization of capacity for a collection of flexible loads. This characterization is in terms of the Power Spectral Density (PSD) of the reference signal. Two advantages of our characterization are: (i) it easily allows for a BA to use the characterization for resource allocation of flexible loads and (ii) it allows for precise definitions of the power and energy capacity for a collection of flexible loads.
△ Less
Submitted 4 May, 2020;
originally announced May 2020.
-
Aggregation and Data Driven Identification of Building Thermal Dynamic Model and Unmeasured Disturbance
Authors:
Zhong Guo,
Austin R. Coffman,
Jeffrey Munk,
Piljae Im,
Teja Kuruganti,
Prabir Barooah
Abstract:
An aggregate model is a single-zone equivalent of a multi-zone building, and is useful for many purposes, including model based control of large heating, ventilation and air conditioning (HVAC) equipment. This paper deals with the problem of simultaneously identifying an aggregate thermal dynamic model and unknown disturbances from input-output data. The unknown disturbance is a key challenge sinc…
▽ More
An aggregate model is a single-zone equivalent of a multi-zone building, and is useful for many purposes, including model based control of large heating, ventilation and air conditioning (HVAC) equipment. This paper deals with the problem of simultaneously identifying an aggregate thermal dynamic model and unknown disturbances from input-output data. The unknown disturbance is a key challenge since it is not measurable but non-negligible. We first present a principled method to aggregate a multi-zone building model into a single zone model, and show the aggregation is not as trivial as it has been assumed in the prior art. We then provide a method to identify the parameters of the model and the unknown disturbance for this aggregate (single-zone) model. Finally, we test our proposed identification algorithm to data collected from a multi-zone building testbed in Oak Ridge National Laboratory. A key insight provided by the aggregation method allows us to recognize under what conditions the estimation of the disturbance signal will be necessarily poor and uncertain, even in the case of a specially designed test in which the disturbances affecting each zone are known (as the case of our experimental testbed). This insight is used to provide a heuristic that can be used to assess when the identification results are likely to have high or low accuracy.
△ Less
Submitted 24 January, 2020;
originally announced January 2020.
