Showing 1–9 of 9 results for author: Chakrabortty, A

Robust and Scalable Game-theoretic Security Investment Methods for Voltage Stability of Power Systems

Authors: Lu An, Pratishtha Shukla, Aranya Chakrabortty, Alexandra Duel-Hallen

Abstract: We develop investment approaches to secure electric power systems against load attacks where a malicious intruder (the attacker) covertly changes reactive power setpoints of loads to push the grid towards voltage instability while the system operator (the defender) employs reactive power compensation (RPC) to prevent instability. Extending our previously reported Stackelberg game formulation for t… ▽ More We develop investment approaches to secure electric power systems against load attacks where a malicious intruder (the attacker) covertly changes reactive power setpoints of loads to push the grid towards voltage instability while the system operator (the defender) employs reactive power compensation (RPC) to prevent instability. Extending our previously reported Stackelberg game formulation for this problem, we develop a robust-defense sequential algorithm and a novel genetic algorithm that provides scalability to large-scale power system models. The proposed methods are validated using IEEE prototype power system models with time-varying load uncertainties, demonstrating that reliable and robust defense is feasible unless the operator's RPC investment resources are severely limited relative to the attacker's resources. △ Less

Submitted 4 September, 2023; v1 submitted 10 March, 2022; originally announced March 2022.

Comments: 6 pages, 6 figures, accepted by IEEE CDC 2023

Distributed Multi-Agent Reinforcement Learning Based on Graph-Induced Local Value Functions

Authors: Gangshan **g, He Bai, Jemin George, Aranya Chakrabortty, Piyush K. Sharma

Abstract: Achieving distributed reinforcement learning (RL) for large-scale cooperative multi-agent systems (MASs) is challenging because: (i) each agent has access to only limited information; (ii) issues on convergence or computational complexity emerge due to the curse of dimensionality. In this paper, we propose a general computationally efficient distributed framework for cooperative multi-agent reinfo… ▽ More Achieving distributed reinforcement learning (RL) for large-scale cooperative multi-agent systems (MASs) is challenging because: (i) each agent has access to only limited information; (ii) issues on convergence or computational complexity emerge due to the curse of dimensionality. In this paper, we propose a general computationally efficient distributed framework for cooperative multi-agent reinforcement learning (MARL) by utilizing the structures of graphs involved in this problem. We introduce three coupling graphs describing three types of inter-agent couplings in MARL, namely, the state graph, the observation graph and the reward graph. By further considering a communication graph, we propose two distributed RL approaches based on local value-functions derived from the coupling graphs. The first approach is able to reduce sample complexity significantly under specific conditions on the aforementioned four graphs. The second approach provides an approximate solution and can be efficient even for problems with dense coupling graphs. Here there is a trade-off between minimizing the approximation error and reducing the computational complexity. Simulations show that our RL algorithms have a significantly improved scalability to large-scale MASs compared with centralized and consensus-based distributed RL algorithms. △ Less

Submitted 11 April, 2024; v1 submitted 25 February, 2022; originally announced February 2022.

Comments: This paper has been accepted by IEEE Transactions on Automatic Control as a full paper and published online. Different from the published paper, the arxiv version contains more results. Moreover, we will continuously update the arxiv version if we find any typos in the published paper. So we suggest you to read this arxiv paper instead of the published one. Thank you for your interest in our work

Distributed Cooperative Multi-Agent Reinforcement Learning with Directed Coordination Graph

Authors: Gangshan **g, He Bai, Jemin George, Aranya Chakrabortty, Piyush. K. Sharma

Abstract: Existing distributed cooperative multi-agent reinforcement learning (MARL) frameworks usually assume undirected coordination graphs and communication graphs while estimating a global reward via consensus algorithms for policy evaluation. Such a framework may induce expensive communication costs and exhibit poor scalability due to requirement of global consensus. In this work, we study MARLs with d… ▽ More Existing distributed cooperative multi-agent reinforcement learning (MARL) frameworks usually assume undirected coordination graphs and communication graphs while estimating a global reward via consensus algorithms for policy evaluation. Such a framework may induce expensive communication costs and exhibit poor scalability due to requirement of global consensus. In this work, we study MARLs with directed coordination graphs, and propose a distributed RL algorithm where the local policy evaluations are based on local value functions. The local value function of each agent is obtained by local communication with its neighbors through a directed learning-induced communication graph, without using any consensus algorithm. A zeroth-order optimization (ZOO) approach based on parameter perturbation is employed to achieve gradient estimation. By comparing with existing ZOO-based RL algorithms, we show that our proposed distributed RL algorithm guarantees high scalability. A distributed resource allocation example is shown to illustrate the effectiveness of our algorithm. △ Less

Submitted 9 January, 2022; originally announced January 2022.

Asynchronous Distributed Reinforcement Learning for LQR Control via Zeroth-Order Block Coordinate Descent

Authors: Gangshan **g, He Bai, Jemin George, Aranya Chakrabortty, Piyush K. Sharma

Abstract: Recently introduced distributed zeroth-order optimization (ZOO) algorithms have shown their utility in distributed reinforcement learning (RL). Unfortunately, in the gradient estimation process, almost all of them require random samples with the same dimension as the global variable and/or require evaluation of the global cost function, which may induce high estimation variance for large-scale net… ▽ More Recently introduced distributed zeroth-order optimization (ZOO) algorithms have shown their utility in distributed reinforcement learning (RL). Unfortunately, in the gradient estimation process, almost all of them require random samples with the same dimension as the global variable and/or require evaluation of the global cost function, which may induce high estimation variance for large-scale networks. In this paper, we propose a novel distributed zeroth-order algorithm by leveraging the network structure inherent in the optimization objective, which allows each agent to estimate its local gradient by local cost evaluation independently, without use of any consensus protocol. The proposed algorithm exhibits an asynchronous update scheme, and is designed for stochastic non-convex optimization with a possibly non-convex feasible domain based on the block coordinate descent method. The algorithm is later employed as a distributed model-free RL algorithm for distributed linear quadratic regulator design, where a learning graph is designed to describe the required interaction relationship among agents in distributed learning. We provide an empirical validation of the proposed algorithm to benchmark its performance on convergence rate and variance against a centralized ZOO algorithm. △ Less

Submitted 2 May, 2024; v1 submitted 26 July, 2021; originally announced July 2021.

Comments: The arxiv version contains proofs of Lemma 3 and Lemma 5, which are missing in the published version

Decomposability and Parallel Computation of Multi-Agent LQR

Authors: Gangshan **g, He Bai, Jemin George, Aranya Chakrabortty

Abstract: Individual agents in a multi-agent system (MAS) may have decoupled open-loop dynamics, but a cooperative control objective usually results in coupled closed-loop dynamics thereby making the control design computationally expensive. The computation time becomes even higher when a learning strategy such as reinforcement learning (RL) needs to be applied to deal with the situation when the agents dyn… ▽ More Individual agents in a multi-agent system (MAS) may have decoupled open-loop dynamics, but a cooperative control objective usually results in coupled closed-loop dynamics thereby making the control design computationally expensive. The computation time becomes even higher when a learning strategy such as reinforcement learning (RL) needs to be applied to deal with the situation when the agents dynamics are not known. To resolve this problem, we propose a parallel RL scheme for a linear quadratic regulator (LQR) design in a continuous-time linear MAS. The idea is to exploit the structural properties of two graphs embedded in the $Q$ and $R$ weighting matrices in the LQR objective to define an orthogonal transformation that can convert the original LQR design to multiple decoupled smaller-sized LQR designs. We show that if the MAS is homogeneous then this decomposition retains closed-loop optimality. Conditions for decomposability, an algorithm for constructing the transformation matrix, a parallel RL algorithm, and robustness analysis when the design is applied to non-homogeneous MAS are presented. Simulations show that the proposed approach can guarantee significant speed-up in learning without any loss in the cumulative value of the LQR cost. △ Less

Submitted 7 March, 2021; v1 submitted 16 October, 2020; originally announced October 2020.

Comments: This paper contains proofs of all the theorems in the conference paper "Decomposability and Parallel Computation of Multi-Agent LQR"

Model-Free Optimal Control of Linear Multi-Agent Systems via Decomposition and Hierarchical Approximation

Authors: Gangshan **g, He Bai, Jemin George, Aranya Chakrabortty

Abstract: Designing the optimal linear quadratic regulator (LQR) for a large-scale multi-agent system (MAS) is time-consuming since it involves solving a large-size matrix Riccati equation. The situation is further exasperated when the design needs to be done in a model-free way using schemes such as reinforcement learning (RL). To reduce this computational complexity, we decompose the large-scale LQR desig… ▽ More Designing the optimal linear quadratic regulator (LQR) for a large-scale multi-agent system (MAS) is time-consuming since it involves solving a large-size matrix Riccati equation. The situation is further exasperated when the design needs to be done in a model-free way using schemes such as reinforcement learning (RL). To reduce this computational complexity, we decompose the large-scale LQR design problem into multiple smaller-size LQR design problems. We consider the objective function to be specified over an undirected graph, and cast the decomposition as a graph clustering problem. The graph is decomposed into two parts, one consisting of independent clusters of connected components, and the other containing edges that connect different clusters. Accordingly, the resulting controller has a hierarchical structure, consisting of two components. The first component optimizes the performance of each independent cluster by solving the smaller-size LQR design problem in a model-free way using an RL algorithm. The second component accounts for the objective coupling different clusters, which is achieved by solving a least squares problem in one shot. Although suboptimal, the hierarchical controller adheres to a particular structure as specified by inter-agent couplings in the objective function and by the decomposition strategy. Mathematical formulations are established to find a decomposition that minimizes the number of required communication links or reduces the optimality gap. Numerical simulations are provided to highlight the pros and cons of the proposed designs. △ Less

Submitted 16 March, 2021; v1 submitted 14 August, 2020; originally announced August 2020.

Comments: This paper proposes a hierarchical learning and control framework for model-free LQR of heterogeneous linear multi-agent systems

A Stackelberg Security Investment Game for Voltage Stability of Power Systems

Authors: Lu An, Aranya Chakrabortty, Alexandra Duel-Hallen

Abstract: We formulate a Stackelberg game between an attacker and a defender of a power system. The attacker attempts to alter the load setpoints of the power system covertly and intelligently, so that the voltage stability margin of the grid is reduced, driving the entire system towards a voltage collapse. The defender, or the system operator, aims to compensate for this reduction by retuning the reactive… ▽ More We formulate a Stackelberg game between an attacker and a defender of a power system. The attacker attempts to alter the load setpoints of the power system covertly and intelligently, so that the voltage stability margin of the grid is reduced, driving the entire system towards a voltage collapse. The defender, or the system operator, aims to compensate for this reduction by retuning the reactive power injection to the grid by switching on control devices, such as a bank of shunt capacitors. A modified Backward Induction method is proposed to find a cost-based Stackelberg equilibrium (CBSE) of the game, which saves the players' costs while providing the optimal allocation of both players' investment resources under budget and covertness constraints. We analyze the proposed game extensively for the IEEE 9-bus power system model and present an example of its performance for the IEEE 39-bus power system model. It is demonstrated that the defender is able to maintain system stability unless its security budget is much lower than the attacker's budget. △ Less

Submitted 11 September, 2020; v1 submitted 20 June, 2020; originally announced June 2020.

Comments: 6 pages in main paper, 6 pages in supplementary material, 5 figs, Main paper has been accepted by CDC 2020

Reduced-Dimensional Reinforcement Learning Control using Singular Perturbation Approximations

Authors: Sayak Mukherjee, He Bai, Aranya Chakrabortty

Abstract: We present a set of model-free, reduced-dimensional reinforcement learning (RL) based optimal control designs for linear time-invariant singularly perturbed (SP) systems. We first present a state-feedback and output-feedback based RL control design for a generic SP system with unknown state and input matrices. We take advantage of the underlying time-scale separation property of the plant to learn… ▽ More We present a set of model-free, reduced-dimensional reinforcement learning (RL) based optimal control designs for linear time-invariant singularly perturbed (SP) systems. We first present a state-feedback and output-feedback based RL control design for a generic SP system with unknown state and input matrices. We take advantage of the underlying time-scale separation property of the plant to learn a linear quadratic regulator (LQR) for only its slow dynamics, thereby saving a significant amount of learning time compared to the conventional full-dimensional RL controller. We analyze the sub-optimality of the design using SP approximation theorems and provide sufficient conditions for closed-loop stability. Thereafter, we extend both designs to clustered multi-agent consensus networks, where the SP property reflects through clustering. We develop both centralized and cluster-wise block-decentralized RL controllers for such networks, in reduced dimensions. We demonstrate the details of the implementation of these controllers using simulations of relevant numerical examples and compare them with conventional RL designs to show the computational benefits of our approach. △ Less

Submitted 29 April, 2020; originally announced April 2020.

Journal ref: Automatica 2021 (full version with proofs)

Digital Grid: Transforming the Electric Power Grid into an Innovation Engine for the United States

Authors: Aranya Chakrabortty, Alex Huang

Abstract: The electric power grid is one of the largest and most complex infrastructures ever built by mankind. Modern civilization depends on it for industry production, human mobility, and comfortable living. However, many critical technologies such as the 60 Hz transformers were developed at the beginning of the 20th century and have changed very little since then.1 The traditional unidirectional power f… ▽ More The electric power grid is one of the largest and most complex infrastructures ever built by mankind. Modern civilization depends on it for industry production, human mobility, and comfortable living. However, many critical technologies such as the 60 Hz transformers were developed at the beginning of the 20th century and have changed very little since then.1 The traditional unidirectional power from the generation to the customer through the transmission-distribution grid has also changed nominally, but it no longer meets the need of the 21st century market energy customers. On one hand, 128m US residential customers pay $15B/per month for their utility bill, yet they have no option to select their energy supplier. In a world of where many traditional industries are transformed by digital Internet technology (Amazon, Ebay, Uber, Airbnb), the traditional electric energy market is lagging significantly behind. A move towards a true digital grid is needed. Such a digital grid requires a tight integration of the physical layer (energy and power) with digital and cyber information to allow an open and real time market akin to the world of e-commerce. Another major factor that is pushing for this radical transformation are the rapidly changing patterns in energy resources ownership and load flow. Driven by the decreasing cost in distributed solar, energy storage, electric vehicle, on site generation and microgrids, the high penetration of Distributed Energy Resource (DER) is shifting challenges substantially towards the edge of grid from the control point of view. The envisioned Digital Grid must facilitate the open competition and open innovation needed to accelerate of the adoption of new DER technologies while satisfying challenges in grid stability, data explosion and cyber security. △ Less

Submitted 4 May, 2017; originally announced May 2017.

Comments: A Computing Community Consortium (CCC) white paper, 3 pages