-
State-action control barrier functions: Imposing safety on learning-based control with low online computational costs
Authors:
Kanghui He,
Shengling Shi,
Ton van den Boom,
Bart De Schutter
Abstract:
Learning-based control with safety guarantees usually requires real-time safety certification and modifications of possibly unsafe learning-based policies. The control barrier function (CBF) method uses a safety filter containing a constrained optimization problem to produce safe policies. However, finding a valid CBF for a general nonlinear system requires a complex function parameterization, whi…
▽ More
Learning-based control with safety guarantees usually requires real-time safety certification and modifications of possibly unsafe learning-based policies. The control barrier function (CBF) method uses a safety filter containing a constrained optimization problem to produce safe policies. However, finding a valid CBF for a general nonlinear system requires a complex function parameterization, which in general, makes the policy optimization problem difficult to solve in real time. For nonlinear systems with nonlinear state constraints, this paper proposes the novel concept of state-action CBFs, which not only characterize the safety at each state but also evaluate the control inputs taken at each state. State-action CBFs, in contrast to CBFs, enable a flexible parameterization, resulting in a safety filter that involves a convex quadratic optimization problem. This, in turn, significantly alleviates the online computational burden. To synthesize state-action CBFs, we propose a learning-based approach exploiting Hamilton-Jacobi reachability. The effect of learning errors on the effectiveness of state-action CBFs is addressed by constraint tightening and introducing a new concept called contractive CBFs. These contributions ensure formal safety guarantees for learned CBFs and control policies, enhancing the applicability of learning-based control in real-time scenarios. Simulation results on an inverted pendulum with elastic walls validate the proposed CBFs in terms of constraint satisfaction and CPU time.
△ Less
Submitted 18 December, 2023;
originally announced December 2023.
-
Approximate Dynamic Programming for Constrained Piecewise Affine Systems with Stability and Safety Guarantees
Authors:
Kanghui He,
Shengling Shi,
Ton van den Boom,
Bart De Schutter
Abstract:
Infinite-horizon optimal control of constrained piecewise affine (PWA) systems has been approximately addressed by hybrid model predictive control (MPC), which, however, has computational limitations, both in offline design and online implementation. In this paper, we consider an alternative approach based on approximate dynamic programming (ADP), an important class of methods in reinforcement lea…
▽ More
Infinite-horizon optimal control of constrained piecewise affine (PWA) systems has been approximately addressed by hybrid model predictive control (MPC), which, however, has computational limitations, both in offline design and online implementation. In this paper, we consider an alternative approach based on approximate dynamic programming (ADP), an important class of methods in reinforcement learning. We accommodate non-convex union-of-polyhedra state constraints and linear input constraints into ADP by designing PWA penalty functions. PWA function approximation is used, which allows for a mixed-integer encoding to implement ADP. The main advantage of the proposed ADP method is its online computational efficiency. Particularly, we propose two control policies, which lead to solving a smaller-scale mixed-integer linear program than conventional hybrid MPC, or a single convex quadratic program, depending on whether the policy is implicitly determined online or explicitly computed offline. We characterize the stability and safety properties of the closed-loop systems, as well as the sub-optimality of the proposed policies, by quantifying the approximation errors of value functions and policies. We also develop an offline mixed-integer linear programming-based method to certify the reliability of the proposed method. Simulation results on an inverted pendulum with elastic walls and on an adaptive cruise control problem validate the control performance in terms of constraint satisfaction and CPU time.
△ Less
Submitted 6 January, 2024; v1 submitted 27 June, 2023;
originally announced June 2023.
-
Approximate Dynamic Programming for Constrained Linear Systems: A Piecewise Quadratic Approximation Approach
Authors:
Kanghui He,
Shengling Shi,
Ton van den Boom,
Bart De Schutter
Abstract:
Approximate dynamic programming (ADP) faces challenges in dealing with constraints in control problems. Model predictive control (MPC) is, in comparison, well-known for its accommodation of constraints and stability guarantees, although its computation is sometimes prohibitive. This paper introduces an approach combining the two methodologies to overcome their individual limitations. The predictiv…
▽ More
Approximate dynamic programming (ADP) faces challenges in dealing with constraints in control problems. Model predictive control (MPC) is, in comparison, well-known for its accommodation of constraints and stability guarantees, although its computation is sometimes prohibitive. This paper introduces an approach combining the two methodologies to overcome their individual limitations. The predictive control law for constrained linear quadratic regulation (CLQR) problems has been proven to be piecewise affine (PWA) while the value function is piecewise quadratic. We exploit these formal results from MPC to design an ADP method for CLQR problems. A novel convex and piecewise quadratic neural network with a local-global architecture is proposed to provide an accurate approximation of the value function, which is used as the cost-to-go function in the online dynamic programming problem. An efficient decomposition algorithm is developed to speed up the online computation. Rigorous stability analysis of the closed-loop system is conducted for the proposed control scheme under the condition that a good approximation of the value function is achieved. Comparative simulations are carried out to demonstrate the potential of the proposed method in terms of online computation and optimality.
△ Less
Submitted 6 April, 2023; v1 submitted 20 May, 2022;
originally announced May 2022.
-
Max-algebraic hybrid automata: Modelling and equivalences
Authors:
A. Gupta,
B. De Schutter,
J. van der Woude,
T. van den Boom
Abstract:
This article introduces the novel framework of max-algebraic hybrid automata as a hybrid modelling language in the max-plus algebra. We show that the modelling framework unifies and extends the switching max-plus linear systems framework and is analogous to the discrete hybrid automata framework in conventional algebra. In addition, we show that the framework serves as a bridge between automata-th…
▽ More
This article introduces the novel framework of max-algebraic hybrid automata as a hybrid modelling language in the max-plus algebra. We show that the modelling framework unifies and extends the switching max-plus linear systems framework and is analogous to the discrete hybrid automata framework in conventional algebra. In addition, we show that the framework serves as a bridge between automata-theoretic models in max-plus algebra and switching max-plus linear systems. In doing so, we formalise the relationship between max-plus automata and switching max-plus linear systems in a behavioural sense. This also serves as another step towards importing tools for analysis and optimal control from conventional time-driven hybrid systems to discrete-event systems in max-plus algebra.
△ Less
Submitted 19 November, 2021;
originally announced November 2021.
-
Framework for Studying Stability of Switching Max-Plus Linear Systems
Authors:
Abhimanyu Gupta,
Ton van den Boom,
Jacob van der Woude,
Bart De Schutter
Abstract:
We propose a framework for studying the stability of discrete-event systems modelled as switching max-plus linear systems. In this framework, we propose a set of notions of stability for generic discrete-event systems in the max-plus algebra. Then we show the loss of equivalence of these notions for switching max-plus linear systems due to the lack of global monotonicity and the accompanying diffi…
▽ More
We propose a framework for studying the stability of discrete-event systems modelled as switching max-plus linear systems. In this framework, we propose a set of notions of stability for generic discrete-event systems in the max-plus algebra. Then we show the loss of equivalence of these notions for switching max-plus linear systems due to the lack of global monotonicity and the accompanying difficulty in rigorous analysis. This serves as a motivation to relax the assumption on monotonicity of the dynamics to positive invariance of max-plus cones. Then we proceed to generalise the notions of stability when the dynamics is restricted to such cones. The stability analysis approach presented in this paper serves as a first step to study the stability of a general class of switching max-plus linear systems.
△ Less
Submitted 6 July, 2020;
originally announced July 2020.
