Closed-Loop Finite-Time Analysis of
Suboptimal Online Control
Abstract
Suboptimal methods in optimal control arise due to a limited computational budget, unknown system dynamics, or a short prediction window among other reasons. Although these methods are ubiquitous, their transient performance remains relatively unstudied. We consider the control of discrete-time, nonlinear time-varying dynamical systems and establish sufficient conditions to analyze the finite-time closed-loop performance of such methods in terms of the additional cost incurred due to suboptimality. Finite-time guarantees allow the control design to distribute a limited computational budget over a time horizon and estimate the on-the-go loss in performance due to suboptimality. We study exponential incremental input-to-state stabilizing policies, and show that for nonlinear systems, under some mild conditions, this property is directly implied by exponential stability without further assumptions on global smoothness. The analysis is showcased on a suboptimal model predictive control use case.
Nonlinear Systems, Optimization Algorithms, Predictive Control
1 Introduction
Optimal control aims to compute an input signal to drive a dynamical system to a given target state, while optimizing a performance cost subject to constraints. In the absence of uncertainty, the problem has been studied using calculus of variations [1] and dynamic programming [2]. However, in many practical applications with limited computational power, it becomes difficult or infeasible to solve due to the curse of dimensionality [2]. This is further exacerbated if there are unknown system and/or cost parameters. As a result, control designers rely on approximate or suboptimal methods [3] to solve the problem. If there are adequate computational resources and an accurate simulator of the true system, the problem can be solved up to an arbitrary accuracy using approximate dynamic programming [4] or reinforcement learning [5] techniques. When this is not the case, e.g. the system has unpredictable dynamics or the cost to be optimized for is changing adversarially, offline methods alone are not sufficient. In such cases, the input or the policy are updated online or adaptively as more data becomes available. Examples of such suboptimal online methods include adaptive controllers [6, 7], receding horizon controllers [8, 9], online control methods [10, 11] or online feedback optimization methods [12, 13].
![Refer to caption](x1.png)
Suboptimal online algorithms become a necessity driven by practical requirements. This motivates research on the performance of such methods, especially in the finite-time or transient domain. Given their implementation in real-time implementation, suboptimal algorithms need to stay computationally efficient while stabilizing the system. Additionally, their performance is measured in terms of the accumulated cost that needs to be kept to a minimum. To quantify this, we fix a benchmark policy that we deem to be close to the desired optimal one, visualized in Figure 1, and study the suboptimality gap of the given algorithm in terms of the additional incurred cost due to suboptimality. Such an analysis provides a relative measure on the performance of the given algorithm, since, in general, the benchmark policy attains a non-zero cost. In this context, we pose the following questions.
-
1.
How does the transient cost performance of an online algorithm scale with a measure of its suboptimality?
-
2.
How should the benchmark policy be chosen to achieve computable and meaningful finite-time bounds?
We consider nonlinear time-varying systems and choose a benchmark policy that renders the closed-loop system exponentially incrementally input-to-state stable (E--ISS). Incremental input-to-state stability has been introduced in [14], in the continuous-time setting and later analyzed in discrete-time in [15]. As opposed to input-to-state stability, E--ISS provides a condition on the deviation of two separate trajectories of the same system. A sufficient condition for E--ISS to hold is that of contraction [16, 17, 18]. When the dynamics are smooth, contraction can be verified by checking uniform negative definiteness conditions [17, 16]. However, smoothness often does not hold for the closed-loop dynamics; this is the case, for example, for linear time-invariant systems in closed-loop with a constrained Model Predictive Controller (MPC) [19]. Here we consider general nonlinear systems that are smooth only in an arbitrarily small region around the origin. Contraction can also be asserted, at least in the continuous-time case, by ensuring the closed-loop system is one-sided Lipschitz continuous [20, 18]. We do not explore this condition for our use cases and instead start with the assumption that the closed-loop dynamics under the benchmark policy are exponentially stable, which is often easier to verify.
There are several notable examples of settings where such finite-time suboptimality analysis of an online algorithm can be applied. These include suboptimal MPC, e.g. [8, 21, 22], when the suboptimality is due to finite computational resources, adaptive controllers with transient performance guarantees, e.g. [6, 11], with suboptimality due to unknown system parameters, or online feedback optimization [12, 13] and online control [10, 11], with suboptimality due to unknown future costs. In this work, we study the suboptimal linear quadratic MPC (LQMPC) setting in detail and show how a nonlinear incremental stability analysis can be used to provide a tighter bound on the suboptimality gap as compared to the one derived in [23].
Our contributions are summarized below:
-
a)
We derive sufficient conditions under which exponential stability (ES)of a non-smooth nonlinear time-varying system implies E--ISS, making the condition on the benchmark policy easier to verify,
-
b)
We show that if the closed-loop dynamics in closed-loop with the benchmark policy are E--ISS, then the suboptimality gap scales with the pathlength of closed-loop suboptimal trajectory,
-
c)
We study suboptimal LQMPC as an example satisfying these assumptions.
Our bounds are asymptotically tight, in the sense that they scale with the level of suboptimality of the given algorithm, converging to zero when the algorithm matches with the benchmark. The bounds also scale with the pathlength of the suboptimal trajectory, allowing an on-the-go calculation of the suboptimality gap that is independent of the optimal states. Moreover, our result is independent of the asymptotic properties of the suboptimal algorithm, providing finite-time performance bounds even when the closed-loop is not exponentially stable.
The article is structured as follows. In Section 2 we provide the preliminaries and the problem setup. In Section 3, we conduct the suboptimality gap analysis. Sufficient conditions for E--ISS are derived in Section 4, and in Section 5, the suboptimal MPC use case with a numerical example is studied.
Notation: The sets of positive real numbers, positive integers and non-negative integers are denoted by , and , respectively. For a given vector , its Euclidean norm is denoted by , and the two-norm weighted by a by . For a square matrix , the spectral radius and the spectral norm are denoted by , and , respectively. Given , and denote the minimum and maximum eigenvalues of ; for any vector , they satisfy . The Euclidean point-to-set distance of a vector from a nonempty, closed, convex set is denoted by , and the projection onto it by .
2 Preliminaries and Problem Setup
We consider the optimal control problem of discrete-time, nonlinear time-varying systems of the form
(1) |
where and denote the state and control input at time , respectively, denotes the unforced nominal dynamics and the controllable dynamics. Given an initial state , the optimal control objective is to find the sequence of control inputs that minimizes the finite-time cost
(2) |
where is the stage cost at time . In addition, the control input has to satisfy for all , for some bounded .
Admissable policy maps the current state and time to a control input, generating the control signal and the associated trajectory for a horizon of length . With a slight abuse of notation, its associated cost is denoted by . We consider time-varying systems for generality, and for the introduction of several novel results. Our analysis extends to time-invariant systems directly, as we showcase in Section 5.
We are interested in the relation of a policy , corresponding to a given suboptimal algorithm, with respect to another benchmark policy that is equipped with desirable characteristics, such as optimality. The latter is often obtained as the solution of some optimization problem. The two policies are defined as follows.
Benchmark dynamics: Consider a benchmark policy . In particular, given a , the benchmark dynamics are given by111For readability, we place the time in the subscript of .
(3) |
for all . We assume that the closed-loop dynamics (3) define a forward invariant set , and restrict attention to ; hence for all .
Suboptimal dynamics: The suboptimal state evolution for a given policy can be represented in the following form222We drop the explicit reference to from the superscript of for readability. for any
(4) |
for all . The map** can be thought of as a state-dependent disturbance acting on the optimal state dynamics (3), introduced due to suboptimality. It is assumed to be such that the closed-loop suboptimal dynamics (4) define a forward invariant set ; hence, restricting attention to , for all .
Figure 1 shows the pictorial evolution of the two considered trajectories starting from the same initial condition. For each , denotes the control input generated by and for each , the input generated by the suboptimal policy.
To quantify the relation between and , we define the suboptimality gap of the policy as the additional incurred cost compared to the benchmark policy
(5) |
given some . While closed-loop properties such as asymptotic or exponential stability convey information about the policy’s behavior in the limit, an informative bound on (5) would also capture its transient behavior. Hence, the finite-time analysis and derivation of upper bounds for (5), can provide a quantifiable tradeoff between the effort needed to compute the suboptimal policy and the additional cost incurred by using it instead of .
We assume, the benchmark policy, , itself has good performance, since otherwise, can be uninformative. We characterize this performance in terms of E--ISS.
Definition 1.
A dynamical system is said to be E--ISS in some forward invariant , if there exist and , such that for any , and , the perturbed dynamics satisfy
where the disturbances are such that for all . If the system is called globally E--ISS.
E--ISS for continuous-time systems has been introduced in [14]. For an in-depth discussion, analysis, and comparison of incremental stability, contraction, and convergent dynamics [24] in discrete-time we refer the reader to [15]. We assume the following for the benchmark policy.
Assumption 1.
The Lipschitz condition is standard in the nonlinear control literature, excluding policies with abrupt changes. The second assumption limits how fast the benchmark policy changes given the same state between two timesteps. Since is bounded, such an always exists, for all , and can be set equal to the diameter of . However, it can also encode additional information, such as stationarity of the benchmark policy, in which case we can take for all . The condition of E--ISS is used in deriving the bounds for the suboptimality gap. In Section 4, we show that under further mild conditions on , exponential stability (ES) is enough to guarantee E--ISS.
Next, we impose a time-varying contractivity condition on the suboptimal policy .
Assumption 2.
(Suboptimal Policy) Given the closed loop system (4), there exist , such that for all , and some the suboptimal policy satisfies
where .
![Refer to caption](x2.png)
Assumption 2 imposes a contractivity-like condition on the suboptimal policy evaluated on the suboptimal trajectory, as visualized in Figure 2. In words, it implies that the input generated by at time is closer to the optimal input generated by at the same state, compared to the input generated by at the previous timestep and the preceding state. In some cases, the contraction constant, of the suboptimal policy can be thought of as a design parameter that can be tuned to control the desired level of suboptimality depending on the available computational budget. Gradient descent-based methods where the suboptimal policy performs a finite number of iterations of an optimization problem [25] is a notable case where Assumption 2 holds. We direct the reader to Section 5 for further details on this.
We restrict our attention to systems where the controllable dynamics, , are Lipschitz continuous with respect to uniformly in and .
Assumption 3.
There exists a , such that for any , for all
This is satisfied, for instance, in linear time-invariant systems or nonlinear systems in certain feedback-linearizable forms (see [26, 27] for details). Finally, we restrict our analysis to local Lipschitz continuous stage costs.
Assumption 4.
(Stage costs). For all there exist , such that for all and
3 Suboptimality Gap Analysis
In this section, we analyze the suboptimality gap for a given policy, and show that scales with the product of the path length of the suboptimal dynamics and a vector dependent on the contractive constants. We define the backward difference path vector, , to be
where , where is the state at time for the suboptimal dynamics (4). The path length of the suboptimal trajectory is then defined as and the Euclidean path length as .
The policy contraction rate vector, is defined as
where
Note that and provides a weighting on the influence of the on . This is analyzed further in Section 3.2. We denote the Euclidean norm of the suboptimality vector by . The rate of change of the optimal input is captured by the vector , defined as
3.1 Upper Bound
The bound in the following theorem captures the tradeoff between suboptimality and the additional cost in closed-loop.
Theorem 1.
The bound in the theorem tends to zero as decreases. This is intuitive, as smaller suggests that the benchmark and suboptimal trajectories are closer to each other. Additionally, the suboptimality gap is a relative measure, but the bound is fully decoupled from the performance of and only depends on the performance of the suboptimal state evolution if . The above bound can also be represented in terms of the pathlength, . The path length, , captures the transient behavior of the suboptimal system and is well-defined in the limit as , for example when (4) is exponentially stable.
Before we prove Theorem 1, we introduce the following supporting lemmas. In the subsequent proofs, we make use of the Cauchy Product inequality defined for two finite series and
(6) |
Proof.
For all , define . Then
where the inequality follows directly from Assumption 2, and follow from the triangle inequality for vector norms and from the uniform Lipschitz condition in Assumption 1.i. and Assumption 1.ii.. Applying the above inequality recursively leads to
for all . Summing
where the second inequality follows from Assumption 2, by denoting , and the equality from the definition of and . ∎
The following lemma provides an upper bound for the finite-time suboptimality due to trajectory mismatch.
Lemma 2.
Proof.
Given the boundedness of , the uniform Lipschitz continuity of in , and recalling the definition of from (4), it follows that there exists a , such that . Then, under Assumption 1, and considering (4) to be the perturbed version of the optimal dynamics (3), there exist and , such that for all and
where the second inequality follows from Assumption 3, by recalling . Summing up over the whole trajectory and noting the resultant finite geometric series
where we have used the Cauchy product for the last inequality. Finally, the result follows by using the bound in Lemma 1. ∎
Similarly, the suboptimality due to the difference in applied inputs can be bounded in the following Lemma.
Lemma 3.
Proof.
Using the triangle inequality for vector norms and defining
where the last inequality follows from Lipschitz continuity of from Assumption 1.i.. Summing up over the trajectory horizon and using the results from Lemmas 1 and 2 completes the proof. ∎
Proof of Theorem 1.
By Assumption 4, for all
Then, using Lemmas 2 and 3 for the two respective sums
The result follows by taking . ∎
For the special case when the contraction rate of the suboptimal policy is constant, the bound in Theorem 1 can be simplified.
Corollary 1.
Proof.
In the special case when , and is bounded by
The complexity term then satisfies
The rest of the proof follows directly from Theorem 1 by replacing the complexity term with the new bound. ∎
3.2 Interpretation of the Upper Bound
The term in the bound of Theorem 1 captures the error due to the initial mismatch in the control input, . This term in general cannot be avoided, unless the initial “guess” of the input is correct, or , so that the suboptimal and optimal policies match at the initial timestep.
The second term, , scales with the magnitude of the rate of change of the time-varying benchmark policy , as defined in Assumption 1.ii.. It vanishes either when is stationary, or when the benchmark and suboptimal policies coincide.
![Refer to caption](x3.png)
![Refer to caption](x4.png)
The main complexity term of interest is the last one. This captures the suboptimality of the policy through the inner product of the path vector and the suboptimality vector . To study the interplay of these two quantities in more detail, let us consider the case when the benchmark dynamics, (3) under the policy have an equilibrium at some . If the suboptimal policy makes the closed-loop (4) exponentially stable with , then the backward difference vector norm , for some , as visualised in Figure 2(a). In such a setting, the finite value of the suboptimality gap is captured by the complexity term as
(7) |
This example coincides with the suboptimal LQMPC use case discussed in detail in Section 5. Among other possibilities, one can also consider the case when the benchmark dynamics (3) converge to a limit cycle. Since (3) is E--ISS it follows from (4) that if at a given point in time , the suboptimal policy becomes optimal, i.e. , then the trajectories will necessarily coincide, as visualised in Figure 2(b). This is captured by the complexity term as
(8) |
Even though the path length keeps increasing, the norm of the suboptimality vector is finite, resulting in a finite suboptimality gap, containing only the additional cost due to suboptimality at the first timesteps.
4 Exponentially Stable Policies and E--ISS
In this section, we analyze the E--ISS property of the closed-loop system (3), and derive conditions under which the exponential stability of a nonlinear system implies E--ISS for non-smooth dynamics.
We treat (3) as a general nonlinear time-varying system of the form
(9) |
where is continuous with respect to both arguments and for some and . Note that by definition of , is forward invariant. The solution of the system (9) at time is characterized by the function map** the current time, initial time and the initial state to the current state, i.e. for all . We consider the origin to be an equilibrium point for (9), i.e. for all . Although this restricts the attention to regulation problems, one can convert a tracking problem into a regulation one given the form of the system (1). We impose the following assumption.
Assumption 5.
Remark 1.
The conditions on the Jacobian matrix of in Assumption 5 are required only in the time-varying case [26].
Before presenting the main results of this section, we present a series of auxiliary definitions and theorems for (incremental) exponential stability, for completeness.
4.1 Preliminaries on Exponential Stability
We formally define uniform exponential stability for discrete-time, nonlinear time-varying systems [27].
Definition 2.
If the origin is locally/globally exponentially stable, we also refer to the system (9) as such. Lyapunov theory provides necessary and sufficient conditions for the exponential stability of nonlinear systems. Below are the discrete-time Lyapunov theorems for exponential stability.
Theorem 2.
The converse Lyapunov theorem for the discrete-time case shows the implication in the opposite direction.
Theorem 3.
The above theorems generalize to uniform global exponential stability if [26, 29]. In continuous-time, the rate of change of the Lyapunov function with respect to the state is bounded by the norm of the state [26, Thm 4.14]. The following Lemma is the discrete-time equivalent of this bound.
Lemma 4.
The proof of Lemma 4 is provided in the appendix.
When the system is linear time-varying, the Lyapunov function has a quadratic structure.
Theorem 4.
[30, Thm. 23.3] The linear time-varying system
is uniformly exponentially stable, if and only if there exists a sequence of positive-definite, bounded matrices , , satisfying the following difference Lyapunov equation
(13) |
for some .
4.2 Preliminaries on Exponential Incremental Stability
Exponential incremental stability shows the exponential convergence of two trajectories generated by the same system to each other [15, 17].
Definition 3.
The nonlinear system (9) is uniformly semiglobally exponentially incrementally stable in if there exists a and , such that for all initial states and for all
In the case , the system is said to be uniformly globally exponentially incrementally stable.
The theory of exponential convergence of two trajectories of the same system has first been studied as uniform convergence by Demidovich [24, 31], and later extended through contraction theory [16, 17]. Contraction is a sufficient condition for exponential incremental stability, that when is smooth, can be checked by the following condition.
Theorem 5.
We also state the following converse Lyapunov theorem. The proof, extended from [14] and [15], is provided in the appendix.
Theorem 6.
If the system (9) is uniformly semiglobally exponentially incrementally stable in , then there exists a function and constants and , such that for all and
(15) | |||
(16) |
Moreover, under Assumption 5.i., there exists a constant , such that for all and
(17) |
4.3 Main Results
In this subsection we show that if the nonlinear dynamics (9) are uniformly locally exponentially stable in a given forward invariant region and satisfy Assumption 5, then they are also E--ISS in the same region. First we show that under the local Lipschitz continuity assumption, exponential incremental stability implies E--ISS.
Theorem 7.
Under Assumptions 5.i. and 5.ii., if the nonlinear system (9) is uniformly semigobally exponentially incrementally stable in , then it is E--ISS in the same region.
Proof.
Given the nonlinear system (9), consider the evolution of two, respectively unperturbed and perturbed trajectories
for some , where , for some is such that for all . Given is uniformly semiglobally exponentially incrementally stable in , then from Theorem 6, there exists a Lyapunov function satisfying (15)-(17). Then, for any , and
where the first inequality follows from Theorem 6, and the second from the triangle inequality. Completing the square, and denoting it follows from above that
where the second inequality follows by the fact that for any , and the last inequality from Theorem 6. Finally, rearranging the Lyapunov equations it follows that
for all , with , since , and . Unrolling the recursion and using (15)
Dividing by and taking the square root, completes the proof
∎
Next we show, that if the following theorem states that if in addition to local Lipschitz continuity, the nonlinear dynamics are also locally differentiable in some arbitrarily small region, , around the equilbrium, then uniform local exponential stability implies uniform semiglobal exponential incremental stability and E--ISS by Corollary 2.
Theorem 8.
Under Assumption 5, if the nonlinear system (9) is uniformly locally exponentially stable in , then it is also uniformly semiglobally exponentially incrementally stable in the same region.
Proof.
We start by showing that the exponential stability of implies that the linearized dynamics around the origin are also stable by following similar arguments to [26]. Let
which is well-defined given Assumption 5. Moreover, there exists a , such that , . It follows from Theorem 3, that there exists a continuous map** satisfying (11)-(12). Let us consider as a candidate Lyapunov function for , then for all , there exist constants , such that
where the first inequality follows from Theorem 3, the second from Lemma 4 and the last from Definition 2 and properties of induced norms. Denoting , it follows from the Lipschitz continuity of the Jacobian of [26, Chpt. 4.6] that there exists a , such that , . Using this
where the second inequality follows from the converse Lyapunov Theorem 3 for some . Defining , for all , it holds that . Hence, using Theorem 2 the linearized dynamics are uniformly locally exponentially stable. Then, by Theorem 4 there exists a sequence of uniformly positive definite that solves the difference Lyapunov equation (13) for some .
Considering now equation (14), for any and if we denote then
where the first equality follows from the definition of and the last one from Theorem 4.
Following the same arguments as in [26, Chpt. 4.6], there exists a , such that for all and , . Pre- and post-multiplying the above with some and , respectively then yields
where , and . Note that the rate of exponential stability of the linear system is . Then, choosing , adding to both sides of the above inequality and defining ensures that
uniformly in and , for all . This implies by Theorem 5 that for all the system (9) is uniformly semiglobally exponentially incrementally stable with rate of .
To show that the system is also uniformly semiglobally exponentially incrementally stable in , consider any , then for all , and some , the following two inequalities hold
(18) | ||||
(19) |
The bound in (18) follows from the exponential stability of , and the one in (19) from its Lipschitz continuity. Combining the two
Define such that both . Then, from the above analysis, there exists a , such that for all for all
It then follows that
Note that for all
where
where is a constant independent of since is finite and also independent of it. Combining the bounds
which is the definition of uniform semiglobal exponential incremental stability. ∎
Combining Theorems 7 and 8 the following corollary follows directly.
Corollary 2.
Under Assumption 5, if the nonlinear system (9) is uniformly locally exponentially stable in , then it is also E--ISS in the same region.
In the sequel we use these insights to address the closed-loop dynamics under a specific, notable policies.
5 Model Predictive Control - A Use Case
In Section 3 we showed that under certain assumptions on the suboptimal policy (Assumption 2) and the benchmark policy (Assumption 1), the suboptimality gap of can be bounded for a certain family of costs. A key condition on the benchmark policy is that of E--ISS. We now exploit the results of Section IV to study the case of linear quadratic MPC.
Consider the control of linear time-invariant dynamical systems, modeled by
where , and are the known system matrices. We consider the finite horizon linear quadratic regulator (LQR) whose objective is to minimize the finite-time quadratic cost
(20) |
where and are design matrices and is taken to be the solution of the discrete Algebraic Riccati Equation, , with , and the control inputs must satisfy for all , where is a constraint set. The following standard assumptions ensure a unique minimizer for (20) always exists [32].
Assumption 6.
(Well posed problem)
-
i.
The pair is stabilizable, , .
-
ii.
The input constraint set is compact, convex, and contains the origin.
The model predictive controller solves this problem in a receding horizon fashion, solving the following parametric optimal control problem (POCP) at each timestep , having measured a state
(21) |
Here is the prediction horizon length, and denotes the predicted input vector. We refer to the minimiser of (21) for a given initial state (parameter) , , as the optimal map**. In this setting, solving the above POCP to optimality is taken to be the benchmark policy. The optimal cost attained by this map** is denoted by , which serves as an approximate value function for the problem. For each , the model predictive controller applies the first element of to the system, and the process is repeated in a receding horizon fashion. The optimal state evolution under this optimal MPC policy is then given by
(22) |
where , , and is the selector matrix. Note that the optimal MPC policy, is time-invariant due to the structure of the problem.
Problem (21) is a parametric quadratic program and for a given parameter can be represented in an equivalent condensed form , where
(23) |
and the definitions of , and, can be found in [33].
As the optimal may often be prohibitive to compute exactly, suboptimal schemes are often considered. In our setting, a suboptimal policy is computed by performing only a finite number of optimization steps for (21). In particular, given and an input vector , consider the operator that performs one step of the projected gradient method
(24) |
where is a step size. Applying (24) iteratively times provides an approximation for the optimal input, and hence the optimal policy. The combined dynamics of the system and the approximate optimizer are then given by333The subscript of is dropped when it is taken to be a constant.
(25a) | ||||
(25b) |
where is an initialization vector, and for some and , we define
(26) |
The dynamics under the suboptimal policy are (25b) by taking for all , i.e.
Note that is also a function of the previous input state . However, since the closed loop evolution is noise free, it can be uniquely determined given the initialization vector, the current time and the current state. Hence, the dependence on is encoded in the subscript of . The suboptimal policy can in general be defined as a function of the information vector ; as long as Assumption 2 is satisfied, the results in this manuscript hold.
5.1 Optimal MPC
In this subsection, we review the properties of the optimal map** . As shown in [34, 35], system (3) is asymptotically stable with the forward invariant ROA estimate
where , , and is such that the following set is non-empty
Moreover, as shown in [21] the closed loop system (3) is exponentially stable in with an explicit formulation for the decay rate derived in [23]. The function is a Lyapunov function for the optimal MPC algorithm, satisfying
(27) | |||
(28) |
where is the exponential decay rate. The Lipschitz continuity of the optimal map** is formalized in the following lemma.
Lemma 5.
5.2 Suboptimal MPC
The suboptimal policy in this setting is defined by (5). The following well-established result shows the linear rate of convergence of the PGM method.
The suboptimal MPC scheme is treated by considering the combined evolution of the system-optimizer dynamics (5). The stability of such a scheme, also referred to as TD-MPC or as real-time implementation of MPC is shown [21, 22, 33] for a fixed number of iterations and in [23] for a time varying . In particular, if for all , where
where is the same as in Section 5.1, , , and
Then, the dynamics (5) are exponentially stable in the following forward invariant ROA estimate
recalling from Section 5.1. The exponential stability result from [23] is summarized in the following Lemma.
If the same number of optimization iterations are taken at all times the bound reduces to the following.
5.3 Suboptimality Gap
We define the suboptimality vector, , for this use case as
where , and . We denote the Euclidean norm of the suboptimality vector by . The main result for a suboptimal LQ-MPC scheme is summarized in the following Theorem.
Theorem 10.
Proof.
We start by showing that Assumptions 1-4 are satisfied in the LQ-MPC use-case. In this setting, the linear dynamics to be controlled are given by
(29) |
with the input . First, we note that Assumption 3 is satisfied trivially with . The benchmark controller is the optimal policy that solves the POCP (21), and is given in (22).
Taking to be the forward invariant ROA estimate , it follows from Lemma 5 that Assumption 1.i. is satisfied. To show that the Assumption 1.iii. is satisfied, we use the analysis from Section 4. Exploiting the structure of the optimization problem (21), it has been shown by Bemporad et. al. [19] that the solution of the LQ-MPC problem is piecewise-affine in the state, under Assumption 6. Moreover, there is a polyhedral non-empty set around the origin where the solution is affine in the state [19, Corollary 2]. This and Lemma 5 imply that Assumption 5 is satisfied, and hence, by Corollary 2 since the optimal solution is exponentially stabilizing [21, 23], we conclude that it is also E--ISS. Since the policy is time-invariant, Assumption 1.ii. is satisfied trivially with .
The suboptimal policy is given by (5). The results in [23] show that for all , is a forward invariant ROA estimate for the combined dynamics (5). Given this and an initial , consider . Then Assumption 2 is satisfied directly from Theorem 9.
Finally, to reconcile the quadratic cost defined in (20) and the modified dynamics (29), we redefine the cost, as
where . For the quadratic costs in (20), and for any , and
where and are such that, and for all . Then the condition in Assumption 4 is satisfied with and .
Since 6 implies Assumptions 1-4 are satisfied, we can invoke the bound in Theorem 1 for the suboptimality gap. As the suboptimal dynamics are exponentially stable, its path length is finite. In particular
where the first inequality follows by the triangle inequality, the second from Lemma 6 and by denoting , and the last one by bounding the geometric series and denoting . Noting that in this example, the suboptimality gap is bounded by .
The theorem shows that the suboptimality gap of the LQ-MPC suboptimal controller (5) scales with or , where the number of iterations are design parameters. Note that the higher the more computation is required at each timestep, but the lower the suboptimality gap; in the limit, as , the suboptimality gap is zero. This is a tighter result than the one derived in [23], as in the latter no incremental properties of the optimal controller are used. Specifically, when looking at the limit case of , the suboptimality gap in [23] is strictly positive, while the bound in Theorem 10 vanishes, reflecting the exact matching of the suboptimal and benchmark trajectories. The derived bounds can be used by control designers to give a quantifiable measure of the finite-time suboptimality of the controller. This can then be utilized to find the best sequence of to deliver a desired tradeoff between suboptimality and computational power.
In practice, the suboptimal MPC can be asymptotically stable even when the number of optimization iterations is less than . In this case, the existence of a forward invariant region of attraction given, but the bounds in Theorem 1 and Corollary 1 still hold, as long as the closed-loop system stays recursively feasible in practice. This is shown in the following numerical example.
5.4 Numerical Example
The suboptimal TD-MPC scheme described in this section is implemented for the following linearized, continuous-time model of an inverted pendulum from [35], [23]
where the state is , is the angle relative to the equilibrium position and the control input is the applied torque. We consider the control of the discretized model of the plant with a sampling time of . The input constraint set is taken to be , the cost matrices are , and and the initial state is .
![Refer to caption](x5.png)
The left panel of Figure 4 shows the evolution of two trajectories in closed-loop with the TD-MPC policy with and with an optimal MPC. For this example . However, even with the low value of the closed-loop system stays stable, as also observed in [33, 35]. Although the asymptotic/exponential stability cannot be proven, the finite time bounds can still be computed online using only the suboptimal states as per Corollary 1. The order of this upper bound, , as well as the empirically observed suboptimality gap, , are plotted in the right panel of Figure 4 for for a range of values of , increasing from to . The decrease of the suboptimality gap for increasing values of is juxtaposed with the increase of simulation/computational time in the same figure. The simulation time for each is calculated as the sum of the times it takes to solve the TD-MPC for each timestep, over the horizon . To obtain an averaged value for this time, its average over repeated independent runs from the same initial conditions is taken. The initial states are intialized in following the procedure described in [35]. In the right panel of the figure, only the order of the suboptimality gap upper bound is plotted. The true constant is much larger; our aim here is not to compare the very conservative theoretical bound with the practical performance, but to give an estimate whether the bound captures the order correctly. And indeed, it can be observed from the figure that the order of the true suboptimality gap is approximately captured by the upper bound with an underestimation.
Among other possible uses of the closed-loop suboptimality analysis, the insights in the figure can be used to design the allocation of finite computational resources. The right-side plot in the figure can be used to estimate the relative gain in computational time and loss in optimality for a given change in . For example, a change of to , or equivalently to results in a times increase in computational time and a times decrease in the suboptimality gap bound, a conservative estimate of the true suboptimality gap.
6 Conclusions
We study the finite-time suboptimality gap of policies for discrete-time nonlinear, time-varying systems. We show that when the benchmark policy is chosen to be exponentially incrementally stable, then given a geometric improvement condition on the suboptimal policy, its suboptimality gap scales with its path length and improvement factor. We further show, that for non-smooth nonlinear systems E--ISS is implied by exponential stability under certain conditions. The assumptions are verified on the suboptimal linear quadratic MPC use case and on a numerical example. The generality of the provided analysis enables the study of other examples where the suboptimality is due to unknown system parameters or cost functions, for example in the fields of adaptive and online control. We leave the analysis of these use cases to future work.
Proof of Lemma 4
Proof.
Given a state and some , let
Then
and, from Definition 2
Thus, (11) is satisfied with and . To show that (12) holds, consider
Choosing large enough such that ensures (12) hold with since and . Finally, for some sand , denote and consider
where the last inequality follows from the local exponential stability and -Lipschitz continuity of the nonlinear map**. Taking completes the proof. ∎
Proof of Theorem 6
The proof hinges on extending results from [29],[36], [14] and [15] . We provide all the definitions and lemmas required for the proof of Proposition 7 first.
Definition 4.
The extension of the converse Lyapunov results from [14] and [29] to the case of semiglobal stability is included below for completeness.
Theorem 11.
If the system (9) is uniformly semiglobally exponentially stable with respect to some closed set and a ROA , then there exists a function and constants and , such that
(30) | ||||
(31) |
Proof.
Following the same line of argument from the proof of Lemma 4, consider a and some and let
(32) |
for some . Then,
and, from Definition 4
Next, consider
Choosing large enough such that completes the proof with , , and since and . ∎
Proof of Theorem 6.
Consider the augmented system
The diagonal is the set . Let , then it is shown in [14] that
(33) |
Then, considering the evolution of the combined system
(34) |
one can note that for all , for any , since is forward invariant. It follows that system (9) is locally exponentially incrementally stable if and only if (34) is locally exponentially stable with respect to the diagonal .
Moreover, using Theorem 11, the combined system (34) admits a Lyapunov function satisfying (30)-(31). It follows from the equivalence in (33), and the proof of Theorem 11, that (15)-(16) are satisfied with , and .
To show the inequality (17), we consider the same Lyapunov function (32) used to show (15)-(16) with , i.e. given a
where is the state transition matrix for the system (34). Then, for any , , and defined as above
where the last two inequalities follow from the Lipschitz continuity and exponential stability (with respect to ) of the solutions. The first inequality follows from the following. For any two vectors and closed set
where is such that , which exists and is unique, since is closed. Finally, choosing completes the proof. ∎
References
- [1] L. S. Pontryagin, Mathematical theory of optimal processes. CRC press, 1987.
- [2] R. Bellman, “Dynamic programming,” Science, vol. 153, no. 3731, pp. 34–37, 1966.
- [3] D. Bertsekas, Lessons from AlphaZero for optimal, model predictive, and adaptive control. Athena Scientific, 2022.
- [4] D. Bertsekas, Dynamic programming and optimal control: Volume I, vol. 4. Athena scientific, 2012.
- [5] R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction. MIT press, 2018.
- [6] N. Hovakimyan and C. Cao, adaptive control theory: Guaranteed robustness with fast adaptation. SIAM, 2010.
- [7] K. S. Narendra and A. M. Annaswamy, Stable adaptive systems. Courier Corporation, 2012.
- [8] P. O. Scokaert, D. Q. Mayne, and J. B. Rawlings, “Suboptimal model predictive control (feasibility implies stability),” IEEE Transactions on Automatic Control, vol. 44, no. 3, pp. 648–654, 1999.
- [9] B. Kouvaritakis and M. Cannon, “Model predictive control,” Switzerland: Springer International Publishing, vol. 38, 2016.
- [10] Y. Li, X. Chen, and N. Li, “Online optimal control with linear dynamics and predictions: Algorithms and regret analysis,” Advances in Neural Information Processing Systems, vol. 32, 2019.
- [11] N. M. Boffi, S. Tu, and J.-J. E. Slotine, “Regret bounds for adaptive nonlinear control,” in Learning for Dynamics and Control, pp. 471–483, PMLR, 2021.
- [12] G. Belgioioso, D. Liao-McPherson, M. H. de Badyn, S. Bolognani, R. S. Smith, J. Lygeros, and F. Dörfler, “Online feedback equilibrium seeking,” arXiv preprint arXiv:2210.12088, 2022.
- [13] Z. He, S. Bolognani, J. He, F. Dörfler, and X. Guan, “Model-free nonlinear feedback optimization,” arXiv preprint arXiv:2201.02395, 2022.
- [14] D. Angeli, “A Lyapunov approach to incremental stability properties,” IEEE Transactions on Automatic Control, vol. 47, no. 3, pp. 410–421, 2002.
- [15] D. N. Tran, B. S. Rüffer, and C. M. Kellett, “Convergence properties for discrete-time nonlinear systems,” IEEE Transactions on Automatic Control, vol. 64, no. 8, pp. 3415–3422, 2018.
- [16] W. Lohmiller and J.-J. E. Slotine, “On contraction analysis for non-linear systems,” Automatica, vol. 34, no. 6, pp. 683–696, 1998.
- [17] H. Tsukamoto, S.-J. Chung, and J.-J. E. Slotine, “Contraction theory for nonlinear stability analysis and learning-based control: A tutorial overview,” Annual Reviews in Control, vol. 52, pp. 135–169, 2021.
- [18] F. Bullo, Contraction Theory for Dynamical Systems. Kindle Direct Publishing, 1.1 ed., 2023.
- [19] A. Bemporad, M. Morari, V. Dua, and E. N. Pistikopoulos, “The explicit linear quadratic regulator for constrained systems,” Automatica, vol. 38, no. 1, pp. 3–20, 2002.
- [20] A. Davydov, S. Jafarpour, and F. Bullo, “Non-euclidean contraction theory for robust nonlinear stability,” IEEE Transactions on Automatic Control, vol. 67, no. 12, pp. 6667–6681, 2022.
- [21] A. Zanelli, Q. T. Dinh, and M. Diehl, “A Lyapunov function for the combined system-optimizer dynamics in nonlinear model predictive control,” arXiv preprint arXiv:2004.08578, 2020.
- [22] D. Liao-McPherson, M. M. Nicotra, and I. Kolmanovsky, “Time-distributed optimization for real-time model predictive control: Stability, robustness, and constraint satisfaction,” Automatica, vol. 117, p. 108973, 2020.
- [23] A. Karapetyan, E. C. Balta, A. Iannelli, and J. Lygeros, “On the finite-time behavior of suboptimal linear model predictive control,” arXiv preprint arXiv:2305.10085, 2023.
- [24] B. P. Demidovich, “Lectures on stability theory (in Russian),” 1967.
- [25] A. B. Taylor, J. M. Hendrickx, and F. Glineur, “Exact worst-case convergence rates of the proximal gradient method for composite convex minimization,” Journal of Optimization Theory and Applications, vol. 178, no. 2, pp. 455–476, 2018.
- [26] H. Khalil, “Nonlinear systems, third edition, vol. 115,” Upper Saddle River, NJ, USA: Patience-Hall, 2002.
- [27] W. M. Haddad and V. Chellaboina, Nonlinear dynamical systems and control: a Lyapunov-based approach. Princeton university press, 2008.
- [28] Z.-P. Jiang, Y. Lin, and Y. Wang, “Nonlinear small-gain theorems for discrete-time feedback systems and applications,” Automatica, vol. 40, no. 12, pp. 2129–2136, 2004.
- [29] Z.-P. Jiang and Y. Wang, “A converse Lyapunov theorem for discrete-time systems with disturbances,” Systems & control letters, vol. 45, no. 1, pp. 49–58, 2002.
- [30] W. J. Rugh, Linear system theory. Prentice-Hall, Inc., 1996.
- [31] A. Pavlov, A. Pogromsky, N. van de Wouw, and H. Nijmeijer, “Convergent dynamics, a tribute to boris pavlovich demidovich,” Systems & Control Letters, vol. 52, no. 3-4, pp. 257–261, 2004.
- [32] D. Q. Mayne, J. B. Rawlings, C. V. Rao, and P. O. Scokaert, “Constrained model predictive control: Stability and optimality,” Automatica, vol. 36, no. 6, pp. 789–814, 2000.
- [33] D. Liao-McPherson, T. Skibik, J. Leung, I. Kolmanovsky, and M. M. Nicotra, “An analysis of closed-loop stability for linear model predictive control based on time-distributed optimization,” IEEE Transactions on Automatic Control, vol. 67, no. 5, pp. 2618–2625, 2021.
- [34] D. Limón, T. Alamo, F. Salas, and E. F. Camacho, “On the stability of constrained mpc without terminal constraint,” IEEE transactions on automatic control, vol. 51, no. 5, pp. 832–836, 2006.
- [35] J. Leung, D. Liao-McPherson, and I. V. Kolmanovsky, “A computable plant-optimizer region of attraction estimate for time-distributed linear model predictive control,” in 2021 American Control Conference (ACC), pp. 3384–3391, IEEE, 2021.
- [36] E. D. Sontag and Y. Wang, “New characterizations of input-to-state stability,” IEEE transactions on automatic control, vol. 41, no. 9, pp. 1283–1294, 1996.