Online Stackelberg Optimization via Nonlinear Control

A number of properties under the name “local controllability” have been considered for various continuous-time and asymptotic control settings (Aoki, 1974; Kuhn and Wohltmann, 1989; Barbero-Liñán and Jakubczyk, 2013; Boscain et al., 2021), generally relating to the notion that all states in a neighborhood around a given state are reachable. We give two formulations of local controllability for our setting, which we take as properties of the dynamics $D$ holding over all inputs.

We often refer to weak local controllability simply as local controllability. This property ensures that there is always some action $x_{t}$ which results in the next state $y_{t}$ staying fixed at $y_{t-1}$, as well as some action which moves the state to any point in a surrounding ball; in the weak case, the size of the reachable ball is allowed to decay as $y_{t}$ approaches the boundary of $\operatorname{\mathcal{Y}}$. The parameter $\rho$ controls the speed at which we can navigate the state space: when $\rho=1$ in the weak case (or $\rho\geq R$ in the strong case), we can always immediately reach some point on the boundary of $\operatorname{\mathcal{Y}}$, yet for $\rho$ close to zero we may only be able to move in a small neighborhood. Our results use local controllability to minimize regret over $\operatorname{\mathcal{Y}}$ by reduction to online convex optimization. As we prove in Appendix A, up to a quantifier alternation which vanishes as $\rho$ approaches $0$, a property of this form is essentially necessary: competing with the best state $y$ is impossible if we cannot remain in its neighborhood.

While regret benchmarks in online control are typically expressed in terms of a reference class of policies, we note that there is a class of “state-targeting” policies which track the reward of fixed states (asymptotically, and up to the influence of disturbances), and which can be implemented if $D$ is known; we maintain the formulation in terms of fixed states for clarity with respect to our motivations for Stackelberg optimization. Existing no-regret algorithms for online control typically compete with linear policies, and choose actions each round by implementing policies which are linear in multiple past states (as in e.g. Agarwal et al. (2019a)). Here, we show that all such policies can be arbitrarily suboptimal when compared to state-targeting policies, even for dynamics which are linear up to projection and with fixed convex losses over states, as they may yield actions and states which remain fixed at $\mathbf{0}$ in every round even if the optimal state is always immediately accessible under the dynamics. We prove Proposition 2 in Appendix A.

If dynamics are linear up to projection with $D(x_{t},{y_{t-1}})=\Pi_{\operatorname{\mathcal{Y}}}(By+Ax)$ for full-rank $A$, and $\dim(\operatorname{\mathcal{X}})=\dim(\operatorname{\mathcal{Y}})$, note that $P_{\hat{y}}(y)=A^{-1}(\hat{y}-By)$ implements any $P_{\hat{y}}$ for sufficiently large $\operatorname{\mathcal{X}}$.

When dynamics satisfy local controllability and $y_{t-1}$ is not too close to $\operatorname*{bd}(\operatorname{\mathcal{Y}})$, all points $y_{t}$ in a ball around $y_{t-1}$ are feasible with an appropriate $x_{t}$; this enables execution of an online convex optimization (OCO) algorithm over $\operatorname{\mathcal{Y}}$ by playing the action $x_{t}$ which yields a state update to the target $y_{t}$ chosen at each iteration, computed via offline non-convex optimization. Here we assume that $D$ is known and can be queried for any inputs, and that disturbances to the state are not present. We allow the dynamics to change over time, potentially as a function of previous actions $x_{s}$ and losses $f_{s}$ for $s<t$, provided that $D_{t}$ can be determined in each round. We use Follow the Regularized Leader ($\operatorname{\textup{{FTRL}}}$) as our OCO subroutine (Shalev-Shwartz and Singer, 2006; Abernethy et al., 2008), yet we note that it may be substituted for any OCO algorithm whose per-round step size is guaranteed to be sufficiently small (such as OGD with a constant learning rate); statements of the $\operatorname{\textup{{FTRL}}}$ algorithm and its key properties are provided in Appendix B. We instantiate $\operatorname{\textup{{FTRL}}}$ over a contracted space $\tilde{\operatorname{\mathcal{Y}}}\subseteq\operatorname{\mathcal{Y}}$, calibrated to ensure that the minimum loss over $\tilde{\operatorname{\mathcal{Y}}}$ is close to that for $\operatorname{\mathcal{Y}}$, yet where each step of $\operatorname{\textup{{FTRL}}}$ lies within the feasible region ensured by (weak) local controllability.

The proof for Theorem 1 is given in Appendix C.

While $\operatorname{\textup{{NestedOCO}}}$ requires no assumptions on the dynamics beyond local controllability, there are large classes of dynamics for which the oracle call can be removed. We say that dynamics are action-linear if $y_{x}=D(x,y)$ is linear in $x$, for $y_{x}\in\operatorname*{int}(\operatorname{\mathcal{Y}})$ (and arbitrary for $y_{x}\in\operatorname*{bd}(\operatorname{\mathcal{Y}}))$.

• Proof

  For $y=y_{t-1}\in\widetilde{\operatorname{\mathcal{Y}}}\subseteq\operatorname*{int}(\operatorname{\mathcal{Y}})$, we have $D(x,y)=A_{y}\cdot x+b_{y}$ for some matrix $A_{y}$ and vector $b_{y}$, and so we can solve $x_{t}=\operatorname*{argmin}_{x\in\operatorname{\mathcal{X}}}\left\lVert A_{y}\cdot x+b_{y}-y^{*}\right\rVert^{2}$ efficiently. ∎

The class of action-linear dynamics is quite general, owing to the flexibility permitted by nonlinear parameterizations of $(A_{y},b_{y})$ in terms of $y$; in Appendix D, we show that local controllability holds for multiple explicit families of instances when appropriate eigenvalue conditions are satisfied. We can further relax this condition to accommodate dynamics where action-linearity holds only locally in the neighborhood of stabilizing actions (i.e. actions $x^{*}$ where $D(x^{*},y)=y$).

By this condition, for any $x$ in a sufficiently small neighborhood around $x^{*}$, the deviation of dynamics (and thus the resulting $y_{t+1}$) from action-linearity vanishes. Note that our algorithm always chooses a target $y_{t}$ will always be near $y_{t-1}$; as such, these deviations from non-action-linearity can be modeled as disturbances with magnitude strictly less than our per-round step size $\left\lVert y_{t+1}-y_{t}\right\rVert$ (along with universal constant factors). The existence of an efficient implementation follows as a straightforward corollary of Theorem 2 in Section 3.3, which extends $\operatorname{\textup{{NestedOCO}}}$ to accommodate bounded adversarial disturbances, as we can then select actions by disregarding the influence of $q_{y}$ and only considering the local approximation $D(x,y)=A_{y}x+b_{y}$ at each state $y$ (assuming that each decomposition between $q_{y}$ and the action-linear component is known).

Our algorithm $\operatorname{\textup{{NestedOCO}}}$ can be extended to accommodate adversarial disturbances, where the state is updated as $y_{t}=D(x_{t},y_{t-1})+w_{t}$, with $\{w_{t}\}$ chosen adversarially. In the weak local controllability case, we show a sharp threshold effect in terms of whether or not $\left\lVert w_{t}\right\rVert$ is allowed to exceed the undisturbed distance from the boundary by a factor of $\frac{\rho}{1+\rho}$: if disturbances are bounded below this threshold, regret minimization remains feasible with a tight $\Theta(E)$ dependence on the total disturbance magnitude, yet if disturbances may exceed this, no sublinear regret rate is attainable even for a constant total disturbance magnitude. When $\rho$ is small, an adversary can push us to the boundary faster than we can “undo” past disturbances, causing our feasible range to decay.

The maximum disturbance bound can be removed when dynamics are strongly locally controllable, as the ensured feasible range of the dynamics does not vanish at the boundary of the state space. For such instances, we can minimize regret (with tight $O(E\cdot\rho^{-1})$ dependence) even if disturbances are only implicitly bounded by the state space diameter (which is at least $\rho$, without loss of generality).

In each case, our lower bounds in terms of $E$ hold for the same constants obtained by our algorithms, and our algorithms obtain the stated regret guarantees even when $E$ is not known in advance. We present the algorithms and analysis for each theorem in Appendix E; both operate by tracking deviations from an idealized trajectory without disturbances, and calibrating parameters to preserve sufficient reachability margin for applying corrections towards this trajectory in each round. The lower bounds both proceed by considering an instance with a fixed target state $y^{*}$ and losses which track the distance from $y^{*}$, along with an adversary whose goal is to maximize this distance by selecting disturbances which push the current state away from $y^{*}$.

Up until this point, we have assumed that the dynamics $D$ can be queried arbitrarily in each round. While this has required minimal assumptions on $D$ beyond local controllability, accommodation of unknown dynamics is often desired in online control (Cassel et al., 2022; Minasyan et al., 2022) and for several of our applications (Roth et al., 2015; Agarwal and Brown, 2023). Here we give conditions under which regret minimization can be implemented without advance knowledge of $D$ by an algorithm $\operatorname{\textup{{ProbingOCO}}}$, which maintains continuously-updating local linear approximations of $D$ near $y_{t}$ across rounds. Crucially, we assume that $D$ is time-invariant and locally action-linear with sufficiently small Lipschitz parameters, and that for the initial state $y_{0}$ some near-stabilizing action $x_{1}$ is known, i.e. $\left\lVert D(x_{1},y_{0})-y_{0}\right\rVert\leq\epsilon$, for some $\epsilon=o(\sqrt{T})$.

We state $\operatorname{\textup{{ProbingOCO}}}$ and prove Theorem 4 in Appendix F, along with additional details on the regularity and near-stability assumptions. The crux of our analysis, beyond that from our previous results, hinges on being able to maintain and update local linear approximations of $D$ throughout our optimization which are sufficiently accurate to allow us to discard the effects of both learned representation errors and action non-linearity from $q_{y}(x)$ as bounded disturbances. We implement each update from our nested regret minimization algorithm as a series of $O(\dim(\operatorname{\mathcal{X}}))$ steps involving small near-orthogonal perturbations to our targets $y_{t}$, which we then use to update our local estimate for $D$.

We can extend our approach from $\operatorname{\textup{{NestedOCO}}}$ to accommodate bandit feedback for convex losses by replacing $\operatorname{\textup{{FTRL}}}$ with the $\operatorname{\textup{{FKM}}}$ algorithm (Flaxman et al., 2004) and appropriately recalibrating parameters. $\operatorname{\textup{{FKM}}}$ obtains ${O}(T^{3/4})$ regret, which is the best currently-known bound for bandit convex optimization without additional assumptions (e.g. strong convexity), and we obtain an analogous bound here for nested optimization. We note that this extension to bandit feedback can again be applied for any algorithm with a small per-round step-size bound, though this property does not hold for algorithms which sample from larger sets to reduce variance of gradient estimators (e.g. those from Abernethy et al. (2008); Hazan and Levy (2014)).

We present the $\operatorname{\textup{{NestedBCO}}}$ algorithm and prove Theorem 5 in Appendix G.

Performative Prediction was introduced by Perdomo et al. (2020) to capture settings in which the data distribution may shift as a function of the classifier itself. We consider the online formulation of Performative Prediction introduced in Kumar et al. (2022) as an instance of online convex optimization with unbounded memory, which we extend to accommodate a stateful variant of the problem (as in Brown et al. (2022)) in which the update to the distribution is a function of both the classifier and the current distribution itself. Let $\operatorname{\mathcal{X}}\subseteq\operatorname{\mathbb{R}}^{n}$ denote our space of classifiers, and let $p_{0}$ be the initial distribution over $\operatorname{\mathbb{R}}^{n}$. When a classifier $x_{t}$ is deployed, the distribution is updated to

where $\operatorname{\mathcal{D}}(x_{t},y)=A(x_{t},y_{t-1})+\xi$, for a random variable $\xi\in\operatorname{\mathbb{R}}^{n}$ with mean $\mu$ and covariance $\Sigma$, and with $y_{t}=A(x_{t},y_{t-1})$, where $A$ satisfies $\rho$-local controllability for some $\rho>0$ and appropriate smoothness notions. We also assume there is some linear $s:\operatorname{\mathcal{X}}\rightarrow\operatorname{\mathcal{Y}}$ such that $A(x,y)=s(x)$ if $y=s(x)$. We then receive loss $\tilde{f}_{t}(x_{t},p_{t})=\operatorname*{\mathbb{E}}_{z\sim p_{t}}[f_{t}(x_{t},z)]$, where each $f_{t}$ is convex and Lipschitz.

This generalizes the model of Kumar et al. (2022), in which $A(x,y)=A\in\operatorname{\mathbb{R}}^{n\times n}$ is taken to be a fixed matrix; there, $\rho$-local controllability is satisfied for some $\rho>0$ provided that $A$ is nonsingular. Their aim is to compete with the best fixed classifier by running regret minimization over $\operatorname{\mathcal{X}}$. Here we run $\operatorname{\textup{{NestedOCO}}}$ over $\operatorname{\mathcal{Y}}$, taken over the range of $s$, which allows us to compete against the best fixed classifier as well by the properties of $s$; while the classifiers $x_{t}$ we play will generally not result in stabilizing points of $A$, their excess loss compared to each $s^{-1}(y_{t})$ is bounded.

Online interactions with economic agents of various types are ubiquitous, and the resulting control problems tend to be manifestly nonlinear; here we treat two diverse examples from this space. The Adaptive Recommendations problem, as introduced by Agarwal and Brown (2022), is about providing menu recommendations repeatedly to an agent, whose choice distribution is a function of their past selections, while the controller’s reward in each round depends on adversarial losses over the choice. In each round $t\in[T]$, we show the agent a (possibly randomized) menu $K_{t}$ containing $k$ (out of $n$) items, and the agent’s instantaneous choice distribution conditioned on seeing $K_{t}$ is

where each $s_{i}:\Delta(n)\rightarrow[\lambda,1]$ is the agent’s preference scoring function for item $i$, for some $\lambda>0$, taking as input the agent’s memory vector $v\in\Delta(n)$. The memory vector updates each round as

where $\theta_{t}\in[\theta,1]$ for $\theta>0$ is a possibly time-dependent update speed, and we receive loss $f_{t}(p_{t})$, where each $f_{t}$ is convex and $L$-Lipschitz. Note that the set of feasible choice distributions when considering all menu distributions $x_{t}\in\Delta({n\choose k})$ depends on the memory vector $v_{t}$. The regret benchmark considered by Agarwal and Brown (2022) is the intersection of all such sets, denoted the “everywhere instantaneously-realizable distribution” set $\operatorname{\textup{{EIRD}}}=\cap_{v\in\Delta}\operatorname{\textup{{IRD}}}(v)$, where $\operatorname{\textup{{IRD}}}(v)$ is the “instantaneously realizable distribution” set for $v$, given as the convex hull of the choice distributions $p(K_{t})$ resulting from each menu $K_{t}\in[{n\choose k}]$ when $v$ is the memory vector. It is shown that the set is non-empty when $\lambda$ is not too small, and algorithms which minimize regret with respect to any distribution in $\operatorname{\textup{{EIRD}}}$ are given in Agarwal and Brown (2022) and Agarwal and Brown (2023) under varying assumptions regarding the scoring functions and update speed.