Achieving Tractable Minimax Optimal Regret in Average Reward MDPs

In this paper, we answer the above question affirmatively, by proposing a polynomial time algorithm with regret guarantees $\widetilde{\operatorname*{{\rm O}}}\left(\!\!\sqrt{{\mathrm{sp}\left(h^{*}\right)}SAT}\right)$ for average-reward MDPs. Our method can further incorporate almost arbitrary prior bias information $\mathcal{H}_{*}\subseteq\mathbf{R}^{\mathcal{S}}$ to improve its regret.

The geometry of the prior bias region $\mathcal{H}_{*}$ is discussed later (see 4). It can be taken trivial with $\mathcal{H}_{*}=\mathbf{R}^{\mathcal{S}}$ to obtain a completely prior-less algorithm. To the best of our knowledge, this is the first tractable algorithm with minimax optimal regret bounds (up to logarithmic factors). The algorithm does not necessitate any prior knowledge of ${\mathrm{sp}\left(h^{*}\right)}$, thus circumventing the potentially high cost associated with learning ${\mathrm{sp}\left(h^{*}\right)}$. On the technical side, a key novelty of our method is the subroutine named PMEVI (see Algorithm 2) that improves and can replace EVI Auer et al. (2009) in any algorithm that relies on it Auer et al. (2009); Fruit et al. (2018); Filippi et al. (2010); Fruit et al. (2020); Bourel et al. (2020) to boost its performance and achieve minimax optimal regret.

Here is a short overview of the learning theory of average reward MDPs. For communicating MDPs, the notable work of Auer et al. (2009) proposes the famous UCRL2 algorithm, a mature version of their prior UCRL Auer and Ortner (2006), achieving a regret bound of $\widetilde{\operatorname*{{\rm O}}}(DS\!\!\sqrt{AT})$. This paper pioneered the use optimistic methods to learn MDPs efficiently. A line of papers Filippi et al. (2010); Fruit et al. (2020); Bourel et al. (2020) developed this direction by tightening the confidence region that UCRL2 rely on, and sharpened its analysis through the use of local properties of MDPs, such as local diameters and local bias variances, but none of these works went beyond regret guarantees of order $S\!\!\sqrt{DAT}$ and suffer from an extra $\!\!\sqrt{S}$. A parallel direction was initiated by Bartlett and Tewari (2009), that design REGAL to attain ${\mathrm{sp}\left(h^{*}\right)}$-dependent regret bounds (instead of $D$) while extending the regret bounds to weakly-communicating MDPs. The computational intractability of REGAL is addressed by Fruit et al. (2018) with SCAL, while Zhang and Ji (2019) further enhance the regret analysis by evaluating the bias differences with EBF, eventually reaching optimal minimax regret but loosing tractability.

Another successful design approach is Bayesian-flavored sampling, derived from Thompson Sampling Thompson (1933), that usually replaces optimism. The regret guarantees of these algorithms usually stick to the Bayesian setting however Ouyang et al. (2017); Theocharous et al. (2017), although Agrawal and Jia (2023) also enjoys $\widetilde{\operatorname*{{\rm O}}}(S\!\!\sqrt{DAT})$ high probability regret by coupling posterior sampling and optimism. Another line of research focuses on the study of ergodic MDPs, where all policies mix uniformly according to a mixing time. To name a few, the model-free algorithm Politex Abbasi-Yadkori et al. (2019) attains a regret of $\widetilde{\operatorname*{{\rm O}}}((t_{\mathrm{mix}})^{3}t_{\mathrm{hit}}\!\!\sqrt{SA}T^{\frac{3}{4}})$. By leveraging an optimistic mirror descent algorithm, Wei et al. (2020) achieve an enhanced regret of $\widetilde{\operatorname*{{\rm O}}}(\!\!\sqrt{(t_{\mathrm{mix}})^{2}t_{\mathrm{hit}}AT})$ .

We refer the readers to Table 1 for a (non-exhaustive) list of existing algorithms.

An element $M\in\mathcal{M}$ is a tuple $(\mathcal{S},\mathcal{A},p,r)$ where $p$ is the transition kernel and $r$ the reward function. The random state-action pair played by the agent at time $t$ is denoted $X_{t}\equiv(S_{t},A_{t})$, and the achieved reward is $R_{t}$. A policy is a deterministic rule $\pi:\mathcal{S}\to\mathcal{A}$ and we write $\Pi$ the space of policies. Coupled with a $M\in\mathcal{M}$, a policy properly defines the distribution of $(X_{t})$ whose associated probability probability and expectation operators are denoted $\mathbf{P}^{\pi}_{s},\mathbf{E}^{\pi}_{s}$, where $s\in\mathcal{S}$ is the initial state. Under $M$, a fixed policy has a reward function $r^{\pi}(s):=r(s,\pi(s))$, a transition matrix $P^{\pi}$, a gain $g^{\pi}(s):=\lim\frac{1}{T}\mathbf{E}^{\pi}_{s}[R_{0}+\ldots+R_{T-1}]$ and a bias $h^{\pi}:=\lim\sum_{t=0}^{T-1}(R_{t}-g(S_{t}))$, that all together satisfy the Poisson equation $h^{\pi}+g^{\pi}=r^{\pi}+P^{\pi}h^{\pi}$, see Puterman (1994). The Bellman operator of the MDP is:

$M$ is weakly-communicating Puterman (1994); Bartlett and Tewari (2009) if the state space can be divided into two sets: (1) the transient set, consisting in states that are transient under all policies; (2) the non-transient set, where every state is reachable starting from any other non-transient. In this case, $L$ has a span-fixpoint $h^{*}$ (see Puterman (1994)), i.e., there exists $h^{*}\in\mathbf{R}^{\mathcal{S}}$ such that $Lh^{*}-h^{*}\in\mathbf{R}e$ where $e$ is the vector full of ones. We write $h^{*}\in\operatorname{{Fix}}(L)$. Then $g^{*}:=Lh^{*}-h^{*}$ is the optimal gain function and every policy $\pi$ satisfies $r^{\pi}+P^{\pi}h^{*}\leq g^{*}+h^{*}$. We accordingly define the Bellman gaps:

Another important concept is the diameter, that describes the maximal distance from one state to another state. It is given by $D:=\sup_{s\neq s^{\prime}}\inf_{\pi}\mathbf{E}_{s}^{\pi}[\inf\left\{t\geq 1:S_{t}=s^{\prime}\right\}].$ An MDP is said communicating if its diameter $D$ is finite.

The learner is only aware that $M\in\mathcal{M}$ but doesn’t have a clue about what $M$ further looks like. From the past observations and the current state $S_{t}$, the agent picks an available action $\mathcal{A}(S_{t})$, receives a reward $R_{t}$ and observe the new state $S_{t+1}$. The regret of the agent is:

Its expected value satisfies $\mathbf{E}[\operatorname{{Reg}}(T)]=\mathbf{E}[\sum_{t=0}^{T-1}\Delta^{*}(X_{t})]+\mathbf{E}[h^{*}(S_{0})-h^{*}(S_{T})]$ and the quantity $\sum_{t=0}^{T-1}\Delta^{*}(X_{t})$ will be referred to as the pseudo-regret. This paper focuses on minimax regret guarantees. Specifically, for $c\geq 1$, denote $\mathcal{M}_{c}:=\left\{M\in\mathcal{M}:\exists h^{*}\in\operatorname{{Fix}}(L(M)),{\mathrm{sp}\left(h^{*}\right)}\leq c\right\}$ the set of weakly-communicating MDPs that admit a bias function with span at most $c$, where the span of a vector $u\in\mathbb{R}^{\mathcal{S}}$ is ${\mathrm{sp}\left(u\right)}:=\max(u)-\min(u)$. Following Auer et al. (2009), every algorithm $\mathbf{A}$, for all $c>0$, we have

The goal of this work is to reach this lower bound with a tractable algorithm.

To solve (5) efficiently, the celebrated Auer et al. (2009) introduced the extended value iteration algorithm (EVI). Assume that $\mathcal{M}_{t}$ is a $(s,a)$-rectangular confidence region, meaning that $\mathcal{M}_{t}\equiv\prod_{s,a}(\mathcal{R}_{t}(s,a)\times\mathcal{P}_{t}(s,a))$ where $\mathcal{R}_{t}(s,a)$ and $\mathcal{P}_{t}(s,a)$ are respectively the confidence region for $r(s,a)$ and $p(s,a)$ after $t$ learning steps. EVI is the algorithm computing the sequence defined by:

until ${\mathrm{sp}\left(v_{i+1}-v_{i}\right)}<\epsilon$ where $\epsilon>0$ is the numerical precision. When the process stops, it is known that any policy $\pi$ such that $\pi(s)$ achieves $\mathcal{L}_{t}v_{i}$ in (6) satisfies $g^{\pi}(\mathcal{M}_{t})\geq g^{*}(\mathcal{M})-\epsilon$, hence is nearly optimistically optimal. This process gets its name from the observation that $\mathcal{L}_{t}$ is the Bellman operator of $\mathcal{M}_{t}$ seen as a MDP, hence EVI is just the Value Iteration algorithm Puterman (1994) ran in $\mathcal{M}_{t}$. A choice of action from $s\in\mathcal{S}$ in $\mathcal{M}_{t}$ consists in (1) a choice of action $a\in\mathcal{A}(s)$, (2) a choice of reward $\tilde{r}(s,a)\in\mathcal{R}_{t}(s,a)$ and (3) a choice of transition $\tilde{p}(s,a)\in\mathcal{P}_{t}(s,a)$; It is an extended version of $\mathcal{A}(s)$.

Obviously, the regret of an OFU-algorithm is directly related to the quality of the confidence region $\mathcal{M}_{t}$. That is why most previous works tried to approach the regret lower bound $\!\!\sqrt{DSAT}$ of Auer et al. (2009) by refining $\mathcal{M}_{t}$. The older works of Auer et al. (2009); Bartlett and Tewari (2009); Filippi et al. (2010) have been improved with a variance aware analysis Talebi and Maillard (2018); Fruit et al. (2018, 2020); Bourel et al. (2020) that essentially make use of tightened kernel confidence regions $\mathcal{P}_{t}$. While all these algorithms successively reduce the gap between the regret upper and lower bounds, they fail to achieve optimal regret $\!\!\sqrt{DSAT}$. Meanwhile, the EVI algorithm of Zhang and Ji (2019) achieves the lower bound but (1) the algorithm is intractable because it relies on an oracle to retrieve optimistically optimal policies and (2) needs prior information on the bias function. Nonetheless, the method of Zhang and Ji (2019) strongly suggests that inferring bias information from the available data is key to achieve minimax optimal regret.

Rather surprisingly and in opposition to this previous line of work, our work suggests that the choice of the confidence region $\mathcal{M}_{t}$ has little importance. Instead, our algorithm takes an arbitrary (well-behaved) confidence region in, infer bias information similarly to EBF Zhang and Ji (2019) and makes use of it to heavily refine the extended Bellman operator (6) associated to the input confidence region. Our algorithm can further take arbitrary prior information (possibly none) on the bias vector to tighten its bias confidence region. The pseudo-code given in Algorithm 1 is the high level structure our algorithm PMEVI-DT. In Section 3.1, we explain how (6) is refined using bias information and in Section 3.2, we explain how bias information is obtained.