Catastrophic-risk-aware reinforcement learning with extreme-value-theory-based policy gradients

Abstract

This paper tackles the problem of mitigating catastrophic risk (which is risk with very low frequency but very high severity) in the context of a sequential decision making process. This problem is particularly challenging due to the scarcity of observations in the far tail of the distribution of cumulative costs (negative rewards). A policy gradient algorithm is developed, that we call POTPG. It is based on approximations of the tail risk derived from extreme value theory. Numerical experiments highlight the out-performance of our method over common benchmarks, relying on the empirical distribution. An application to financial risk management, more precisely to the dynamic hedging of a financial option, is presented.

Keywords: Risk-aware reinforcement learning, catastrophic risk, extreme value theory, peaks-over-threshold (POT), hedging.

1   Introduction

Reinforcement learning (RL) consists in a set of methods allowing to optimize sequential decision processes through interactions with an environment. In traditional RL (see Sutton and Barto,, 2018), the primary objective is to maximize expected rewards. However, a subset of RL techniques, referred to as risk-aware reinforcement learning, aim to take risk also into account (i.e. departure from the expected case), see for example Wu and Lin, (1999), Borkar, (2001), Tamar et al., (2012), La and Ghavamzadeh, (2013), Chow et al., (2018), Greenberg et al., (2022) and Vijayan and Prashanth, (2023). Integrating risk mitigation within the RL framework is of paramount importance in several areas, as policies producing high expected rewards together with a high risk might be unacceptable in certain circumstances. Financial risk management is an example key area where risk-aware RL methods are developed, see for instance Buehler et al., (2019), Carbonneau and Godin, (2021), Cao et al., (2023) and Wu and Jaimungal, (2023) for a few examples.

The present work is concerned with problems involving the minimization of catastrophic risk in a sequential decision process, which represents outcomes that are very rare but of extreme magnitude. Since such extreme events can cause very undesirable outcomes depending on the area of application, such as financial ruin, health-impeding consequences or accidents, mitigating their impact is very important. Another example of application is the measurement of capital requirements in finance or insurance, which are based on the average of outcomes in the very worst-case scenarios; minimizing capital requirements for a financial institution is a key determinant of its probability, as capital is costly to hold.

Here, extreme risk is quantified through risk measures, reflecting the far tail of the distribution of total costs incurred by the agent. In particular, we consider the special case of the conditional Value-at-Risk, CVaR_α, which represents the average outcome among the worst possible set of scenarios with probability $1-\alpha$. The main motivation of the work is that CVaR with a very high confidence level $\alpha$ is very poorly approximated with the empirical distribution, due to the scarcity of observations in the far tail. Such paucity can be caused either by the lack of extreme observations, or the inability to generate a sufficient number of scenarios that include enough extreme data points in a reasonable time frame. In most acute cases, extreme outcomes might even be outside the data range, by not having materialized yet. The scarcity of tail observations can be exacerbated if the cost outcomes from the problem have fat-tailed distributions.

Our main contribution is to develop a policy gradient method for risk-aware RL problems that is tailor-made for cases where catastrophic-level risk must be minimized. We refer to our algorithm as POTPG, as it relies on the peaks-over-threshold (POT) approach of extreme value theory (EVT) that allows extrapolating the far tail behavior of a distribution through asymptotic approximations leveraging the distribution from large (but not extreme) outcomes. To the best of our knowledge, we are the first to incorporate EVT results within reinforcement learning algorithms to tackle general sequential decision problems; our work can be seen as an extension of Troop et al., (2022) that explores catastrophic risk minimization within the multi-armed bandits framework, but did not tackle the more general Markov decision problem setting.

The paper is divided as follows. Section 2 describes the risk-aware sequential decision making problem considered here, and provides a conventional policy gradient algorithm to tackle the problem. Section 3 proposes our POT policy gradient (POTPG) approach, a modified policy gradient algorithm based on extreme value theory estimates of the tail risk of a distribution. This algorithm is tailor-made to tackle catastrophic-level risk minimization. Section 4 benchmarks the performance of POTPG against the conventional approach in a controlled environment, whereas Section 5 assesses its performance in a financial risk management application, namely option hedging optimization. The paper concludes with some final remarks in Section 6. The Python code to replicate the various numerical experiments of this paper is available at https://github.com/parisadavar/EVT-policy-gradient-RL.

2   A risk-aware reinforcement learning problem and policy gradients

We herein consider the framework of Markov decision processes to represent sequential decision problems. Such problems are represented by a set of time steps $\mathcal{T}=\{0,\ldots,T\}$, a state space $\mathcal{S}$, an action space $\mathcal{A}$, a cost space $\mathcal{C}$ and a sequence of transition probabilities characterizing the joint distribution of the next-step reward and state, given the current state and action, namely $\mathbb{P}[S_{t+1}\leq s^{\prime},C_{t+1}\leq c|A_{t}=a,S_{t}=s]$ for $s,s^{\prime}\in\mathcal{S}$, $a\in\mathcal{A}$, $c\in\mathcal{C}$ and $t\in\mathcal{T}\backslash\{T\}$.

Without loss of generality, deterministic policies $\pi:\mathcal{S}\rightarrow\mathcal{A}$ are considered in this work. Such framework gives rises to random state-action-cost sequences of the form $S_{0},A_{0},C_{1}$, $S_{1},A_{1},C_{2},\ldots$, $S_{T-1},A_{T-1},C_{T}$, where at any time point $t$ the agent takes action $A_{t}=\pi(S_{t})$ when encountering state $S_{t}$, at a cost of $C_{t}$, then the next-stage state $S_{t+1}$ is drawn randomly from the probability measure $\mathbb{P}$.

2.1   A risk-aware reinforcement learning problem

The risk-aware reinforcement learning problem considered here is to find the optimal policy minimizing the risk associated with the cumulative discounted cost: denoting costs as $C^{(\pi)}_{t}$ to highlight their dependence on policy $\pi$, the problem considered can be written as

$$\underset{\pi}{\min}\,\rho\left(\sum^{T}_{t=1}\gamma^{t}C^{(\pi)}_{t}\right)$$ (2.1)

for some discount factor $\gamma\in(0,1]$ and a risk measure $\rho$ map** random variables into perceived risk. In the classic non-risk-aware case (see for instance Sutton and Barto,, 2018), the risk measure is the expectation operator, namely $\rho=\mathbb{E}$. However, more general risk measures can be used to depict preferences of risk-aware agents. Since this work is concerned with catastrophic risk mitigation, we consider the specific case of the CVaR risk measure (Rockafellar and Uryasev,, 2002) depicting tail risk and defined as

$$\text{CVaR}_{\alpha}(X)=\frac{1}{1-\alpha}\int_{\alpha}^{1}q_{\gamma}(X)d\gamma,\quad\text{with }\,q_{\alpha}(X)=\inf\{x\in\mathbb{R}|F(x)\geq\alpha\},$$

where $F$ is the cumulative distribution function (CDF) of $X$ and $q_{\alpha}(X)$ is its quantile at level $\alpha$. In what follows we write $q_{\alpha}=q_{\alpha}(X)$. When $X$ is an absolutely continuous random variable, as in this work, then CVaR has the alternative representation $\text{CVaR}_{\alpha}(X)=\mathbb{E}[X|X\geq q_{\alpha}]$, which can be interpreted as the average outcome among the set of the $100(1-\alpha)\%$ worst-case scenarios. This work considers catastrophic risk minimization, and as such, we consider very high levels for $\alpha$, i.e. $\alpha$ very close to one.

2.2   A policy gradient solution approach

A natural approach to solve (2.1) is policy gradient methods. Policies are first restricted to a set of parametric policies $\pi_{\theta}$ with parameter vector $\theta$. In that case, Problem (2.1) reduces to solving

$$\theta^{*}=\underset{\theta}{\arg\max}\ J(\theta)\quad\text{with }\,\,J(\theta)=\text{CVaR}_{\alpha}\left(\sum^{T}_{t=1}\gamma^{t}C^{(\pi_{\theta})}_{t}\right).$$ (2.2)

A common solution approach to the above problem is to use batch stochastic gradient descent, which leads to a sequence of parameter vectors $\{\theta^{(j)}\}_{j\geq 1}$ obtained through

$$\theta^{(j+1)}=\theta^{(j)}+\eta_{j}\widehat{\nabla J}(\theta^{(j)}),$$

with $\{\eta_{n}\}$ representing the learning schedule and $\widehat{\nabla J}(\theta^{(j)})$ representing a suitable (stochastic) approximation of the gradient. Here we use the celebrated ADAM algorithm of Kingma and Ba, (2014), with a step size parameter of $0.01$ to determine learning rate sequences $\{\eta^{(j)}\}_{j\geq 1}$.

To approximate the gradient, a (forward) finite difference approach is used here: for some small $\epsilon>0$,

$$\widehat{\frac{\partial J}{\partial\theta_{i}}}(\theta^{(j)}) \approx \frac{\widehat{J}(\theta^{(j)}+\epsilon\mathds{1}_{i})-\widehat{J}(\theta^{(j)})}{\epsilon},$$ (2.3)

$$\widehat{\nabla J}(\theta^{(j)}) = \left[\widehat{\frac{\partial J}{\partial\theta_{1}}}(\theta^{(j)})\ldots\widehat{\frac{\partial J}{\partial\theta_{p}}}(\theta^{(j)})\right]^{\top},$$ (2.4)

with $p$ being the dimension of the parameter vector and $\mathds{1}_{i}$ being the dummy vector containing zeroes, except for its $i^{th}$ element that is equal to one. The objective function $\widehat{J}(\theta)$ is approximated by sampling $n$ independent copies $X_{1},\ldots,X_{n}$ of the cumulative costs $X=\sum^{T}_{t=1}\gamma^{t}C^{(\pi_{\theta})}_{t}$ obtained from a Monte-Carlo simulation, if a simulator of the environment is available, or alternatively through the application of the policy $\pi_{\theta}$ to real data, either in an online or offline fashion.

The most natural approach to obtain the estimate of the objective function consists in assuming that the empirical distribution of cumulative costs obtained through the mini-batch is close to the true distribution. Such method is referred to as the sample averaging method and relies on

$$\widehat{J}=\widehat{CVaR}_{\alpha}(X)=\frac{\sum_{i=1}^{n}X_{i}\mathds{1}_{\{X_{i}\geq\hat{q}_{\alpha}\}}}{\sum_{j=1}^{n}\mathds{1}_{\{X_{j}\geq\hat{q}_{\alpha}\}}},$$ (2.5)

where, denoting by $X_{(1)},\ldots,X_{(n)}$ the order statistics of the sample (i.e. the sample sorted in increasing order), $\hat{q}_{\alpha}$ is the empirical quantile given by:

$$\hat{q}_{\alpha}=\widehat{VaR}_{\alpha}(X)=\inf\{x\in\mathbb{R}|\hat{F}(x)\geq\alpha\}=X_{(\lceil{\alpha n}\rceil)},$$ (2.6)

with $\widehat{F}$ being the empirical CDF of $X$ given by $\widehat{F}(x)=\frac{1}{n}\sum_{s=1}^{n}\mathds{1}_{\{X_{s}\leq x\}}$.

Unfortunately, when $\alpha$ is high and very close to one, the scarcity of observations can make the sample averaging approach very unstable in estimating the objective function, a problem which is exacerbated if the distribution of $X$ is heavy-tailed. This justifies the development of the EVT-based estimator described in the next section.

3   Integrating extreme value theory estimates into policy gradients

This section first discusses the construction of CVaR estimates based on the peaks–over–threshold (POT) approach rooted in extreme value theory (EVT). The POT approach is discussed more in-depth in Coles et al., (2001) or McNeil et al., (2015). The procedure integrating such estimates into policy gradient approaches is subsequently detailed.

3.1   Estimation of CVaR with the peaks-over-threshold approach

A wide set of distributions satisfy the following condition.

Definition 3.1.

A CDF $F$ is said to be in the maximum domain of attraction of the generalized extreme value distribution (GEVD) $H_{\xi}$ with parameter $\xi$, denoted $F\in MDA(H_{\xi})$, if there exist a sequence a positive numbers $\{a_{n}\}_{n\in\mathbb{N}}$ and a sequence of real numbers $\{b_{n}\}_{n\in\mathbb{N}}$, such that

$$\lim_{n\to\infty}F^{n}(a_{n}x+b_{n})=H_{\xi}(x).$$ (3.1)

Note that $F^{n}$ is the CDF of the maximum of $n$ i.i.d. copies of a random variable with CDF $F$.

The $F\in MDA(H_{\xi})$ property characterizes the asymptotic behavior of distribution $F$. Indeed, define $F_{u}$, the distribution of excesses above threshold $u$, as

$$F_{u}(y)=P(X-u\leq y|X>u)=P(X\leq y+u|X>u)=\frac{F(y+u)-F(u)}{1-F(u)}.$$ (3.2)

Define also the generalized Pareto distribution (GPD) as follows.

Definition 3.2.

The GPD with scale parameter $\sigma$ and shape parameter $\xi$ has a CDF

$$G_{\xi,\sigma}(x)=\begin{cases}1-(1+\frac{\xi x}{\sigma})^{\frac{-1}{\xi}},&\mbox{if }\xi\neq 0,\\ 1-e^{\frac{-x}{\sigma}},&\mbox{if }\xi=0,\end{cases}$$ (3.3)

where the support is $x\geq 0$, for $\xi\geq 0$, and $0\leq x\leq-\frac{\sigma}{\xi}$, for $\xi\leq 0$, and a probability density function (PDF)

$$g_{\xi,\sigma}(x)=\begin{cases}\frac{1}{\sigma}(1+\frac{\xi x}{\sigma})^{\frac{-1}{\xi}-1},&\mbox{if }\xi\neq 0,\\ \frac{1}{\sigma}e^{\frac{-x}{\sigma}},&\mbox{if }\xi=0.\end{cases}$$ (3.4)

Then the following result from Balkema and De Haan, (1974) or Pickands III, (1975) states that when $F\in MDA(H_{\xi})$, the excess distribution $F_{u}$ is well-approximated asymptotically by a GPD distribution when $u$ is near to the essential supremum of distribution $F$.

Theorem 3.1 (Pickands–Balkema–de Haan).

If $F\in MDA(H_{\xi})$, there exists a positive measurable function $\sigma(u)$ such that

$$\lim_{u\to y_{0}}\sup_{y_{0}\in[0,y_{0}-u]}|F_{u}(y)-G_{\xi,\sigma(u)}(y)|,$$ (3.5)

where $y_{0}=sup\{y\in\mathbb{R};F(y)<1\}\leq\infty$ and $G_{\xi,\sigma(u)}(y)$.

As described in Section 7.2 of McNeil et al., (2015), such a result allows defining the following approximation for the CVaR of the variable $X$ with CDF $F$, which is based on the assumption that $F_{u}(y)\approx G_{\xi,\sigma(u)}(y)$ for $y>u$, i.e. if $u$ is large enough.

Corollary 3.1.

Assume that $F\in MDA(H_{\xi})$ for some $\xi\in[0,1)$, and that $q_{\alpha}>u$. Let $\sigma=\sigma(u)$ satisfy conditions of Theorem 3.1. Then for $s_{u,\alpha}=\frac{1-F(u)}{1-\alpha}$,

$${CVaR}_{\alpha}(X)\approx c_{u,\alpha}=\begin{cases}u+\frac{\sigma}{1-\xi}(1+\frac{s^{\xi}_{u,\alpha}-1}{\xi}),&\mbox{if }\xi\neq 0,\\ u+\sigma(\log s_{u,\alpha}+1),&\mbox{if }\xi=0.\end{cases}$$ (3.6)

This points toward the following procedure, called the peaks-over-threshold approach to estimate ${CVaR}_{\alpha}(X)$ based on a sample of i.i.d. copies $X_{1},\ldots,X_{n}$ of $X$:

1. Select a proper threshold $u$.

2. Calculate sample values of excesses over threshold $u$, denoted $Y_{1},\ldots,Y_{k}$ and defined as $Y_{i}=X_{(n+1-i)}-u$, where $k$ is the number of sample observations $X_{i}$ above $u$.

3. Fit a GPD distribution to the sample $Y_{1},\ldots,Y_{k}$ to get estimates $(\hat{\xi},\hat{\sigma})$.

4. Replace $(\xi,\sigma)$ and $F(u)$ with respective estimates $(\hat{\xi},\hat{\sigma})$ and $\hat{F}(u)$ into (3.6) to get an approximation for ${CVaR}_{\alpha}(X)$.

Since for any fixed $u$, excesses $Y_{1},\ldots,Y_{k}$ are independent, Step 3 can be performed through maximum likelihood by solving numerically

$$(\hat{\xi},\hat{\sigma})=\underset{\xi,\sigma}{\arg\max}\sum_{i=1}^{k}\ln g_{\xi,\sigma}(Y_{i}).$$ (3.7)

Alternatively, a method-of-moments (MOM) estimator matching the first two moments of the GPD distribution with those of the empirical distribution of excesses would lead to

$$\hat{\xi}=\frac{S^{2}-\bar{Y}^{2}}{2S^{2}},\quad\hat{\sigma}=\bar{Y}\left(\frac{S^{2}+\bar{Y}^{2}}{2S^{2}}\right),$$ (3.8)

with $\bar{Y}=\frac{1}{k}\sum^{k}_{j=1}Y_{j}$ and $S^{2}=\frac{1}{k}\sum^{k}_{j=1}(Y_{j}-\bar{Y})^{2}$.

The task in Step $1$, namely the selection of a suitable choice of threshold $u$ is challenging, as it entails seeking a proper bias-variance trade-off. Indeed, if $u$ is too low, the distribution tail behavior might not be well-approximated by its asymptotic GPD distribution, leading to high bias. Conversely, choosing a $u$ that is too large will imply a low number of excesses, which will lead to high variance for the GDP parameter estimators. A common approach in the literature is to manually select $u$ through visual inspection of the so-called Hill plot (see McNeil et al.,, 2015). However, such a method is not appropriate in our setup since the choice of threshold $u$ needs to be repeated a very large number of times through the learning phase. As such, we rely on the Bader et al., (2018) algorithm that performs automated selection of the threshold based on a sequence of Anderson-Darling goodness-of-fit tests. Such a procedure tests for a set of candidate values $u^{(1)}<\ldots<u^{(\ell)}$, and the smallest among these is selected as the threshold, which leads to a proper fit of the GPD to excesses over $u$. The implementation from Troop et al., (2021) of the procedure is considered here and is detailed in Appendix Appendix A     The automated threshold selection procedure. This modifications allows stabilizing estimates, for instance by not allowing estimated values of $\xi$ too close to one (to avoid the CVaR estimate exploding) and by using the sample averaging estimator as fallback, when none of the thresholds lead to a satisfactory fit of the GPD.

3.2   Our proposed EVT policy gradient algorithm

The POT-based CVaR estimation method from Section 3.1 is now integrated into the policy gradient estimation formula in (2.3) to obtain a complete policy gradient learning procedure for the policy parameters $\theta$. This procedure, which we call the POTPG algorithm (standing for for peaks-over-threshold policy gradient), is summarized in the Algorithm 1 box below.

Algorithm 1 POTPG

$\epsilon$ (finite difference step), n (number of episodes), M (number of iterations)

Initialize $\theta_{0}$ through random sampling

for $j=0,\ldots,M-1$  do

     Sample $n$ episodes of the MDP with policy $\pi_{\theta_{j}}$, and denote by $X_{i}=\sum^{T}_{t=1}\gamma^{t}C^{(\pi_{\theta^{(j)}})}_{t}$ the total discounted costs for the $i^{th}$ episode,

     Based on sample $X_{1},\ldots,X_{n}$, obtain the estimates $\hat{\xi}$, $\hat{\sigma}$ and $\hat{F}(u)$ where the automated threshold selection method of Appendix Appendix A     The automated threshold selection procedure is applied to determine $u$,

     Obtain the EVT-based estimate of $\widehat{J}(\theta^{(j)})$ through (3.6),

     for $i=1,\ldots,p$  do

         Sample $n$ episodes of the MDP with policy $\pi_{\theta^{(j)}+\epsilon\mathds{1}_{i}}$, and denote by $X_{i}=\sum^{T}_{t=1}\gamma^{t}C^{(\pi_{\theta^{(j)}+\epsilon\mathds{1}_{i}})}_{t}$ the total discounted costs for the $i^{th}$ episode,

         Based on sample $X_{1},\ldots,X_{n}$ and threshold $u$, obtain the estimates $\hat{\xi}$, $\hat{\sigma}$ and $\hat{F}(u)$.

         Obtain the EVT-based estimate of $\widehat{J}(\theta^{(j)}+\epsilon\mathds{1}_{i})$ through (3.6),      

     Estimate the gradient $\widehat{\nabla J}(\theta^{(j)})$ through the finite difference scheme (2.3)-(2.4),

     $\theta^{(j+1)}\leftarrow\theta^{(j)}+\eta_{j}\widehat{\nabla J}(\theta^{(j)})$, with $\eta_{j}$ as determined by the ADAM algorithm. Return $\theta^{(M)}$

If a simulator of the environment is available, it can be desirable, within a given iteration $j$, to use the same random seed to perform the simulation of episodes under policy $\pi_{\theta^{(j)}}$ and these under policies $\pi_{\theta^{(j)}+\epsilon\mathds{1}_{i}}$, $i=1,\ldots,p$. This approach offers the advantage of isolating the impact of the policy alteration (from $\pi_{\theta^{(j)}}$ to $\pi_{\theta^{(j)}+\epsilon\mathds{1}_{i}}$) from the randomness associated with the generation of episodes; the latter can add noise to the gradient estimate. The same seed is used throughout all the experiments presented here to simulate episodes under the original and shocked policies.

Note also that we propose to use the same threshold $u$ to estimate $\widehat{J}(\theta^{(j)})$ and all $\widehat{J}(\theta^{(j)}+\epsilon\mathds{1}_{i})$ in the POTPG algorithm to enhance the stability in the gradient estimation.

4   Simulation experiments in a controlled environment

Several simulation experiments in a controlled environment are first conducted to assess the performance of the POTPG algorithm from Section 3.2 and compare it to the conventional sample averaging (SA) benchmark based on (2.5). A simple simulation setting is considered to establish a proof of concept and highlight the potential usefulness of the POTPG algorithm. In such setting, we consider a single-dimension policy vector $\theta$ (i.e. $p=1$), and we assume the cumulative discounted cost is distributed according to a given family of distributions whose parameters depend on $\theta$.

More precisely, assume that under policy $\pi_{\theta}$, $X\sim$ GPD$(\xi,\varsigma=(\theta-\vartheta)^{2}+b)$ for some $\xi\in(0,1)$, $b>0$ and $\vartheta\in\mathbb{R}$. Fix $\vartheta=0.4$ and $b=2$ and consider values $\xi=0.4$, $0.6$ or $0.8$ in subsequent experiments. Note that if $X\sim GPD(\xi,\varsigma)$, then the conditional exceedance $X-u|X>u\sim GPD(\xi,\varsigma+\xi u)$, meaning that the excess distribution of a GPD random variable is a GPD with the same shape parameter, and a scaling parameter that grows linearly with the threshold $u$. In that case, representing the tail distribution with a GPD is exact and not merely an asymptotic approximation. Such setting is used to test the POTPG algorithm in an ideal case with no misspecification of the tail distribution. The CVaR of the distribution with policy $\pi_{\theta}$ is then

$$\text{CVaR}_{\alpha}(X)=q_{\alpha}+\frac{\varsigma+\xi q_{\alpha}}{1-\xi}=\frac{\varsigma}{1-\xi}\left(1+\frac{\left(1-\alpha\right)^{-\xi}-1}{\xi}\right),$$

since $q_{\alpha}=\frac{\varsigma}{\xi}\left((1-\alpha)^{-\xi}-1\right)$. Therefore,

$$\theta^{*}=\underset{\theta}{\arg\min}\,\text{CVaR}_{\alpha}(X)=\vartheta,\quad\underset{\theta}{\min}\,\text{CVaR}_{\alpha}(X)=\frac{b}{1-\xi}\left(1+\frac{\left(1-\alpha\right)^{-\xi}-1}{\xi}\right),$$

i.e. the optimal policy is to set $\theta=\vartheta$ to minimize the scale parameter of the cumulative discounted costs.

In each simulation run, we consider $M=500$ iterations, in each of which $n=2,\!000$ cumulative discounted cost realizations are generated. A total of $R=50$ runs are performed. The finite difference step size for the gradient computation is $\epsilon=0.01$. We set the initial policy parameter to $\theta^{(0)}=1$. Define $\theta^{(j,r)}$ and $\widehat{J}^{(j,r)}$ respectively as estimates of the policy parameter and the objective function (CVaR) estimate on the $j^{th}$ iteration of run $r$. We report the root-mean-square-error (RMSE) across the various runs for each iteration of policy parameters and the objective function associated as:

$$\text{RMSE}_{\theta}=\sqrt{\frac{1}{R}\sum^{R}_{r=1}\left(\theta^{(j,r)}-\theta^{*}\right)^{2}},\quad\text{RMSE}_{\widehat{J}}=\sqrt{\frac{1}{R}\sum^{R}_{r=1}\left(\widehat{J}^{(j,r)}-J(\theta^{*})\right)^{2}}.$$

The CVaR level $\alpha=0.998$ is chosen to depict catastrophic risk levels.

Figure 1 reports metrics $\text{RMSE}_{\theta}$ and $\text{RMSE}_{\widehat{J}}$ with respect to iteration $j=1,\ldots,M$ for the three different values of tail parameter $\xi$. The POTPG outperforms the SA benchmark in all experiments as the RMSE on the optimal policy parameter decreases faster for the former approach. The extent of out-performance increases when the tail thickness (i.e. parameter $\xi$) increases. This is because sample averaging relies on only four observations, i.e. $n(1-\alpha)=4$, coming from the tail of distribution, which is increasingly unstable as tail thickness increases. The POT approach better alleviates this issue by using many more observations from the body of the distribution to extrapolate tail behavior. Moreover, even when having converged to the optimal policy, both methods (POTPG and SA) exhibit residual estimation error in the objective function (i.e. cumulative discounted costs) CVaR estimate. Though $RMSE_{\hat{J}}$ is generally smaller for the POTPG approach, such methods out-perform SA significantly for the thicker tail case $\xi=0.8$. In conclusion, the thicker the tail of the costs distribution is, the more useful the POTPG approach is.

(a) $\text{RMSE}_{\theta}$, $\xi=0.4$

(b) $\text{RMSE}_{\widehat{J}}$, $\xi=0.4$

(c) $\text{RMSE}_{\theta}$, $\xi=0.6$

(d) $\text{RMSE}_{\widehat{J}}$, $\xi=0.6$

(e) $\text{RMSE}_{\theta}$, $\xi=0.8$

(f) $\text{RMSE}_{\widehat{J}}$, $\xi=0.8$

Figure 1: Training performance for the POTPG algorithm and the sample averaging (SA) benchmark. Left column: RMSE of policy parameter estimate $\text{RMSE}_{\theta}$. Right column: RMSE of the objective function (the CVaR) $\text{RMSE}_{\widehat{J}}$. RMSE metrics are computed over $R=50$ runs.

5   Application to financial hedging

We present the application of the POTPG algorithm to a financial risk management problem, namely the dynamic Delta-Gamma hedging of an option. The problem of finding the optimal proportion of the Gamma to neutralize when options are very expensive is discussed.

5.1   The hedging framework

Time elapsed between consecutive time points are assumed to be weeks (period of length $1/52$ year). The periodic continuously compounded interest rate is $r=0.02/52$. With $S_{0}=1,\!000$, let $S_{t}$ denote the time-$t$ price of a non-dividend-paying stock, whose dynamics is assumed to be a discrete-time version of an exponential normal-inverse Gaussian (NIG) Lévy process: $S_{t}=S_{0}e^{\sum_{m=1}^{t}Z_{m}}$, with $\{Z_{t}\}^{T}_{t=1}$ being, under the physical measure $\mathbb{P}$, i.i.d. random variables with a NIG$(\mathtt{a}^{\mathbb{P}},\beta^{\mathbb{P}},\delta^{\mathbb{P}},\mu^{\mathbb{P}})$ distribution whose PDF is given by

$$\phi^{NIG}(x;\mathtt{a},\beta,\mu,\delta)=\frac{\mathtt{a}\delta e^{\delta\gamma}}{\pi}\frac{K_{1}(\mathtt{a}\sqrt{\delta^{2}+(x-\mu)^{2}}}{\sqrt{\delta^{2}+(x-\mu^{2})}}e^{{\beta(x-\mu)}},\quad x\in\mathbb{R},$$ (5.1)

where $K_{\lambda}(x)$ represents the modified Bessel function of the second kind with index $\lambda$, defined as:

$$K_{\lambda}(x)=\frac{1}{2}\int_{0}^{\infty}u^{\lambda-1}e^{-\frac{1}{2}x(u^{-1}+u)}du,\quad x>0.$$ (5.2)

Such distribution is known to exhibit fat tails and is therefore well-suited to study the extreme risk minimization framework of this study.

Parameters considered are taken from Godin, (2016), namely $\mathtt{a}^{\mathbb{P}}=35.7$, $\beta^{\mathbb{P}}=-10.8$, $\delta^{\mathbb{P}}=2.04\times 10^{-2}$ and $\mu^{\mathbb{P}}=6.7\times 10^{-3}$.

We consider a market with high volatility risk premium where options are costly; as such we assume risk-neutral parameters and identical to the physical ones, except for the delta parameter driving the returns variance, which is inflated by a factor of 4: $\mathtt{a}^{\mathbb{Q}}=\mathtt{a}^{\mathbb{P}}$, $\beta^{\mathbb{Q}}=\beta^{\mathbb{P}}$, $\delta^{\mathbb{Q}}=4\delta^{\mathbb{P}}$ and $\mu^{\mathbb{Q}}=\mu^{\mathbb{P}}$. In such a market, fully neutralizing the gamma of the option being hedged is most likely sub-optimal, due to high option cost, and thus determining the best hedge ratio yielding the optimal cost versus risk reduction tradeoff is a non-trivial endeavor which is the problem considered in this section.

We assume than any European call option on such stock is priced according to the formula provided in Godin et al., (2012) which is based on the mean-correcting martingale measure described in Schoutens, (2003). The time-$t$ price of a European call option with strike $E$ providing the time $t^{\prime}$ payoff $\max(0,S_{t^{\prime}}-E)$ is

$$\Pi(t,\tau,E) =S_{t}\left(1-\Phi^{NIG}\left(\ln\left(\frac{E}{S_{t}}\right);\ \mathtt{a}^{\mathbb{Q}},\ \beta^{\mathbb{Q}}+1,\ \delta^{\mathbb{Q}}\tau,\ [\mu^{\mathbb{Q}}+\zeta^{\mathbb{Q}}]\tau\right)\right)$$

$$\quad-Ee^{-r\tau}\left(1-\Phi^{NIG}\left(\ln\left(\frac{E}{S_{t}}\right);\ \mathtt{a}^{\mathbb{Q}},\ \beta^{\mathbb{Q}},\ \delta^{\mathbb{Q}}\tau,\ [\mu^{\mathbb{Q}}+\zeta^{\mathbb{Q}}]\tau\right)\right),$$ (5.3)

with $\tau=t^{\prime}-t$ weeks, $\zeta^{\mathbb{Q}}=r-\mu^{\mathbb{Q}}+\delta^{\mathbb{Q}}(\sqrt{(\mathtt{a}^{\mathbb{Q}})^{2}-(\beta^{\mathbb{Q}}+1)^{2}}-\sqrt{(\mathtt{a}^{\mathbb{Q}})^{2}-(\beta^{\mathbb{Q}})^{2}})$ and $\Phi^{NIG}$ denoting the CDF of the NIG distribution. It is straightforward to compute the Delta and the Gamma of such options:

$$\Delta(t,\tau,E) = \frac{\partial\Pi(t,\tau,E)}{\partial S_{t}}=1-\Phi^{NIG}\left(\ln\left(\frac{E}{S_{t}}\right);\ \mathtt{a}^{\mathbb{Q}},\ \beta^{\mathbb{Q}}+1,\ \delta^{\mathbb{Q}}\tau,\ [\mu^{\mathbb{Q}}+\zeta^{\mathbb{Q}}]\tau\right).$$

$$\Gamma(t,\tau,E) = \frac{\partial^{2}\Pi(t,\tau,E)}{\partial(S_{t})^{2}}=\frac{1}{S_{t}}\phi^{NIG}\left(\ln\left(\frac{E}{S_{t}}\right);\ \mathtt{a}^{\mathbb{Q}},\ \beta^{\mathbb{Q}}+1,\ \delta^{\mathbb{Q}}\tau,\ [\mu^{\mathbb{Q}}+\zeta^{\mathbb{Q}}]\tau\right).$$

We consider a financial institution (the hedging agent) which holds a short position in a call option with a strike price $E=S_{0}$ and maturity $T=0.5\times 52=26$ weeks. Such option is referred to as the target option. To mitigate the risk associated with the uncertainty related to its payoff, a self-financing hedging portfolio is used. At any time point, the portfolio is invested in three hedging assets, namely a risk-free account, the stock and an option on the stock. The time-$t$ value of the hedging portfolio is denoted $V^{\theta}_{t}$ (the superscript $\theta$ refers to its dependence on the policy) and evolves according to

$$V^{\theta}_{t+1}=\underbrace{(V^{\theta}_{t}-\psi^{(S)}_{t}S_{t}-\psi^{(O)}_{t}H^{beg}_{t})}_{\text{cash investment}}e^{r}+\psi^{(S)}_{t}(S_{t+1}-S_{t})+\psi^{(O)}_{t}(H^{end}_{t+1}-H^{beg}_{t}),$$

with $(\psi^{(S)}_{t},\psi^{(O)}_{t})$ being the respective portfolio positions on time interval $[t,t+1)$ in the stock and an option used for hedging, and $H^{beg}_{t}$ and $H^{end}_{t+1}$ being the respective time-$t$ and time-$(t+1)$ price of the hedging option purchased at $t$. The positions are thus rebalanced at each period, and option positions are rolled-over, with the hedging options currently in the portfolio being liquidated at the end of the period while new ones are being purchased. At the start of any period $[t,t+1)$, the option considered for purchase is at-the-money (its strike is $S_{t}$ and its maturity is $\tau=0.1\times 52=5.2$, meaning $10$% of a year). As such, $H^{beg}_{t}=\Pi(t,0.1\times 52,S_{t})$ and $H^{end}_{t+1}=\Pi(t+1,0.1\times 52-1,S_{t})$. Note that unless $S_{t}=S_{t+1}$, $H^{end}_{t+1}\neq H^{beg}_{t+1}$ since options being included in the hedging portfolio change on the various periods. Moreover, $V^{\theta}_{0}=\Pi(0,T,S_{0})$ is the option premium that is initially invested in the hedging portfolio.

The optimal policy should characterize the selection of positions $\psi^{(S)}_{t},\psi^{(O)}_{t}$, $t=0,\ldots,T$ to be included in the hedging portfolio. Assume that the agent wants to be fully Delta-neutral, which is obtained with

$$\psi^{(S)}_{t}=\underbrace{\Delta(t,T-t,S_{0})}_{\text{target option }\Delta}-\psi^{(O)}_{t}\underbrace{\Delta(t,0.1\times 52,S_{t})}_{\text{hedging option }\Delta}.$$

However, we assume that the agent might prefer not fully neutralizing the Gamma of the target option due to purchases of hedging options being too costly in a market with large volatility risk premium. The agent shall therefore only neutralize a portion $\theta\in(0,1)$, called the hedge ratio, of the target option Gamma. This leads to $\psi^{(O)}_{t}\Gamma(t,0.1\times 52,S_{t})=\theta\Gamma(t,T-t,S_{0})$, and thus to $\psi^{(O)}_{t}=\theta\frac{\Gamma(t,T-t,S_{0})}{\Gamma(t,0.1\times 52,S_{t})}$.

The objective of the hedging agent is therefore to find the optimal hedge ratio, which is the optimal policy parameter $\theta$. A single terminal cost is considered for the agent: $C_{t}=\mathds{1}_{\{t=T\}}\left(\max(0,S_{T}-E)-V^{\theta}_{T}\right)$ and no discount factor is considered $\gamma=1$. The agent thus attempts minimizing risk associated with catastrophic hedging shortfalls at maturity: hence consider $\alpha=0.999$.

Before applying the reinforcement learning procedure, we want to approximate the objective function $J(\theta)$, the CVaR_0.999 of the hedging shortfall, for various hedge ratios $\theta$. Such approximations are produced with brute force Monte-Carlo simulations, where for several values of $\theta$, $1,\!000,\!000$ realizations of the hedging shortfall $\max(0,S_{T}-E)-V^{\theta}_{T}$ are produced and sample averaging is applied, i.e. $J$ is estimated by the $1,\!000$ largest realizations. Figure 2 reports such estimates, with the optimal hedge ratio being estimated to be $\theta^{*}=0.5991$ and the corresponding objective function being $\hat{J}(\theta^{*})=40.37$.

Figure 2: Objective function (CVaR_0.999 of the hedging shortfall) versus the hedge ratio $\theta$, representing the percentage of the target option Gamma being neutralized. Estimates are obtained by brute force calculations, i.e. through sample averaing over $1,\!000,\!000$ simulated paths. Red point: optimal value.

Now apply the POTPG algorithm to the policy optimization problem, and compare its performance to that of the sample averaging method (SA). Such methods are applied with either $n=1,\!000$ or $n=10,\!000$ simulated paths of weekly stock returns. $R=100$ independent runs are conducted, each comprised of $M=500$ iterations for the case $n=1,\!000$, or $M=150$ iterations when $n=10,\!000$. In each run, the initial policy is set to $\theta^{(0)}=0$. The finite difference shock is $\epsilon=0.05$. The method of moments is used to estimate tail parameters $\xi,\sigma$ in the POTPG algorithms since such method exhibited (in unreported tests) greater stability than maximum likelihood estimates in the presented framework.

Figure 3 reports the performance of the POTPG and SA policy gradient algorithms for the hedging problem, by displaying the evolution of the RMSE (across runs) of the estimate of the optimal policy parameter (RMSE_θ) and the corresponding objective function (RMSE${}_{\hat{J}}$) versus the number of iteration conducted. The POTPG algorithm exhibits materially superior performance by exhibiting much lower errors on estimates for the optimal policy parameter and objective function. The gap in performance between the POTPG and the benchmark is greater for the lower sample size $n=1,\!000$, which highlights that our method has more added value in the context of more severe distribution tail data scarcity. Note that none of the two methods have the estimated policy parameter converge to the true optimal value (i.e. RMSE_θ does not converge to zero), which can be explained by the fact that both methods are biased in finite sample $n$. Nevertheless, we see that higher sample size $n$ increases the precision, with lower RMSEs for the estimates of the policy parameter $\theta^{*}$ and of the objective function $J$.

(a) RMSE_θ, $n=1,\!000$

(b) $\text{RMSE}_{\hat{J}}$, $n=1,\!000$

(c) RMSE_θ, $n=10,\!000$

(d) RMSE${}_{\hat{J}}$, $n=10,\!000$

Figure 3: Evolution of the RMSE of the estimate of the optimal policy parameter (RMSE_θ) and the corresponding objective function (RMSE${}_{\hat{J}}$) over iterations of the POTPG algorithm and the sample averaging (SA) benchmark. Top row: sample size $n=1,\!000$. Bottow row: $n=10,\!000$. Left panels: RMSE_θ. Right panels: RMSE${}_{\hat{J}}$.

6   Conclusion

We propose a policy gradient algorithm based on estimators of tail risk borrowed from extreme value theory to tackle the difficult task of catastrophic risk minimization within a sequential decision making framework. The peaks-over-threshold procedure is used to estimate the CVaR of cumulative costs by leveraging the asymptotic convergence of the tail distribution to a generalized Pareto distribution. We have shown in several simulation experiments, including an application to financial options hedging, that our method can outperform conventional benchmarks relying on the empirical distribution of the cumulative costs. Indeed, such benchmarks can perform quite poorly to mitigate extreme risk when observations in the tail are scarce.

Our method relies on finite difference approximations for the gradient, and as such it work for low-dimensional policies relying on a small number of parameters. An extension of our approach could consist in develo** a high-dimensional EVT-based policy gradient framework to tackle more complex problems. This would for instance allow using policies represented by deep neural networks and combine the EVT-based policy gradient with deep reinforcement learning.

Appendix A     The automated threshold selection procedure

The automated threshold selection procedure involves testing several thresholds $u_{i}=\hat{F}^{-1}(q_{i})$, $i=1,\ldots,l$, which we choose to be quantiles of pre-determined levels $q_{1},\ldots,q_{\ell}$ of the empirical distribution of the sample $X_{1},\ldots,X_{n}$. For each $i$, denote by $k_{i}$ the number of threshold excesses. Assuming that a threshold $u_{i}$ leads to GPD parameter estimates $(\hat{\xi}_{u_{i}},\hat{\sigma}_{u_{i}})$ for the distribution of the excesses $\mathcal{Y}_{u_{i}}=\{X_{i}-u_{i}:X_{i}>u_{i}\}$, the Anderson-Darling test statistic for such threshold is

$$A_{i}^{2}=-k_{i}-\frac{1}{k_{i}}\sum_{j=1}^{k_{i}}(2j-1)\left[\log\left(Z_{(j,i)}\right)+\log\left(1-Z_{(k_{i}+1-j,i)}\right)\right].$$ (6.1)

where $Z_{(j,i)}=G_{\hat{\xi}_{u_{i}},\hat{\sigma}_{u_{i}}}(Y_{(j,i)})$ with $Y_{(j,i)}$ being the $j^{th}$ smallest excess value, i.e. among values in $\mathcal{Y}_{u_{i}}$. The automated selection procedure attempts using the smallest possible threshold for which no threshold above would be deemed inadequate. In the application, we choose $l=20$, $q_{1}=0.79$, $q_{2}=0.80,\ldots,q_{20}=0.98$ and $\xi_{max}=0.9$.

Remark 6.1.

If the automated threshold selection procedure is unsuccessful, i.e. $I=\emptyset$, the sample averaging estimate in (2.5) is used for CVaR, as a fallback estimate.

Algorithm 2 Automated threshold selection procedure (from Troop et al., (2021))

Significance parameter $\gamma$, cutoff $\xi_{\text{max}}<1$, i.i.d. sample $X_{1},\ldots,X_{n}$, threshold percentiles $0<q_{1},\ldots,q_{l}<1$.

$I\leftarrow\emptyset$

for $i=1,\ldots,l$ do

     Set $u_{i}=\hat{F}^{-1}(q_{i})$

     Compute $(\hat{\xi}_{u_{i}},\hat{\sigma}_{u_{i}})$ from the $k_{i}$ threshold excesses over $u_{i}$

     if $\hat{\xi}_{u_{i}}\leq\xi_{\text{max}}$ then

         Compute $A_{i}^{2}$ using (6.1)

         Set $p_{i}$ to $p$-value for $A_{i}^{2}$ using a lookup table

         $I\leftarrow I\cup\{i\}$      

if $I\neq\emptyset$ then

     Set $W=\{w\in I\,|\,-\frac{1}{w}\sum_{i=1}^{w}\log(1-p_{i})\leq\gamma\}$

     if $W\neq\emptyset$ then

         Compute $\hat{w}_{F}=\max W$

         if $\hat{w}_{F}=\max(I)$ then

              $v\leftarrow\max(I)$

         else

              $v\leftarrow\min\{w\in I\,|\,w>\hat{w}_{F}\}$          

     else

         $v\leftarrow\min(I)$      $u\leftarrow u_{v}$

     Return $u_{v}$, $(\hat{\xi}_{u_{v}},\hat{\sigma}_{u_{v}})$