Stock Volume Forecasting with Advanced Information by Conditional Variational Auto-Encoder

Abstract.

We demonstrate the use of Conditional Variational Encoder (CVAE) to improve the forecasts of daily stock volume time series in both short and long term forecasting tasks, with the use of advanced information of input variables such as rebalancing dates. CVAE generates non-linear time series as out-of-sample forecasts, which have better accuracy and closer fit of correlation to the actual data, compared to traditional linear models. These generative forecasts can also be used for scenario generation, which aids interpretation. We further discuss correlations in non-stationary time series and other potential extensions from the CVAE forecasts.

1. Introduction

1.1. Motivation

Let $Y_{t+k|t}$ be the forecast of variable $Y$ at time $t+k$, given information available up to time $t$. For instance, in the case of daily stock data where $Y_{t}$ indicates the end of day stock price of day $t$, $Y_{t+1|t}$ would be the forecasted stock price on day $t+1$ based on information up to time $t$.

Advanced information concerns with the notion of information available up to time $t$. In linear time series, such information is mainly the past lags of the data (autoregressive terms) and error terms, together with other features observed up to time $t$. Advanced information acknowledges the future state of some variables at time $t$. For instance, there is an explicit rule of when Stoxx index rebalancing would occur (STOXX, 2024) and this information may concern the future state $t+k$, but is known at time $t$. If we write $RB_{t}$ as the indicator of whether day $t$ is a rebalancing day for Stoxx index, then $RB_{t+k}$ is known at time $t$ for all $k$. It can therefore be helpful to utilise such advanced information at time $t$ to construct a forecast of $Y$ at time $t+k$, especially when $k$ is large. However, how to incorporate such an advanced information to the time series model becomes a crucial question, in light of the potential non-linear effect this information has on $Y_{t}$. In this paper, we use Conditional Variational Auto-Encoder (CVAE) to account for this in a non-linear modelling and forecasting setting.

The notion of long term forecasting concerns $k$ being large, for instance, the stock price two weeks (ten trading days) later, $Y_{t+10|t}$. In the analysis of linear stationary time series (e.g. ARMA and VARMA), the long term forecasts converge to the unconditional expectation due to the assumption of stationarity. However, in the case of non-stationary time series, such convergence is not guaranteed, and can be of interests.

Empirically, in the case of stock price, it may be convenient to put $Y_{t+10|t}$ or even $Y_{t+100|t}$ to some trend-stationary expectation or equivalent, as the difference of stock prices (either $Y_{t}-Y_{t-1}$ or $\log(Y_{t})-\log(Y_{t-1})$) are often modelled as a stationary time series. In the case of stock volume, however, it is not as convenient to assert such stationary expectation, as stock volumes are often affected by extraordinary shocks, e.g. company announcements and index rebalancing. It is therefore a harder task, both in modelling and forecasting, to provide analysis for stock volume time series.

Furthermore, forecasting stock volume in a non-linear fashion helps the pricing of various financial derivatives, such as stock buyback contracts, which heavily depend on market volumes for periods covered by such contracts.

1.2. Literature Review

There are some recent literature on time series with the use of neural network, such as Time Series generative adversarial networks (Yoon et al., 2019) and Generative Time Series with bi-directional Variational Auto-Encoder (VAE) (Li et al., 2022). We take a similar approach in designing the latent space and building Gaussian assumptions conditional on latent space. However, we take into account the advanced information and utilise such information to improve the quality of our forecast, with the incorporation of latent variables. We also use CVAE to enable scenario generation and interpretation. To this end, we engage with technical papers on CVAE (Doersch, 2016; Sohn et al., 2015) for modelling and derivations under a time series setting.

In terms of the literature of advanced information, there are no direct reference of this terminology in time series, however, we find similar concepts in Bayesian Time Series (Berliner, 1996; Tsay and Chen, 2018), where one updates the state equations based on some signals. Such signal may be learnt or stated in advanced for the model — advanced information may be considered as a set of variables where we know the existence of its value in advance, and need not be learnt for the duration of present and future forecasting period.

The empirical need for stock volume forecasting can be motivated in the general financial machine learning literature (De Prado, 2018), though most of the recent literature focus on the microstructure statistics, such as in the case of limit order book (Cont et al., 2013). The role of stock volume time series in buyback contract pricing has been mentioned in recent literature (Guéant et al., 2020; Hamdouche et al., 2022), which introduce practical forecasting problems that we aim to empirically contribute to.

1.3. Contributions

Our overall contributions can be summarised into threefold. Firstly, we identify the class of problem of forecasting with advanced information, which can be considered as an expectation computation based on a richer information set — this is practically modelled with non-linear interactions in a CVAE architecture of neural networks, which additionally enables generative forecasting. This is elaborated in section 2.

Secondly, we demonstrate the capability for event-driven interpretations and alternative scenario generation. This utilises the generator aspect of CVAE to answer questions such as what happens on the special occasion and what are the alternative scenarios. By generating the forecast paths under different conditional values, we are able to answer these questions and henceforth provide interpretation to the model. These are provided in addition to the traditional model evaluation metrics such as Mean Squared Error (MSE) and correlation matrix, which are detailed in section 3.

Lastly, we contribute to the empirical literature on daily stock volume forecasting, which is studied across the incumbents of EURO STOXX 50 index, which is a cluster of 50 high market-capitalisation stocks listed in the Eurozone. This contributes to the demand of long term forecasting in empirical finance, such as for buyback contracts.

1.4. Notations

Most of the notations are explained when first introduced. Common annotations are as follows. $N(\cdot,\cdot)$ indicates the Gaussian Distribution, $I_{q}$ indicates a q-dimensional identity matrix, upper case letters usually denote objects being a random variable, whereas lower case letters usually denote data observed at specific values. $\mathbb{E}[\cdot]$ denotes the expectation operator, $V(\cdot)$ denotes the variance operator, with $Cov(\cdot,\cdot)$ as the covariance operator, and $Corr(\cdot,\cdot)$ as the correlation operator. Conditional expectation $\mathbb{E}_{X\sim P}[Q(X)]$ refers to the expectation of $Q(X)$ conditional on $X\sim P$.

2. Methodology: From Non-Linear Modelling To Generative Algorithm For Forecasting

2.1. Time series in a CVAE context

We first give an overview of the modelling assumptions of CVAE, and put time series in such a context.

Let $X,Y$ be random variables. $Y\in\mathbb{R}^{d}$ and $X\in\mathbb{R}^{p}$. Let $Z\in\mathbb{R}^{q}$ be a latent variable with distribution $Z\sim N(0,I_{q})$. Let $D$ be the dataset consisting paired observations of $(X,Y)$. The key assumptions are: conditional distribution of the output given input and latent variable as Gaussian with non-linear and unknown mean, written as

(1) $$Y|X,Z\sim N(f(X,Z),\sigma^{2}I_{d})$$

for some unknown function $f:\mathbb{R}^{p}\times\mathbb{R}^{q}\to\mathbb{R}^{d}$, and conditional distribution of latent variable given the observed data ($X,Y$) is Gaussian, written as

(2) $$Z|X,Y\sim N(\mu(X,Y),\Sigma(X,Y))$$

A CVAE is a tuple $(\hat{f}^{en},\hat{f}^{de})$ where $\hat{f}^{de}:\mathbb{R}^{p}\times\mathbb{R}^{q}\to\mathbb{R}^{d}$ is a decoder which induces a probability distribution $\hat{P}(Y|X,Z)$ according to Equation 1 and an encoder is a function $\hat{f}^{en}:\mathbb{R}^{p}\times\mathbb{R}^{d}\to\mathbb{R}^{q}\times(0,\infty)^{q}$ which induces the moments of the Gaussian distribution $Z|X,Y$ according to Equation 2, with the assumption that $\Sigma(X,Y)$ can be written as a diagonal matrix with positive entries on all diagonals.

Further to this, time series $y_{t},x_{t}$ are seen as observation of random variables $Y,X$ at time $t$, and by deliberate modelling, we aim to train a CVAE $(\hat{f}^{en},\hat{f}^{de})$ that enables the forecast of future states of $y_{t+k}$ conditional on $x_{t+1}$. For example, let $x_{t}=y_{t-1}$, then $y_{t+1|t}=\mathbb{E}_{Z\sim N(0,I_{q}),X=y_{t}}[N(\hat{f}^{de}(X,Z),\sigma^{2}I_{d})]=\mathbb{E}_{Z\sim N(0,I_{q})}[N(\hat{f}^{de}(y_{t},Z),\sigma^{2}I_{d})]=\mathbb{E}_{Z\sim N(0,I_{q})}[\hat{f}^{de}(y_{t},Z)]$. Iteratively for $k\geq 2$, we have $y_{t+k|t}=\mathbb{E}_{Z\sim N(0,I_{q})}[\hat{f}^{de}(y_{t+k-1|t},Z)]$.

The training of a CVAE is done by using two Neural Network architectures $F_{1},F_{2}$ to define the functions ${f}^{en}\in F_{1},{f}^{de}\in F_{2}$, followed by gradient methods and other classical optimisation techniques to maximise $\mathbb{E}_{X,Y\sim D}[\log(P(Y|X))]$ through conditional marginalisation over $Z$. Derivations and other technical remarks are written in appendix A.1.

2.2. Generative scheme from CVAE

For a given decoder $\hat{f}^{de}$ and conditional variable $x_{t}$, the distribution $\mathbb{E}_{Z\sim N(0,I_{q}),X=x_{t}}[N(\hat{f}^{de}(X,Z),\sigma^{2}I_{d})]$ can be approximated by the generative scheme of $S$ samples

$$\forall s\in[S], \texttt{ draw }z_{s}\sim N(0,I_{q}),$$

(3) $$\texttt{ then draw }y_{t}^{s}\sim N(\hat{f}^{de}(x_{t},z_{s}),\sigma^{2}I_{d})$$

Now, through Equation 3, we may compute the approximated expectation by taking the average across samples $\{y_{t}^{s}\}_{s\in[S]}$.

Putting this scheme in a forecasting regime, we may generate forecasts $\{y_{t+1|t}^{s}\}_{s\in[S]}$ and thereafter $\{y_{t+k|t}^{s}\}_{s\in[S]}$ for a generalised $k$. For a given horizon of forecast $K$, we call a forecast path as the vector $y^{s}_{t+\cdot|t}:=(y_{t+1|t}^{s},y_{t+2|t}^{s},...,y_{t+K|t}^{s})$, and the average forecast path $\bar{y}_{t+\cdot|t}:=(\bar{y}_{t+1|t},\bar{y}_{t+2|t},...,\bar{y}_{t+K|t})$ where each entry $\bar{y}_{t+k|t}$ is the arithmetic average over the generated samples $\{y_{t+k|t}^{s}\}_{s\in[S]}$.

2.3. A special forecasting scenario: advanced information

In this section, we formaly introduce, and give examples to, the notion of advanced information.

Given $K\geq 1$, we are interested to forecast random variable $Y_{t+k},k\in[K]$ given information availble up to time $t$. Let $X_{t}=(X_{t}^{0},X_{t}^{1})$ where $X_{t+k+1}^{0}$ is always known at time $t$, for $k\in[K]$ but only up to $X_{t+1}^{1}$ is known at time $t$, not $X_{t+2}^{1}$ or anything further. We say $X^{0}$ is an advanced information and $X^{1}$ is an ordinary information. Intuitively, $X^{1}$ is the part of information where we only know up to the time we are requested to forecast, as was commonly assumed in classical linear time series models, whereas $X^{0}$ is the nuanced part where we know some information ahead of time. Then, we consider filtration ${\mathscr{F}}^{*}_{t}:={\mathscr{F}}^{0}_{t+K}\times{\mathscr{F}}^{1}_{t}$ where ${\mathscr{F}}^{0}_{t+K}$ is the filtration generated by $X_{t+K}^{0}$ and ${\mathscr{F}}^{1}_{t}$ is the filtration generated by $X_{t+1}^{1}$. Forecasting with advanced information concerns investigating the conditional distributions $\mathbb{Q}^{*k}(\cdot|{\mathscr{F}}^{*}_{t})$ where $k\in[K]$ and $\mathbb{Q}^{*k}(S|{\mathscr{F}}^{*}_{t})$ measures the probability of $Y_{t+k}\in S$ given ${\mathscr{F}}^{*}_{t}$. The expected forecast takes the form of $Y_{t+k|t}:=\int Y_{t+k}d\mathbb{Q}^{*k}(Y_{t+k}|{\mathscr{F}}^{*}_{t})$ .

To give some contextual remark, the inclusion of $X^{0}$ can often be considered as properties of the time series, such as category of variables. Say $Y\in\mathbb{R}$ and some of the data belong to group 1 whereas the other belong to group 2, then $X^{0}\in\{0,1\}^{2}$ can be a one-hot indicator function which outputs $(1,0)$ for group 1 and $(0,1)$ for group 2. In this case, when forecasting, $X^{0}$ is always known ahead of time (with effectively infinitely large $K$). This means many panel data models fall into the scenario of advanced information, as the category of the response can be considered as a known property, henceforth advanced information. Likewise for seasonality, where such category could be considered as an information exploitable ahead of the forecasting time.

Another example is rebalancing, as was motivated in section 1.1, that rebalancing dates are known ahead of the desired forecasting time, so $X^{0}$ may be an indicator of rebalancing date which outputs $1$ if the date is a rebalancing date, and $0$ otherwise.

As a remark, $X^{0}$ may also be a special case for cointegration, as a source of drift. This may fall into the wider literature of ’common trend removal’ in cointegration analysis.

2.4. Forecasting algorithms using advanced information and CVAE

Algorithm 1 Iterative Forecasting with Advanced Information (General)

Input: $t$ (time the forecast is requested), $S$ (number of samples), $\hat{f}^{de}$ (a trained decoder), $K$ (desired forecasting horizon), $\{x_{\tau}^{0}\}_{\tau=t+1}^{t+K}$ (advanced information), $x_{t+1}^{1}$ (ordinary information).

Output: Simulated forecast paths $\{(k,y_{t+k|t}^{s}):k\in[K]\}_{s\in[S]}$

1:  Draw S samples from the distribution $\mathbb{E}_{Z\sim N(0,I_{q}),X^{0}=x^{0}_{t+1},X^{1}=x^{1}_{t+1}}[N(\hat{f}^{de}(X^{0},X^{1},Z),\sigma^{2}I_{d})]$ according to Equation 3. These samples are denoted as $y_{t+1|t}^{s}$ for $s\in[S]$.

2:  for $\tau\in\{2,3,...,K\}$ do

3:     Update $x^{1}_{t+\tau|t}$ with $\{y_{t+\tau-1|t}^{s}:s\in[S]\}$

4:     Draw S samples from the distribution $\mathbb{E}_{Z\sim N(0,I_{q}),X^{0}=x^{0}_{t+\tau},X^{1}=x^{1}_{t+\tau|t}}[N(\hat{f}^{de}(X^{0},X^{1},Z),\sigma^{2}I_{d})]$ according to Equation 3. These samples are denoted as $y_{t+\tau|t}^{s}$ for $s\in[S]$.

5:  end for

6:  return  $\{(k,y_{t+k|t}^{s}):k\in[K]\}_{s\in[S]}$

In algorithm 1, we present the iterative forecasting algorithm with advanced information, under the setting where decoder ${f}^{de}$ takes $X^{0},X^{1},Z$ as input. The algorithm uses Equation 3 for the CVAE generation and serves as a practical algorithm for forecasting with advanced information.

Another algorithm as a special case for AR(1)-type of forecasting (the case where $X^{1}_{t}=Y_{t-1}$) is presented in algorithm 2, where there are more specified approach in handling $X^{1}$, as would be done in classical linear time series models. This is also the exact algorithm used in the empirical applications section next.

Algorithm 2 Iterative Forecasting with Advanced Information and 1-lag Autoregressive Ordinary Information

Input: $t$, $S$, $\hat{f}^{de}$, $K$, $\{x_{\tau}^{0}\}_{\tau=t+1}^{t+K}$, $x_{t+1}^{1}$, and $y_{t}$ (last observation of $Y$ at the time of forecast)

Output: $\{(k,y_{t+k|t}^{s}):k\in[K]\}_{s\in[S]}$

1:  Draw S samples from the distribution $\mathbb{E}_{Z\sim N(0,I_{q}),X^{0}=x^{0}_{t+1},X^{1}=y_{t}}[N(\hat{f}^{de}(X^{0},X^{1},Z),\sigma^{2}I_{d})]$ according to Equation 3. These samples are denoted as $y_{t+1|t}^{s}$ for $s\in[S]$.

2:  for $\tau\in\{2,3,...,K\}$ do

3:     Average $\hat{y}_{t+\tau-1|t}=\frac{\sum_{s\in[S]}y_{t+\tau-1|t}^{s}}{S}$

4:     Draw S samples from the distribution $\mathbb{E}_{Z\sim N(0,I_{q}),X^{0}=x^{0}_{t+\tau},X^{1}=\hat{y}_{t+\tau-1|t}}[N(\hat{f}^{de}(X^{0},X^{1},Z),\sigma^{2}I_{d})]$ according to Equation 3. These samples are denoted as $y_{t+\tau|t}^{s}$ for $s\in[S]$.

5:  end for

6:  return  $\{(k,y_{t+k|t}^{s}):k\in[K]\}_{s\in[S]}$

3. Empirical Application: Daily Stock Volume Forecasting

In this section, we model and forecast daily stock volume for 50 European stocks which were components of Euro Stoxx 50 as of the end of year 2023.

3.1. Data Availability and Processing

Daily stock volume data were obtained from Yahoo Finance. We split the training and testing as start of year 2021 to end of year 2022, and start of year 2023 to end of June 2023, respectively. We use the traning data to normalise the time series — that is, for each stock, we find mean and variance in training period, then de-mean and unify the variance, as one of the standard Machine Learning data processing procedure.

Figure 1. Illustrations of raw data and processed data for training and testing

In addition to the volume data, we obtain stoxx rebalancing dates according to the STOXX index guide, and individual stock categories (Location and Sector) from the index-tracking ETF, EUE. These are seen as advanced information in modelling and foreacsting, as the information attributed are either known ahead of time, or believed to be unchanged throughout the forecasting horizon.

As an illustration on some of these data, in Figure 1, we plot two of the stocks (ASML.AS and BNP.PA) in both row data and processed data. The processed data rows (the second row for training and the third row for testing) also contains vertical red lines highlighting the rebalancing dates. It is visually clear that rebalancing dates do tend to co-occur with a higher-than-average volume — this is something we would wish our model to capture, both in in-sample modelling and out-of-sample forecasting.

We annotate the scalar $y(i)_{t}$ as the normalised observed stock volume in day $t$ and for stock $i\in[N]$ where $N=50$, and the vector $\bm{y}_{t}\in\mathbb{R}^{50}$ as the vector of observed stock volume in day $t$ across all stocks. Where there are missing observations in a day, we drop such an observation.

As for features, $RB_{t}\in\{0,1\}^{3}$ denotes a three-dimensional one-hot encoder indicating the day relative to rebalancing day — in particular, $RB_{t}=(1,0,0)$ indicates the last observation before rebalacing day, $RB_{t}=(0,1,0)$ indicates the rebalacing day, and $RB_{t}=(0,0,1)$ indicates the next observation after rebalacing day. $RB_{t}=(0,0,0)$ for all other days. The feature $DoW_{t}\in\{0,1\}^{5}$ denotes the day of the week of the observation. Another set of features are $Sector(i)_{t}\in\{0,1\}^{10}$ and $Location(i)_{t}\in\{0,1\}^{7}$ which are one-hot encoders for the sector of the stock $i$ and location of stock $i$ respectively. They are invariant over time and henceforth considered as advanced information.

3.2. Forecasting Tasks

We would like to accomplish two forecasting tasks through our models.

The long term forecasting concerns the scenario where we are given large $K$ and a fixed $t$ (end of the training sample), and we request to forecast $y(i)_{t+k|t}$ for $k\in[K],i\in[N]$. In this dataset, we have $K=120$ representing the first half of 2023 (business days and after removal of days with empty observations due to holidays or data source error).

The short term rolling forecast concerns the scenario of week-long forecasts on multiple periods, hence $K\leq 5$ ($K=5$ is often the case where the forecast run from Monday to Friday — in some cases such as the public holiday, it would reduce to $K<5$). Given a set of time points $t_{1},...,t_{u}$ indicating end of the current week, and some $k_{1},...,k_{u}\leq 5$ which indicates the duration of forecasts from the start of the upcoming week to the end of the upcoming week, we request to forecast $y(i)_{t+k^{\prime}|t}$ for $k^{\prime}\in[k],(t,k)\in\{(t_{1},k_{1}),...,(t_{u},k_{u})\},i\in[N]$.

3.3. Summary of Models

Recall the setting of CVAE as per Equation 1 and Equation 2. We model a univariate CVAE (U-CVAE) on each individual $y(i)_{t}$ and a multivariate CVAE (M-CVAE) on the vector $\bm{y}_{t}$. W keep $q=1$ for simplicity.

For the U-CVAE, we have $Y(i)_{t}\in\mathbb{R}$ and model

(4) $$Y(i)_{t}|X(i)_{t},Z\sim N(f(X(i)_{t},Z),\sigma^{2}),$$

(5) $$X(i)_{t}=(X(i)^{0}_{t},X(i)^{1}_{t}),\ \ Z\sim N(0,1)$$

with $X(i)^{0}_{t}=(Sector(i)_{t},Location(i)_{t},DoW_{t},RB_{t})$ and $X(i)^{1}_{t}=Y_{t-1}(i)$.

For the M-CVAE, we have $\bm{Y}_{t}\in\mathbb{R}^{50}$ and model

(6) $$\bm{Y}_{t}|X_{t},Z\sim N(f(X_{t},Z),\sigma^{2}I_{50}),$$

(7) $$X_{t}=(X_{t}^{0},X_{t}^{1}),\ \ Z\sim N(0,1)$$

with $X_{t}^{0}=(DoW_{t},RB_{t})$ and $X_{t}^{1}=\bm{Y}_{t-1}$.

For comparison, we also provide two baseline models: for the univariate baseline, we model and forecast $y(i)_{t}$ using ARMA(1,1); and for the multivariate baseline, we do so on $\bm{y}_{t}$ using VAR(1).

3.4. Evaluation Metrics

Figure 2. Short Term Rolling Forecasts: M-CVAE and VAR(1) Illustrations

Figure 3. Long Term Forecasts: U-CVAE and ARMA(1,1) Illustrations

To evaluate the forecasts, we employ two core concepts in time series: Mean Squared Errors (MSE) and correlations. The computation of MSE is straightforward: given forecasts $\hat{y}(i)_{t+k|t}$ generated by the models, we compute the average MSE by comparing against the actual observation in the testing period: $y_{t+k}$, namely $(\hat{y}(i)_{t+k|t}-y_{t+k})^{2}$ averaged across the testing period (the precise notation varies depending on whether it is summed over short-time rolling forecast and long term forecast) for each stock $i$, denoted $MSE(i)$. Summary of statistics (mean and median) can then be obtained for $MSE(i)$.

The correlation matrix may also be produced for $\hat{\rho}_{i,j}=corr(\hat{y}(i)_{t+k|t},\hat{y}(j)_{t+k|t})$ against the correlation of the data ${\rho_{i,j}}=corr(y(i)_{t+k},y(j)_{t+k})$ (exact formulations are drawn in section 4.1 where we further discuss the correlation of non-stationary time series). To summarise the difference, we apply average absolute value to obtain the difference, denoted CD for correlation difference

$$CD(i)=\frac{1}{50}\times\sum_{j\in[50]}|\hat{\rho}_{i,j}-\rho_{i,j}|$$

Summary of statistics (mean and median) can then be obtained for $CD(i)$.

The cross-correlation matrix is also considered, as one may wish to observe the difference between lagged correlation amongst the forecasts. In particular, we produce $\hat{\rho}^{*}_{i,j}=corr(\hat{y}(i)_{t+k|t},\hat{y}(j)_{t+k+1|t})$ and compare against the one by data ${\rho^{*}_{i,j}}=corr(y(i)_{t+k},y(j)_{t+k+1})$. We then summarise the difference in the same fashion as was done in $CD(i)$. We denote this statistic as $CCD(i)$ for cross correlation difference.

3.5. Summary of Performance

As a summary of the performance in forecasting, we provide evaluation metrics in Table 1 and Table 2 for the long term forecasting task and short term rolling forecasts respectively. Illustrations of these forecasts are plotted in Figure 2 and Figure 3, where we can see immediate difference between CVAE forecasts and their baseline models.

	U-CVAE	M-CVAE	ARMA(1,1)	VAR(1)
mean MSE	0.887	0.888	0.923	0.981
median MSE	0.876	0.884	0.922	1.001
mean CD	0.466	0.391	0.441	0.492
median CD	0.435	0.374	0.421	0.458
mean CCD	0.106	0.146	0.755	3.669
median CCD	0.096	0.136	0.837	3.644

Table 1. Performance of Long Term Forecasts

	U-CVAE	M-CVAE	ARMA(1,1)	VAR(1)
mean MSE	0.793	0.788	0.971	1.070
median MSE	0.737	0.761	0.989	1.118
mean CD	0.240	0.275	0.093	0.193
median CD	0.227	0.258	0.083	0.179
mean CCD	0.124	0.262	0.377	0.420
median CCD	0.105	0.271	0.392	0.421

Table 2. Performance of Short Term Rolling Forecasts

As a summary, we see that the CVAE forecasts do better job in both long term and rolling short term forecasting tasks, with the out-performance in MSE — more significantly so in the short term forecasts, and good fit of CCD in all cases. Correlation matrix fit well, though under-performs their baseline counterparts in short term forecasting tasks. Illustrations of correlation matrices are plotted in Figure 9 in the appendix.

From the illustrations, we can further appreciate such out-performance in two folds: quite significantly so in long term forecasts, the CVAE takes into account the advanced information in both modelling and forecasts, and are able to project spikes in a non-linear time series fashion that matches some of the spikes in the actual observation. Additionally, in short term forecasts, baseline models tend to have heavy reliance on their lagged dependent variables, which creates problems as to over- or under-forecasts in their forecasting horizon, these are partly mitigated in CVAE forecasts as they moderate these with the trained parameters of advanced information and other features.

There are further potential improvements in short-term forecasts: linear baselines do well in the correlation fitting — this may be seen in the lower part of Figure 9 and the CD entries in Table 2. Despite having low accuracies, linear models still preserve the correlation structures in their forecasts, resulting in a better fit — whereas the CVAE models, despite better fit, tend to have more variabilities in their forecasts, and consequentially tend to over-fit in correlation when comparing to the actual data. In cross correlation, however, CVAE models still significantly outperform linear baselines.

We further some of these interpretations by zooming into the forecasts and discuss alternative scenarios of the feature values in the next section.

3.6. Decoder as Generator: feature interpretation and scenario generation

In this section, we address two questions which help to appreciate the value of CVAE forecasts: How does RB affect the forecasts? And what’s the effect of the lagged dependent variable (similar to the IRF analysis in linear time series)? To engage with empirical data, we zoom into the long term forecast of ASML in March - April 2023 (illustrated in Figure 4) to answer the first question, and the short term rolling forecast of BNP in the same period (Figure 6) for the second question.

Figure 4. Long Term Forecasts: A zoomed-in plot for all models in March - April 2023, for ticker ASML.AS

From Figure 4, we can closely observe the ability of CVAE-generated forecasts to match the spike on the 17th March 2023, which is benefited from the advanced information of rebalancing date indicators (RB). Their corresponding baselines stay flat as the convergence of stationary forecasts would yield, when forecast period becomes large (the last observation being the end of year 2022).

Further to this, we may analyse the counterfactual of CVAE-generated forecasts in the case where there would be no rebalancing events in that week. To do this, we may augment the advanced information $X^{0}$ such that the $RB_{t}=(0,0,0)$ for the period of forecasts. We then generate the new forecast paths using algorithm 2 with the augmented advanced information $X^{0}$.

Figure 5. RB interpretation: ASML illustration (generated by U-CVAE)

In Figure 5, we plot the paths (in grey) and their mean (in red), 97.5% quantile (in blue) and 2.5% quantile (in black). On the left panel are the forecasts with the actual $X^{0}$ where rebalancing happens on Friday, whereas on the right are the newly generated forecasts under the counterfactual with absence of rebalancing. There are slight difference on Thursday (16th March) paths as the one with rebalancing moves up and slightly tighter on the day before rebalancing date, and a huge difference on Friday (17th March) where rebalancing kicks up the forecast on the left panel — the counterfactual in the absence of rebalancing date shows a relatively normal mean and upper and lower quantiles.

Figure 6. Short Term Rolling Forecasts: A zoomed-in plot for all models in March - April 2023, for ticker BNP.PA

Using a similar convention, we may observe from Figure 6 that a spike in the data was observed on the 24th March. This comes in the episode of a higher-than-usual volume in March 2023 (mainly due to the European banking sector distress during that period). We zoom in to the last week of March (week commencing 27th March) for the analysis of lagged impact. As it is rolling forecasts, we can see U-CVAE and its baseline all picked up the last observed spike in their upcoming short term forecasts of 5 days. The spike on the 24th March was recorded at just below 5 — it is tempting to seek for the impulse of this extraordinary observation by comparing against a counterfactual of the observation being at zero or at the negative of what was observed (just above -5).

To do this, two alternative paths are generated by replacing the $X^{1}_{t}=y_{t}$ part of algorithm 2 to our desired value of the counterfactual states. We plot such paths in Figure 7 with the augmented $y_{t}$ values. As visualised, the U-CVAE generated forecasts have a similar shape as are in the stationary time series analysis, where the paths converge back to the longer-term stationarity (around 0) when facing a upward or downward shock — in reality, the upward shock was in effect, under which U-CVAE responded well, similar to how the baseline performed.

Figure 7. Lagged volume feature interpretation: BNP illustration (generated by U-CVAE)

4. Further Discussions

4.1. Path correlation in non-stationary time series

Recall that for a given k, a forecast path can be written as $y_{t+\cdot|t}:=(y_{t+1|t},...,y_{t+k|t})$ and likewise for $y(i)_{t+\cdot|t}$, which takes the $i$-th entry, namely $y(i)_{t+\cdot|t}:=(y(i)_{t+1|t},...,y(i)_{t+k|t})$. The correlation statistics takes the form of

$$\rho_{i,j}=\frac{Cov(y(i)_{t+\cdot|t},y(j)_{t+\cdot|t})}{\sqrt{V(y(i)_{t+\cdot|t})V(y(j)_{t+\cdot|t})}}$$

between paths $y(i)_{t+\cdot|t}$ and $y(j)_{t+\cdot|t}$. Now, to obtain this statistics, we consider the correlation of the average paths (CAP), defined over the average path $\bar{y}_{t+\cdot|t}$ generated. This can be expressed as

(8) $$\hat{\rho}^{CAP}_{i,j}=\frac{Cov(\bar{y}(i)_{t+\cdot|t},\bar{y}(j)_{t+\cdot|t})}{\sqrt{V(\bar{y}(i)_{t+\cdot|t})V(\bar{y}(j)_{t+\cdot|t})}}$$

Throughout this paper so far, we used CAP to report the correlation statistics. However, it can also be of interests to report another statistics, which concerns per-path correlation. To this end, we define the average correlation of paths (ACP), denoted $\hat{\rho}^{ACP}_{i,j}$, as below:

(9) $$\forall s,\hat{\rho}^{s}_{i,j}:=\frac{Cov(y(i)^{s}_{t+\cdot|t},y(j)^{s}_{t+\cdot|t})}{\sqrt{V(y(i)^{s}_{t+\cdot|t})V(y(j)^{s}_{t+\cdot|t})}};\ \ \hat{\rho}^{ACP}_{i,j}=\frac{\sum_{s\in S}\hat{\rho}^{s}_{i,j}}{|S|}$$

To empirically showcase the difference, we compute such statistics on the correlation and cross correlation between ASML and BNP, and plot the rolling estimations (expanding window of samples) on Figure 8 for short term rolling forecasts and under $\sigma=1$ configuration. Clearly, ACP and CAP converge to different values, with CAP having less bias approximating the true correlation and ACP having a better try on approximating the true cross correlation. A generalised set of cross correlation matrices are plotted in Figure 10 in the appendix, where ACP tends to fit well with the actual data while CAP mostly over-estimates the values.

Figure 8. Correlation estimates (left) and cross correlation (right), with red line indicating the true data estimation

It is in general not easy to derive the limiting distribution of the correlation variables when the sample size goes to infinity. Fixing $k$, under linear time series models (stationary ARMA or VAR in particular), each sample would be drawn from a Gaussian distribution, hence both covariance and variance would converge to $\chi^{2}$ in distribution. However, non-stationary time series does not have such convergence, making it hard to analytically showcase the limiting distribution.

Likewise, it may be tempting to discuss the limit when $k$ goes to infinity. Under linear time series, the forecast will be gradually concentrating to a point as $y_{t+k|t}\to\mathbb{E}[Y]\text{ as }k\to\infty$ under the stationarity assumption, which leads to the unbiasedness of correlation estimate when $k\to\infty$, should all variables be stationary. However, non-stationary variables do not benefit from this, and ACP is conjectured to be different from CAP under the non-linear setting. One may derive the limiting distribution under simple regime-switching for this case.

4.2. Other Extensions

There are many other aspects of the CVAE forecasting schemes which are not discussed extensively in this paper due to the length restriction. First and foremost, we have alternative generative schemes that worth consideration and demonstration — instead of drawing from

$$\mathbb{E}_{Z\sim N(0,I_{q}),X=x_{t}}[N(\hat{f}^{de}(X,Z),\sigma^{2}I_{d})]$$

we may wish to draw $Z$ from conditional distribution based on the previous observation, that is, from

$$\mathbb{E}_{(Z\sim Z|X=x_{t-1},Y=y_{t-1}),\ X=x_{t}}[N(\hat{f}^{de}(X,Z),\sigma^{2}I_{d})]$$

This makes it possible to interact with the encoder, as the distribution of $Z|X,Y$ is learnt by the encoder function. In fact, in some literature, this is the main generating method (Li et al., 2022).

Additionally, as was introduced in section 2.3, advanced information is a more generalised concept of forecast measures, instead of just point forecasts — hence one could look into interval forecasting, in addition to our investigation of mean forecast. This will then be comparable to traditional linear models. An attempt is made in section 3.6 when plotting the upper and lower quantiles of the simulated paths, though more thorough extension would be desired.

There are many other aspects of machine learning techniques on the neural network architectures that could be extended, including the increase of latent dimension and alternative architectures such as convolutional neural networks. These could act as better forms of approximation to the true non-linear and non-stationary nature of the data.

5. Conclusion

In this paper, we first identify the class of problem of time series forecasting with advanced information, which can be related to many problems in time series and finance, including stationarity, panel data, and stock volume forecasting. A CVAE architecture is introduced to model such time series, We further investigate the case for daily stock volume forecasting, and found the CVAE generated forecasts more competitive than traditional linear models. The CVAE forecasts may be further generated for different scenarios of input features, creating a possibility to interpret feature inputs and generate various scenarios. Various extensions are then discussed in light of potential challenges which can be encountered in non-linear time series.

Figure 9. Cross Correlation Matrix of Long Term Forecasts (Upper) and Correlation Matrix of Short Term Rolling Forecasts (Lower): Data (left), U-CVAE (centre), and ARMA(1,1) (right)

Appendix A Appendix

A.1. Technical remarks on the traninig of CVAE

A.1.1. ERM for training

The empirical risk minimisation for training the CVAE goes as follows. We aim to minimise $\mathbb{E}_{X,Y\sim D}[\log(P(Y|X))]$ based on the assumptions in Equation 1 and Equation 2. Let $P(Y|X,Z)$ denote the probability distribution as specified in Equation 1, let $P(Z)=N(0,I_{q})$ be the distribution of $Z$, let $Q(Z|X,Y)$ be the distribution of $Z$ conditional on $X,Y$ as specified by Equation 2, and let $P(Z|X,Y)$ be the conditional distribution obtained by Bayes rule, i.e. $P(Z|X,Y)=\frac{P(Y|X,Z)P(Z|X)}{P(Y|X)}$

Write $KL(\cdot||\cdot)$ as the KL divergence, then observe, by Bayes’ Rule

$$KL(Q(Z|X,Y)||P(Z|X,Y))$$

$$= \mathbb{E}_{Z\sim Q(Z|X,Y)}[\log(Q(Z|X,Y))-\log(P(Z|X))$$

$$+\log(P(Y|X))-\log(P(Y|X,Z))]$$

$$= KL(Q(Z|X,Y)||P(Z|X))$$

$$+\log(P(Y|X))-\mathbb{E}_{Z\sim Q(Z|X,Y)}[\log(P(Y|X,Z))]$$

Rearranging get

$$\log(P(Y|X))= \mathbb{E}_{Z\sim Q(Z|X,Y)}[\log(P(Y|X,Z))]$$

$$+KL(Q(Z|X,Y)||P(Z|X,Y))$$

$$-KL(Q(Z|X,Y)||P(Z|X))$$

$$\geq \mathbb{E}_{Z\sim Q(Z|X,Y)}[\log(P(Y|X,Z))]$$

$$-KL(Q(Z|X,Y)||P(Z|X))$$

The last line is also known as Variational Lower Bound. Instead of interacting with $\log(P(Y|X))$, we interact with the Variational Lower Bound. Same as the literature, we assume $P(Z|X)=P(Z)$ for latent variable to be independent of the input (Sohn et al., 2015). This enables the KL divergence term to be written in explicit form, which is

$$KL(Q(Z|X,Y)||P(Z))$$

$$= \frac{||\mu(X,Y)||^{2}_{2}+tr(\Sigma(X,Y))-\log(\det(\Sigma(X,Y)))}{2}$$

Now, given dataset $D=\{(x_{i},y_{i})\}_{i\in I}$, we minimise the empirical version of the bound, namely

$$\mathbb{E}_{X,Y\sim D}[\mathbb{E}_{Z\sim Q(Z|X,Y)}[\log(P(Y|X,Z))]-KL(Q(Z|X,Y)||P(Z))]$$

The first term can be further approximated with simulated samples $z^{i}_{1},...,z^{i}_{S}\sim Q(Z|x_{i},y_{i})\forall i$

$$\sum_{i\in I}\sum_{s\in S}\frac{\log(P(y_{i}|x_{i},z^{i}_{s}))}{|I||S|}$$

And by Equation 1, the probability term can be further derived into

$$\log(P(y_{i}|x_{i},z^{i}_{s}))=c+\frac{||y_{i}-f(x_{i},z_{i}^{s})||^{2}_{2}}{\sigma^{2}}$$

where $c$ is a constant term independent of $x_{i},y_{i},z^{i}_{s}$

Figure 10. Cross correlation from data (left), ACP estimate of cross correlation (centre), and CAP estimate of cross correlation (right)

A.1.2. Architecture and optimisation techniques

As for the architecture of the neural networks (formally speaking the space $F_{1},F_{2}$), we use two layers of RELU network for the encoder function, with dimensions linking the input space ($\mathbb{R}^{p}\times\mathbb{R}^{d}$) to $\mathbb{R}^{q}\times\mathbb{R}^{q}$ for the first layer, an untrained identity map from $\mathbb{R}^{q}\to\mathbb{R}^{q}$ to map to the expectation term and an untrained softplus from $\mathbb{R}^{q}\to(0,\infty)^{q}$ to map the variance.

We use two layers of RELU and one layer of linear network for the decoder function. The dimensionality depends on the dimensionality of the input — when $d=50$ (the MCVAE model), the dimensions for the two layers of RELU are input dimension to 64 ($\mathbb{R}^{59}\times\mathbb{R}^{64}$), followed by $\mathbb{R}^{64}\times\mathbb{R}^{64}$ and then linear layer $\mathbb{R}^{64}\times\mathbb{R}^{50}$ for output. When $d=1$ (the UCVAE model), the layers are $\mathbb{R}^{27}\times\mathbb{R}^{16}$ and $\mathbb{R}^{16}\times\mathbb{R}^{8}$ for RELU, then a linear layer of $\mathbb{R}^{8}\times\mathbb{R}$ for output.

The optimisation procedure can be summarised as a combination of ADAM (default setting in tensorflow version 2.16.1) and validation early stop**. The training is hated when the validation loss exceeds 1% of the local minimum within the last 3 steps of the training.

A.1.3. Technical note regarding $\sigma$ calibration

There are various ways to estimate or calibrate $\sigma$. As the focus of this paper is more on the point forecast $y_{t+k|t}$ instead of interval forecasting, we aim to explore the centre of sampling as opposed to the tails, hence calibration is used for the value of $\sigma$. In section 3.5, we used $\sigma=0.1$ for efficient sampling to produce overall results, and in section 3.6 and 4, we used $\sigma=1$ to have wider range of samples and to further the investigation to correlation estimates.