Few-Shot Learning Patterns in Financial Time-Series for Trend-Following Strategies
Abstract
Forecasting models for systematic trading strategies do not adapt quickly when financial market conditions rapidly change, as was seen in the advent of the COVID-19 pandemic in , causing many forecasting models to take loss-making positions. To deal with such situations, we propose a novel time-series trend-following forecaster that can quickly adapt to new market conditions, referred to as regimes. We leverage recent developments from the deep learning community and use few-shot learning. We propose the Cross Attentive Time-Series Trend Network – X-Trend – which takes positions attending over a context set of financial time-series regimes. X-Trend transfers trends from similar patterns in the context set to make forecasts, then subsequently takes positions for a new distinct target regime. By quickly adapting to new financial regimes, X-Trend increases Sharpe ratio by over a neural forecaster and 10-fold over a conventional Time-series Momentum strategy during the turbulent market period from to . Our strategy recovers twice as quickly from the COVID-19 drawdown compared to the neural-forecaster. X-Trend can also take zero-shot positions on novel unseen financial assets obtaining a 5-fold Sharpe ratio increase versus a neural time-series trend forecaster over the same period. Furthermore, the cross-attention mechanism allows us to interpret the relationship between forecasts and patterns in the context set.
Keywords Trend-Following Time-Series Momentum Few-Shot Learning Deep Learning Machine Learning Transfer Learning Change-point Detection Quantitative Finance Portfolio Construction
1 Introduction
When financial market conditions change the forecasting models that are used to take positions in the markets perform very poorly [1, 2]. The recent success of deep learning for learning representations from data has translated into better financial forecasting models [3]. However deep learning models also rely on large stationary datasets for representation learning. Financial markets can be highly non-stationary due to changing market conditions. When a financial market enters a new regime, augmenting the inputs with indicators of the time and severity of regime change improves returns [4]. This finding is important since it shows that there is a benefit to training deep learning models that have additional supervision about when regimes change. Furthermore, it raises an important question: can we selectively use historical patterns to transfer knowledge of the past to make forecasts for new regimes and new markets?
In recent years it has been observed, that the risk-adjusted returns of conventional Time-series Momentum (TSMOM) models [5], which exploit trends in financial time-series, have deteriorated by from to compared to the period from to . This can potentially be attributed to a concept known as ‘factor crowding’ [6], where arbitrageurs trade the same assets based on similar factors. This causes market inefficiencies to quickly disappear and can increase the risk of liquidity-driven tail events [7]. To mitigate this, one alternative has been to explore new assets where we typically do not have sufficient data for deep learning approaches.
In deep learning, quick adaptation to new data or few-shot learning has seen recent advances in computer vision [8, 9, 10]. Few-shot learning involves training neural networks (NNs) such that they are able to adapt and learn from minimal data. Few-shot learners are tested on completely unseen classes of images using a few to no examples or are required to solve unseen reinforcement learning environments [11, 12, 13]. Few-shot learners have the desirable quality of being able to adapt and learn from very few data points. This is advantageous for a systematic trading strategy, allowing it to adapt quickly to new financial regimes or new markets. Broadly, we can categorize the market regimes, which systematic strategies aim to exploit, as trending or mean-reverting. Mean-reversion is a market phenomenon whereby, under certain circumstances, the price has a tendency to return its long-term mean [14]. A detailed study of the trending and mean-reverting financial market anomalies can be found in [15].
In this work, we leverage advances in few-shot learning and time-series momentum strategies to develop a model that can make predictions in new market regimes and unseen markets. In practice, our experiments backtest on continuous futures contracts of various asset classes: equities, foreign exchange, commodities and fixed income. Our model obtains significantly higher risk-adjusted returns in terms of Sharpe ratio [16], a measure of returns per unit volatility. Our model is able to learn transferable patterns, then subsequently learn to take positions in markets or regimes distinct from those used for training. The idea of universal patterns in financial markets, which are transferable, is motivated by the works [17, 18, 19].
Our model uses a cross-attention mechanism over a context-set [20, 21]. Attention mechanisms [22] have been shown to improve returns by attending over an asset’s history [19], this work generalizes this finding by extending the temporal attention mechanism over other assets. This enables our model to transfer knowledge from the context set to enable better predictions for a new regime or new market with little data. Different regimes from financial assets are segmented using change-point detection methods [23, 24]. Additionally, the attention maps provide a degree of interpretability in the resulting predictions [25, 19]. We also use the latest insights from deep time-series momentum strategies to train our model to produce positive returns over our baselines [3]. We call our method the Cross Attentive Time-Series Trend Network or X-Trend for short. The code is available at https://github.com/kieranjwood/x-trend.
We summarize our contributions as follows:
-
•
We leverage few-shot learning, and change-point detection to develop an agent which is able to produce returns in futures markets with minimal data. Our X-Trend model is able to successfully respond to “momentum crashes” [1], and “momentum turning points” [2]. We improve Sharpe ratio by in comparison to the benchmark neural forecaster over to , an extremely turbulent period in various financial markets. Our X-Trend strategy recovers from the initial COVID-19 drawdown twice as quickly.
-
•
X-Trend learns to make predictions in challenging, low-resource, zero-shot settings where the model has never seen a financial asset during training. It improves over the loss-making neural forecaster to achieve an average Sharpe of 0.47 over the period to . This turbulent period is particularly challenging for unseen financial assets.
-
•
X-Trend makes interpretable predictions. It is able to learn relationships between similar assets using an interpretable cross-attention mechanism over a context set of different assets. For a given target sequence, a similarity score with patterns in the context set via cross-attention can be visualized. Additionally, by outputting a forecast as an auxiliary output step, we reveal the relationship between optimal trading signal and forecast.
2 Preliminaries
Let us denote a time-series of daily close prices, where denotes a particular asset from a basket of assets and denotes a time index where is the final observation for the asset. We work with returns , which linearly de-trends the price series:
(1) |
where is the number of days we calculate returns over and for brevity we use to denote .
Our objective is to trade a position which we hold for the next day conditioned on a target sequence of the last days, where is a vector of factors measuring trends at different timescales or the relationship between trends at different timescales. These factors are constructed for time using returns data and price data, which we elaborate on in Section 2.2 and Section 2.3. We aim to choose positions such that we maximize portfolio returns:
(2) |
for each of the assets where assets and is the transaction cost. We use volatility targeting [5, 26, 27], which introduces a leverage factor , where we normalize our holdings by the ex-ante volatility, , then scale by the annual target volatility . Aligning our work with the literature [5, 26], is calculated using an exponentially weighted moving standard deviation of returns over the span , where the contribution has decreased to zero by 60 days in the past. For this paper we set rather than assuming a cost for each asset, focusing on pure predictive power of our model. The volatility targeting approach to portfolio construction ensures that each asset contributes approximately equal risk to the portfolio111We assume a diagonal covariance, which is a reasonable assumption outside of the tail for the basket of futures contracts we consider in this paper, which is a typical basket for trend-following strategies.. As such, we perform regression to estimate the probability distribution of volatility scaled next-day returns, .
2.1 Episodic Learning
We use episodic learning [8] which trains models in the same way as they are used for testing; models are trained to produce few-shot and zero-shot predictions. Traditional deep learning puts data from all assets together and is trained using mini-batch stochastic gradient descent [28, 29]. In episodic learning, we want to make forecasts given a sequence’s history for a specific asset, and leverage sequences from other assets for additional non-parametric similarity learning.
Our learner selects a position given target: , which we shorten to for brevity, and where belongs to our test set of assets. Additionally, the learner leverages a context set of prices where belongs to our training set of assets , is the length of each time-series in the context set and is the total number of contexts in the context set. In all circumstances, the context sets are time-series that have occurred before the time-series in the target: for all in the context set so that our predictions are causal. Throughout the paper we will work with two distinct problem scenarios:
-
•
Few-shot: where the context set can contain the same asset as the target (but in the past) .
-
•
Zero-shot: where the target set asset (at test time) is not contained in the context set .
We sample a target time-series, and we sample an associated context set per target point in the mini-batch. As opposed to our target problem, we also include the volatility scaled next-day returns as inputs in the context set, denoting the combined inputs as . We describe how we construct our context sets for the financial assets in Section 3.2.
2.2 Classical Momentum Approaches
Time-series Momentum (TSMOM) strategies [5], or trend-following strategies, are based on the idea that strong price trends have a tendency to persist. The phenomena of trend-following has been observed historically for more than a century222It should be noted that returns have started to suffer since the introduction of electronic trading [4] this century., outperforming a simple buy-and-hold Long strategy [30]. TSMOM strategies aim to forecast trends and then map them to a trading signal, or position. For instance, we can calculate returns over the past year ( trading days), and map this to a position :
(3) |
where is the sign function, with corresponding to a full long position and a full short position [5]. The Long strategy takes a full long position at each time-step: .
We employ volatility scaling for portfolio construction Eq. 2. This approach to portfolio construction is often used in practice by Commodity Trading Advisors (CTAs), where trend-following is a major component of their overall strategy. While this TSMOM approach has proven to be successful over time [30], the -year momentum indicator particularly suffers during periods where market conditions change rapidly. We call these regime changes throughout this paper (also referred to as momentum crashes [1]). An attempt to mitigate regime changes is to combine weighted signals at different timescales, for example, 1-month (21-day), 3-month (63-day), half-year (126-day) and 12-month momentum, , where represents the weighting of each respective factor.
MACD (Moving Average Convergence Divergence) factors compare exponentially weighted signals at two different timescales [31]. These popular momentum indicators aim to balance the trade-off between capturing trends and responding to potential regime changes:
(4a) | ||||
(4b) |
where function is an exponentially weighted moving average. The inputs are a short timescale , with half-life of and a long timescale , defined similarly. The MACD signal indicates buy if and sell if . The magnitude provides a measure of conviction or signal strength. It is common to blend multiple MACD indicators at different timescales with a typical choice being [31]. Funds typically convert the MACD signal to a position via response function: [31].
2.3 Deep-learning Momentum Approaches
In financial markets, we often observe different trends and mean-reversions, which we can also think of as a collection of shorter-term trends, occurring concurrently at multiple timescales. Furthermore, the added complexity of regime change means that it is a daunting task to successfully blend various trading signals. This motivated Deep Momentum Networks (DMNs), as a solution to such a complex forecasting problem [3, 19, 4, 32], which have been shown to outperform TSMOM in terms of risk-adjusted returns [3].
We opt to use factors that are commonly used in trend-following strategies [31, 30, 3]. Concretely, we use returns aggregated and normalized over different time scales:
(5) |
and include MACD indicators:
(6) |
The DMN framework simultaneously learns asset price trends and position sizes. The position sizes are estimated using a neural network:
(7) |
where is a neural network followed by a linear map**, , and activation function such that traded position . The neural network hidden state dimension is denoted .
The primary innovation of DMNs, as introduced in [3], was to output traded positions and directly optimize the Sharpe ratio: a risk-adjusted return metric measuring returns per unit volatility. Typically most fund managers or CTAs will have a predefined risk tolerance and aim to maximize returns given this constraint333The loss function can easily be tailored to instead use drawdown, VaR (Value at Risk), or even some combination of these, instead of volatility as the risk measure.. With the aim to optimize neural network parameters , the Sharpe loss function is defined as:
(8) |
where is a batch of pairs . In practice, when training the model, we select sequences such that , with warm-up period . That is, our loss function ignores the first predictions for each sampled sequence.
The Long Short-Term Memory cell (LSTM) [33] is a popular Recurrent Neural Network (RNN) tailored to modeling sequences and is the primary DMN component [3, 4]. In addition to the output for each time-step , the LSTM maintains , a cell state, which stores longer-term information:
(9) |
The LSTM modulates information through a series of gates, clearing when necessary, such as during regime change, and maintains as a localized summary of the sequence. The initialization can be specific per contract, and we provide details of our implementation Section 3.1.
2.4 Attention
The attention mechanism computes a weighted average of elements where the weights depend on an input query and the elements’ keys [22]. It dynamically decides on which input elements to focus on or “attend” to. Specifically, it computes a similarity between the query vector and key vectors. These vectors can either be model inputs, some deep-learning hidden state, or, in the case of our work, a hidden state summarising a sequence. If we have a query vector and we want to compute the relative importance of each key in , we calculate soft weights with a softmax function:
(10) |
with learnable weight matrices and attention dimension . The inner-product is the mechanism used to compute similarity. The primary benefit of attention is that it provides a direct connection with all of the keys and, furthermore, these weights are interpretable. We then use each weight to scale the values , where each value has a corresponding key, and aggregate as:
(11) |
We elaborate on the specific details of our implementation in Section 3.3. We can generalize equation 11 to multiple parallel attention heads to make the model bigger, allowing the model to capture representations for different patterns and time-scales. The cross-attention mechanism in X-Trend uses heads.
3 Cross-Attentive Time-Series Momentum Forecaster
3.1 Sequence Representations and Baseline Neural Forecaster
We want to create sequence summaries, of sequence length , with a learnable function , where is our hidden dimension. Our representation is created to summarise not just input sequences , but also side information , namely the category (ticker) of the futures contracts. We include this category because we know the dynamics of different contracts can vary significantly; for example crude oil compared to the 5-year treasury bond. For each time-step, and each asset, we input features , as defined in Eq. 6.
We encode side information , with an entity embedding: a learnable map** of each category into , which can automatically learn to group similar assets in the embedding space [34]. We use a feedforward (FFN) network to fuse the entity embeddings and time-series representations. Throughout our paper we use (Exponential Linear Unit) [35] activations which are continuously differentiable everywhere and avoid dead neurons (which are completely deactivated). We indicate the ability to optionally include static information via embeddings in blue:
(12a) |
where is a learnable linear transformation into and is a vector which can either be a hidden-state or a vector. Additionally, we use the Variable Selection Network (VSN) [3], which weights out different lagged normalized returns and MACD features.
We associate a learnable nonlinear function with the -th element of , and scale by the associated learnable weight , of :
(13a) | ||||
(13b) |
where the -th element of the softmax is defined as: .
Our model consists both of an encoder and a decoder. It is important to note that in the decoder we also include the output of the encoder, , as an input, where (Section 3.4). Our encoder sequence representations, , can be summarised as:
(14a) | ||||
(14b) | ||||
(14c) | ||||
(14d) |
The LSTM initial state is learnable and specific to each contract, setting . The skip connections, in equation 19c and equation 19d, allow for the respective components to be suppressed, enabling the model to be as complex as necessary. See Fig. 4 for an overview of our model.
The Baseline neural forecaster which we compare to uses as a model i.e. from equation 7. The baseline architecture consists of only the decoder. The X-trend model adds an encoder and the cross-attentive steps.
3.2 Context Set Construction
When backtesting, we randomly sample a context set of size sequences from the past444During training we do not enforce causality when we construct our context set, as the aim is to teach the model to best transfer patterns; however, the training set only contains information prior to the test date to ensure our model is causal at test time. to enforce causality at the time of prediction for our target problem. We explore three different approaches to constructing our context sequences and choosing the point at which we condition on, illustrating these in Fig. 2:
-
1.
Final hidden state and random sequences of fixed length. We sample random sequences across time and assets of fixed length and we condition on the final hidden states of the sequences, which summarises the sequence.
-
2.
Time-equivalent hidden state. We sample random context sequences that are the same length as the target sequence: . For each time-step in the target sequence, we condition on the time-equivalent hidden states i.e. the -th target step attends to the -th context steps. This allows the model to incorporate additional adjacent regimes by conditioning on different representations at each time-step of the target sequence.
-
3.
Change-point detection (CPD) segmented representations. We use a Gaussian Process change-point detection algorithm, detailed in Appendix A, to segment the context set into regimes. An example of this segmentation for the British Pound can be seen in Fig. 3. We randomly sample change-point segmented sequences and condition on the final hidden states of these change-point time-series segments. We limit to a maximum sequence size. We test both day ( month) and day ( month) maximum length sizes, using a higher CPD severity threshold for the day version, calibrating a change-point threshold in both cases such that the average sequence length is approximately half of the maximum length.
3.3 Cross Attention
In order for our predictions to leverage a context set we use a cross-attention mechanism between our target sequence and a context set of sequences. This allows our target to attend to different sequences from different assets across time and across different regimes. The rationale is that this context set contains a much broader range of patterns in comparison to the near-term history of the same asset, which was successfully leveraged with attention by [19] (Section 6).
We create the set of key vectors, , and set of value vectors, :
(15) |
where denotes context sequence without the target returns. For a given query, we calculate attention weights for each key (see Eq. 10). Then adapting Eq. 11, we weight the value vectors by the attention weights and expand to multiple concurrent heads to increase the representational space. We leverage the context set via the following steps:
(query representation) | (16) | ||||
(self-attention) | (17) | ||||
(cross-attention) | (18) |
We illustrate this in Fig. 4. The self-attention step, which outputs the updated set of values , helps to identify similarities between regimes within the context set.
3.4 Decoder and Loss Function
Similarly to our sequence representations in the encoder, Eq. 14, we summarize our target sequence in the decoder. This time our sequence representation, , fuses the output of the encoder, . Highlighting the additional components in magenta, our decoder representation is computed as:
(19a) | ||||
(19b) | ||||
(19c) | ||||
(19d) |
We want to model the next-day volatility scaled return. This is a regression task where we parameterize the predictive mean via and volatility via ; the likelihood is . Thus our loss function is to minimize the log-likelihood under a Gaussian distribution of future returns:
(20) |
It is important to note that the Gaussianity of returns in financial time-series is a convenient approximation, which in practice does not always hold, especially in the tail. However, since we are focused on optimizing the Sharpe ratio in this paper555Due to the non-Gaussianity of returns observed in practice, the Sharpe ratio often may not be the best metric for measuring risk-adjusted returns., predictive mean and volatility estimates of next-day return outputs are very useful since Sharpe is dependent on realized returns and volatility.
Rather than directly optimizing Sharpe with a DMN, we propose to jointly optimize the likelihood for next-day predictions and the Sharpe. The Sharpe loss requires an additional neural network head, Predictive distribution (mean and standard deviation) To Position: head . This is a single FFN followed by a activation. Our joint loss function is:
(21) |
where is Eq. 8 applied to the PTP outputs and is a tunable hyperparameter to balance the two loss functions.
As an alternative to assuming Gaussianity of returns, we can instead perform Quantile Regression (QRE). The aim of QRE is to learn the full probability distribution of next-day returns. QRE has proven successful for multi-step time-series regression [36]. For quantiles , pairs and target , our QRE loss function is:
(22) |
where and is a neural network head which we replace and with. Our set of quantiles, , includes quantiles in the left and right tail – the market movements which typically have the largest impact on our strategy risk-adjusted returns. We define the joint QRE loss function, as:
(23) |
with the PTP for the Sharpe loss again a FFN, .
Utilizing the outputs of our encoder, , our decoder outputs a trading signal as follows:
(24) |
We refer to the Gaussian MLE (Maximum Likelihood Estimation) variant of the architecture as X-Trend-G, the QRE variant as X-Trend-Q and the Sharpe loss variant as X-Trend.
It is important to note that in the zero-shot setting we exclude the ticker-type embedding of for the target sequence. This is because we are trading a previously unseen asset and we have not yet seen any contract-specific dynamics. We do however still include this information in the context set, where the aim is to quickly identify similarities with previously seen contracts.
4 Experiments
The cross-attention mechanism is key to obtaining low error forecasts for few-shot learning for a toy dataset of Gaussian Process (GP) draws [37]. We implemented the recurrent attentive neural process [38] for this toy dataset, and we ablate certain components and we show that the cross-attention mechanism is key to obtaining low-error few-shot forecasts. This result motivates using a cross-attention for momentum. See Appendix B for further details.
We backtest our X-Trend variants on a portfolio of of the most liquid, continuous futures contracts666The futures contracts are chained together using the backwards ratio-adjusted method., , over the period from 1990 to 2023, extracted from the Pinnacle Data Corp CLC Database777https://pinnacledata2.com/clc.html. The futures contracts we have selected are amongst the most liquid and typical for backtesting TSMOM strategies [5, 30, 3, 19]. To test-out-of-sample across the entire history, we use an expanding window approach, where we initially train on to , test out-of-sample on the period from to , expand the training window to from to , then test out-of-sample on the subsequent years and so on. We take particular note of performance over the to period, which covers the COVID-19 crisis, exhibiting dynamics that are significantly different to the training set.
For our zero-shot experiments, we randomly selected of the Pinnacle dataset assets as the test set, , leaving the other , for the context set and training. Despite the fact these futures contracts had historical data available in reality, we constructed this experiment as an artificial zero-shot setting to provide insight into transferability to unseen assets. Furthermore, this low-resource setting is particularly challenging because is also small at only contracts. The dataset is described in further detail in Appendix D. The details of training the neural networks are provided in Appendix C.
5 Results
Context Time-step | Average Annual Sharpe Ratio | |||||||
Method | Loss | Final/Time/CPD | – | – | – | |||
Reference | Long | 0.48 | 0.40 | 0.60 | ||||
TSMOM | 0.23 | 0.71 | 1.01 | |||||
MACD | 0.27 | 0.45 | 0.71 | |||||
Baseline | Sharpe | 2.27 | 1.93 | 2.91 | ||||
J-Gauss | 2.43 (+7.2%) | 2.06 (+7.0%) | 3.04 (+4.5%) | |||||
J-QRE | 2.26 (-0.5%) | 1.96 (+1.6%) | 2.89 (-0.9%) | |||||
X-Trend | Sharpe | T | 10 | 126 | 2.28 (+0.2%) | 1.97 (+2.4%) | 2.93 (+0.5%) | |
Sharpe | F | 10 | 21 | 2.35 (+3.4%) | 1.99 (+3.2%) | 3.01 (+3.4%) | ||
Sharpe | F | 20 | 21 | 2.38 (+4.9%) | 1.99 (+3.4%) | 3.02 (+3.6%) | ||
Sharpe | F | 30 | 21 | 2.25 (-0.9%) | 1.99 (+3.1%) | 3.03 (+4.0%) | ||
Sharpe | F | 10 | 63 | 2.31 (+1.7%) | 1.97 (+2.3%) | 3.11 (+6.9%) | ||
Sharpe | C | 10 | 21 | 2.30 (+1.1%) | 2.02 (+4.7%) | 3.04 (+4.5%) | ||
Sharpe | C | 20 | 21 | 2.65 (+16.9%) | 2.17 (+12.5%) | 3.17 (+8.8%) | ||
Sharpe | C | 30 | 21 | 2.38 (+5.0%) | 2.02 (+4.6%) | 3.08 (+5.8%) | ||
Sharpe | C | 10 | 63** | 2.50 (+10.1%) | 2.14 (+11.2%) | 3.11 (+6.8%) | ||
X-Trend-G | J-Gauss | T | 10 | 126 | 2.47 (+8.8%) | 2.14 (+11.1%) | 3.16 (+8.5%) | |
J-Gauss | F | 10 | 21 | 2.25 (-1.0%) | 1.90 (-1.3%) | 3.10 (+6.5%) | ||
J-Gauss | F | 20 | 21 | 2.52 (+11.0%) | 2.10 (+8.8%) | 3.05 (+4.8%) | ||
J-Gauss | F | 30 | 21 | 2.26 (-0.5%) | 2.08 (+7.9%) | 3.04 (+4.2%) | ||
J-Gauss | F | 10 | 63 | 2.42 (+6.5%) | 2.07 (+7.4%) | 3.17 (+8.8%) | ||
J-Gauss | C | 10 | 21 | 2.42 (+6.8%) | 2.09 (+8.1%) | 3.06 (+5.1%) | ||
J-Gauss | C | 20 | 21 | 2.42 (+6.5%) | 2.03 (+5.2%) | 3.11 (+6.8%) | ||
J-Gauss | C | 30 | 21 | 2.51 (+10.4%) | 2.12 (+9.8%) | 3.07 (+5.4%) | ||
J-Gauss | C | 10 | 63** | 2.32 (+2.1%) | 2.00 (+3.6%) | 3.18 (+9.2%) | ||
X-Trend-Q | J-QRE | T | 10 | 126 | 2.53 (+11.6%) | 2.12 (+10.1%) | 3.08 (+5.9%) | |
J-QRE | F | 10 | 21 | 2.38 (+4.9%) | 1.98 (+2.5%) | 3.07 (+5.3%) | ||
J-QRE | F | 20 | 21 | 2.21 (-2.8%) | 1.86 (-3.4%) | 2.94 (+0.8%) | ||
J-QRE | F | 30 | 21 | 2.42 (+6.6%) | 2.05 (+6.5%) | 3.03 (+4.2%) | ||
J-QRE | F | 10 | 63 | 2.49 (+9.6%) | 2.04 (+5.8%) | 3.06 (+5.1%) | ||
J-QRE | C | 10 | 21 | 2.26 (-0.5%) | 2.03 (+5.3%) | 3.07 (+5.3%) | ||
J-QRE | C | 20 | 21 | 2.53 (+11.6%) | 2.08 (+7.7%) | 3.07 (+5.5%) | ||
J-QRE | C | 30 | 21 | 2.4 (+5.6%) | 2.02 (+4.5%) | 3.01 (+3.2%) | ||
J-QRE | C | 10 | 63** | 2.70 (+18.9%) | 2.14 (+10.9%) | 3.11 (+6.8%) |
Context Time-step | Average Annual Sharpe Ratio | |||||||
Method | Loss | Final/Time/CPD | – | – | – | |||
Reference | Long | 0.28 | 0.02 | 0.28 | ||||
TSMOM | -0.26 | 0.05 | 0.61 | |||||
MACD | -0.14 | 0.11 | 0.32 | |||||
Baseline | Sharpe | -0.11 | 0.02 | 1.00 | ||||
J-Gauss | ||||||||
J-QRE | ||||||||
X-Trend | Sharpe | C | 20 | 21 | ||||
J-Gauss | C | 20 | 21 | |||||
J-QRE | C | 10 | 63** |
5.1 Few-shot Setting
We plot the few-shot strategy returns over the past years ( to ) in Fig. 5, which is where we start to see larger gains when backtesting our X-Trend strategy, in comparison to the baseline. In particular, we draw attention to the results over the past five years ( to ), which is an extremely interesting period for few-shot learning because it exhibits significant market turbulence, previously unseen market dynamics, and numerous regime shifts. It includes the Bull market of /, followed by the COVID-19 pandemic in / and then the beginning of the Russia-Ukraine war in . During this -year period X-Trend improves upon the Sharpe of the baseline strategy by % and X-Trend-Q improves upon the baseline by %. It is evident that the cross-attention step is the primary driver of the improvement in risk-adjusted returns. We observe that X-Trend-Q and X-Trend outperform X-Trend-G (Table 1), which suggests that the model is able to learn and benefit from a more complex returns distribution, compared to assuming Gaussian returns. We argue that this is likely because we explicitly force the model to pay attention to large movements by performing QRE on quantiles in the left (and right) tail of the returns distribution.
The few-shot results in Table 1 show the impact of changing the context set size, context sequence (maximum) length, and different methodologies for selecting the context hidden state to attend to. Notably, for X-Trend, we demonstrate that by segmenting the context set with CPD, we improve Sharpe by a further %, showing the benefit of constructing context sets with regimes. In effect, we are constructing a context set with only the most informative observations.
In Fig. 5 we plot the strategy drawdown of the X-Trend-Q strategy compared to our baseline learner over the period to , focusing on the COVID-19 drawdown – the most significant “momentum crash” or drawdown in the TSMOM strategies over the entire years of historical data. Noting that the drawdown begins on Jan 2020 for both strategies, we observe a maximum drawdown (MDD) of % for X-Trend-Q compared to an MDD of % for the baseline. Furthermore, the drawdown ends after trading days for X-Trend-Q compared to days for the Baseline which is an entire year and almost twice as long as X-Trend-Q. This quick recovery demonstrates the ability of our agent to adapt to new regimes.
5.2 Zero-shot Setting
For our zero-shot experiments in Table 2. The strategy improvement over the baseline is more pronounced than in the few-shot setting. We plot the strategy returns over the entire backtest ( to ) in Fig. 6, to demonstrate viability across time. We note a degradation of performance to and, again, we plot drawdowns over the period to , to draw attention to the COVID-19 “momentum crash”. Despite the fact that the strategy degrades, trading unseen assets during this period is an extremely challenging setting and not only is the strategy still profitable, it outperforms all benchmarks.
Unlike the few-shot setting, it is evident that the joint loss function is the key driver of the improvement of results in the zero-shot setting. Furthermore, in the zero-shot setting, X-Trend-G outperforms X-Trend-Q, indicating that the simpler assumption of Gaussian returns is favourable in a low-resource setting.
5.3 Dissecting the X-Trend Architecture
In Fig. 7 and Fig. 8 we explore the relationship between our predictive returns output distribution and the trading signal. In both examples, we observe that the predictive mean and median only make small movements above and below zero. This indicates that the model has learnt that we are operating in a low signal-to-noise environment, aiming to capitalize on slight market inefficiencies. Despite observing that there are small spikes and dips in predictive volatility, the estimate typically remains close to , which suggests that our volatility targeting step, which targets in this example, is functioning as expected.
For X-Trend-G (Fig. 7), we observe that there is an almost linear relationship between the predictive mean and trading signal, indicating that the strategy uses the predictive mean as a measure of conviction and sizes the position accordingly. The relationship between predictive volatility and position is less clear. This suggests that it is the predictive mean driving the strategy and the predictive volatility is being used for volatility scaling. We can observe that the volatility scaling pre-processing job of altering leverage based on the -day ex-ante volatility is doing a good job because we can observe that the % confidence interval is fairly stable around when targeting a daily volatility of .
For X-Trend-Q (Fig. 8), we observe that the trading signal deviates more from the predictive median than X-Trend-G, indicating that it incorporates the full predictive distribution into the trading signal. This is likely the reason that the Sharpe for X-Trend-Q outperforms X-Trend-G by % in the few-shot setting over the turbulent period of to . This supports our hypothesis that, for the best results, Gaussianity of returns should not be assumed in the few-shot setting. The assumption of Gaussian returns may be more appropriate in the low-resource setting.
We provide an illustrative example of how we can interpret the cross-attention weights in Fig. 9 for the natural gas futures contract in ; a period exhibiting significant trends due to the Russia-Ukraine conflict. We examine points in time, which correspond to different regimes, and in all cases, the top attention weights are highly intuitive. The target point at the beginning of an uptrend correctly identifies another commodity uptrend with the highest weight, with the other top being a commodity mean-reversion and a large equity uptrend. The target point at the beginning of the large downtrend clearly identifies another commodity sequence with a large downtrend, with almost double the weighting of the next highest. The target point during the beginning of a reversal identifies a slight downtrend with reversion, an extremely short downtrend, and an uptrend with significant reversion for the top three weights. Interestingly, in this case, the model actually identifies similarities with equities sequences instead of commodities.
6 Related Works
Transfer Learning. We can transfer features learned from one dataset to enable learning from new datasets more quickly. Transferring NN weights can also enable learning from new smaller datasets. This can be done by fine-tuning the weights of a NN on a new task with SGD or linear-probing which freezes lower layers of the network and only updates the top layers [39].
Few-shot learning. Few-shot learning is characterized by enabling NNs to be able to learn to make predictions using little data. One key idea for enabling this is to train the few-shot learning agent in the same way that it is used for testing. This is the idea of episodic learning where if we wish to train a model to make predictions using only examples or shots then we need to train the model by performing -shot learnings [8, 13]. Non-parametric methods are a natural fit for few-shot learning [40]. Learning distances between a few context points and a new target point has been fruitful for few-shot image classification [10, 41, 42]. Neural processes learn to sample functions like Gaussian Process [43, 37, 20]. Automatic context construction for few-shot learning using change-point detection methods has also been explored for image datasets [44]. Recently there have been works that challenge some of the common assumptions for few-shot image classification [45, 46, 47].
Few-shot learning for time-series. Neural Processes [37] which parameterize a distribution over functions have been employed for few-shot learning for time-series forecasting [48]. Neural processes algorithmically use a latent variable to enable drawing a new function for each point by sampling; the latent variable can employ autoregressive transition dynamics for time-series [49]. One can condition the target time-series predictions on a context of time-series with a cross-attention mechanism [38]. The cross-attention mechanism has been shown to be very effective for Neural Processes [20]. Gradient-based few-shot learning has also been successfully employed for time-series [50]. From the literature and from our own initial experiments on toy data (Appendix B) we found the cross-attention very effective for few-shot learning for time-series.
Deep-learning for financial time-series. Building on the work of the vanilla DMNs, [3], the Momentum Transformer [19], which is a variant of the Temporal Fusion Transformer [25], incorporates an attention mechanism to attend to prior LSTM hidden-states from the same sequence. The attention pattern naturally segments the time-series into regimes, where significant importance is placed on the final hidden state of each regime, motivating the cross-attention step in this paper. Other works demonstrate that causal convolutional filters can be be utilized to automatically generate features [51, 32], as an alternative to the momentum factor approach used in this paper. The work of [18] utilizes convolutions followed by an LSTM to generate features from limit order book (LOB) data. There is growing evidence to suggest that we can benefit from a cross-section of assets to assist our forecasting, where the work by [32] implements convolutional filters across assets and [52] uses a graph learning model to reveal momentum spillover. Meta-learning has been used in finance to construct a partial index portfolio to track a benchmark index where the asset allocation is meta-learned [53].
7 Conclusions and Future Work
We introduce X-Trend: the Cross Attentive Time-Series Trend Network. It leverages few-shot learning and change-point detection to enable adaptation to new financial regimes. We show that it is able to recover from the COVID-19 draw-down almost twice as quickly as an equivalent neural time-series agent. Over the -year period to , we are able to improve risk-adjusted returns by compared to the baseline agent and around 10-fold compared to a conventional Time-series Momentum (TSMOM) strategy. This boost in performance is largely driven by our cross-attention step which transfers trends, from similar patterns in a context set. Furthermore, our model can generate profitable zero-shot trading signals in an extremely challenging low-resource setting where we trade a previously unseen asset we achieve a Sharpe of compared to loss-making time-series momentum baselines, both deep-learning based and conventional TSMOM. In this work we withheld assets from a standard dataset to test zero-shot performance; a future avenue of work could be applying this framework to an emerging asset class such as cryptocurrencies.
We illustrate the importance of constructing a good context set, where we improve Sharpe by after segmenting the sequences with change-point detection. A future avenue of work would be to further investigate context set construction. One possibility could be considering a cross-sectional approach where we attend across a universe of assets at the same time, motivated by the works [32, 56] with the option of including lead-lag [57]. We could consider generating synthetic data for the context set [58]. Another direction of work, inspired by the Neural Process literature, is to reconcile optimizing a Sharpe ratio with optimizing the evidence lower bound for variational inference required for latent variable Neural Process time-series models [38, 48]. Finally, this work could be combined with other innovations that expand upon the standard Deep Momentum Network framework such as bringing transaction costs into the loss function [3] and including change-point features in the decoder [4]. We could also use self-attention in the temporal dimension [19, 25], introducing it in the decoder, and automatically generating features from our assets [32].
8 Acknowledgements
The authors would like to thank the Oxford-Man Institute of Quantitative Finance for its generous support. SR would like to thank the U.K. Royal Academy of Engineering.
References
- [1] Kent Daniel and Tobias J. Moskowitz. Momentum crashes. Journal of Financial Economics, 122(2):221 – 247, 2016.
- [2] Ashish Garg, Christian L Goulding, Campbell R Harvey, and Michele Mazzoleni. Momentum turning points. Available at SSRN 3489539, 2021.
- [3] Bryan Lim, Stefan Zohren, and Stephen Roberts. Enhancing time-series momentum strategies using deep neural networks. The Journal of Financial Data Science, 1(4):19–38, 2019.
- [4] Kieran Wood, Stephen Roberts, and Stefan Zohren. Slow momentum with fast reversion: A trading strategy using deep learning and changepoint detection. The Journal of Financial Data Science, 4(1):111–129, 2022.
- [5] Tobias J Moskowitz, Yao Hua Ooi, and Lasse Heje Pedersen. Time series momentum. Journal of financial economics, 104(2):228–250, 2012.
- [6] Nick Baltas. The impact of crowding in alternative risk premia investing. Financial Analysts Journal, 75(3):89–104, 2019.
- [7] Gregory W Brown, Philip Howard, and Christian T Lundblad. Crowded trades and tail risk. The Review of Financial Studies, 35(7):3231–3271, 2022.
- [8] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
- [9] Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. 2016.
- [10] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017.
- [11] Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. Rl2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.02779, 2016.
- [12] Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint arXiv:1611.05763, 2016.
- [13] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, pages 1126–1135. PMLR, 2017.
- [14] James M Poterba and Lawrence H Summers. Mean reversion in stock prices: Evidence and implications. Journal of financial economics, 22(1):27–59, 1988.
- [15] Dimitri Vayanos and Paul Woolley. An institutional theory of momentum and reversal. The Review of Financial Studies, 26(5):1087–1145, 2013.
- [16] William F. Sharpe. The sharpe ratio. The Journal of Portfolio Management, 21(1):49–58, 1994.
- [17] Justin Sirignano and Rama Cont. Universal features of price formation in financial markets: Perspectives from deep learning. SSRN, 2018.
- [18] Zihao Zhang, Stefan Zohren, and Stephen Roberts. Deeplob: Deep convolutional neural networks for limit order books. IEEE Transactions on Signal Processing, 67(11):3001–3012, 2019.
- [19] Kieran Wood, Sven Giegerich, Stephen Roberts, and Stefan Zohren. Trading with the momentum transformer: An intelligent and interpretable architecture. arXiv preprint arXiv:2112.08534, 2021.
- [20] Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. arXiv preprint arXiv:1901.05761, 2019.
- [21] Carl Doersch, Ankush Gupta, and Andrew Zisserman. Crosstransformers: spatially-aware few-shot transfer. Advances in Neural Information Processing Systems, 33:21981–21993, 2020.
- [22] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
- [23] Roman Garnett, Michael A Osborne, Steven Reece, Alex Rogers, and Stephen J Roberts. Sequential bayesian prediction in the presence of changepoints and faults. The Computer Journal, 53(9):1430–1446, 2010.
- [24] Yunus Saatçi, Ryan D Turner, and Carl E Rasmussen. Gaussian process change point models. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pages 927–934, 2010.
- [25] B Lim, SO Arik, N Loeff, and T Pfister. Temporal fusion transformers for interpretable multi-horizon time series forecasting. arxiv. arXiv preprint arXiv:1912.09363, 2019.
- [26] Abby Y. Kim, Yiuman Tse, and John K. Wald. Time series momentum and volatility scaling. Journal of Financial Markets, 30:103 – 124, 2016.
- [27] Campbell R. Harvey, Edward Hoyle, Russell Korgaonkar, Sandy Rattray, Matthew Sargaison, and Otto van Hemert. The impact of volatility targeting. SSRN, 2018.
- [28] Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pages 400–407, 1951.
- [29] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http://www.deeplearningbook.org.
- [30] Brian Hurst, Yao Hua Ooi, and Lasse Heje Pedersen. A century of evidence on trend-following investing. The Journal of Portfolio Management, 44(1):15–29, 2017.
- [31] Jamil Baz, Nicolas Granger, Campbell R. Harvey, Nicolas Le Roux, and Sandy Rattray. Dissecting investment strategies in the cross section and time series. SSRN, 2015.
- [32] Tom Liu, Stephen Roberts, and Stefan Zohren. Deep inception networks: A general end-to-end framework for multi-asset quantitative strategies. arXiv preprint arXiv:2307.05522, 2023.
- [33] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.
- [34] Cheng Guo and Felix Berkhahn. Entity embeddings of categorical variables. arXiv preprint arXiv:1604.06737, 2016.
- [35] Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289, 2015.
- [36] Ruofeng Wen, Kari Torkkola, Balakrishnan Narayanaswamy, and Dhruv Madeka. A multi-horizon quantile recurrent forecaster. arXiv preprint arXiv:1711.11053, 2017.
- [37] Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint arXiv:1807.01622, 2018.
- [38] Shenghao Qin, Jiacheng Zhu, Jimmy Qin, Wenshuo Wang, and Ding Zhao. Recurrent attentive neural process for sequential data. arXiv preprint arXiv:1910.09323, 2019.
- [39] Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. arXiv preprint arXiv:2202.10054, 2022.
- [40] Massimiliano Patacchiola, Jack Turner, Elliot J Crowley, Michael O’Boyle, and Amos J Storkey. Bayesian meta-learning for the few-shot setting via deep kernels. Advances in Neural Information Processing Systems, 33:16108–16118, 2020.
- [41] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1199–1208, 2018.
- [42] Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. arXiv preprint arXiv:1904.04232, 2019.
- [43] Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In International Conference on Machine Learning, pages 1704–1713. PMLR, 2018.
- [44] James Harrison, Apoorva Sharma, Chelsea Finn, and Marco Pavone. Continuous meta-learning without tasks. Advances in neural information processing systems, 33:17571–17581, 2020.
- [45] Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: a good embedding is all you need? In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIV 16, pages 266–282. Springer, 2020.
- [46] Guneet S Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A baseline for few-shot image classification. arXiv preprint arXiv:1909.02729, 2019.
- [47] Steinar Laenen and Luca Bertinetto. On episodes, prototypical networks, and few-shot learning. Advances in Neural Information Processing Systems, 34:24581–24592, 2021.
- [48] Timon Willi, Jonathan Masci, Jürgen Schmidhuber, and Christian Osendorfer. Recurrent neural processes. arXiv preprint arXiv:1906.05915, 2019.
- [49] Gautam Singh, Jaesik Yoon, Youngsung Son, and Sung** Ahn. Sequential neural processes. Advances in Neural Information Processing Systems, 32, 2019.
- [50] Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. Deeptime: Deep time-index meta-learning for non-stationary time-series forecasting. arXiv preprint arXiv:2207.06046, 2022.
- [51] **gwen Jiang, Bryan T Kelly, and Dacheng Xiu. (re-) imag (in) ing price trends. Chicago Booth Research Paper, (21-01), 2020.
- [52] Xingyue Stacy Pu, Stephen Roberts, Xiaowen Dong, and Stefan Zohren. Network momentum across asset classes. Stephen and Dong, Xiaowen and Zohren, Stefan, Network Momentum across Asset Classes (August 7, 2023), 2023.
- [53] Yongxin Yang and Timothy Hospedales. Partial index tracking: A meta-learning approach. In Conference on Lifelong Learning Agents, pages 415–436. PMLR, 2023.
- [54] Bryan T Kelly and Dacheng Xiu. Financial machine learning. Technical report, National Bureau of Economic Research, 2023.
- [55] Bryan Lim and Stefan Zohren. Time-series forecasting with deep learning: a survey. Philosophical Transactions of the Royal Society A, 379(2194):20200209, 2021.
- [56] Wee Ling Tan, Stephen Roberts, and Stefan Zohren. Spatio-temporal momentum: Jointly learning time-series and cross-sectional strategies. arXiv preprint arXiv:2302.10175, 2023.
- [57] Yichi Zhang, Mihai Cucuringu, Alexander Y Shestopaloff, and Stefan Zohren. Robust detection of lead-lag relationships in lagged multi-factor models. arXiv preprint arXiv:2305.06704, 2023.
- [58] Magnus Wiese, Robert Knobloch, Ralf Korn, and Peter Kretschmer. Quant gans: deep generation of financial time series. Quantitative Finance, 20(9):1419–1440, 2020.
- [59] Taesup Kim, Jaesik Yoon, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sung** Ahn. Bayesian model-agnostic meta-learning. arXiv preprint arXiv:1806.03836, 2018.
- [60] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
- [61] Ronald J. Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks, 1989.
- [62] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
- [63] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15:1929–1958, 2014.
- [64] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In Autodiff Workshop – Advances in Neural Information Processing (NeurIPS), 2017.
Supplementary Material
Appendix A Gaussian Process Change Point Segmentation
Even after using returns to linearly de-trending the price-series and by employing volatility scaling to target a consistent volatility, we still encounter occasional and significant periods of disequilibrium or regime change. The work in [4] explores how a Gaussian Process (GP) based online change-point detection (CPD) [23, 24] module can be inserted to help address any transitions in regime, which motivates the episodic approach taken in this paper. The work by [4], assumes that there is a single change-point in some pre-specified lookback window (LBW), then calculates the change-point location and severity, therefore an approximately stationary regime will correspond with a very low severity. The severity is computed by the improvement in log marginal-likelihood using a change-point covariance kernel, in comparison to a simple Gaussian Processes, 888A typical choice could be an Ornstein–Uhlenbeck process, which is the Matérn Kernel 1/2 kernel. For consistency with the motivating work [4], we use the Matérn 3/2 kernel.. The Change-point kernel assumes that there are instead two underlying Gaussian Processes of the same kernel, and (which we refer to together as ), either side of the assumed change-point with a soft transition from the first to the second. That is, we measure the benefit of using two separate kernels instead of one. We leave the location of the transition as a free variable, which we tune when we maximize the likelihood. In this work we use a LBW of as a compromise between speed of detection and robustness to noise [4]. This is not to be confused with the fact that we can have regimes longer than because we identify regime change when the severity threshold is reached. We detail our CPD algorithm in Algorithm 1. We use a change-point severity of for the experiments where we set maximum regime length as and we set for the experiments where we use . We disregard regimes with lengths less than .
Appendix B Recurrent Attentive Neural Process Experiments
B.1 Model Architecture
We use the recurrent attentive neural process [38] as a benchmark to study the importance of certain model components. In this section, we will summarize the model and provide an overview in Fig. 10.
We use contexts which are sequences of length . The targets have a length and are required to produce a prediction . Causally the contexts are all observed at time-steps prior to the targets. The contexts are passed into an LSTM encoder with parameters . The final hidden states are passed into a self-attention module over contexts before the cross-attention branch.
We perform cross attention between the context encodings and the targets . The targets are encoded by a separate LSTM network with parameters and the final hidden state is the query vector in the cross-attention model. The encoded contexts are the values. The keys are the contexts, which are encoded similarly to the values but with another LSTM encoder with parameters . The output of the cross attention is and is used to condition the decoder [59, 38].
The encodings from the encoder are then aggregated with a sum operation in Fig. 10. These encodings are then passed into two separate linear layers which are used to parameterize the mean and variance of a latent variable which is distributed according to a Gaussian distribution and trained using the reparameterization trick [60]. The context encodings and the draw from , are stacked on top of the targets before producing predictions with an LSTM decoder network with parameters [38, 48].
B.2 Results
We generate a dataset of GP draws for the recurrent attentive neural process [38] Fig. 10 to produce forecasts. The GP curves are generated using an RBF (Radial Basis Function) kernel with a length scale of and noise variance of . Each context is drawn from a different GP draw. The length of each sequence in the context , the input and output are temporally connected sequences of length . Likewise, the target is taken from a separate GP draw where the input and the output are temporally connected sequences of length . The target decoder is trained for k iterations using teacher forcing and during testing unrolls the forecast using the previous prediction as a new input [61]. We use a Gaussian likelihood and assess test performance using the MSE.
From Fig. 11 we can see that the baseline LSTM forecaster with no context , suffers from overfitting after training for a fixed number of iterations. The full sequential NP with latent variable, self-attention over the context hidden states, and cross-attention between target and contexts obtains good forecasting performance for all different experimental settings for different context set sizes.
When we ablate away certain components, we see that performance remains the same when removing the latent variable and the self-attention. However, If we remove the cross-attention between the contexts and the target sequences then the sequential NP underfits severely. This underfitting becomes more severe as more context sequences are used to condition the sequential NP (Fig. 11).
Appendix C Training Details
We calibrate our model using the training data by optimizing the Sharpe loss function via minibatch Stochastic Gradient Descent (SGD) [28], using the Adam optimizer [62]. We employ dropout [63], which helps to prevent the model from overfitting by randomly removing hidden nodes in the training phase. We list the fixed model parameters for each architecture in Table 4, including early stop** patience. We keep the last 10% of the training data, for each asset, as a validation set. We implement 10 iterations of random grid search, as an outer optimization loop, to select the best hyperparameters, based on the validation set. The hyperparameter search grid for each architecture is listed in Table 4. We perform 10 full repeats of the outer optimization loop and ensemble the 10 models for our experiments to reduce noise.
Our model was implemented in the deep-learning framework PyTorch [64]. It was trained on a NVIDIA GeForce RTX 3090 GPU.
Hyperparameters | Random Grid |
---|---|
Dropout Rate | 0.3, 0.4, 0.5 |
Hidden Layer Size, | 64, 128 |
Minibatch Size, | 64, 128 |
Max Gradient Norm |
Parameter | Value |
---|---|
Learning Rate | |
Target, training warm-up steps, | |
Target, total LSTM steps, | |
Early stop** patience | 10 |
Maximum SGD iterations | 100 |
Number attention heads | 4 |
Attention dimension, | |
Joint loss weight, | 1 |
Joint loss weight, | 5 |
Appendix D Datasets
We backtest our X-Trend variants on a portfolio of 50 liquid, continuous futures contracts over the period 1990–2023, extracted from the Pinnacle Data Corp CLC Database. The futures contracts are chained together using the backwards ratio-adjusted method. For our few-shot experiments, we use the same assets for both training and testing, using all assets in both Table 6 and Table 6. For our zero-shot experiments, we randomly selected 20 of the 50 Pinnacle assets as the target set , which we detail in Table 6, leaving the other 30 for , which we detail in Table 6. It should be noted that we chose to not include any fixed income contracts in the zero-shot portfolio, due to the fact that they are more correlated than the other futures contracts.
Identifier | Description |
---|---|
Commodities (CM) | |
CC | COCOA |
DA | MILK III, composite |
LB | LUMBER |
SB | SUGAR #11 |
ZA | PALLADIUM, electronic |
ZC | CORN, electronic |
ZF | FEEDER CATTLE, electronic |
ZI | SILVER, electronic |
ZO | OATS, electronic |
ZR | ROUGH RICE, electronic |
ZU | CRUDE OIL, electronic |
ZW | WHEAT, electronic |
ZZ | LEAN HOGS, electronic |
Equities (EQ) | |
EN | NASDAQ, MINI |
ES | S&P 500, MINI |
MD | S&P 400 (Mini electronic) |
SC | S&P 500, composite |
SP | S&P 500, day session |
XX | DOW JONES STOXX 50 |
YM | Mini Dow Jones ($5.00) |
Fixed Income (FI) | |
DT | EURO BOND (BUND) |
FB | T-NOTE, 5yr composite |
TY | T-NOTE, 10yr composite |
UB | EURO BOBL |
US | T-BONDS, composite |
Foreign Exchange (FX) | |
AN | AUSTRALIAN $$, composite |
DX | US DOLLAR INDEX |
FN | EURO, composite |
JN | JAPANESE YEN, composite |
SN | SWISS FRANC, composite |
Identifier | Description |
---|---|
Commodities (CM) | |
GI | GOLDMAN SAKS C. I. |
JO | ORANGE JUICE |
KC | COFFEE |
KW | WHEAT, KC |
NR | ROUGH RICE |
ZG | GOLD, electronic |
ZH | HEATING OIL, electronic |
ZK | COPPER, electronic |
ZL | SOYBEAN OIL, electronic |
ZN | NATURAL GAS, electronic |
ZP | PLATINUM, electronic |
ZT | LIVE CATTLE, electronic |
Equities (EQ) | |
CA | CAC40 INDEX |
ER | RUSSELL 2000, MINI |
LX | FTSE 100 INDEX |
NK | NIKKEI INDEX |
XU | DOW JONES EUROSTOXX50 |
Foreign Exchange (FX) | |
BN | BRITISH POUND, composite |
CN | CANADIAN $$, composite |
MP | MEXICAN PESO |