metadata: authors: 0: author: Foresti,Andrea
count: United States
docna: 0: docna: Estimation of the ex ante Distribution of
            Returns for a Portfolio of U.S. Treasury Securities via Deep Learning
docty: Policy Research Working Paper
subtopic: Public Sector Economics,Public Finance Decentralization and
            Poverty Reduction
teratopic: Public Sector Development
abstracts: cdata!: This paper presents different deep
            neural network architectures designed to forecast the
            distribution of returns on a portfolio of U.S. Treasury
            securities. A long short-term memory model and a
            convolutional neural network are tested as the main building
            blocks of each architecture. The models are then augmented
            by cross-sectional data and the portfolio's empirical
            distribution. The paper also presents the fit and
            generalization potential of each approach.
geo_reg_key: 119228,517191
repnme: repnme: Estimation of the ex ante Distribution of
            Returns for a Portfolio of U.S. Treasury Securities via Deep Learning
display_title: Estimation of the ex ante
            Distribution of Returns for a Portfolio of U.S. Treasury
            Securities via Deep Learning
listing_relative_url: /research/2019/03/30921728/estimation-ex-ante-distribution-returns-portfolio-treasury-securities-deep-learning
topicv3: Treasury m1327009 1045
publishtoextweb_dt: 2019-03-21T00:00:00Z
historic_topic: Public Sector Development
no_of_pages: 28
author_keys: 0: author_key: 1278218
model_extractions: Policy Research Working Paper 8790 Estimation of the ex ante Distribution of Returns for a Portfolio of U.S. Treasury Securities via Deep Learning _Andrea Foresti_ Market and Counterparty Risk Team March 2019
Policy Research Working Paper 8790

# **Abstract**

This paper presents different deep neural network architectures designed to forecast the distribution of returns on a portfolio of U.S. Treasury securities. A long short-term memory model and a convolutional neural network are tested as the main building blocks of each architecture. The models are then augmented by cross-sectional data and the portfolio’s empirical distribution. The paper also presents the fit and generalization potential of each approach.

This paper is a product of the Market and Counterparty Risk Team. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/research. The author may be contacted at aforesti@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
Estimation of the ex ante Distribution of Returns for a Portfolio of U.S. Treasury Securities via Deep Learning Andrea Foresti The World Bank

**JEL** : C45, C58, G17

**Keywords** : Machine Learning, Neural Networks, Convolution, LSTM, Market Risk
# 1 Introduction

Deep learning is a branch of machine learning that may be loosely defined as a collection of statistical algorithms which feature different layers of operations (usually more than three). These algorithms can be used in either a supervised or unsupervised manner.

Deep learning methods are being actively explored for a variety of practical purposes and have, in recent years, found considerable success. In particular, these methods can be used for both regression (e.g. see [Lathuili`ere et al., 2018]) and classification tasks (e.g. see [Krizhevsky et al., 2012]). Both these fields of application are very significant in the context of Moreover, deep learning is of particular interest in all those tasks that present complex input and output data, as the data coming into the model, flowing from one layer to another and finally exiting as output can be arranged in complex multi-dimensional structures.

As we will later see, for instance, classification tasks often entail the specification of the level of confidence across the available classes, thus allowing to give a more nuanced answer than a simple binary (yes/no) attribution.

As a result of all these advantages, in the past few years Deep Learning has started to see fervent research and actual application in finance (as, for instance, recognized in [Board, 2017]).

As an illustration, we have so far seen examples in:

- time series forecasting (e.g. [Sun et al., 2018])

- portfolio construction (e.g. [Heaton et al., 2017])

- credit analysis (e.g. for default prediction see [Hosaka, 2019] or [Hamori et al., 2018])

- There has been, however, limited publicly available research in the field of Market Risk.

2
# 2 Deep neural networks

A deep neural network is a deep learning model composed of several layers of computations that are loosely inspired by the neurological structure of the brain.

The layers consist of different components (neurons) that are connected to the output of the previous layer according to some pattern. Every layer enacts a transformation on both the content and the shape of the data that flow through it.

The oldest and most common type of layer is the so called _dense_ or _fully connected_ layer. This type of layer transforms the data by multiplying their inputs by a weight matrix (and adding - optionally - a fixed number (called the _bias_ ) to its results): where _xl_ is the _l_<sup>_t_</sup> _h_ layer in the network, _Wl_<sup>_l_</sup> _−_ 1<sup>istheweightsmatrixthattransformthe</sup> data from layer _l −_ 1 to layer _l_ and _bl_ is the bias vector. From equation 1 we see that only one bias is applied to every layer, while the weight matrix size is determined by the input and desired output data dimensions.

The output data of a layer are then usually transformed by a function operating element-wise (called the _activation function_ ). The most commonly used activation functions are: <!-- Start of picture text --> 0 for x < 0<br>Binary<br>1 for x > = 0<br>�<br><!-- End of picture text -->

|Binary|�<br>0 for _x <_0<br>1 for _x >_= 0|
|---|---|
|Sigmoid|1<br>1 +_e_<sup>_−x_</sup>|
|TanH|_e_<sup>_x _</sup>_−e_<sup>_−x_</sup><br>_e_<sup>_x_ </sup>+_e_<sup>_−x_</sup>|
|ReLu|�<br>0 for _x <_0<br>_x_ for _x >_= 0|

<!-- Start of picture text --> 0 for x < 0<br>ReLu<br>x for x > = 0<br>�<br><!-- End of picture text --> Table 1: Common activations 3
When the outputs of a layer are instead directly passed to the following layers, this is normally referred to as a _linear activation_ .

What is interesting is that both continuous and discontinuous activation functions are attractive for different purposes during learning.

The fully connected layer works very well for a variety of different tasks but is poorly suited to the analysis of data that are structured in multiple dimensions (such as time-dependent multivariate variables).

While, in theory, a deep learning neural network comprised only of fully connected layers would be able to tackle such a problem, it would in reality present two main problems:

1. in order to be able to learn effectively, the model would need to be of a considerable size (at a minimum hundreds of thousands of parameters);

2. such a model design would have a tendency to overfit the training data.

The first obstacle has been successfully tackled by recent research developments (starting with Geoffrey Hinton’s seminal paper [Hinton et al., 2006]) and the availability of ever-greater computing power. The second one, however, has prompted research into different learning architectures that are better suited to extract meta-features from the input data.

One particular solution has been to introduce specialized types of layers to work alongside the fully connected ones.

# 2.1 Recurrent neural networks and LSTM

A first approach to model time-dependent factors is to use a so-called recurrent neural network (RNN). Recurrent neural networks have the ability to consider time-dependent data, as they work in a way that allows the same set of weights to be applied sequentially to all the items contained in a sample.

RNNs have traditionally been very hard to train and thus they were not used extensively. This changed with the introduction of the Long Short-Term Memory (LSTM) model [Hochreiter and Schmidhuber, 1997]. The LSTM model has, for instance, proven very successful in natural language processing applications and spurred a lot of applications that entail, for instance, text prediction.

The center of the LSTM model is the LSTM cell. A variable number of these cells comprise an LSTM layer, either working in parallel or stacked sequentially.

4
<!-- Start of picture text -->
| ry ry<br>ry<br>=] z o—<br>s<br>hy<br>(x) Elementwise multiplication<br>[> | Dense layer (sigmoid activation)<br>[« | Concatenation<br>Elementwise tanh<br><!-- End of picture text -->
Convolutions are particularly powerful when used in sequence, as they can achieve a significant dimensionality reduction on the data, while automatically identifying and extracting the most significant features (see e.g. [Krizhevsky et al., 2012]).

In particular, there is ample evidence of the effectiveness of using several convolutions paired with data pooling (i.e. applying a filter that iterates on the data similarly to a convolution, but that reduces the data in each window to a scalar using a function - such as the average or maximum value).

Another advantage of Convolutional Neural Networks is that they are faster to train than regular feed-forward networks composed of solely fully connected layers. On the other hand, their power often translates in a tendency to overfit the data (i.e. fit the noise in the training data set while having poor performance in the out-of-sample portion of the data). However, several solutions to the problem of overfitting have been proposed, including:

- better feature selection;

- expansion of the number of test cases;

- input data normalization;

- parameter regularization;

- dropout;

- batch normalization.

# 2.3 Loss functions

The models presented in this paper all belong to the family of supervised learning methods. This means that the parameters of the model are determined as to minimize a specific error (’loss’) quantity.

The loss function allows to quantify this error in terms of the distance between a value estimated by the model and the actual (’true’) value recorded for the observation. Common functions used in Machine Learning are:

- Mean Square Error (L2 loss)

- Mean Absolute Error (L1 loss)

- Hinge Loss (Maximum margin loss)

- Cross Entropy Loss

The actual function used depends on the particular problem being studied. For instance, the first two losses are used in regression tasks whereas the last two are used in classification models.

6
# 2.4 Data and parameter regularization

Input data are routinely scaled in deep learning in order to achieve faster and better convergence of the model weights. This technique is effectively used to prevent model underfit (as opposed to parameter regularization which aims to achieve the opposite effect). It is, for instance, common practice either to use either _MinMax_ scaling or _Standard_ scaling. In the first case, the data are rescaled between two arbitrary values (usually zero and one). In the second case, the data are rescaled to have zero mean and unit variance.

## 2.4.1 L1 and L2 norms

When the problem, on the other hand is model overfit, kernel weights, biases and layer outputs are instead regularized. A common way to do that is via _L_ 1 and _L_ 2 norms. These are added to the penalty function (scaled by an arbitrary parameter _λ_ ) in order to discourage their target from becoming too big.

More precisely (given a vector _β_ of parameters):

## 2.4.2 Batch normalization

Batch normalization (see [Ioffe and Szegedy, 2015]) works by normalizing (i.e. subtracting the batch mean and dividing by the batch standard deviation) a layer’s activations using their recorded values during each mini-batch. It is another method that has shown results in allowing greater model generalization.

## 2.4.3 Dropout

Introducing node dropout is yet another method used to achieve better model robustness. In this case a random percentage _p_ of outputs for a pre-determined layer is selected at every gradient update and the values for those nodes is set to zero.

In order to achieve the best effectiveness the dropout percentage _p_ is normally set to a value around 0.5.

7
# 3 An application of deep learning on a portfolio of U.S. Treasury bonds

I here analyze the ability of several deep learning architectures to accurately predict the future distribution of a linear portfolio of US Treasury exposures. Such a problem is of particular interest in the field of Market Risk management.

It has to be noted here that a classical value-at-risk (VaR) analysis is not immediately amenable to be performed using the types of models here presented. This is because the use of classical model backtesting performance (i.e. distance between predicted and actual breaches over the observations) as a loss function during training would invariably lead the optimizer to the trivial scalar solution.

In fact, if we define VaR for a variable X as: and I backtest over a set of training data using the actual portfolio returns, we see that a model that consistently predicted the _n · α_ worst return would indeed achieve a perfect score.

In other words, the model would not learn from the data instead moving to the (fixed) solution that corresponds to the number of breaches that should be observed during training. This clearly would not have any generalization value. For this reason, it is a lot more appealing to consider the whole future distribution, instead of a single statistic calculated on it.

## 3.1 Portfolio

The portfolio weights for this analysis are fixed in advance as follows: They will be kept constant across all experiments. A relaxation of this constraint is certainly deserving of future research.

8
# 3.2 Market data

I consider 10 years of daily data for US Treasury zero rates (provided by Bloomberg):

|Term|Ticker|
|---|---|
|3M|I02503M Index|
|6M|I02506M Index|
|1Y|I0251Y Index|
|2Y|I0252Y Index|
|3Y|I0253Y Index|
|5Y|I0255Y Index|
|7Y|I0257Y Index|
|10Y|I02510Y Index|
|30Y|I02530Y Index|

Table 2: Zero rates Using these rates, I compute the daily returns on the corresponding zero-coupon bonds: These returns are then arranged in a [500 x 9] matrix _for every observation_ . In other words, every matrix contains the latest 500 observations of these returns.

The choice of the past 500 observations is motivated by the widely adopted convention in market risk of using the previous two years of returns for the estimation of the parameters used for ex ante risk calculations.

In addition to the return matrix, some model versions presented here augment the returns data with cross-sectional data. These models are referred to in this document as ’mixed’ models.

The basic data used in these cases are presented in table 3.

9
|Description|Ticker|
|---|---|
|US CPI YOY|CPI YOY Index|
|EU CPI YOY|ECCPEMUY Index|
|JP CPI YOY|JNCPIYOY Index|
|EUR 10yr swap rate|EUSA10 Curncy|
|GBP 10yr swap rate|BPSW10 Curncy|
|USD 10yr swap rate|USSW10 Curncy|
|JPY 10yr swap rate|JYSWAP10 Curncy|
|AUD 10yr swap rate|ADSWAP10 Curncy|
|3M USD Zero rate|I02503M Index|
|6M USD Zero rate|I02506M Index|
|1Y USD Zero rate|I0251Y Index|
|2Y USD Zero rate|I0252Y Index|
|3Y USD Zero rate|I0253Y Index|
|5Y USD Zero rate|I0255Y Index|
|7Y USD Zero rate|I0257Y Index|
|10Y USD Zero rate|I02510Y Index|
|30Y USD Zero rate|I02530Y Index|
|10Y US Treasury yield|USGG10YR Index|
|2Y US Treasury yield|USGG2YR Index|

Table 3: Base cross-sectional data In addition, some simple computed features are constructed using the cross-sectional variables as building blocks (Table 4).

10
Formula

|10Y US Treasury yield|-|2Y US Treasury yield|
|---|---|---|
|USD CPI|avg|50d|
|USD CPI|avg|100d|
|USD CPI|avg|200d|
|3M USD Zero rate|avg|50d|
|6M USD Zero rate|avg|50d|
|1Y USD Zero rate|avg|50d|
|2Y USD Zero rate|avg|50d|
|3Y USD Zero rate|avg|50d|
|5Y USD Zero rate|avg|50d|
|7Y USD Zero rate|avg|50d|
|10Y USD Zero rate|avg|50d|
|30Y USD Zero rate|avg|50d|

Table 4: Calculated cross-sectional data The rationale behind pre-computing certain features is to achieve a better economy in model estimation.

Another data source provided to some versions of the models is the empirical (historical) portfolio distribution. It is calculated using the returns matrix associated to each observation to compute a vector of portfolio returns. The histogram of these returns is then used (the portfolio returns having equal weights).

For T observations of a returns vector with members _rt_ and a vector L [ _l_ 1 _. . . ln_ ] of bucket delimiters, the empirical distribution is defined as this vector: The buckets are delimited according to this rule: where R is the vector of one-day portfolio returns (defined as: _rt ∗ positions_ where position is the vector in (5)).

# 3.3 Target data

The model will try to forecast the distribution of the actual (forward-looking) 10-day returns of the portfolio.

11
These returns are calculated as: The returns range is then divided into buckets that are delimited using the same rule defined in equation (8). These buckets can be seen as the classes to which the models will have to assign probabilities.

# 3.4 Softmax activation

The final layer of all the architectures presented here is a fully connected layer with a _softmax_ activation function. This particular activation ouputs a vector whose values sum to one.

The _softmax_ transformation of a vector x (with n elements) into an output vector y is as: The softmax activation function is widely used in classification applications. It allows the model output to be interpreted as the confidence for the state of the world (i.e. future portfolio returns) to belong to any of the available classes.

This means that, in the context of this analysis, the output classes are defined as the discrete bins of the returns distribution.

# 3.5 Loss function

The loss function used to evaluate the model is the categorical cross-entropy. Over n observations on a model that outputs C classes (i.e. return buckets), entropy (E) is defined in this context as: The choice of this particular loss function results in a loss of zero for a model with perfect foresight while having an infinite value in case a zero probability is wrongly assigned to a class.

12
<!-- Start of picture text --> Istm_1_ (None, 500, 9)<br>input: InputLayer (None, 500, 9)<br>Istm_1: LSTM P (None, 500,9)<br>(None, 500, 1000)<br>Istm_ 2: LSTM I aone, 500, 1000)<br>- (None, 500, 500)<br>Istm_3: LSTM P (None, 500, 500)<br>dense_1: Dense P (None, 1000)<br>dense_2: Dense I aone, 1000)<br>~ (None, 500)<br>dense _3: Dense I a )<br>~ (None, 200)<br>dense_4: Dense E a<br>(None, 100)<br>dense_5: :Dense (None, 100)<br><!-- End of picture text -->
/ ; (None, 500, 9, 1) Convolution_One_input: InputLayer = (None, 500, 9, 1) <!-- Start of picture text -->
| (None, 500, 9, 1)<br>Convolution_One: Conv2D (None, 481,7, 128)<br>; (None, 481,<br>Convolution_Two: Conv2D 7, 128)<br>- (None, 479,5, 128)<br>Convolution_Three:/ Conv2D (None, 479,5, 128)<br>(None, 477, 3, 128)<br>/ / (None, 477, 3, 128)<br>max_pooling2d_1: MaxPooling2D (None, 238,1, 128)<br>flatten_1: Flatten I ohone, 238, 1, 128)<br>(None, 30464)<br>dense_1: Dense H aone, 30464)<br>(None, 1000)<br>activation_1: Activation H a )<br>(None, 1000)<br>dense_2: Dense (None, 1000)<br>(None, 500)<br>activation_2: Activation- (None, 500)<br>~ (None, 500)<br>dense_3: Dense H w )<br><!-- End of picture text -->
<!-- Start of picture text --> ; (None, 500, 9)<br>input_2: InputLayer (None, 500, 9)<br>LSTM_One: LSTM P w ) input_1: InputLayer P w<br>LSTM_Two: LSTM P a ) dense_1: Dense us w )<br>- (None, 500, 500) - (None, 1000)<br>LSTM_Three: LSTM (None,(None,500,250)7 500) dense_2: Dense (None,(None, 1000)500)=<br>concatenate_1: Concatenate [(None, 250), (None, 500)]<br>dense_3: Dense u a )<br>— (None, 1000)<br>dense_4: Dense (None, 1000)<br>(None, 500)<br>dense_5: Dense (None, 500)<br>(None, 200)<br>dense_6: Dense H a )<br>~ (None, 100)<br>dense_7::Dense (None, 100)<br><!-- End of picture text -->
<!-- Start of picture text --> input_2: (None, 500, 9, 1)<br>= InputLayer (None, 500, 9, 1)<br>Convolution _One: Conv2D “p wone, 500,9,1 )<br>(None, 481, 7, 128)<br>Convolution_Two: Conv2D P &one, 481, 7, 128 )<br>— (None, 479, 5, 128)<br>Convolution_Three:- Conv2D (None, 479, 5, 128) input_1: InputLayer (None, 33)<br>(None, 477,3, 128) = (None, 33)<br>max_pooling2d_1: MaxPooling2D P (None, 477, 3, 128) dense_1: Dense ! (None, 33)<br>(None, 238, 1, 128) (None, 1000)<br>flatten_1: Flatten P wone, 238, 1, 128 ) dense_2: Dense “p wjone, 1000)<br>(None, 30464) (None, 500)<br>concatenate_1: Concatenate [(None, 30464), (None, 500)]<br>(None, 30964)<br>dense_3: Dense u (None, 30964)<br>(None, 1000)<br>dense_4: Dense P al(None,one, 1000)500)<br>dense_5: Dense (None, 500)<br>(None, 200)<br>dense_6: Dense H (None, 200)<br>(None, 100)<br>dense_7: Dense (None, 100)<br><!-- End of picture text -->
<!-- Start of picture text --> LSTM_Input: (None, 500,9)<br>= InputLayer (None, 500,9)<br>LSTM_One: LSTM CrossSectional_Input: InputLayer Historical_Input: InputLayer<br>LSTM_Two: LSTM [_inpot:] (None, 500, 500) CrossSectional_One: Dense w (None, 1000) Historical_One: Dense v (None, 1000)<br>LSTM_Three: LSTM (None, 500, 500) CrossSectional_Two:; Dense (None, 1000) Historical_Two:—_ Dense (None, 1000)<br>concatenate_1: Concatenate [@None, 250), (None, 500), (None, 500)]<br>(None, 1250)<br>Dense_One:7 Dense | put | ‘(None, 1000))<br>Dense_Two: Dense (None, 1000)<br>~ (None, 500)<br>Dense_Three: Dense P as »<br>Dense_Four: Dense (None, 200)<br>(None, 100)<br>(None, 100)<br>Output: Dense<br><!-- End of picture text -->
<!-- Start of picture text --> Convolution_Input: (None, 500, 9, 1)<br>7 InputLayer (None, 500, 9, 1)<br>ConvolutionA_One:; Conv2D (None, 500,9, 1) ConvolutionB_One: Conv2D (None, 500, 9, 1)<br>(None, 481, 7, 128) (None, 461,7, 128)<br>ConvolutionA_Two: Conv2D [ioe [os (None,jone, 481,479, 7,5,- 128)128) ConvolutionB_Two: ConvzpD |=” (None,(None, 461,459,——7,5, 128)128)<br>ConvolutionA_Three: Conv2D ConvolutionB_Three: Conv2D CrossSectional_Input: InputLayer Historical_Input: InputLayer<br>ConvolitionA, MaxPooling:MaxPoolingxp | "2 | (None, 238, 1, 128)) | | convotition MaxPooling MasPootingap | 2PM | (None, 228, 1, 128)» | | crosssectional_One Dense | (None,x 1000)» Historical_One: Dense }— (None,ws 1000))<br>flatten_1: Flatten (None, 238,1, 128) flatten_2: Flatten (None, 228,1, 128) CrossSectional Two: Dense (None, 1000) Historical_Two: Dense (None, 1000)<br>concatenate_1: Concatenate [(None, 30464), (None, 29184), (None, 500), (None, 500)]<br>(None, 60648)<br>Dense_One Dense | “PM |&(None,jone, 60648}1000))<br>Dense_Two:: Dense (None, 1000)<br>Danse Thee Dense |“| (None,(None, 200)500)<br>Dense Four Dense | SP | (None, 100)»<br>Oniput: Dense | 2PM | »<br><!-- End of picture text -->
<!-- Start of picture text --> Training<br>LSTM (training)<br>2D convolution (training)<br>2.3 Mixed LSTM (training)<br>Mixed convolution (training)<br>Mixed historical LSTM (training)<br>2.2 \ Mixed historical convolution (training)<br>\<br>\<br>2.1 \<br>=228\ . . = i 7<br>[o]oaB22.0 ¢ -<br>19<br>18<br>17<br>oO 10 20 30 40 50<br>Epoch (no.)<br>Validation<br>LSTM (validation)<br>2.05 2D convolution (validation)<br>2 Mixed LSTM (validation)<br>2.00: WayXi \ ‘ . : ¥ . MixedMixedMixed convolutionhistoricalhistorical convolutionLSTM(validation)(validation)(validation)<br>1.95<br>><br>e \<br>5°ra9o2s 190185 v \<br>1.80<br>175<br>1.70<br>oO 10 20 30 40 50<br>Epoch (no.)<br><!-- End of picture text -->
I now choose as the candidate model versions using these criteria (in order of decreasing importance):

- least validation loss

- least training loss

- latest epoch

After training the models presented here, this resulted in choosing these models:

|Model|Epoch|Training loss|Validation loss|
|---|---|---|---|
|LSTM|07|2.05|2.00|
|Convolutional|22|2.05|2.00|
|Mixed LSTM|22|1.98|1.95|
|Mixed convolutional|19|2.05|2.00|
|Mixed historical LSTM|44|1.76|1.80|
|Mixed historical convolutional|49|1.66|1.76|
|Parametric|-|2.19|2.17|
|Historical|-|2.07|2.02|

Table 5: Optimal model performance In order to better judge the models, the shape of the produced distribution is considered. It is indeed very important to have models that produce meaningful distribution shapes and that react to data.

Machine learning models have a normal tendency to quickly converge onto degenerate solutions that nevertheless have a strong explanatory power (e.g. constant solutions). This, in a way, could be considered as a special case of overfit.

So, to measure the variability of the distributions produced by each model, I consider the Bhattacharyya distance (introduced in [Bhattacharyya, 1946]): For every model, I compute the Bhattacharyya distance between the latest available returns distribution and all the other distributions generated over the rest of the data. The results are illustrated in Figure 9.

20
<!-- Start of picture text --> isTM<br>oot<br>£ 002<br>E000<br>2-002<br>-o08<br>3 00 1000 ay 1500 2000 2500<br>20 convolution<br>a0<br>£ 002<br>E 000<br>3 002<br>008<br>3 360 1000 bay 1500 3000 2500<br>Mixed LSTM<br>03s<br>030<br>£ aes<br>% coro<br>2e oois<br>§ ooo<br>«200s<br>«2000 3 360 1000 Day 1500 2000 2500<br>lea? Mixed Convolution<br>20<br>° |<br>Bas |<br>Zio5 h<br>as<br>ao 3 360 1000 ay 1500 2000 2500<br>Mixed historical LST<br>axe<br>ous<br>on<br>3 oxo<br>3 o0e<br>g8 0.06<br>3 0.04<br>002<br>200 3 500 1000 bay 1500 3000 2500<br>Mixed historical convolution<br>os<br>£03<br>Z oa<br>EB<br>é<br>a<br>oo 3 360 1000 Day 3500 2000 3500<br>030 Historical<br>02s<br>£ 020<br>foisA<br>5 10<br>005<br>000 3 30 1000 bay 1500 2000 3500<br><!-- End of picture text -->
In order to offer a better comparison, the same distance is applied also to the empirical portfolio distributions (i.e. the historical model distribution) results.

It is clearly possible to appreciate how the complexity of the model, while maybe not significantly improving the training and validation performance, allows the creation of more reactive models.

This gives us a hint about the possibility that more complex models might offer better generalization power.

In particular, we see that the naive application of both the LSTM and convolutional model result in degenerate behavior of the distance among the predicted distributions.

For the sake of illustration, figure 10 presents the distributions predicted by each model for the latest available observation in the data set.

22
<!-- Start of picture text --> Latest prediction (Parametric) Latest prediction (Historical)<br>s<br>20<br>x»<br>22<br>E1 E,<br>»<br>5<br>5<br>° “at 2'00-'80:.60: 40-1 20-1 00-0 80-060-040-0.20000 020 040 060 080 100130 140 160 180 200 0 at 200-1 80-,60-190-1.20-100-080-060-040-020 000 020 040 a0 080 00 120 140 1.60 1.80 200<br>Latest prediction (LSTM) Latest prediction (2D convolution)<br>20 20<br>1 1s<br>22<br>aEi aEr<br>55<br>°Gar 2'00..80:1 601 40-120-1.00-0.80-060-0.40-020000 020 040 0.60 080 100 130 140 160 180 200 0 “at 2001 80:,60-190-120-100-080-0.60-0.40-020 000 020 040 a0 080 re0 130 140 160 1.80 200<br>Latest prediction (Mixed LSTM) Latest prediction (Mixed convolution)<br>2<br>20<br>20<br>1<br>E g 0<br>0<br>5 5<br>° 0<br>Comer bucket imi tower bucket mt<br>Py Latest prediction (Mixed historical LSTM) “ Latest prediction (Mixed historical convolution)<br>3<br>2<br>x»<br>20<br>23<br>2 2<br>Bis B20<br>E E<br>ra<br>»<br>0<br>5<br>5<br>° 0<br>ower bucket it ‘tower bucket mt,<br><!-- End of picture text -->
# 5 Conclusion

The results I obtained corroborate the notion that the use of deep-learning methods for financial application is viable. In particular, a finding which is consistent with existing deep learning literature is that models need to cross a complexity threshold in order to achieve significant efficacy.

Whether the models here presented already achieved this critical mass is, though, not clear. Furthermore, it has to be underlined that these tools are still poorly understood and much more difficult to calibrate than more traditional methods. It is not immediately apparent which features are more important than others (especially considering the limited nature of these experiments).

Moreover, the long time to convergence for some of the models indicates that more could be done in order to speed up learning.

Finally, these results suggest a few opportunities for further research:

- Introducing a bigger number of factors onto which the portfolio is mapped;

- Testing different model architectures, hyper-parameters and regularization techniques. The use of auto-encoders for dimensionality reduction is, in this context, particularly appealing in light of possible real-world applications (i.e. models featuring hundreds or thousands of different factors);

- Exploring different cross-sectional data features;

- Verifying the soundness of the model(s) for different portfolio structures;

- Investigating the use of different penalty functions that would be more of interest in the field of risk measurement (e.g. weighted cross-entropy).

24