Stock Embeddings Acquired from News Articles and Price History, and an Application to Portfolio Optimization (2020)

Abstract: Previous works that integrated news articles to better process stock prices used a variety of neural networks to predict price movements. The textual and price information were both encoded in the neural network, and it is therefore difficult to apply this approach in situations other than the original framework of the notoriously hard problem of price prediction. In contrast, this paper presents a method to encode the influence of news articles through a vector representation of stocks called a stock embedding . The stock embedding is acquired with a deep learning framework using both news articles and price history. Because the embedding takes the operational form of a vector, it is applicable to other financial problems besides price prediction. As one example application, we show the results of portfolio optimization using Reuters & Bloomberg headlines, producing a capital gain 2.8 times larger than that obtained with a baseline method using only stock price data. This suggests that the proposed stock embedding can leverage textual financial semantics to solve financial prediction problems.

## introduction news articles influence the dynamics of financial markets. for example, after the release of breaking news, the share prices of related stocks are often observed to move. this suggests the possibility of using natural language processing (nlp) to aid traders by analyzing this influence between news article texts and prices. recent studies @xcite @xcite have indeed reported that news articles can be leveraged to improve the accuracy of predicting stock price movements. these previous works have used deep learning techniques. they train neural networks with article texts and financial market prices, attempting to improve price prediction. in these approaches, the overall mutual effect between texts and prices is distributed over the neural network, which makes it difficult to extract this effect and apply it to tasks other than price prediction. therefore, we take a new approach by explicitly describing this mutual effect in terms of a vector. a stock is represented by a vector so that its inner product with an embedding of a text produces a larger value when the text is more related to the stock. in the rest of the paper, we call this vector a stock embedding. the names of stocks, such as "aapl" (the ticker symbol for apple inc.), typically appear in a financial news article text. because these names form part of the text, usual nlp techniques can be applied to acquire an embedding of a stock. such general textual embedding, however, does not incorporate the financial reality of stock price changes. hence, the proposed stock embedding represents the price as well as the semantics of the text, as we acquire it by training on both news articles and stock prices. precisely, our stock embedding is trained through a binary classification problem, namely, whether a stock price goes up or down in comparison with the previous day's price. as a result, an acquired stock embedding captures the relation between a stock name and a news article even when the article has no direct mention of the stock. our stock embedding can be considered as one technique to specialize, or ground, a symbol that has a practical reality outside of text. furthermore, two major advantages come with the vector form of our stock embedding. the first is that the training can be effectuated for all stocks at once, rather than stock by stock. this is an important advantage to alleviate data sparseness and prevent overfitting, as discussed in section 4. the second advantage lies in the portability of a vector. in contrast to previous works, in which stock-specific information was distributed among the parameters of a neural network, a vector representing all the characteristics of a stock is much easier to extract and apply to other uses besides price prediction. hence, this paper shows an example of portfolio optimization, one of the most important applications in finance. to the best of our knowledge, this is the first report of incorporating nlp into modern portfolio theory @xcite . our method differs from previous works that used nlp to enhance investment strategies. many previous works focused on stock price forecasting only @xcite and did not attempt to apply the learned results to other financial tasks. another previous work @xcite investigated portfolios with texts. it obtained a ranking of stocks from texts by using a neural network technique and then evaluated investment in the highest/lowest ranked stocks. that work was not based on modern portfolio theory, however, nor did it integrate price and text data. in contrast, our method uses nlp in addition to price data to acquire a general representation in the form of an embedding applicable to different targets. in our experiments, a portfolio generated using stock embeddings achieved an annual gain 2.8 times greater than that of a portfolio generated with price data only. this provides evidence that the stock embedding well encodes both text and price information. ## related work the main idea of this article is based on important techniques of nlp. it is now common to represent discrete entities in natural language by continuous vectors. these vectors are called "embeddings" and usually obtained from neural network models. examples include the word embedding @xcite , phrase embedding @xcite , sentence embedding @xcite , and event embedding @xcite . one advantage of these continuous representations is that the geometry of an embedding system contains rich semantic information, as has been discovered at many levels @xcite . the acquisition of stock embeddings in this paper is based on the original idea developed for linguistic entities. here, we extend the idea further so that the embeddings reflect the reality of a stock market outside text. a stock embedding is trained using the attention mechanism @xcite , which is another current nlp technique. the basic idea of the original attention mechanism is to assign higher weights to more relevant word vectors and make the weights adaptive to different contexts. our framework is based on the classification task for text-driven stock price movement, which has been studied intensely as follows. early research on exploiting financial news articles for better stock price prediction dates back to @xcite , in which financial indicators were extracted manually from financial statements. later, in @xcite , nlp methods were adopted for automatic text feature extraction. since the 2000s, twitter and other text-centered social media platforms have become essential sources of financial signals. @xcite found evidence for causality between the public mood extracted from tweets and the dow jones industrial average index. in @xcite , post texts collected from the yahoo! finance message board were used to predict whether the prices of 18 us stocks would rise or drop on the next trading day. as deep learning methods for nlp have become more common, many papers have reported the use of neural networks for text-driven stock classification (or prediction) tasks. @xcite proposed an event embedding to represent a news headline with a vector and used a convolutional neural network for classification. in that work, all the event embeddings of news articles published on the same day were simply averaged to summarize that day's market information. @xcite was among the first works that applied the attention mechanism to the task of news-driven stock price movement classification. they developed a dual-level attention framework, in which news articles were assigned different weights depending on the output of a logistic regression component with a bias term, so that the most informative news articles were "highlighted." the method of weighting news articles in this paper is similar to that previous work. the stock-specific information in @xcite was encoded in the neural network, however, making it focused on the price prediction task. in contrast, we represent such stock-specific information by the stock embedding, i.e., a vector, which is easy to interpret geometrically and extract for other applications. for one such application, we evaluated our stock embedding in terms of portfolio optimization. to the best of our knowledge, this is the first paper applying nlp techniques to modern portfolio theory. we use the mean-variance minimization portfolio model (introduced in section 7) proposed in @xcite , which directly led to the capital asset pricing model @xcite . ## news-driven stock price classification in this paper, the stock embedding is trained with a deep learning system through binary classification of price movements. let p t be the stock price on day t, and let y t be the desired output of the system. here, @xmath0, 2, . . . , t }, and t is the number of trading days in the considered time period. the binary classification problem indicates that y t is classified in the following way: to train such a deep learning system, news articles are used as the input. in this work, news articles are considered daily (i.e., treated in units of days). we denote the set of articles published on day t by n t , and each article by n @xmath1, with @xmath2, . . . , |n t |. this paper considers a time window around day t, denoted as [t -d 1 , t + d 2 ] given two constants d 1 , d 2 . let n [t-d 1 ,t+d 2 ] be the set of news articles published within the time window. when d 2 = -1, indicating the use of articles until day t -1, the task is called prediction, as the training does not use any articles published on or after day t. in general, this task is acknowledged as very hard @xcite @xcite according to the efficientmarket hypothesis (emh) foot_0 , and such prediction provides only a limited gain, if any. note that previous nlp studies concerning stock prices were all aimed at this hard problem @xcite @xcite . on the other hand, when d 2 ≥ 0, this paper refers to the task as classification. the performance on classification shows how well the model understands a news article. because the prediction problem is too hard and offers limited gain, as proven by many previous works, our target lies in classification. the aims are thus to acquire embeddings that are highly sensitive to textual context and to apply them to tasks other than price prediction. therefore, in this paper, we set d 1 = 4 and d 2 = 0. let the classification model be represented by a mapping f . the probability that the price of a stock j, where @xmath3, . . . , j, goes up on day t is ŷj in the process of model optimization, the model should reduce the mean cross-entropy loss between every true label y j t and its corresponding estimate ŷj t , as follows: this function describes the loss for only one stock, but a stock market includes multiple stocks. this work considers all stocks in a market equally important. the overall loss function is therefore a simple average of the cross-entropy loss for all stocks, i.e., @xmath4. ## method to acquire stock embeddings let s j represent a stock embedding, where @xmath5, 2, ..., j. this is initialized as a random vector and then trained via a neural model to obtain s j , whose inner product with the embedding of a related text becomes large. this section describes the proposed method to acquire stock embeddings by building up a neural network for price movement classification. the neural network consists of two parts: a text feature distiller and a price movement classifier. text feature distiller. the text feature distiller first converts every news article n i into a pair of vectors (n k i , n v i ) corresponding to "key" and "value" vectors, respectively. let let α j i denote the weight put on news article i with respect to stock j, to classify whether the stock price will go up or down. with the use of score i,j defined above, α j i is given as the following: by using α j i as the weights put on news articles, we compute the market status of stock j on day t as the following, which is the input to the classifier: price movement classifier. the input of the price movement classifier is a sequence of vectors, m j [t-4,t] = [m j t-4 , m j t-3 , . . . , m j t ], with respect to stock j. this is processed by a recurrent neural network using a bidirectional gated recurrent unit (bi-gru). the choice of a bi-gru was made by considering the model capacity and training difficulty. the classifier estimates the probability ŷj t : and this h o becomes the input of the second line instead of h o t . here, β τ , the weight for day τ , decides how much one day is considered in the classification. in our implementation, figure 1 illustrates the difference between models with and without classifier sharing. the upper figure (a) shows the conventional setting without sharing, in which j classifiers are generated, one for each stock. in contrast, the lower figure (b) shows one classifier generated for all stocks. this setting enables learning of the correlation among stocks, in addition to avoiding overfitting and the problem of small sample sizes. specifically, the classifier is shared across all stocks, thus achieving a sample size about 50 to 100 times larger. 1 ). following previous studies reporting that the main body of a news text produces irrelevant noise @xcite , we extracted only the headlines in both datasets. as for the stocks, we selected two subsets of the stocks in standard & poor's s&p 500 index, one for each of the wsj and r&b datasets. these subsets consisted only of stocks that were mentioned in no fewer than 100 different news articles, so that mutual effects between the articles and the price history would appear pretty often in the texts. more importantly, this ensured that keyword retrieval-based methods that locate related articles by explicit keyword matching could be applied for comparison. for the wsj and r&b datasets, the subsets had 89 and 50 stocks, respectively. all other stocks were removed from consideration. as seen in formula (2), the input for the neural network is n [t-4,t] , the set of articles around day t, and the output is y j t . the label y j t is the binarized price movement of stock j at day t. this is measured by the log-return between two subsequent days: log return j @xmath6. the distribution of log-returns is typically bell shaped with a center close to 0, as also mentioned in @xcite . the return values of the days were separated into three categories of "negative," "ambiguous," and "positive" by the use of thresholds foot_2 . here, "ambiguous" refers to those samples close to 0.0, which were removed. thus, by using only the clearly negative and positive days, the returns were binarized. through such filtering, the number of samples for each stock became about two-thirds of the number of all trading days, or around foot_3 2600 and 1200 samples for each stock, for the wsj and r&b datasets, respectively. ## effect of stock embedding on price movement classification the basic effect of the stock embedding was evaluated through the performance on the price movement classification task, as stated in section 3. the whole dataset described in section 5.1 was randomly divided into nonoverlapping training/validation/test sets in the ratios of 0.6/0.2/0.2. the training/validation/test parts did not share any samples from the same dates. every method below was tested for 10 different random divisions, and the average performance is reported here. the proposed model is abbreviated as wa+cs+dvr, for weighted average with classifier sharing and dual-vector representation. for an ablation test, four models were considered, which varied the market vector of the day (defined in formula (3) in section 4 @xcite ) and were with or without the dual-vector representation and classifier sharing @xcite @xcite , as follows. simple average: the simple average of the text representations of the same day is taken as the market vector of the day, as proposed by @xcite . weighted average (wa): as stated in formula furthermore, to examine the effect of the data size, we tested different dataset portions: 1 year, 3 years, and the whole dataset. therefore, the experimental variants involved five methods (four comparison + our proposal) and three data sizes, or a total of 15 experiments. figure 2 summarizes the complete experimental results. the uppermost bar of each bar group, in red, corresponds to our model with classifier sharing (cs) and the dual-vector representation (dvr). the other bars, in orange, blue, purple, and green, correspond to the four ablation variants. the ablation datasets with only 1-year data contained around 150 training samples and were too small for most variants to work well, yet our proposed model, wa+cs+dvr, could still obtain positive results (classification accuracy over 50%). with the 3-year datasets, our wa+cs+dvr model widened the performance gap, whereas the simple average and weighted average models still failed to work better than random guessing. these results show the superiority of our model in handling the overfitting problem with small datasets. finally, the significant differences between wa+cs+dvr (in red) and wa+cs (in blue) and between wa+dvr (in orange) and wa (in purple) strongly supported the advantage of adopting the dual-vector representation (dvr), especially when classifier sharing was combined. overall, our model successfully achieved 68.8% accuracy for the r&b dataset, which was significantly better than any of the other four variants. ## portfolio optimization thus far, the evaluation on classification has shown the capability of our framework in understanding news articles. for financial applications, however, the task must be in the form of prediction; that is, it must produce some gain ahead of the time when a news article is published. as one such predictive example, we present portfolio optimization, one of the most important financial tasks, and we show how our stock embedding can be applied to it. a portfolio is essentially a set of weights assigned to stocks, representing the proportions of capital invested in them. intuitively, a portfolio bears a bigger risk if a large proportion is invested in two highly positively correlated stocks, rather than two uncorrelated or negatively correlated stocks. based on this idea, the mean-variance minimization model in @xcite is formulated as follows: in the original model of markowitz, σ is the covariance matrix of the historic return time series of stocks, σ @xmath7, {r j } t ) (i, @xmath8, ..., j}). according to @xcite , the solution of this optimization problem, which can be obtained via quadratic programming, gives the portfolio with the smallest risk for an expected overall return e. using the covariance matrix as the risk matrix σ is limited, however, for two reasons. first, the overwhelming noise in price movements prevents accurate estimation of the covariance. more importantly, it ignores the events described in news articles that indeed cause price movements. on the other hand, the stock embeddings built here provide much abundant textual information for defining σ. concretely, σ i,@xmath9, s j ). this should work because the stock embedding reflects a stock's responsiveness to a certain class of news events. in other words, close stock embeddings indicate a correlated response pattern to an event described in news articles. stock embeddings capture this correlation much better than the covariance matrix does, and this correlation is what a good portfolio relies on. by solving the same optimization problem but with a different matrix σ, we get another vector of investment ratios, w, with respect to the stocks. by virtually investing according to w and observing the result within a certain period, σ can be evaluated. for each of the wsj and r&b datasets, we ran one investment simulation for various definitions of σ, as follows. s&p 500 index: as a market baseline, we used an s&p 500 index portfolio, in which all 505 stocks in the index were considered and the investment weight w j was in proportion to the market capitalization of stock j. the price history of the portfolio was provided by dow jones. this method did not use σ to form the portfolio. s&p 89*/50*: this approach was the same as above but with the set of stocks reduced to those tested in our work, as explained in section 5.1: 89 stocks for the wsj dataset foot_6 , and 50 for the r&b dataset. covariance matrix of historic stock returns: σ was the covariance matrix as originally proposed by markowitz. word2vec-general: (text only) σ was the cosine matrix of the word embeddings trained on general corpora (fasttext word embeddings @xcite were used in our experiments). for each stock, we used the word embedding of its ticker symbol, e.g., the word embedding of "aapl" for apple inc. word2vec-news: (text only) σ was the cosine matrix of the word embedding vectors trained on news text corpora. we used the full text of the r&b dataset for training, in which all mentions of a stock in the text were replaced by the stock's ticker symbol. covariance • stock embedding: (text and price) σ was the result of element-wise multiplication of the covariance matrix and the cosine matrix of the stock embeddings. weighted bert: (text only) σ was the cosine matrix of stock vectors acquired as follows, where the bert-based text representation n v i was used. for a stock j, the vector was obtained as a weighted average of n v i for which the text mentioned the stock or company. here, the weight of article i was defined as follows: stock embeddings: σ was the cosine matrix of the stock embeddings. the portfolio evaluation was conducted in a yearly setting, as illustrated in figure 3 . at the beginning of each year, given some expected gain e, the portfolio was computed by using all news articles and historic prices until the end of the previous year. in other words, for each year, the training set in the experiment consisted of samples strictly earlier than those constituting the test set. therefore, the evaluation was conducted in a prediction setting. then, investments were made according to the yearly renewed portfolio as in figure 3 ; that is, capital was allocated to stocks according to w. the realized annual gain of the portfolio followed this equation: in this way, for each of the wsj and r&b, we obtained results over 16 and 7 years, respectively. for different expected gains @xmath10.05, 0.06, ..., 0.29}, which cover typical cases in real-world portfolio construction, the average annual gain was computed. figure 4 shows the experimental results. the upper graphs show the annual gain with respect to different values of e (horizontal axes) for (a) the wsj and (b) the r&b, averaged over years. every curve corresponds to a different definition of σ. it can be seen that the proposed stock embedding method outperformed the other methods, except for larger e with wsj 8 . especially for the r&b dataset, stock embedding greatly outperformed all other methods at all e. the lower bar graph summarizes the overall aggregate gain for each method. the values in the bars indicate the average realized annual gains, while those above the bars are the ratios of the gains in comparison with that of the standard covariance method (in blue). the leftmost two bars in each bar graph show the gains of the s&p 500 portfolio and the s&p 89*/50* portfolio, respectively. as described above, the s&p 500 portfolio consisted of an average of around 500 stocks traded in the us, while the s&p 89*/50* portfolio, which was calculated with the same method but on a smaller set of stocks (89 for the wsj, and 50 for the r&b), achieved higher gains than its s&p 500 sibling did. the values of the s&p portfolios generally went up during the periods of both datasets, and therefore, the gains were positive. the dashed horizontal line in each bar graph indicates the result for the standard covariance method as a baseline. its gains were only 12.5% and 12.7% for the wsj and r&b, respectively, but with stock embeddings, the gains increased to 17.2% and 35.5%, or 1.37 and 2.80 times greater than the baseline results, respectively. this per8 our method did not perform well only for large e. the mean-variance minimization model has been reported to become unstable under the two conditions of large e and low overall market gain @xcite . the return of the wsj period @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite was lower than that of the r&b period @xcite @xcite @xcite @xcite , and therefore, these two conditions were more likely to be met for wsj. formance largely beat all other variants and gives evidence of how well the stock embeddings integrated both price and textual information. the results for the method that integrated the covariance matrix and stock embedding (in green) did not much outperform the baselines. a possible reason is that the stock embedding had already integrated the price information. as for the other variants based on pure text (in purple, orange, and brown), the results improved slightly. among them, weighted bert outperformed the other methods for both datasets. this indicates the potential of bert and other recent neural language models for portfolio optimization. ## conclusion this paper has proposed the idea of a stock embedding, a vector representation of a stock in a financial market. a method was formulated to acquire such vectors from stock price history and news articles by using a neural network framework. in the framework, the stock embedding detects news articles that are related to the stock, which is the essence of the proposed method. we trained stock embeddings for the task of binary classification of stock price movements on two different datasets, the wsj and r&b. the improvements in classification accuracy with our framework, due to the clas-sifier sharing and dual-vector text representation proposed in this paper, implied that the stock embeddings successfully incorporated market knowledge from both the news articles and price history. because the stock embedding is a vector that can be separated from the other components of the classification model, it can be applied to other tasks besides price movement classification. as an example, we showed the use of stock embeddings in a portfolio optimization task by replacing the risk matrix in the portfolio objective function with a cosine matrix of stock embeddings. in investment simulations on the r&b dataset, our stock embedding method generated 2.80 times the annual return obtained using the covariance matrix of the historic return series. this significant gain suggests further potential of our stock embedding for modeling the correlation among stocks in a financial market, and for further applications, such as risk control and asset pricing.