Quantitative Day Trading from Natural Language using Reinforcement Learning (2021)

Abstract: It is challenging to design profitable and practical trading strategies, as stock price movements are highly stochastic, and the market is heavily influenced by chaotic data across sources like news and social media. Existing NLP approaches largely treat stock prediction as a classification or regression problem and are not optimized to make profitable investment decisions. Further, they do not model the temporal dynamics of large volumes of diversely influential text to which the market responds quickly. Building on these shortcomings, we propose a deep reinforcement learning approach that makes time-aware decisions to trade stocks while optimizing profit using textual data. Our method outperforms state-of-the-art in terms of risk-adjusted returns in trading simulations on two benchmarks: Tweets (English) and financial news (Chinese) pertaining to two major indexes and four global stock markets. Through extensive experiments and studies, we build the case for our method as a tool for quantitative trading.

## introduction the stock market, a financial ecosystem involving quantitative trading and investing, observed a market capitalization exceeding $us 60 trillion as of the year 2019. stock trading presents lucrative opportunities for investors to utilize the market as a platform for investing funds and maximizing profits. however, making profitable investment decisions is challenging due to the market's volatile, noisy, and chaotic nature @xcite . research at the intersection of natural language processing (nlp) and finance presents encouraging prospects in stock prediction @xcite . conventional work forecasts future trends by modeling numerical historical stock data (lu *equal contribution. figure 1 : here, we show how tweets about tesla and moderna influence investors' opinions and impact the stocks over a day and the upcoming week. the tweets by the tesla ceo elon musk lead to massive price drops in tesla's stock, and moderna's positive news attracts investments in its stock. a profitable trading decision would entail selling off tesla's shares (if already held) and buying moderna's stock in such a scenario. @xcite . however, price signals alone can not capture market surprises, mergers, acquisitions, and company announcements. such events, often reported across financial news and social media, have strong influence over market dynamics @xcite . for instance, prices immediately react to breaking news about the related company @xcite . such reactions conform to the efficient market hypothesis (emh), a hypothesis in finance which states that financial markets are informationally efficient and prices reflect all available market information at any given time @xcite . the abundance of stock affecting information across news and social media online inspires the adoption of natural language processing to study the interplay between textual data and stock prices @xcite . however, unlike numerical data, the study of natural language is more challenging. individual tweets or news headlines may not be informative enough, and analyzing them together can provide a greater context, as shown in figure 1 . moreover, the timing of their release plays a critical role as stock markets rapidly react to new information @xcite . furthermore, not each news story or tweet holds the potential to influence stock trends as texts have a diverse influence on prices @xcite . these observations suggest benefits in factoring in the time-aware dependence and diverse influence of text while analyzing natural language. despite profitability being the prime objective of quantitative trading, existing natural language processing methods for stock prediction @xcite @xcite ) are commonly formulated as classification or regression tasks, and are not directly optimized towards profit generation. such methods face fundamental drawbacks. first, they do not innately incorporate the decision making and strategies involved in quantitative trading, in turn limiting potential profitability. second, they have limited practical applicability as they do not factor in the monetary resources available and financial assets (stocks) held with a trader at each trading time step. this gap presents a new research direction where profit generation can be directly optimized by modeling the complex sequential decision-making process in quantitative trading as a reinforcement learning (rl) task. owing to its nature, rl formulation is directly suitable to the problem of quantitative trading as it provides the potential to automatically learn the adjustment of investment budgets across stocks in portfolios while taking into account the configuration of investments made in the past. ## problem description we formulate stock trading as a reinforcement learning problem. let @xmath0, s 2 , . . . , s n } denote a set of n stocks. we aim to design a trading agent that learns to interact with the stock market environment by leveraging stock-affecting signals present across financial news items and tweets to trade stocks. in the context of an agent, an interaction comprises observing the environment state at any particular time-step to generate an action, and reach the next time-step to receive a reward along with the next state. the typical markov decision process (mdp) description is widely adopted for rl tasks where environments are fully-observable. however, in the stock market, prices are influenced by numerous macro-and micro-economic factors, investor opinions about stocks formed through social media, financial news, and countless other sources. thus, it becomes pragmatically and computationally impractical to observe and incorporate stock affecting information from all possible sources to make trading decisions. as the stock markets and the underlying factors that drive stock prices are not fully-observable, partially observable mdp (pomdp) provides a natural generalization of the mdp to model the stock trading environment @xcite . hence, the key components of the stock trading environment considered and developed in this study are as follows: the text input in o m is structured such that it comprises all stock relevant text in a lookback window of length t in a hierarchical fashion within and across days. the orders made through the trading actions taken by the reinforcement learning agent would have minute impacts on the overall market trends, thus having little to no direct influence on the market-information observations. trading actions: the agent can buy, sell, or hold the shares for each stock at the time-step τ . we compute a vector of actions a τ over the set of stocks s as decisions made by the agent, which result in an increase, decrease, or no change in the number of stocks shares h. one of three possible actions is taken on each stock s: note that the trading actions at time-step τ directly impact the trading-account observation at time-step τ + 1, o τ +1 . rewards: we define the reward as the change in the value when an action is taken at state s τ to arrive at new state s τ +1 . corresponding to each state change, we define a return r, as: ## proposed approach: profit we adopt reinforcement learning to optimize profitability in quantitative trading. to this end, we introduce profit, a deep reinforcement learning approach for text-based stock trading, as shown in figure 2 . for this study, we make use of a custom policy network that hierarchically and attentively learns time-aware representations of news and tweets to trade stocks. in practice, profit's proposed policy network is generalizable across various actor-critic reinforcement learning methods that exploit neural networks as function approximators. moreover, profit is compatible with any custom policy network of the same nature that can handle textual time-series data. ## conclusion we propose profit, a deep rl approach for quantitative trading using textual data across online news and tweets. to model the market information, profit hierarchically learns temporally relevant signals from texts in a time-aware fashion, and directly optimizes trading actions towards profit generation. through extensive analyses on english tweets and chinese news spanning four markets, we highlight profit's real-world applicability. in trading simulations on the s&p 500 and china ashares indexes, profit outperforms baselines in terms of profitability and risk in investment.