Stock Price Volatility Prediction: A Case Study with A uto ML (2022)

Abstract: Accurate prediction of the stock price volatility, the rate at which the price of a stock increases or decreases over a particular period, is an important problem in finance. Inaccurate prediction of stock price volatility might lead to investment risk and financial loss, while accurate prediction might generate significant returns for investors. Several studies investigated stock price volatility prediction in a regression task by using the transcripts of earning calls (quarterly conference calls held by public companies) with Natural Language Processing (NLP) techniques. Existing studies use the entire transcript and this degrades the performance due to noise caused by irrelevant information that might not have a significant impact on stock price volatility. In order to overcome these limitations, by considering stock price volatility prediction as a classification task, we explore several denoising approaches, ranging from general-purpose approaches to techniques specific to finance to remove the noise, and leverage AutoML systems that enable auto-exploration of a wide variety of models. Our preliminary findings indicate that domain-specific denoising approaches provide better results than general-purpose approaches, moreover AutoML systems provide promising results.

## introduction predicting stock price volatility is of great interest to researchers and seems to remain one of the interesting open problems. volatility is about information disclosure, and how unexpected the information is to the market; therefore, volatility persists in the market until the future values of stock reflect the information provided. according to @xcite , markets are informationally efficient if prices at each moment incorporate all available information about future values. such that if there is an information disclosure, not yet incorporated in market prices, the future values will be volatile until the price fully reflects the disclosed information. @xcite any information disclosed to the market by competitors, suppliers, customers, and regulators creates volatility, in addition to the internal information the company voluntarily discloses. every quarter the executive leadership of a public company holds an earnings call meeting with investors and analysts to inform them about the company's status. as executives inform the investors about the company's current status and future outlook, earnings conference calls may result in stock price volatility. in finance and accounting research, the high volatility following an earnings conference call is conceptualized as post-earnings announcement drift (pead) @xcite , which refers to the drift of a company's stock price for an extended period. stock prices tend to drift upward (or downward) when the announcements are above (or below) expectations following an earnings conference call. depending on how unexpected the information shared during earnings conference calls is, stock prices and firm valuations change, and markets become more 'informationally efficient' by absorbing this information over the long run @xcite . in this work, we study the problem of leveraging the textual transcripts of companies' earnings calls and building natural language processing (nlp) models to predict the volatility of their stock prices for a period of time following the earnings calls. while this problem has been studied in the literature, prior works exhibit these limitations: in this work, we address the aforementioned challenges as follows: to improve the signal coming from the transcripts, we propose and experiment with an entire spectrum of denoising approaches, ranging from domain-agnostic denoising techniques to domain-specific approaches that utilize domain knowledge to improve the denoising process further. our experimental evaluation shows that domain-specific denoising approaches outperform domain-agnostic techniques, which points to the importance of incorporating domain knowledge into the denoising process. the rest of this paper is structured as follows: we start by reviewing related work in section 2. in section 3, we describe the problem definition and data preparation. we propose a range of denoising approaches in section 4 and explain how we discover appropriate nlp models by leveraging an automl system in section 5. finally, we present the experimental evaluation results and associated insights in section 6 and conclude the paper in section 7. ## related work information is one of the most valuable and highly sought assets in financial markets @xcite @xcite and is found to be impacting stock price volatility in several studies. moreover, as posited by the mixture of distributions hypothesis, the sequential arrival of new information generates trading volume and price movements @xcite @xcite (i.e. information shocks). briefly stated, the impact of information disclosure on the volatility of stock prices has been investigated from several angles in the literature. four types of textual data have been mainly used for stock volatility prediction: annual statements, news, social media data, and earnings calls transcripts. annual statements (10-k reports): annual statements include historical data about a company's financial performance and a future outlook that can be valuable in predicting the volatility of its stock. for instance, @xcite formulate a "text regression problem", where information from the 10-k reports is used to predict the volatility of stock returns in the periods following the reports. @xcite ) financial lexicon generated from fourteen years of historical annual statements (10-k reports) is one of the major and initial attempts that utilize language resources to predict stock price volatility. news data: news data often provide important information about events related to a company. for instance, tetlock (2007) uses daily content from the wall street journal to predict volatility. in a similar vein, @xcite adapt open ie technology for event-based stock price movement prediction by extracting structured events from large-scale public news. social media data: social media data often capture public sentiment about a company that can be an important indicator for the future price of its stock. @xcite use behavioral economics to investigate how societal moods affect collective decision-making for dow jones industrial average (djia)'s values. earnings calls transcripts: several works have found that earnings calls (as captured through their transcripts) can be predictive of investor sentiment for stock price volatility prediction tasks @xcite @xcite . recent studies also combined the textual transcripts with additional verbal and vocal cues from audio recordings of earnings call events and leveraged multi-modal learning to predict stock price volatility @xcite . most of the existing research models the stock price volatility prediction as a regression task, with the exception of keith and stent's work @xcite , whichsimilar to our work -models it as a classification task. however, they use classification to predict the analysts' recommendations to buy/sell/hold a stock. in contrast, we predict the market reaction itself by predicting the actual stock price volatility (classified as low, medium, and high volatility). our work makes multiple novel contributions: first, we consider stock price volatility prediction as a text classification task different from existing work @xcite . second, instead of using the earnings call transcripts as-is, we employ denoising techniques designed to separate the signal from the noise caused by irrelevant information in the transcripts and improve the performance of the resulting nlp models. we design and test several denoising approaches and report the results of their effectiveness. third, automl systems have been used in several text classification tasks @xcite @xcite , however, there is little effort in the literature to use automl systems for the stock price volatility prediction task. using automl systems to predict stock price volatility enables non-ai expert users (who may not be proficient in ai/nlp techniques) to create nlp models for the stock volatility prediction task quickly. ## problem definition & data preparation in this paper, we aim to predict the magnitude of volatility from earnings call transcripts and formulate this as a classification problem. in line with that purpose, we combine earnings call transcripts (text data) with their corresponding volatility labels (financial data). ## denoising approaches earnings call transcripts are typically long documents containing a lot of information. while some of this information is valuable for predicting stock price volatility, another part of it can be irrelevant for the stock price volatility prediction task and can thus introduce unwanted noise. to address this problem, we experiment with several approaches of denoising the transcripts as a pre-processing step. we propose a spectrum of approaches, ranging from generic domain-agnostic approaches that are used in different tasks to more domain and taskspecific approaches related to finance. we start by using raw earnings call transcripts without further processing (which we use as our baseline). in the second approach, we use a general domain-agnostic denoising approach by leveraging the t5 summarization model @xcite to create a summary of the earnings call transcript. in the third approach, we experiment with a more domain and task-specific approach by borrowing a finance domain-specific dictionary @xcite , which we use to identify the sentences with important information. in the fourth approach, we create an intermediate domain-specific nlp model to identify the sentences containing important information that has the potential of affecting the stock price volatility. ## building classification models through autoai for text building models for nlp tasks, such as the stock price volatility classification task considered in this work, requires significant technical expertise, effort, and resources. to lower the barrier of entry and accelerate the model development process, the research and industrial community have developed automl/autoai techniques to automate parts of this process @xcite @xcite . multiple automl techniques suggested in the literature target different parts of the model development process. these include neural architecture search @xcite , hyperparameter optimization (weidele et al., 2020), and others. while previous works on stock volatility prediction using textual data leverage a small set of hand-picked nlp models, we explore automl techniques to select the best nlp model for the stock price volatility prediction task in this work. the goal behind this choice is twofold: first, we want to investigate how domain experts (in our case, financial analysts) can create nlp models for their tasks. second, we want to explore multiple nlp model architectures and gain insights into which model architectures work best for the stock volatility prediction task. we feed the denoised earnings calls transcripts and their corresponding labels into autoai for text @xcite . autoai for text is a comprehensive end-to-end automl system for text classification tasks, which given a labeled text classification dataset explores a large search space of models for the provided dataset. during this search, autoai for text explores multiple featurizers (such as glove, tfidf, etc.), estimators/transformers (such as svc, cnn, lstm, etc.), and hyperparameters. the result of this optimization process is a set of nlp models for the given dataset (referred to as pipelines), ranked based on a chosen optimization metric (such as accuracy, precision, recall, f1, etc.). as we explain when describing the experimental evaluation in section 6, autoai for 6 experimental evaluation experimental setting. for each earnings call transcript, we compute its volatility label for four different time periods, corresponding to 3, 7, 15, and 30 days following the earnings call. through this process we obtain four different sets of labels for our earnings call transcripts (one per time period). in parallel, we run each transcript through the four denoising approaches outlined in section 4. this leads to four sets of documents (one per denoising approach). for each (time period, denoising approach) pair, we combine the denoised transcript with the corresponding volatility label to generate a labeled dataset corresponding to the given time period and denoising approach. this labeled dataset is then fed into autoai for text, which is tasked with discovering the best nlp model for the given pair. each labeled dataset is split into a 90% combined train and validation split and 10% test split. this split is done sequentially based on the timestamp of the earnings calls to ensure that we do not include in the train split any information about the time periods included in the test split (i.e., we want to avoid giving the model at training time information about the future). the combined train and test split is then further split by autoai for text into train and validation utilizing another 90/10 split. in addition to the train/validation split ratio, we use the following configuration for autoai for text: we assign an optimization time budget of 2 hours (i.e., instructing it to use up to 2 hours for optimization purposes) and ask it to explore and train at most 81 candidate models. we also select accuracy as the metric used both internally for optimization purposes and externally to report model performance. finally, we instruct autoai for text to explore a variety of estimators/transformers, which include svc, cnn, lstm, and bert, and a variety of featurizers, which include glove and tfidf 4 . for each (time period, denoising approach) pair, we select the model that autoai for text has identified as having the highest accuracy on the validation set (which we refer to as the best model). the best model is then evaluated on the test set, computing its accuracy, which is the metric that we present in our evaluation results. baseline. existing works on stock price volatility prediction from earnings call transcripts model the problem as a regression problem, however, as a baseline we use a simple approach that assigns to all transcripts the same label. for each (time period, denoising approach) pair, we report three versions of this baseline, depending on which is the common label assigned to all transcripts: l (low), m (medium), or h (high). for instance, in the l-baseline, all transcripts are predicted as being low volatility. while this is an admittedly simple baseline, it still allows us to understand whether the discovered models have identified a signal in the data or have simply learned to predict the most common label. results. the accuracy of the best model discovered by autoai for text for each (time period, denoising approach) pair is shown in figures 2 and 3 . we next present and discuss the results by focusing on a few main questions that we hope to answer from this experimental evaluation. are the models able to identify a signal in the data? the first question we hope to answer is whether the discovered models have identified a signal in the input transcripts that allows them to predict the stock price volatility or whether they have simply learned to predict the most common label. this is a non-trivial question, as transcripts are long documents that in addition to information that may affect the stock price often contain a lot of irrelevant information. to answer this question, the results). as we can see, for all time periods, the former is always higher than the latter. thus the best models seem to have successfully identified a signal in the transcripts that allows them to perform better than the baseline. for instance, the best model for the 3-day time period has an accuracy of 0.52, which is higher than the best baseline accuracy of 0.43 (which corresponds to predicting for every input transcript the most common label, which in this case is the high volatility). the gap between the best model and the baseline closes as the time period increases. while this is an interesting phenomenon that needs to be investigated further, a potential explanation is that transcripts may be more useful in predicting volatility for time periods immediately following the earnings calls, rather than for longer time periods 5 . which denoising approaches perform best? the next question is identifying the best denoising approach for the studied problem. which of the proposed denoising approaches should one choose for predicting stock price volatility and does the choice of the approach make a difference? comparing the performance of the denoising approaches provides some interesting insights: first, utilizing the full document (without any denoising) always yields the lowest performance. this shows that denoising approaches are important for distilling the long transcripts and making them more amenable for being used as training data for an nlp model. second, domain-agnostic denoising (such as the one provided by the t5 summarization approach) consistently underperforms domain-specific denoising approaches (such as the use of the domainspecific dictionary or the intermediate model). this shows that further applying domain knowledge to distill input documents can improve model performance. finally, while the best denoising approaches are the two domain-specific approaches, we observe that using the domain-specific dictionary of loughran -mcdonald is better for shorter time periods (i.e., time periods of 3 and 7 days), while using the intermediate denoising model based on figure 3 : accuracy of best model discovered by autoai for text (together with the architecture of the model) for different denoising approaches and time periods. the baseline shows the accuracy that would be obtained if we assigned to all transcripts the same label of l: low, m: medium, or h: high volatility. which model architectures perform best? finally, leveraging autoai for text, we want to identify which model architectures perform best for the stock volatility prediction task. to aid in answering this question, figure 3 includes the description of the best model discovered by autoai for text. each model is shown as f + e, where f is the featurizer and e is the estimator/transformer. for instance, glove+cnn is a model combining a glove featurizer with a convolutional neural network. as described above, in our experiments autoai for text explored the glove and tfidf featurizers. similarly it searched among the following estimators/transformers: svc, cnn, lstm, and bert. by comparing the models reported in figure 3 , we can make the following observations: in all cases the models that perform best are based either on svc and cnn combined with either glove or tfidf featurizers. we cannot observe any systematic difference between svc and cnn, leading us to believe that both work equally well for the studied problem. however, an important observation is that lstm and bert never appear among the best models. 6 however, both this as well as the performance of lstm in this case our important results that we think are worth investigating further. 6 all the experiments were done on a cpu-only machine. as such, we instructed autoai for text to explore only cpufriendly types of models. these are types of models that can be trained fast with cpu-only resources and include classical ones like svc as well as faster deep-learning based models (cnn, lstm). we left out bert from the exploration space, since bert works better when given gpu resources. ## conclusion and future work compared to existing work in the area, our work makes three main contributions: first, it models the problem as a text classification task (in contrast to the regression task considered before) and explores how one can leverage text classification models. second, instead of just utilizing the long earnings call transcripts as-is, it explores the use of denoising approaches to distill the information found in the input documents and improve the performance of the learned models. we propose and explore an entire spectrum of denoising approaches, ranging from domain-agnostic techniques (such as generalpurpose summarization models) to domain-specific techniques and compare their performance. third, we leverage automl approaches to explore a range of nlp models and understand which model architectures perform best for the stock price volatility prediction task. our preliminary findings lead to several important insights. denoising is shown to improve model performance with domain-specific denoising leading to bigger gains than domain-agnostic denoising approaches. moreover, the use of automl leads to interesting insights on which model architectures perform best for the stock volatility task. we believe that these insights point to new interesting research directions both in developing better domainspecific denoising approaches, as well as further investigating which model architectures work best for long financial documents, which are some of the directions we plan to further explore in our future work.