Predict the Future from the Past? On the Temporal Data Distribution Shift in Financial Sentiment Classifications (2023)

Abstract: Temporal data distribution shift is prevalent in the financial text. How can a financial sentiment analysis system be trained in a volatile market environment that can accurately infer sentiment and be robust to temporal data distribution shifts? In this paper, we conduct an empirical study on the financial sentiment analysis system under temporal data distribution shifts using a real-world financial social media dataset that spans three years. We find that the fine-tuned models suffer from general performance degradation in the presence of temporal distribution shifts. Furthermore, motivated by the unique temporal nature of the financial text, we propose a novel method that combines out-of-distribution detection with time series modeling for temporal financial sentiment analysis. Experimental results show that the proposed method enhances the model’s capability to adapt to evolving temporal shifts in a volatile financial market.

## introduction natural language processing (nlp) techniques have been widely adopted in financial applications, such as financial sentiment analysis, to facilitate investment decision-making and risk management @xcite @xcite . however, the nonstationary financial market environment can bring about significant changes in the data distribution between model development and deployment, which can degrade the model's performance over time and, consequently, its practical value. for example, a regime shift in the stock market refers to a significant change in the underlying economic or financial conditions. a regime shift, which may be triggered by changes in interest rates or political events, can significantly affect the market behavior and investor sentiment @xcite . there has been limited research on the temporal dataset shift in the financial context. existing nlp works on financial sentiment analysis follow the conventional approach that randomly splits a dataset into training and testing so that there is no distribution shift between training and testing @xcite . however, in a real-world financial sentiment analysis system, there could be unpredictable distribution shifts between the data that is used to build the model (in-sample data) and the data that the model runs inference on (out-of-sample data). as a result, the practitioners often face a dilemma. if the model fits too well to the in-sample data, it may experience a drastic drop in the out-of-sample data if a distribution shift happens (such as a regime shift from a bull market to a bear market); if the model is built to minimize performance disruption, its performance may be unsatisfactory on the in-sample data as well as the out-of-sample data. in this paper, we raise our first research question rq1: how does temporal data shift affect the robustness of financial sentiment analysis? the question is not as trivial as it seems. for example, @xcite find that language models are robust to temporal shifts in healthcare prediction tasks. however, financial markets may exhibit even more drastic changes. to answer this question, we systematically assess several language models, from bert to gpt-3.5, with metrics that measure both the model capacity and robustness under temporal distribution shifts. in our monthly rollingbased empirical analysis, this dilemma between in-sample performance and out-of-sample performance is confirmed. we find that fine-tuning a pretrained language model (such as bert) fails to produce robust sentiment classification performance in the presence of temporal distribution shifts. moreover, we are interested in rq2: how to mitigate the performance degradation of financial sentiment analysis in the existence of temporal dis-tribution shift? motivated by the unique temporal nature of financial text data, we propose a novel method that combines out-of-distribution (ood) detection with autoregressive (ar) time series modeling. experiments show that ood detection can effectively identify the samples causing the model performance degradation (we refer to those samples as ood data). furthermore, the model performance on the ood data is improved by an autoregressive time series modeling on the historical model predictions. as a result, the model performance degradation from in-sample data to out-ofsample data is alleviated. this work makes two contributions to the literature. first, while sentiment analysis is a very well-studied problem in the financial context, the long-neglected problem is how to build a robust financial sentiment analysis model under the pervasive distribution shift. to our knowledge, this paper provides the first empirical evidence of the impact of temporal distribution shifts on financial sentiment analysis. second, we propose a novel approach to mitigate the out-of-sample performance degradation while maintaining in-sample sentiment analysis utility. we hope this study contributes to the continuing efforts to build a more robust and accountable financial nlp system. ## temporal distribution shift in financial sentiment analysis in this section, we first define the task of financial sentiment analysis on temporal data. we then introduce two metrics for model evaluation under the data distribution shift. ## experiment setup this section describes the evaluation setups on the dataset and models for temporal model analysis. ## empirical evaluation of temporal data shift in financial sentiment analysis our first research question aims to empirically examine if temporal data shift affects the robustness of financial sentiment analysis and to what extent. prior literature has studied temporal distribution shifts in the healthcare domain and finds that pretrained language models are robust in the presence of temporal distribution shifts for healthcarerelated prediction tasks such as hospital readmission @xcite . however, financial markets and the financial text temporal shift are much more volatile. we empirically answer this research question using the experiment setup discussed in section 3. ## mitigating model degradation under temporal data shift in the previous section, our analysis reveals a consistent performance degradation in financial sentiment classification models on out-of-sample data. in this section, we explore possible ways to mitigate the degradation. as our previous analysis shows that the performance degradation is correlated with the distribution shift, it is reasonable to infer that the performance degradation is caused by the failure of the out-of-distribution (ood) examples. this observation motivates us to propose a two-stage method to improve the model robustness under temporal data distribution shifts. firstly, we train an ood sample detector to detect whether an upcoming sample is out of distribution. if not, we still use the model trained on in-sample data on this sample. if yes, we propose using an autoregressive model from time series analysis to simulate the model prediction towards the future. ## conclusions in this paper, we empirically study the problem of distribution shift over time and its adverse impacts on financial sentiment classification models. we find a consistent yet significant performance degradation when applying a sentiment classification model trained using in-sample (past) data to the out-of-sample (future) data. the degradation is driven by the data distribution shift, which, unfortunately, is the nature of dynamic financial markets. to improve the model's robustness against the ubiquitous distribution shift over time, we propose a novel method that combines out-of-distribution detection with autoregressive (ar) time series modeling. our method is effective in alleviating the out-of-sample performance drop. given the importance of nlp in real-world financial applications and investment decision-making, there is an urgent need to understand the weaknesses, safety, and robustness of nlp systems. we raise awareness of this problem in the context of financial sentiment classification and conduct a temporal-based empirical analysis from a practical perspective. the awareness of the problem can help practitioners improve the robustness and accountability of their financial nlp systems and also calls for developing effective nlp systems that are robust to temporal data distribution shifts.