Measuring Consistency in Text-based Financial Forecasting Models (2023)

Abstract: Financial forecasting has been an important and active area of machine learning research, as even the most modest advantages in predictive accuracy can be parlayed into significant financial gains. Recent advances in natural language processing (NLP) bring the opportunity to leverage textual data, such as earnings reports of publicly traded companies, to predict the return rate for an asset. However, when dealing with such a sensitive task, the consistency of models – their invariance under meaning-preserving alternations in input – is a crucial property for building user trust. Despite this, current methods for financial forecasting do not take consistency into consideration. To address this issue, we propose FinTrust, an evaluation tool that assesses logical consistency in financial text. Using FinTrust, we show that the consistency of state-of-the-art NLP models for financial forecasting is poor. Our analysis of the performance degradation caused by meaning-preserving alternations suggests that current text-based methods are not suitable for robustly predicting market information.

## introduction nlp techniques have been used in various financial forecasting tasks, including stock return prediction, volatility forecasting, portfolio management, and more @xcite @xcite @xcite @xcite . despite the increased performance of nlp models on financial applications, there has been pushback questioning their trustworthiness, and robustness @xcite . recently, the causal explanation has been viewed as one of the promising directions for measuring the robustness and thus improving the transparency of models @xcite . among them, consistency has been viewed as a crucial feature, reflecting the systematic ability to generalize in semantically equivalent contexts and receiving increasing attention in tasks such as text classification and entailment @xcite . previous text-based financial forecasting methods have mostly considered stock movement prediction based on various sources of data, including financial news @xcite , analyst reports @xcite , and earnings conference calls @xcite @xcite . while most work evaluates their methods using accuracy and profit gains based on the final outcome in the market @xcite , consistency evaluation remains largely unexplored. the only exception @xcite focuses on evaluating the implicit preferences in pre-trained language models (plms) but not the consistency in predictive models. the lack of evaluation in behavior consistency, an important characteristic of human decisions, hinders the deployment of financial forecasting models in real-world scenarios. the main objective of this work is to explore a wholistic measure for stock movement prediction, integrating consistency as a criterion of trustworthiness. to this end, we define behavior consistency of text-based models in the financial domain. regarding the intrinsic characteristics of financial text data, we consider four types of logical consistency tests. as shown in figure 1 , these transformations include negation consistency, symmetric consistency, additive consistency, and transitive consistency. taking negation consistency as an example, given an input "the cost of raw materials has been greatly decreased", if the token "decreased" is changed to "increased", the model prediction is expected to be flipped accordingly. based on the above logical transformations, we introduce fintrust � � � � � � � � �, a new evaluation tool that enables researchers to measure consistency in plms and text-based financial forecasting models. using fintrust � � � � � � � � �, we design three tasks to investigate the influence of these logical transformations. first, we assess implicit preference in plms such as bert @xcite and finbert @xcite , especially for economic words. second, we measure the accuracy of stock movement prediction on a real-world earnings call dataset after the meaning-preserving modifications. finally, we propose a realistic trading simulation to see if simple meaning-preserving modifications can wipe out positive returns. experiments on several baseline models, including previous best-performing architectures @xcite @xcite and the machine learning classifier @xcite show that all current methods exhibit a significant decline in the performance of stock movement predictions when evaluating on fintrust compared to their original results. notably, some models demonstrate a level of accuracy that is even lower than that of a random guess after undergoing logical consistency transformation, and most methods fail to surpass the performance of the simplest buy-all strategy in the trading simulation. these results suggest that existing text-based financial models have robustness and trustworthiness issues, which can limit their use in practical settings. to our knowledge, fintrust � � � � � � � � � is the first evaluation tool for probing if the relatively accurate stock movement prediction is based on the right logical behavior. we release our tool and dataset at github † , which can assist future research in developing trustworthy finnlp methods. in addition to building pre-trained models specially trained for financial domains, researchers have recently proposed myriad neural network architectures aimed at more accurate predictions to produce profitable gains including financial risk (volatility) and return predictions. for example, researchers @xcite @xcite have considered predicting the volatility of publicly traded companies based on multi-model earnings conference call datasets. also, @xcite cohen (2018); duan et al. (2018); yang et al. (2018); feng et al. (2019) leverage different textual data sources for predicting the stock movement based on the daily closing price. unfortunately, despite the alarm over the reliance of machine learning systems on spurious patterns that have been found in many classical nlp tasks, the topic of text-based financial forecasting lacks a systematical evaluation regarding the robustness analysis from either an adversarial or consistency perspective. to this end, we present the first critical investigation of popular benchmarks by using fintrust from the consistency perspective. consistency measurement. the inductive bias of machine learning systems is greatly affected by the patterns in training data due to the nature of inductive reasoning. while a flurry of research has highlighted this issue @xcite @xcite shows that possible artefacts in data are more influential than the model design when leading to the problem of lacking trustworthiness. thus, assessing the influence of data artefacts, such as consistency, becomes a crucial problem for trustworthy nlp. @xcite study the consistency of plms (e.g., bert, albert, and roberta) with regard to their knowledge extraction ability and conclude that the consistency of these models is generally low. @xcite aim to raise awareness of potential implicit stock preferences based on the finding that consistent implicit preference of the stock market exists in plms at the whole market. in addition to evaluating preferences in plms, previous methods also attempt to evaluate the consistency of models in downstream nlp tasks, such as visual question answering (ribeiro et al., 2018), qa @xcite @xcite , named entity recognition @xcite , and natural language inference @xcite @xcite . besides, ribeiro et al. (2020) consider using consistency for building the behavioural testing benchmark beyond accuracy. surprisingly, these discussions have not yet been extended to text-based financial forecasting models, which require strong robustness to assist decision-making in the financial market, with the exception of our work. ## method we define the pipeline of fintrust � � � � � � � � � in figure 2 . for text-based financial models, there are two salient components, namely text representations and financial behavior. for the former, using plms has become a dominant approach, improving the quality of text representations in many domains. for the latter, various neural models can be built on plms. correspondingly, we have two setups in the consistency evaluation, representation (setup 1) and behavior (setup 2), respectively. setup 1. in the first stage, we assess the implicit preferences in plms via masked token predictions. in particular, we first mask a predictable word from the original input extracted from earning conference call transcripts, such as "the cost of raw materials has been greatly decreased...so the expected return for the next quarter is [mask]". then, we predict the masked token using plms and compare the probability of predicting "increased" and "decreased" for contexts from different transcripts. a higher probability of predicting "increased" would indicate that the given plm hold logical consistency with human predictions. conversely, it suggests that the prediction of the plm may be influenced by spurious patterns such as favoritism towards a particular stock. setup 2. we evaluate text-based financial forecasting models after fine-tuning plms on a popularly used earnings conference call dataset @xcite . however, the consistency measurement faces significant challenges in defining the relationship between two texts, particularly when the text is a long transcript with complex logical connections, such as earnings conference call transcripts. incorrectly defining this relationship can render consistency judgments meaningless. in line with prior research @xcite , we develop four logical consistency transformations customized for financial text in this work. by meaning-preserving altering the original text, we ensure generated samples have a logical relationship to the original text, thus ensuring the consistency judgment is meaningful. below we define our text level consistency transformation first (sec 3.1), before introducing the financial tasks for behavior study (sec 3.2) and a wholistic metric (sec 3.3) to integrate performance and trustworthiness. ## experiments we first evaluate the explicit preferences in plms. then we assess the ability of text-based models to make consistent predictions on the stock movement and finally test the profitability of these predictions using a trading simulation. ## results and discussion we report the results of three tasks defined in section 3.2 and the consistency score calculated by the consistency evaluation metrics in this section. furthermore, we present extensive ablation studies and discussions to support in-depth analyses of each component in fintrust. ## conclusion we proposed fintrust, an evaluation tool that assesses the trustworthiness of financial forecasting models in addition to their accuracy. results on fintrust show that (1) the consistency of state-ofthe-art models falls significantly short of expectations when applied to stock movement prediction;