Financial Sentiment Analysis: An Investigation into Common Mistakes and Silver Bullets (2020)

Abstract: The recent dominance of machine learning-based natural language processing methods has fostered the culture of overemphasizing model accuracies rather than studying the reasons behind their errors. Interpretability, however, is a critical requirement for many downstream AI and NLP applications, e.g., in finance, healthcare, and autonomous driving. This study, instead of proposing any “new model”, investigates the error patterns of some widely acknowledged sentiment analysis methods in the finance domain. We discover that (1) those methods belonging to the same clusters are prone to similar error patterns, and (2) there are six types of linguistic features that are pervasive in the common errors. These findings provide important clues and practical considerations for improving sentiment analysis models for financial applications.

## introduction natural language processing has been widely used for financial applications in recent years. these applications include stock market / foreign exchange market prediction, volatility modeling, asset allocation, business taxonomy construction, credit scoring, initial public offering valuation (ipo), and more @xcite @xcite @xcite . among them, there are mainly two threads of method to process textual inputs: the first is to directly encode financial texts by neural nets and to use the representations for learning downstream tasks @xcite ; the second is to analyze financial texts with critical linguistic features such as content semantics @xcite or investors' sentiment @xcite , for the sake of interpretability. the objective of financial sentiment analysis (fsa) is to classify a piece of financial text as expressing bullish or bearish opinions toward certain arguments. although sentiment analysis in the general domain is extensively studied in the past decades @xcite , fsa is a challenging task because of the lack of large-scale training data and the difficulty in labeling after acquiring the texts, which involves expert knowledge. as a result, the model performance for fsa are usually significantly worse than using the same sentiment analysis model for the general domain (which is referred to as a problem of domain adaptation). in addition to the above-mentioned challenges, fsa requires more interpretability comparing to sentiment analysis in other domains @xcite . early sentiment analysis studies leveraged on textual data from movie reviews, product reviews, and social media posts. in these applications, the purpose being roughly understanding customer feedbacks, statistically aggregating straightforward opinions suffices and a single mistake does not make much difference. whereas for financial applications, a fraction of bad sentiment analysis results may cause extreme loss, hence have to be carefully treated with exception. therefore, the goal of fsa is more than obtaining a high accuracy number: understanding when and why the method would fail is equally important. in this paper, we explore the fsa behavior of some of the most common sentiment analysis models and the interpretability problem by using error visualization and linguistic analysis. unlike many of the domain adaptation-based fsa effort where only distant supervision is available @xcite , we resort to self-labeling. noticing that some financial social platforms allow users to simultaneously label their post as bullish (positive) or bearish (negative), such platforms make it possible for us to collect financial texts with high-quality sentiment labels in a "crowd-sourced" manner. we also compare performance of the same models on both the finance domain (stocksen) and the general business review domain (yelp). we are specifically interested in the following research questions. rq1: do different sentiment analysis methods step into the same pitfalls (i.e., make the common errors) or each of them makes errors on different examples of financial texts? rq2: how reliable and consistent are the usually reported sentiment analysis metrics, i.e., f-score and accuracy measure, in evaluating methods across datasets from different language domains? rq3: is the performance deterioration for financial texts due to the same reasons as those found in other language domains? if so, the problem is expected to be significantly mitigated with common domain adapted methods. otherwise, we suspect that to understand financial sentiment, more resources (jargon, time, complex reference) are required than in other domains, such that domain adaptation alone cannot surmount the bottleneck. the major contributions of this paper are as follows: formally, each financial text t consists of k sentences s 1 , s 2 , ..., s k , where s i is a word sequence w i1 , w i2 , ..., w it . the fsa model outputs an average of binary sentence polarities: p(t ) = avg(p(s i )). ## experiments in this section, we provide further details on the datasets, experimental settings that produce the classification results, and evaluation metrics. ## discoveries and the answer to research questions we observe from bert has the highest scores on all metrics for the yelp dataset thanks to its strong expressive power. however, on the stocksen dataset, figure 2 shows the subtle phenomenon that the pairwise correlations are higher inside "model clusters". we observe the highest correlation from the left matrix between svm and fasttext, hence forms the machine learning-based cluster. similarly, the deep learning model cluster (bi-lstm, s-lstm, bert) has a darker color. opinionlex and l&m (@xmath0.40) are also highly correlated, though senticnet makes the lexicon-based cluster loose. this may attribute to the fact that senticnet leverages syntactic rules for sentiment classification. considering that all the models are trained on the same stocksen training data, the highest correlation (between svm and fasttext, @xmath1.56) is surprisingly low. figure 1 also confirms no clear pattern. for rq1, it seems that the errors are model-specific and even less consensus are on the bearish samples (the upper part is darker in the right matrix of figure 2 ). for rq2, though each metric roughly tell the model performance, inside clusters they may not rank models in the same way. 2 ). as an answer to rq3, simply using a domain adapted lexicon does not necessarily solve the fsa problem. ## reasons behind classification errors since this paper studies the reasons for classification errors, especially for fsa, we take a deeper look into 384 "overlapping errors" in sum, whereby 237 positive examples are correctly predicted by less than four models, and 147 negative examples are wrongly predicted by all eight models experimented. this error set contains some extremely difficult cases, such as "what goes up goes down and vice versa", which flips its polarity for many times; and "the downtrend is not because [...] is a bad investment", which express a positive sentiment with all negative words. although not all the errors from the stocksen dataset are explainable, the majority are explainable and we found the six interesting error types as follows. unlike the error analysis by @xcite , which generally focused on machine learning features and is specific to twitter data, our analysis emphasizes more on linguistic phenomena. one may ask whether these error types really afflict the sentiment analysis performance and are specific only to the finance domain. therefore, we also look into 237 positive and 147 negative examples randomly sampled from the "overlapping errors", and 384 random texts from the test population on the yelp dataset. indeed, we find more significant concentrations of these linguistic features in the errors from financial domain. ## conclusion in this paper, we evaluated the performance of some of the widely acknowledged lexicon-based, machine learning-based, and deep models for financial sentiment analysis. we went beyond simply comparing model metrics by visualizing the error patterns and conducting linguistic analysis. results confirm that models within the same cluster are prone to similar error patterns. we also conclude six reasons that cause financial sentiment analysis to fail, i.e., irrealis mood, rhetoric, dependent opinion, unspecified aspects, unrecognized words, and external reference. irrealis mood, in particular, also accumulates in the failed cases of sentiment analysis in the business review domain. the error types, of course not exclusive for financial sentiment analysis and may appear elsewhere, are possible avenues for improving model performance. these findings also contribute to the understanding of semantics and pragmatics of financial tweets. it is frustrating to see a number of errors, which are not difficult for human, whereas all the experimented methods failed on them. therefore, we argue that having 'silver bullets' for financial sentiment analysis at the current stage is an illusion. more broadly speaking, sentiment analysis is a suitcase problem, which cannot be perfectly solved by an end-to-end model without handling the many nlp sub-problems. a divide-and-conquer approach is particularly needed from financial sentiment analysis, which has limited training data and unique language styles. in future work, we plan to extend our experiments to more datasets, and to incorporate separate nlp modules that deal with the discovered error types to improve existing sentiment analysis models.