Generating Plausible Counterfactual Explanations for Deep Transformers in Financial Text Classification (2020)

Abstract: Corporate mergers and acquisitions (M&A) account for billions of dollars of investment globally every year and offer an interesting and challenging domain for artificial intelligence. However, in these highly sensitive domains, it is crucial to not only have a highly robust/accurate model, but be able to generate useful explanations to garner a user’s trust in the automated system. Regrettably, the recent research regarding eXplainable AI (XAI) in financial text classification has received little to no attention, and many current methods for generating textual-based explanations result in highly implausible explanations, which damage a user’s trust in the system. To address these issues, this paper proposes a novel methodology for producing plausible counterfactual explanations, whilst exploring the regularization benefits of adversarial training on language models in the domain of FinTech. Exhaustive quantitative experiments demonstrate that not only does this approach improve the model accuracy when compared to the current state-of-the-art and human performance, but it also generates counterfactual explanations which are significantly more plausible based on human trials.

## introduction and related work in recent years, large-scale, pre-trained transformer models have led to massive improvements on a wide range of natural language processing (nlp) tasks @xcite , including financial technology applications @xcite @xcite . however, this impressive ability also coincides with an inherent lack of robustness and transparency, which undermines human trust in the prediction outcome. in the highly sensitive (and financially lucrative) area of fintech, explainable financial text classification remains an open, and highly alluring question. to tackle this problem, this paper advances a novel approach which first applies robust transformer models (by leveraging adversarial training) on a real-world, up-to-date, self-collected mergers and acquisitions (m&a) dataset, and then generating plausible, post-hoc, counterfactual explanations. in the remainder of this section, we describe relevant work to both of these areas before detailing our contributions. ## the novel mergers and acquisitions dataset description number #processed deal news total (2007-2019) 4,098 #train (2007-2014) 3,120 #validation (2015 -2016) 478 #test (2017 -aug 2019) 500 #unique companies and institutions 1,406 for this study we adopted a large-scale, up-to-date m&a dataset collected from zephyr, a comprehensive database of deal data from the "real world". the dataset foot_0 contains 14,539 news articles or tweets on m&a events between january 1st 2007, and august 12th 2019. each instance corresponds to a specific editorial m&a article which describes a possible deal between an acquirer and a target company (also including a few ipo rumours). additionally, each datapoint also includes the deal outcome (see below), and the deal announcement data, if relevant. in this work, the deal outcome corresponds to the target class, and the raw dataset contains the following outcome types: complete -a deal between the acquirer and target companies concluded successfully; rumour -no deal materialized between the acquirer and target company; pending -a desired deal between the acquirer and target company has been confirmed, and at the time of data collection was deemed to be in-progress, but not yet complete; cancelled -a past potential deal between the acquirer and target companies has been confirmed, but it did not complete, and is no longer being pursued. in order to prepare the raw dataset for use in this study, a number of pre-processing steps were carried out: the result is a dataset of 4,098 instances (news articles and meta-data) which we split into training, validation, and testing sets on a year-by-year basis (see ## methodology the pipeline of our method is shown in fig. 1 . first, as a prerequisite, a transformer variant is finetuned on the m&a prediction task, alongside adversarial training (which as we shall see is shown to be promising in this domain). second, important words in the test instances are identified using a sampled contextual decomposition technique after the prediction. third, a counterfactual explanation is generated by replacing these words with grammatically plausible substitutes. as we shall see, although this method does not always guarantee a plausible counterfactual will be found, we propose two alternative methods which will, albeit with the possible trade-off of plausibility. these steps are detailed next. ## experiment 1: financial text classification with robust transformers in this section we describe the results of a comprehensive evaluation of classification accuracy, comparing a variety of different classification baselines (including a human baseline) to our adversarial transformer approach. ## experiment 2: generating plausible counterfactual explanations interpretability is an increasingly important property for many deep learning techniques, including computer vision and natural language processing (kenny and keane, 2019), especially in critical tasks such as financial text classification; high-value investment decisions demand a reasonable level of interpretability if investors are to trust the predictions that come for a system such as the one described in this work. in this section, we describe the qualitative analysis for each of our methods. subsequently, we show the evaluation of user studies compared to the existing example-based explanation methods. ## conclusion and future work in this work, we pursued a new research problem of m&a prediction. our transformer-based classifier leveraged the regularization benefits of adversarial training to enhance model robustness. more importantly, we built upon previous techniques to quantify the importance of words and help guarantee the generation of plausible counterfactual explanations with a masked language model in financial text classification. the results demonstrate superior accuracy and explanatory performance compared to state-of-the-art techniques. an obvious extension would be to include canceled deals into the classifier, or to predict novel m&a events based on market descriptions of companies (e.g., scale, finances, and target markets). moreover, additional financial events (e.g., misstatement detection and earnings call analysis) is yet another related task to be considered for further research.