UPB at F in C ausal-2020, Tasks 1 & 2: Causality Analysis in Financial Documents using Pretrained Language Models (2020)

Abstract: Financial causality detection is centered on identifying connections between different assets from financial news in order to improve trading strategies. FinCausal 2020 - Causality Identification in Financial Documents – is a competition targeting to boost results in financial causality by obtaining an explanation of how different individual events or chain of events interact and generate subsequent events in a financial environment. The competition is divided into two tasks: (a) a binary classification task for determining whether sentences are causal or not, and (b) a sequence labeling task aimed at identifying elements related to cause and effect. Various Transformer-based language models were fine-tuned for the first task and we obtained the second place in the competition with an F1-score of 97.55% using an ensemble of five such language models. Subsequently, a BERT model was fine-tuned for the second task and a Conditional Random Field model was used on top of the generated language features; the system managed to identify the cause and effect relationships with an F1-score of 73.10%. We open-sourced the code and made it available at: https://github.com/avramandrei/FinCausal2020 .

## introduction financial news contain various descriptions of causes and effects between financial objects that influence sales, employment, earnings, or stock prices. causal analysis can be used to detect correlation between news and prices of different assets @xcite by identifying different consequence, that in return can lead to major events as a financial crisis @xcite or possible arbitrage opportunities @xcite . for example, given the follow-up statement regarding brexit "the leave win led to an 11 percent drop in gbp/usd overnight.", we can infer that "the leave win" is the cause for "an 11 percent drop in gbp/usd overnight", which represents the effect and indicates a change in price of the gbp/usd currency pair @xcite . nowadays, the high volume of published news and financial documents hinders and renders unfeasible the manual analysis of all articles in order to extract implications on the economy. thus, a need for developing new tools for causality extraction emerged to facilitate this process. methods based on pattern detection @xcite , on combinations of patterns and rules @xcite , and on attention mechanisms with different embeddings @xcite were introduced in the field of causal detection. a shared task was proposed at the financial narrative processing workshops, namely fincausal-2020 @xcite , to evaluate the performance of each participant system on two tasks defined on a financial corpus proposed by yseoplab. the first task consisted in building a classifier to identify whether a sentence contains a financial causality or not, whereas the second task extracted causes and also their corresponding effects from a given text. the rest of the paper is organized as follows. the second section introduces our proposed methods for tackling the two sub-tasks, while the third section presents our results, together with experimental setup and an error analysis. the paper ends with conclusions and proposals of future work. ## dataset the organizers made available 3 datasets for both tasks: trial, practical, and evaluation. the trial dataset for the first task contained 8580 samples, out of which only 569 (i.e., 6,69% from total samples) included causal events. the practice dataset has approximately the same distribution as trial, with 13478 samples out of which 1010 were labeled as causal (i.e., 7,49% from total samples). the evaluation dataset was kept blind, but it has the same distribution as trial and practice. the dataset for the second taks largely corresponds to samples from task 1 that are labelled as causal, with the specification of causal and effect sub-strings. the trial dataset contains 641 samples, while the practical dataset contains 1109 samples. the evaluation dataset contained 638 samples with removed annotations, i.e., the text is revealed, but there are no causal or effect representations. ## proposed solutions five transformer-based language models @xcite were considered for the the first task and were fine-tuned on the fincausal-2020 dataset, namely: bert (base and large) @xcite , al-bert (base and large) @xcite , roberta (base and large) @xcite , scibert (base) @xcite , and finbert (base) @xcite . the fine-tuning mechanism was the one proposed by @xcite who take the embedding of the first token and project it into a scalar that represents a probability. then, we minimize the kullback-leibler divergence between the distribution of the dataset and the distribution predicted by the model by using the adam optimizer @xcite . we also experimented with various classifier ensembles taking into account different combinations of the fine-tuned language models as these ensembles surpass in general the performance of individual models; majority voting was considered for the final labeling of each sentence. the second task was operationalised as a token labeling, similar to part-of-speech tagging @xcite or named entity recognition (dumitrescu and avram, 2019), by marking the corresponding tokens of the causes and effects sequences with cause and effect labels, whereas the rest of the tokens were marked as o. inside-outside-beginning (iob) format was used for the labels to support the identification of the sequences at inference time. bert-base models were used to obtain contextualized embeddings for each token, and then a conditional random field (crf) @xcite was trained to predict the most probable sequence of labels for a given input text. although the iob format and the crf aided in modeling the problem as a token labeling task, the extraction of causes and effects is harder at inference because the model can make an incorrect prediction in the middle of a cause/effect sequence, or predict a cause/effect token in the middle of o tags. several heuristics were introduced to mitigate this issue when extracting the causes and effects, namely: ## conclusion and future work as the financial volume of data grows, it becomes harder and harder to analyze chain of events, with corresponding cause and effects. our solution for the first task at fincausal-2020 was an ensemble of five transformer-based language models that placed second on the evaluation leader-board with an 97.55% f1-score. the second task considered a bert-base model to extract contextualized embeddings for each token, that were further used to train a crf; an f1-score of 73.10% was achieved on the evaluation set. possible directions of research include using more powerful language models like xlnet @xcite or exploring if an additional bidirectional long-short term memory (bilstm) on top of the language model to improve the results on either task. in addition, we envision the consideration of specific discourse markers, as well as transfer learning using similar tasks -for example, a model trained on the snli corpus from stanford @xcite .