F i NLP at F in C ausal 2020 Task 1: Mixture of BERT s for Causal Sentence Identification in Financial Texts (2020)

Abstract: This paper describes our system developed for the sub-task 1 of the FinCausal shared task in the FNP-FNS workshop held in conjunction with COLING-2020. The system classifies whether a financial news text segment contains causality or not. To address this task, we fine-tune and ensemble the generic and domain-specific BERT language models pre-trained on financial text corpora. The task data is highly imbalanced with the majority non-causal class; therefore, we train the models using strategies such as under-sampling, cost-sensitive learning, and data augmentation. Our best system achieves a weighted F1-score of 96.98 securing 4th position on the evaluation leaderboard. The code is available at https://github.com/sarthakTUM/fincausal

## introduction a causal relation involves two events e 1 and e 2 where one, referred to as 'cause', is responsible for triggering the other, referred to as the 'effect'. the causation can be expressed implicitly or explicitly by causal verbs, punctuation, conjunctions, prepositions, or any other linguistic cue that establishes a trigger between the cause and the effect. explicit causation in a text segment can usually be identified by the presence of causative keywords or characteristic grammatical patterns whereas implicit causality is comparatively harder to detect, usually only through the context. many studies on causality extraction in the text are motivated by @xcite in which they identify five different constructs to express explicit cause-effect relation. @xcite extended the third construct by identifying lexical syntactic patterns including noun-phrases (np) such as n p 1 , cause-verb, n p 2 to find causality in the wordnet 1 definitions. @xcite extends the first and second construct by introducing four categories of the cue phrases for the causality expressions to build an automatic causality extraction framework. more recently, @xcite proposed an lstm-crf architecture for labeling cause and effect tokens in a text segment. many prior works on causality extraction primarily use the data labeled using weak supervision techniques due to scarcity of the human-labeled data. @xcite use transfer learning to tackle this issue. to this end, @xcite introduced a human-labeled data for causality extraction in financial news text crawled by qwam. 2 shared task: this shared task consists of two sub-tasks: (i) binary classification of text segment as 'causal (c)' if the text contains causality or 'non-causal (nc)' otherwise, and (ii) extracting spans of 'cause' and 'effect' within the text segment if it is labelled 'causal'. the 'effect' can only be a quantified fact, whereas a 'cause' can be a fact or a quantified fact. we propose a system for the first sub-task. ## experiments and results we split the annotated corpus as 80% for training, 10% for validation, and 10% for testing. the final system is retrained on all the data combined for the number of training epochs obtained from the validation set using early stopping @xcite with patience of 3 epochs on the f1 score. we use pytorch @xcite ) models trained on nvidia gtx1050 gpu. we aim to answer the following research questions (rq) using precision (p), recall (r), weighted f1-score (f1), confusion matrix (tp, tn, fp, fn) on our own test set (columns without *-e) and official evaluation set (columns with *-e). ## conclusion we proposed a system to detect causality in the financial texts using an ensemble of domain-specific and generic bert classifiers, trained using various datasets and training strategies to tackle the imbalanced class distribution. among many other experiments that did not improve the performance and are not mentioned in this paper are the convolutional neural networks (cnns), roberta language model @xcite , and non-neural classifiers trained on hand-crafted linguistic features. we would like to continue the improvement by jointly training the causality classifier (sub-task-1) with the causal span extraction (sub-task-2).