Domino at F in C ausal 2020, Task 1 and 2: Causal Extraction System (2020)

Abstract: Automatic identification of cause-effect relationships from data is a challenging but important problem in artificial intelligence. Identifying semantic relationships has become increasingly important for multiple downstream applications like Question Answering, Information Retrieval and Event Prediction. In this work, we tackle the problem of causal relationship extraction from financial news using the FinCausal 2020 dataset. We tackle two tasks - 1) Detecting the presence of causal relationships and 2) Extracting segments corresponding to cause and effect from news snippets. We propose Transformer based sequence and token classification models with post-processing rules which achieve an F1 score of 96.12 and 79.60 on Tasks 1 and 2 respectively.

## introduction with the rise of the internet came an unprecedented growth in the amount of financial news being produced everyday. in its raw form however, this data though large, has limited utility. from this raw data, identifying relationship mapping between an external cause and its consequence can be of great value. the simplest way to define a causal relationship is expressing it in the form of "e 1 causes e 2 " where e 1 and e 2 are two linked events -e 1 being the cause and e 2 being the effect. though this forms a simple illustrative example, real world causal relationships are often expressed with different lexical constructs, span across multiple sentences or are coupled together with other effects and causes. we can now formally define our tasks as follows : our first task is framed as a text classification task and has been widely studied in the nlp community @xcite . long-short term memory networks, convolutional neural networks @xcite , and more recently pre-trained transformer models @xcite have shown success on a wide range of text classification tasks. particularly, for the task of cause-effect detection, bert based models have achieved state of the art performance @xcite on the semeval task @xcite on detecting semantic relations in text such as cause-effect, entity-origin, etc. to further leverage the benefits of pre-training, we use finbert @xcite , a bert based model fine-tuned on financial data for better language understanding and domain adaptation. our second task is concerned with extracting spans corresponding to cause and effect from a causal news snippet. with the advent of large scale question answering datasets @xcite , deep learning models and specifically transformers have been used extensively to identify answer spans. this task can also be framed as a token classification task where each token is classified as cause/effect/other and the tokens are grouped to form cause/effect spans. additionally, token classification models such as bilstm-crf @xcite and feature-based crf @xcite have demonstrated good performance for causality extraction. we cast our task as a token classification problem and apply certain post-processing rules to reconstruct the spans from individual token outputs. in this work, we present our results on task 1 using a variant of bert -finbert pre-trained on financial data. our finbert model achieve f 1 scores of 95.60 and 96.12 on the validation and test set respectively. for task 2, we use bert with a linear layer over token embeddings for token classification. our model achieves a validation f 1 score of 75.40, beating the baseline crf model and a test set f 1 score 79.60. the code for our approaches will be made available on github foot_0 . ## data in this work, we use the two datasets provided by fincausal 2020 workshop @xcite to train and evaluate our models. in comparison to datasets for other nlp tasks, the datasets for tasks 1 and 2 are relatively small. ## proposed approach we now present our systems for cause-effect detection and extraction. given the small size of our dataset, it might not be possible for deep learning models to completely fit our tasks. we combat this in two wayspre-training and post-processing. we leverage pre-training by utilizing finbert (araci, 2019) -a bert model trained on reuters trc2 corpus, financial phrasebank and fiqa datasets for easier adaptation to the financial domain and use a post-processing module for task 2 to incorporate some of the rules followed by the annotators as opposed to relying solely on the model to learn them. ## experimental setup we finetune models for tasks 1 and 2 on an nvidia tesla t4 gpu and use the adam optimizer @xcite . the data partition and hyperparameters used for the individual tasks are described below. ## results and analysis we compare our approaches against baselines and models provided by the task organizers. for task 1, we compare our finbert model against a 1d cnn model @xcite and vanilla bert based task baseline. tables 1 , 3 show the strong performance of our model on validation and test splits. our finbert model placed 7 th on the official fincausal leaderboard 9 . for task 2, we compare our approach against a zero-shot model based on squad bert 10 (we frame synthetic questions asking for the cause/effect, and extract spans produced by the model), a crf baseline provided by the organizers and our sequence labelling approach. we can see from ## conclusion in this work, we leverage bert for sequence and token classification to detect and extract cause-effect relations in financial documents. we show the efficacy of domain-specific pre-training and address the low resource problem by applying post-processing rules over the model's predictions. in the future, we plan to experiment with data augmentation techniques such as synthetic data created from the cause and effect of different examples, and syntax-aware post-processing rules for more effective span extraction.