GB e at F in C ausal 2020, Task 2: Span-based Causality Extraction for Financial Documents (2020)

Abstract: This document describes a system for causality extraction from financial documents submitted as part of the FinCausal 2020 Workshop. The main contribution of this paper is a description of the robust post-processing used to detect the number of cause and effect clauses in a document and extract them. The proposed system achieved a weighted-average F1 score of more than 95% for the official blind test set during the post-evaluation phase and exact clauses match for 83% of the documents.

## introduction the fincausal 2020 shared task @xcite focuses on the identification of causality in financial documents. beyond factual data, causality detection and characterization in financial documents help to identify the reasons leading to a given quantified financial event. fincausal 2020 proposes two shared tasks: the first one focuses on the detection of causality in documents (binary classification), while the second task consists of the actual extraction of the cause and effect spans from a document expressing causality (token classification or span extraction). this document describes a system submitted for the second task. based on a deep learning language model, it leverages an architecture similar to question answering and therefore addresses the task as a span extraction problem. the post-processing and filtering of unrealistic spans are of high importance given that several clauses (at least a cause and an effect) need to be extracted from the documents. the extracted clauses are further refined to align with the task labeling guidelines. the proposed system ranked 2 nd for the fincausal 2020 task 2, achieving a weighted-average f1 score of 94.7% and predicted exact matches for the cause and effect clauses for 73.7% of documents. further training refinement led to an improved f1 score of 95% and exact match for 83.3% of the documents in the official blind test set in the post-evaluation phase. the code for the proposed solution is available at https://github.com/guillaume-be/financial-causality-extraction ## system description the core system used for all submissions is described in figure 1 . features are generated from an input text using the transformers library @xcite with high-performance tokenizers (hugging face, 2020). while the task dataset contains only rather short (2 to 3 sentences) texts, the system can handle longer documents exceeding the language model context using a sliding window mechanism. the tokenized input is then passed through a language model based on the transformers @xcite architecture. the last layer of hidden states is connected to a dense layer with 4 output dimensions to generate logits for the start and end positions of the cause and effect span. this mechanism is similar to question answering state-of-the-art architectures with two spans extracted instead of one. the output of the model is an array of 4 vectors per document representing the logits for the start and end of the cause and effect. a first step cuts-off the logits vectors to the top-n elements, limiting the potential solution space to a size of n 4 . several of these solutions are unrealistic and must be this work is licensed under a creative commons attribution 4.0 international license. license details: http:// creativecommons.org/licenses/by/4.0/ . filtered out. some of these filtering steps are leveraged from the bert @xcite question answering system for squad @xcite ). this includes for example filtering out start and end combinations where the end is before the start or the span length is longer than the maximum allowed length. the cause and effect extraction tasks add complexity in that two spans need to be extracted. additional filters ensuring that the cause and effect clauses do not overlap have been added (for example start @xmath0. feature generation input text sentence boundary detection sentence boundaries language model dense cause start cause end effect start effect end logits impossible spans filtering heuristics scoring threshold possible cause effect span tuples cause effect pairs finally, task-specific heuristics aligned with the labeling guidelines @xcite have been implemented. in addition to the filtered logits, these heuristics take the sentence boundaries as an input (these are extracted with the pysbd library @xcite ). the following heuristics have been implemented: ## conclusion a system for the second task of the fincausal 2020 workshop has been described. the problem was framed as a span extraction task leveraging pre-trained model and post-processing elements from stateof-the-art question answering systems. this strong baseline was complemented with extensions to handle the extraction of two spans per document. the ability of the system to not only extract cause and effect, but also to predict the number of cause/effect relationships in a document was also demonstrated. the proposed model accurately extract causes and effect from documents.