The Financial Document Causality Detection Shared Task ( F in C ausal 2020) (2020)

Abstract: We present the FinCausal 2020 Shared Task on Causality Detection in Financial Documents and the associated FinCausal dataset, and discuss the participating systems and results. Two sub-tasks are proposed: a binary classification task (Task 1) and a relation extraction task (Task 2). A total of 16 teams submitted runs across the two Tasks and 13 of them contributed with a system description paper. This workshop is associated to the Joint Workshop on Financial Narrative Processing and MultiLing Financial Summarisation (FNP-FNS 2020), held at The 28th International Conference on Computational Linguistics (COLING’2020), Barcelona, Spain on September 12, 2020.

## introduction in an effort to automatically interpret the semantics of written languages, the analysis and understanding of causal relationships between facts stand as a key element. a major difficulty regarding automation is that causality can be expressed using many different syntactic patterns as well as contrasted semantic representations. this difficulty is reinforced by the existence of both explicit and implicit cause-effect links. early works in this field, such as @xcite , aim at detecting causal relations using linguistic patterns. however, these applications are often restricted to a specific domain, limited to explicit causal relationships only (causal links, causative verbs, resultative constructions, conditionals and causative adverbs and adjectives), and do not take into account the ambiguities of the connectors. the semi-automatic method developed by @xcite goes a step further by creating lexicosyntactic patterns based on wordnet semantic relations between nouns, then using semantics constraints to test ambiguous causal relations. despite better results, the exclusive use of linguistic patterns prevents a fully efficient coverage of all cause-effect links. consequently, various machine learning techniques were tested for this task. @xcite developed naive bayes causality extraction models based on lexical pair probability and cue phrase probability. by focusing on the dynamics of causal relationships, the system prepost developed by @xcite stands as a viable system to detect causal relationships between one event and a consequent state of this event, training a classifier to identify events' preconditions and/or postconditions. in parallel, hybrid methods were also developed such as the expanded semantic parsing of @xcite . this system combines an scl approach, patternbased methods and a neural network architecture, offering more flexibility than exclusive pattern based approaches. overall, the management of linguistic ambiguities as well as the existence of implicit connections appear to be the main brakes in the identification and extraction of causal relationships. in this paper, we present the fincausal corpus and the associated featured tasks, as a contribution to the research effort addressing implicit and multiple causalities detection automation in financial documents. all the datasets created for this shared task are publicly available to support further research on causality modelling 3 , and the detailed annotation scheme is provided in the appendix a. next, sec-tion 2 describes the fincausal corpus and section 3 presents the tasks. section 4 provides the baseline proposed to participants and details their results, with a high-level description of the approaches they adopted. finally, section 5 concludes this report and discusses some future directions. ## fincausal corpus the data are extracted from a corpus of 2019 financial news provided by qwam, collected on 14.000 economics and finance websites. the original raw corpus is an ensemble of html pages corresponding to daily information retrieval from financial news feed. these news mostly inform on the 2019 financial landscape, but can also contain information related to politics, micro economics or other topic considered relevant for finance information. data are released under the cc0 license foot_4 . all collected html files were initially split into sentences according to their punctuation, then were grouped into text sections of 1 to 3 sentences after the data annotation process has been completed. below are the principle global metrics gathered during the annotation process. the metrics are defined with respect to the annotation scheme. for consistency in our references, we refer to a file as the original document to process, a text section as a multi-sentenced text string (1 to 3 sentences that may or may not contain causality) and a chunk as a substring (consisting either of a part of a sentence, a whole sentence or multiple sentences) within a text section. we also retain statistics related to the main tags used in our annotation scheme during the preparation of the datasets. these tags are defined as follows: in addition, metrics related to fact alignment (i.e., trimming sentences according to preset priority rules in the annotation scheme) are also included to consistently reflect the preprocessing carried out at this step. all statistics are provided for the 3 datasets provided to participants: trial, practice, evaluation as well as global statistics to present a general outlook on the overall annotation phase. the resulting statistics are collected in ## evaluation a baseline was provided on the trial samples for both tasks 1 and 2 foot_5 . participating systems were ranked on blind evaluation datasets based on a weighted f1 score, recall, precision for task 1, plus an additional exact match for task 2. regarding official ranking, weighted metrics from the scikit-learn package foot_6 were used for both tasks, and the official evaluation script is available on github foot_7 . participating teams were allowed to submit as many runs as they wished, while only their highest score was withheld to represent them during evaluation. in addition, they were proposed to enhance their system in a postevaluation phase foot_8 . only the scores validated during the evaluation phase of the competition are displayed below. amongst the 13 participating teams, six choose to address tasks 1 and 2 and one (prosperamnet) proposed an integrated pipeline for both. details on the methods and features used by different systems are provided in ## conclusion in this paper, we present the framework and the results for the fincausal shared task. in addition , we present the new fincausal dataset built specifically for this shared task. we plan to run similar shared tasks in the near future, possibly with some augmented data, in association with the fnp workshop.