Probing into the Root: A Dataset for Reason Extraction of Structural Events from Financial Documents (2021)

Abstract: This paper proposes a new task regarding event reason extraction from document-level texts. Unlike the previous causality detection task, we do not assign target events in the text, but only provide structural event descriptions, and such settings accord more with practice scenarios. Moreover, we annotate a large dataset FinReason for evaluation, which provides Reasons annotation for Financial events in company announcements. This task is challenging because the cases of multiple-events, multiple-reasons, and implicit-reasons are included. In total, FinReason contains 8,794 documents, 12,861 financial events and 11,006 reason spans. We also provide the performance of existing canonical methods in event extraction and machine reading comprehension on this task. The results show a 7 percentage point F1 score gap between the best model and human performance, and existing methods are far from resolving this problem.

## introduction why does the event happen? people are always eager to find the reasons for an event. automatically extracting the causal explanations of the given events from texts is useful and important for common users and downstream applications. for example, in the financial domain, returning the reasons of a concerned financial event in an information retrieval system can free analysts from reading the enormous company announcements and help investors make financial decisions. previous work on event causality @xcite @xcite the identification of causal relations between two given events that are usually presented as event trigger words. however, in reality, users may only know a particular event happened but without knowing its mention or trigger in the documents, and they just wonder the reasons for it. therefore, we propose a new task aiming at extracting the causal explanations of the given structurally presented events from document-level texts. specifically, a structural event defined here is a structural description that contains all necessary roles for an event type. such a description can completely represent a specific happened event in reality. for example, in figure 1 , the pledge event has four predefined roles name, org, num, beg to represent an occurred pledge event. then, our task is to extract the reasons of the structural events as textual spans from the document. to investigate the solution for this challenging task, we construct a large-scale chinese dataset finreason 1 . specifically, we automatically collect the formal financial documents with their corresponding structural events the same as @xcite . then, crowd workers are employed to annotate the reasons in the documents for each structural event. in order to guarantee annotation quality and high inter-annotator agreement (iaa), we set several annotation principles and define 3 types of possible causal explanations (motivation, cause, enable) as reasons for the events in company announcements to guide annotators. finally, there are 8,794 documents, 12,861 collected financial events, and 11,006 reason spans in total. the cohen's kappa of annotations is 83.87%. moreover, to understand this task's difficulties, we regard this task as an event extraction (ee) or a machine reading comprehension (mrc) task. we also try some canonical models, such as bil-srm+crf @xcite , and bert @xcite on this task and set benchmarks. empirical results show that this task is challenging, and there is still an overall gap of 7pp (percentage points) in the f1 score between the best model and human performance. ## related work much nlp research has focused on identifying causality relations from text, including knowledge bases (wordnet @xcite , framenet @xcite and conceptnet @xcite ), semantic related evaluations (semeval-2007 task 04 @xcite , copa @xcite , red @xcite ), and event-related systems @xcite @xcite @xcite . these work tried to identify real-world causality in lexicons or texts from different aspects. however, they have found it is difficult to agree on if a causal relationship exists in reality due to the ambiguity of causality definition. our dataset mitigates this problem by only identifying contextual causality and do not check with reality. in addition, plenty of work also only identify context-level causal relationships, such as general causality detection tasks pdtb @xcite 1 http://www.nlpr.ia.ac.cn/cip/ ˜liukang/dataset/finreason1.html 2007) and because 2.0 @xcite , and emotion causality detection task eca @xcite . some work @xcite @xcite also tries to identify the causal relations between events at the contextual-level. however, our task is different because we focus on extracting the reasons for well-defined structural events, which is more close to practice scenarios. ## conclusion in this work, we propose a dataset finreason for a new event causality extraction task. our experiments show that this task is still challenging for current models. future work may consider breaking the challenging cases (multiple-events, multiplereasons, and implicit-reasons) to achieve a more satisfying performance.