Fraunhofer IAIS at F in C ausal 2020, Tasks 1 & 2: Using Ensemble Methods and Sequence Tagging to Detect Causality in Financial Documents (2020)

Abstract: The FinCausal 2020 shared task aims to detect causality on financial news and identify those parts of the causal sentences related to the underlying cause and effect. We apply ensemble-based and sequence tagging methods for identifying causality, and extracting causal subsequences. Our models yield promising results on both sub-tasks, with the prospect of further improvement given more time and computing resources. With respect to task 1, we achieved an F1 score of 0.9429 on the evaluation data, and a corresponding ranking of 12/14. For task 2, we were ranked 6/10, with an F1 score of 0.76 and an ExactMatch score of 0.1912.

## introduction despite the many advances in the field of natural language processing (nlp) in recent years, many challenges dealing with contextual relationships still persist, one of which being causality. causality, as conveyed in written english, relates a textual description of a cause to a description of its effect(s). while the identification of subordinating conjunctions -such as "because", "since" and "if" -can help in identifying relevant pieces of text forming causal relationships, often richer contextual information is needed. the goal of the fincausal 2020 shared task @xcite is to explore causal relationships from financial and economic context and to determine the likelihood of whether or not a given text contains a causal relationship and then detect the cause and effect parts of the sentence. ## data the data sets are composed of 23808 paragraphs from financial news. 1 the data sets contain paragraphs, with an average size of 214 (+/-161) characters, each with annotations relevant to the corresponding task. for task 1, the size of the complete data is 22058 paragraphs, each of which is annotated with a "1" (a causal relationship exists in this paragraph -7.2% of the paragraphs), or "0" (no causal relationship in the paragraph -92.8% of the paragraphs). the data set for task 2 has a size of 1750 paragraphs, with every paragraph containing exactly one cause and one effect character sequence. these are annotated as character ranges. since we treat task 2 as a sequence tagging task, we transform these annotations to token level annotations, so that every token in the paragraph gets the label cause, effect or, 0 (for all tokens which belong neither to the cause nor the effect part). this yields a label distribution of 40.8% cause, 40.8% effect, and 18.4% "0". for the experimentation and the development of the models for task 1 and task 2, a 70% training -30% validation split of the respective data-set (joined practice and trial) was used. ## system tasks 1 and 2 were treated independently. the former was treated as a document classification task (a document being a paragraph in this case) and the latter as a sequence tagging task. ## results the reported results were evaluated on the holdout test set, which was not included in the validation set. the evaluation metrics are reported as defined by the shared task organizers. foot_4 ## discussion our present system relies heavily on the choice of embeddings, as well as on large pre-trained language models. this could possibly be changed if we had a more extensive and consistent data set for training. one of our observations in this regard was, that the provided data included a number of irregularly formatted paragraphs (for example headlines or bullet points). we believe that our algorithms would benefit from further analysis and possible cleaning of such instances. additionally, the current data does not allow us to explore other approaches that are based on syntactic features. regarding task 1, the results of the cnn could be potentially improved by additional hyperparameter tuning. the cnn model showed good results already in early epochs, but was not able to outperform the less complex ml models, which were extensively tuned. with respect to task 2, it would have been interesting to test more combinations of embeddings, data augmentation, and post-processing, which was not possible due to limited time and computing resources. ## conclusion and outlook identifying causality in text can be a helpful tool for many empirical use cases. in the context of financial document analysis, it could be used to identify important facts and developments in the annual report of a company. we can potentially extend our previous work on various downstream-tasks on financial reports by incorporating causality, for instance for key performance indicator extraction @xcite , contradiction detection @xcite , or content-based text classification and consistency checks @xcite . we can also exploit causality detection in the context of text summarization. causal sentences may indicate content richness, which is useful not only to extract the most relevant sentences of an original text within an extractive summarization setting (as presented in @xcite ) but also when we evaluate the quality of a generated summary from a wide range of features @xcite . in our future work, we are planning to address such applications, building on our experiments and insights from this challenge.