Event Causality Identification with Causal News Corpus - Shared Task 3, CASE 2022 (2022)

Abstract: The Event Causality Identification Shared Task of CASE 2022 involved two subtasks working on the Causal News Corpus. Subtask 1 required participants to predict if a sentence contains a causal relation or not. This is a supervised binary classification task. Subtask 2 required participants to identify the Cause, Effect and Signal spans per causal sentence. This could be seen as a supervised sequence labeling task. For both subtasks, participants uploaded their predictions for a held-out test set, and ranking was done based on binary F1 and macro F1 scores for Subtask 1 and 2, respectively. This paper summarizes the work of the 17 teams that submitted their results to our competition and 12 system description papers that were received. The best F1 scores achieved for Subtask 1 and 2 were 86.19% and 54.15%, respectively. All the top-performing approaches involved pre-trained language models fine-tuned to the targeted task. We further discuss these approaches and analyze errors across participants’ systems in this paper.

## introduction a causal relation represents a semantic relationship between a cause argument and an effect argument, in which the occurrence of the cause leads to the occurrence of the effect @xcite . extracting causal information from text has many downstream natural language processing (nlp) applications, for summarization and prediction @xcite @xcite , question answering @xcite @xcite , inference and understanding @xcite . however, data for causal text mining is limited @xcite @xcite @xcite . there are also not many benchmarks to allow for fair model comparisons @xcite . therefore, in this paper, we continue our efforts with the creation of the causal news corpus (cnc). cnc is a corpus of news articles annotated with causal information suitable for causal text mining. additionally, we introduce a shared task to promote modelling for two causal text mining tasks: (1) causal event classification and (2) cause-effect-signal span detection. figure 1 provides examples from the cnc in this shared task. to our knowledge, we are the first dedicated causal text mining dataset and benchmark to include signal span detection as an objective. the rest of the paper is organized as follows: section 2 presents literature on event causality datasets. section 3 describes the dataset and annotation of the corpus. section 4 formally introduces the two subtasks for the shared task. section 5 describes the evaluation metrics and competition set-up. subsequently, section 6 summarizes the methods used by participants during the competition, while section 7 analyzes the participants' submissions. finally, section 8 concludes this paper. ## related work in many papers about event causality identification (eci) @xcite @xcite @xcite , the two datasets used for benchmarking are causaltime-bank @xcite and eventstoryline @xcite . these datasets are unsuitable for span detection since their arguments are event headwords only. there are two other efforts that intentionally introduce datasets for benchmarking causal text mining systems. fincausal @xcite is a recurring shared task held within the finnlp workshop focusing on financial news. in the first subtask, participants also aim to identify if sentences contain causal relations. in the second subtask, participants to identify the cause and effect spans in the causal sentences. unicausal @xcite foot_0 is an open-source repository for causal text mining that has consolidated six corpora for three causal text mining tasks. the six corpora included in unicausal are: altlex @xcite , because 2.0 @xcite , causaltimebank @xcite , eventstoryline v1.0 @xcite , penn discourse treebank v3.0 @xcite , and semeval 2010 task 8 @xcite . the three tasks are: causal sentence classification, causal pair classification and cause-effect span detection. similar to fincausal and unicausal, we included a signal span detection objective. our annotation guidelines differ slightly, in that our arguments must contain events, and the spans are annotated in a manner that is minimally sufficient. in general, we notice that spans from fincausal are much longer. spans from unicausal depend on the original data source. additionally, for cause-effect span detection in fincausal, their approach to handle multiple causal relations per unique sentence was to include index numbers at the start of each sentence to differentiate the cause-effect predictions. this approach is problematic because (1) it leaks information that the sentence contains multiple causal relations to the model, and (2) predictions that are submitted in a different order from the true labels are unnecessarily penalised. therefore, we differ from fincausal when evaluating multiple causal relations in span detection since we group relations by its sentence index. this is described further in section 5.1. ## task description the shared task is comprised of two subtasks related to event causality identification. the objective of each task is described in detail as follows: ## conclusion in conclusion, our shared task investigated two important tasks in causal text mining, namely: (1) causal event classification, and (2) cause-effect-signal span detection. our shared task attracted 29 registered participants and 17 active participants who made over 100 submissions on the test set. based on the 12 description papers received, many novel methods that exceeded our initial baseline were proposed. the best f1 scores achieved for subtask 1 and 2 were 86.19% and 54.15% respectively. we intend to re-launch this shared task next year with even more data for subtask 2. additionally, we will also investigate the challenging examples in subtask 1 that are predicted wrongly by many teams.