IDIAP ers @ Causal News Corpus 2022: Extracting Cause-Effect-Signal Triplets via Pre-trained Autoregressive Language Model (2022)

Abstract: In this paper, we describe our shared task submissions for Subtask 2 in CASE-2022, Event Causality Identification with Casual News Corpus. The challenge focused on the automatic detection of all cause-effect-signal spans present in the sentence from news-media. We detect cause-effect-signal spans in a sentence using T5 — a pre-trained autoregressive language model. We iteratively identify all cause-effect-signal span triplets, always conditioning the prediction of the next triplet on the previously predicted ones. To predict the triplet itself, we consider different causal relationships such as cause→effect→signal. Each triplet component is generated via a language model conditioned on the sentence, the previous parts of the current triplet, and previously predicted triplets. Despite training on an extremely small dataset of 160 samples, our approach achieved competitive performance, being placed second in the competition. Furthermore, we show that assuming either cause→effect or effect→cause order achieves similar results.

## introduction causality links the relationship between two arguments -cause and effect @xcite . figure 1 shows examples extracted from the causal news corpus (cnc) @xcite . cause clauses appear in yellow, effect in green, and signals in pink; hereafter referred to as ces triplets. as shown in the example, "the bombing created panic among villagers", illustrates that the event "bombing" caused the event "panic among villagers" termed as effect. the linkage among the cause and effect, i.e., the word "created", is termed as signal and can be expressed explicitly or implicitly. the alliance claimed 4,000 took part last year. figure 1 : examples from the causal news corpus, causes are in :::::: yellow, effects in . . . . . . green, and signals in pink. if a sentence has both -cause and effect -it is referred to as casual (a), otherwise, as non-casual (b). automatically detecting and extracting causality relations plays a vital role in many natural language processing (nlp) works to tackle inference and understanding @xcite @xcite . it has applications in various down-streaming nlp tasks, namely, causal question-answering generation, explaining social media behavior, political phenomena, effective education, and gender bias in the research community @xcite @xcite @xcite . in this paper, we describe our methodology for case-2022 cause-effect-signal span detection shared task (subtask 2). overall, our main contributions are listed below: ## related work the problem of causality extraction from text is a challenging task as it requires semantic understanding and contextual knowledge. there were many attempts in the domain of linguistics for corpora creation for event extraction but with limited size such as causaltimebank (ctb) @xcite from news with 318 pairs, caters @xcite ) from short stories with 488 casual links, eventstoryline @xcite from online news articles with 1,770 casual event pairs, semantic relation corpora pdtb-3 @xcite with over 7, 000 causal relations and cnc corpus @xcite with 1,957 casual events with multiple event pairs. compared to previous datasets, cnc differs by focusing on event sentences, accepting arguments which does not need to form a clause, and not limiting itself to pre-defined list of connectives, but instead including causal examples in more varied linguistic constructions. the previous work in this domain can be broadly classified into knowledge-based approaches, statistical ml, and deep-learning-based approaches. the knowledge-based approach uses linguistic patterns by predefining hand-crafted or keywords @xcite @xcite @xcite @xcite @xcite . statistical techniques @xcite rely on building probabilistic models over features extracted via third-party nlp tools such as wordnet @xcite . deep-learning techniques map words and features into low-dimensional dense vectors, which may alleviate the feature sparsity problem. the most frequent used sequence to sequence models are feed-forward network @xcite , long short-term memory networks @xcite @xcite convolutional neural networks @xcite @xcite , recurrent neural networks @xcite , gated recurrent units @xcite which embed semantic and syntactic information in local consecutive word sequences @xcite . later unsupervised training model such as bert @xcite , roberta @xcite , graph convolution network @xcite , graph attention networks and joint model for entity relation extraction @xcite @xcite . in this work, we base our model on t5 @xcite , a sequence-to-sequence transformer model, pre-trained on a mixture of denoising objective and 25 supervised tasks such as machine translation, linguistic acceptability, abstractive summarization or question answering. the unsupervised denoising objective randomly replaces spans of the input with different mask tokens, and generates contents of these masked spans prefixed with these special mask tokens. furthermore, our work shares similarities with pointer-network @xcite based generative framework for various ner subtasks introduced by @xcite . contrastively, our work is more adapted to lowresource scenarios, as no extra parameters were added to our system, at the cost of errors, which can happen in the postprocessing matching step. ## problem description case-2022 shared task challenge @xcite aimed for event causality identification, and extraction in casual news corpus @xcite . it comprised of two subtasks, namely casual event classification (subtask 1) and cause-effect-signal span detection (subtask 2) foot_1 . subtask 2 aims on extracting the spans corresponding to cause-effectsignal (ces) triplets, as shown in figure 1 . we trained a generative seq2seq model to address this challenge and extracted the ces triplets using an iterative procedure (see section 4.1). the dataset statistics are presented in ## results & discussion we now report the results obtained from averaging at least ten measured performances from 10 checkpoints trained with different seeds foot_5 . we studied 4 different variants of our system. system t5-ces is our vanilla model described in 4.1, based on t5-base. system t5-ces large is the same model based on t5-large. unlike t5-ces, system t5-ecs reverses the generation order by generating the first effect and cause, followed by the signal (assuming causal order @xmath0, hence the suffix ecs). lastly, we studied the effect of conditioning the model on the history of already generated triplets. we remove the history from the input at all times in training and predict the four identical ces triplets for each example in test time. our ablated results are available in firstly, the model with no history at input performs significantly worse, validating our hypothesis that the model can learn to decrease the probability of the triplets already contained within the input, even from just 160 samples. secondly, we observed a general trend that in the cause f1 t5-ces outperforms t5-ecs and in effect f1, t5-ecs outperforms t5-ces. this leads to the hypothesis that whichever part of the triplet, cause or effect, is generated first, the language model performs better in its case. thirdly, we observed that the large model achieved the best results on average. it also achieved our best single-checkpoint performance on the dev set (78.3 overall f1). however, given the sample size of the dev set, the differences between t5-ces, t5-ecs, and t5-ces large can hardly be deemed significant. next we present our results on the test set in additionally, we include extra statistics (dev 0 f1, dev 1 f1, dev es acc) for our best checkpoints. we expected the performance on the dev subset with two triplets (dev 2 f1) per example to be worse than on the dev subset with one triplet per sentence (dev 1 f1). performance-wise this does not always seem to be the case. upon manual analysis, we found that the model often failed in the second round of triplet extraction. we found 2 lm hallucinations out of 18 dev samples in the second generation round. ## inference speed measuring the inference speed on test set, we used intel i5-based 2080ti gpu workstation. the inference of 4 ces triplets without postprocessing per 1 sentence example took 1.46 seconds on average. the postprocessing runtime was negligible, taking 0.025 seconds per sentence example on average. ## conclusion in this work, we have analyzed our case-2022 2nd place submissions on subtask 2. we showed that a generative model could extract cause-effectsignal triplets at the competitive level using just 160 annotated samples. we investigated causal assumptions about the generation order of cause and effect to answer the research question "should cause be identified first, and only then effect, or vice-versa?" and found that while the overall f1 won't change significantly, whichever component was generated first achieved better performance on average (cause first achieved better cause-f1, and effect first effect-f1 respectively). finally, we showed the f1 difference between the dev subset with 1 or 2 causal triplets per sentence is negligible.