IDIAP ers @ Causal News Corpus 2022: Efficient Causal Relation Identification Through a Prompt-based Few-shot Approach (2022)

Abstract: In this paper, we describe our participation in the subtask 1 of CASE-2022, Event Causality Identification with Casual News Corpus. We address the Causal Relation Identification (CRI) task by exploiting a set of simple yet complementary techniques for fine-tuning language models (LMs) on a few annotated examples (i.e., a few-shot configuration).We follow a prompt-based prediction approach for fine-tuning LMs in which the CRI task is treated as a masked language modeling problem (MLM). This approach allows LMs natively pre-trained on MLM tasks to directly generate textual responses to CRI-specific prompts. We compare the performance of this method against ensemble techniques trained on the entire dataset. Our best-performing submission was fine-tuned with only 256 instances per class, 15.7% of the all available data, and yet obtained the second-best precision (0.82), third-best accuracy (0.82), and an F1-score (0.85) very close to what was reported by the winner team (0.86).

## introduction causal relation identification aims to predict whether or not there exists a cause-effect relation between a pair of events mentioned in a given text. for example, in the sentence "protests spread to 15 towns and resulted in the destruction of property", the automatic causal identification system must be able to realize that there is cause-effect relation between the events "protest" and "destruction". hence, understanding causal relations within a text is an essential aspect of natural language processing (nlp) and understanding (nlu) @xcite @xcite . once the causal information is identified within a 1 code available at https://github.com/idiap/cncsharedtask . text, such knowledge becomes beneficial for many other downstream nlp tasks, e.g., information extraction, question answering, text summarization @xcite . however, due to the ambiguity and diversity in written documents, causality identification is not easy and remains a challenging problem. the event causality identification with causal news corpus (case-2022) shared task @xcite addresses this problem on a recently created corpus named the causal news corpus (cnc) @xcite . contrary to previous existing causality corpora, the cnc dataset, manually annotated by experts, incorporates a broader set of causal linguistic constructions, i.e., not only limited to explicit constructions, resulting in a more challenging dataset. in this paper, we describe our followed methodology for addressing the causal event classification shared task (subtask 1) during the case-2022 competition @xcite ). 2 our primary method, based on a few-shot configuration, follows a prompt-based approach for fine-tuning the language model (lm). the intuitive idea of this approach is to allow the lm to directly auto-complete natural language prompts. following this technique, we leverage the lm's knowledge and let it decide the correct label of the input sequence. additionally, we evaluate the performance of ensemble techniques trained using the entire dataset available. our results demonstrate that our fewshot, prompt-based, fine-tuning approach can generalize well even when using as few as 256 samples per class for training, outperforming ensemble techniques trained with the entire dataset, as well as most of other teams' submissions. the rest of the paper is organized as follows. section 2 describes relevant related work, section 3 describes the components of our main method, namely the prompt-based approach. section 4 describes the experimental setup, i.e., datasets, additional baselines, experiments configuration and obtained results. finally, section 5 depicts our main conclusions and future work directions. ## related work previous work on causal relation identification varies from knowledge-based to deep neural network approaches (deep-nn). knowledge-based systems rely on linguistic patterns extracted using an exhaustive exploration of the data, where lexico-semantic and syntactic analysis lead to the identification of relevant structures and keywords that depict the presence of a causal relation in the text @xcite . although interpretable, these methods require a lot of human effort to generate relevant patterns and result in models that are not readily applicable in different domains. statistical machine learning (ml) approaches leave to the selected algorithm to find patterns in the data on the basis of the manual annotation. traditionally, using different nlp tools, it is possible to compute various features for a given collection and apply any ml pipeline to train a causality relation classifier, e.g., @xcite . however, one main disadvantage of these techniques is the language dependency and error propagation of the nlp tools, e.g., syntactic parsers. finally, recent approaches based on deep-nn have become popular, given their powerful representation learning ability. typical approaches include convolutional neural networks @xcite , long short-term memory networks @xcite , and pre-trained transformer-based lms such as bert @xcite , where following a standard fine-tuning approach makes possible the detection of causality relations @xcite @xcite . normally, these methods involve high computational costs and large amounts of labeled data. however, in this work, we show that pre-trained lms can still be effective even when fine-tuned with very few instances. contrary to previous work, we evaluate the effectiveness of very recent prompt-based prediction approaches under a few-shot configuration for causal relation identification. ## prompt-based approach in the "pre-train, prompt, and predict" paradigm, unlike the standard "pre-train and fine-tune" paradigm, instead of adapting pre-trained lms to downstream tasks via objective engineering, foot_1 downstream tasks are reformulated to look more like those solved during the lm pre-training phase @xcite . more precisely, promptbased prediction treats the downstream task as a masked language modeling problem, where the model directly generates a textual response (referred to as a label word) to a given prompt defined by a task-specific template @xcite . for instance, when identifying the sentiment of a movie review like "i love this movie." we may continue with "overall, it was a [mask] movie." and ask the lm to fill the mask with a sentimentbearing word. in this example, the original input text x ("i love this movie.") is modified using the template "[x] overall, it was a [mask] movie." into a textual string prompt x ′ in which the mask will be filled with a label word. some examples of label words for this example could be "fantastic" or "boring". in the case of classification tasks, in addition to defining a set of possible label words, it is necessary to define a mapping between each one and the actual output labels. for instance, if labels + andrefer to positive and negative sentiment, respectively, "fantastic" in previous example could be mapped to output label +, and "boring" to -. formally, let l be a pre-trained language model, f t (x) a function that converts the input x into a prompt by instantiating template t which contains one [mask] token, mask. let word : @xmath0, y, to the label words set, w. then, the classification task is converted to a masked language modeling (mlm) task in which the probability of predicting class @xmath1: this is [mask].[sep]the bombing created panic. this is causal.[sep]the strike is still on. this is random.[sep] roberta's mlm head input instance x:"soldiers were hurt in the attacks." template t:[x]this is [mask][sep] logits average causal (output positive class) ✓ random (output negative class) template instantiated with x demonstration for positive class demonstration for negative class f t '(x) = f t ("soldiers were hurt in the attacks.") + f t←causal (a positive instance) + f t←random (a negative instance) figure 1 : augmented prompt-based classification for causality identification task. first, the input instance @xmath2. then, these three prompts are given to a roberta model, and one logit vector is obtained for each. these vectors are then averaged, and the word with the highest score, "causal", is selected. finally, this word is mapped to its corresponding class, and x is classified as positive. note that, in this example, we have the following word-toclass label mapping word(positive) = "causal" and word(negative) = "random". when fine-tuning l to minimize the cross-entropy loss, the pre-trained weights w v are re-used, and there's no need to introduce any new parameter. on the contrary, with standard fine-tuning a taskspecific head, sof tmax(w o h @xcite ), has to be added, with new task-specific learnable parameters w @xmath3, which increases the gap between pre-training and fine-tuning. hereafter we will refer to the "causal" and "noncausal" classes as "positive" (+) and "negative" (-) respectively. in addition, and following previous work by @xcite , we append one answered prompt for each class to the input prompt as demonstrations. 4 more precisely, let @xmath4, -} be the set of labels for the binary causality identification task, let t ← v be the template t in which its [mask] token has been filled with word v, and w @xmath5, then we redefine f t (x) in equation 1 as f ′ t (x) defined as: step 1: we created a new training set, τ k , by extracting k instances per class from the original train partition, and used the remaining 2925-2×k instances as a large evaluation set δ t -k (dataset stats are given in step 2: in order to add demonstrations to a given input x (see equation 1 ), we uniformly sampled x - and x + from the top-50% most similar instances in τ k . 5 to do so, we pre-computed the sentence embeddings of training instances using a pre-trained sbert @xcite ) model, and cosine distance was used as a similarity metric. step 3: using "causal" and "random" as word labels, foot_3 the next step was to generate candidate templates automatically using t5. first, each training instance x of class y in τ k was converted to "[x]<@xmath6, and used a 100 wide beam search to decode multiple template candidates by @xmath7. step 4: next step was sorting all 100 final candidate templates by f1 score. however, since this is a time-consuming step, a subset of the evaluation set was used by sampling 256 unique positive and negative instances from δ t -k . note that no finetuning is used at this point, just the out-of-the-box pre-trained lm. step 5: we selected the top-10 best-performing templates as final candidates. for each candidate template we fine-tuned the lm as a mlm task (see equation 1 ) on the training set, τ k , evaluating it on the complete evaluation set, δ t -k . step 6: finally, the model with the best f1 score on the official dev set was selected as a candidate for submission -we also checked that the f1 score on δ t -k was among the first ones too (if not first). note that in this step we're evaluating the model on unseen data since the official dev set is being used as an unofficial test set. the above process was repeated varying the number k of training instances, with @xmath8, 356, 512, and 1000; 7 the number d of input prompts to ensemble during classification stage, with d from 1 to 9; and using roberta (large and base), and deberta v3 (base) as pre-trained lms. in step 5, models were fine-tuned for a maximum of 1000 steps using adamw @xcite ) optimizer (β 1 =0.9, β 2 =0.999, ϵ=1e-8) with a learning rate of γ=1e-5 with no weight de-7 inspired by evidence showing a performance saturation when @xmath9, compared to standard fine-tuning on the entire dataset, we decided to start from this value. ## results & discussion in this section we provide the details of the employed dataset, a set of additional experiments based on recent ensemble techniques, and the final configuration of our submitted runs to the subtask 1 of case 2022. ## conclusions this paper describes our participation in the case-2022 subtask 1. our proposed approach uses a fewshot configuration in which a prompt-based model is fine-tuned using only 256 instances per class and yet was able to obtain remarkable results among all 16 participant teams. the comparison against traditional fine-tuning techniques, ensemble approaches, as well as the other participating models, show the potential of the proposed approach for better generalizing the posed task. for future work, we plan to perform further ablation studies when we have access to test set ground truth labels. for instance, measuring the dev-to-test performance drop in relation to k or the robustness against different training and demonstration sampling given a fixed k.