N oisy A nnot@ Causal News Corpus 2022: Causality Detection using Multiple Annotation Decisions (2022)

Abstract: The paper describes the work that has been submitted to the 5th workshop on Challenges and Applications of Automated Extraction of socio-political events from text (CASE 2022). The work is associated with Subtask 1 of Shared Task 3 that aims to detect causality in protest news corpus. The authors used different large language models with customized cross-entropy loss functions that exploit annotation information. The experiments showed that bert-based-uncased with refined cross-entropy outperformed the others, achieving a F1 score of 0.8501 on the Causal News Corpus dataset.

## introduction a causal relationship in a sentence implies an underlying semantic dependency between the two main clauses. the clauses in these sentences are generally connected by markers which can have different parts of tags in the sentence. moreover, the markers can be either implicit or explicit and for these reasons, one cannot rely on regex or dictionary-based systems. thus, there is a need to investigate the context of the sentences. for the given task, we exploited different large language models that provide a contextual representation of sentences to tackle causality detection. shared task 3 in case-2022 @xcite ) aims for causality detection in news corpus, which can be structured as a text classification problem with binary labels. pre-trained transformer-based models @xcite have shown success on tackling a wide range of nlp tasks including text generation, text classification, etc. the authors look into inter-annotation agreements and number of experts and how they can be included in the loss to improve the performance of the pre-trained models. the main contributions of the paper are as follows: the remaining paper is formulated as follows: section 2 reviews the related work, section 3 describes the dataset on which the work has been done, section 4 discusses the methodology used in the paper, the following section discusses the results and provides an ablation of the various loss functions introduced and finally, section 6 concludes the paper and suggests future works. ## related work multiple annotations on a single sample reduce the chances of the labelling to be incorrect or bias being incorporated into the dataset @xcite . including multiple annotators also leads to disagreement among the labels that have been provided by them. the final or gold annotation is then usually determined by majority voting @xcite or by using the label of an "expert" @xcite . there are also different methodologies which do not use majority voting to select the "ground truth". expectation maximization algorithm has been used to account for the annotator error (dawid and @xcite . entropy metrics have been developed to identify the performance of the annotators @xcite @xcite . multi-task learning is also used to deal with disagreement in the labels @xcite @xcite . there are methods which include the annotation disagreement into the loss function for part of speech tagging @xcite on svms and perceptron model. the present work considers the inter-annotator agreement as well as the number of annotators into the loss function for any model. the work also compares the performance when the annotators who disagree with the majority voting has been ignored. ## dataset the causal news corpus dataset @xcite consists of 3,559 event sentences extracted from protest event news. each sample in the dataset contains the text, the corresponding label, the number of experts who annotated the label and the degree of agreement among the experts. figure 1 shows a sample from the provided training set. the training data is fairly balanced, containing 1603 sentences with a causal structure and 1322 sentences without a causal structure. also, the number of causal and non-causal sentences in the validation set does not differ significantly. finally, 311 news articles have been used as test set for evaluation. besides the binary labels, the causal news corpus dataset also provides additional information regarding the number of experts who labeled the sentence and the percentage of agreement between them. figure 1 shows that the number of experts who annotated the text "the farmworkers'strike resumed on tuesday when their demands were not met." is 3 (@xmath0. also, all of the experts labeled the sentence to be causal so the agreement is 1.0 (100% agreement) and the label is 1. in case only one of three experts assigned label 1 to the previous text, the three predictors num_votes, agreement, label would now become 3, 2 3 , 0 respectively. in this paper, the authors exploit this information to give the model more prior and thus potentially improve the model's performance, which has been described in more detail in section 4. ## methodology the section discusses the pipeline, the different types of loss functions that were implemented, and the experimental details that have been used in the third shared task for case 2022 @xcite . ## results and discussion in this section, the results of the different models and the different losses are discussed. the models with noisy and refined cross-entropy loss utilizes the annotated information and thus performs better. the noisy cross-entropy loss is similar to restricting the highest probability output that a model can predict. however, in almost all cases, the degree of agreement was either 1 or 2 3 . in general, the smooth labelling has a value in the range of 0.9 to 1. different contradicting annotations of labels might make the model face difficulties in learning and yielding an accurate prediction for each sentence. the refined cross-entropy solely considers the labels that do not contradict each other, thus it performs the best. moreover, the experiments show that robertabased models achieve lower performance compared to bert-based models, especially bert-baseuncased models. the model pretrained on bertlarge-cased has been fine-tuned for only one epoch due to computation limitations. their f1 scores are worse than those of bert-base-cased and bert-baseuncased models. bert-base models result in better performance, as compared to models fine-tuned on roberta-base. the reason could be that robertabased models had not been trained on next sentence prediction (nsp) while bert-based models were. causality detection can benefit from nsp. a sentence can be considered to be two relevant clauses that are joined by a causal effect. thus, knowing if the clauses are relevant or not benefits the task of causality detection. figure 3 shows the confusion matrix resulting from bert-base-uncased models which result the best f1 scores in all implemented models. models are generally good at predicting non-causal sentences regardless of the loss function used. in fact, true negatives and true positives are always the highest measures compared to the others. on the other hand, there is a clear trend in the number of true positives when we shift the loss function from vanilla to noisy and refined cross-entropy. in particular, the model yields 145 true positives and is improved to 152 and 149 true positives when we replaced vanilla cross-entropy loss with noisy and