NLP 4 ITF @ Causal News Corpus 2022: Leveraging Linguistic Information for Event Causality Classification (2022)

Abstract: We present our submission to Subtask 1 of theCASE-2022 Shared Task 3: Event CausalityIdentification with Causal News Corpus as partof the 5th Workshop on Challenges and Applicationsof Automated Extraction of SociopoliticalEvents from Text (CASE 2022) (Tanet al., 2022a). The task focuses on causal eventclassification on the sentence level and involvesdifferentiating between sentences that include acause-effect relation and sentences that do not. We approached this as a binary text classificationtask and experimented with multiple trainingsets augmented with additional linguisticinformation. Our best model was generated bytraining roberta-base on a combination ofdata from both Subtasks 1 and 2 with the additionof named entity annotations. During thedevelopment phase we achieved a macro F1 of0.8641 with this model on the development setprovided by the task organizers. When testingthe model on the final test data, we achieved amacro F1 of 0.8516.

## introduction causal event classification can be categorized as a part of the natural language processing (nlp) task of event extraction. when extracting event information from text, the general aim is to identify answers to the 5w1h questions (who, what, when, where, why, how; @xcite . some of the questions can be answered easily by means of open source nlp tools-a named entity tagger can facilitate the extraction of locations (where) and times or dates (when), for example. however, some event information remains more difficult to identify reliably in texts, such as answers to why questions, which is also the type of question that causal event classification addresses. this task presents an opportunity to develop models that detect information about the reason behind a particular event. for this process, a binary classifier is used to determine whether a cause-effect relation is present in the input sentence. in an nlp pipeline, the output of such a classification process is often used as input for a span detection system, which identifies the particular cause and effect text spans in each causal sentence. as described by @xcite , causality can be expressed either explicitly or implicitly. the authors illustrate this by providing the following examples: example 1 displays explicit causality, made apparent by the presence of the causal marker "due to". the organizers of the current shared task also refer to this marker as the signal. in contrast, the causal relation between the sit-in and worker dissatisfaction in example 2 is implicit, as the sentence does not contain a causal marker. in their survey of causal relation extraction in natural language texts, yang et al. (2022) emphasize the potential of domain-specific pre-trained models in combination with graph-based models. they also stress the importance of leveraging linguistic information in order to identify both implicit and explicit causal relations. for this reason, the current study focuses primarily on experiments regarding the integration of linguistic information in the training data, to be used as input for the fine-tuning of pre-trained transformer models. the remainder of this paper is structured as follows: section 2 introduces the shared task and the dataset. in section 3, we describe the training process, model configuration details, and the linguistic dataset alterations that we tested. results are presented in section 4 and discussed in section 5, followed by concluding remarks and a summary of our findings in section 6. ## dataset and task the case-2022 shared task 3 on event causality identification is divided into two subtasks. the data provided by the organizers stems from the causal news corpus, a collection of 3,559 english annotated event sentences from 869 news articles about protests @xcite . the goal of subtask 1 is to determine whether an event sentence contains a cause-effect relation. subtask 2 is concerned with identifying the spans that correspond to cause, effect, or signal in each causal sentence. we developed and submitted models for the first of these two subtasks. for the development phase, the task organizers provided a training dataset consisting of 2925 training instances. sentences with the label 0 (@xmath0, while sentences with the label 1 included a causal relation and were in the majority (@xmath1. in addition, an unlabeled development set of 323 sentences was made available in order to allow for model testing via the codalab submission portal. preliminary exploratory analysis of the data provided for the development phase revealed an average inter-annotator agreement of 88.27% for causal sentences, while sentences labeled as containing no causal relation had an average agreement of 77.89%. between 1 and 3 annotators labeled each sentence, coming to a consensus of 100% agreement for 70.31% (@xmath2.35% (@xmath3. for the test phase, the previously unlabeled development set was re-released with annotations so that it could be used as additional training data. an unlabeled test set of 311 previously unseen sentences was made available for the final testing and scoring process. ## methodology we fine-tuned pre-trained language models (plms) on the training data and adjusted the model hyperparameters accordingly. we then tested four different methods of augmenting the training data with linguistic information and compared their efficacy. ## discussion the discrepancies between the scores for the development and final testing phases call for a closer investigation of the model input and output. the results from the development phase suggest that model performance increases when training data includes linguistic information in the form of 1) named entity annotations for the per and loc classes, or 2) as a combination of both per named entity annotations and cause-effect-signal spans. adding only cause-effect-signal spans, however, appears to have had a negative impact on model test scores. the fact that the roberta+per+loc model outperformed the roberta+per+ces model also suggests that named entity information may prove more useful than cause-effect-signal spans. it is frequently the case that named entities of type per, i.e. proper nouns, have the semantic role of agent or patient in a sentence. replacing these nouns, along with location names, with named entity tags distills this important information and reduces the number of superfluous words in the data. we suggest that this creates a clearer pattern for the model, which in turn improves performance. in the final test phase, however, only roberta+per+ces outperformed our established roberta baseline by a small margin, while roberta+per+loc and roberta+ces had the lowest macro f1 scores. according to these results, it seems that adding linguistic information to the training data in the form of named entity annotations or cause-effect-signal spans only leads to minute increases in model performance. it may be that our roberta baseline model is able to extract this particular linguistic information on its own without the need for additional feature engineering. further experimentation with linguistic features is needed in this area. with only 2925 sentences, the size of the original amount of training data is also a potential factor that affected model performance. more training data would most likely increase model performance. a closer investigation of the data revealed that some annotations also leave room for discussion, such as the sentence in example 8: the sentence is labeled as belonging to the positive class, but we were unable to identify a cause-effect relation. this shows that identifying causality can pose difficulties for expert human annotators. such instances may negatively influence the detection of causal patterns during training. interestingly, re-training our best model with the additional training data in the final testing phase did not improve performance. furthermore, the testing data used for evaluation in the development phase appears to consist of sentences from only two news articles. the data basis for development was accordingly very homogeneous and most likely did not provide an accurate representation of all possible articles that the model might need to classify in a real-world application. optimization based on homogeneous data can lead to a preference for models that work well with that specific data but fail to generalize to more diverse data. the difference in model performance between the development and test phases might be evidence of this phenomenon. ## conclusion training data-including the source, domain, amount, and any added features-plays an important role when it comes to model optimization for nlp tasks, and the subfield of event causality is no exception. our findings show that the generalizability of a model depends heavily on the quality and content of the model input. in our case, adding linguistic information to the training data only led to a minute increase in model performance as compared to our established roberta baseline. it is possible that a larger training dataset would improve results. in addition, a larger, more diverse testing dataset is necessary in order to adequately evaluate the robustness of the model and predict its effectiveness for real-world applications. future directions might also include a greater focus on strategies for the identification of implicit cause-effect relations.