CSECU - DSG @ Causal News Corpus 2023: Leveraging R o BERT a and D e BERT a Transformer Model with Contrastive Learning for Causal Event Classification (2023)

Abstract: Cause-effect relationships play a crucial role in human cognition, and distilling cause-effect relations from text helps in ameliorating causal networks for predictive tasks. There are many NLP applications that can benefit from this task, including natural language-based financial forecasting, text summarization, and question-answering. However, due to the lack of syntactic clues, the ambivalent semantic meaning of words, complex sentence structure, and implicit meaning of numerical entities in the text make it one of the challenging tasks in NLP. To address these challenges, CASE-2023 introduced a shared task 3 task focusing on event causality identification with causal news corpus. In this paper, we demonstrate our participant systems for this task. We leverage two transformers models including DeBERTa and Twitter-RoBERTa along with the weighted average fusion technique to tackle the challenges of subtask 1 where we need to identify whether a text belongs to either causal or not. For subtask 2 where we need to identify the cause, effect, and signal tokens from the text, we proposed a unified neural network of DeBERTa and DistilRoBERTa transformer variants with contrastive learning techniques. The experimental results showed that our proposed method achieved competitive performance among the participants’ systems.

## introduction a causal relation is a semantic relationship between two arguments known as cause and effect, where the occurrence of one (cause argument) incurs the occurrence of the other (effect argument). causal relation extraction from text is also known as the study of causality extraction (ce) which gain attention in different domains including biomedical, media, emergency management @xcite @xcite , etc. such causal relation plays an important role in various contemporary nlp tasks including question-answering (q/a), product recommendation based on user comments, and other textual entailments @xcite . however, the implicit causal relationship between sentences, numerical connectives impact, and ambivalent semantic meaning of the text make ce one of the most challenging tasks in nlp. ## proposed method in this section, we describe our proposed approach for case 2023 task 3, subtask 1 and subtask 2. the overview of our proposed framework for subtask 1 is depicted in figure 1 . to extract the diverse contextual features from the text, we employ two transformer models including deberta @xcite and one of roberta variants twit-ter_roberta @xcite . later, a linear feed-forward layer is utilized in each model to estimate the probability score of each class. finally, for the effective fusion of the scores, we take the weighted arithmetic mean of the prediction scores of these models. a class that contains the highest probability scores is considered the final label. on the other hand for subtask 2 we utilized two different transformer models deberta and dis-tilroberta independently to exploit cause-effect and signal span features respectively. then we concatenate both transformers model features and feed to a stacked bilstm network to distill long-term relations among the tokens. followed by the bil-stm network we incorporate a max-pooling and classifier layer to predict tokens label. to improve system performance we calculate the contrastive loss for cause-effect token classification whereas we utilized cross-entropy loss for signal token classification since it may or may not contain in text. however, figure 2 illustrates our proposed method for subtask 2. ## experiment and evaluation in this section, we now describe the dataset and hyper-parameters settings with the finetuning strategy that we have employed to design our proposed system for the case 2023 shared task 3. ## conclusion and future work in this paper, we present two approaches to identifying causal events and extraction of causal relations from text. for the identification task, we proposed a unified neural network of two finetuned transformer models including deberta and twitterroberta with a late-fusion technique. similarly, for the extraction task, we utilize two transformers models but this time we incorporate the distilroberta model instead of the twitterroberta. here, we design our model differently, we use deberta with contrastive learning to train the cause-effect spans of text whereas distilroberta is used to train the signal span. then we utilized an early fusion technique and pass the fused features to max-pooling and the final classifier label to get the predictions. in the future, we intend to explore the challenges of nested causality extraction task where we will design a model to predict the multi-label of a single token at a time.