I nteros ML @Causal News Corpus 2023: Understanding Causal Relationships: Supervised Contrastive Learning for Event Classification (2023)

Abstract: Causal events play a crucial role in explaining the intricate relationships between the causes and effects of events. However, comprehending causal events within discourse, text, or speech poses significant semantic challenges. We propose a contrastive learning-based method in this submission to the Causal News Corpus - Event Causality Shared Task 2023, with a specific focus on SubTask1 centered on causal event classification. In our approach we pre-train our base model using Supervised Contrastive (SuperCon) learning. Subsequently, we fine-tune the pre-trained model for the specific task of causal event classification. Our experimentation demonstrates the effectiveness of our method, achieving a competitive performance, and securing the 2nd position on the leaderboard with an F1-Score of 84.36.

## introduction understanding the intricate relationships between cause and effect within events is a fundamental aspect of language comprehension. causal events, which provide insights into these connections, present semantic challenges when it comes to their classification and analysis in discourse, text, or speech. we tackle the specific problem of causal event classification in subtask 1 of the causal news corpus -event causality shared task 2023 @xcite in our submission. this task involves accurately identifying and categorizing causal events, which plays a vital role in unraveling the underlying mechanisms behind real-world phenomena. successful classification enables a wide range of applications, such as information extraction, summarization, and knowledge graph construction. to address this challenge, we propose an innovative approach that leverages supercon learning and source-aware sampling. contrastive learning has shown promising results in computer vision to learn a better and robust visual representations @xcite and various natural language processing task like knowledge graph embeddings @xcite ), text classification @xcite , entity linking @xcite and entity resolution @xcite etc. it allows the models to learn by contrasting positive and negative pairs, capturing informative representations. the use of contrastive learning in text classification has been investigated in various contexts. for instance, the study by @xcite employed self-supervised learning techniques to address event causality identification in scenarios with limited annotated datasets. similarly, @xcite took an approach to incorporate contrastive learning with synthesized counterfactuals for data augmentation, demonstrating notable improvements in aspects such as counterfactual robustness, cross-domain generalization. in this paper we apply the idea of supercon learning introduced by @xcite to the causal event classification task. further, we loosely connect to the idea of source-aware sampling strategy introduced by @xcite and modify it to suite the classification subtask for pre-training the base encoder architecture. our methodology involves pre-training a transformer based encoder model using supercon loss with naive source-aware sampling, followed by fine-tuning the pre-trained model on the causal event classification task. through extensive experimentation and evaluation on the causal news corpus dataset, we demonstrate the effectiveness of our approach. this paper's contributions can be summarized as follows: (1) introducing contrastive learning as a method for causal event classification. (2) achieving competitive performance in the causal news corpus -event causality shared task 2023, the violence in yuen long has heightened tensions and fears of further attacks nhai case : cops fail to produce accused in court negative samples "that night , doordarshan telecast news of protests the nia wanted to catch akram alive as he could provide clues in resolving the blast case positive samples input batch positive representation negative representation supervised contrastive learning the nia wanted to catch akram alive as he could provide clues in resolving the blast case phase-1 trained roberta encoder feed-forward layer causal event prediction fine-tuning classification architecture nhai case : cops fail to produce accused in court the violence in yuen long has heightened tensions and fears of further attacks input batch phase-1 phase-2 roberta encoder with the f1 score of 84.36. foot_0 ## methodology in this section, we present the methodology employed to address the causal event classification task. our approach utilizes contrastive learning and consists of two main phases (figure 1 ): (1) pre-training the baseline transformer architecture with supercon, and (2) fine-tuning the pre-trained model on the downstream classification task. for the encoder architecture, we adopt the roberta base model foot_1 which has been shown to achieve strong results across different benchmark tasks @xcite . ## results and analysis this section presents the outcomes of our model architecture in the context of the causal event classification task. we conducted the model training on an a10 gpu with 24gb ram, utilizing the available computational resources effectively. ## conclusion and future work in this study, we have delved into the application of supercon learning for the task of causal event classification. by harnessing the power of super-con, our model achieved competitive performance, securing the 2nd position in key evaluation metrics such as f1-score, accuracy, and matthew's correlation coefficient (mcc). these competitive results provide strong evidence for the efficacy of our approach in comprehending intricate causal relationships within textual data. additionally, our comparative analysis highlights the model's learning strength and the benefits of this learning approach. in the future we could explore the use of a large dataset from a distinct domain during the pre-training phase. this would enable us to gauge the inductive capacity of our learning paradigm on the causal news corpus domain dataset. such investigations hold the potential for promising implications in the realms of low-resource, few-shot, and domain-specific causality event understanding.