1 C ademy @ Causal News Corpus 2022: Leveraging Self-Training in Causality Classification of Socio-Political Event Data (2022)

Abstract: This paper details our participation in the Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE) workshop @ EMNLP 2022, where we take part in Subtask 1 of Shared Task 3 (CITATION). We approach the given task of event causality detection by proposing a self-training pipeline that follows a teacher-student classifier method. More specifically, we initially train a teacher model on the true, original task data, and use that teacher model to self-label data to be used in the training of a separate student model for the final task prediction. We test how restricting the number of positive or negative self-labeled examples in the self-training process affects classification performance. Our final results show that using self-training produces a comprehensive performance improvement across all models and self-labeled training sets tested within the task of event causality sequence classification. On top of that, we find that self-training performance did not diminish even when restricting either positive/negative examples used in training. Our code is be publicly available at https://github.com/Gzhang-umich/1CademyTeamOfCASE .

## introduction task 1 of the case workshop @ emnlp 2022 works to identify and classify event causality in socio-political event (spe) data, with subtask 1 being a binary classification of causality. in other words, participants are tasked with answering: does an event sentence contain cause-effect meaning? the workshop provides data from causal news corpus (cnc) @xcite for training and evaluation of the subtask. causality itself aims to identify a semantic relationship between two events where one event (the cause) is responsible for the production of the other event (the effect). utilizing the cnc dataset serves as a benchmark for participants to evaluate the ability of a given model or process to identify causality in event data. we approach the problem of causality sequence classification by applying self-training @xcite @xcite as a means to improve the performance of language models in this task. the goal of selftraining is to generate proxy labels for previously unlabeled data to enhance the learning process. the self-training process works by iteratively labeling previously unpredicted data, and then using the new pseudo-labels as truthful labels in the next training stage. the intuition behind self-training comes from the fact that it can pseudo-expand the training space to basically an unlimited size in a very cheap manner, as no hand-labeling is required in the process. additionally, we run supplementary experiments to test the effectiveness of self-training against various transformer-based data augmentation techniques @xcite and separate multi-task learning approaches @xcite that we originally designed for the competition. the description and results of these additional experiments can be found in the appendix. in summary, our main contributions are as follows. ## conclusion and discussion this paper explores how training a classifier on self-labeled data can improve the performance of sequence classification tasks. in our case, we examine the effect of self-training on the task of event causality in socio-political event data as part of subtask 1 of shared task 3 of the case workshop @ emnlp 2022. our results show that training a classifier on selflabeled data using a teacher-student approach comprehensively improves task performance. furthermore, we find that performance improvement from self-training did not differ significantly between self-labeled training sets with varying levels of example polarity. this indicates that the model is capable of reaping the full benefits of self-training despite having limited access to positive or negative samples. one thing that could help explain this is our relatively high threshold t of 0.9 which determines whether or not to keep an example during the next, given that our self-labeled examples are gathered from an assortment of articles from wikipedia, it should be well noted that the benefits of self-training are apparent even when the selflabeled examples are not domain specific to the original labeled data. we decide to use wikipedia as the source of our self-labeled examples as we view it as a more accessible source with far greater amounts available unlabeled data. thus, our findings indicate that performance improvements from self-training work with non-domain specific data, which alleviates us from the restriction of confining our self-labeled data to the single domain of the original labeled data. finally, one more aspect of our experiments that should be further explored is the classifier's actual dependence on the self-labeled data versus the originally provided training data. in our setup, we choose to train our models on one epoch of selflabeled data and then on five epochs of the original training data in order to prioritize the true labeled training data. we believe that it would be worthwhile to explore training classifiers with a higher training priority on the self-labeled data, or even to test the performance of classifiers trained solely on the self-labeled data, without the original true data. event detection classification head (pre-trained) event detection classification head (pre-trained) entailment detection classification head (pre-trained) entailment detection classification head (pre-trained) causality classification head linear layer final causality prediction final causality prediction architecture 1 architecture 2 ... ... i 1 i 2 i n i 1 i 2 i n task-specific head outputs linear layer figure 2: multi-task learning architectures used in supplementary testing.