M ars E clipse at S em E val-2023 Task 3: Multi-lingual and Multi-label Framing Detection with Contrastive Learning (2023)

Abstract: This paper describes our system for SemEval-2023 Task 3 Subtask 2 on Framing Detection. We used a multi-label contrastive loss for fine-tuning large pre-trained language models in a multi-lingual setting, achieving very competitive results: our system was ranked first on the official test set and on the official shared task leaderboard for five of the six languages for which we had training data and for which we could perform fine-tuning. Here, we describe our experimental setup, as well as various ablation studies. The code of our system is available at https://github.com/QishengL/SemEval2023 .

## introduction framing involves highlighting certain elements of the perceived reality over others, which can promote problem definition, causal interpretation, moral evaluation, and/or treatment recommendation @xcite . for example, events such as the outbreak of the covid-19 pandemic can be discussed from different perspectives: health, economic, political, legal, quality of life, etc. recently, machine learning has led to significant progress in framing detection. previously, naïve bayes, support vector machines, and deep learning models based on recurrent neural networks (rnns), long short-term memory networks (lstm), and transformers have been used to analyze large amounts of text data and to capture complex patterns of framing @xcite . here, we describe our system for semeval-2023 task 3 @xcite subtask 2 on framing detection, which is defined as a multi-label text classification task at the article level. the participants were required to build a system that can identify the frames used in an input article. below, we describe our approach, which combines pretrained language models (plms) and contrastive learning. our contributions are as follows: ## related work contrastive learning has been used in computer vision for many years. it works as follows: a random example is picked as an anchor, and then positive and negative examples are selected with respect to that anchor. next, the positive examples are pulled towards the anchor, while the negative ones are pushed away from it. simclr is a popular contrastive learning model @xcite , introduced in computer vision. it uses two views of the same image, generated using augmentations, and these are considered as positive examples. all other images are negative examples. this is a semi-supervised model, and it does not need labeled data for training. the result of this training is a good encoder with a strong capability for contrastive tasks. then, the parameters are frozen, and the model is fine-tuned for a downstream task for which labeled data is available. the loss function for contrastive learning is shown as equation 1 below. here, i denotes an image, j is a positive example for i, k denotes all other images, τ is a temperature parameter, and l is an indicator function whose value is 1 if k ̸ = i and it is 0 if @xmath0. after simclr, a supervised contrastive learning model based on simclr was proposed by @xcite . in supervised contrastive learning, each image is labeled. images with the same labels are considered as positive examples, while such with different labels are taken as negative examples. the loss function is modified according to equation 2 below, where p (i) is the set of all positive examples for i, and a(i) is the set of all negative examples for i. ## system overview our system uses xlm-roberta as the backbone, and implements a multi-lingual and multi-label contrastive model on top of it. xlm-roberta is well-suited for handling multi-lingual data. we trained a general model using data from various languages, with the aim of grouping examples with similar labels together, even if they are in different languages, and pushing apart examples with dissimilar labels. this strategy may allow the model to more easily identify a better decision boundary, as the number of clusters decreases. moreover, by using contrastive loss, we expect increased robustness compared to models trained on just one language; it also helps fight class imbalance. ## ablation study in ## conclusion and future work we have described our system for semeval-2023 task 3 subtask 2, which used a combination of a plm encoder and contrastive loss. on the official test set, our system was ranked 1st for five out of the six languages for which training data was available and thus we could perform fine-tuning, and we were ranked 3rd for english. we further explored alternative options for each component of our system and showed the individual contributions of each component. we should note though that, while our system was the best overall on the six languages with training data, our results were not as good for the three zero-shot languages. in future work, we plan to explore ways to improve our model in a zero-shot learning scenario. we further plan to apply our model to other tasks, e.g., to subtask 3 of this semeval-2023 task 3. it would be also interesting to experiment with adapters and to study their interaction with contrastive learning.