m CPT at S em E val-2023 Task 3: Multilingual Label-Aware Contrastive Pre-Training of Transformers for Few- and Zero-shot Framing Detection (2023)

Abstract: This paper presents the winning system for the zero-shot Spanish framing detection task, which also achieves competitive places in eight additional languages. The challenge of the framing detection task lies in identifying a set of 14 frames when only a few or zero samples are available, i.e., a multilingual multi-label few- or zero-shot setting. Our developed solution employs a pre-training procedure based on multilingual Transformers using a label-aware contrastive loss function. In addition to describing the system, we perform an embedding space analysis and ablation study to demonstrate how our pre-training procedure supports framing detection to advance computational framing analysis.

## introduction approaches for computational framing detection are diverse @xcite , as the framing concept itself is often just casually defined @xcite . consequently, framing detection is challenging on its own, but also suffers from a lack of sufficient data @xcite , especially in multilingual settings. the semeval 2023 task 3 subtask 2 @xcite aims at predicting 14 distinct media frames @xcite present within news articles in 9 languages. due to label imbalances, as a result of the high dimension of the label space compared to the number of samples, traditional paradigms, e.g., per-label binary classification, do not apply well to the given setting without adaptions @xcite . we introduce mcpt, the label-aware contrastive pre-training of transformers based on a multilingual encoder model (original team name on the leaderboard 1 : polarice). we exploit two features of the task: (i) multi-label information and (ii) multilingual data for pre-training. first, we leverage the label information by adopting a contrastive loss function, i.e., @xcite , for natural language processing that optimizes the embedding space with respect to the similarities of the label space. therefore, samples with more similar labels occupy similar regions in the embedding space, whereas mostly dissimilar samples regarding their shared labels are pushed apart (refer to figures 1 top and 2a ). second, we design a custom two-phase procedure with multiple stages for multilingual training to maximize the available data (see figure 1 ). in phase one, we train on all languages, while in phase two, we further fine-tune the model on the target language if such data exist i.e. few-shot setting, or continue training on all languages if not i.e. zeroshot setting. our system performs competitively (top 10) on all six few-shot (i.e., english, german, french, italian, polish, and russian) and three zero-shot (i.e., spanish, greek, and georgian) settings, beating the baselines on all languages. on spanish, which is the only zero-shot language with a common language family and alphabet as the training languages, our system is the winning contribution with a micro-f1 of 0.571 (compared to 0.120 of the baseline). therefore, we argue that our system generalizes well to unseen data, even when no training data is available in similar target languages. in sum, our contribution is three-fold foot_0 : c1 we adopt a multi-label contrastive loss function for natural language processing to optimize the embeddings of textual data. c2 we describe a two-phase multi-stage training procedure for multilingual scenarios with limited data, i.e., few-and zero-shot predictions. c3 we demonstrate the effectiveness of our winning system for framing detection supported by embedding and ablation studies. framing detection. according to @xcite , to frame is to select and emphasize some aspects of reality to encourage particular interpretations. that is, messages centered around a common topic may draw the receiver's attention to distinct features, thus suggesting different courses of action, causal interpretations, etc. as such, computational framing detection requires natural language processing (nlp) methods that capture nuances of how content is presented rather than just what topic is present. therein, studies focus on detecting vastly different conceptualizations of framing, such as blame frames @xcite , war frames (wicke and bolognesi, 2020), moral frames @xcite , or media frames @xcite . regarding media frames, boydstun et al. (2013) identified a set of relevant frames that formed the basis for the media frame corpus @xcite . within this supervised frame detection scenario, @xcite indicate that transformer-based approaches vastly outperform approaches using less powerful architectures such as lstms. as such, we also employ transformer models with label-aware contrastive pre-training. supervised contrastive learning. contrastive learning, originally mainly used in computer vision settings (e.g., @xcite , has recently found increased attention in the nlp research community due to its efficacy on tasks with limited amounts of data and its applicability to transformer embeddings (e.g., @xcite . the general concept of supervised contrastive learning @xcite is that latent representations (or embeddings in nlp) of samples with the same labels should be close in embedding space, while samples with different labels should be further apart. su et al. (2022) and zheng et al. (2022) have independently proposed contrastive learning methods for multi-label settings that weight similarities of samples by the similarity of their label vectors, i.e. hidden representations of samples with similar label vectors should be more similar than hidden representations of samples with less similar label vectors. su et al. (2022) weight a euclidean distance-based measure of embeddings by a normalized dot product of the label vectors. herocon (zheng et al., 2022) generalizes supervised contrastive loss (khosla et al., 2020b) and beats previous state-of-the-art contrastive learning paradigms in multi-label settings on multiple image data sets. tunstall et al. (2022) introduce setfit, an algorithm for the data-efficient fine-tuning of sentence embeddings, primarily on binary labels. setfit first fine-tunes sentence embeddings in a contrastive manner before training a classification head. we combine the idea of the contrastive pretraining stage from setfit and adopt herocon for nlp loss to improve performance on multi-label datasets. ## methods at the core of our system lies a multilingual transformer model with dense neural layers comprising the head. contrastive fine-tuning is performed as part of a multi-stage training procedure. ## experiments we now present the results of our mcpt system, supported by embedding space and ablation studies (c3). we base mcpt on the multilingual foot_1 sentence transformer model paraphrase-multilingual-minilm-l12-v2 @xcite and demonstrate that competitive results can be achieved with a relatively small amount of parameters, i.e., 117m parameters (wang et al., 2020), given a training method tailored to the task. the model was chosen for its sentence embedding performance on multiple languages and its small size relative to similar state-of-the-art multilingual transformer models. our model architecture comprises mean-pooling, no normalization of embeddings, a dense head with one hidden layer of size 256, and a dropout of 0.5. we train the model with separate learning rates for the classification head (1e-3) and the body (2e-5), a weighting parameter α of 0.01, a batch size of baseline models. we compare the performance of our system against two baselines. first, we consider the results of the official baseline base (i.e., n-grams and support vector classification; @xcite . second, we compare against setfit @xcite with the same base encoder as ours, on the post-challenge test set (details in appendix b). ## conclusion in this paper, we describe our system (mcpt) for the framing detection shared task @xcite . we introduce an approach based on a labelaware contrastive loss and training procedure for transformers to deal with the challenges of multilingual multi-label prediction with few or even zero samples. the generalization ability of our system is demonstrated by providing the winning contribution for the spanish framing detection subtask where no training samples were available foot_2 . hence, we believe that our system is a notable advancement for computational framing research. lewis tunstall, nils reimers, unso eun seo jo, luke bates, daniel korat, moshe wasserblat, and oren pereg. 2022. efficient few-shot learning without prompts. arxiv preprint arxiv:2209.11055. wenhui wang, furu wei, li dong, hangbo bao, nan yang, and ming zhou. 2020. minilm: deep selfattention distillation for task-agnostic compression of pre-trained transformers. advances in neural information processing systems, 33:5776-5788. philipp wicke and marianna m bolognesi. 2020. framing covid-19: how we conceptualize and discuss the pandemic on twitter. plos one, 15(9):e0240010. ben wu, olesya razuvayevskaya, freddy heppell, joão a. leite, carolina scarton, kalina bontcheva, and xingyi song. 2023. sheffieldveraai at semeval-2023 task 3: mono and multilingual approaches for news genre, topic and persuasion technique classification. in proceedings of the 17th international workshop on semantic evaluation, semeval 2023, toronto, canada. lecheng zheng, jinjun xiong, yada zhu, and jingrui he. 2022. contrastive learning with complex heterogeneity.