Retrofitting Light-weight Language Models for Emotions using Supervised Contrastive Learning (2023)

Abstract: We present a novel retrofitting method to induce emotion aspects into pre-trained language models (PLMs) such as BERT and RoBERTa. Our method updates pre-trained network weights using contrastive learning so that the text fragments exhibiting similar emotions are encoded nearby in the representation space, and the fragments with different emotion content are pushed apart. While doing so, it also ensures that the linguistic knowledge already present in PLMs is not inadvertently perturbed. The language models retrofitted by our method, i.e., BERTEmo and RoBERTaEmo, produce emotion-aware text representations, as evaluated through different clustering and retrieval metrics. For the downstream tasks on sentiment analysis and sarcasm detection, they perform better than their pre-trained counterparts (about 1% improvement in F1-score) and other existing approaches. Additionally, a more significant boost in performance is observed for the retrofitted models over pre-trained ones in few-shot learning setting.

## introduction despite the emergence of powerful models like gpt4 and lamda, lightweight pre-trained language models (plms) such as bert and roberta remain relevant since they are computationally less expensive and open. being trained on massive amount of data available over the web, these models are extremely good at encoding general language properties, resulting in highly accurate word, sentence, and paragraph representations. these representations are then typically fine-tuned for given tasks using task-specific datasets. however, the representations learned using these plms are general purpose, and they do not capture the affective aspects (such as emotions, affects, etc.) of human communication well. consider a few exemplar pairs of text fragments, along with the emotion they evoke, in one straightforward way to inject emotions into plms is through transfer learning by fine-tuning them on emotion recognition task. multi-task learning is also a potential alternative wherein the affective task of interest is paired with the emotion recognition task and optimized jointly. however, these approaches are not that effective in capturing emotion aspects (see section 5). the community has focussed on using resources such as sentiment lexicons, emoticons, etc., to impart affective aspects into plms, by modifying pre-training objectives such as masked language modeling @xcite and next sentence prediction @xcite . these approaches either learn plms from scratch or continually pre-train them on domain-specific corpora. a few works @xcite have also explored attention-based network modules over contextualized embeddings to capture affective semantics. the approaches mentioned above, however, are highly sensitive to the training corpus and the lexical resources used, and do not generalize well across tasks. retrofitting methods that use external knowledge to improve representations have been well explored in the community for static embeddings @xcite . there are also attempts to retrofit plms to learn robust contextualized word embeddings text fragment 1 text fragment 2 bert ours worst advice ever. this would piss me off. -anger she sounds abusive and narcissistic. delete your facebook, hit the gym and kill her. -anger 0.2013 0.1085 i love how this sub is gender less -joy 0.0702 0.5254 i'm so glad i watched this entire thing. that's incredible. -joy glad to see we are still around :). -joy 0.1063 0.1746 i'm sorry it happened like that for you. i really can't imagine. -sadness 0.082 0.3429 in this work, we present a novel retrofitting method to learn emotion-aware plms using supervised contrastive learning as a transfer learning tool. we use go_emotions @xcite , the largest publicly available emotion recognition dataset, as a retrofitting corpus. our method updates plm network weights such that the text fragments exhibiting similar emotions are encoded nearby in the representation space, and fragments with different emotion content are pushed apart. while doing so, it also ensures that the linguistic knowledge originally present in the plm is preserved. we refer to the resulting emotion-aware models as * emo foot_0 . our contributions are: the sentence embeddings from bert and roberta do not have this property (quantitatively shown in ## retrofitting corpus: go_emotions to retrofit plms for emotions, we need a corpus that provides emotion annotations for text fragments, i.e., emotion recognition datasets. such datasets, varying in size, labeling scheme, domain, etc., have been proposed in the field (see @xcite . being the largest publicly available dataset, we use go_emotions @xcite in this work. it provides fine-grained emotion annotations (27 categories) for 54,263 english reddit comments. the dataset has been carefully created to avoid offensive, identity and religion terms. though the text fragments number of examples 58,009 number of raters per example 3 number of examples with 2+ raters agreeing on at least 1 label 54,263 number of emotions 27+neutral number of examples with unique labels 45,446 in the dataset are marked with multiple emotion categories, we only use a subset of 45,446 examples that are annotated with a single emotion label. ## retrofitting method the application of supervised contrastive learning in our retrofitting method is inspired by @xcite , albeit applied in a transfer learning setting. our method also ensures that the linguistic knowledge already present in plm network weights is not inadvertently perturbed. figure 1 shows the architecture for our retrofitting method. 1. training data generation: we sample training examples from go_emotions to create a minibatch b of size n (i.e., {x i , y i } @xmath1..n ; x @xmath2; y @xmath3. we consider two sampling alternatives: (1 ) mperclasssampler samples an equal number of examples from each emotion category; (2) stratifiedsampler samples examples from an emotion category proportional to the total number of examples of that category in go_emotions. 2. encoder: the plm under consideration is treated as an encoder function. it takes text fragments x i from batch b as input and returns the [cls] i embeddings as output, i.e., enc(x i ) = [cls] i . 3. projection head: it maps [cls] i to an l 2 -normalized vector z @xmath4, i.e., proj([cls] i ) = z i . we consider two variants for proj: (1) linear: [cls] i projected linearly to r 64 ; (2) mlp: [cls] i projected non-linearly to r 64 using a single hidden layer of size 768. 4. loss function: in self-supervised contrastive learning, the normalized temperature-scaled cross entropy (nt-xent) loss proposed by chen et al. (2020) has been shown to learn robust latent representations by implicitly providing hard positive/negative mining. khosla et al. (2020) extended it further for scl by incorporating label information into the loss function. we use the supervised version of the loss function foot_1 which is as follows, here, b(i): @xmath5; p (i): {@xmath6; y @xmath7; and τ : temperature scaling factor. setting τ to a low value gives more importance to hard positives/negatives, whereas a high value weighs all pairs nearly equally. vector space preservation (vsp): being trained on a vast amount of data, plms are extremely good at encoding lexical, syntactic, and semantic relations, concept similarities, and so on @xcite . we do not want to lose such valuable information while retrofitting them for emotions. while retrofitting literature on static embeddings has recognized and addressed this issue by introducing a regularization term that preserves the topology of pre-trained vector space @xcite , such regularization has not been considered by retrofitting methods for plms. to address this, we use the following regularization term, while enc(•) comes from step 2, enc fixed (•) is a fixed version where the network weights from plm are frozen, providing the snapshot of the plm before the application of contrastive learning. with this regularization, the network weights in enc(•) are updated such that the sentence embeddings in [cls] do not deviate much from their pre-trained version. the final loss function used by our method is then: @xmath8, where λ is a hyperparameter that determines how strictly the original pre-trained vector space is preserved. post training, we discard the projection head. while the encoder enc(•), with its updated weights, provides us with emotion-aware plm. ## experiments we evaluate our retrofitting method on two plms in bert @xcite and roberta @xcite {base, uncased versions}. the emotion-aware plms retrofitted by our method are referred to as * emo, i.e., bertemo and robertaemo. to select the best hyper-parameter configuration, we consider two aspects: (1) quality of * emo in learning emotion-aware embeddings, measured using clustering and retrieval metrics (refer section 5.2 for details) (2) vector space preservation: mean cosine distance between sentence embeddings obtained from * emo and its pre-trained version (cosine @xmath9; @xmath10, 2]). we first filter vector space preserving configurations. from the filtered set, we then choose the configuration with the highest ami (clustering quality metric) as the best configuration. we detail this process and report the complete hyper-parameter grid search in appendix a. ## conclusion and future work we present a novel retrofitting method to learn emotion-aware plms in a contrastive learning setting using emotion labeling in the go_emotions @xcite corpus. it updates plm network weights such that the text fragments exhibiting similar emotions are encoded nearby in the representation space, and fragments with different emotions are pushed apart while preserving the linguistic knowledge originally captured during the plm pre-training phase. the emotion-aware models ( * emo) learned by our method perform better than their pre-trained counterparts (about 1% improvement in f1-score) and other benchmarks, with significant gains in few-shot learning setting. going forward, we want to extend our retrofitting method to other affective resources such as the nrc vad lexicon, emoticons, etc. we also plan to investigate affective content in large language models such as gpt4.