Contrastive Learning of Sociopragmatic Meaning in Social Media (2023)

Abstract: Recent progress in representation and contrastive learning in NLP has not widely considered the class of sociopragmatic meaning (i.e., meaning in interaction within different language communities). To bridge this gap, we propose a novel framework for learning task-agnostic representations transferable to a wide range of sociopragmatic tasks (e.g., emotion, hate speech, humor, sarcasm). Our framework outperforms other contrastive learning frameworks for both in-domain and out-of-domain data, across both the general and few-shot settings. For example, compared to two popular pre-trained language models, our model obtains an improvement of 11.66 average F1 on 16 datasets when fine-tuned on only 20 training samples per dataset. We also show that our framework improves uniformity and preserves the semantic structure of representations. Our code is available at: https://github.com/UBC-NLP/infodcl

## introduction meaning emerging through human interaction such as on social media is deeply contextualized. it extends beyond referential meaning of utterances to involve both information about language users and their identity (the domain of sociolinguistics @xcite ) and the communication goals of these users (the domain of pragmatics @xcite ). from a sociolinguistics perspective, a message can be expressed in various linguistic forms, depending on user background. for example, someone might say 'let's watch the soccer game', but they can also call the game 'football'. in real world, the game is the same thing. while the two expressions are different ways of saying the same thing @xcite , they do carry information about the user such as their region (i.e., where they could be coming from). from a pragmatics perspective, the meaning of an utterance depends on its interactive context. for example, while the utterance 'it's really hot here' (said in a physical meeting) could be a polite way of asking someone to open the window, it could mean 'it's not a good idea for you to visit at this time' (said in a phone conversation discussing travel plans). we refer to the meaning communicated through this type of socially embedded interaction as sociopragmatic meaning (sm). while sm is an established concept in linguistics @xcite , nlp work still lags behind. this issue is starting to be acknowledged in the nlp community @xcite , and there has been calls to include social aspects in representation learning of language @xcite . arguably, pre-trained language models (plms) such as bert @xcite learn representations relevant to sm tasks. while this is true to some extent, plms are usually pre-trained on standard forms of language (e.g. bookcorpus) and hence miss (i) variation in language use among different language communities (social aspects of meaning) in (ii) interactive settings (pragmatic aspects). in spite of recent efforts to rectify some of these limitations by plms such as bertweet on casual language @xcite , it is not clear whether the masked language modeling (mlm) objective employed in plms is sufficient for capturing the rich representations needed for sociopragmatics. another common issue with plms is that their sequence-level embeddings suffer from the anisotropy problem @xcite . that is, these representations tend to occupy a narrow cone on the multidimensional space. this makes it hard for effectively teasing apart sequences belonging to different classes without use of large amounts of labeled data. work on contrastive learning (cl) has targeted this issue of anisotropy by attempting to bring semantic representations of instances of a given class (e.g., positive pairs of the same objects in images or same topics in text) closer and representations of negative class(es) instances farther away @xcite . a particularly effective type of cl is supervised cl @xcite , but it (i) requires labeled data (ii) for each downstream task. again, acquiring labeled data is expensive and resulting models are task-specific (i.e., cannot be generalized to all sm tasks). in this work, our goal is to learn effective taskagnostic representations for sm from social data without a need for labels. to achieve this goal, we introduce a novel framework situated in cl that we call infodcl. infodcl leverages sociopragmatic signals such as emojis or hashtags naturally occurring in social media, treating these as distant/surrogate labels. 1 since surrogate labels are abundant (e.g., hashtags on images or videos), our framework can be extended beyond language. to illustrate the superiority of our proposed framework, we evaluate representations by our infodcl on 24 sm datasets (such as emotion recognition @xcite and irony detection @xcite ) and compare against 11 competitive baselines. our proposed framework outperforms all baselines on 14 (out of 16) in-domain datasets and seven (out of eight) out-of-domain datasets (sec. 4). furthermore, our framework is strikingly successful in few-shot learning: it consistently outperforms baselines by a large margin for different sizes of training data (sec. 4). our framework is also language-independent, as demonstrated on several tasks from three languages other than english (sec. e.3). our major contributions are as follows: ## related work our work combines advances in representation learning and contrastive learning. representation learning. plms encode discrete language symbols into a continuous representation space that can capture the syntactic and the semantic information underlying the text. since bert is pre-trained on standard text that is not ideal for social media, @xcite propose bertweet, a model pre-trained on tweets with mlm objective and without intentionally learning sm from social media data. previous studies @xcite have also utilized distant supervision (e.g., use of emoji) to obtain better representations for a limited number of tasks. our work differs in that we make use of distant supervision in the context of cl to acquire rich representations suited to the whole class of sm tasks. in addition, our methods excel not only in the full data setting but also for few-shot learning and diverse domains. contrastive learning. there has been a flurry of recent cl frameworks introducing selfsupervised @xcite @xcite , semi-supervised @xcite , weakly-supervised @xcite , and strongly supervised @xcite @xcite learning objectives. 2 although effective, existing supervised cl (scl) frameworks @xcite @xcite suffer from two major drawbacks. the first drawback is scl's dependence on task-specific labeled data (which is required to identify positive samples in a batch). @xcite introduced a weaklysupervised cl (wcl) objective for computer vision, which generates a similarity-based 1-nearest neighbor graph in each batch and assigns weak labels for samples of the batch (thus clustering vertices in the graph). it is not clear, however, how much an wcl method with augmentations akin to language would fare for nlp. we propose a framework that does not require model-derived weak labels, which outperforms a clustering-based wcl approach. the second drawback with scl is related to how negative samples are treated. @xcite ; @xcite treat all the negatives equally, which is sub-optimal since hard negatives should be more informative @xcite . @xcite attempt to rectify this by introducing a label-aware contrastive loss (lcl) where they feed the anchor sample to a taskspecific model and assign higher weights to confusable negatives based on this model's confidence on the class corresponding to the negative sample. lcl, however, is both narrow and costly. it is narrow since it exploits task-specific labels. we fix this by employing surrogate labels generalizable to all sm tasks. in addition, lcl is costly since it requires an auxiliary task-specific model to be trained with the main model. again, we fix this issue by introducing a light lcl framework (lcl-lit) where we use our main model, rather than an auxiliary model, to derive the weight vector w i from our main model through an additional loss (i.e., weighting is performed end-to-end in our main model). also, lcl only considers instancelevel information to capture relationships between individual sample and classes. in comparison, we introduce a novel corpus-aware contrastive loss (ccl) that overcomes this limitation (sec. 3.3). ## proposed framework our goal is to learn rich and diverse representations suited for a wide host of sm tasks. to this end, we introduce our novel infodcl framework. infodcl is a distantly supervised cl (dcl) framework that exploits distant/surrogate label (e.g., emoji) as a proxy for supervision and incorporates corpus-level information to capture inter-class relationships. ## main results few-shot learning results. since dcl and infodcl exploit an extensive set of cues, allowing them to capture a broad range of nuanced concepts of sm, we hypothesize they will be particularly effective in few-shot learning. we hence fine-tune our dcl, infodcl, strongest two baselines, and the original roberta with varying amounts of downstream data. 13 as we then extract weights of the last classification layer and use these weights as class embeddings, @xmath2, e 2 , . . . , e c }, where e @xmath3, d is hidden dimension (i.e., 768), and |c| is the size of classes (i.e., 1, 067). the correlation of each pair of emojis is computed using cosine similarity, i.e., 15 as ## conclusion we proposed infodcl, a novel framework for adapting plms to sm exploiting surrogate labels in contrastive learning. we demonstrated effectiveness of our framework on 16 in-domain and eight out-of-domain datasets as well as nine non-english datasets. our model outperforms 11 strong baselines and exhibits strikingly powerful performance in few-shot learning.