Improving Social Meaning Detection with Pragmatic Masking and Surrogate Fine-Tuning (2022)

Abstract: Masked language models (MLMs) are pre-trained with a denoising objective that is in a mismatch with the objective of downstream fine-tuning. We propose pragmatic masking and surrogate fine-tuning as two complementing strategies that exploit social cues to drive pre-trained representations toward a broad set of concepts useful for a wide class of social meaning tasks. We test our models on 15 different Twitter datasets for social meaning detection. Our methods achieve 2.34% F_1 over a competitive baseline, while outperforming domain-specific language models pre-trained on large datasets. Our methods also excel in few-shot learning: with only 5% of training data (severely few-shot), our methods enable an impressive 68.54% average F_1 . The methods are also language agnostic, as we show in a zero-shot setting involving six datasets from three different languages.

## introduction masked language models (mlms) such as bert @xcite have revolutionized natural language processing (nlp). these models exploit the idea of self-supervision where sequences of unlabeled text are masked and the model is tasked to reconstruct them. knowledge acquired during this stage of denoising (called pretraining) can then be transferred to downstream tasks through a second stage (called fine-tuning). although pre-training is general, does not require labeled data, and is task agnostic, fine-tuning is narrow, requires labeled data, and is task-specific. for a class of tasks τ , some of which we may not know in the present but which can become desirable in the future, it is unclear how we can bridge the learning objective mismatch between these two stages. in particular, how can we (i) make pre-training more tightly related to downstream task learning objective; and (ii) focus model pre-training representation on an all-encompassing range of concepts of general affinity to various downstream tasks? we raise these questions in the context of learning a cluster of tasks to which we collectively refer as social meaning. we loosely define social meaning as meaning emerging through human interaction such as on social media. example social meaning tasks include emotion, irony, and sentiment detection. we propose two main solutions that we hypothesize can bring pre-training and finetuning closer in the context of learning social meaning: first, we propose a particular type of guided masking that prioritizes learning contexts of tokens crucially relevant to social meaning in interactive discourse. since the type of "meaning in interaction" we are interested in is the domain of linguistic pragmatics @xcite , we will refer to our proposed masking mechanism as pragmatic masking. we explain pragmatic masking in section 3.1. second, we propose an additional novel stage of fine-tuning that does not depend on gold labels but instead exploits general data cues possibly relevant to all social meaning tasks. more precisely, we leverage proposition-level user assigned tags for intermediate fine-tuning of pre-trained language models. in the case of twitter, for example, hashtags naturally assigned by users at the end of posts can carry discriminative power that is by and large relevant to a wide host of tasks. although cues such as hashtags and emojis have been previously used as surrogate lables before for one task or another, we put them to a broader use that is not focused on a particular (usually narrow) task that learns from a handful of cues. in other words, our goal is to learn extensive concepts carried by tens of thousands of cues. a model endowed with such a knowledge-base of social concepts can then be further fine-tuned on any narrower task in the ordinary way. we refer to this method as surrogate fine-tuning (section 3.2). another migration from previous work is that our methods excel not only in the full-data setting but also for few-shot learning, as we will explain below. in order to evaluate our methods, we present a social meaning benchmark composed of 15 different datasets crawled from previous research sources. we perform an extensive series of methodical experiments directly targeting our proposed methods. our experiments set new state-of-the-art (sota) in the supervised setting across different datasets. moreover, our experiments reveal a striking capacity of our models in improving downstream task performance in few-shot and severely few-shot settings (i.e., as low as 1% of gold data), and even the zero-shot setting on languages other than english (i.e., as evaluated on six different datasets from three languages in section 6). to summarize, we make the following contributions: (1) we propose a novel pragmatic masking strategy that makes use of social media cues akin to improving social meaning detection. (2) we introduce a new effective surrogate fine-tuning method suited to social meaning that exploits the same simple cues as our pragmatic masking strategy. (3) we report new sota on eight out of 15 supervised datasets in the full-data setting. (4) our methods are remarkably effective for few-shot and zero-and learning. we now review related work. intermediate fine-tuning. although pretrained language models (plm) have shown significant improvements on nlp tasks, intermediate training of the plm on one or more data-rich tasks can further improve performance on a target downstream task. most previous work (e.g., @xcite @xcite @xcite ) focus on intermediate fine-tuning on a given goldlabeled dataset related to a downstream target task. different to these works, our surrogate fine-tuning method is agnostic to narrow downstream tasks and fine-tunes an plm on large-scale data with tens of thousands of surrogate labels that may be relevant to all social meaning. we now introduce our methods. ## results and analysis we report performance of our models trained with our pm strategy in section 5.1, where we investigate two types of pragmatic signals (i.e., hashtag and emoji) and the effect of their locations (anywhere vs. at the end). section 5.2 shows the results of our sft method with hashtags and emojis. moreover, we combine our two proposed methods and compare our models to the sota models in sections 5.3 and 5.4, respectively. ## zero-and few-shot learning since our methods exploit general cues in the data for pragmatic masking and learn a broad range of social meaning concepts, we hypothesize they should be particularly effective in few-shot learning. to test this hypothesis, we fine-tune our best models (i.e., prags1 and prags2) on varying percentages of the train set of each task as explained in section 4.2. figure 2 shows that our two mod- we compare using the same metrics employed on each dataset. metrics: m-f 1 : macro f 1 , w-f 1 : weighted and may fare well on languages other than english. although we do not test this claim directly in this work, we do score our english-language best models on six datasets from three other languages (zero-shot setting). we fine-tune our best english model (i.e., prags2 in ## model analyses to better understand model behavior, we carry out both a qualitative and a quantitative analysis. for the qualitative analysis, we encode all the dev and test samples from one emotion downstream task using two plms (roberta and bertweet) and our two best models (i.e., prags1 and prags2) 8 . we then use the hidden state of the [cls] token from the last transformer encoder layer as the representation of each input. we then map these tweet representation vectors (768 dimensions) to a 2-d space through t-sne technique (van der maaten and hinton, 2008) and visualize the results. comparing our models to the original roberta and bertweet, we observe that the representations from our models give sensible clustering of emotions before fine-tuning on downstream dataset. recent research @xcite @xcite has identified an anisotropy problem with the sentence embedding from plms, i.e., learned representations occupy a narrow cone, which significantly undermines their expressiveness. hence, several concurrent studies @xcite seek to improve uniformity of plms. however, wang and liu (2021) reveal a uniformity-tolerance dilemma, where excessive uniformity makes a model intolerant to semantically similar samples, thereby breaking its underlying semantic structure. following wang and liu (2021), we investigate the uniformity and tolerance of our models. the uniformity metric indicates the embedding distribution in a unit hypersphere, and the tolerance metric is the mean similarities of samples belonging to the same class. formulas of uniformity and tolerance are defined in section c in appendix. we calculate these two metrics for each model using development data from our 8 note that we use these representation models without downstream fine-tuning. ## conclusion we proposed two novel methods for improving transfer learning with plms, pragmatic masking and surrogate fine-tuning, and demonstrated the effectiveness of these methods on a wide range of social meaning datasets. our models exhibit remarkable performance in the few-shot setting and even the severely few-shot setting. our models also establish new sota on eight out of fifteen datasets when compared to tailored, task-specific models with access to external resources. our proposed methods are also language independent, and show promising performance when applied in zeroshot settings on six datasets from three different languages. in future research, we plan to further test this language independence claim. we hope our methods will inspire new work on improving language models without use of much labeled data. we then use adam with a weight decay of 0.01 to optimize the model and fine-tune each task for 20 epochs with early stop (@xmath0. we fine-tune the peak learning rate in a set of {1e -5, 5e -6} and batch size in a set of {8, 32, 64}. we find the learning rate of 5e -6 performs best across all the tasks. for the downstream tasks whose train set is smaller than 15, 000 samples, the best mini-batch size is eight. the best batch size of other larger downstream tasks is 64. for fine-tuning bertweet, we use the hyperparameters identified in nguyen et al. (2020), i.e., a fixed learning rate of 1e -5 and a batch size of 32. we use the same hyperparameters to run three times with random seeds for all downstream finetuning (unless otherwise indicated). all downstream task models are fine-tuned on four nvidia v100 gpus (32g each). at the end of each epoch, we evaluate the model on the dev set and identify the model that achieved the highest performance on dev as our best model. we then test the best model on the test set. in order to compute the model's overall performance across 15 tasks, we use same evaluation metric (i.e., macro f 1 ) for all tasks. we report the average test macro f 1 of the best model over three runs. we also average the macro f 1 scores across 15 tasks to present the model's overall performance.