LMSOC : An Approach for Socially Sensitive Pretraining (2021)

Abstract: While large-scale pretrained language models have been shown to learn effective linguistic representations for many NLP tasks, there remain many real-world contextual aspects of language that current approaches do not capture. For instance, consider a cloze test “I enjoyed the _____ game this weekend”: the correct answer depends heavily on where the speaker is from, when the utterance occurred, and the speaker’s broader social milieu and preferences. Although language depends heavily on the geographical, temporal, and other social contexts of the speaker, these elements have not been incorporated into modern transformer-based language models. We propose a simple but effective approach to incorporate speaker social context into the learned representations of large-scale language models. Our method first learns dense representations of social contexts using graph representation learning algorithms and then primes language model pretraining with these social context representations. We evaluate our approach on geographically-sensitive language modeling tasks and show a substantial improvement (more than 100% relative lift on MRR) compared to baselines.

## introduction language models are at the very heart of many modern nlp systems and applications @xcite . representations derived from largescale language models are used widely in many downstream nlp models @xcite . however, an implicit assumption made in most modern nlp systems (including language models) is that language is independent of extra-linguistic context such as speaker/author identity and their social setting. while this simplifying assumption has undoubtedly encouraged remark-able progress in modeling language, there is overwhelming evidence in socio-linguistics that language understanding is influenced by the social context in which language is grounded @xcite @xcite @xcite . in fact, language use on social media where every utterance is grounded in a specific social context (like time, geography, social groups, communities) reinforces this often ignored aspect of language. when nlp applications ignore this social context, they may perform sub-optimally underscoring the need for a richer integration of social contexts into nlp models @xcite @xcite @xcite @xcite . prior attempts to better leverage the social context surrounding language while learning language representations have mostly focused on learning social context dependent word embeddings and have been primarily used to characterize language variation across many dimensions (time, geography, and demographics). these methods learn word embeddings for each specific social context and can capture how word meanings vary across these dimensions @xcite @xcite . however, word embedding based approaches in general suffer from two fundamental limitations: (a) word embeddings are not linguistically contextualized as noted by @xcite word embedding learning is transductive -they can only generate embeddings for words observed during training and usually assume a finite word vocabulary and a set of social contexts all of which need to be seen during training. recent approaches have addressed the first limitation by learning word representations that are contextualized by their tokenspecific usage context @xcite @xcite . we propose lmsoc to (a) learn representations of tokens that are both linguistically contextualized and socially sensitive and (b) enable the language model to inductively generate representations for language grounded in social contexts it has never observed during the language model pre-training process. as an example, our model can enable nlp systems to associate the right entity being referred to based on the broader user/social context in which an utterance like "our prime minister visited the uk last week." is grounded. ## conclusion we proposed a method to learn socially sensitive contextualized representations from large-scale language models. our method embeds social context in continuous space using graph representation algorithms and proposes a simple but effective socially sensitive pre-training approach. our approach thus enables language models to leverage correlations between social contexts and thus generalize better to social contexts not observed in training. more broadly, our method sets the stage for future research on incorporating new types of social contexts and enabling nlp systems like personalized predictive typing systems and entity-linking systems to better accommodate language variation.