A Computational Framework for Slang Generation (2021)

Abstract: Slang is a common type of informal language, but its flexible nature and paucity of data resources present challenges for existing natural language systems. We take an initial step toward machine generation of slang by developing a framework that models the speaker’s word choice in slang context. Our framework encodes novel slang meaning by relating the conventional and slang senses of a word while incorporating syntactic and contextual knowledge in slang usage. We construct the framework using a combination of probabilistic inference and neural contrastive learning. We perform rigorous evaluations on three slang dictionaries and show that our approach not only outperforms state-of-the-art language models, but also better predicts the historical emergence of slang word usages from 1960s to 2000s. We interpret the proposed models and find that the contrastively learned semantic space is sensitive to the similarities between slang and conventional senses of words. Our work creates opportunities for the automated generation and interpretation of informal language.

## introduction slang is a common type of informal language that appears frequently in daily conversations, social media, and mobile platforms. the flexible and ephemeral nature of slang @xcite poses a fundamental challenge for computational representation of slang in natural language systems. as of today, slang constitutes only a small portion of text corpora used in the natural language processing (nlp) community, and it is severely under-represented in standard lexical resources @xcite . here we propose a novel framework for automated generation of slang with a focus on generative modeling of slang word meaning and choice. existing language models trained on large-scale text corpora have shown success in a variety of nlp tasks. however, they are typically biased toward formal language and under-represent slang. consider the sentence ''i have a feeling he's gonna himself someday''. directly applying a state-of-the-art gpt-2 @xcite based language infilling model (e.g., @xcite would result in the retrieval of kill as the most probable word choice (@xmath0.7%). however, such a language model is limited and near-insensitive to slang usage, for example, ice-a common slang alternative for killreceived virtually 0 probability, suggesting that existing models of distributional semantics, even the transformer-type models, do not capture slang effectively, if at all. our goal is to extend the capacity of nlp systems toward slang in a principled framework. as an initial step, we focus on modeling the generative process of slang, specifically the problem of slang word choice that we illustrate in figure 1 . given an intended slang sense such as ''to kill'', we ask how we can emulate the speaker's choice of slang word(s) in informal context. 1 we are particularly interested in how the speaker chooses existing words from the lexicon and makes innovative use of those words in novel slang context (such as the use of ice in figure 1 ). our basic premise is that sensible slang word choice depends on linking conventional or established senses of a word (such as ''frozen water'' for ice) to its emergent slang senses (such as ''to kill'' for ice). for instance, the extended use of ice to express killing could have emerged from the use of cold ice to refrigerate one's remains. a principled semantic representation should adapt to such similarity relations. our proposed framework is aimed at encoding slang that relates informal figure 1 : a slang generation framework that models speaker's choice of a slang term (ice) based on the novel sense (''to kill'') in context and relations with conventional senses (e.g., ''frozen water''). and conventional word senses, hence capturing semantic similarities beyond those from existing language models. in particular, contextualized embedding models such as bert would consider ''frozen water'' to be semantically distant or irrelevant from ''to kill'', so they cannot predict ice to be appropriate for expressing ''to kill'' in slang context. the capacity for generating novel slang word usages will have several implications and applications. from a scientific view, modeling the generative process of slang word choice will help explain the emergence of novel slang usages over time-we show how our framework can predict the emergence of slang in the history of english. from a practical perspective, automated slang generation paves the way for automated slang interpretation. existing psycho-linguistic work suggests that language generation and comprehension rely on similar cognitive processes (e.g., @xcite . similarly, a generative model of slang can be an integral component of slang comprehension that informs the relation between a candidate sense and a query word, where the mapping can be unseen during training. furthermore, a generative approach to slang may also be applied to downstream tasks such as naturalistic chatbots, sentiment analysis, and sarcasm detection (see work by @xcite and @xcite ). we propose a neural-probabilistic framework that involves three components: 1) a probabilistic choice model that infers an appropriate word for expressing a query slang meaning given its context, 2) an encoder based on contrastive learning that captures slang meaning in a modified embedding space, and 3) a prior that incorporates different forms of context. specifically, the slang encoder we propose transforms slang and conventional senses of a word into a slang-sensitive embedding space where they will lie in close proximity. as such, senses like ''frozen water'', ''unfriendliness'' and ''to kill'' will be encouraged to be in close proximity in the learned embedding space. furthermore, the resulting embedding space will also set apart slang senses of a word from senses of other unrelated words, and hence contrasting within-word senses from across-word senses in the lexicon. a practical advantage of this encoding method is that semantic similarities pertinent to slang can be extracted automatically from a small amount of training data, and the learned semantic space will be sensitive to slang. our framework also captures the flexible nature of slang usages in natural context. here, we focus on syntax and linguistic context, although our framework should allow for the incorporation of social or extra-linguistic features as well. recent work has found that the flexibility of slang is reflected prominently in syntactic shift @xcite . for example, ice-most commonly used as a noun-is used as a verb to express ''to kill'' (in figure 1 ). we build on these findings by incorporating syntactic shift as a prior in the probabilistic model, which is integrated coherently with the contrastive neural encoder that captures flexibility in slang sense extension. we also show how a contextualized language infilling model can provide additional prior information from linguistic context (c.f. erk, 2016). to preview our results, we show that our framework yields a substantial improvement on the accuracy of slang generation against state-ofthe-art embedding methods including deep contextualized models, in both few-shot and zero-shot settings. we evaluate our framework rigorously on three datasets constructed from slang dictionaries and in a historical prediction task. we show evidence that the learned slang embedding space yields intuitive interpretation of slang and offers future opportunities for informal natural language processing. computational studies of slang. slang has been extensively studied as a social phenomenon @xcite , where social variables such as gender @xcite , ethnicity @xcite , and social-economic status @xcite have been shown to play important roles in slang construction. more recently, an analysis of social media text has shown that linguistic features also correlate with the survival of slang terms, where linguistically appropriate terms have a higher likelihood of being popularized @xcite . recent work in the nlp community has also analyzed slang. @xcite studied slang comprehension as a translation task. in their study, both the spelling of a word and its context are provided as input to a translation model to decode a definition sentence. @xcite proposed end-to-end neural models to detect and identify slang automatically in natural sentences. @xcite have proposed computational models that derive novel word forms of slang from spellings of existing words. here, we instead explore the generation of novel slang usage from existing words and focus on word sense extension toward slang context, based on the premise that new slang senses are often derived from existing conventional word senses. our work is inspired by previous research suggesting that slang relies on reusing words in the existing lexicon @xcite . previous work has applied cognitive models of categorization to predict novel slang usage @xcite . in that work, the generative model is motivated by research on word sense extension @xcite . in particular, slang generation is operationalized by categorizing slang senses based on their similarities to dictionary definitions of candidate words, enhanced by collaborative filtering @xcite to capture the fact that words with similar senses are likely to extend to similar novel senses @xcite . however, this approach presupposes that slang senses are similar to conventional senses of a word represented in standard embedding space, an assumption that is not warranted and yet to be addressed. our work goes beyond the existing work in three important aspects: 1) we capture semantic flexibility of slang usage by contributing a novel method based on contrastive learning. our method encodes slang meaning and conventional meaning of a word under a common embedding space, thereby improving the inadequate existing methodology for slang generation that uses common, slang-insensitive embeddings. 2) we capture syntactic and contextual flexibility of slang usage in a coherent probabilistic framework, an aspect that was ignored in previous work. 3) we rigorously test our framework against slang sense definition entries from three large slang dictionaries and contribute a new dataset for slang research to the community. contrastive learning. contrastive learning is a semi-supervised learning technique used to extract semantic representations in data-scarce situations. it can be incorporated into neural networks in the form of twin networks, where two exact copies of an encoder network are applied to two different examples. the encoded representations are then compared and back-propagated. alternative loss schemes such as triplet @xcite and quadruplet loss @xcite have also been proposed to enhance stability in training. in nlp, contrastive learning has been applied to learn similarities between text @xcite and speech utterances @xcite with recurrent neural networks. the contrastive learning method we develop has two main differences: 1) we do not use recurrent encoders because they perform poorly on dictionary-based definitions; 2) we propose a joint neural-probabilistic framework on the learned embedding space instead of resorting to methods such as nearest-neighbor search for generation. ## computational framework our computational framework for slang generation comprises three interrelated components: 1) a probabilistic formulation of word choice, extending that in @xcite to leverage encapsulated slang senses from a modified embedding space; 2) a contrastive encoder-inspired by variants of twin network @xcite -that constructs a modified embedding space for slang by adapting the conventional embeddings to incorporate new senses of slang words; 3) a contextually informed prior for capturing flexible uses of naturalistic slang. ## conclusion we have presented a framework that combines probabilistic inference with neural contrastive learning to generate novel slang word usages. our results suggest that capturing semantic and contextual flexibility simultaneously helps to improve the automated generation of slang word choices with limited training data. to our knowledge this work constitutes the first formal computational approach to modeling slang generation, and we have shown the promise of the learned semantic space for representing slang senses. our framework will provide opportunities for future research in the natural language processing of informal language, particularly the automated interpretation of slang.