Toward Informal Language Processing: Knowledge of Slang in Large Language Models (2024)

Abstract: Recent advancement in large language models (LLMs) has offered a strong potential for natural language systems to process informal language. A representative form of informal language is slang, used commonly in daily conversations and online social media. To date, slang has not been comprehensively evaluated in LLMs due partly to the absence of a carefully designed and publicly accessible benchmark. Using movie subtitles, we construct a dataset that supports evaluation on a diverse set of tasks pertaining to automatic processing of slang. For both evaluation and finetuning, we show the effectiveness of our dataset on two core applications: 1) slang detection, and 2) identification of regional and historical sources of slang from natural sentences. We also show how our dataset can be used to probe the output distributions of LLMs for interpretive insights. We find that while LLMs such as GPT-4 achieve good performance in a zero-shot setting, smaller BERT-like models finetuned on our dataset achieve comparable performance. Furthermore, we show that our dataset enables finetuning of LLMs such as GPT-3.5 that achieve substantially better performance than strong zero-shot baselines. Our work offers a comprehensive evaluation and a high-quality benchmark on English slang based on the OpenSubtitles corpus, serving both as a publicly accessible resource and a platform for applying tools for informal language processing.

## introduction large language models (llm) are the core engines of widely used applications such as chat-gpt. while the technology is becoming increasingly pervasive, it is important to understand its abilities and limitations with input from diverse forms of language use. here, we focus on the 1 code and dataset available at: https://github.com/ amazon-science/slang-llm-benchmark case of slang -a common type of informal language that is ubiquitous across day-to-day conversations @xcite . figure 1 illustrates the relevance of slang in natural language processing (nlp). when describing a good jacket, one can make different word choices such as excellent and blazing. even though the intended meaning is the same across both word choices, we might expect a significant difference in performance caused by an llm's lack of knowledge about slang. recent work in computational modeling of slang has suggested that pre-trained llms assign much lower probabilities to slang compared to their literal equivalents @xcite , suggesting that models such as bert @xcite lack knowledge of slang. knowledge of slang in llms has important implications beyond automated processing of informal language. this is the case because the use of slang explicitly reflects one's social identity @xcite @xcite . for example, the use of blazing to express 'something excellent' emerges from the us whereas it expresses 'anger' in the uk @xcite . previous work has shown that the performance of nlp systems can substantially differ across language generated by different demographic groups stratified by age, gender, region, or ethnicity @xcite @xcite @xcite and can potentially introduce representational harm @xcite . given slang's close ties with social identity, a competent language model may also accurately reveal a slang user's identity. while such information can be used to improve nlp performance @xcite , the use of slang may also lead to an increased risk of personal information exposure. despite these important implications, llms have not been rigorously evaluated across a wide range of models on tasks pertaining to slang. the main challenge lies in the lack of high-quality datasets that are publically accessible. furthermore, existing dictionary-based data sources (e.g., @xcite do not include useful meta-data such as the literal paraphrase of a slang usage. for example, having a pair of sentences as illustrated in figure 1 where the only difference lies in the slang and its paraphrase (blazing and excellent respectively) allows us to probe the llms in a controlled setting. to address these challenges, we collect a new publically accessible dataset of slang usages based on the opensubtitles corpus @xcite . using this dataset, we systematically evaluate the llms' knowledge of slang, with a particular focus on the widely adopted gpt models @xcite . 2 we show that while the llms contain considerable knowledge about slang, task-specific finetuning is still essential in achieving state-of-the-art performance. we focus on two core tasks for informal language processing, illustrated in figure 1 . first, we evaluate the extent to which llms can reliably detect slang usages in natural sentences. second, we assess whether llms can be used to identify regional-historical sources foot_1 of slang via a text classification task. finally, we examine the semantic knowledge of slang in llms to understand the differences in their representation of slang versus conventional language use. throughout our evaluation, we also draw close attention to performance discrepancies based on different demographic variables and discuss their implications toward fairness and privacy. we make the following contributions in this paper: 1) a dataset containing thousands of human annotated english slang usages in movie subtitles, contributing a novel publically available benchmark of slang for evaluation and finetuning; 2) a rigorous evaluation of large language models' knowledge of slang, including important tasks such as slang detection; 3) a discussion of the implications of such knowledge and how it may affect fairness and privacy in nlp. ## conclusion we offered a comprehensive investigation of slang knowledge in large language models. we show that larger gpt models are more knowledgeable about slang compared to bert-like models in 1) better detecting slang in natural sentences, 2) more accurately identifying the regional source and time period of slang usages, and 3) better predicting slang usages relative to their literal counterparts. despite the superiority of gpt in these slang processing tasks, we did not find evidence that it represents or encodes slang as a special form of language. it is conceivable that gpt has learned to process slang by treating slang usages as rare meanings of words expressed in appropriate linguistic contexts. in the identification of region and age of slang usages, we observed that all models tend to perform poorly on slang from the uk (compared to us slang) and more contemporary slang (compared to historical slang), likely due to impoverished training data. however, we found that gpt models are no more biased compared to earlier bert-based models and that it shows comparable discrepancy in processing slang across regions. additionally, we observed that gpt models contain good knowledge about the demographic identities of a slang usage in context. this capability may have implications for privacy in many scenarios (e.g., automatic data annotation), and users should be aware of the increased risk of identity exposure when using slang in llm-based applications. we have provided the first comprehensive probing analysis of large language models on knowledge of slang and have contributed an open benchmark dataset to facilitate future efforts in evaluating and improving large language models on informal language processing.