Characterizing E nglish Variation across Social Media Communities with BERT (2021)

Abstract: Much previous work characterizing language variation across Internet social groups has focused on the types of words used by these groups. We extend this type of study by employing BERT to characterize variation in the senses of words as well, analyzing two months of English comments in 474 Reddit communities. The specificity of different sense clusters to a community, combined with the specificity of a community’s unique word types, is used to identify cases where a social group’s language deviates from the norm. We validate our metrics using user-created glossaries and draw on sociolinguistic theories to connect language variation with trends in community behavior. We find that communities with highly distinctive language are medium-sized, and their loyal and highly engaged users interact in dense networks.

## introduction internet language is often popularly characterized as a messy variant of ''standard'' language @xcite . however, work in sociolinguistics has demonstrated that online language is not homogeneous @xcite @xcite . instead, it expresses immense amounts of variation, often driven by social variables. online language contains lexical innovations, such as orthographic variants, but also repurposes words with new meanings @xcite . there has been much attention on which words are used across these social groups, including work examining the frequency of types @xcite . however, there is also increasing interest in how words are used in these online communities as well, including variation in meaning @xcite . for example, a word such as python in figure 1 has different usages depending on the community in which it is used. our work examines both lexical and semantic variation, and operationalizes the study of the latter using bert @xcite . social media language is an especially interesting domain for studying lexical semantics because users' word use is far more dynamic and varied than is typically captured in standard sense inventories like wordnet. online communities that sustain linguistic norms have been characterized as virtual communities of practice @xcite @xcite . users may develop wiki pages, or guides, for their communities that outline specific jargon and rules. however, some communities exhibit more language variation than others. one central goal in sociolinguistics is to investigate what social factors lead to variation, and how they relate to the growth and maintenance of sociolects, registers, and styles. to enable our ability to answer these types of questions from a computational perspective, we must first develop metrics for measuring variation. our work quantifies how much the language of an online community deviates from the norm and identifies communities that contain unique language varieties. we define community-specific language in two ways, one based on word choice variation, and another based on meaning variation using bert. words used with community-specific senses match words that appear in glossaries created by users for their communities. finally, we test several hypotheses about user-based attributes of online english varieties drawn from sociolinguistics literature, showing that communities with more distinctive language tend to be mediumsized and have more loyal and active users in dense interaction networks. we release our code, our dataset of glossaries for 57 reddit communities, and additional information about all communities in our study at https://github .com/lucy3/ingroup lang . ## related work the sheer number of conversations on social media platforms allow for large-scale studies that were previously impractical using traditional sociolinguistics methods such as ethnographic interviews and surveys. earlier work on computermediated communication identified the presence and growth of group norms in online settings @xcite , and how new and veteran community members adapt to a community's changing linguistic landscape @xcite . much work in computational sociolinguistics has focused on lexical variation @xcite . online language contains an abundance of ''nonstandard'' words, and these dynamic trends rise and decline based on social and linguistic factors @xcite @xcite @xcite . online communities' linguistic norms and differences are often defined by which words are used. for example, @xcite quantify the distinctiveness of a reddit community's identity by the average specificity of its words and utterances. they define specificity as the pmi of a word in a community relative to the entire set of communities, and find that distinctive communities are more likely to retain users. to identify community-specific language, we extend @xcite 's approach to incorporate semantic variation, mirroring the sense-versus-type dichotomy of language variation put forth by previous work on slang detection @xcite . there has been less previous work on cross-community semantic variation. @xcite use social networks to address sentiment variation across twitter users, accounting for cases such as sick being positive in this sick beat or negative in i feel tired and sick @xcite adapt the model of @xcite for learning dialectaware word vectors to reddit communities discussing programming and football. they find that sub-communities for each topic share meaning conventions, but also develop their own. a line each marker on the t-sne plot is a bert embedding of python, case insensitive, in r/cscareerquestions (where it refers to the programming language) and r/elitedangerous (where it refers to a type of spacecraft). bert also predicts different substitutes when python is masked out in these communities' comments. the strength of bert to capture word senses presents a new opportunity to measure semantic variation in online communities of practice. bert embeddings have been shown to capture word meaning @xcite , and different senses tend to be segregated into different regions of bert's embedding space @xcite . clustering these embeddings can reveal sense variation and change, where distinct senses are often represented as cluster centroids @xcite . for example, @xcite use a nearest-neighbor classifier for word sense disambiguation, where word embeddings are assigned to the nearest centroid representing a word sense. using bert-base, they achieve an f1 of 71.1 on semcor @xcite , beating the state of the art at that time. part of our work examines how well the default behavior of contextualized embeddings, as depicted in figure 1 , can be used for identifying niche meanings in the domain of internet discussions. as online language may contain semantic innovations, our domain necessitates word sense induction (wsi) rather than disambiguation. we evaluate approaches for measuring usage or sense variation on two common wsi benchmarks, sem-eval 2013 task 13 @xcite and semeval 2010 task 14 @xcite , which provide evaluation metrics for unsupervised sense groupings of different occurrences of words. the current state of the art in wsi clusters representations consisting of substitutes, such as those shown in figure 1 , predicted by bert for masked target words @xcite . we also adapt this method on our reddit dataset to detect semantic variation. ## data our data is a subset of all comments on reddit made during may and june 2019 @xcite . reddit is broken up into forum-based communities called subreddits, which discuss different topics, such as parenting or gaming, or target users in different social groups, such as lgbtq+ or women. we select the top 500 most popular subreddits based on number of comments and remove subreddits that have less than 85% english comments, using the language identification method proposed by @xcite . this process yields 474 subreddits, from which we randomly sample 80,000 comments each. the number of comments per subreddit originally ranged from over 13 million to over 80,000, so this sampling ensures that more popular communities do not skew comparisons of word usage across subreddits. each sampled subreddit had around 20k unique users on average, where a user is defined as a unique username associated with comments. 1 we lowercase the text, remove urls, and replace usernames, numbers, and subreddit names each with their own special token type. the resulting dataset has over 1.4 billion tokens. to understand how users in these communities define and catalog their own language, we also manually gather all available glossaries of the subreddits in our dataset. these glossaries are usually written as guides to newcomers to the community and can be found in or linked from community 1 some reddit users may have multiple usernames due to the creation of ''throwaway'' accounts @xcite , but we define a single user by its account username. ## word sense induction we develop and evaluate word sense induction models using semeval wsi tasks in a manner that is designed to parallel their later use on larger reddit data. ## glossary analysis new members may spend 8 to 9 months acquiring a community's linguistic norms @xcite , and some reddit communities have such distinctive language that their posts can be difficult to understand to outsiders. this makes the manual annotation of linguistic norms across hundreds of communities difficult, and so for the purposes of our study, we use user-created glossaries to provide context for what our metrics find. still, glossaries only contain words deemed by a few users to be important for their community, and the lack of labeled negative examples inhibits their use in a supervised machine learning task. therefore, we focus on whether glossary words, on average, tend to have high scores using our methods. to see how highly our metrics tend to score glossary terms, we calculate their mean reciprocal rank (mrr), an evaluation metric often used to evaluate query responses @xcite : we have five different possible metrics for scoring community-specific word types: type pmi, type npmi, tf-idf, textrank, and jsd. of these, textrank has the lowest mrr, but still scores a competitive percentage of glossary words in the 98th percentile. this is because the textrank algorithm only determines how important a word is within each subreddit, without any comparison to other subreddits to determine how a word's frequency in a subreddit differs from the norm. type npmi has the highest mrr, followed by jsd. though jsd has more glossary words in the 98th percentile than type npmi, we notice that many high-scoring jsd terms include words that have a very different probability in a subreddit compared to the rest of reddit, but are not actually distinctive to that subreddit. for example, in r/justnomil, words such as husband, she, and her are within the top 10 ranked words by jsd score. this contrasts the words in figure 2 shows the normalized distributions of type npmi and sense npmi. though glossary words tend to have higher npmi scores than non-glossary words, there is still overlap between the two distributions, where some glossary words have low scores and some non-glossary words have high ones. sometimes this is because many glossary words with low type npmi instead have high sense npmi. for example, the glossary word envy in r/competitiveoverwatch refers to an esports team and has low type npmi (t * = 0.1876) but sense npmi in the 98th percentile (m * = 0.2640, m † = 0.2136). only 21 glossary terms, such as aha, a popular type of skin exfoliant in r/skincareaddiction, are both in the 98th percentile of t * and the 98th percentiles of m * and m † scores. thus, examining variation in the meaning of broadly used words provides a complementary metric to counting distinctive word types, and overall provides a more comprehensive understanding of community-specific language. other cases of overlap are due to model error. manual inspection reveals that some glossary words that actually have unique senses have low m scores. sometimes a wsi method splits a glossary term in a community into too many senses or fails to disambiguate different meanings. for example, the glossary word spawn in r/childfree refers to children, but the embeddingbased method assigns it to the same sense used in gaming communities, where it instead refers to the creation of characters or items. as another example of a failure case, the substitution-based method splits the majority of occurrences of rep, an exercise movement, in r/bodybuilding into two large but separate senses. though new methods using bert have led to performance boosts, wsi is still a challenging task. the use of glossaries in our study has several limitations. some non-glossary terms have high scores because glossaries are not comprehensive. for example, dips (m * = 0.2920, m † = 0.2541) is not listed in r/fitness's glossary, but it regularly refers to a type of exercise. this suggests the potential of our methods for uncovering possible additions to these glossaries. the vast majority of glossaries contain community-specific words, but a few also include common internet terms that have low values across all metrics, such as lol, imo, and fyi. in addition, only 71.12% of all single-token glossary words occurred often enough to have scores calculated for them. some words are relevant to the topic of the community (e.g., christadelphianism in r/christianity), but are actually rarely used in discussions. we do not compute scores for rarely occurring words, so they are excluded from our results. despite these limitations, however, user-created glossaries are valuable resources for outsiders to understand the terminology used in niche online communities, and offer one of the only sources of in-domain validation for these methods. ## communities and variation in this section, we investigate how language variation relates to characteristics of users and communities in our dataset. for these analyses, we use the metrics that aligned the most with user-created glossaries ( ## ethical considerations the reddit posts and comments in our study are accessible by the public and were crawled by @xcite . our project was deemed exempt from institutional review board review for human subjects research by the relevant administrative office at our institution. even so, there are important ethical considerations to take when using social media data @xcite . users on reddit are not typically aware of research being conducted using their data, and therefore care needs to be taken to ensure that these users remain anonymous and unidentifable. in addition, posts and comments that are deleted by users after data collection still persist in the archived dataset. our study minimizes risks by focusing on aggregated results, and our research questions do not involve understanding sensitive information about individual users. there is debate on whether to include direct quotes of users' content in publications @xcite . we include a few excerpts from comments in our paper to adequately illustrate our ideas, especially since the exact wording of text can influence the predictions of nlp models, but we choose examples that do not pertain to users' personal information. ## conclusion we use type-and sense-based methods to detect community-specific language in reddit communities. our results confirm several sociolinguistic hypotheses related to the behavior of users and their use of community-specific language. future work could develop annotated wsi datasets for online language similar to the standard semeval benchmarks we used, since models developed directly on this domain may better fit its rich diversity of meanings. we set a foundation for further investigations on how bert could help define unknown words or meanings in niche communities, or how linguistic norms vary across communities discussing similar topics. our community-level analyses could be expanded to measure linguistic similarity between communities and map the dispersion of ideas among them. it is possible that the preferences of some communities towards specific senses is due to words being commonly polysemous and one meaning being particularly relevant to the topic of that community, while others might be linguistic innovations created by users. more research on semantic shifts may help untangle these differences.