Conditional Language Models for Community-Level Linguistic Variation (2022)

Abstract: Community-level linguistic variation is a core concept in sociolinguistics. In this paper, we use conditioned neural language models to learn vector representations for 510 online communities. We use these representations to measure linguistic variation between commu-nities and investigate the degree to which linguistic variation corresponds with social connections between communities. We find that our sociolinguistic embeddings are highly correlated with a social network-based representation that does not use any linguistic input.

## introduction linguistic communication requires that speakers share certain linguistic conventions, such as syntactic structure, word meanings, and patterns of interaction. speakers assume that these conventions are common ground among their interlocutors, based on joint membership in a community @xcite . such speech communities @xcite range in size from the very small, like members of a friend group, to the very large, like speakers of english. however, as @xcite point out, it is communities of practice -defined by mutual social engagement in a common activity -that are the primary locus of linguistic variation. variation is an important object of study in sociolinguistics, and is naturally amenable to computational analysis @xcite . most previous computational work on linguistic variation has considered variation at the level of macrosocial categories, such as gender @xcite @xcite , age @xcite , and geographic location @xcite .in the present work, however, we investigate linguistic variation across online communities in the social media website reddit. for this purpose, we introduce (section 2) various community-conditioned language models (cclms for short). these models are conditioned on a vector representation (or embedding), which varies by community. hence, they learn community embeddings. we report which architectures make best use of the community information (section 3), however our our primary purpose is not to improve language models in terms of perplexity, but rather to extract community embeddings that capture linguistic similarities between communities and test how the resulting embeddings correspond to the social structure of subreddits. to that end, we test the how well the community embeddings correlate with a social network-based representation of communities (section 4). the contributions of this work are twofold. first, we develop a language model-based community embedding that we show is correlated with (but still different from) an embedding based on community membership alone. second, the method we describe for testing the correlation between two embeddings from different models is, to our knowledge, novel to computational linguistics. we experiment with two kinds of model architecture: simple unidirectional lstm @xcite and a masked transformer @xcite . although transformerbased language models are considered state-of-theart, they achieve dominance partly thanks to the availability of very large data sets (e.g., @xcite , which are not available to us. 1 thus the lstm is a worthy model to test for us. in either case, the model is organised as a standard 3-layer neural sequence encoder, where the input for the tth timestep of the n + 1st layer is the tth hidden state of the nth layer. as usual, the input to the first layer, is a sequence of tokens, encoded with a trainable embedding layer over a predetermined vocabulary. at the other end, word tokens are predicted using a softmax projection layer. what we have described so far does not take community into account and as such we call them unconditioned models, but the same encoder architecture also forms the core of our conditioned models. in the cclms, we add a community embedding parameter, which varies depending on the community of origin of the input sample. this parameter is concatenated (at each time step) with the hidden layer of the sequence encoder, at some layer l @xmath0, and passed through a linear layer which projects the resulting vector back to the original hidden layer size. for l @xmath1, the output of this linear layer is passed directly to the softmax function, just as the final hidden layer of the sequence encoder is in other models. for l @xmath2, the community embedding is concatenated with the token embedding. for this reason, we set the hidden size of the sequence encoder and the size of the token embedding to be equal for all models. ## cclm performance in this section, we report the performance of the conditioned and un-conditioned models on the held out test set. first, we define two performance metrics: perplexity and information gain. in the following, we use m to refer to messages in the combined test set, and m j for the partition of the test set originating from community c j . ## comparison of cclm community embeddings with a social network embedding in this section we investigate the degree to which cclm community embeddings correlate with the social network structure of reddit. to this end, we compare the cclm-learned community embeddings foot_5 with the community embedding created by kumar et al. (2018), 7 which were generated using using a negative-sampling optimization algorithm, with the authorcommunity co-occurrence matrix as ground truth, using data from january 2014 to april 2017. we refer the reader to @xcite for details, but the important point is that no linguistic information is used to create these embeddings: they only reflect the social relationship between communities via community membership. in contrast, cclm community embeddings depend in no way on which user is the author of any given message: we only use the contents of messages, not authorship data. ## related work we have presented results using conditional neural language models to model variation between speech communities. the architecture of these models concatenates a vector representation of the conditioned variable to the input of the sequence model. this approach has been applied in various conditioned text generation domains such as image captioning @xcite , machine translation (kalchbrenner and blunsom, 2013), but it has not, to our knowledge, been used extensively to study linguistic variation. there are, however, related applications of conditional neural language models. lau et al. @xcite presents a neural language model that jointly learns to predict words on the sentence-level and represent topics on the document level. the topic representation is then fed back into the language model, improving its performance on next word prediction. this is similar to how our model experiences improved performance by learning community representations. unlike our model, topics are inferred in an unsupervised way, raising the question of whether communities could be identified from unlabeled data as well. a piece of work with similar goals as ours is that of @xcite , which uses a bayesian generative model to infer communities from variation in text data. in contrast to our work, this model treats words as independent events, ignoring the structure (and variation) in the construction of sequences. it does further suggest, however, that community-level variation can be modeled in an unsupervised way. del @xcite , use a modified skip-gram model to community-level linguistic variation. they show that lexical semantic variation occurs even across different communities organised around the same topic. their approach does not result in community level representations, however. there are several other recent studies that aim to measure linguistic distinctiveness at the level of speech community @xcite @xcite . distinctiveness is one possible interpretation of the community-stratified information gain of the cclm over its unconditioned counterpart (section 3.1). whereas the metrics in previous work are based on lexical frequency (and in the case of @xcite , word sense distributions), cclm information gain is capable of capturing distinctiveness at multiple levels of linguistic analysis. however, further work is needed to investigate exactly what kinds of variation are captured. while the focus of this paper is sociolinguistic aspects, computational models of variation can also support robust, equitable language technology. previous work has shown that speaker demographics can improve performance on standard nlp tasks @xcite ## discussion and conclusion to sum up our findings, we have defined community-conditioned language models (cclms). these models are generally able to attune to community-specific language, as witnessed by the information gain that they exhibit over baseline unconditioned models. we find that the layer depth of the community embedding (l c ) has a weak effect on the information gain and the perplexity of the cclms. for lstm models, the perplexity per word, averaged over messages from all communities, was between 66.01 and 66.35 (with 68.74 for the unconditioned model). for transformer models, it varies a bit more, between 75.66 and 83.53, but this seems to be mainly due to the poor performance of the models where the community embedding is inserted between transformer layers (l @xmath3.13). the pattern of information gain by community is similar across architectures; communities that benefit most from the conditioned model behave that way for both the lstm and transformer. however, there are some differences. for example, many of the communities with the biggest difference in information gain between the l @xmath4.g., /r/pokemongiveaway, /r/actrade, and /r/svexchange). these communities tended to have highly conventionalized ways negotiating trades and coordinating meetups. it would be interesting to investigate these differences further in future work, since it could reveal differences in the kind of linguistic variation the different model architectures capture. our main result is that community representations learned by cclms are positively correlation with user co-occurrence patterns. even though such homophilic correlation is a core hypothesis of sociolinguistics (see @xcite , for example), we believe that this study is the first to test it at the level of communities of practice using computational methods. furthermore, it appears that our method (correlating linguistic embeddings and social embeddings) is novel. indeed, even though the procrustes method has been used to correlate two sets of linguistic embeddings for the same model, we find no evidence of the method being applied to embeddings for widely different models, as we have done.