Leveraging Similar Users for Personalized Language Modeling with Limited Data (2022)

Abstract: Personalized language models are designed and trained to capture language patterns specific to individual users. This makes them more accurate at predicting what a user will write. However, when a new user joins a platform and not enough text is available, it is harder to build effective personalized language models. We propose a solution for this problem, using a model trained on users that are similar to a new user. In this paper, we explore strategies for finding the similarity between new users and existing ones and methods for using the data from existing users who are a good match. We further explore the trade-off between available data for new users and how well their language can be modeled.

## introduction recent work has suggested that there are several benefits to personalized models in natural language processing (nlp) over one-size-fits-all solutions: they are more accurate for individual users; they help us understand communities better; and they focus the attention of our evaluations on the enduser @xcite . generation tasks in particular benefit from a personalized approach, for example, @xcite argue that user intention is more often difficult to recover from the context alone. we study personalization in language modeling, a core task in nlp. direct applications of language models (lm) include predictive text, authorship attribution, and dialog systems used to model the style of an individual or profession (e.g., therapist, counselor). lms are increasingly used as the backbone of models for a range of tasks in nlp, increasing the potential impact of personalization even further @xcite . the standard non-personalized approach is to use pretrained models trained on a large volume *authors contributed equally and work was performed while at the university of michigan. previous work on personalized and demographic word embeddings has seen successful application in downstream tasks. @xcite look at location and gender and how they affect associations with words like "health" and many other stimulus words like "stack"-does it make you think of books or pancakes? @xcite discuss other associations, for instance, "embodying" an idea may more often refer to a religious or economic concept depending on your beliefs. similarly, "wicked" may mean "evil" or may function as an intensifier depending on where you live @xcite . these exemplify how personalized representations can help make distinctions in meaning, however, static representations have limitations. for example, @xcite find that in some contexts "testing" refers to seeing if a device works and "sanitation" refers to a pest control issue, while in another context both refer to conditions of the covid-19 pandemic. personalized lms, or language models built to better predict what an individual will say, could better address these cases, as lms learn dynamic encodings of words. in this paper, we consider approaches to finetuning and interpolation that are novel in that they leverage data from similar users to boost personalized lm performance. we consider the case of users with a small number of available tokens and propose ways to (1) find similar users in our corpus and (2) leverage data from similar users to build a personalized lm for a new user. we explore the trade-offs between the amount of available data from existing users, the number of existing users and new users, and how our similarity metrics and methods scale. we then show an analysis to explore what types of words our method predicts more accurately and are thus more important to consider in personalization methods. ## related work personalized language modeling. @xcite examined methods for creating personalized lms and their work is most similar to ours. they consider interpolating, fine-tuning, and priming lms as methods of personalization, though they use these methods with a large generic model. in contrast, our work shows that performance can be improved by leveraging data from similar users. they also analyzed model adaptation for models trained on users with similar demographics, inspired by @xcite , who showed that these demographic factors could help model a variety of tasks, and found that personalized models perform better than those adapted from similar demographics. @xcite have also explored models for personalization but focused on handling oov tokens. @xcite proposed a framework to learn user embeddings from reddit posts. their user embeddings were built on the sentence embeddings generated by a bert model. by using the learned user embeddings to predict gender, detect depression and classify mbti personality, they concluded that their embeddings incorporate intrinsic attributes of users. in our work, user embeddings are learned in a different approach, and we focus on how to use similarity calculated from user embeddings to build better lms. ## dataset we examine a corpus of publicly available reddit comments and select users active on reddit between the years of 2007-2015 who have at least 60k tokens of text. 1 we refer to the existing users with at least 250k tokens of text as anchor users. these are users that are leveraged through interpolation or fine-tuning in order to improve performance on new users. reddit posts are mostly in english. we experiment with two settings: in the small anchor setting, there are 100 anchor users, with a 200k, 25k, 25k split for training, validation, and test, and 50 new users, with 2k tokens for training, and 25k for each of validation and test. in the large anchor setting, there are 10k anchor users and 100 new users, each having 2k tokens for training and validation and 20k for test. preprocessing reddit data can be noisy, containing urls, structured content (e.g., tables, lists), subreddit-specific emoticons, generated, or deleted content. we first extract all posts for each user in our dataset. during this process we remove noisy posts, where a post is considered "noisy" if it matches one of ten rules. these rules and examples of each are shown in ## experiments our method for constructing personalized lms consists of a similarity metric and a method for leveraging similar user data to train a personalized lm. the similarity metric measures which anchor users are most similar to a new user. that is, given a set of users (anchors), a new user (n), and a similarity function (sim), we compute @xmath0, anchors); z ⊂ anchors to get a set of similar users z. we explore three similarity metrics and two methods of applying them to the construction of personalized models. figure 1 shows how user data is used for each step. ## results we divide our results into separate subsections for each of the anchor sets. on the small anchor set we were able to perform more exploration of the weighted fine-tuning method, as it does not scale as well to the large anchor set. we present results using standard perplexity measurements as a function of the probability of a correct prediction of a token. we also present results with accuracy at n, where a prediction is counted as correct if the correct token occurs within the top n most probable words given by the model. ## ethical considerations work on personalized lms could be used for surveillance by detecting language from individuals or groups @xcite . we recommend against such applications, as they threaten intellectual freedom and risk discrimination @xcite . there may be a risk in storing private data necessary to construct these models, as data may not be properly secured or used. furthermore, a personalized model could reinforce incorrect language usage, which may be an issue for individuals learning to speak a new language, making it more difficult to learn. learning personal language patterns in a given context and suggesting these patterns in other contexts may lead to potentially incorrect or offensive results and we recommend that if this type of personalization is deemed appropriate, users are made aware of how their data is being used and potential consequences. ## conclusions in this paper, we addressed the issue of language modeling in a low data setting where a new user may not have enough data to train a personalized lm and presented a novel approach that leverages data from similar users. we considered three similarity metrics and two methods of leveraging data from similar anchor users to improve the performance of language modeling over a standard fine-tuning baseline, and showed how our results vary with the amount of data available for anchor users and the number of available anchor users. we found that the most easily scalable and highest performing method was to use user embedding similarity and to interpolate similar user fine-tuned models. additionally, we provided an analysis of the kind of words that our personalized models are able to more accurately predict and further discussed limitations of our methods.