Exploring the Value of Personalized Word Embeddings (2020)

Abstract: In this paper, we introduce personalized word embeddings, and examine their value for language modeling. We compare the performance of our proposed prediction model when using personalized versus generic word representations, and study how these representations can be leveraged for improved performance. We provide insight into what types of words can be more accurately predicted when building personalized models. Our results show that a subset of words belonging to specific psycholinguistic categories tend to vary more in their representations across users and that combining generic and personalized word embeddings yields the best performance, with a 4.7% relative reduction in perplexity. Additionally, we show that a language model using personalized word embeddings can be effectively used for authorship attribution.

## introduction word embeddings have become ubiquitous in natural language processing applications. usually, embeddings are trained from a large corpus of news or web data that contains writing from many sources and authors @xcite . these embeddings capture syntactic and semantic properties of the language of all authors who contributed to this corpus. multi-source corpora provide large volumes of data, but they may not lead to the ideal representations for individuals. for instance, the word "hometown" may have a different representation for different individuals. for some, it may relate to words such as "hills," "trees," and "family," whereas for others may be more strongly connected to "ocean," "beach," and "friends." these personalized representations differ among individuals, and also differ from a more generic representation that often tends to capture words that are semantically related at concept level, such as "city," "town," or "place." in this paper, we present the idea of personalized word embeddings. we explore differences in personalized word representations using a corpus of english reddit posts that contains a large number of posts per author. we use the embeddings to initialize a language model and show that personalization leads to better results than generic embeddings. one potential application of this work is personalized text generation for auto-completion to speed up text entry. another application is dialog systems that follow the speaking style of certain professionals (e.g., counselors, advisors). finally, personalized word representations could particularly help users with atypical writing styles that are not currently well served by models trained to suit the majority. ## related work prior work has considered user embeddings, where one vector is learned for each user in the data (we learn a set of vectors per user, one for each word in the vocabulary). user embeddings have been used for dialog generation @xcite , query auto-completion @xcite , authorship attribution @xcite , and sarcasm detection @xcite . amer et al. (2016) learn a set of embeddings from the books that a user adds to their profile. some approaches also use network information @xcite . this work is licensed under a creative commons attribution 4.0 international license. license details: http:// creativecommons.org/licenses/by/4.0/ . ## differences across individual word representations and usages individuals use the same word in different ways in different contexts. examining these differences can give insight into individual topic and style preferences, or their word associations. to illustrate these differences, in to gain a deeper understanding of these differences, we analyze personal and generic embeddings for specific word groups based on the linguistic inquiry and word count (liwc) lexicon @xcite and part-of-speech tags. this analysis can help us understand what types of words tend to have different representations across users and are therefore more personal in nature. we part-of-speech tag the messages with the stanford corenlp tagger @xcite . for each word in the vocabulary, we assign a tag if the tagger gives the same tag at least 95% of the time, otherwise the word is ignored. liwc categories are looked up in the lexicon that contains words and word stems and a word may have multiple categories, in which case it counts toward each. we look at the proportion of word types in the 5k most dissimilar words for each user. we define word similarity as the cosine distance between a word's generic embedding and its author-specific embedding. note that theses are unique words and that the x-axis is the percentage of the way through the 5k dissimilar words, with the most dissimilar at 100%. we break this into subsets and look at how the distribution changes as we approach the most dissimilar words. a visualization is provided in figure 1 . we find the set of most dissimilar words includes more function words, words relating to space and time (relativ), cognitive processes (cogproc), and social words, as well as more adjectives and nouns. this suggests that these types of words may tend to have more personal usage than other types. these results are consistent with prior work that has found function words are effective for recognizing style and measuring style similarity @xcite and for authorship attribution @xcite @xcite . ## language modeling our first test case for personalized word embeddings is language modeling. we train language models on our data with our embeddings as input. we use awd-lstm language model from previous work as our baseline @xcite . it is an autoregressive language model that has state-of-the-art results by combining regularization techniques and has been widely used. although more recent models can achieve better perplexities on standard benchmarks @xcite ; dai lm attribution model mrr ppl mrr accuracy merity et al. (2018b) 0.364 65.53 0.452 34.8% single user vectors 0.361 66.70 0.450 34.9% personalized embeddings 0.371 62.43 0.462 36.1% to use our personalized embeddings, we modify the architecture to take as input the concatenation of the personalized user-specific embedding and the generic embedding for each word. the same embedding dropout mask is applied to both word embeddings. the embeddings are trained on all available user data, but the more computationally expensive language models are not. we use a subsample of our dataset with 1,000 posts for each user and an 80/10/10 split for training, validation, and testing. the same splits are used for generic and personalized models, varying only the embedding layer. to measure the ability of our models to predict the next word, we use two metrics: (1) mean reciprocal rank (mrr), calculated as one divided by the rank of the correct word choice in the descending list of next word probabilities and averaged over all instances, and (2) perplexity. single user embeddings. we also consider an approach in which just one vector is learned for each user (rather than one for each user-word pair). this is an approach widely used in previous work @xcite . this user vector is concatenated to the generic word embedding. the results in we can also analyze the accuracy when predicting words belonging to particular parts of speech and liwc categories. our intuition is that a model that uses personalized word embeddings would be better at predicting words belonging to the four liwc categories whose words are most distant from the generic space. tables 3a and 3b show that for almost all categories, the personalized word embeddings lead to the best performance, although for relativity words, single user vectors performs slightly better. ## authorship attribution we also use a language model trained with personalized word embeddings to perform the task of authorship attribution. 2 we build a language model for each author using a sample of 10k posts for training and 1k for validation. we then hold out another sample of 1k posts to use for authorship attribution. the language models for all authors are separately run on the held out set, and the model with the lowest perplexity is then chosen as the assigned author. ## limitations and ethical considerations in applying our method to text prediction systems, users may experience unintended negative effects. for instance, embeddings may become unintentionally biased toward language that becomes inappropriate when later suggested in another context. additionally, users who are learning a language may bias embeddings toward improper language use, reinforcing errors and making it more difficult for the user to learn the language. it may be appropriate to use our embeddings if users consent and are made aware of the possible consequences of doing so. it is possible that our method could be used for authorship attribution and surveillance of individuals online @xcite . such an application risks potential discrimination, coercion, and threats to intellectual freedom @xcite . personalized language models could also be used to develop a tool that tells the user who their writing most resembles, or if their writing resembles their past writing, with the objective of obfuscating the author's identity @xcite . a tool like this could also be used maliciously to impersonate a particular author. although we believe the difficulty of this task currently makes these minor risks, we advocate against the use of our methods for these tasks. our method requires more computation and memory than the baseline method we compared to. the additional computation is relatively small, as learning the embeddings takes around 3 hours using 30 threads on a machine with 16 intel xeon silver 4108 cpus. the memory required to store embeddings for n users is n + 1 times the amount of storage required by a generic matrix only. ## conclusion in this paper, we explored personalized word embeddings. using a large corpus of reddit posts, we generated personalized word embeddings for 100 individuals, and performed analyses of the differences between personalized and generic embeddings for specific groups of words. we showed that using personalized word embeddings to initialize a language model improves perplexity over a model that uses generic word embeddings, or a model that only learns single vectors for each user as has been frequently done in previous work. further, we showed that the embeddings can be used to improve performance on authorship attribution. we cannot release the data due to licensing restrictions but our code is available online with instructions for how to obtain and process the data in order to support future work on personalization. 3