Examination and Extension of Strategies for Improving Personalized Language Modeling via Interpolation (2020)

Abstract: In this paper, we detail novel strategies for interpolating personalized language models and methods to handle out-of-vocabulary (OOV) tokens to improve personalized language models. Using publicly available data from Reddit, we demonstrate improvements in offline metrics at the user level by interpolating a global LSTM-based authoring model with a user-personalized n-gram model. By optimizing this approach with a back-off to uniform OOV penalty and the interpolation coefficient, we observe that over 80% of users receive a lift in perplexity, with an average of 5.4% in perplexity lift per user. In doing this research we extend previous work in building NLIs and improve the robustness of metrics for downstream tasks.

## introduction natural language interfaces (nlis) have become a ubiquitous part of modern life. such interfaces are used to converse with personal assistants (e.g., apple siri, amazon alexa, google assistant, microsoft cortana), to search for and gather information (google, bing), and to interact with others on social media. one developing use case is to aid the user during composition by suggesting words, phrases, sentences, and even paragraphs that complete the user's thoughts @xcite . personalization of these interfaces is a natural step forward in a world where the vocabulary, grammar, and language can differ hugely user to user @xcite . numerous works have described personalization in nlis in audio rendering devices @xcite , digital assistants @xcite , telephone interfaces @xcite , etc. we explore an approach for personalization of language models * indicates equal contributions (lms) for use in downstream nlis on composition assistance, and replicate previous work to show that interpolating a global long short-term memory network (lstm) model with user-personalized n-gram models provides per-user performance improvements when compared with only a global lstm model @xcite . we extend that work by providing new strategies to interpolate the predictions of these two models. we evaluate these strategies on a publicly available set of reddit user comments and show that our interpolation strategies deliver a 5.2% perplexity lift. finally, we describe methods for handling the crucial edge case of out-of-vocabulary (oov) tokens foot_0 . specifically, the contributions of this work are: ## related work language modeling is a critical component for many nlis, and personalization is a natural direction to improve these interfaces. several published works have explored personalization of language models using historical search queries @xcite , features garnered from social graphs @xcite @xcite , and transfer learning techniques @xcite . other work has explored using profile information (location, name, etc.) as additional features to condition trained models @xcite . specifically, in the nli domain, google smart compose @xcite productized the approach described in @xcite by using a linear interpolation of a general background model and a personalized n-gram model to personalize lm predictions in the email authoring setting. we view our work as a natural extension to this line of research because strategies that improve personalization at the language modeling level drive results at the user interface level. ## personalized interpolation model the goal of text prediction is strongly aligned with language modeling. the task of language modeling is to predict which words come next, given a set of context words. in this paper, we explore using a combination of both large scale neural lms and small scale personalized n-gram lms. this combination has been studied in the literature @xcite and has been found to be performant. we describe mechanisms for extending this previous work in this section. once trained, we compute the perplexity of these models not by exponentiation of the cross entropy, but rather by explicitly predicting the probability of test sequences. in practice this model is to be used to rerank sentence completion sequences. as a result, it is impossible to ignore the observation of oov tokens. ## data the data for our model comes from comments made by users on the internet social media website reddit 4 . reddit is a rich source of naturallanguage data with high linguistic diversity due to posts about a variety of topics, informality of language, and sheer volume of data. as a linguistic resource, reddit comments present in a heavily 3 we consider φ to be a hyperparameter which must be tuned for each use case. in our experiments we assign φ to be 1 v , where v is the vocabulary size. 4 we retrieved copies of www.reddit.com user comments from https://pushshift.io/ . ## results here we discuss the results observed when evaluating the interpolated global lstm and userpersonalized n-gram model on users' comments using various oov mitigation and α interpolation strategies. ## conclusion & future work in this paper we presented new strategies for interpolating personalized lms, discussed strategies for handling oov tokens to give better vision into model performance, and evaluated these strategies on public data allowing the research community to build upon these results. furthermore, two directions could be worth exploring: investigate on when personalization is useful at a user level to better interpret the results; research on user-specific vocabularies for personalized models instead of using a shared vocabulary for both the personalized and global background models. as nlis move closer to the user, personalization mechanisms will need to become more robust. we believe the results we have presented form a natural step in building that robustness.