Pearl: Personalizing Large Language Model Writing Assistants with Generation-Calibrated Retrievers (2024)

Abstract: Powerful large language models have facilitated the development of writing assistants that promise to significantly improve the quality and efficiency of composition and communication. However, a barrier to effective assistance is the lack of personalization in LLM outputs to the author’s communication style, specialized knowledge, and values. In this paper, we address this challenge by proposing Pearl, a LLM writing assistant personalized with a retriever that is trained to be generation-calibrated for personalization. Generation calibration ensures that our retriever selects historic user authored documents to augment an LLM prompt such that they are likely to help an LLM generation better adhere to a users’ preferences. We propose two key novelties for training such a retriever: (1) A training data selection method that identifies user requests likely to benefit from personalization and documents that provide that benefit; and (2) A scale-calibrating KL-divergence objective that ensures that our retriever scores remain proportional to the downstream generation quality from using the document for personalized generation. In a series of holistic evaluations, we demonstrate the effectiveness of Pearl in generating long-form texts on multiple social media datasets. Finally, we demonstrate how a generation-calibrated retriever can double as a performance predictor – detecting low quality retrieval, and improving potentially under-performing outputs via revision with LLMs.

## introduction machine-assisted writing has seen a long history of development, progressing from providing simple syntactic checks, to revising human authored text, to recent assistants being able to fully compose texts on direction from authors @xcite . the text-generation capabilities of current llms and has led current research to explore a new frontier of writing assistants for complex applications such as knowledge synthesis @xcite , peer review @xcite , and journalism @xcite ). an important element of effective writing assistants is being able to personalize generated text to retain the knowledge, style, and values of a user -an essential element of interpersonal communication @xcite . with current llms prone to generating overly generic text @xcite , author personalization of llms is an important problem. personalizing llm outputs may be seen as a form of alignment to individual users of the llm @xcite . however, leveraging fine-tuning for alignment in a personalization setup poses challenges to serving trained per-user models and obtaining sufficient per-user alignment training data. therefore, we pursue in-context alignment through retrieval augmentation @xcite . first, we assume access to a set of historic user-authored documents (e.g. emails, social media posts, etc.) and a user request for a personalized generation. to personalize llm outputs we propose an approach to train a retrieval model that selects historic user documents to augment an llm's prompt. historic documents capture users' personal style, knowledge, and values and can serve as useful context for personalized generation. while training retrievers for non-personalized applications have been explored in prior work @xcite , this exploration has been limited in personalized text generation. finally, we pursue personalization of llms only accessible via prompt-based apis since this represents a common form of accessing performant large scale llms. the starting point for our retriever in prior work examining effective prompts for non-personalized applications: @xcite show the best prompts to be those with the highest conditional likelihood of generating a target text, and @xcite use these likelihoods to train retrieval models for non-personalized retrieval augmentation of llms. while this approach performs well in non-personalized setups, personalized text generation presents unique challenges and opportunities: there are fewer historic documents per user (∼hundreds) than common non-personalized retrieval collections, and user requests may diverge from their history as users' preferences change. a smaller retrieval corpus and shifting interests mean that all requests cannot be satisfied by retrieval from a users' historical documents -as a result, all historic requests and documents are unlikely to be useful for training a retriever. our first contribution addresses this: we present a novel difference of likelihoods-based method that identifies only the personalizable user requests and associated documents that are likely to personalize downstream generations, and use these to train our retriever. next, the personalization setup offers an opportunity: fewer historical documents per user permits the use of expressive cross-encoder retrievers instead of scalable but less expressive biencoders commonly used for non-personalized tasks @xcite . however, cross-encoders produce skewed scores at the ends of their score ranges @xcite , hampering their ability to closely track the utility of a document for personalized generation. we remedy this with our second contribution -a personalized scale-calibrating training objective @xcite . this ensures that scores from our retriever are generation-calibrated for personalization -i.e. the score it produces for request-document pairs is proportional to the output quality of an llm prompted with the pair. in a case study, we show how generation calibration enables the retriever's scores to be used for retrieval performance prediction -detecting low-quality retrievals, and revising potentially low-quality generations. we instantiate pearl with multiple llms, davinci-003 and gpt-35-turbo, at privacy compliant enterprise api endpoints and evaluate it on a private dataset of workplace communications and a public dataset of reddit comments. for evaluation, we use a variety of evaluation methods spanning intrinsic, extrinsic, and personalized llm-as-judge evaluations to demonstrate the value of pearl. further, since we train calibrated retrieval models, we present additional evaluations for calibration, ablations, and analysis in appendices. our evaluations demonstrate that pearl consistently matches or outperforms strong baseline approaches. ## related work example selection for llms early work on training retrievers for augmenting llm contexts in non-personalized applications was proposed by @xcite . they train retrieval models by distilling llm likelihoods of the target completions conditioned on the prompt. similarly wang et al. (2023b) train retrieval models on finer-grained feedback from a trained reward model through distillation. @xcite train instances selection models on rewards from a downstream evaluation metric using reinforcement learning. parallel with our work, @xcite train bi-encoders for personalized classification and short text generation and find knowledge distillation from downstream llms to outperform reinforcement learning based training of retrievers. in this regard, @xcite and @xcite are closely related and represent closest work to ours -we compare to such an approach in ablations (appendix c.2). despite similarities to our work, all prior work has explored training retrievers for document selection while assuming that satisfactory predictions can be made for all inputs/requests. in addition to selecting documents for training, we also select training requests that benefit from retrieval augmentation -a necessity in personalization where retrieval is performed over a smaller historical document set instead of a large shared corpus. further, no prior approaches explore calibration for retrievers and their ability to identify low-quality retrievals, and selectively revise llm outputs -we explore this. appendix d discusses additional work on optimizing prompts, robustness to prompt errors, and calibrated retrievers. personalized writing assistants while writing assistants have seen considerable exploration, only some prior work has focused on author personalization. these applications range from email @xcite , to social media @xcite , and grammatical error correction (gec) @xcite . these systems commonly leverage nearestneighbor models @xcite and user-group level parameterefficient fine-tuning for personalization @xcite . in contrast, we explore retrieval models for in-context alignment/personalization with llms. parallel work has also explored personalized writing with llms. @xcite construct prompts with pre-trained retrieval and summarization models and fine-tune an llm for personalized completion. follow-on work has explored training a prompt-re-writer to tune prompts for a fixed llm @xcite . prompt rewriting is a complementary approach to a trained retriever, with future systems likely to benefit from both. appendix d discusses non-personalized writing assistants and reader personalization. ## problem definition we consider a request-conditional, personalized text generation task. as input to the system, we assume a user u who is associated with a set of given d u , q u , and f llm , our retriever, f retr is trained to select a subset of historical documents d ′ u ⊂ d u as few-shot examples for the llm. then the llm generates a target text t u of up to 300 words: ## proposed approach we present pearl, an in-context aligned llmbased model for personalized writing assistance. our approach (figure 2 ) consists of an offline retriever training stage and an online llm inference stage. offline, we train a retriever f retr : (q u , d u ) → r that scores the user's historical documents for their ability to personalize the output for a user request. further, we ensure that f retr is generation calibrated i.e. the scores it produces for (q u , d u ) pairs are proportional to the quality of the generated text from using (q u , d u ) in a prompt. we train such a retriever through two key novelties: (1) training data selection based on a novel difference of likelihoods from an auxiliary text generation model -we identify requests which benefit from personalization and documents which likely help personalize a target, and (2) a scale-calibrating training objective which ensures that retrievers closely track the benefit of request-document pairs for generation. given a new request, our llm is prompted to generate a target text t u conditioned on the request and the documents retrieved by f retr . next, we describe the retriever training set construction (algorithm 1), how we optimize the retriever, and the details of our implementation. ## experiments we demonstrate the effectiveness of pearl on two personalized text generation datasets from social media platforms. for evaluation, we employ standard intrinsic evaluations, extrinsic evaluation based on downstream tasks using the generated text, and recently proposed personalized llm-asjudge @xcite . then, in §5.3 we show how a calibrated retriever can be used for selective revision of underperforming requests. we present ablations in §c.2 and we demonstrate the calibration performance for our retriever in §c.3. ## conclusion in this paper we present pearl-an llm based writing assistant personalized with generation calibrated retrievers. we propose a method for training generation calibrated retrievers through a careful selection of training data and a scale calibrated objective. in a series of holistic evaluations, we demonstrate the effectiveness of our approach in datasets of social media communication compared to baselines ( §5.2) as well as ablated models (appendix c.2). we demonstrate the calibration performance for our retriever (appendix c.3), and show how our retrieval model can double as a performance predictor ( §5.3) and can identify outputs which can benefit from llm revision.