F ed P er C : Federated Learning for Language Generation with Personal and Context Preference Embeddings (2023)

Abstract: Federated learning is a training paradigm that learns from multiple distributed users without aggregating data on a centralized server, promising the ability to deploy machine-learning to a diverse population of users without first collecting large, labeled datasets. As federated learning involves averaging gradient updates across a decentralized population, there is a growing need for personalization of federated learning systems (i.e. conversational agents must personalize to individual users and the context of an interaction).In this work, we propose a new direction for personalization research within federated learning, leveraging both personal embeddings and shared context embeddings.We also present an approach to predict these “preference” embeddings, enabling personalization without backpropagation. Compared to state-of-the-art personalization baselines, our approach achieves a 50% improvement in test-time perplexity using 0.001% of the memory required by baseline approaches, and achieving greater sample- and compute-efficiency.

## introduction as conversational agents and dialog systems are deployed to real-world scenarios, these systems require data-efficient personalization paradigms such that language systems such as conversational agents can be effectively adapted on-device. the benefits of on-device optimization are two-fold; we leverage the insight that a client's data distribution is informed by both individual preferences and additional contextual information. for example, while each user may have their own individual style, there may be more general population-wide trends that inform the style of personalized predictions (e.g., dialogue assistants helping patients with cognitive disorders, whereby agents can personalize to individual patients and broader conditionwide trends). while individual preferences may be unique to each client (e.g. a user's taste or affect), we can more accurately personalize to client preferences with the addition of context, as sharedcontext parameters carry beneficial stylistic information across clients @xcite . stylistic or situational context provides additional information to curate relevant language figure 1: overview of our personalized federated learning setup, fedperc. language models within client devices, such as individual agents deployed to communicate with people at hospitals, homes, or construction sites, pull down global model parameters and context embeddings. local, on-device data is then paired with both personal and context embeddings to produce personalized predictions with global model parameters. in this work, we contribute a new approach to personalized federated learning that is both easier to learn and more effective than prior work, and investigate the utility of personalization via individual preferences and contexts. while prior language generation approaches have developed personal or persona-based generative systems @xcite or context-based generative systems @xcite individually, none have combined them to personalize outputs in a low-data setting under stylized preferences. we show that our approach is more sample-efficient than state-of-the-art baselines, while requiring less time to train. we additionally present an inference-only version of our approach, personalizing without backpropagation for new users. finally, we directly test the potential for personalization with users who have been held-out from training (i.e., testing with new users). an overview of our approach is given in figure 1 . ## related work federated learning enables machine-learning atscale to a diverse population of end-users without first collecting a large, labeled dataset for all possible tasks. after the introduction of federated averaging @xcite , focus has shifted to different ways of personalizing to individual users. prior personalization approaches for federated learning have typically involved learning personal network heads and a shared global en-coder (i.e., "split-learning" approaches @xcite ), or learning a separate local model from a global initialization (i.e., a "meta-learning" approach @xcite ). ## approach in this section, we present our novel approach to personalization in federated learning with fedperc. fedperc produces personal and contextual preference embeddings either via backpropagation (i.e., learning preference embeddings), or by inference (i.e., predicting preference embeddings). a visual overview of our federated learning architecture is in figure 2 , and a step-by-step walk-through of our training algorithm is given in algorithm 1. ## experiments we conduct several experiments to evaluate the sample efficiency, generalization, and runtime of our approach relative to baseline federated learning frameworks. in our experiments, we compare: because our experimental datasets do not contain labeled personas for all users, we do not compare directly to prior works that assume access to such information (e.g., fednlg @xcite ). we conduct two sets of experiments to compare the above approaches on both sample efficiency and runtime efficiency. for the sample efficiency experiments, we present perplexity numbers for all methods across two versions of the dataset: known users and withheld users. for our known user experiments, all users are present in the training and testing set. for our withheld user experiments, a subset of users from each dataset is withheld entirely from training, and performance results are presented only for the held-out users. perplexity is calculated over unseen utterances with the first three tokens of each utterance given as a prompt. finally, we present qualitative results from our method, demonstrating the power of stylized generation for individual users. all models are initialized with the distilgpt2 pre-trained model @xcite , with all layers frozen. we note that the use of large language models for federated language generation is a significant improvement over prior work @xcite which instead learned seq2seq models from scratch. for our split-learning and meta-learning baselines, the last layer of the model is unfrozen. training details are in the appendix. ## conclusion we present fedperc, a new approach to personalized federated learning, enabling efficient and highperformance personalization to client devices by leveraging individual and shared preference embeddings. combining shared contexts with individual personal preferences, fedperc outperforms baselines even when allotted a lower computational budget, and is the first federated language generation approach to build on large language models rather than training sequence generation models from scratch. we also provide a method of generating preference embeddings through inference alone, providing personalization with no on-device gradient computation, and we show comparable performance to fedperc using learned embeddings. we presented experiments on two datasets, tv show scripts and reddit user data, presenting empirical evidence of the utility of fedperc towards personalizing to unseen users in a federated learning setting, i.e. a 50% improvement in terms of runtime and perplexity when fine-tuning on with new users. we also demonstrated qualitative results, showing the power of separate personal and context embeddings and enabling stylization of users in new contexts. our results show that fedperc offers a promising path forward for personalization within federated learning, achieving superior quantitative results and requiring significantly less training time and data relative to baseline approaches.