Efficient- F ed R ec: Efficient Federated Learning Framework for Privacy-Preserving News Recommendation (2021)

Abstract: News recommendation is critical for personalized news access. Most existing news recommendation methods rely on centralized storage of users’ historical news click behavior data, which may lead to privacy concerns and hazards. Federated Learning is a privacy-preserving framework for multiple clients to collaboratively train models without sharing their private data. However, the computation and communication cost of directly learning many existing news recommendation models in a federated way are unacceptable for user clients. In this paper, we propose an efficient federated learning framework for privacy-preserving news recommendation. Instead of training and communicating the whole model, we decompose the news recommendation model into a large news model maintained in the server and a light-weight user model shared on both server and clients, where news representations and user model are communicated between server and clients. More specifically, the clients request the user model and news representations from the server, and send their locally computed gradients to the server for aggregation. The server updates its global user model with the aggregated gradients, and further updates its news model to infer updated news representations. Since the local gradients may contain private information, we propose a secure aggregation method to aggregate gradients in a privacy-preserving way. Experiments on two real-world datasets show that our method can reduce the computation and communication cost on clients while keep promising model performance.

## introduction with the explosion of online information, the large quantities of news generated every day may overwhelm users and make them difficult to find the news they are interested in. to tackle this problem, many news recommendation methods @xcite @xcite have been proposed to display news according to users' personalized interests. these methods are usually composed of two core modules, i.e., user model and news model. the user model is used to learn user representations from user historical click behaviors. for example, @xcite use a candidate-aware attention network as the user model to help capture user interests in candidate news. the news model is used to learn news representations from news content. for example, @xcite apply multi-head self attention network to capture the interactions between words in news model. with the success of pre-trained language models (plm) in nlp, a few plm-empowered news recommendation methods have been proposed and achieve remarkable performance. for example, @xcite apply pre-trained language models to enhance news modeling. however, these methods require centralized storage of user behaviors, which are highly privacysensitive @xcite . collecting private user data has raised many concerns @xcite . moreover, due to the adoption of some data protection regulations such as gdpr foot_0 , it might not be able to analyze centralized user data in the future. federated learning @xcite is a privacy-preserving method to train models on the private data decentralized on a large number of clients. in federated learning, each user keeps a local copy of model, and compute local model gradients with their local private data. a central server coordinates the clients and aggregates local gradients to update the global model. recently, @xcite proposed a fedrec method to train news recommendation models using federated learning. however, the model sizes of many existing news recommendation methods are large, especially their news models. for example, plm-nr @xcite has 110.7m parameter in total, 110m of which are in the news model (bert-base version). thus, the communication and computation costs of fedrec can be too high for clients with rather limited computation resource. in this paper, we propose an efficient federated learning framework for privacy-preserving news recommendation named efficient-fedrec 2 . in our framework, we decompose the news recommendation model into a large news model and a lightweight user model. instead of training and communicating the whole model, in our approach the clients only request the user model and the representations of news involved in their local behaviors from the server. the clients locally compute the gradients of the user model and news representations on their local data, and send them to the server for aggregation. the central server uses the aggregated user model gradients to update its maintained global user model, and update the news model based on the aggregated news representation gradients. the updated news model is further used to infer updated news representations. the above process is repeated for multiple rounds until the model gets converges. in order to protect user privacy in model training, we develop a secure aggregation protocol based on the multi-party computation framework for privacy-preserving gradient aggregation. we exchange the news representations in the union news set involved by a group of user behaviors to protect the click history of a specific user. we conduct plenty of experiments on two real-world datasets and the results show that our approach can effectively reduce the computation and communication cost on clients for federated news recommendation model training. the main contributions of this work include: ## methodology in this section, we introduce our efficient-fedrec method for privacy-preserving news recommendation. we first introduce the problem formulation and news recommendation framework. then we introduce the details of our efficient-fedrec framework. the details of secure aggregation are demonstrated in the last subsection. ## experiments in this section, we demonstrate the efficiency and effectiveness of our efficient-fedrec. we conduct several experiments to answer the following research questions: ## conclusion in this paper, we propose an efficient federated learning framework for privacy-preserving news recommendation named efficient-fedrec. we decompose the news recommendation model into a large news model maintained by server and a lightweight user model. users request news representations and user model from the central server and compute gradients with user local data. the cen-tral server aggregates gradients to update the user model and news model. the updated news model is further used to infer news representation by server. in order to protect the private information in user local gradients, we apply secure aggregation to aggregate gradients. in order to protect user interacted news history, we exchange the news representations in the union news set involved by a group of user behaviors. experiments on two real-world datasets validate our method can effectively reduce both communication and computation cost on user side while keep the model performance.