Tiny- N ews R ec: Effective and Efficient PLM -based News Recommendation (2022)

Abstract: News recommendation is a widely adopted technique to provide personalized news feeds for the user. Recently, pre-trained language models (PLMs) have demonstrated the great capability of natural language understanding and benefited news recommendation via improving news modeling. However, most existing works simply finetune the PLM with the news recommendation task, which may suffer from the known domain shift problem between the pre-training corpus and downstream news texts. Moreover, PLMs usually contain a large volume of parameters and have high computational overhead, which imposes a great burden on low-latency online services. In this paper, we propose Tiny-NewsRec, which can improve both the effectiveness and the efficiency of PLM-based news recommendation. We first design a self-supervised domain-specific post-training method to better adapt the general PLM to the news domain with a contrastive matching task between news titles and news bodies. We further propose a two-stage knowledge distillation method to improve the efficiency of the large PLM-based news recommendation model while maintaining its performance. Multiple teacher models originated from different time steps of our post-training procedure are used to transfer comprehensive knowledge to the student model in both its post-training stage and finetuning stage. Extensive experiments on two real-world datasets validate the effectiveness and efficiency of our method.

## introduction with the explosion of information, massive news is published on online news platforms such as microsoft news and google news @xcite , which can easily get the users overwhelmed when they try to find the information they are interested in @xcite . many personalized news recommendation methods have been proposed to alleviate the information overload problem for users @xcite @xcite . since news articles usually contain abundant textual content, learning high-quality news representations from news texts is one of the most critical tasks for news recommendation @xcite . as pre-trained language models (plms) have been proved to be powerful in text modeling and have empowered various nlp tasks @xcite , a few recent works delve into employing plms for better news modeling in news recommendation @xcite @xcite . for example, @xcite propose to replace shallow nlp models such as cnn and attention network with the plm to capture the deep contexts in news texts. however, these methods simply finetune the plm with the news recommendation task, which may be insufficient to cope with the domain shift problem between the generic pre-training corpus and downstream news texts @xcite . moreover, plms usually have a large number of parameters. for example, the bert-base model @xcite ) contains 12 layers with 110m parameters. deploying these plm-based news recommendation models to provide low-latency online services requires extensive computational resources. in this paper, we propose a tiny-newsrec approach to improve both the effectiveness and the efficiency of plm-based news recommendation 1 . in our approach, we first utilize the natural matching relation between different parts of a news article and design a self-supervised domain-specific posttraining method to better adapt the general plm to the news domain. the plm-based news encoder is trained with a contrastive matching task between news titles and news bodies to make it better capture the semantic information in news texts and generate more discriminative representations, which are beneficial to both news content understanding and user interest matching in the following news recommendation task. in addition, we propose a two-stage knowledge distillation method to compress the large plm-based model while maintaining its performance foot_1 . domain-specific knowledge and task-specific knowledge are transferred from the teacher model to the student in its post-training stage and finetuning stage respectively. besides, multiple teacher models originated from different time steps of our post-training procedure are used to provide comprehensive guidance to the student model in both stages. for each training sample, we adaptively weight these teacher models based on their performance, which allows the student model to always learn more from the best teacher. extensive experiment results on two real-world datasets show that our approach can reduce the model size by 50%-70% and accelerate the inference speed by 2-8 times while achieving better performance. the main contributions of our paper are as follows: ## methodology in this section, we introduce the details of our tiny-newsrec method. we first briefly introduce the structure of our plm-based news recommendation model. then we introduce the design of our selfsupervised domain-specific post-training method and the framework of our two-stage knowledge distillation method. some notations used in the paper are listed in ## conclusion in this paper, we propose a tiny-newsrec method to improve the effectiveness and efficiency of plmbased news recommendation with domain-specific post-training and a two-stage knowledge distillation method. specifically, before the task-specific finetuning, we propose to conduct domain-specific post-training towards the plm-based news encoder with a self-supervised matching task between news titles and news bodies to make the general plm better capture and match the high-level semantic information in news texts. in our two-stage knowledge distillation method, the student model is first adapted to the news domain and then finetuned on the news recommendation task with the domainspecific and task-specific knowledge transferred from multiple teacher models in each stage. we conduct extensive experiments on two real-world datasets and the results demonstrate that our ap-proach can effectively improve the performance of the plm-based news recommendation model with considerably smaller models.