MTR ec: Multi-Task Learning over BERT for News Recommendation (2022)

Abstract: Existing news recommendation methods usually learn news representations solely based on news titles. To sufficiently utilize other fields of news information such as category and entities, some methods treat each field as an additional feature and combine different feature vectors with attentive pooling. With the adoption of large pre-trained models like BERT in news recommendation, the above way to incorporate multi-field information may encounter challenges: the shallow feature encoding to compress the category and entity information is not compatible with the deep BERT encoding. In this paper, we propose a multi-task method to incorporate the multi-field information into BERT, which improves its news encoding capability. Besides, we modify the gradients of auxiliary tasks based on their gradient conflicts with the main task, which further boosts the model performance. Extensive experiments on the MIND news recommendation benchmark show the effectiveness of our approach.

## introduction online news platforms such as google news and msn news have become a prevalent way for users to access news information @xcite . to alleviate information overload and improve the reading experience, personalized news recommendation has become an essential part of these platforms @xcite . traditional recommendation models focus on modeling feature interactions @xcite @xcite . accurate modeling of news and users is critical for news representation. previous neural methods usually learn news representation vectors solely based on news titles and then learn a user representation by aggregating the previously browsed * equal contribution. this work was done when qiwei bi was an intern at huawei noah's ark lab. news via sequential or attentive models @xcite @xcite . though effective, these methods only utilize the title information and neglect other valuable news information such as categories and entities, which we call multi-field information. to fully utilize this information, as shown in figure 1 , existing methods usually transform each field of information (e.g., title, category, and entities) into a feature vector and combine different representations via attentive multi-field learning @xcite . ## method given i historical clicked news of a user n @xmath0, n h 2 , ..., n h i ] and a set of candidate news n @xmath1, n c 2 , ..., n c j ]. our goal is to calculate the user interest score s j of each candidate news according to the historical behavior of the user, then the candidate news with the highest interest score is recommended to the user. for each news, we have its title text t , category label p c , and entity set e. ## conclusion we propose a novel multi-task learning framework over bert for news recommendation, named mtrec, to effectively incorporate the multi-field information. we also modify the gradient surgery technique to reduce gradient conflicts and further improve the model performance. finally, we find that combining multi-task learning with traditional attentive multi-field learning achieves the best results. extensive experiments on the mind dataset show the effectiveness of our approach. in the future, we will also combine mtrec with more advanced user modeling methods @xcite . qi zhang, qinglin jia, chuyuan wang, jingjie li, zhaowei wang, and xiuqiang he. 2021. amm: attentive multi-field matching for news recommendation. in sigir, pages 1588-1592. dataset we evaluate our approach on a realworld news recommendation dataset mind @xcite , which is collected from the user behavior logs of microsoft news. there are two versions of the dataset, namely mind-large and mind-small. the mind-large contains more than 15 million impression logs generated by 1 million users, from which the mind-small randomly samples 50,000 users. an impression log records the clicked and non-clicked news that are displayed to a user at a specific time and his historical news click behaviors before this impression. besides, mind contains off-the-shelf category labels and a set of entities of each news.