Boosting Deep CTR Prediction with a Plug-and-Play Pre-trainer for News Recommendation (2022)

Abstract: Understanding news content is critical to improving the quality of news recommendation. To achieve this goal, recent studies have attempted to apply pre-trained language models (PLMs) such as BERT for semantic-enhanced news recommendation. Despite their great success in offline evaluation, it is still a challenge to apply such large PLMs in real-time ranking model due to the stringent requirement in inference and updating time. To bridge this gap, we propose a plug-and-play pre-trainer, namely PREC, to learn both user and news encoders through multi-task pre-training. Instead of directly leveraging sophisticated PLMs for end-to-end inference, we focus on how to use the derived user and item representations to boost the performance of conventional lightweight models for click-through-rate prediction. This enables efficient online inference as well as compatibility to conventional models, which would significantly ease the practical deployment. We validate the effectiveness of PREC through both offline evaluation on public datasets and online A/B testing in an industrial application.

## introduction personalized news recommendation has become a ubiquitous channel in various online applications, such as google news and msn news, which helps users discover their interested news information. to deal with the massive amounts (usually millions) of daily news, industrial recommender systems usually apply a multi-stage recommendation pipeline as illustrated in figure 1 . it generally involves two phases, matching and ranking @xcite . the matching phase first generates hundreds or thousands of news candidates from multiple channels (e.g., popularity-based channel, content-based channel @xcite , and collaborative filtering channel (linden et al., 2003; * corresponding author. figure 1 : a multi-stage pipeline for news recommendation. @xcite ). subsequently, the ranking phase employs a news ranking task, such as clickthrough rate (ctr) prediction, to rank the candidate news and finally return the top dozens of news to the user interface. the matching and ranking phases have different goals and requirements. concretely, the matching phase aims for efficient and high-recall candidate retrieval from millions of news corpus. therefore, most studies construct two-tower networks (e.g., npa @xcite , lstur @xcite , and nrms @xcite ) to learn user representations and item representations separately, and then apply simple dot product to compute similarity scores. as such, both user and item representations can be pre-computed offline and cached online for fast nearest neighbour search (e.g., using faiss @xcite ). in contrast, the ranking phase targets at accurate scoring of each candidate item based on ctr prediction and thus requires networks to capture complex feature interactions between users and news (e.g., deepfm @xcite , pnn @xcite , and dcn @xcite ). these widely adopted ctr prediction models in industry are usually small (e.g., 3 ∼ 5 layers) to meet the latency requirements for online inference and enable frequent model updates (e.g., hourly or daily). to leverage powerful pre-trained language models (plms) to better capture the semantics of news content, some recent studies @xcite propose the use of plms for news recommendation. for example, as depicted in figure 1 , plm-nr @xcite replaces original news encoders (e.g., cnn in @xcite and multi-head attention in @xcite ) with plms such as bert @xcite to empower existing content-based matching models (e.g., lstur @xcite , naml @xcite , and nrms @xcite ). yet, these studies typically adopt the common pretrain-finetune strategy to train plm-based news encoder and user encoder jointly, which is extremely time-consuming given the massive amount of click data. as a concrete example illustrated in figure 2 , we estimate the updating time of the end-to-end model and the hierarchically decoupled model on mind @xcite . finetuning "plm-nr + nrms" with the mind click data (4m clicks with 1m users and 100k news) takes 2800 minutes for one epoch. this might be tolerable during matching as reported in @xcite , where user and item representations are pre-computed offline, but is absolutely unacceptable for ctr prediction tasks. in this paper, we explore the use of plms for ctr prediction tasks and propose a plug-and-play pre-trainer, namely prec, to learn both user and news encoders through multi-task pre-training. in particular, our prec model has the following key advantages: 1) this is the first work to integrate both news pre-trainer and user pre-trainer for recommendation. the former fuses multi-view news features (e.g., title, abstract, category) for representation learning, while the latter learns user representations from historical interaction sequences depending on the learned news representations and iter-dependencies among them. 2) the prec model is constructed in a hierarchical decoupled manner, bringing rich news contents and deep user interests to ctr models while meeting the requirements of both inference and updating efficiency. as shown in figure 2 , prec's model updating time only reaches about 50 minutes, which is 50+ times faster than plm-nr under the same setting. 3) the decoupled design makes prec easily compatible with various existing ctr prediction models (e.g., pnn @xcite , dcn @xcite , deepfm @xcite )), where user and news representations learned from prec could be used as features or embedding initialization for downstream models. this allows prec to be easily deployed in industrial recommender systems. we conduct extensive experiments to validate the effectiveness of prec on two open benchmark datasets for news recommendation, mind @xcite and @xcite . the experiments show that our pre-trainer prec can consistently boost the performance of existing lightweight ctr prediction models, even outperforming some state-of-the-art heavyweight models. in addition, we have deployed prec in the production system and an online a/b test shows that prec leads to a 2.4% improvement on the overall ctr metric. while pre-trained models have shown great promise in content understanding, lightweight ctr models @xcite @xcite still dominate industrial applications due to its high efficiency, where plms cannot be deployed due to its large size. our proposed pretrainer can work seamlessly with these industrial ctr models and provide them semantic news embeddings and user interest knowledge with almost no overhead in inference time and deployment cost. ## the proposed method in this section, we present our proposed cascaded pre-trainer for news recommendation (prec). we first introduce the model framework, pre-training tasks and pre-training strategy. then, we describe how to apply the pre-trainer for downstream tasks. ## conclusion in this paper, we have developed a plug-and-play pre-trainer called prec to boost the performance of traditional id-based ctr models for news recommendation. prec is built on transformer layers, utilizes multi-view features of news and users, and is trained with tailored pre-training tasks to learn semantic news and user representations. aside from its ability in content understanding, prec can be easily deployed in industrial recommender systems to improve ctr prediction. offline experiments on public benchmark datasets and online a/b testing in industrial recommender systems demonstrate the effectiveness and efficiency of prec.