N ews BERT : Distilling Pre-trained Language Model for Intelligent News Application (2021)

Abstract: Pre-trained language models (PLMs) like BERT have made great progress in NLP. News articles usually contain rich textual information, and PLMs have the potentials to enhance news text modeling for various intelligent news applications like news recommendation and retrieval. However, most existing PLMs are in huge size with hundreds of millions of parameters. Many online news applications need to serve millions of users with low latency tolerance, which poses great challenges to incorporating PLMs in these scenarios. Knowledge distillation techniques can compress a large PLM into a much smaller one and meanwhile keeps good performance. However, existing language models are pre-trained and distilled on general corpus like Wikipedia, which has gaps with the news domain and may be suboptimal for news intelligence. In this paper, we propose NewsBERT, which can distill PLMs for efficient and effective news intelligence. In our approach, we design a teacher-student joint learning and distillation framework to collaboratively learn both teacher and student models, where the student model can learn from the learning experience of the teacher model. In addition, we propose a momentum distillation method by incorporating the gradients of teacher model into the update of student model to better transfer the knowledge learned by the teacher model. Thorough experiments on two real-world datasets with three tasks show that NewsBERT can empower various intelligent news applications with much smaller models.

## introduction pre-trained language models (plms) like bert @xcite and gpt @xcite have achieved remarkable success in various nlp applications @xcite . these plms are usually in huge size with hundreds of millions of parameters @xcite . for example, the bert-base model contains about 110m parameters and 12 transformer @xcite layers, which may raise a high demand of computational resources in model training and inference. however, many online applications need to provide services for a large number of concurrent users and the tolerance of latency is often low, which hinders the deployment of large-scale plms in these systems @xcite . in recent years, online news websites such as msn news and google news have gained huge popularity for users to digest digital news @xcite . these news websites usually involve a series of intelligent news applications like automatic news topic classification @xcite , news headline generation @xcite and news recommendation @xcite @xcite . in these applications, text modeling is a critical technique because news articles usually contain rich textual content @xcite . thus, these applications would benefit a lot from the powerful language understanding ability of plms if they could be incorporated in an efficient way, which further has the potential to improve the news reading experience of millions of users @xcite . knowledge distillation is a technique that can compress a cumbersome teacher model into a lighter-weight student model by transferring useful knowledge @xcite . it has been employed to compress many huge pre-trained language models into much smaller versions and meanwhile keep most of the original performance @xcite @xcite . for example, @xcite proposed a distil-bert approach, which learns the student model from the soft target probabilities of the teacher model by using a distillation loss with softmaxtemperature @xcite , and they regular-ized the hidden state directions of the student and teacher models to be aligned. @xcite proposed tinybert, which is an improved version of distilbert. in addition to the distillation loss, they proposed to regularize the token embeddings, hidden states and attention heatmaps of both student and teacher models to be aligned via the mean squared error loss. these methods usually learn the teacher and student models successively, where the student can only learn from the results of the teacher model. however, the learning experience of the teacher may also be useful for the learning of student model @xcite , which is not considered by existing methods. in addition, the corpus for pre-training and distilling general language models (e.g., wikipedia) may also have some domain shifts with news corpus, which may not be optimal for intelligent news applications. in this paper, we propose a newsbert approach that can distill plms for various intelligent news applications. in our approach, we design a teacherstudent joint learning and distillation framework to collaboratively learn both teacher and student models in news intelligence tasks by sharing the parameters of top layers, and meanwhile distill the student model by regularizing the output soft probabilities and hidden representations. in this way, the student model can learn from the teacher's learning experience to better imitate the teacher model, and the teacher can also be aware of the learning status of the student model to enhance student teaching. in addition, we propose a momentum distillation method by using the gradients of the teacher model to boost the gradients of student model in a momentum way, which can better transfer useful knowledge learned by the teacher model to enhance the learning of student model. we conduct extensive experiments on two real-world datasets that involve three news intelligence tasks. the results validate that our proposed newsbert approach can consistently improve the performance of these tasks using much smaller models and outperform many baseline methods for plm distillation. the main contributions of this work include: ## related work in recent years, many researchers explore to use knowledge distillation techniques to compress large-scale plms into smaller ones @xcite @xcite @xcite @xcite @xcite . for example, tang et al. (2019) proposed a bilstm soft method that distills the bert model into a single layer bilstm using the distillation loss in downstream tasks. sanh et al. (2019) proposed a distilbert approach, which distills the student model at the pre-training stage using the distillation loss and a cosine embedding loss that aligns the hidden states of teacher and student models. sun et al. (2019) proposed a patient knowledge distillation method for bert compression named bert-pkd, which distills the student model by learning from teacher's output soft probabilities and hidden states produced by intermediate layers. wang et al. (2020b) proposed minilm, which employs a deep self-attention distillation method that uses the kl-divergence loss between teacher's and student's attention heatmaps computed by query-key inner-product and the value relations computed by value-value inner-product. @xcite proposed tinybert, which distills the bert model at both pre-training and finetuning stages by using the distillation loss and the mse loss between the embeddings, hidden states and attention heatmaps. there are also a few works that explore to distill pre-trained language models for specific downstream tasks such as document retrieval @xcite . for example, @xcite proposed a twinbert approach for document retrieval, which employs a two-tower architecture with two separate language models to encode the query and document, respectively. they used the distillation loss function to compress the two bert models into smaller ones. these methods usually train the teacher and student models successively, i.e., distilling the student model based on a well-tuned teacher model. however, the useful experience evoked by the teacher's learning process cannot be exploited by the student and the teacher is also not aware of the student's learning status. in addition, the corpus for pre-training and distilling these language models usually has some domain shifts with news texts. thus, it may not be optimal to apply the off-theshelf distilled language models to intelligent news applications. in this work, we propose a news-bert method to distill pre-trained language models for intelligent news applications, which can effectively reduce the computational cost of plms and meanwhile achieve promising performance. we propose a teacher-student joint learning and distillation framework, where the student model can exploit the useful knowledge produced by the learning process of the teacher model. in addition, we propose a momentum distillation method that integrates the gradient of the teacher model into the student model gradient as a momentum to boost the learning of the student. ## newsbert in this section, we introduce our newsbert approach that can distill plms for intelligent news applications. we will first introduce the teacherstudent joint learning and distillation framework of newsbert by using the news classification task as a representative example, then introduce our proposed momentum distillation method, and finally introduce how to learn newsbert in more complicated tasks like news recommendation. ## conclusion in this paper, we propose a knowledge distillation approach named newsbert to compress pretrained language models for intelligent news applications. we propose a teacher-student joint learning and distillation framework to collaboratively train both teacher and student models, where the student model can learn from the learning experience of the teacher model and the teacher model is aware of the learning of student model. in addition, we propose a momentum distillation method that combines the gradients of the teacher model with the gradients of the student model in a momentum way, which can boost the learning of student model by injecting the knowledge learned by the teacher. we conduct extensive experiments on two real-world datasets with three different news intelligence tasks. the results show that our newsbert approach can effectively improve the performance of these tasks with considerably smaller models.