PENS : A Dataset and Generic Framework for Personalized News Headline Generation (2021)

Abstract: In this paper, we formulate the personalized news headline generation problem whose goal is to output a user-specific title based on both a user’s reading interests and a candidate news body to be exposed to her. To build up a benchmark for this problem, we publicize a large-scale dataset named PENS (PErsonalized News headlineS). The training set is collected from user impressions logs of Microsoft News, and the test set is manually created by hundreds of native speakers to enable a fair testbed for evaluating models in an offline mode. We propose a generic framework as a preparatory solution to our problem. At its heart, user preference is learned by leveraging the user behavioral data, and three kinds of user preference injections are proposed to personalize a text generator and establish personalized headlines. We investigate our dataset by implementing several state-of-the-art user modeling methods in our framework to demonstrate a benchmark score for the proposed dataset. The dataset is available at https://msnews.github.io/pens.html .

## introduction news headline generation @xcite @xcite @xcite @xcite @xcite @xcite , conventionally considered as a paradigm of challenging text summarization task, has been extensively explored for decades. their intuitive intention is to empower the model to output a condensed generalization, e.g., one sentence, of a news article. the recent year escalation of online content vendors such as google news, topbuzz, and etc @xcite propels a new research direction that how to decorate the headline as an irresistible invitation to users for reading through the article @xcite since more readings may acquaint more revenue of these platforms. to this end, specified stylized headline generation techniques were proposed, such as question headline @xcite , sensational headline @xcite generation, and so on @xcite . however, the over-decorate headlines might bring negative effects as click-baits begin to become notorious in ubiquitous online services foot_0 . hence, the question is now changing to how to construct a title that catches on reader curiosity without entering into click-bait territory. inspired by the tremendous success of personalized news recommendation @xcite @xcite where the ultimate goal is to learn users' reading interests and deliver the right news to them, a plausible solution to this question could be producing headlines satisfying the personalized interests of readers. it thus motivates the study of the personalized news headline generation whose goal is to output a user-specific title based on both a user's reading interests and a candidate news body to be exposed to her. analogous to personalized news recommendations, user preference can be learned by leveraging the behavioral data of readers on content vendors, and the representation could personalize text generators and establish distinct headlines, even with the same news body, for different readers. however, it might be difficult to evaluate the approaches of personalized headline generation due to the lack of large-scale available datasets. first, there are few available benchmarks that simultaneously contain user behavior and news content to train models. for example, most available news rec-ommendation datasets may predominately contain user-side interaction data, e.g., exposure impressions and click behaviors, but the textual features usually have already been overly pre-processed @xcite . as a result, advanced nlp techniques that extract useful features from textual data are limited. news headline generation datasets, on the other hand, usually consist of news bodies as well as their headlines, which all come from the news-side @xcite ) rather than the user-side. though the mind dataset @xcite , which was presented by microsoft, simultaneously contains the user-side behavioral data and the news-side original textual data, it was constructed for personalized news recommendations rather than our problem. the more challenging issue for evaluating personalized headline generation approaches is the severe cost during the test phase. it could be intractable and infeasible to do an a/b test for every model in online environments. an efficient and fair testbed to evaluate the models in an offline mode is in urgent demand to make the effectiveness and reproducibility of proposed models comparable. to this end, we publicize a dataset named pens (personalized news headlines) in this paper as a benchmark to testify the performance of personalized news headline generation approaches. the training set of pens is collected from the user impression logs of microsoft news foot_1 , in which 500, 000 impressions over 445, 765 users on more than one hundred thousand english news articles are provided. in addition, we collected 103 english native speakers' click behaviors as well as their more than 20, 000 manually-crafted personalized headlines of news articles on the same news corpus for testing. these manually-written headlines are regarded as the gold standard of the user-preferred titles. then, proposed methods can take prevailing matching metrics, e.g., rouge, bleu and etc., to verify the performance. moreover, we propose a generic framework to inject personalized interests into a proposed neural headline generator to enable a beacon for this area, considering there are few existing works that can generate personalized news headlines. in more detail, we devise three kinds of incorporation methods to inject user interest representation into a proposed neural headline generator with a transformer-based encoder and a pointer network-based @xcite news encoder news encoder news encoder news encoder user encoder … candidate news clicked news … dot click probability click predictor r u r v figure 1: personalized news recommendation framework. 2017) decoder. we implement six state-of-the-arts personalized news recommendation approaches to model user preferences and provide a horizontal standard for the pens dataset. the experimental results show effective personalization modeling and comprehensive injection of user interests can underpin an improvement in the quality of personalized news headline generation. we expect pens can serve as a benchmark for personalized headline generation and bolster the research in this area. ## problem formulation and discussion in this section, we formulate the problem of personalized news headline generation and differentiate it from personalized news recommendations. ## pens dataset in this section, we detail our pens dataset. the dataset was randomly sampled impression logs of microsoft news from june 14 to july 12, 2019. both user behaviors and news contents are involved, and each user was de-linked from the production system when securely hashed into an anonymous id to reserve the data privacy issues. ## our framework in this section, we illustrate our generic framework for resolving personalized news headline generation, and its key issue is how to inject the user preference into a news headline generator. we devise a headline generator with a transformer encoder and a pointer network decoder as our base model and propose three kinds of manners of injecting the user interests to generate personalized headlines. the user interests can be derived following the approaches in news recommendations community, and we omit its details due to the space limitation. the architecture of our proposed framework is shown as figure 3 . candidate news body clicked news … v news encoder news encoder news encoder user encoder … c u n c u 2 c u 1 user embedding transformer encoder attention distribution p t gen encoder hidden states x t x t-1 decoder hidden states f1 f2 context vector vocabulary distribution p vocab (y t ) final distribution ×p t gen ×(1p t gen ) p(y t ) figure 3 : the generic framework of personalized news headline generation. three kinds of user embedding injections are devised. 1 : utilizing user embedding to initialize the decoder's hidden state of the headline generator. 2 : personalizing the attentive values on words in the news body by the user embedding. 3 : perturbing the choice between generation and copying via the user embedding. ## experimental evaluation in this section, we investigate our proposed pens dataset and conduct several comparisons to give benchmark scores of personalized headline generation on this dataset. in the following part, we will introduce the compared methods first, and then detail the experimental setup, and finally present the results and analysis. ## conclusion and future work in this paper, we formulated the problem of personalized news headline generation. to provide an offline testbed for this problem, we constructed a dataset named pens from microsoft news. the news corpus of this dataset contains more than 100 thousand news articles over 15 topic categories. the training set constitutes of 500, 000 impressions of 445, 765 users to learn user interests and construct personalized news headline generator by distant supervisions. the test set was constructed by 103 annotators with their clicked behaviors and manually-written personalized news headlines. we propose a generic framework that injects user interests into an encoder-decoder headline generator in three different manners to resolve our problem. we compared both sota user modeling and headline generating approaches to present benchmark scores on the proposed dataset. for future work, we first believe designing more complex and refined approaches to generated more diversified personalized news headlines will be interesting. more importantly, how to improve personalization while keeping factualness will be another interesting work, and it will propel the methods deployable in practical scenarios. third, news headline personalization might burgeon the news content personalization, which is a more challenging but interesting open problem.