General then Personal: Decoupling and Pre-training for Personalized Headline Generation (2023)

Abstract: Personalized Headline Generation aims to generate unique headlines tailored to users’ browsing history. In this task, understanding user preferences from click history and incorporating them into headline generation pose challenges. Existing approaches typically rely on predefined styles as control codes, but personal style lacks explicit definition or enumeration, making it difficult to leverage traditional techniques. To tackle these challenges, we propose General Then Personal (GTP), a novel framework comprising user modeling, headline generation, and customization. We train the framework using tailored designs that emphasize two central ideas: (a) task decoupling and (b) model pre-training. With the decoupling mechanism separating the task into generation and customization, two mechanisms, i.e., information self-boosting and mask user modeling, are further introduced to facilitate the training and text control. Additionally, we introduce a new evaluation metric to address existing limitations. Extensive experiments conducted on the PENS dataset, considering both zero-shot and few-shot scenarios, demonstrate that GTP outperforms state-of-the-art methods. Furthermore, ablation studies and analysis emphasize the significance of decoupling and pre-training. Finally, the human evaluation validates the effectiveness of our approaches.1

## introduction the task of headline generation aims to produce a concise sentence for expressing the salient information of the document @xcite . in addition to preserving the content information, recent studies have further proposed generating appealing headlines. for instance, some researchers propose to inject specific styles, such as humor and romance, into news headlines or generate interrogative ones to attract readers' attention @xcite @xcite . however, these approaches only consider one type of style to catch the attention, neglecting that individuals may possess different preferences. therefore, personalized headline generation (phg) @xcite has emerged as a new research direction, which customizes news headlines by considering user information. specifically, given body-headline pairs along with users' click history, the goal is to infer the implicit user preference and further incorporate it into generating personalized headlines. besides, the task is usually formulated as zero-shot learning due to the high cost of collecting large-scale personalized headlines @xcite . nevertheless, the task formulation brings the first critical challenge, i.e., generating personalized headlines without ground-truth annotation. in such a scenario, models are required to incorporate the implicit user preference without explicit supervision for personal text styles. the previous work @xcite tackles the task by reinforcement learning, taking the users' click history as a learning signal to construct the style supervision. however, their learning framework does not leverage the news headlines that can contribute to improving headline quality. therefore, we make the first attempt to facilitate personalization learning without or with limited ground-truth annotations while enhancing the generation quality by leveraging the news headlines. in this paper, we propose a framework named general then personal (gtp) to tackle this challenge. specifically, we propose to decouple the generation process into headline generation and headline customization. the goal of headline generation is to produce headlines targeting a general audience, while headline customization aims to further customize them based on the control code foot_1 of a specific user. with the task decoupling, we could pre-train the headline generator using body-headline pairs from a diverse range of news articles, thus improving the quality of generation. afterward, the control code and generated headlines are jointly utilized by the headline customizer. however, constructing the control code poses several challenges. in previous work, the control code typically refers to specific and discrete attributes of the target headlines, such as topics, sentiments, keywords, or descriptive prompts @xcite @xcite . while for phg task, the target attributes are the user preferences encapsulated in the click histories, which cannot be directly defined. as such, we follow a similar approach to @xcite and pre-train recommendation models on impression logs to extract the features of click histories as the control codes. to train the headline customizer utilizing the control codes, we could form the training samples with news articles and corresponding user click histories. however, since a news article can attract multiple users, each with potentially distinct preferences, the pairing between news and user click history is not one-to-one. this limitation impedes the learning of preference control. thus, we construct a new dataset to alleviate the issue, which assumes that users who click on the same news have similar preferences. specifically, we integrate the click histories of these users to synthesize a pseudo click history, which helps build a new user profile with more specific interests in the target news. moreover, due to the distinct latent space of control codes compared to regular text features, models may inadvertently disregard the control codes during the learning process. thus, we design two mechanisms, masked user modeling (mum) and information self-boosting (isb), to alleviate these issues. mum serves as a pre-training objective to make the control code recognizable to the model, while isb leverages the generated headline to recall information from the article, reducing information loss in the two-stage generation process. finally, the last issue is the lack of evaluation metrics. previous works only depend on lexical similarity for evaluation, constraining the models to generate one type of headline. however, given an article, a user could be attracted by various headlines beyond ground-truth ones. this argument is akin to the multi-target summarization problem @xcite , suggesting multiple valid summaries could exist for an article. to benchmark the degree of personalization, we propose the anomaly-based personalization evaluation (ape) metric, inspired by the anomaly detection task and the evaluation metrics used in the field of vision domain. to implement ape, we train auto-encoder models to assess whether an input headline adheres to the same distribution as user-written ones. the detection model is expected to learn text style to distinguish the inputs, enabling us to consider the hidden states as style features. we quantify the results by measuring the feature distance between generated and user-written headlines. however, relying solely on distance measurements may not provide a comprehensive quality assessment. to offer a more intuitive metric, we introduce the editor headlines as reference points for comparison and employ relative values to convey the results. unlike rouge scores, which focus on lexical similarity, ape enables a distribution-wise evaluation, providing a more flexible reference for assessment. the contributions are summarized as follows: ## methodology in this section, we elaborate on how we tackle the phg task as shown in fig. 1 . the following sec-tions are organized as follows. sec. 4.1 describes how to establish the control code from users' click histories. sec. 4.2 introduces the generation framework to incorporate the control code effectively. sec. 4.3 presents the training strategy and the process of pre-training dataset construction. finally, sec. 4.4 introduces an evaluation metric to quantify the degree of personalization. ## conclusion in this paper, we propose a novel framework named general then personal (gtp) to tackle the challenges of constructing and incorporating control code for personalized headline generation. specifically, we propose a late fusion model by decoupling the generation process. two mechanisms are further introduced to facilitate the framework and enable text control. additionally, we construct a pre-training dataset, pens-sh, to build an effective control code, which enhances the connections between click history and target news. moreover, we introduce a novel evaluation metric, ape, to quantify the degree of personalization. the extensive experiments and human evaluation demonstrate the necessity of all designs and show that gtp significantly outperforms state-of-the-art under both zero-shot and few-shot settings.