Contrastive Learning enhanced Author-Style Headline Generation (2022)

Abstract: Headline generation is a task of generating an appropriate headline for a given article, which can be further used for machine-aided writing or enhancing the click-through ratio. Current works only use the article itself in the generation, but have not taken the writing style of headlines into consideration. In this paper, we propose a novel Seq2Seq model called CLH3G (Contrastive Learning enhanced Historical Headlines based Headline Generation) which can use the historical headlines of the articles that the author wrote in the past to improve the headline generation of current articles. By taking historical headlines into account, we can integrate the stylistic features of the author into our model, and generate a headline not only appropriate for the article, but also consistent with the author’s style. In order to efficiently learn the stylistic features of the author, we further introduce a contrastive learning based auxiliary task for the encoder of our model. Besides, we propose two methods to use the learned stylistic features to guide both the pointer and the decoder during the generation. Experimental results show that historical headlines of the same user can improve the headline generation significantly, and both the contrastive learning module and the two style features fusion methods can further boost the performance.

## introduction natural language generation tasks have achieved great success both in research and application, such as neural machine translation @xcite , headline generation @xcite and so on. in many real-life reading scenarios, an attractive headline of the article can immediately grab the readers and then lead them to view the whole article. thus, headline generation (hg) is becoming an important task and draw increasing attention nowadays, which aims to automatically generate the appropriate headline for a given aritcle. earlier research of hg @xcite mainly focus on generating a fluent and relevant headline for a given news article to alleviate the author's work in a machine-aided writing way. more recent works @xcite @xcite intend to generate attractive headlines for articles so as to get higher click-through ratio and further directly improve the profit of the online social media platforms. furthermore, some works @xcite try to generate keyphrase-aware and personalized headlines to meet the requirements of different application scenarios and further satisfy users' personal interests. however, most of these existing methods only use the information of the article to generate the headline but ignore the author's historical headlines. in general, the headlines manually designed by the author are usually more suitable for the author's articles but the design style of the headline may be quite different from the writing style of the article and we can hardly obtain the author's headline style only through the content of the article. therefore, when we integrate existing historical headlines into the hg model to learn the headline style of the author, such as grammar and syntax, the model can generate more appropriate headlines for machine-aided writing. for example, as shown in figure 1 , all of the historical headlines and the reference headline are composed of two clauses. the generated headline of the historical headlines based hg model has the same syntax with the historical headlines, which makes it more likely to be accepted by the author and more attractive than the generated headline with distinctive syntax by the general hg model. to our best knowledge, there is no corpus that contains both news articles and corresponding authorships to meet the requirement of our experiments. therefore, in this paper, we build a new dataset named h3g(historical headlines based headline generation) to explore the research of historical headlines based hg. we collect the h3g dataset from the online social media platform tencent qqbrowser, which contains more than 380k news articles from more than 23k different authors. the detailed introduction of the h3g dataset is discussed in the experiment section. besides, we propose a novel contrastive learning enhanced historical headlines based headline generation (clh3g) model to extract and learn headline styles for hg. inspired by the existing style transfer models @xcite , we represent the headline style of the historical headlines as a single vector. such a design can not only reduce the computation cost of the historical headlines representation, but also facilitate the integration of historical headlines information on the decoder side of the hg model. besides, two different methods are applied to guide the generation of the author-style headlines through the single headline vector. the first style vector fusion method can instruct the decoder to generate author-style target headline representation, and the other controls the generated words of the pointergenerator network. what's more, on the encoder side of sequence-to-sequence (seq2seq) model, we also use contrastive learning (cl) to distinguish headlines from different authors as an auxiliary task, which is consistent with and conduce to the extraction of the headline style. experimental results on automatic metrics rouge and bleu and human evaluation show that the historical headlines can greatly improve the effectiveness of hg compared with general hg models, and both of the two style vector fusion methods and contrastive learning based auxiliary task can also improve the performance. we also train a contrastive learning classifier to distinguish headlines from different authors, and find our clh3g model can generate more author-style headlines than the general hg models and other compared models. to this end, our main contributions are summarized as follows: the general hg models want to generate a fluent and suitable headline given an article. an early work @xcite uses linguisticallymotivated heuristics to generate a matching headline. this method is very safe, because all words in the generated headline are selected from the original article. then, some works @xcite use end-to-end neural networks to generate headlines. these methods achieve the state-of-the-art results and are very convenient for training and inference. besides, @xcite uses a coarse-to-fine model to generate headlines for long articles. the style-based headline generation models aims to generate headline with specific styles. @xcite uses reinforcement learning to generate sensational headlines to capture reader's interest. @xcite proposes dual-attention sequence-to-sequence model to generate questionstyle headlines, because they find question-style headlines can get much higher click-through ratio. besides, @xcite uses parameter sharing scheme to generate general, humorous, romantic, click-baity headlines at the same time. adaptive headline generation models want to generate different headlines for different scenarios. @xcite proposes to generate different headlines with different keywords, which can be used to generate different headlines for different search queries in search engines. @xcite uses the user impression logs of news to generate personalized headlines for different users to satisfy their different interests. contrastive learning is very popular recently for representation learning. cl was first used for vision understanding in @xcite . subsequently, cl is also used in natural language generation, including conditional text generation @xcite , dialogue generation @xcite , report generation @xcite ) and text summarization @xcite . in this paper, we use cl like @xcite , whose framework includes a neural network encoder and a small neural network projection. ## model figure 2 shows our proposed contrastive learning enhanced h3g (clh3g) model, which is an end-to-end seq2seq generation model. we will briefly introduce the entire model in section 3.1 and discuss the encoder and the cl based auxiliary task in section 3.2. finally, the decoder and two headline style vector fusion methods are presented in section 3.3. ## conclusion in this paper, we discuss the effectiveness of historical headlines for headline generation, and aim to generate headlines not only appropriate for the given news articles, but consistent with the author's style. we build a large historical headlines based headline generation dataset, and propose a novel model clh3g to integrate the historical headlines effectively, which contains a contrastive learning based auxiliary task and two headline style vector fusion methods. experimental results show the effectiveness of historical headlines for headline generation and the exceptional performance of both the cl based auxiliary task and the two headline style vector fusion methods of our clh3g model.