D iv HSK : Diverse Headline Generation using Self-Attention based Keyword Selection (2023)

Abstract: Diverse headline generation is an NLP task where given a news article, the goal is to generate multiple headlines that are true to the content of the article but are different among themselves. This task aims to exhibit and exploit semantically similar one-to-many relationships between a source news article and multiple target headlines. Toward this, we propose a novel model called DIVHSK. It has two components:KEYSELECT for selecting the important keywords, and SEQGEN, for finally generating the multiple diverse headlines. In KEYSELECT, we cluster the self-attention heads of the last layer of the pre-trained encoder and select the most-attentive theme and general keywords from the source article. Then, cluster-specific keyword sets guide the SEQGEN, a pre-trained encoder-decoder model, to generate diverse yet semantically similar headlines. The proposed model consistently outperformed existing literature and our strong baselines and emerged as a state-of-the-art model. We have also created a high-quality multi-reference headline dataset from news articles.

## introduction generating diverse and semantically similar multiple outputs in natural language generation (nlg) is an important and challenging task @xcite . the traditional single headline generation task is formulated as a sequence-to-sequence learning problem and has been extensively studied for more than a decade now @xcite @xcite @xcite . recently, researchers are also interested towards diverse output sequence generation tasks. this falls into the one-to-many generation category and is being studied for multiple tasks such as paraphrase generation @xcite , machine translation @xcite , question generation @xcite and summarization @xcite . in this work, we consider the problem of generating diverse headlines given a single news article. diverse headlines present the theme of the article in semantically related yet lexically different short sentences, which may attract different sets of audiences and increase the consumption of the news. the existing approaches for diverse sequence generation mostly diversify the decoding steps through alternative search algorithms @xcite or mixture decoder approaches @xcite where different decoders generate difference output sequences. recently, @xcite proposed a two-stage modeling involving a diversification stage to extract diversifying attributes and a generation stage to guide the encoder-decoder model for diverse generations. the diversifying attributes are keywords extracted from the input text with the expectation-maximization algorithm. they consider text summarization and questiongeneration tasks. in similar lines, @xcite leverage external knowledge graph, i.e., concept-net @xcite to extract diverse yet relevant keywords at diversification stage and generate diverse common sense reasoning texts. these models are not directly applicable for diverse headline generation tasks because the headlines are mostly oriented toward a single common theme @xcite in a short sentence, and these models distract the semantics of generated headlines. our empirical experiments (section-5) validate this point. @xcite used manually extracted keywords with a multi-source transformer for diverse headline generation. the model is not scalable to other datasets/tasks because keyword extraction requires a human annotator. unlike these, we used an automated self-attention-based approach to obtain the most attentive keywords from the article automatically. to overcome the limitations of the existing models, we propose divhsk, a simple yet effective model for diverse headline generation using a selfattention-based keyword selection. the model has two modules/components: (a) keyselect -a pretrained encoder model to extract diversifying attributes i.e. theme and general keywords from input news article and (b) seqgen -a regular pre-trained encoder-decoder architecture guided by diversifying attributes for generating multiple diverse yet semantically similar headlines. ## problem formulation given a news article, the goal is to generate semantically similar, grammatically coherent, fluent and diverse headlines. formally, given a news article x, the goal is to model the conditional distribution for k target outputs p(y k |x) with valid mappings @xmath0, . . . , @xmath1, y 2 , . . . , y k } should be diverse. here we consider @xmath2, i.e., the task is to generate three diverse headlines. ## methodology the proposed divhsk model has two components (1) pre-trained encoder, i.e., keyselect and (2) regular pre-trained encoder-decoder, i.e., seqgen. as per @xcite , multiple headlines should convey the common theme, differing on a lexical level and the headline tokens should be uniformly distributed across the source article. towards these goals, in keyselect, we first cluster the encoders' last-layer self-attention heads to find the most attentive keywords for each cluster from the input news article. we observe that: (a) all the clusters have a few most-attentive common keywords called as theme and (b) cluster-specific most attentive keywords called as general (i.e., non-theme) keywords. we combine theme with cluster-specific general keywords to create diversifying attributes. for each of the k clusters, there is a corresponding diversify-ing attribute. table-4, in appendix, presents a few sample themes and general keywords. the input news article, theme, and general keywords (from diversifying attributes) are concatenated with [sep] tokens to create modified input for the seqgen module. in this way, different cluster leads to generate diverse headlines. the theme and general keywords in the cluster lead to semantically similar and theme-oriented headlines. for pre-trained encoder and pre-trained encoderdecoder models, we use the 'encoder of t5-base' @xcite and t5-base checkpoints, respectively. see figure 1 for an overview of the proposed model. more details about each component are given below: ## conclusion in this work, we present a novel task and dataset for diverse headline generation. we also propose a strong neural architecture for the task. the model, referred to as divhsk, uses self-attentionbased clustering to create diversifying attributes that guide the pre-trained encoder-decoder model to generate diverse headlines. we empirically demonstrate that the divhsk consistently outperforms all baseline models on both automated and human evaluation metrics, while maintaining diversity as a key criterion.