Leveraging Key Information Modeling to Improve Less-Data Constrained News Headline Generation via Duality Fine-Tuning (2022)

Abstract: Recent language generative models are mostly trained on large-scale datasets, while in some real scenarios, the training datasets are often expensive to obtain and would be small-scale. In this paper we investigate the challenging task of less-data constrained generation, especially when the generated news headlines are short yet expected by readers to keep readable and informative simultaneously. We highlight the key information modeling task and propose a novel duality fine-tuning method by formally defining the probabilistic duality constraints between key information prediction and headline generation tasks. The proposed method can capture more information from limited data, build connections between separate tasks, and is suitable for less-data constrained generation tasks. Furthermore, the method can leverage various pre-trained generative regimes, e.g., autoregressive and encoder-decoder models. We conduct extensive experiments to demonstrate that our method is effective and efficient to achieve improved performance in terms of language modeling metric and informativeness correctness metric on two public datasets.

## introduction in an age of information explosion, headline generation becomes one fundamental application in the natural language process (nlp) field @xcite . currently, the headline generation is usually regarded as a special case of general text summarization. therefore, many cutting-edge techniques based on pre-trained models and finetuning methods can be directly adapted by feeding headline generation datasets @xcite . actually, compared with those textual summaries, headline generation aims at generating only one sentence or a piece of short texts given a long document (e.g., a news article). it is challenging to guarantee the generated headline readable and informative at the same time, which is important to attract or inform readers especially for news domain @xcite . recently, some works find that neglecting the key information would degrade the performance of generative models which only consider capturing natural language @xcite . then many works about modeling different kinds of key information have been studied to enhance the information correctness of generative summaries. for example, overlapping salient words between source document and target summary @xcite , keywords @xcite , key phrases @xcite and named entities @xcite ) are involved to design generative models. however, those works are mostly either trained on large-scale datasets or targets for long summaries @xcite . in some real applications, it is expensive to obtain massive labeled data. thus it becomes a much more challenging task that how to generate short headlines which should be both readable and informative under less-data constrained situations. to model the key information, existing works often follow the assumption that a generated summary essentially consists of two-fold elements: the natural language part and the key information part. the former focuses on language fluency and readability, while the later is for information correctness. for this reason, an additional task of key information prediction is leveraged and the multitask learning method is employed @xcite . figure 1 can illustrate the intuitive idea more clearly, and the bold parts can be treated as the key information (overlapping salient tokens), which should be modeled well to inform correct and sufficient information for readers. to achieve the above motivation, technically, applying existing fine-tuning and multi-task learning methods to headline generation can be a natural choice. however they have some drawbacks. ## related work usually, headline generation is regarded as a special task of general abstractive text summarization, and the majority of existing studies could be easily adapted to headline generation by feeding headline related datasets @xcite . for example, sequence-to-sequence based models are investigated for text summarization, which emphasizes on generating fluent and natural summaries @xcite @xcite . in recent years, the large-scale transformer-based models @xcite @xcite and the two-stage (pre-training and fine-tuning) learning paradigm @xcite @xcite have greatly promoted the performance of most nlp tasks. and headline generation can also benefit from those works. since the length of headlines is often short and almost 'every word is precious', compared to general text summarization, modeling the key information is better worth of paying attention @xcite @xcite @xcite . however, to our knowledge, little work focuses on this problem for headline generation, especially under the less-data constrained situations, and mostly they focus on low-resource long text summarization @xcite @xcite . recent years witness the rapid development of transformers-based pre-trained models @xcite and two kinds of regimes of natural language generation (nlg) are prevalent @xcite . one is based on autoregressive language models which have a shared transformer encoder structure for encoding and decoding @xcite @xcite , while the other is based on the standard transformer framework which has two separate encoderdecoder structures @xcite . fine-tuning and multi-task learning on them to reuse the ability of pre-trained models are widely studied for various tasks @xcite @xcite . our work can also align with this research line and we propose a new multi-task fine-tuning method. we leverage the core idea of dual learning, which can fully mine information from limited data and well model multiple tasks by designing duality constraints @xcite . this learning paradigm has been successfully applied to many fields, such as image-to-image translation @xcite , recommendation system @xcite , supervise and unsupervised nlu and nlg @xcite . those works have demonstrated that duality modeling is suitable for small-scale training situations. ## problem definition in this section, we formally present our problem. the training set is denoted as @xmath0, h, k), where d and h are the sets of source documents and target headlines. k is the set of key informa-tion, which indicates the overlapping salient tokens (stopwords excluded) in each pair of document and headline. a training sample is denoted as a tuple (d, h, k). @xmath1 ## conclusion in this paper, we introduce a novel task that how to improve the performance of less-data constrained headline generation. we highlight to explicitly exploit the key information, and propose a novel duality fine-tuning method which firstly integrates dual learning paradigm and fine-tuning paradigm for less-data generation. the proposed method should obey the probabilistic duality constraints, which are critical to model multiple tasks. therefore, the method can model more supervised information, learn more knowledge, and train more powerful generative models. our method can also be generally applied to both autoregressive and encoderdecoder generative regimes. we collect various sizes of small-scale training datasets from two public corpora in english and chinese, and the extensive experimental results prove our method effectively improve the readability and informativeness of generated headlines with different pre-trained models.