Diverse Keyphrase Generation with Neural Unlikelihood Training (2020)

Abstract: In this paper, we study sequence-to-sequence (S2S) keyphrase generation models from the perspective of diversity. Recent advances in neural natural language generation have made possible remarkable progress on the task of keyphrase generation, demonstrated through improvements on quality metrics such as F1-score. However, the importance of diversity in keyphrase generation has been largely ignored. We first analyze the extent of information redundancy present in the outputs generated by a baseline model trained using maximum likelihood estimation (MLE). Our findings show that repetition of keyphrases is a major issue with MLE training. To alleviate this issue, we adopt neural unlikelihood (UL) objective for training the S2S model. Our version of UL training operates at (1) the target token level to discourage the generation of repeating tokens; (2) the copy token level to avoid copying repetitive tokens from the source text. Further, to encourage better model planning during the decoding process, we incorporate K-step ahead token prediction objective that computes both MLE and UL losses on future tokens as well. Through extensive experiments on datasets from three different domains we demonstrate that the proposed approach attains considerably large diversity gains, while maintaining competitive output quality.

## introduction automatic keyphrase generation is the task of generating single or multi-word lexical units that provides readers with high level information about the key ideas or important topics described in a given source text. apart from an information summarization perspective, this task has applications in various downstream natural language processing tasks such as text classification @xcite , document clustering @xcite and information retrieval @xcite . traditionally, keyphrases (kps) were extracted from source documents by retrieving and ranking a set of candidate phrases through rule based approaches. with recent advances in neural natural language generation and availability of larger training corpora, this problem is formulated under a sequence-tosequence (s2s) modelling framework @xcite . this approach has an advantage that it can generate new and meaningful keyphrases which may be absent in the source text. the earliest work in this direction was by @xcite , who train a s2s model to generate one keyphrase at a time. at inference time, they decode with beam sizes as high as 200, to generate a large number of kps and finally de-duplicate the outputs. however, this is computationally expensive and wasteful because @xmath0. an alternative approach is to train a s2s model to generate multiple keyphrases in a sequential manner, where the output kps are separated by a pre-defined delimiter token. this method has an added benefit that the model automatically learns to generate a variable number of keyphrases depending on the input, instead of a user-specified fixed number of keyphrases (top-k) from a large list of candidate outputs. however, some previous approaches @xcite still use exhaustive beam search decoding to over-generate kps and then apply post-processing to remove repetitions. apart from the additional computational requirements, we argue that this method of avoiding information redundancy is a last-minute solution. title semi automated schema integration with sasmint abstract the emergence of increasing number of collaborating organizations has made clear the need for supporting interoperability infrastructures , enabling sharing and exchange of data among organizations . schema matching and schema integration are the crucial components of the interoperability infrastructures , and their semi automation to interrelate or integrate heterogeneous and autonomous databases in collaborative networks is desired . the semi automatic schema matching and integration sasmint system introduced in this paper identifies and resolves (...) ground truth schema integration ; collaboration ; schema matching ; heterogeneity ; data sharing mle baseline schema integration ; sasmint ; schema matching ; schema integration ; schema matching ; sasmint derivation markup language divkgen schema integration ; interoperability infrastructures ; schema matching ; sasmint in this paper, we take a principled direction towards addressing the information redundancy issue in keyphrase generation models. we propose to tackle this problem directly during the training stage, rather than applying adhoc post-processing at inference time. specifically, we adopt the neural unlikelihood training (ul) objective @xcite , whereby the decoder is penalized for generating undesirable tokens. @xcite introduce unlikelihood training for a language model setting. since we work with a s2s setup, our version of ul loss consists of two components: (1) a target token level ul loss based on the target vocabulary to penalize the model for generating repeating tokens; (2) a copy token level ul loss based on the dynamic vocabulary of source tokens required for copy mechanism @xcite , which penalizes the model for copying repetitive tokens. s2s models trained with maximum likelihood estimation (mle) are usually tasked with the next token prediction objective. however, this does not necessarily incentivize the model to plan for future token prediction ahead of time. we observe such lack of model planning capability in our initial experiments with mle models and to overcome this issue we propose to use k-step ahead token prediction. this modified training objective encourages the model to learn to correctly predict not just the current token, but also tokens upto k-steps ahead in the future. we then naturally incorporate ul training on the k-step ahead token prediction task. we summarize our contributions as follows: (1) to improve the diversity of generated keyphrases in a principled manner during training, we adopt the unlikelihood objective for the s2s setting and propose a novel copy token unlikelihood loss. (2) in order to incentivize model planning, we augment our training objective function to incorporate k-step ahead token prediction. additionally, we also introduce the k-step ahead unlikelihood losses. (3) we propose new metrics for benchmarking keyphrase generation models on diversity criterion. we carry out experiments on datasets from three different domains (scientific articles, news and community qa) and validate the effectiveness of our approach. we observe substantial gains in diversity while maintaining competitive output quality. ## proposed approach rather than addressing the information redundancy issue through post-processing, we take a principled approach in this direction during training itself. firstly, we adopt neural unlikelihood training @xcite for sequence-to-sequence setting by directly penalizing the decoder for either generating or copying repeating tokens. secondly, we improve the planning capability of the decoder by incorporating a k-step ahead token prediction loss. this is achieved by using the same decoder hidden state but different attention mechanisms to decide which source tokens to be attended to, for predicting the target at the current time step, 1-step ahead and so on. an illustration of our approach is presented in figure 1 . ## experiment setup evaluation metrics. to measure the quality of generated keyphrases, i.e., its relevance with respect to the source document, we compare the generated set of kps to the kps in the corresponding ground truth data. to this end, we report f 1 @m , where m refers to the number of model predicted keyphrases. we also include the corresponding precision and recall metrics. as justified in previous work @xcite , f 1 @m captures the ability of abstractive s2s models in generating a variable number of kps depending on the source document, in comparison to traditional extractive methods where one is required to specify a cutoff in order to output the top-k keyphrases. however, different from previous work, we report the overall f 1 @m score rather than separately computing this score for keyphrases present vs. absent in the source text. this is because our goal in this work is to overcome the lack of diversity issue in keyphrase generation models, and not necessarily to generate more absent keyphrases. in order to evaluate the model outputs on the criterion of diversity, we define the following metrics: the next three metrics measure the inter-keyphrase similarity among the generated set of keyphrases -a lower value indicates fewer repetitions and thus more diversity in the output. self-bleu: we use self-bleu @xcite which computes pairwise bleu score @xcite between generated kps. this metric captures word level surface overlap. ## results and analysis we report quality and diversity metrics on five baselines and three variants of the proposed approach ( with regards to quality evaluation metrics, it can be observed that divkgen models have slightly lower scores. this can be explained from a quality-diversity trade-off viewpoint. as the model attempts to explore the output space through the generation of more interesting kps, it may output new kps that are not present in the ground truth, thus resulting in lower precision. divkgen generates shorter sequences (and hence may not be able to produce all the kps as per the ground truth) than the baselines, which could explain the lower recall. ## conclusion and future work in this work, we first point out the shortcomings of mle based training for keyphrase generation. we specifically address the lack of output diversity issue via the use of unlikelihood training objective. we adopt a target level unlikelihood loss and propose a novel copy token unlikelihood loss, the combination of which provides large diversity gains. in addition, a k-step ahead mle and ul objective is incorporated into the training. through extensive experiments on datasets from three different domains, we demonstrate the effectiveness of our model for diverse keyphrase generation. for future work, we plan to explore directions that would enable us to simultaneously optimize for quality and diversity metrics.