KPDROP : Improving Absent Keyphrase Generation (2022)

Abstract: Keyphrase generation is the task of generating phrases (keyphrases) that summarize the main topics of a given document. Keyphrases can be either present or absent from the given document. While the extraction of present keyphrases has received much attention in the past, only recently a stronger focus has been placed on the generation of absent keyphrases. However, generating absent keyphrases is challenging; even the best methods show only a modest degree of success. In this paper, we propose a model-agnostic approach called keyphrase dropout (or KPDrop) to improve absent keyphrase generation. In this approach, we randomly drop present keyphrases from the document and turn them into artificial absent keyphrases during training. We test our approach extensively and show that it consistently improves the absent performance of strong baselines in both supervised and resource-constrained semi-supervised settings.

## introduction keyphrase generation (kg) is the task of producing a set of phrases that best summarize a document. it can be leveraged for various applications such as text summarization @xcite , recommendation @xcite , and opinion mining @xcite . accurate identification of keyphrases especially on scientific papers can improve efficiency in paper indexing and paper retrieval @xcite . keyphrases can be divided into two types: (1) present keyphrases and (2) absent keyphrases. present keyphrases appear verbatim in the document, whereas absent keyphrases are topicallyrelevant but missing from the document. many prior works @xcite focused on present keyphrase extraction exclusively. as such, they are not suitable for generating any absent keyphrases. to overcome this limitation, recent approaches @xcite @xcite use sequence-to-sequence (seq2seq) models to generate both present and absent keyphrases. as shown by @xcite , absent keyphrases that are substantially different from the present ones can significantly improve the effectiveness of document-retrieval. @xcite suggested that absent keyphrases can improve document-retrieval by expanding the query terms to alleviate the vocabulary mismatch problem between the query terms and relevant documents @xcite . thus, there is a strong motivation for improving absent keyphrase performance. however, generating absent keyphrases can be very challenging. even the best keyphrase generation models @xcite still only achieve a modest performance in absent keyphrases. in this work, we propose keyphrase dropout or kpdrop as a simple and effective technique to improve the performance of absent keyphrases. unlike the traditional dropout method that focuses on dropping neurons @xcite , we propose a novel dropout method where, instead of neurons, we randomly drop entire phrases (specifically, present gold keyphrases) from a given document during training. thus, the dropped present keyphrases are turned into (artificial) absent keyphrases. as a result, kpdrop has the following two effects: we apply kpdrop into three distinct neural models representing three distinct paradigms: one2many @xcite , one2one @xcite , and one2set @xcite . we observe consistent and substantial improvement in absent keyphrase performance in supervised settings on five standard datasets used for kg evaluation, with little to no drops in present keyphrase performance (and in fact often yielding improved present performance). additionally, kpdrop can be used to create synthetic absent keyphrases for unlabelled data to be used for self-supervised pre-training. we demonstrate that such pre-training augmented with kpdrop brings substantial improvement when fine-tuned in low-resource labelled data. ## methodology of kpdrop in this section, we formally describe our approach of keyphrase dropout (kpdrop). in keyphrase generation (kg), we have a document x as an input sequence of tokens, and a set of n keyphrases, @xmath0, y 2 , . . . , y n }, as the target output. within y , there is a subset of s (0 ≤ @xmath1, p 2 , p 3 , . . . , p s } (p ⊆ y ) and a subset of t (0 ≤ @xmath2, a 2 , a 3 , . . . , a t } (a ⊆ y ). it also holds true that a ∩ @xmath3. similar to @xcite , we separate absent and present keyphrases by checking if the stemmed version of a keyphrase appears in the stemmed version of the input document (in which case it is present) or does not appear (in which case it is absent). when applying kpdrop, any present keyphrase p @xmath4, where r is the dropout rate (or kpdrop rate). let o (o ⊆ p ) be the subset of present keyphrases that are randomly dropped during the application of kp-drop at some training epoch. for every keyphrase p @xmath5, we remove any and all substrings from x that, when stemmed, match the stemmed version of the phrase p k and replace each removed substring with a special mask token @xcite . we call the modified version of x as x new . the sets of present and absent keyphrases are also modified. the new set of present keyphrases becomes p := p \ o and the new set of absent keyphrases becomes a := a ∪ o. thus, the keyphrases in o, which were originally present, become artificially absent after applying kpdrop. the set y becomes y new with the new versions of p and a. an example of applying kpdrop is shown in figure 1 . note that although as a set y new is the same as y , we use y new to convey that the sets of present and absent keyphrases have changed. ## evaluations we use the following evaluation metrics: @xcite @xcite @xcite used to evaluate the performance of keyphrase generation. this is an f 1 based metric where all the predictions by a given model are considered for evaluation. f 1 @5: f 1 @5 is similar to f 1 @m but only the top 5 predicted keyphrases are used for evaluation (if the total number of predictions are less than 5, all the available predictions are used). this metric is used in several prior works @xcite ). 2 this metric is useful in settings where the model is used to overgenerate keyphrases (for example, by using beam search). overgeneration can lead to lower precision when all the predictions are kept (thus, low f 1 @m). so, often in such settings, it is more useful to check the performance of the model when some simple truncation policy is used, for example, selecting some top k (e.g., top 5 in f 1 @5) predictions. r@10 and r@50: r@10 and r@50 represents macro recall of the top 10 predictions and the top 50 predictions, respectively. in some applications (such as retrieval) high recall can be sometimes more useful than precision. r@10 and r@50 are used in prior works @xcite @xcite to measure absent keyphrase 2 note that the f1@5 metric as used by us and as introduced by @xcite for keyphrase generation is different from the f1@5 metric as introduced by @xcite and used in some other works @xcite . we report results with @xcite 's formulation of f1@5 in appendix a.2 performance after overgenerating them using beam search with high beam size. like prior works, we use macro-f 1 and macrorecall for all the above metrics. ## baselines we use the following baselines with our kpdrop-r and kpdrop-a approaches: gru one2many: gru one2many (also known as catseq) represents a simple seq2seq baseline based on grus. it takes a document input and generates a concatenated sequence of keyphrases similar to @xcite . for this baseline, we concatenate the ground truth keyphrases based on the best performing ordering procedure @xcite : we first concatenate the present keyphrases according to the order of their first appearance in text and then we append the absent keyphrases in their natural order. however, when using kpdrop, we start with the ordering as mentioned but then shift the dropped present keyphrases (artificial absent keyphrases) after the fully present keyphrases but keep them before the naturally absent keyphrases. we maintain the internal order of the dropped present (artificially absent) keyphrases. we use ";" as the delimiter for separating keyphrases. ## supervised experiments in the supervised setting, we explore the effects of applying both kpdrop-r and kpdrop-a on the baselines that we discussed in §4. note that one2many models @xcite had been explored in both greedy search based generations where only a single concatenated sequence of keyphrases is generated and also in beam search based generations where multiple beams of concatenated sequence of keyphrases are generated. we too explore both greedy and beam search for the one2many models. in one2one models, greedy search is ineffective because it only generates a single keyphrase. thus, following meng et al. (2017), we only use beam search to overgenerate multiple beams of keyphrases for one2one models. for one2set models, only greedy search was explored @xcite . here, in addition to greedy search, we also explore using beam search for each control code in one2set models. for all models, we investigate the effect of applying kpdrop in all of their applicable decoding settings (be it beam search or greedy search). following prior works @xcite , we mainly use beam search to demonstrate recall performance ( ## semi-supervised experiments we also investigate whether kpdrop has something to offer in a low-resource semi-supervised setting. we simulate a semi-supervised setting by randomly splitting the training set of kp20k into two parts. in one part, we keep 5000 samples with their original keyphrases intact, but in the other part, we keep the rest of the data after removing the original keyphrases. thus, we are left with a low-resource (5000 samples) author-annotated training data, and a huge unlabelled corpus (rest of the kp20k training set). henceforth, we refer to the former low-resource labelled dataset as lr, and the latter unlabelled corpus as uc. we investigate a pre-training based approach to utilize uc. essentially, we pre-train our models first on uc and then fine-tune them on lr. for pretraining, similar to ye and wang (2018), we create synthetic labels using an unsupervised keyphrase extraction model. unlike @xcite , we use a contemporary embedding-based keyphrase extraction model @xcite to generate the synthetic keyphrases. particularly, we rank the candidate keyphrases and keep the top 10. note that in this pre-training setup, the synthetic keyphrases will only be present keyphrases because they are extracted from the input text. this is where kpdrop can make the unsupervised pre-training more interesting by creating artificial absent keyphrases by dropping of the synthetic present keyphrases (and simulating real data). we hypothesize that the application of kpdrop can help our models to learn more effective weights from uc in the selfsupervised pre-training stage. we test the effectiveness of using kpdrop to augment self-supervised pre-training by testing the pre-trained model on the labelled test sets after fine-tuning on lr. during fine-tuning on lr, for all models, we always use kpdrop-a as we have already shown this to be beneficial in most supervised contexts. hyperpa-rameters are detailed in the appendix a.1. ## preliminary experiments on pre-trained models we also did a few preliminary experiments on applying kpdrop-r to a large pre-trained seq2seq language model, in particular, t5 @xcite . we present the results in appendix a.4. consistent with previous results, we find that kpdrop-r increases absent performance for t5 as well. however, overall, we found the perfor-mance of t5 baseline to be limited compared to trained-from-sratch models like transformer one2set. similarly, other reported performances on pre-trained models have been generally lower than transformers one2set too. for example, even after large scale specialized pre-training for keyphrases, @xcite reports only comparable performance on present keyphrases to transformer one2set @xcite and much less in absent performance. on the other hand, using pre-trained models, wu et al. (2021) foot_1 achieve comparable on absent performance with one2set under greedy search, but much less in present performance. however, it should not be too difficult to adapt a pre-trained model into a one2set framework during fine-tuning. this can be a promising future direction and form a stronger base model for kpdrop. ## related work there is a wide variety of approaches @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite for keyphrase extraction exclusively. meng et al. (2017) diverged from pure extractive methods by introducing a seq2seq model (copyrnn) for generation of both present and absent keyphrases. chen et al. (2018) extended copy-rnn with keyphrase correlation constraints and zhao and zhang (2019) extended it with linguistic constraints. ye and wang (2018); wu et al. (2022) investigated keyphrase generation (kg) in semisupervised or resource-constrained settings. chen et al. (2019b) used a title-guided encoding method for better kg. wang et al. (2019) incorporated a topic-model to enhance kg. yuan et al. (2020) extended copyrnn by introducing the catseq model that can generate a concatenation of dynamically determined variable number of keyphrases. chan et al. (2019); luo et al. (2021) improved kg using reinforcement learning whereas swaminathan et al. (2020a,b); lancioni et al. (2020) do so using gans. a few approaches @xcite @xcite augmented kg with information from retrieved documents. multiple approaches @xcite @xcite @xcite took a joint-training or multitasking approach to do both present keyphrase extraction and absent keyphrase generation. @xcite and @xcite changed the decoder to better respect the structure of keyphrases. @xcite changed the encoder to better respect the input document structure. @xcite proposed a new beam-search-based adaptive decoding method. @xcite ; @xcite investigated pre-training objectives for kg. both, however, rely on labelled pre-training data. ## conclusion our proposal, kpdrop, randomly drops present keyphrases from a document to turn them artificially absent. this encourages the model to learn to better exploit the context in the input to be able to infer keyphrases that are absent from the text but otherwise topically relevant. the results show that kpdrop serves as a simple model-agnostic method to substantially improve absent (and sometimes, present) keyphrase performance in both supervised and semi-supervised (low resource) settings when large annotated datasets for keyphrase generation are not available. in future, we would like to explore integration of kpdrop with largescale pre-training. in addition, kpdrop-r can potentially harm the performance of present keyphrases in some contexts (especially when using greedy search in a supervised setting). to address this, we can simply use the absent predictions of the model trained with kpdrop and the present predictions of the model trained without kpdrop.