Select, Extract and Generate: Neural Keyphrase Generation with Layer-wise Coverage Attention (2021)

Abstract: Natural language processing techniques have demonstrated promising results in keyphrase generation. However, one of the major challenges in neural keyphrase generation is processing long documents using deep neural networks. Generally, documents are truncated before given as inputs to neural networks. Consequently, the models may miss essential points conveyed in the target document. To overcome this limitation, we propose SEG-Net , a neural keyphrase generation model that is composed of two major components, (1) a selector that selects the salient sentences in a document and (2) an extractor-generator that jointly extracts and generates keyphrases from the selected sentences. SEG-Net uses Transformer, a self-attentive architecture, as the basic building block with a novel layer-wise coverage attention to summarize most of the points discussed in the document. The experimental results on seven keyphrase generation benchmarks from scientific and web documents demonstrate that SEG-Net outperforms the state-of-the-art neural generative methods by a large margin.

## problem definition keyphrase generation task is defined as given a text document x, generate a set of keyphrases @xmath0, k 2 , . . . , k |k| } where the document seg-net decomposes the keyphrase generation task into three sub-tasks. we define them below. task 1 (salient sentence selection). given a list of sentences s x , predict a binary label (0/1) for each sentence s i task 2 (present keyphrase extraction ). given s sal x as a concatenated sequence of words, predict a label (b/i/o) for each word that indicates if it is a constituent of a present keyphrase. task 3 (absent keyphrase generation). given s sal ## seg-net for keyphrase generation our proposed model, seg-net jointly learns to extract and generate present and absent keyphrases from the salient sentences in a target document. the key advantage of seg-net is the maximal utilization of the information from the input text in order to generate a set of keyphrases that summarize all the key points in the target document. seg-net consists of a sentence-selector and an extractor-generator. the sentence-selector identifies the salient sentences from the target document (task 1) that are fed to the extractor-generator to predict both the present and absent keyphrases (task 2, 3). we detail them in this section. ## results we compare our proposed model seg-net with the baselines on the scientific and web domain datasets. we present the experiment results in present keyphrase prediction from the results, it is evident that seg-net outperforms all the baseline methods by a significant margin (@xmath1.05, t-test) in 3 out of 5 scientific datasets and both web domain datasets. unlike the baseline methods, seg-net extracts the present keyphrases from the salient sentences, contributing most to the performance improvement. in the krapivin dataset, the performance is on par, while in the inspec dataset, seg-net performs worst in terms of f1@5. the perofrmance drop is explainable as inspect dataset consists of shorter documents (see the average lengths in absent keyphrase prediction unlike present phrases, absent phrases do not appear exactly in the target document. hence, predicting them is more challenging and requires a comprehensive understanding of the underlying document semantic. from ## analysis the differences between the transformer baseline and seg-net are (1) decoupling keyphrase extraction and generation, (2) use salient sentences for keyphrase prediction, and (3) layer-wise coverage attention. we perform ablation on the three design choices and present the results in decoupling extraction and generation seg-net extracts present keyphrases and generates absent keyphrases as suggested in @xcite with a difference in the extractor. seg-net employs a 3-way classifier (to predict bio tags) that enables consecutive present keyphrases extraction. the ablation study shows that separating extraction and generation boosts present keyphrase prediction (as much as 2.8, 1.0, and 2.2 f1@m points in nus, semeval, and kptimes datasets, respectively). salient sentence selection one of seg-net's key contributions is the sentence-selector that identifies the salient sentences to minimize the risk of missing critical points due to truncating long target documents (e.g., web documents). the contribution of sentence-selector in present keyphrase prediction is evident from the ablation study. the impact of using salient sentences to generate absent keyphrases is significant for the web domain datasets (e.g., 2.8 f1@m points in kptimes). we show the performances on kptimes test documents with different length in figure 3 and the results suggest that seg-net improves absent keyphrase prediction significantly for longer documents, and we credit this to the sentence selector. the selector's accuracy on the kp20k and kptimes datasets are 78.2 and 73.7 in terms of (macro) f1 score. we evaluate seg-net by providing the groundtruth salient sentences to quantify the improvement achievable with a perfect sentence-selector. we found that the present keyphrase prediction performance would have increased by 3.2 and 4.1 f1@m points with a perfect sentence-selector. we compare the sentence selector with the baselines that select the first n sentences from the target document, and the results are presented in ## related work keyphrase extraction approaches identify important phrases that appear in a document. the existing approaches generally work in two steps. first, they select a set of candidate keyphrases based on heuristic rules @xcite @xcite . the selected keyphrases are scored as per their importance in the second step, which is computed by unsupervised ranking approaches @xcite or supervised learning algorithms @xcite @xcite @xcite . finally, the top-ranked candidates are returned as the keyphrases. another pool of extractive solutions follows a sequence tagging ap-proach @xcite @xcite . however, the extractive solutions are only able to predict the keyphrases that appear in the document and thus fail to predict the absent keyphrases. keyphrase generation methods aim at predicting both the present and absent phrases. @xcite proposed the first generative model, known as copyrnn, which is composed of attention @xcite and copy mechanism @xcite . multiple extensions of copyrnn were proposed in subsequent works @xcite . different from these approaches, @xcite proposed copycnn that utilizes convolutional neural network (cnn) @xcite to form sequenceto-sequence architecture. however, these generation methods are trained to predict one keyphrase from the target document. in contrast, @xcite proposed to concatenate all the ground-truth keyphrases and train models to generate them as one output sequence. other noteworthy approaches in literature utilize data from external source @xcite , syntactic supervision @xcite , semisupervised learning @xcite , reinforcement learning @xcite , adversarial training @xcite , unlikelihood training @xcite to improve keyphrase generation. ## conclusion this paper presents seg-net, a keyphrase generation model that identifies the salient sentences in a target document to utilize maximal information for keyphrase prediction. in seg-net, we incorporate a novel layer-wise coverage attention to cover all the critical points in a document and diversify the present and absent keyphrases. we evaluate seg-net on seven benchmarks from scientific and web documents, and the experiment results demonstrate seg-net's effectiveness over the state-of-the-art methods on both domains.