Keyphrase Generation Beyond the Boundaries of Title and Abstract (2022)

Abstract: Keyphrase generation aims at generating important phrases (keyphrases) that best describe a given document. In scholarly domains, current approaches have largely used only the title and abstract of the articles to generate keyphrases. In this paper, we comprehensively explore whether the integration of additional information from the full text of a given article or from semantically similar articles can be helpful for a neural keyphrase generation model or not. We discover that adding sentences from the full text, particularly in the form of the extractive summary of the article can significantly improve the generation of both types of keyphrases that are either present or absent from the text. Experimental results with three widely used models for keyphrase generation along with one of the latest transformer models suitable for longer documents, Longformer Encoder-Decoder (LED) validate the observation. We also present a new large-scale scholarly dataset FullTextKP for keyphrase generation. Unlike prior large-scale datasets, FullTextKP includes the full text of the articles along with the title and abstract. We release the source code at https://github.com/kgarg8/FullTextKP.

## introduction keyphrases of scientific papers provide important topical information about the papers in a highly concise form and are crucial for understanding the evolution of ideas in a scientific field @xcite . keyphrases of scientific papers can be also useful for mining and analyzing the literature @xcite and for multiple downstream tasks such as index construction @xcite , summarization @xcite , query formulation @xcite , recommendation @xcite , reviewer matching for paper submissions @xcite and for clustering papers for fast retrieval @xcite . identifying keyphrases from the scientific documents has been popularly studied under the paradigm of keyphrase extraction for over a decade @xcite @xcite @xcite @xcite @xcite @xcite . however, one of the major limitations is that these approaches are not able to capture the semantic information in the document, particularly the absent keyphrases (keyphrases that are not present in the document). in this paper, we therefore focus on a newer paradigm of keyphrase generation introduced by @xcite . instead of extracting keyphrases from the document, we model the problem as a neural sequence-to-sequence (seq2seq) approach where we learn to generate keyphrases in an autoregressive way. this approach can not only predict the exact keyphrases that are present in a document (present keyphrases) but also those that are semantically relevant but absent from the document (absent keyphrases). while many works @xcite @xcite @xcite adopt this approach, their focus is on architectural innovations to improve the generation of both present and absent keyphrases. in contrast, our focus is different from that of the prior works in keyphrase generation: we propose to extend the source of information of the input sequence, which is thus far limited to only title and abstract. to this end, we curate a large-scale dataset of papers published by acm foot_0 which contains not only the title and abstract of the documents but also the full text from the documents. we call this dataset fulltextkp. owing to the scarcity of such large-scale datasets with full text and the difficulty of parsing and understanding the entire text in this paper, we provide innovative ways of using certain parts of the documents that could be rich source of information, e.g., using citation sentences from the content of the document or an extractive summary of the document (as illustrated in our experiments are performed with three widely used models for keyphrase generation as well as, to our knowledge, for the first time in keyphrase generation, with a transformer model suited for long documents, longformer encoder-decoder (led) @xcite . overall, we make the following contributions: keyphrase generation. recent approaches to keyphrase generation have been dominated by neural seq2seq models because they provide a mechanism to also generate absent keyphrases. meng et al. (2017) originally proposed an rnn-based seq2seq model along with copynet (copyrnn) for keyphrase generation. chen et al. (2018) improved copyrnn by taking correlations between the predicted keyphrases into account (corrrnn). yuan et al. (2020) extended the copyrnn model to dynamically generate a variable number of diverse keyphrases. chen et al. (2020) proposed an exclusive hierarchical decoding framework to capture the hierarchical compositionality of keyphrases (exhird). swaminathan et al. (2020) explored generative adversarial networks for keyphrase generation. ye et al. (2021b) modeled the target keyphrases as a set instead of a predefined sequence to improve the biased matching of the generated and target keyphrases. these methods use only title and abstract as the input sequence. our work can aid all of these methods to further enhance the performance of keyphrase generation by augmenting the source texts with information from the body of the documents. external information. some of the previous methods try to incorporate external information to enhance the performance of keyphrase extraction. for example, maui (medelyan et al., 2009) uses semantic information from wikipedia, ceke (caragea et al., 2014) and citetextrank (gollapalli and caragea, 2014) use information from citation networks, and singlerank and expandrank (wan and xiao, 2008a) use information from local textual neighborhoods. however, these methods are limited to extraction-based methods whereas we focus on generating keyphrases to have a holistic or semantic understanding of the document when the natural language text is fed to the neural model. several works in keyphrase generation started to use external information to generate keyphrases. for example, chen et al. (2019a), santosh et al. (2021) proposed methods where they augmented a set of additional keyphrases from semantically similar documents and use hidden state representations built from these keyphrases, diao et al. (2020) used cross-document attention networks, ye et al. (2021a) used information from retrieved documentkeyphrase pairs that are similar to the source document, shen et al. ( 2021 ) used a phrase bank by pooling all keyphrases from all the articles. similar to these works, one of our methods (retrieval-augmentation) retrieves relevant sentences from semantically similar documents and directly appends them to the title and abstract. datasets. there have been numerous efforts in the direction of curating large-scale datasets for either keyphrase extraction or keyphrase generation. in the non-scholarly domain, the datasets such as kptimes @xcite , jptimes @xcite , kpcrowd @xcite , @xcite , were curated from news articles, openkp @xcite was mined from webpages from a search engine. in the scholarly domain, there has been ongoing research with the datasets such as krapivin @xcite , inspec @xcite , semeval @xcite , nus (nguyen and @xcite , kp20k @xcite and @xcite . the limitation of these datasets is that they are either very small (of the magnitude of a few hundreds or thousands) or they are bounded to title and abstract. for instance, the widely used kp20k dataset @xcite consists of only the title and abstract of the research articles and the average words per document is as low as 176. to leverage and understand the benefit of additional data, we mined a new dataset based on the articles primarily from the acm database which contains not just the titles and abstracts but also the full text of the scholarly documents. in our dataset, the typical length of full papers is 5000-10000 tokens. ## methods we use four different models, three of which are popular models in the keyphrase generation task, viz. catseq @xcite , one2set @xcite , exhird @xcite ; and the fourth model is based on one of the latest transformer models, i.e., longformer encoder-decoder (led) @xcite , suitable for long documents. the baseline in each model takes the title and abstract (t+a) as input and predicts a sequence of keyphrases. we explore different extensions (also illustrated in ## experimental setup 4.1 fulltextkp dataset to evaluate the performance of models leveraging different types of information, we construct a new dataset, which we call fulltextkp. our full-textkp dataset consists of research papers that are published by acm foot_2 and are available in the acm digital library in its international conference proceedings series (icps). we used only the articles which have at least the five fields, viz., title, abstract, keywords, full text and references. we lowercased the text and constructed numerous regexes to remove the escape sequences, html tags, urls, emails, etc. we also replaced all the numbers and roman numerals with @xmath2. further, we used punktsentencetokenizer foot_3 to segment the document into sentences and nltk's word_tokenizer 4 to tokenize the sentences into tokens. we constructed suitable regex to extract the citation sentences and we considered all other sentences as non-citation sentences. the articles without any citation sentences and duplicates were further removed from the collection. thus, we were able to collect 142,844 articles. we split the fulltextkp dataset into 80/10/10 for train, test and validation sets. statistical information about the construction of this dataset addresses the sparsity of large-scale datasets for keyphrase extraction/generation from scientific papers and is aimed at enabling deep learning modeling. currently, there exist only a couple of large-scale datasets for this task, e.g., kp20k @xcite , and oagk (cano and bojar, 2018). unlike these datasets, which contain only the title and abstract of each paper, our dataset provides access to the full text of each paper. ## results in this section, we first present the results for present and absent keyphrase generation performance in §5.1 for the different methods described in §3. we further explore our best performing method, i.e., t+a+summary into two subcategories: extractive and abstractive summarization methods in §5.2. finally, in §5.3 we explore the performance of the best performing model, i.e., longformer encoder-decoder with sequence lengths much higher than 800. we run all the experiments three times and report the average. ## conclusion in this paper, we explore numerous ways of incorporating additional data from the body of the scholarly documents for improving the performance of the keyphrase generation task. we conclude from the results that the extractive summary sentences, that are more central to the paper, provide the most topical information for boosting both present and absent keyphrase generation performance. the citation, non-citation, and random sentences also bring complementary information to improve the performance modestly. some other ways such as augmenting semantically similar sentences from other papers or augmenting abstractive summary sentences, that bring repetitive or redundant information to titles and abstracts, are not effective. we present a comprehensive analysis with four models including led, yet unexplored for the task. our work aims at breaking apart the barriers of using only titles and abstracts, and presents a large-scale dataset with full texts for keyphrase generation.