A Survey on Recent Advances in Keyphrase Extraction from Pre-trained Language Models (2023)

Abstract: Keyphrase Extraction (KE) is a critical component in Natural Language Processing (NLP) systems for selecting a set of phrases from the document that could summarize the important information discussed in the document. Typically, a keyphrase extraction system can significantly accelerate the speed of information retrieval and help people get first-hand information from a long document quickly and accurately. Specifically, keyphrases are capable of providing semantic metadata characterizing documents and producing an overview of the content of a document. In this paper, we introduce keyphrase extraction, present a review of the recent studies based on pre-trained language models, offer interesting insights on the different approaches, highlight open issues, and give a comparative experimental study of popular supervised as well as unsupervised techniques on several datasets. To encourage more instantiations, we release the related files mentioned in this paper.

## introduction keyphrase extraction is a fundamental task in nlp for identifying and extracting a set of keyphrases from the document that could summarize the important information discussed in the source document @xcite . keyphrases have enabled accurate and fast searching for the document from a large text corpus and have exhibited their potential in improving many nlp tasks, such as text summarization @xcite . various information filtering and extracting techniques are becoming critical with the ever-increasing amount of text data. owing to its potential importance, keyphrase extraction has received more and more attention from nlp researchers. however, the keyphrase extraction task is far from being solved: state-of-the-art performance on keyphrase extraction is still lower than other core nlp tasks. our goal in this paper is to investigate the state-of-the-art models in keyphrase extraction, examine the primary sources of errors made by existing systems, and discuss the challenges ahead. the first keyphrase extraction task was organized by @xcite , which defines the keyphrase extraction task as "the automatic selection of important and topical phrases from the body of a document". since then, there have been numerous keyphrase extraction models @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . in the past two decades, keyphrase extraction methods have experienced the development from traditional approaches to deep learning methods @xcite . with the recent development of pre-trained language models (plms) @xcite , many nlp tasks have significantly changed, that is, how to adopt and leverage pre-trained language models in the specific task. therefore, many keyphrase extraction models @xcite adopt plms as the embedding layer. we present a comprehensive survey of recent advances in neural keyphrase extraction. we describe the neural keyphrase extraction systems based on pre-trained language models, which depend on different paradigms (e.g., one-stage @xcite and two-stage @xcite ), various tasks (e.g., classification and ranking @xcite ), different learning strategies (e.g., supervised @xcite and un-supervised @xcite ), and variants of pre-trained language models (e.g., bert @xcite and roberta @xcite ). furthermore, we re-implement and collect the results of the mentioned models on several benchmark keyphrase extraction datasets. we illustrate the results in overall, this paper first discusses previous surveys on keyphrase extraction in section 2.1 and give a briefly introduction about pre-trained language models in section 2.2. then we highlight standard, past, and recent benchmark keyphrase extraction datasets (from shared tasks and other research) in section 3 and evaluation metrics in section 4. we then describe neural keyphrase extraction systems in section 5. next, we give the analysis and discussion in section 6. finally, we summarize the conclusions and future directions of neural keyphrase extraction in section 7. the limitations of our work is presented in section 8. ## preliminary in this section, we claim the differences between the current survey and the existing surveys. next, we present the background of pre-trained language models and their importance in nlp. ## keyphrase extraction dataset since the first shared task on ke (turney, 1999), many shared tasks and benchmark datasets for ke have been created. specifically, openkp @xcite , inspec @xcite , nus @xcite , krapivin @xcite , semeval2010 @xcite , se-meval2017 @xcite , and kp20k @xcite were created from scientific articles in english. compared with other datasets, kp20k contains a large amount of annotation data, so it is often used as the dataset to train the neural-based ke models recently. meanwhile, in recent papers @xcite , inspec @xcite , nus @xcite , krapivin @xcite datasets are often used as the zero-shot test sets to verify the robustness of the ke models trained by the kp20k dataset. furthermore, ke tasks have also been organized on newswire articles in english, e.g., duc2001 @xcite . ## keyphrase extraction evaluation this section describes evaluation metrics for measuring recent state-of-the-art keyphrase extraction baselines on commonly-used datasets. designing a suitable evaluation metric for the keyphrase extraction task is by no means an easy study @xcite . to score the output of a keyphrase extraction model, the traditional approach, which is also adopted by the @xcite shared task on keyphrase extraction, is (1) to create a mapping between the keyphrases in the ground-truth keyphrases and those in the model output adopting exact and partial matching (papagiannopoulou and tsoumakas, 2019), and then (2) score the output using evaluation metrics such as precision (p), recall (r), and f1-score (f1). as mentioned earlier, such evaluation usually operates based on exact matches between the predicted and ground-truth keyphrases. however, such a strategy cannot account for partial matches or semantic similarity. for example, if the prediction is "keyphrase extraction model" and the ground truth is "keyphrase extraction system", despite both semantic similarity and partial matching, the score will be 0. these minor deviations are ubiquitous in keyphrase extraction, yet they are harshly penalized by the "exact match" evaluation metrics. ## neural keyphrase extraction models with pre-trained language models there are two popular pipelines in the keyphrase extraction task, including one-stage and two-stage frameworks, as illustrated in figure 1 . the former mainly refers to using the task reformulation to address the keyphrase extraction task, which often treats the keyphrase extraction task as a sequence labeling task. the latter represents a more general framework, which usually operates in two procedures: (1) extracting a set of words/phrases that serve as candidate phrases using some heuristics and (2) determining which candidate phrases are keyphrases using supervised or unsupervised methods @xcite . typically, supervised methods perform better on specific domain tasks. however, this kind of method takes a lot of labor to annotate the corpus, and the model after training may overfit and not work well on other ke datasets. on the contrary, unsupervised methods do not need to annotate the corpus and usually have better data generalization in different domains. still, the performance is often insufficient due to the lack of annotated data. overall, we defined the above two procedures as the candidate keyphrase extraction and keyphrase importance estimation. in this paper, we distinguish the existing methods into three categories depending on the recent state-of-the-art baselines (with pre-trained language models as the backbone), including two-stage unsupervised, two-stage supervised, and one-stage supervised models. ## discussion in this section, we report the results of the recent unsupervised and supervised keyphrase extraction baselines, which all adopt pre-trained language models as the backbone, as shown in our second finding is those embedding-based systems with plms outperform embedding-based approaches with static embeddings in most cases. however, not all embedding-based systems with plms are superior to embedding-based systems with static embeddings. the former generally outperforms the latter when adopting the same importance estimation strategy, but the estimation strategy can significantly affect the results of keyphrase extraction. to sum up, effectively using pre-trained embeddings to estimate the importance score of each candidate is a critical part of improving the performance of keyphrase extraction. furthermore, there is still interesting progress to be made by leveraging a self-supervised learning strategy to optimize embedding-based systems. mderank uses a simple yet effective contrastive learning strategy to optimize embedding-based systems, achieving better performance. our third finding is that the embedding-based methods have slight improvement on long document datasets (e.g., semeval2010), and all unsupervised methods have poor effects on long document datasets. this demonstrates that keyphrase extraction from long documents is still a challeng-ing problem. our final finding is that two-stage supervised keyphrase extraction methods are superior to onestage supervised keyphrase extraction methods, as illustrated in ## conclusion and future directions we summarize the recent neural keyphrase extraction models based on pre-trained language models. our survey of models for keyphrase extraction, covering both unsupervised and supervised models, has yielded several important insights. the analysis revealed that there are at least six major challenges ahead.