An Analysis of Datasets, Metrics and Models in Keyphrase Generation (2025)

Abstract: Keyphrase generation refers to the task of producing a set of words or phrases that summarises the content of a document. Continuous efforts have been dedicated to this task over the past few years, spreading across multiple lines of research, such as model architectures, data resources, and use-case scenarios. Yet, the current state of keyphrase generation remains unknown as there has been no attempt to review and analyse previous work. In this paper, we bridge this gap by presenting an analysis of over 50 research papers on keyphrase generation, offering a comprehensive overview of recent progress, limitations, and open challenges. Our findings highlight several critical issues in current evaluation practices, such as the concerning similarity among commonly-used benchmark datasets and inconsistencies in metric calculations leading to overestimated performances. Additionally, we address the limited availability of pre-trained models by releasing a strong PLM-based model for keyphrase generation as an effort to facilitate future research.

## introduction keyphrase generation involves generating a set of words or phrases that summarise the content of a source document. these so-called keyphrases concisely and explicitly encapsulate the core content of a document, which makes them valuable for a variety of nlp and information retrieval tasks. for instance, keyphrases were proven useful for improving document indexing @xcite @xcite @xcite , summarization @xcite @xcite and question-answering @xcite @xcite , analyzing topic evolution @xcite @xcite or assisting with reading comprehension @xcite . keyphrase generation expands on keyphrase extraction by enabling the production of keyphrases absent from the source text @xcite . this ability is critical when dealing with short documents that often lack appropriate keyphrase candidates. @xcite provided the seminal work on keyphrase generation, introducing a sequence-to-sequence learning approach. their model builds upon an rnn encoder-decoder architecture @xcite and incorporates a copying mechanism @xcite to identify important phrases within the source text. equally importantly, they introduced kp20k, a dataset that laid the groundwork for end-to-end training of neural models for keyphrase generation. over the past few years, continuous efforts have been devoted to improve the effectiveness of keyphrase generation models. these efforts have been spread across different lines of research, such as model architectures, data resources, and use-case scenarios, often pursued separately. this analysis paper presents an overview of the current state of keyphrase generation, discussing recent progress, remaining limitations and open challenges. more specifically, we compiled and analysed a collection of over 50 papers on keyphrase generation, identifying the type(s) of contribution these papers made ( §3.1), examining the most frequently used benchmark datasets ( §3.2) and evaluation metrics ( §3.3), providing descriptions of proposed models while highlighting important milestones ( §3.4), and investigating how proposed models perform against each other ( §3.5). our findings are that: 1) commonly used benchmark datasets are so similar that reporting results on more than one adds no value, 2) the performance of models is often overestimated due to discrepancies in evaluation protocols, and 3) while dedicated models have been superseded by fine-tuned pretrained language models (plms), the overall performance gain since early models remains limited. our work goes beyond surveying the existing literature and addresses some of the aforementioned concerns by training, documenting and releasing a strong plm-based model for keyphrase generation along with an evaluation framework to facilitate future research ( §4). finally, we discuss some of the open challenges in keyphrase generation and propose actionable directions to address them ( §5). ## scope of the study our analysis encompasses a total of 52 research papers selected based on the following criteria: they are accessible through the acl anthology, they contain the phrase "keyphrase generation" either in their titles or abstracts, and they have been published after the seminal work of @xcite . for a more comprehensive coverage, we also include papers from other nlp-related venues, comprising aaai (4 papers), sigir (1 paper), and cikm (1 paper). to keep the number of papers manageable, we arbitrarily disregard papers from pre-print servers (e.g. arxiv) or those published in non-acl venues. nonetheless, we are confident that our sample represents a comprehensive portion of the research on keyphrase generation, encompassing all papers published at major nlp venues in the last seven years. this includes, for instance, the ten most cited articles in the field. 1 for each paper in our sample, we manually collect the following information: all the data collected in the course of this study is available at https://github.com/boudinfl/ kg-datasets-metrics-models. ## analysis in this section, we analyze the selected papers across five key dimensions: types of contribution ( §3.1), benchmark datasets ( §3.2), evaluation metrics ( §3.3), model architectures ( §3.4), and the best reported performances ( §3.5). ## a strong baseline model our analysis highlights the progress achieved by current keyphrase generation models, while drawing attention to the lack of standardized evaluation procedures and the limited availability of pretrained models. to address these challenges, we provide a strong baseline model for keyphrase generation, along with an evaluation framework to facilitate future research. upon analysing the proposed model scores (see §3.5), we find that fine-tuning a plm for the task consistently yields the best performance. based on this observation, we adopt this approach for our baseline model, leveraging bart-large @xcite as the initial plm, in line with recent studies @xcite . the model is fine-tuned on the kp20k training set for 10 epochs in a one2many setting @xcite , that is, given a source text as input, the task is to generate keyphrases as a single sequence of delimiter-separated phrases. during finetuning, gold keyphrases are arranged in the presentabsent order which was found to give the best results @xcite . notably, we do not apply any pre-processing to either the source texts or the ground-truth keyphrases, thereby fixing the issues we identified in §3.3. at test time, we evaluate the model using greedy decoding to generate the most probable keyphrases, or beam search (@xmath0. to select the bestperforming model, we save a checkpoint at the end of each epoch and evaluate its performance on the validation set of kp20k, using f 1 @{m, 5, 10} scores against the ground truth keyphrases. overall, fine-tuning for 9 epochs produces the highest scores (see figure 8 ), leading us to select the corresponding checkpoint as our baseline model. code for training, inference and evaluation is available on github. additionally, all model weights are accessible on hugging face. implementation details are given in appendix a.5. here, we evaluate the performance of our baseline model on kp20k test set and compare it against previously proposed models. ## open challenges and discussion we wrap up this paper by highlighting two challenges in keyphrase generation and suggesting actionable strategies to address them. finally, we discuss what llms can do for the task.