KPE val: Towards Fine-Grained Semantic-Based Keyphrase Evaluation (2024)

Abstract: Despite the significant advancements in keyphrase extraction and keyphrase generation methods, the predominant approach for evaluation mainly relies on exact matching with human references. This scheme fails to recognize systems that generate keyphrases semantically equivalent to the references or diverse keyphrases that carry practical utility. To better assess the capability of keyphrase systems, we propose KPEval, a comprehensive evaluation framework consisting of four critical aspects: reference agreement, faithfulness, diversity, and utility. For each aspect, we design semantic-based metrics to reflect the evaluation objectives. Meta-evaluation studies demonstrate that our evaluation strategy correlates better with human preferences compared to a range of previously proposed metrics. Using KPEval, we re-evaluate 23 keyphrase systems and discover that (1) established model comparison results have blind-spots especially when considering reference-free evaluation; (2) large language models are underestimated by prior evaluation works; and (3) there is no single best model that can excel in all the aspects.

## introduction building automated keyphrase prediction systems has been a long-lasting research interest of information retrieval (ir) and natural language processing (nlp) @xcite @xcite . while a large number of keyphrase prediction systems have been proposed, the majority of them are assessed using a simplistic method: comparing the stemmed predictions with human references for exact matches. an extensive review of 76 recent keyphrase extraction and generation papers published in major conferences reveals a predominant reliance on exact matching, with 75 of 76 papers employing it and 51 treating it as the sole evaluation criterion (appendix a). this over-reliance brings two major concerns. first, it has been established that the evaluation accuracy of exact matching is inadequate @xcite . although a number of heuristics are proposed to relax the matching criteria @xcite @xcite or enrich the label set @xcite , they still struggle to accurately capture phrase semantics and have not been validated by systematic meta-evaluation. second, solely relying on reference-based evaluation is an incomplete strategy that overlooks critical aspects such as diversity of the predicted keyphrases (bahuleyan and el asri, 2020) or their utility in practical applications of keyphrase systems such as indexing for ir applications @xcite . in this paper, we undertake a systematic approach to advance keyphrase evaluation. for reference-based evaluation, we propose a phraselevel semantic matching metric with a high quality embedding trained on large-scale keyphrase data. based on the human evaluation corpus annotated on kp20k @xcite and kptimes @xcite , the meta-evaluation on five keyphrase systems shows that our metric significantly outperforms existing metrics by more than 0.15 absolute points in kendall's tau. by contrast, many proposed improvements to exact matching surprisingly fail to improve its human agreement ( §5.3). further analyses reveal that the proposed metric exhibits enhanced stability under the label variations commonly present in the keyphrase annotations. next, we move beyond reference agreement and holistically consider the desiderata for evaluating keyphrase systems. three crucial aspects are introduced: (1) faithfulness, whether the predictions are grounded to the document ( §6.1); (2) diversity, whether the predictions represent distinct concepts ( §6.2); and (3) utility for downstream ir applications ( §6.3). to accurately assess each aspect, we propose semantically-oriented metric designs 1 1959 including embedding similarity, model-based consistency evaluation, and dense retrieval. together, these aspects and metrics form kpe-val, a fine-grained semantic-based keyphrase evaluation framework (figure 1 ). in §7, we employ kpeval to evaluate 23 keyphrase systems, producing intriguing insights: ## related work in this section, we review the relevant literature on evaluating keyphrase systems. reference-based evaluation the major metrics for evaluating keyphrase systems are precision, recall, and f1 based on exact-match between the stemmed predictions and references @xcite @xcite . this method indiscriminately penalizes unmatched predictions, including synonyms or parent/child concepts of the reference. later works attempt to improve the metric by relaxing the matching criterion. @xcite propose to use r-precision with approximate matching, tolerating a prediction to be a substring of a reference and vice versa. @xcite employ n-gram matching metrics such as bleu @xcite and rouge @xcite . @xcite expand the references with name variations. @xcite propose a fine-grained score that combines token-level matching, edit distance, and duplication penalty. @xcite and @xcite use the semantic-based bertscore @xcite with predictions and references concatenated into two strings. meanwhile, ranking-based metrics such as mean reciprocal rank, mean averaged precision, and normalized discounted cumulative gain are introduced to evaluate the ranking provided by keyphrase extraction models @xcite @xcite . these metrics also compute exact matching to the references during their evaluation. reference-free evaluation directly evaluating keyphrase predictions without references is less common. early studies conduct human evaluation @xcite . later work evaluates the predictions' utility in applications such as retrieval @xcite or summarization @xcite . bahuleyan and el asri (2020) conduct reference-free evaluation of the predictions' diversity. meta-evaluation meta-evaluation studies that compare keyphrase metrics with human evaluations have been limited in scope, with a focus on reference-based evaluation. @xcite compare five lexical matching metrics and concluded that r-precision has the highest spearman correlation with human judgments. @xcite annotated a meta-evaluation corpus with 400 documents in french, evaluating 3 keyphrase models on "appropriateness" and "silence", approximately corresponding to precision and false negative rate. discussion building upon existing literature, this work systematically rethinks the goals of keyphrase evaluation and advances the evaluation methodology. we introduce kpeval, a holistic evaluation framework encompassing four key aspects ( §4). kpeval incorporates semantic-based metrics validated via rigorous meta-evaluation ( §5.3 and §6.1). finally, we conduct a large-scale evaluation of 21 keyphrase systems and offer novel insights into existing model comparisons and llms ( §7). ## background this section formulates the keyphrase prediction and evaluation tasks and outlines the scope of study. ## kpeval: evaluation aspects we introduce kpeval, a fine-grained framework for keyphrase evaluation. kpeval posits to evaluate p's quality across four crucial aspects: kp-set input reference corpus reference agreement ✔ ✔ faithfulness ✔ diversity ✔ utility ✔ ✔ ✔ ## kpeval: reference-based evaluation with semantic matching to begin with, we focus on reference agreement, the most extensively investigated aspect. recognizing the limitations of previous approaches, we introduce a semantic matching formulation and conduct meta-evaluation to confirm its effectiveness. ## 1962 reference agreement and kpg model, and includes unsupervised, supervised, and few-shot prompting methods. then, three crowd-source annotators are asked to rate on likert-scale the semantic similarity between (1) each prediction keyphrase p i and the most semantically similar keyphrase in y, and (2) each reference keyphrase y i and the most semantically similar keyphrase in p. we report the details of the annotator recruitment process, the annotation instructions, and the interface in appendix d. a total of 1500 document-level annotations with 13401 phrase-level evaluations are collected. as presented in ## kpeval: reference-free evaluation for a range of text generation tasks, the optimal output is often highly aspect-specific @xcite @xcite . as such, reference-based evaluation is incomplete as it does not always align with the evaluation goals. to address this gap, kpeval introduces three novel evaluation aspects, along with corresponding reference-free metrics, aimed at aligning closer with real-world application requirements. ## fine-grained benchmarking of keyphrase systems finally, we benchmark 23 keyphrase systems with kpeval, with the full evaluation results on kp20k and kptimes presented in we use four superscripts to mark the results that are significantly better than catseq (a), catseqtg-2rf1 (b), exhird-h (c), and settrans (d) with @xmath3.01 via a paired t-test. we use rr@5 to represent utility and 1 -dup_token_ratio to represent diversity. ## conclusion we introduce kpeval, a fine-grained evaluation framework that conducts semantic-based evaluation on reference agreement, faithfulness, diversity, our framework marks the first step towards systematically evaluating keyphrase systems in the era of llms. we hope kpeval can motivate future works to adopt more accurate evaluation metrics and further advance the evaluation methodology. future studies might also explore the development of utility metrics tailored to the specific requirements of applications in niche domains. 9 1967