M eta KP : On-Demand Keyphrase Generation (2024)

Abstract: Traditional keyphrase prediction methods predict a single set of keyphrases per document, failing to cater to the diverse needs of users and downstream applications. To bridge the gap, we introduce on-demand keyphrase generation, a novel paradigm that requires keyphrases that conform to specific high-level goals or intents. For this task, we present MetaKP, a large-scale benchmark comprising four datasets, 7500 documents, and 3760 goals across news and biomedical domains with human-annotated keyphrases. Leveraging MetaKP, we design both supervised and unsupervised methods, including a multi-task fine-tuning approach and a self-consistency prompting method with large language models. The results highlight the challenges of supervised fine-tuning, whose performance is not robust to distribution shifts. By contrast, the proposed self-consistency prompting approach greatly improves the performance of large language models, enabling GPT-4o to achieve 0.548 SemF1, surpassing the performance of a fully fine-tuned BART-base model. Finally, we demonstrate the potential of our method to serve as a general NLP infrastructure, exemplified by its application in epidemic event detection from social media.

## introduction keyphrase prediction is an nlp task that has attracted long-lasting research interest @xcite @xcite . given documents from various domains such as academic writing, news, social media, or meetings, keyphrase extraction and keyphrase generation models output short phrases aiming at encapsulating the key entities and concepts mentioned by the document. beyond a number of information retrieval applications @xcite @xcite , keyphrase prediction methods are widely incorporated into the pipelines of other nlp tasks such as natural language generation @xcite ), text summarization @xcite , and text classification @xcite . despite their wide application in diverse scenarios, which may have diverse requirements on the types of keyphrases, existing keyphrase prediction methods generally follow a suboptimal assumption: for every document, the model shall predict a single application-agnostic set of keyphrases, which is then evaluated against a monolithic set of references @xcite . this one-size-fits-all approach fails to cater to both downstream applications' varied requirements of the keyphrase predictions' topic and level of specificity and different expectations from human users with diverse backgrounds. to properly handle such diverse feedback, current approach could only rely on the samplererank strategy @xcite , which is largely inefficient. besides, the singlereference setting also biases the intrinsic evaluation, of keyphrase prediction models, as high-frequency topics in keyphrase labels may significantly outweigh the long-tail keyphrases. ## metakp benchmark in this section, we formulate the on-demand keyphrase generation task and introduce the metakp evaluation benchmark. ## discussion in this section, we further discuss the real-world use cases of the on-demand keyphrase generation. then, we apply it to solve event detection as a proof-of-concept case study. ## conclusion we introduce on-demand keyphrase generation, targeting the need for dynamic, goal-oriented keyphrase prediction tailored to diverse applica3 we solicited the dataset and outputs from the authors. tions and user demands. a large-scale, multidomain, human-verified benchmark metakp was curated and introduced. we designed and evaluated both supervised and unsupervised methods on metakp, highlighting the strengths of selfconsistency prompting with large language models. this approach significantly outperformed traditional fine-tuning methods under domain shifts, showcasing its robustness and the broader applicability of our methodology. finally, we underscore the versatility of on-demand keyphrase generation in practical applications such as epidemic event extraction, promising a new direction for keyphrase generation as general nlp infrastructure.