O ne2 S et + Large Language Model: Best Partners for Keyphrase Generation (2024)

Abstract: Keyphrase generation (KPG) aims to automatically generate a collection of phrases representing the core concepts of a given document. The dominant paradigms in KPG include one2seq and one2set. Recently, there has been increasing interest in applying large language models (LLMs) to KPG. Our preliminary experiments reveal that it is challenging for a single model to excel in both recall and precision. Further analysis shows that: 1) the one2set paradigm owns the advantage of high recall, but suffers from improper assignments of supervision signals during training; 2) LLMs are powerful in keyphrase selection, but existing selection methods often make redundant selections. Given these observations, we introduce a generate-then-select framework decomposing KPG into two steps, where we adopt a one2set-based model as generator to produce candidates and then use an LLM as selector to select keyphrases from these candidates. Particularly, we make two important improvements on our generator and selector: 1) we design an Optimal Transport-based assignment strategy to address the above improper assignments; 2) we model the keyphrase selection as a sequence labeling task to alleviate redundant selections. Experimental results on multiple benchmark datasets show that our framework significantly surpasses state-of-the-art models, especially in absent keyphrase prediction.

## introduction the keyphrase generation (kpg) task involves creating a set of phrases to encapsulate the core con-*equal contribution. in this paper, to deal with the above issue, we introduce a generate-then-select framework that de-composes the kpg task into two steps, each handled by a separate sub-model. this framework includes a generator that aims to recall correct keyphrases and a selector that eliminates incorrect candidates. to identify the most suitable models for the generator and selector, we conduct further experiments in the preliminary study to investigate the potential of conventional kpg models and llms for these roles. our experimental results lead to two conclusions: 1) settrans, a one2setbased kpg model, has a significant advantage in recall and thus is well-suited as the generator, and 2) llms with their superior semantic understanding, are more effective than small language models (slms) for keyphrase selection and are suitable as the selector. furthermore, we improve the generator and selector of our framework in two aspects. the generator assigns each ground-truth to only one control code. however, the number of control codes generally exceeds that of ground-truths, leading to insufficient training for many control codes. to address the above improper assignments, we propose an ot-based assignment strategy for one2set. this strategy converts the matching of candidates and ground truth into an ot problem, allowing a ground-truth to be assigned to multiple candidates. as for the selector, existing studies @xcite @xcite employ reranking methods to individually score candidates and then select those with high scores, which, however, results in many semantically similar candidates being selected. to address this issue, we convert keyphrase reranking into an llmbased sequence labeling task. leveraging the long sequence modeling capability of llms, we feed all candidates into the selector and have it autoregressively generate decision labels indicating whether to keep or discard the corresponding candidate. in this way, we can not only reduce the decoding search space of llms, but also alleviate semantic repetition by enabling the selector to fully consider the correlation between the current candidate and previous selections. particularly, to ensure robustness to the order of candidates, we feed them into the selector in random order during instruction tuning. this encourages the selector to develop a deeper understanding of the candidates' semantics. during inference, candidates are sorted by quality for the selector to prioritize candidates more likely to be correct. overall, the major contributions of our work can be summarized as follows: ## preliminary study to verify the necessity of decomposing kpg, we first explore the performance of dominant models in terms of recall and precision. then, through more experiments, we analyze which models are best suited as the generator and selector. trade-off in keyphrase generation. we measure the performance of dominant models on the testset of kp20k, including settrans, finetuned bart-large, scibart-large, flan-t5-large, and llama-2-7b and report the results in figure 1. as shown in figure 1 , achieving high accuracy and recall simultaneously is challenging for a single model. whether it is a conventional kpg model or an llm, as the number of predicted keyphrases increases, the recall of the model inevitably increases while its accuracy decreases, and vice versa. this result proves the necessity of decomposing kpg into two steps. evaluating the recall performance of llama-2-7b and settrans. as revealed above, settrans and llama-2-7b exhibit excellent recall in commonly-used setting. in figure 2 , we further investigate their recall under various settings for a full comparison. we can clearly observe that as the number of generated candidates increases, settrans consistently exhibits better recall performance than llama-2-7b. given its stronger recall performance and lower computational consumption, we choose settrans as the generator. additionally, settrans tends to recall more correct keyphrases along with more incorrect candidates (see appendix b.2), highlighting the necessity of a selector with strong filtering capabilities to improve accuracy. evaluating slm and llm for keyphrase selection. to identify a suitable selector, we first use settrans as the generator to output candidates, and then compare multiple representative keyphrase reranking methods. the methods we consider include 1) slm-scorer, the reranker from @xcite , which is an slm-based one and achieves sota performance in keyphrase reranking, and 2) llm-scorer, a llama-2-7b reranker, which is fined-tuned as detailed in appendix d.2. as shown in ## our framework as described above, our framework involves an improved one2set-based generator and an llmbased selector. unlike the conventional one2set paradigm, our generator improves the supervision signal assignment during training by modeling it as an optimal transport (ot) problem. distinct from previous studies on keyphrase reranking @xcite and llmbased reranking @xcite , our selector autoregressively generates decision labels for keeping or discarding each candidate. this approach not only reduces the decoding search space but also fully considers the correlation between selections, thus effectively minimizing semantically repetitive selections. moreover, we design an r-tuning s-infer strategy to help the selector comprehend the semantics of candidates. ## related work the related works to ours mainly include keyphrase generation and keyphrase selection. ## conclusion and future work this paper introduces a generate-then-select framework that integrates a one2set model and an llm selector together, so as to fully leverage the high recall of the one2set paradigm and powerful semantic understanding of llm. the one2set model acts as the generator and is optimized by our ot-based assignment to recall more correct candidates. the llm acts as the selector that models the selection of keyphrase candidates as a sequence labeling task and reduces the semantic repetition through its long sequence modeling capability. experimental results show that our framework achieves significant performance improvements compared to existing state-of-theart models. in the future, we tend to combine the generation and selection tasks into a multi-task learning framework, which further improves the synergy between the two tasks.