Rethinking Model Selection and Decoding for Keyphrase Generation with Pre-trained Sequence-to-Sequence Models (2023)

Abstract: Keyphrase Generation (KPG) is a longstanding task in NLP with widespread applications. The advent of sequence-to-sequence (seq2seq) pre-trained language models (PLMs) has ushered in a transformative era for KPG, yielding promising performance improvements. However, many design decisions remain unexplored and are often made arbitrarily. This paper undertakes a systematic analysis of the influence of model selection and decoding strategies on PLM-based KPG. We begin by elucidating why seq2seq PLMs are apt for KPG, anchored by an attention-driven hypothesis. We then establish that conventional wisdom for selecting seq2seq PLMs lacks depth: (1) merely increasing model size or performing task-specific adaptation is not parameter-efficient; (2) although combining in-domain pre-training with task adaptation benefits KPG, it does partially hinder generalization. Regarding decoding, we demonstrate that while greedy search achieves strong F1 scores, it lags in recall compared with sampling-based methods. Based on these insights, we propose DeSel, a likelihood-based decode-select algorithm for seq2seq PLMs. DeSel improves greedy search by an average of 4.7% semantic F1 across five datasets. Our collective findings pave the way for deeper future investigations into PLM-based KPG.

## introduction keyphrases encapsulate the core information of a document. due to their high information density, they have been found valuable in areas such as information retrieval @xcite @xcite , document clustering @xcite , summarization @xcite , and text classification @xcite . a keyphrase is termed a present keyphrase if it is explicitly found within the document and an absent keyphrase otherwise. the task of identifying present keyphrases is defined as keyphrase extraction (kpe), whereas keyphrase generation (kpg) involves predicting both types of keyphrases. recently, pre-trained language models (plms) have been widely incorporated in kpg @xcite via sequence-tosequence (seq2seq) generation, with promising performance on zero-shot @xcite , multilingual @xcite , and low-resource @xcite kpg. however, existing literature typically focuses on a specific subset of important components in this pipeline, such as data construction and loss design, while making arbitrary choices for the others @xcite @xcite . as a result, kpg systems are often compared under different assumptions and the effect of the arbitrary design choices remains unclear. to bridge this gap, this paper focuses on two crucial questions that have not been systematically explored: to begin with, we posit that seq2seq plms are inherently suitable to kpg ( §3). by drawing the correlations with a strong graph-based kpe algorithm, we show that these plms implicitly compute phrase centrality @xcite in their decoder attention patterns. this knowledge is also directly translatable to a strong ranking function for kpe. on the other hand, encoder-only models fail to carry such centrality information. next, we search for the best seq2seq plm for kpg fine-tuning ( §4). while common strategies for other nlp tasks might advocate for (1) scaling up the model size, (2) in-domain pre-training, or (3) task adaptation, do these approaches hold the same merit for kpg? our findings reveal that a singular emphasis on scaling or task adaptation does not ensure efficient performance improvement. in contrast, in-domain pre-training consistently bolsters performance across both keyphrase types and can benefit from task adaptation. a robustness analysis reveals that a proper model choice and data-oriented training approaches are complementary: without the latter, stronger plms are more vulnerable to perturbed input, with over 14% recall drop under named variation substitutions and over 5% recall drop under input paraphrasing. decoding strategy is also an essential component in plm-based kpg, but much under-explored by current literature. in §5, we thoroughly compare six decoding strategies including greedy search, beam search, and sampling-based methods. results suggest that when only generating a single sequence consisting of concatenated keyphrases, greedy search achieves a strong f1 score. however, aggregating the predictions from multiple sampled sequences outperforms greedy search due to a much higher recall. based on these findings, we introduce desel, a likelihood-based selection strategy that selects from sampled phrases to augment the greedy search predictions. desel utilizes the probability of phrases from greedy search's predictions as the baseline to filter out noisy predictions from a set of sampled keyphrase candidates. experiments on five kpg datasets show that desel consistently improves greedy decoding by 7.9% f 1@m for present keyphrases, 25% f 1@m for absent keyphrases, and 4.7% semantic f1 for all keyphrases, achieving state-of-the-art kpg performance, underscoring the importance of carefully examining the design choice of kpg. to summarize, our primary contributions are: ## do plms inherently carry significant knowledge of keyphrases? existing studies have justified their use of seq2seq plms by drawing the close relationship between the pre-training tasks of bart (denoising language modeling) or t5 (unified text-to-text transfer) and the formulation of kpg @xcite @xcite or kpe @xcite . however, there is a lack of an in-depth understanding of why seq2seq plms should be chosen for keyphrase-related tasks. in this section, we reason based on phrase centrality @xcite and show that plms with autoregressive decoders, including seq2seq plms, carry attention heads that approximately function as centrality assigners and naturally as potent keyphrase rankers. ## influence of plm choice for kpg three crucial design options exist for using seq2seq plms for kpg: the choice of plm to fine-tune, the fine-tuning data and objective, and the decoding strategy. previous work focuses on fine-tuning objective and data construction @xcite @xcite while often making the other two choices in an ad hoc way, making it difficult to compare among approaches. this section dives into the first question by evaluating three "conventional wisdoms": ## decoding strategy for kpg while it is well-known that decoding strategies can strongly affect text generation quality @xcite , surprisingly there has been little study about decoding plm-based kpg models. previous studies often directly use greedy search and variants of beam search @xcite @xcite , limiting the understanding of plms fine-tuned for kpg. to bridge this knowledge gap, we first carefully evaluate six decoding strategies on the strongest plm-based kpg model. we then propose a simple yet effective decode-select strategy to mitigate the observed deficiencies of greedy search. ## conclusion this paper systematically investigated model selection and decoding for building kpg models with seq2seq plms. our analyses suggested much more nuanced patterns beyond the "conventional wisdom" assumed by the majority of current literature. our novel decoding strategy, desel, significantly improved the performance of greedy search across multiple datasets. more broadly, this study underscores the distinct nature of the kpg task. one should not blindly transpose conclusions or assumptions from other text generation tasks. instead, they warrant careful re-evaluation and empirical validation. our work also opens up exciting directions for future work, with deep groundings in keyphrase literature. for instance, making kpg models more robust, interpreting a kpg model, and designing better decoding algorithms for kpg.