S im CKP : Simple Contrastive Learning of Keyphrase Representations (2023)

Abstract: Keyphrase generation (KG) aims to generate a set of summarizing words or phrases given a source document, while keyphrase extraction (KE) aims to identify them from the text. Because the search space is much smaller in KE, it is often combined with KG to predict keyphrases that may or may not exist in the corresponding document. However, current unified approaches adopt sequence labeling and maximization-based generation that primarily operate at a token level, falling short in observing and scoring keyphrases as a whole. In this work, we propose SimCKP, a simple contrastive learning framework that consists of two stages: 1) An extractor-generator that extracts keyphrases by learning context-aware phrase-level representations in a contrastive manner while also generating keyphrases that do not appear in the document; 2) A reranker that adapts scores for each generated phrase by likewise aligning their representations with the corresponding document. Experimental results on multiple benchmark datasets demonstrate the effectiveness of our proposed approach, which outperforms the state-of-the-art models by a significant margin.

## introduction keyphrase prediction (kp) is a task of identifying a set of relevant words or phrases that capture the main ideas or topics discussed in a given document. prior studies have defined keyphrases that appear in the document as present keyphrases and the opposites as absent keyphrases. high-quality keyphrases are beneficial for various applications such as information retrieval @xcite ), text summarization @xcite , and translation @xcite . kp methods are generally divided into keyphrase extraction (ke) (witten figure 1 : an example of keyphrase prediction. present and absent keyphrases are colored blue and red, respectively. overlapping keyphrases are in bold. @xcite @xcite @xcite @xcite and keyphrase generation (kg) models @xcite @xcite @xcite @xcite , where the former only extracts present keyphrases from the text and the latter generates both present and absent keyphrases. recently, several methods integrating ke and kg have been proposed @xcite @xcite @xcite . these models predict present keyphrases using an extractor and absent keyphrases using a generator, thereby effectively exploiting a relatively small search space in extraction. however, current integrated models suffer from two limitations. first, they employ sequence labeling models that predict the probability of each token being a constituent of a present keyphrase, where such token-level predictions may be a problem when the target keyphrase is fairly long or overlapping. as shown in figure 1 , the sequence labeling model makes an incomplete prediction for the term "multi-ply connected problem" because only the tokens for "multiply connected" have yielded a high probability. we also observe that the model is prone to miss the keyphrase "integral equations" every time because it overlaps with another keyphrase "boundary integral equation" in the text. secondly, integrated or even purely generative models are usually based on maximum likelihood estimation (mle), which predicts the probability of each token given the past seen tokens. this approach scores the most probable text sequence the highest, but as pointed out by @xcite , keyphrases from the maximumprobability sequence are not necessarily aligned with target keyphrases. in figure 1 , the mle-based model predicts "magnetostatic energy analysis", which is semantically similar to but not aligned with the target keyphrase "nonlinear magnetostatic analysis". this may be a consequence of greedy search, which can be remedied by finding the target keyphrases across many beams during beam search, but it would also create a large number of noisy keyphrases being generated in the top-k predictions. existing ke approaches based on representation learning may address the above limitations @xcite @xcite @xcite @xcite . these methods first mine candidates that are likely to be keyphrases in the document and then rank them based on the relevance between the document and keyphrase embeddings, which have shown promising results. nevertheless, these techniques only tackle present keyphrases from the text, which may mitigate the overlapping keyphrase problem from sequence labeling, but they are not suitable for handling mle and the generated keyphrases. in this work, we propose a two-stage contrastive learning framework that leverages context-aware phrase-level representations on both extraction and generation. first, we train an encoder-decoder network that extracts present keyphrases on top of the encoder and generates absent keyphrases through the decoder. the model learns to extract present keyphrases by maximizing the agreement between the document and present keyphrase representations. specifically, we consider the document and its corresponding present keyphrases as positive pairs and the rest of the candidate phrases as negative pairs. note that these negative candidate phrases are mined from the document using a heuristic algorithm (see section 4.1). the model pulls keyphrase embeddings to the document embedding and pushes away the rest of the candidates in a contrastive manner. then during inference, top-k keyphrases that are semantically close to the document are predicted. after the model has finished training, it generates candidates for absent keyphrases. these candidates are simply constructed by overgenerating with a large beam size for beam search decoding. to reduce the noise introduced by beam search, we train a reranker that allocates new scores for the generated phrases via another round of contrastive learning, where this time the agreement between the document and absent keyphrase representations is maximized. overall, major contributions of our work can be summarized as follows: ## problem definition given a document x, the task of keyphrase prediction is to identify a set of keyphrases @xmath0 ## simckp in this section, we elaborate on our approach to building a contrastive framework for keyphrase prediction. in section 4.1, we delineate our heuristic algorithm for constructing a set of candidates for present keyphrase extraction; in section 4.2, we describe the multi-task learning process for extracting and generating keyphrases; and lastly, we explain our method for reranking the generated keyphrases in section 4.3. figure 2 illustrates the overall architecture of our framework. ## conclusion this paper presents a contrastive framework that aims to improve the keyphrase prediction performance by learning phrase-level representations, rectifying the shortcomings of existing unified models that score and predict keyphrases at a token level. to effectively identify keyphrases, we divide our framework into two stages: a joint model for extracting and generating keyphrases and a reranking model that scores the generated outputs based on the semantic relation with the corresponding document. we empirically show that our method significantly improves the performance of both present and absent keyphrase prediction against existing state-of-the-art models.