Unsupervised Keyphrase Extraction by Jointly Modeling Local and Global Context (2021)

Abstract: Embedding based methods are widely used for unsupervised keyphrase extraction (UKE) tasks. Generally, these methods simply calculate similarities between phrase embeddings and document embedding, which is insufficient to capture different context for a more effective UKE model. In this paper, we propose a novel method for UKE, where local and global contexts are jointly modeled. From a global view, we calculate the similarity between a certain phrase and the whole document in the vector space as transitional embedding based models do. In terms of the local view, we first build a graph structure based on the document where phrases are regarded as vertices and the edges are similarities between vertices. Then, we proposed a new centrality computation method to capture local salient information based on the graph structure. Finally, we further combine the modeling of global and local context for ranking. We evaluate our models on three public benchmarks (Inspec, DUC 2001, SemEval 2010) and compare with existing state-of-the-art models. The results show that our model outperforms most models while generalizing better on input documents with different domains and length. Additional ablation study shows that both the local and global information is crucial for unsupervised keyphrase extraction tasks.

## introduction keyphrase extraction (ke) task aims to extract a set of words or phrases from a document that can represent the salient information of the document @xcite . ke models can be divided into supervised and unsupervised. supervised methods need large-scale annotated training data and always perform poorly when transferred to different domain or type datasets. compared with the supervised method, the unsupervised method is more universal and adaptive via extracting phrases based on information from input document itself. in this paper, we focus on the unsupervised keyphrase extraction (uke) model. uke has been widely studied @xcite @xcite @xcite in the keyphrase extraction field. recently, with the development of text representation, embedding-based models @xcite have achieved promising results and become the new state-of-the-art models. usually, these methods compute phrase embeddings and document embedding with static word2vec models (e.g. glove @xcite @xcite ) or dynamic pre-trained language models (e.g. bert @xcite ). then, they rank candidate phrases by computing the similarity between phrases and the whole document in the vector space. though, these methods performed better than traditional methods @xcite @xcite , the simple similarity between phrase and document is insufficient to capture different kinds of context and limits in performance. figure 1 shows an intuitive explanation for the importance of context modeling. the nodes are candidate phrase embeddings, the star is the document embedding. each black circle represents one local context. nodes in the same black circle mean that these candidate phrases are all related to one vital local information (e.g. one topic/aspect of the document). nodes in the red circle mean that these candidate phrases are similar with the document semantics. if only model the global context via computing similarity between candidate phrases and the document, the model will tend to select red nodes, which will ignore local salient information in three clusters. in order to get the keywords accurately, we should take the local context (black circles) and global context (red circle) into consideration. to obtain information from context adequately, in this paper, we proposed a novel method which jointly models the local and global context of the input document. specifically, we calculate the similarity between candidate phrases and the whole document for modeling global context. for local context modeling, we first build a graph structure, which represents each phrase as nodes and the edges are similarity between nodes. then, we proposed a new centrality computation method, which is based on the insight that the most important information typically occurs at the start or end of documents (document boundary) @xcite @xcite , to measure salience of local context based on the graph structure. finally, we further combine the measure of global similarity and local salience for ranking. to evaluate the effectiveness of our method, we compare our method with recent state-of-the-art models on three public benchmarks @xcite . the results show that our model can outperform most models while generalizing better on input documents with different domains and length. it is deservedly mentioned that our models have a huge improvement on long scientific documents. ## methodology the overall framework of our model is shown in fig. 2 . we follow the general process of unsuper-vised keyphrase extraction. the main steps are as follows: (1) we tokenize the document and tag the document with part-of-speech (pos) tags. (2) we extract candidate phrases based on part-of-speech tags. we only keep noun phrases (np) that consist of zero or more adjectives followed by one or multiple nouns @xcite . (3) we use a pre-trained language model to map the document text to low-dimension vector space and extract vector representation of candidate phrases and the whole document. (4) we score each candidate phrase with a rank algorithm which jointly models the global and local context. (5) we extract phrases with scores from the rank algorithm. the main contribution of the whole process is the rank algorithm we proposed in step (4), which can be divided into three components: 1) phrasedocument similarity for modeling global context; 2) boundary-aware centrality for modeling local context; 3) the combination of global and local information. we will introduce the details of these components in this section. ## conclusion and future work in this paper, we point out that embedding-based models ignore the local information and propose a novel model which jointly models global and local context. our model revisited degree centrality and modified it with boundary function for modeling local context. we combine global similarity with our proposed boundary-aware centrality to extract keyphrases. experiments on 3 public benchmarks demonstrate that our model can effectively capture global and local information and achieve remarkable results. in the future work, we will focus on how to introduce our boundary-aware mechanism into supervised end2end keyphrase extraction/generation models.