MDER ank: A Masked Document Embedding Rank Approach for Unsupervised Keyphrase Extraction (2022)

Abstract: Keyphrase extraction (KPE) automatically extracts phrases in a document that provide a concise summary of the core content, which benefits downstream information retrieval and NLP tasks. Previous state-of-the-art methods select candidate keyphrases based on the similarity between learned representations of the candidates and the document. They suffer performance degradation on long documents due to discrepancy between sequence lengths which causes mismatch between representations of keyphrase candidates and the document. In this work, we propose a novel unsupervised embedding-based KPE approach, Masked Document Embedding Rank (MDERank), to address this problem by leveraging a mask strategy and ranking candidates by the similarity between embeddings of the source document and the masked document. We further develop a KPE-oriented BERT (KPEBERT) model by proposing a novel self-supervised contrastive learning method, which is more compatible to MDERank than vanilla BERT. Comprehensive evaluations on six KPE benchmarks demonstrate that the proposed MDERank outperforms state-of-the-art unsupervised KPE approach by average 1.80 F1@15 improvement. MDERank further benefits from KPEBERT and overall achieves average 3.53 F1@15 improvement over SIFRank.

## introduction keyphrase extraction (kpe) automatically extracts a set of phrases in a document that provide a concise summary of the core content. kpe is highly beneficial for readers to quickly grasp the key information of a document and for numerous downstream tasks such as information retrieval and summarization. previous kpe works include supervised and unsupervised approaches. supervised approaches model kpe as sequence tagging @xcite @xcite @xcite or sequence generation tasks @xcite and require large-scale annotated data to perform well. since kpe annotations are expensive and largescale kpe annotated data is scarce, unsupervised kpe approaches, such as textrank @xcite , yake @xcite , em-bedrank @xcite , are the mainstay in industry deployment. among unsupervised kpe approaches, embedding-based approaches including em-bedrank @xcite and sifrank @xcite yield the state-of-theart (sota) performance. after selecting keyphrase (kp) candidates from a document using rule-based methods, embedding-based kpe approaches rank the candidates in a descending order based on a scoring function, which computes the similarity between embeddings of candidates and the source document. then the top-k candidates are chosen as the final kps. we refer to these approaches as phrase-document-based (pd) methods. pd methods have two major drawbacks: in this paper, we propose a novel unsupervised embedding-based kpe method, denoted by masked document embedding rank (mderank), to address above-mentioned drawbacks of pd methods. the architecture of mderank is shown in figure 1 . the basic idea of mderank is that a keyphrase plays an important role in the semantics of a document, and its absence from the document should cause a significant change in the semantics of the document. therefore, we propose to compare the embeddings of the original document and its variant where the occurrence(s) of some candidate kps are masked. this leads to a new ranking principle based on the increasing order of the resulting similarities, i.e., a lower semantic similarity between the original document and its masked variant indicates a higher significance of the candidate. our proposed method can be deemed as document-document method and it addresses the two weaknesses of the phrase-document methods: (i) since the sequence lengths of the original document and the masked document are the same, comparing their similarities in the semantic space is more meaningful and reliable. (ii) the embedding of the masked document is computed from sufficient amount of context information and hence can capture the semantics reliably using the sota contextualized representation models such as bert. inspired by @xcite @xcite , where pre-trained language models (plms) trained on objectives close to final downstream tasks achieve enhanced representations and improve fine-tune performance, we further propose a novel self-supervised contrastive learning method on top of bert-based models (dubbed as kpebert). the main contributions of this work include: as analyzed in section 1, for embedding-based methods, using contextualized embedding models to compute candidate embeddings could be unreliable due to lack of context, and these methods lack robustness to different lengths of keyphrases and documents. our proposed mderank approach could effectively address these drawbacks. contextual embedding models early emebdding models include static word embeddings such as word2vec @xcite , glove (pen-1 https://maartengr.github.io/keybert/ nington et al., 2014), and fasttext @xcite , phrase embedding model hcpe @xcite , and sentence embeddings such as sent2vec @xcite and doc2vec (lau and baldwin, 2016), which render word or sentence representations that do not depend on their context. in contrast, pre-trained contextual embedding models, such as elmo @xcite , incorporate rich syntactic and semantic information from context for representation learning and yield more expressive representations. bert @xcite captures better context information through a bidirectional transformer encoder than the bi-lstm based elmo, and has established sota in a wide variety of nlp tasks. in one line of research, roberta @xcite , xlnet @xcite and many other bert variant plms have been proposed to further improve the language representation capability. in another line of research, longformer @xcite , bigbird @xcite and other efficient transformers are proposed to reduce the quadratic complexity of transformer on sequence length in order to model long-range dependencies. in this paper, we mainly use bert as the default contextual embedding model. we also evaluate the performance of mderank with these efficient transformers on long documents. ## mderank in this section, we describe the proposed masked document embedding rank (mderank) approach. given a document @xmath0, w 2 , . . . , w n }, @xmath1, and a set of selected candidate kps @xmath2, . . . , c i , . . . , c m }, where a candidate c i consists of one or multiple tokens, as c @xmath3, . . . , c l i }, and @xmath4, kpe aims to select k candidates from c, where @xmath5. following the common practice @xcite , after tokenization and pos tagging, candidates are selected with continuous regular @xmath6. * |@xmath7. * >, which are mostly noun phrases. to address the mismatch between sequence lengths of a document and a candidate phrase as well as lack of contextual information in pd methods as mentioned in section 1, we hypothesize that it is more reasonable to conduct the similarity comparison at the document-document level rather than at the phrase-document level. based on this hypothesis, for each candidate kp c i for a document d, given its occurrence positions in d as [p 1 , p 2 , . . . , p t ], mderank replaces all occurrences of p t @xmath8. it is noted the number of mask used for masking p t is as same as the number of tokens in c i . and then we construct a masked variant of the original document as d c i m . we define the similarity score f (c i ) for ranking the significance of candidates as the cosine similarity between e(d) and e(d c i m ), where e(•) represents the embedding of a document. note that a higher f (c i ) value indicates a lower ranking for c i , which is opposite to the pd methods. this is because the higher similarity the less important the candidate c i is. the semantic of masked document is not changed much compared with original one as only a trivial phrase is masked. we use bert @xcite as the default embedding model and investigate other contextual embedding models in section 5.4. bert is pretrained through self-supervised tasks of masked language modeling (mlm) and next sentence prediction (nsp), on large-scale unlabeled text of english wikipedia (2500m words) and bookscorpus (800 words). a document @xmath9, w 2 , . . . , w n } is prepended with a special token @xcite and then encoded by bert to obtain the hidden representations of tokens as {h 1 , h 2 , . . . , h n }. the document embedding e(d) is computed as follows: we also investigate average pooling in section 5.4 and other masking methods in appendix a. ## kpebert: kpe-oriented self-supervised learning bert and many other bert variant plms can effectively capture syntactic and semantic information in language representations for downstream nlp tasks, through self-supervised learning objectives such as mlm. however, these self-supervised learning objectives neither explicitly model the significance of kps nor model ranking between kps. in this paper, we propose a novel plm kpebert trained with a novel self-supervised learning objective to improve the capabilities of plms for ranking kps. this new task is defined as minimizing the triplet loss between positive and negative examples (see figure 2 ). after obtaining a set of pseudo-kps for documents using another unsupervised kpe method θ, we define documents masking out pseudo-kps as positive examples while those masking out "non-pseudo-kps" as negative examples. following simcse @xcite , we encode the original document d (anchor), the positive example d + , and negative example d -through a plm encoder, respectively, and obtain their hidden representations as h d , h d + , and h d -. finally, we define the triplet loss as: absolute sampling for a document d, we first select candidate keyphrases c using pos tags with regular expressions as described in section 3. then we obtain a set of keyphrases c ′ extracted by another unsupervised kpe approach θ on d, as pseudo labels. we define "keyphrases" as c ′ and "nonkeyphrases" as c \ c ′ . we mask a "keyphrase" from a document with a mask to construct a positive example d + for d. we select a "non-keyphrase" and perform the same mask operation to construct a negative example d -. ## conclusion we propose a novel embedding-based unsupervised kpe approach, mderank, to improve reliability of similarity match compared to previous embeddingbased methods. we also propose a novel selfsupervised learning method and develop a kpeoriented plm, kpebert. experiments demonstrate mderank outperforms sota on diverse datasets and further benefits from kpebert. analyses further verify the robustness of mderank to different lengths of keyphrases and documents, and that mderank benefits from longer context and stronger embedding models. future work includes improving kpebert for mderank by optimizing sampling strategies and pre-training methods.