Hyperbolic Relevance Matching for Neural Keyphrase Extraction (2022)

Abstract: Keyphrase extraction is a fundamental task in natural language processing that aims to extract a set of phrases with important information from a source document. Identifying important keyphrases is the central component of keyphrase extraction, and its main challenge is learning to represent information comprehensively and discriminate importance accurately. In this paper, to address the above issues, we design a new hyperbolic matching model (HyperMatch) to explore keyphrase extraction in hyperbolic space. Concretely, to represent information comprehensively, HyperMatch first takes advantage of the hidden representations in the middle layers of RoBERTa and integrates them as the word embeddings via an adaptive mixing layer to capture the hierarchical syntactic and semantic structures. Then, considering the latent structure information hidden in natural languages, HyperMatch embeds candidate phrases and documents in the same hyperbolic space via a hyperbolic phrase encoder and a hyperbolic document encoder. To discriminate importance accurately, HyperMatch estimates the importance of each candidate phrase by explicitly modeling the phrase-document relevance via the Poincaré distance and optimizes the whole model by minimizing the hyperbolic margin-based triplet loss. Extensive experiments are conducted on six benchmark datasets and demonstrate that HyperMatch outperforms the recent state-of-the-art baselines.

## introduction keyphrase extraction (ke) aims to extract a set of phrases related to the main points discussed in the source document, a fundamental task in natural language processing (nlp). because of their succinct and accurate expression, keyphrase extraction is helpful for a variety of applications such as information retrieval @xcite ) and text summarization @xcite . a large region large region of region of land a large region of land . . . . . . 1-gram 2-gram 3-gram 5-gram … … the great plateau is a large region of land that is secluded from … figure 1 : sample partial of the document in openkp dataset. for ease of presentation, we assume "a large region of land" is a 5-gram candidate phrase as an example in the document. typically, most existing keyphrase extraction models mainly include two procedures: candidate keyphrase extraction and keyphrase importance estimation. specifically, the former extracts candidate phrases from the document via some heuristics @xcite @xcite @xcite , and the latter determines which candidate phrases are keyphrases via unsupervised or supervised models @xcite @xcite . the keyphrase importance estimation procedure usually plays a more critical role than the candidate keyphrase extraction procedure in the supervised setting. in the supervised neural keyphrase extraction models, the keyphrase importance estimation procedure can be subdivided into information representation and importance discrimination. specifically, the information representation part focuses on modeling the encoding procedure, and the importance discrimination part focuses on measuring the important scores of candidate phrases. to represent information comprehensively, recent keyphrase extraction studies have been proposed to build better representations via bi-lstm @xcite , gcns @xcite , and the pre-trained language models (e.g., elmo @xcite , bert, and roberta @xcite ). to discriminate the importance of candidate phrases precisely, most existing supervised keyphrase extraction models @xcite @xcite estimate and rank the importance of candidate phrases to extract keyphrases by using different approaches, such as classification and ranking models. although the existing keyphrase extraction models mentioned above have achieved significant performance, the keyphrase extraction task still needs improvement. among them, there are the following two main issues. the first issue lies in the information representation. typically, candidate phrases often exhibit the inherent hierarchical structures ingrained with complex syntactic and semantic information @xcite . in general, the longer phrases contain more complex structures. (as shown in figure 1 , the phrase "a large region of land" has more complex inherent structures than "region" or "a large region". similarly, the phrase "a large region" is more complex than "region"). besides the phrases, since linguistic ontologies are intrinsic hierarchies @xcite , the conceptual relations between phrases and their corresponding document can also form the hierarchical structures. therefore, the hierarchical structures need to be considered when representing both phrases and documents and estimating the phrase-document relevance. however, it is difficult to capture such structural information even with infinite dimensions in the euclidean space @xcite . the second issue lies in distinguishing the importance of phrases. keyphrases are typically used to retrieve and index their corresponding document, so they should be highly related to the main points of the source document (hasan and ng, 2014). however, most existing supervised keyphrase extraction methods ignore explicitly modeling the relevance between candidate keyphrases and their corresponding document, resulting in biased keyphrase extraction. motivated by the above issues, in this paper, we explore the potential of hyperbolic space for the keyphrase extraction task and propose a new hyperbolic relevance matching model (hypermatch) for supervised neural keyphrase extraction. firstly, to capture hierarchical syntactic and semantic structure information, hypermatch integrates the hidden representations in all the intermediate layers of roberta to collect the adaptive contextualized word embeddings via an adaptive mixing layer based on the self-attention mechanism. and then, considering the hierarchical structure hidden in the natural language content, hypermatch represents both phrases and documents in the same hyperbolic space via a hyperbolic phrase encoder and a hyperbolic document encoder. meanwhile, we adopt the poincaré distance to calculate the phrase-document relevance by considering the latent hierarchical structures between candidate keyphrases and the document. in this setting, the keyphrase extraction task can be regarded as a matching problem and effectively implemented by minimizing a hyperbolic margin-based triplet loss. to the best of our knowledge, we are the first work to explore the supervised keyphrase extraction in hyperbolic space. experiments on six benchmark datasets demonstrate that hypermatch outperforms the state-of-the-art keyphrase extraction baselines. ## preliminaries hyperbolic space is an important concept in hyperbolic geometry, which is considered as a special case in the riemannian geometry @xcite . before presenting our model, this section briefly introduces the basic information of hyperbolic space. in a traditional sense, hyperbolic spaces are not vector spaces; one cannot use standard operations such as summation, multiplication, etc. to remedy this problem, one can utilize the formalism of möbius gyrovector spaces allowing the generalization of many standard operations to the hyperbolic spaces @xcite . similarly to the previous work @xcite @xcite , we adopt the poincaré ball and use an additional hyper-parameter c which modifies the curvature of poincaré ball; it is then defined as the corresponding conformal factor now takes the form λ c x := adaptive contextualized word embedding word index hyperbolic phrase representation hyperbolic document representation scaled phrase-document relevance hyperbolic document encoder hyperbolic cnn network hyperbolic phrase encoder hyperbolic word representation hyperbolic average pooling interactions in poincaré disk source document layer 1 layer 2 layer 3 layer 12 … adaptive mixing layer roberta euclidean word representation i. information representation ii. importance discrimination hyperbolic projection möbius addition. for a pair x, @xmath0, the möbius addition is defined as, möbius matrix-vector multiplication. for a linear map m : r @xmath1, if @xmath2, then the möbius matrix-vector multiplication is defined as, poincaré distance. the induced distance function is defined as, note that with @xmath3, while with @xmath4, y) = 2 xy . exponential and logarithmic maps. to perform operations in hyperbolic space, one first needs to define a mapping function from r n to d n c to map the euclidean vectors to the hyperbolic space. let t x d n c denote the tangent space of d n c at x. the exponential map exp c hyperbolic averaging pooling. the average pooling, as an important operation common in natural language processing, is averaging of feature vectors. in the euclidean setting, this operation takes the following form: extension of this operation to hyperbolic spaces is called the einstein midpoint and takes the most simple form in klein coordinates: therefore, given points in the poincaré ball, we can first map them to the klein model via eq.( 9 ), compute the average using eq.( 7 ), and then move it back to the poincaré model via eq.( 8 ). ## hypermatch given a document @xmath5, ..., w i , ..., w m }, the candidate phrases are first extracted from the source document by the n-gram rules, where m indicates the max length of the input document. then, to determine which candidate phrases are keyphrases, we design a new hyperbolic relevance matching model (hypermatch), which mainly consists of two components: information representation and importance discrimination. figure 2 illustrates the overall framework of hypermatch. ## results and analysis in this section, we test the performance of hyper-match on six widely-used benchmark keyphrase extraction datasets (openkp, kp20k, inspec, krapivin, nus, and semeval) from three facets. the first one demonstrates its superiority by comparing hypermatch with the recent baselines in terms of several metrics. the second one is to verify the effect of each component via ablation tests. the last one is to test the sensitivity of the hyperbolic margin-based triplet loss with different margins. ## related work this section briefly describes the related work from two fields: keyphrase extraction and hyperbolic deep learning. ## conclusions and future work a new hyperbolic relevance matching model hy-permatch is proposed to map candidate phrases and documents representations into the hyperbolic space and model the relevance between candidate phrases and the document via the poincaré distance. specifically, hypermatch first combines the intermediate layers of roberta via the adaptive mixing layer for capturing richer syntactic and semantic information. then, phrases and documents are encoded in the same hyperbolic space to cap-ture the latent hierarchical structures. next, the phrase-document relevance is estimated explicitly via the poincaré distance as the importance scores of all the candidate keyphrases. finally, we adopt the hyperbolic margin-based triplet loss to optimize the whole model for extracting keyphrases. in this paper, we explore keyphrase extraction in hyperbolic space and implicitly model the latent hierarchical structures hidden in natural languages when representing candidate keyphrases and documents. in the future, it will be interesting to introduce external knowledge (e.g., wordnet) to explicitly model the latent hierarchical structures when representing candidate keyphrases and documents. in addition, our code is publicly available to facilitate other research foot_4 .