H yper R ank: Hyperbolic Ranking Model for Unsupervised Keyphrase Extraction (2023)

Abstract: Given the exponential growth in the number of documents on the web in recent years, there is an increasing demand for accurate models to extract keyphrases from such documents. Keyphrase extraction is the task of automatically identifying representative keyphrases from the source document. Typically, candidate keyphrases exhibit latent hierarchical structures embedded with intricate syntactic and semantic information. Moreover, the relationships between candidate keyphrases and the document also form hierarchical structures. Therefore, it is essential to consider these latent hierarchical structures when extracting keyphrases. However, many recent unsupervised keyphrase extraction models overlook this aspect, resulting in incorrect keyphrase extraction. In this paper, we address this issue by proposing a new hyperbolic ranking model (HyperRank). HyperRank is designed to jointly model global and local context information for estimating the importance of each candidate keyphrase within the hyperbolic space, enabling accurate keyphrase extraction. Experimental results demonstrate that HyperRank significantly outperforms recent state-of-the-art baselines.

## introduction keyphrase extraction is a natural language processing task that involves extracting a set of representative keyphrases from the source document @xcite . this task is valuable for generating a concise summary or snippet of a web page. the core objective of keyphrase extraction is to condense information from the input document and preserve only the most essential details in the output. distinguishing what information is important is a challenge in the keyphrase extraction task, primarily due to the fact that candidate keyphrases often exhibit latent hierarchical structures @xcite , as * corresponding author function error function a multiplicative error function multiplicative error function relations binary relations representation of binary relations numerical representation of binary relations a error function illustrated in figure 1 . consequently, when extracting keyphrases, it is necessary to consider these latent hierarchical structures to accurately estimate the importance scores of candidate keyphrases. typically, most existing unsupervised keyphrase extraction models that rely on pre-trained word embeddings, as discussed in @xcite @xcite @xcite , can be categorized into two primary steps: candidate keyphrase generation and keyphrase importance estimation @xcite . specifically, the former step involves extracting a list of words or phrases from the source document that serve as candidate keyphrases through heuristic methods @xcite @xcite @xcite . the latter step mainly consists of two components: text representation and importance calculation. text representation is obtained using pre-trained language models such as bert @xcite . subsequently, the importance scores of candidate keyphrases are estimated based on the pre-trained embeddings. this is achieved by calculating textual similarities between candidate keyphrases and the document using various distance measures, such as manhattan distance, euclidean distance, and cosine distance. these measures are employed to determine which candidate keyphrases are the real keyphrases @xcite @xcite @xcite . in recent years, a significant breakthrough in natural language processing has been the widespread adoption of heavily pre-trained transformers designed for natural language modeling, exemplified by bert @xcite and roberta @xcite . these pre-trained language models (plms) have proven to be formidable tools for various downstream tasks in natural language processing and information retrieval. as a result, they have evolved into indispensable components, serving as embedding layers to obtain contextualized representations in most nlp downstream tasks. leveraging the advancements in text representation, embedding-based unsupervised keyphrase extraction models @xcite @xcite @xcite have demonstrated promising results and have now established themselves as the new state-of-the-art benchmarks. candidate keyphrases frequently present an inherent hierarchical structure infused with intricate syntax and semantics @xcite @xcite . in addition to candidate keyphrases, the relations between keyphrases and the document can also give rise to hierarchical structures. however, capturing such hierarchies, even in infinite dimensions within euclidean space, poses a considerable challenge @xcite . fortunately, hyperbolic space naturally lends itself to modeling these hierarchical structures, including tree-like characteristics @xcite . this property makes hyperbolic space a valuable tool applied in various downstream natural language processing tasks @xcite @xcite . motivated by the phenomena mentioned above, in this paper, we explore the task of embedding-based unsupervised keyphrase extraction in the hyperbolic space. more specifically, we propose a new hyperbolic ranking model (hyperrank) designed to model the importance estimation in the hyperbolic space. in hyperrank, we first leverage the pre-trained language model bert as the backbone of our model to enhance the quality of text representations, projecting these representations into the hyperbolic space. to estimate the importance of each candidate keyphrase accurately, we simultaneously consider two perspectives: phrase-document relevance and cross-phrase relevance. concretely, the former calculates the relevance scores between candidate keyphrases and their corresponding document, while the latter calculates the relevance scores between all candidate keyphrases. we then combine these two relevance scores to determine the final importance scores of candidate keyphrases to rank and extract keyphrases. our experimental results demonstrate that the proposed model hy-perrank outperforms recent strong baselines significantly. we conduct various analyses on existing datasets to explore the characteristics of our model. additionally, we discover that merely mapping representations to the hyperbolic space may likely mitigate the anisotropic issues in text representation in the keyphrase extraction task. ## related work we briefly introduce the recent progress on unsupervised keyphrase extraction and hyperbolic representation learning in this section. ## preliminary generally, hyperbolic space can be characterized using riemannian geometry, as described in @xcite . in line with previous studies @xcite @xcite , we employ the poincaré ball model and introduce an additional hyper-parameter denoted as c to modify the curvature of the poincaré ball. this curvature modification is defined as exponential and logarithmic maps. to perform operations within the hyperbolic space, it is necessary to establish a mapping function from r n to d n c in order to transfer euclidean vectors into the hyperbolic space. we denote the tangent space of d n c at x as t x d n c . the exponential map exp c poincaré distance. the induced distance function is defined as, here, d c (•) denotes the poincaré distance. note that with @xmath1, while with @xmath2, represented as, lim @xmath3, y) = 2 xy . ## methodology the overall architecture of our model hyperrank is illustrated in figure 2 . hyperrank comprises two main parts: candidate keyphrase generation and keyphrase importance estimation. in the first part, we employ natural language linguistic techniques to generate candidate keyphrases from the input document, as illustrated in figure 3 . the second part involves embedding candidate keyphrases and their corresponding document into a lowdimensional semantic space using the pre-trained candidate keyphrases input document 𝑝 ! 𝑝 " … 𝑝 # … tokenize + pos tagger 𝑤 @xmath0 𝑤 " 𝑤 % … 𝑤 & in jj nn nnp 𝑤 $ 𝑤 " 𝑤 % … 𝑤 & figure 3: the procedure of our candidate keyphrase generation. (1) get tokenized document and pos tags. ## experiments and results in this section, we conduct experiments to demonstrate the effectiveness of our proposed model hy-perrank. concretely, we first introduce our experimental settings and then present the experimental results and analysis. ## conclusion in this paper, we propose a novel hyperbolic ranking model for unsupervised keyphrase extraction (hyperrank). hyperrank is designed to concurrently model text representation and importance estimation in the hyperbolic space. to ensure the accuracy of the importance estimation of each candidate keyphrase, hyperrank simultaneously utilizes global and local contextual information to estimate the relevance scores between candidate keyphrases and their corresponding documents. afterward, hy-perrank combines these two aspects of relevance to derive the final importance score for each candidate keyphrase, which is then used for ranking and keyphrase extraction. our extensive experiments conclusively demonstrate that the proposed model hyperrank surpasses the performance of existing state-of-the-art unsupervised keyphrase extraction baselines.