AGR ank: Augmented Graph-based Unsupervised Keyphrase Extraction (2022)

Abstract: Keywords or keyphrases are often used to highlight a document’s domains or main topics. Unsupervised keyphrase extraction (UKE) has always been highly anticipated because no labeled data is needed to train a model. This paper proposes an augmented graph-based unsupervised model to identify keyphrases from a document by integrating graph and deep learning methods. The proposed model utilizes mutual attention extracted from the pre-trained BERT model to build the candidate graph and augments the graph with global and local context nodes to improve the performance. The proposed model is evaluated on four publicly available datasets against thirteen UKE baselines. The results show that the proposed model is an effective and robust UKE model for long and short documents. Our source code is available on GitHub.

## introduction the mainstream unsupervised keyphrase extraction (uke) approaches fall into one of three types: statistical, graph-based, and deep learning approaches. the statistical methods include the tf-idf-based approach and other recent works @xcite , which utilize term frequency, document frequency, word offsets and the number of n-grams to calculate the importance of the candidates. the graph-based methods treat the candidates as the nodes in a graph @xcite . the edges are calculated based on candidates' co-occurrences, semantic similarity, or other relations. graph-based algorithms then determine the importance of candidates. several recent studies have shown that embedding-based methods can achieve excellent performance on unsupervised keyphrase extraction, such as jointmodeling @xcite , atten-tionrank @xcite , sifrank @xcite , keygames @xcite and embedrank @xcite . these approaches base candidates' importance on the distance or similarity of candidate embeddings, and some consider the global or local context. we propose an augmented graph-based unsupervised model to identify keyphrases from documents. the model extracts attention from the pre-trained bert model to generate a candidate keyphrase graph, then augments the attention graph with nodes that present the global and local context. similar to the baseline approaches, noun phrases are extracted as candidates representing the nodes on the graph. the co-occurrence of candidates determines graph edges within the sentential context. edge weights between the candidates are calculated based on the mutual attention extracted from the pre-trained bert model and the indexes of the sentences where the candidates are located. the candidate graph adds the global and local contexts as document and sentence nodes. the edge weights between the document node and candidates and the edge weights between sentence nodes and candidates are calculated based on the cosine similarity between their embeddings. the graph is then adjusted by removing nodes and edges based on the document frequency and edge weights. finally, the ranking of each candidate is calculated using the weighted pagerank algorithm. we summarize our contributions as follows: ## methodology our model has three main parts: (1) candidate graph generation, in which we convert each document into a weighted graph with candidates as nodes and attention between candidates in a sentential context as weighted edges; ## related works the unsupervised keyphrase extraction approaches can be categorized into statistical, graph-based, and deep learning-based or mixed methods. the models based on statistical techniques convert contextual information into statistical features of candidates and then calculate candidate scores for ranking. rose et al. (2010) utilized the ratio of word frequency and the number of co-occurring neighbors to evaluate the importance of the candidates. besides term frequency and neighbor co-occurrence, campos et al. ( 2020 ) also considered more contextual features to identify keyphrases, including the offsets of the candidates, the sentence positions of the candidates first shown, etc. models based on graph methods treat candidates as nodes of the graph, convert certain relations between candidates into edges of the graph, then use a graph algorithm to calculate the candidates' scores @xcite . @xcite utilized a clustering method to select k-nearest-neighbor documents to create a graph for a single document and used a graph sorting algorithm to generate keyphrases. @xcite employed a clustering method to generate several topics of a document and assign the topics to candidates, then utilized the textrank model to rank topics; the most representative candidates of the top-ranked topics are extracted as keyphrases. @xcite utilized the word embedding and word frequency to generate weighted edges between words, then used the weighted pagerank algorithm to compute candidate scores and rankings. @xcite proposed the position-biased pagerank algorithm, which incorporates the candidate positions in the document into the ranking calculation. @xcite proposed the multipartite graph model, which encodes the topic information within a multipartite graph to utilize candidate mutual relations. yeon sung and kim (2020) extracted hierarchical relationships to determine which edges and phrases should be used and evaluated the nodes according to their inflowing edges. @xcite proposed the embedrank, which uses a pre-trained language model to generate the document and candidate embeddings and calculate the similarity between them to select more representative keyphrases. @xcite proposed sifrank, which invokes both the similarity between candidate and document embeddings and the candidate position and frequency to calculate the correlation between candidates and the document. @xcite investigated an evolutionary game theory model that uses candidate embeddings and statistics to calculate confidence scores to determine whether a candidate is a keyphrase. @xcite extracted attention mapping weights and then integrated accumulated attention weights with the cross-attention similarity to rank the candidates. @xcite integrated bounded sentences and candidate local relations based on document-to-candidate global relations, then used both jointly to determine the importance of candidates. @xcite proposed mderank, which ranked candidates using the similarity between the bert embeddings of the source document and the masked document. ## limitations, conclusions, and future work although our augmented graph-based model performs better than the compared baselines, the graph augmentation process is designed with quite a few hyperparameters that need to be tuned for datasets of different domains to obtain optimal performance. we believe this can be further improved by automating the hyperparameter tuning process. our research investigated the integration of graph-based and deep learning-based models for unsupervised keyphrase extraction. the pre-trained bert model is utilized to extract candidates' mu-tual attention to build the initial graph. global and local context information are added through graph augmenting. pagerank algorithm is used to calculate the ranking scores. we compared the proposed model against 13 baseline unsupervised keyphrase extraction models on four benchmark datasets. the ablation study shows that the edge weight adjustment based on sentence position has a higher impact on the long document sets. adding the document and sentence nodes improves the performance for short document sets. future work includes investigating possible solutions to reduce the number of parameters and improve efficiency. we also plan to compare our unsupervised model against supervised keyphrase extraction models to demonstrate the advantages and performances.