A ttention R ank: Unsupervised Keyphrase Extraction using Self and Cross Attentions (2021)

Abstract: Keyword or keyphrase extraction is to identify words or phrases presenting the main topics of a document. This paper proposes the AttentionRank, a hybrid attention model, to identify keyphrases from a document in an unsupervised manner. AttentionRank calculates self-attention and cross-attention using a pre-trained language model. The self-attention is designed to determine the importance of a candidate within the context of a sentence. The cross-attention is calculated to identify the semantic relevance between a candidate and sentences within a document. We evaluate the AttentionRank on three publicly available datasets against seven baselines. The results show that the AttentionRank is an effective and robust unsupervised keyphrase extraction model on both long and short documents. Source code is available on Github.

## introduction a vast amount of scientific or non-scientific articles are published online every year. although some of them have associated keyphrases specified, which made them easy to index and search, a considerable amount of them has no keyphrase defined, making indexing and information retrieval challenging. given so many articles available, it is not feasible to manually extract keyphrases from each. automate the keyphrases extraction becomes crucial. keyphrase extraction has immense value to the downstream text mining tasks, such as text segmentation, text summarization, query expansion, indexing, and so on. keyphrase extraction methods can be supervised or unsupervised. traditional supervision methods use decision tree @xcite or naive bayes @xcite to identify whether the input word/phrase is a keyphrase. with the advancing of neural networks, various supervised deep learning models @xcite @xcite are developed to extract keyphrases. the supervised method requires a large labeled training dataset and is often domainspecific, whereas unsupervised methods do not need labeled datasets. the traditional unsupervised methods use statistical and graph-based approaches. the statistical-based methods @xcite @xcite utilize the candidate position, frequency, length, and capitalization to determine the importance of a word. the graph-based approaches @xcite ) construct a graph with the candidates as nodes. the edges indicate the similarity or co-occurrences of the candidates. graph-based algorithms can be applied to calculate the importance of the nodes (candidates) on the graph. driven by recent deep language models for natural language processing and text analysis, text embedding-based or mixed statistical and embedding-based unsupervised keyphrase extraction methods have emerged, such as embedrank @xcite , sifrank @xcite , and keygames @xcite . in this research, we propose an attention-based unsupervised model -attentionrank for keyphrase extraction. attentionrank is motivated by the self-attention mechanism of the bert model @xcite , and hierarchical attention retrieval (har) mechanism @xcite . at-tentionrank model calculates the accumulated selfattention and cross-attention of a candidate to rank the importance. the accumulated self-attention value for each candidate is extracted from a pretrained bert model. it is calculated as adding the received attention from other words within a sentence then summing up these self-attention values on all sentences within the document. the accumulated self-attention addresses the importance of a candidate within the sentences where it locates. the cross-attention identifies the semantic relevance between a candidate and the document. it calculates the word-level bidirectional attention between the embeddings of a candidate and sentences within a document then generate an enhanced document embedding for the candidate. the final ranking of a candidate is determined by the linear integration of the accumulated self-attention value and the cross-attention relevance value. a postprocessing step based on the document frequency is used to remove the generic terms of a specific corpus. attentionrank is the first to use the attention values extracted from a pre-trained bert model for keyphrase extraction. no additional information or domain knowledge is needed. it is generalizable to documents of any domain. this research focused on investigating whether the pre-trained none domain-specific language model can be used for keyphrase extraction. attentionrank is compared against seven stateof-the-art unsupervised keyphrase extraction methods on three benchmark datasets. two datasets contain short documents, and one contains long documents. the results show that attentionrank performs better than or as competitive as the baselines. the main contributions of this paper are summarized as follows: ## methodology in this section, we introduce the attentionrank model in detail. the overview architecture of our model is shown in figure 1 . attentionrank integrates the accumulated self-attention component with the cross-attention component to calculate the final score of a candidate. the proposed model has four main steps: (1) generate a candidate set c from a document; (2) calculate the accumulated self-attention value a c for each candidate c, @xmath0; (3) calculate the cross-attention relevancy value (r c ) between a candidate c and the document d; (4) calculate the final score s c for each candidate through a linear combination of a c and r c . 2.1 candidates generation we use the candidate extraction module implemented in embedrank (bennani-smires et al., 2018). this module first use part-of-speech (pos) to identify words tagged as nn, nns, nnp, nnps, jj, etc. then, the python package nltk 2 is used to generate the noun phrases, which are candidates. given a sentence 'most parameters of the program are controllable by experimenter-edited text files or simple switches in the program code, thereby minimizing the need for programming to create new experiments', the extracted candidates are 'program code', 'experimenter-edited text files', 'need', 'parameters', 'simple switches', 'program', and 'new experiments'. ## related work there are supervised and unsupervised keyphrase extraction approaches. the unsupervised approaches can be grouped into two categories: traditional unsupervised methods and deep learningbased methods. rose et al. @xcite identified the keywords based on the word frequency, the number of co-occurring neighbors, and the ratio between the co-occurrence and the frequency. campos et al. @xcite calculated the importance of each candidate using frequency, offsets, and co-occurrence. alrehamy et al. @xcite first clustered the candidates based on the semantic similarity. candidates with similar semantic relevance with the centroids were selected as keywords. rabby et al. @xcite ) took a binary tree approach. its statistical relevance determined the importance of a subtree with its root. then, they extracted all the paths from the roots to the leaves to find the keyphrases. mihalcea et al. @xcite converted the candidates into nodes on a graph, then used the pagerank algorithm to calculate the importance of the candidates wan wang et al. @xcite made use of the pre-trained word embedding and the frequency of each word to generate weighted edges between words in a document. a weighted pagerank algorithm was used to compute the final scores of words. mahata et al. @xcite used a similar approach using the phrase embeddings for representing the candidates and ranking the importance of the phrases by calculating the semantic similarity and co-occurrences of the phrases. papagiannopoulou et al. @xcite also used word embeddings for unsupervised keyphrase extraction. different from the previous two methods, the embeddings were trained from a single document called local embedding. bennani-smires et al. @xcite used a document embedding model, embedrank, to measure the similarity between documents and candidates to select more representative keyphrases. sun et al. @xcite proposed sifrank, integration of statistical model and pre-trained language model, to calculate the relevance between candidates and document topic. saxena et al. @xcite employed the concept of evolutionary game theory and utilized the embeddings, position, and frequency of the candidate to calculate the confidence score to determine whether a candidate is a keyphrase. different from previous methods, this study focuses on integrating self-attention weights extracted from the pre-trained deep language model with the calculated cross-attention relevancy value to identify the keyphrases that are important to local sentence context and also have strong relevancy to all sentences within the whole document. ## conclusions this research investigated the accumulated selfattention mechanism integrated with a crossattention model for unsupervised keyphrase extraction. a pre-trained bert model is utilized to calculate the self-attention and cross-attention values. a candidate is processed through a self-attention calculation and a cross-attention relevancy calculation to gain a final score towards ranking. we compared the proposed attentionrank model with seven different baselines on three benchmark datasets, including two short document datasets and one long document dataset. attentionrank gains a better or competitive f1@5, 10, and 15 on all datasets. the ablation study shows that accumulated selfattention has a higher contribution than the crossattention relevancy score on the long document set. for a short document set, the linear integration of both attention mechanisms shows good performance. the future work includes fine-tuning the bert on a target domain and comparing against more baselines on domain-specific datasets.