SAMR ank: Unsupervised Keyphrase Extraction using Self-Attention Map in BERT and GPT -2 (2023)

Abstract: We propose a novel unsupervised keyphrase extraction approach, called SAMRank, which uses only a self-attention map in a pre-trained language model (PLM) to determine the importance of phrases. Most recent approaches for unsupervised keyphrase extraction mainly utilize contextualized embeddings to capture semantic relevance between words, sentences, and documents. However, due to the anisotropic nature of contextual embeddings, these approaches may not be optimal for semantic similarity measurements. SAMRank as proposed here computes the importance of phrases solely leveraging a self-attention map in a PLM, in this case BERT and GPT-2, eliminating the need to measure embedding similarities. To assess the level of importance, SAMRank combines both global and proportional attention scores through calculations using a self-attention map. We evaluate the SAMRank on three keyphrase extraction datasets: Inspec, SemEval2010, and SemEval2017. The experimental results show that SAMRank outperforms most embedding-based models on both long and short documents and demonstrating that it is possible to use only a self-attention map for keyphrase extraction without relying on embeddings. Source code is available at https://github.com/kangnlp/SAMRank.

## introduction keyphrase extraction refers to process of identifying the words or phrases that signify the primary themes of a document. it has a wide range of applications, including document summarization, information retrieval, and topic modeling. the methodologies used for keyphrase extraction are typically categorized into supervised and unsupervised approaches. although supervised keyphrase extraction yields excellent performance, it relies heavily on large quantities of labeled data and tends * corresponding author. to be domain-specific. in contrast, unsupervised keyphrase extraction uses only the information intrinsic to the document. traditional methods rely on statistical information, such as tf-idf @xcite , and graphs based on the co-occurrence of words @xcite . however, these methods often fall short of deciphering latent meanings in the text, resulting in suboptimal performance. the introduction of deep learning-based language models changed this paradigm, by enabling the extraction of phrases closely connected semantically to the document through the calculation of the degree of similarity between the document and phrase embeddings. however, this paper does not utilize embeddings. the rationale behind this is that contextualized representations are not well-suited to tasks based on similarity. a study that has geometrically analyzed the representations of plms such as elmo @xcite , bert @xcite , and gpt-2 @xcite , have postulated that the contextualized representations of these language models exhibit anisotropy @xcite . these representations tend to concentrate on specific directions of the vector space, becoming increasingly anisotropic in higher layers. indeed, the average cosine similarity of two word representations randomly chosen from bert and gpt-2 markedly increases with the layer depth. given that embeddings typically come from the last layer's representation of a plm, anisotropic embeddings can compromise tasks that require similarity-based rankings. to counteract this, bwrank @xcite improved the performance of keyphrase extraction by isotropically reconstructing bert's embeddings by reducing dimensions through whitening operation. in an earlier work entitled "what does bert look at? an analysis of bert's attention" @xcite , the authors sought to understand how bert's attention mechanism operates by determining how much attention a specific word gives to other words in each head of bert. their findings indicate that certain bert heads were skilled in identifying grammatical structures and coreferences such as direct objects of verbs and objects of prepositions. this suggests that in transformer-based plms, such as bert that employ a multi-head attention mechanism to handle linguistic complexity, certain heads may specialize in particular tasks. building on this research, we can posit that some heads of a transformer-based plm may specialize in capturing keyphrases given their importance in providing an understanding of the thematic essence of the text and considering the likelihood that some heads may have been trained to detect keyphrases during the pre-training process of the plm. therefore, this paper shifts the focus from plm embeddings to the self-attention map afforded by each head in the plm. we propose samrank, a novel approach that leverages only the selfattention map of a transformer-based plm to extract keyphrases. samrank calculates a global attention score by summing the attention weights received by the token from all tokens within the document. it also redistributes the global attention score proportionally to other tokens that are focusing on globally significant tokens based on the attention weights they have allocated, yielding a proportional attention score. the combination of these two scores determines the final score for tokens. by aggregating the final scores of each token within a candidate phrase, samrank calculates the total score for the phrase and determines its overall significance. this results in ranking of candidate phrases based on their total scores, thus allowing the extraction of phrases that both receive high attention and are highly attentive to phrases receiving global attention. an overview of samrank is presented in figure 1 and the final token scores calculated through samrank can be visualized, as depicted in figure 2 . in this paper, we independently analyze the selfattention map of each head in each layer to identify heads that excel in keyphrase extraction. we extract the self-attention map from all 144 heads in bert and gpt-2, using three representative keyphrase extraction datasets: inspec, semeval2010, and se-meval2017. the performance of each head is evaluated, with experimental results showing that sam-rank outperforms most embedding-based methods in specific heads of both bert and gpt-2. this paper's main contributions are three-fold: the transformer model proposed in "attention is all you need" @xcite introduced the attention mechanism, bringing about a revolution in the field of natural language processing. in this section, we introduce a self-attention map, which is the computational output of self-attention, the core mechanism of the transformer. additionally, we discuss two transformer-based plms, bert and gpt-2, which have demonstrated superior performance in various natural language processing tasks. ## methodology our proposed methodology involves ranking phrases based on the self-attention map (sam) of transformer-based language models, in this case bert and gpt-2. each row of sam, which is a self-attention weights vector, is perceived as each token allotting its attention according to its relevance with other tokens. the tokens that are semantically connected to many other tokens are more likely to receive this 'attention'. hence, the global attention score of each token is computed by summing the attention weight it received from all the tokens in the text. then, similar to the attention operation (multiplying the v vector by the attention weight), the global attention score is distributed in proportion to the attention weight assigned by the token. this results in the calculation of the proportional attention score, where tokens strongly associated with globally significant tokens are assigned high scores. the final score for a token is calculated by combining the global attention score and the proportional attention score. for phrases, their scores are computed as the accumulated sum of the scores of their constituent tokens across all occurrences in the document. based on these phrase-level scores, the phrases are then ranked to determine their overall importance. ## analysis of heads to identify heads that specialized in keyphrase extraction, we evaluate the performance of each of the 144 heads in bert and gpt-2. the f1@15 performance of each head across all layers for the three datasets is detailed in appendix a. we find that the performance of specific heads shows consistent patterns across all three datasets. in bert, heads mainly located in the third and fifth layers demonstrate high performance, whereas in gpt-2, heads in the eleventh layer are particularly performant. however, the head yielding the highest performance varies across the datasets. this indicates that specific heads in transformer-based plms do pay substantial attention to phrases perceived as keyphrases by humans, but their performance varies depending on the type and length of the text, suggesting that each head might adopt different perspectives or attention patterns when identifying keyphrases. moreover, we observe that the intra-layer performance variance in bert is larger than in gpt-2. this might suggest that some heads in bert specialize in tasks unrelated to semantic keyphrase extraction, possibly focusing on syntactic or other non-semantic tasks. despite the relatively small performance differences among the top-performing heads across different layers in bert, we identify some particular heads (first and eleventh head) in the eleventh layer of gpt-2 that significantly outperforms all other heads, including those in bert. this suggests the possibility of certain heads in gpt-2 being highly specialized for keyphrase identification. ## samrank on llms samrank is based on the self-attention mechanism, so it can be used in transformer models other than bert or gpt-2. we conduct keyphrase extraction experiments applying samrank on the large language models (llms) that have recently attracted significant attention in the field of natural language processing. we utilize llama 2, recently released by meta @xcite . we per-form experiments on: llama 2 7b foot_5 , llama 2 13b foot_6 , and llama 2 70b foot_7 . llama 2 7b consists of 32 layers, each with 32 heads. llama 2 13b consists of 40 layers, each with 40 heads. llama 2 70b consists of 80 layers, each with 64 heads. the results are presented in in our experiments, llama 2 models mostly show comparable performance to bert and gpt-2. however, the llama 2 70b model surpasses both bert and gpt-2 in the inspec and semeval2017 datasets, achieving the highest scores in f1@5 across all datasets. these superior performances in f1@5 are also observed in our experiments with bert and gpt-2, indicating a consistent trend across different plm-based models. importantly, among the llama 2 models, there is a clear trend that an increase in the number of parameters leads to better performance. this observation supports the prevalent notion in llm research that increasing model size can lead to improved performance. ## related work traditional unsupervised keyphrase extraction (uke) approaches broadly divided into two categories: statistical-based and graph-based methods. statistical approaches, like tf-idf @xcite and yake @xcite , use information such as word frequency and distribution. graphbased methods, like textrank @xcite , singlerank @xcite , topi-crank @xcite , and positionrank @xcite , represent a document as a graph and rank phrases based on the graph's characteristics. traditional unsupervised keyphrase extraction approaches that rely solely on surface-level features often fail to extract semantically significant keyphrases. to address these limitations, methods utilizing embeddings emerged. embedrank @xcite is an example that employs static embeddings such as sent2vec and doc2vec. subsequently, a new trend in keyphrase extraction emerged with the advent of plms. sifrank @xcite utilizes elmo, a plm capable of generating contextual embeddings. more recently, transformer-based models like bert have been widely adopted. attentionrank @xcite ) applies bert's self-attention and cross-attention mechanisms to keyphrase identification, examining the relevance of a candidate keyphrase within its sentence and the entire document. masked document embedding rank (mderank) @xcite introduces a masking strategy, ranking keyphrases by comparing the similarity of the source and masked document embeddings. this approach benefits from a customized bert model trained via a selfsupervised contrastive learning method. jointgl @xcite merges local and global contexts for keyphrase extraction, creating a graph structure to capture local information and calculating semantic similarity for a global perspective. ## conclusion in this paper, we present a new perspective on keyphrase extraction that moves away from the conventional embedding-based approach and solely leverages the information from the self-attention map of plm. the proposed samrank extracts keyphrases based on a combination of the global attention score, which identifies tokens that attract attention within a document, and the proportional attention score, which finds tokens deeply associated with important tokens. experimental results on three representative keyphrase extraction datasets demonstrate that samrank consistently outperforms embedding-based models, improving f1@5 and f1@10 scores by approximately 11.5% and 3% on average, respectively. these results suggest that some heads in plms, particularly bert and gpt-2, could be specialized in capturing keyphrases. we also demonstrate that the self-attention map of these heads can provide more useful information for keyphrase extraction than embeddings. future work could potentially utilize the samrank approach to enhance the interpretability by thoroughly analyzing the specific roles and functionality of each head in transformer-based plms, beyond the scope of keyphrase extraction.