Keyword Extraction Using Unsupervised Learning on the Document’s Adjacency Matrix (2021)

Abstract: This work revisits the information given by the graph-of-words and its typical utilization through graph-based ranking approaches in the context of keyword extraction. Recent, well-known graph-based approaches typically employ the knowledge from word vector representations during the ranking process via popular centrality measures (e.g., PageRank) without giving the primary role to vectors’ distribution. We consider the adjacency matrix that corresponds to the graph-of-words of a target text document as the vector representation of its vocabulary. We propose the distribution-based modeling of this adjacency matrix using unsupervised (learning) algorithms. The efficacy of the distribution-based modeling approaches compared to state-of-the-art graph-based methods is confirmed by an extensive experimental study according to the F1 score. Our code is available on GitHub.

## introduction automatic keyword extraction (ake) intends to discover a limited but concise set of words that reflect the main topics discussed within a text document, avoiding the expensive and time-consuming process of manual annotation by experts @xcite . besides, many keyphrase extraction methods form and rank the candidate phrases using the previously scored candidate unigrams by a keyword extractor, as keyphrases consist of n-grams with @xmath0. both supervised and unsupervised approaches are quite famous for the ake task @xcite . during the last two years, the research community pays significant attention on (supervised) deep learning methods @xcite @xcite as the performance of the unsupervised ones shows a relative stagnation (or minimal improvements) compared to the supervised techniques. the use of standard external tools for grammatical/syntactic analysis and information sources such as (pre-trained) static word embeddings @xcite that have a bias over the corpora domains used for training may also exacerbate the problem. moreover, most methods suggest a fixed or relative with the document length number of keywords dissociating the number of returned keywords from the number of topics discussed in the document @xcite . however, unsupervised methods are of timeless interest. they often are domain or language-independent and do not need any labelled data to train models compared to the supervised ones. the graph-based approaches (i.e., the most popular category of the unsupervised ake) consider the "central" nodes of a graph-of-words as the most representative ones usually according to (variations of) the pagerank @xcite ) centrality, i.e., the most effective graph-based ranking method employed by the majority of the state-of-the-art @xcite @xcite . additionally, traditional (semi-)supervised, or even deep learning approaches @xcite @xcite utilize the unsupervised methods mentioned earlier to improve their performance. this work takes a novel unsupervised path to keyword extraction revisiting the information provided by the graph-of-words and its conventional utilization via pagerank. inspired by the recent approach of @xcite , we investigate the effectiveness of the distributionbased modeling of the adjacency matrix, that corresponds to various versions of the (unweighted, weighted or/and enhanced with positional information) graph-of-words for the target document. we also propose the use of more advanced unsupervised algorithms to model the main distribution of the adjacency matrix, determine the number of the retrieved keywords at document level, and score/rank the corresponding candidate keywords. to the best of our knowledge, this is the first work that proposes such modeling of the adjacency matrix using unsupervised machine learning approaches in the context of keyword extraction. our empirical study confirms the efficacy of the distribution-based modeling approaches (regarding the f 1 score) on six datasets (full-texts of scientific publications, paper abstracts and news articles) compared to state-of-the-art graph-based methods. the main contributions of this work are as follows: (a) we propose multiple ways for the distribution-based modeling of the adjacency matrix that corresponds to various versions of the graph-of-words for a target document. specifically, we investigate the use of unsupervised (learning) algorithms to model the distribution of the adjacency matrix, score the candidate words, and (b) discover the appropriate number of keywords (section 3). (c) our empirical study provides strong evidence about the relationship between the graphbased techniques and the proposed ones emphasizing the cases that the distribution-based modeling is more promising (section 4). finally, section 2 and 5 present related work on unsupervised ake (issues/trends) as well as conclusions and future directions of our work, respectively. then, biswas et al. (2018) and vega-oliveros et al. (2019) proposed graph-based keyword extraction methods that combine multiple centrality measures. another important issue is choosing the right number of keywords for a document. @xcite apply the concept of k-core on the graph-of-words of a document retaining only the nodes from the main core as keywords. their method is parameter-free as the k-core principle adjusts the number of keywords concerning each graph's structure. later, @xcite show that retaining only the main core (or truss @xcite ) is suboptimal as the complete set of a document's gold keywords cannot appear within a single subgraph and propose alternative heuristics (stopping criteria) to remove undesired words. trends. information coming from word embeddings @xcite proved useful for the ake task. numerous ake methods use word embeddings @xcite @xcite as an (external) semantic knowledge source. representative graphbased approaches are the one of @xcite and key2vec @xcite that incorporate semantic information from pre-trained distributed word representations and word embeddings trained on a domain-specific corpus, respectively. both methods utilize the information from word embeddings through the usual way of graph-based ranking without giving to the vector representation of terms the primary role. on the contrary, papagiannopoulou and tsoumakas (2018) present the reference vector algorithm (rva) that uses local glove @xcite word vectors (i.e., trained only on the target document). em-bedrank @xcite uses pretrained sentence embeddings, sent2vec @xcite , to embed both candidate terms and documents in the same high-dimensional vector space. finally, @xcite proposed an unsupervised ake method that uses the weighted adjacency matrix's rows as word vectors to model their distribution. the authors show that the centre of the distribution is closer to the non-keywords, as the main bulk of words are neutral or slightly relevant to the documents' topics. ## conclusions and future work in this article, we address the ake task via the distribution-based modeling of the adjacency matrix that corresponds to various versions of the graph-of-words for a target document. more specifically, we propose capitalizing on unsupervised learning algorithms for the distribution-based modeling and scoring of the candidate words. based on our performance evaluation, the if approach shows the best effectiveness results in almost all datasets, concerning the f 1 score determining the number of keywords in document level. there are many interesting future research directions, such as i) improving the scoring functions of the unsupervised learning approaches used in the context of the keyword extraction task, ii) adapting the proposed approach to the keyphrase extraction task , iii) developing novel distributionbased modeling methods that simultaneously utilize the information from one/multiple adjacency matrices , and iv) applying the adjacency matrix's distribution-based modeling in other tasks where only graph-based methods are used to date.