Back to the Basics: A Quantitative Analysis of Statistical and Graph-Based Term Weighting Schemes for Keyword Extraction (2021)

Abstract: Term weighting schemes are widely used in Natural Language Processing and Information Retrieval. In particular, term weighting is the basis for keyword extraction. However, there are relatively few evaluation studies that shed light about the strengths and shortcomings of each weighting scheme. In fact, in most cases researchers and practitioners resort to the well-known tf-idf as default, despite the existence of other suitable alternatives, including graph-based models. In this paper, we perform an exhaustive and large-scale empirical comparison of both statistical and graph-based term weighting methods in the context of keyword extraction. Our analysis reveals some interesting findings such as the advantages of the less-known lexical specificity with respect to tf-idf, or the qualitative differences between statistical and graph-based methods. Finally, based on our findings we discuss and devise some suggestions for practitioners. Source code to reproduce our experimental results, including a keyword extraction library, are available in the following repository: https://github.com/asahi417/kex

## introduction keyword extraction has been an essential task in many scientific fields as a first step to extract relevant terms from text corpora. despite the simplicity of the task, it still poses practical problems, and often researchers resort to simple but reliable techniques such as tf-idf @xcite . in turn, term weighting schemes such as tf-idf paved the way for developing large-scale information retrieval (ir) systems @xcite . its simple formulation is still widely used nowadays, not only for keyword extraction but also as an important component in ir @xcite and natural language processing (nlp) tasks @xcite . while there exist supervised and neural techniques @xcite @xcite , as well as ensembles of unsupervised methods @xcite that can provide competitive performance, in this paper we go back to the basics and analyze in detail the single components of unsupervised methods for keyword extraction. in fact, it is still common to rely on unsupervised methods for keyword extraction given their versatility and the lack of training sets in specialized domains. in order to fill this gap, in this paper we perform an extensive analysis of single unsupervised keyword extraction techniques in a wide range of settings and datasets. to the best of our knowledge, this is the first large-scale empirical evaluation performed across base statistical and graphical keyword extraction methods. our analysis sheds light on some properties of statistical methods largely unknown. for instance, our experiments show that a statistical weighting scheme based on the hypergeometric distribution such as lexical specificity @xcite can perform at least as well as or better than tf-idf @xcite , while having additional advantages with respect to flexibility and efficiency. as for the graph-based methods, they can be more reliable than statistical methods without being considerably slower in practice. in fact, graph-based methods initialized with tf-idf or lexical specificity performs best overall. ## keyword extraction given a document with m words [w 1 • • • w m ], keyword extraction is a task to find n noun phrases, which can comprehensively represent the document. as each of such phrases consists of contiguous words in the document, the task can be seen as an ordinary ranking problem over all candidate phrases appeared in the document. a typical keyword extraction pipeline is thus implemented as, first, to construct a set of candidate phrases p d for a target document d and, second, to compute importance scores for all of individual words in d. 2 finally, the top-n phrases {y j |@xmath0. . . n} ⊂ p d in terms of the aggregated score are selected as the prediction @xcite . figure 1 shows an overview of the overarching methodology for unsupervised keyword extraction. to compute word-level scores, there are mainly two types of approach: statistical and graph-based. there are also contributions that focus on training supervised models for keyword extraction @xcite . however, due to the absence of large labeled data and domainspecificity, most efforts are still unsupervised, which is the focus of this paper. ## experimental setting in this section, we explain our keyword extraction experimental setting. all our experiments are run on a 16-core ubuntu computer equipped with 3.8ghz i7 core and 64gib memory. 5 datasets. to evaluate the keyword extraction methods, we consider 15 different public datasets in english. 6 each entry in a dataset consists of a source document and a set of gold keyphrases, where the source document is processed through the pipeline described in section 3 and the gold keyphrase set is filtered to include only phrases 5 all the details to reproduce our experiments are available at https://github.com/asahi417/kex 6 all the datasets were fetched from a public data repository for keyword extraction data: https://github.com/ liaad/keywordextractor-datasets : kpcrowd @xcite , inspec @xcite , krapivin2009 @xcite , semeval2017 @xcite , kdd (gollapalli and caragea, 2014), www (gollapalli and caragea, 2014), wiki20 @xcite , pubmed @xcite , @xcite , citeulike180 @xcite , fao30 and fao780 @xcite , guyen2007 (nguyen and @xcite , and @xcite . preprocessing. before running keyword extraction on each dataset, we apply standard text preprocessing operations. the documents are first tokenized into words by segtok foot_3 , a python library for tokenization and sentence splitting. then, each word is stemmed to reduce it to its base form for comparison purpose by porter stemmer from nltk @xcite , a widely used python library for text processing. part-of-speech annotation is carried out using nltk tagger. to select a candidate phrase set p d , following the literature @xcite , we consider contiguous nouns in the document d that form a noun phrase satisfying the regular expression (adjective)*(noun)+. 9 we then filter the candidates with a stopword list taken from the official yake implementation 10 (campos et al., 2020). finally, for the statistical methods and the graph-based methods based on them (i.e., lexrank and tfidfrank), we compute comparison models. as statistical models, we include keyword extraction methods based on tf, tf-idf, and lexical specificity referred as tf, tfidf, and lexspec foot_6 respectively. 12 each model uses its statistics as a score for the individual words and then aggregates them to score the candidate phrases (see section 2.1). we also add a heuristic baseline which takes the first n phrases as its prediction (firstn). as graph-based models, we compare five distinct methods: textrank @xcite , singlerank (wan and xiao, 2008a), posi-tionrank @xcite , singletpr @xcite , and topicrank @xcite . additionally, we propose two extensions of singlerank, which we call tfidfrank and lexrank, where a word distribution computed by tf-idf or lexical specificity is used for p b (•). as implementations of graph operations such as pagerank and word graph construction, we use networkx @xcite , a graph analyzer in python. ## results in this section, we report our main experimental results comparing unsupervised keyword extraction methods. moreover, tf gives a remarkably low accuracy on every metric and the huge gap between tf and tfidf can be interpreted as the improvement given by the normalization provided by the inverse document frequency. however, as we discuss in section 6.1, this idf normalization relies on a corpus partition, which may not be available in all cases. on the other hand, a measure such as lexical specificity only needs overall term frequencies, which may have advantages in practical settings. in the following sections we perform a more in-depth analysis on these results and the global performance of each type model. ## analysis following the main results presented in the previous section, we perform an analysis on different aspects of the evaluation. in particular, we focus on the agreement among methods, overall performance (section 5.1), and the features related to each dataset leading to each method's performace (section 5.2). ## discussion in this section, we provide a focused comparison among the different types of model, highlighting their main differences, advantages and disadvantages. first, we discuss the two main statistical methods analyzed in this paper, namely lexspec and tfidf (section 6.1). then, we analyze graphical methods, and in particular singlerank and topicrank (section 6.2). finally, we provide an overview of the main differences between statistical and graph-based methods (section 6.3). shows the performance metric considered; the central columns illustrate the statistically significant variables that positively affect the performance of each model. the significance of the variables is indicated between parenthesis, according to the following scale: 0 '***' 0.001 '**' 0.01 '*' 0.05. only models having adjr 2 > 0.5 are included, as those are the variables that explain most of the differences in performance between the incumbent models. ## conclusion in this paper, we have presented a large-scale empirical comparison of unsupervised keyword extraction techniques. our study was focused on two types of keyword extraction methods, namely statistical relying on frequency-based features, and graph-based exploiting the inter-connectivity of words in a corpus. our analysis on fifteen diverse keyword extraction datasets revealed various insights with respect to each type of method. in addition to well-known term weighting schemes such as tf-idf, our comparison includes statistical methods such as lexical specificity, which shows better performance than tf-idf while being significantly less used in the literature. we have also explored various types of graph-based methods based on pagerank and on topic models, with varying conclusions with respect to performance and execution time. our extensive evaluation and analysis can serve as a reference for future research to understand in detail the advantages and disadvantages of each approach in different settings, both qualitatively and quantitatively. as future work, we plan to extend this analysis to fathom the extent and characteristics of the interactions of different methods and their complementarity. moreover, we will extend this empirical comparison to other settings where the methods are used as weighting schemes for nlp and ir applications, and for languages other than english. xiaojun wan and jianguo xiao. 2008a. collabrank: towards a collaborative approach to single-document keyphrase extraction. in proceedings of the 22nd international conference on computational linguistics (coling 2008), pages 969-976. xiaojun wan and jianguo xiao. 2008b. single document keyphrase extraction using neighborhood knowledge. in aaai, volume 8, pages 855-860. ian h witten, gordon w paynter, eibe frank, carl gutwin, and craig g nevill-manning. 2005. kea: practical automated keyphrase extraction. in design and usability of digital libraries: case studies in the asia pacific, pages 129-152. igi global. ho chung wu, robert wing pong luk, kam fai wong, and kui lam kwok. 2008. interpreting tf-idf term weights as making relevance decisions. acm transactions on information systems (tois), 26(3):1-37. lee xiong, chuan hu, chenyan xiong, daniel campos, and arnold overwijk. 2019. open domain web keyphrase extraction beyond language modeling. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 5175-5184, hong kong, china. association for computational linguistics.