Exploring Linguistically-Lightweight Keyword Extraction Techniques for Indexing News Articles in a Multilingual Set-up (2021)

Abstract: This paper presents a study of state-of-the-art unsupervised and linguistically unsophisticated keyword extraction algorithms, based on statistic-, graph-, and embedding-based approaches, including, i.a., Total Keyword Frequency, TF-IDF, RAKE, KPMiner, YAKE, KeyBERT, and variants of TextRank-based keyword extraction algorithms. The study was motivated by the need to select the most appropriate technique to extract keywords for indexing news articles in a real-world large-scale news analysis engine. The algorithms were evaluated on a corpus of circa 330 news articles in 7 languages. The overall best F1 scores for all languages on average were obtained using a combination of the recently introduced YAKE algorithm and KPMiner (20.1%, 46.6% and 47.2% for exact, partial and fuzzy matching resp.).

## introduction keyword extraction (ke) is the task of automated extraction of single or multiple-token phrases from a textual document that best express all key aspects of its content and can be seen as automated generation of a short document summary. it constitutes an enabling technology for document indexing, clustering, classification, summarization, etc. this paper presents a comparative study of the performance of some state-of-the-art unsupervised linguistically-lightweight keyword extraction methods and combinations thereof applied on news articles in seven languages. the main drive behind the reported work was to explore the usability of these methods for adding another level of indexing of news articles gathered and analysed by the europe media monitor (emm) 1 @xcite , a large-scale multilingual real-time news gathering and analysis system, which processes an average of 300,000 online news articles per day in up to 70 languages and is serving several eu institutions and international organisations. while a vast bulk of research and tools for ke have been reported in the past, the specific focus of our research was to select the most suitable ke methods for indexing news articles taking specifically into account the operational, multilingual and real-time processing character of emm. hence, only unsupervised, scalable vis-a-vis multilinguality and robust algorithms that do not require any sophisticated linguistic resources and are capable of processing single news article in a time-efficient manner were considered. keyword extraction has been the subject of research for decades. both unsupervised and supervised approaches exist, the unsupervised being particularly popular due to the scarcity of annotated data as well as their domain independence. the unsupervised approaches are usually divided in three phases: (a) selection of candidate tokens that can constitute part of a keyword using some heuristics based on statistics and/or certain linguistic features (e.g., belonging to a specific part-ofspeech or not being a stop word, etc.), (b) rank-ing the selected tokens, and (c) generating keywords out of the selected tokens, where the final rank is computed using the scores of the individual tokens. the unsupervised methods are divided into: statistics-, graph-, embeddings-and language model-based ones. the statistics-based methods exploit frequency, positional and co-occurrence statistics in the process of selecting candidate keywords. the graph-based methods create a graph from textual documents with nodes representing the candidate keywords and edges representing some relatedness to other candidate keywords, and then deploy graph ranking algorithms, e.g. pagerank, textrank, to rank the final set of keywords. recently, a third group of methods emerged which are based on word @xcite and sentence embeddings @xcite . linguistic sophistication constitutes another dimension to look at the keyword extraction algorithms. some of the methods use barely any language-specific resources, e.g., only stop word lists, whereas others exploit part-of-speech tagging or even syntactic parsing. the supervised methods are simply divided into shallow and deep learning methods. the shallow methods exploit either binary classifiers to decide whether a token sequence is a keyword, linear regression-based models to rank the candidate keywords, and sequence labelling techniques. the deep learning methods exploit encoder-decoder and sequence-to-sequence labelling approaches. most of the supervised machine-learning approaches reported in the literature deploy more linguistic sophistication (i.e., linguistic features) vis-a-vis unsupervised methods. extensive surveys on keyword extraction methods and comparison of their relative performance are provided in @xcite @xcite . since only a few monolingual corpora with keyword annotation of news articles exist @xcite @xcite that use different approaches to keyword annotation, we have created a new multilingual corpus of circa 330 news articles annotated with keywords covering 7 languages which is used for evaluation purposes in our study. we are not aware of any similar multilingual resource available for research purposes. the paper is organized as follows. first, section 2 introduces the keyword extraction task for news article indexing. section 3 gives an overview of the methods explored. next, section 4 describes the creation of a multi-lingual data set and experiment results. finally, we end up with conclusions and an outlook on future work in section 5. ## keyword extraction task the purpose of ke might vary depending on the domain in which it is deployed. in media monitoring and analysis the main objective is to capture from the text of each news article the main topics discussed therein, the key events reported, the entities involved in these events and what is the outcome, impact and significance thereof. for the sake of specifying what the expected output of ke should be, and in order to guide human annotators tasked to create test datasets, the following constraints on keyword selection were introduced (here in simplified form): ## methods given the specific context of real-time media monitoring, our experiments imposed the following main selection criteria to the keyword extraction techniques to explore and evaluate: consequently, we have selected methods that: (a) do not require any language-specific resources except stop word lists and off-the-shelf pre-computed word embeddings, (b) exploit only information that can be computed in a time-efficient manner, e.g., frequency statistics, co-occurrence, positional information, string similarity, etc., (c) do not require any external text corpora (with one exception for a baseline method). the pool of methods (and variants thereof) explored includes: total keyword frequency (tkf) exploits only frequency information to rank candidate keywords, where candidates are 1-3 word n-grams from text that do not contain punctuation marks, and which neither start nor end with a stop word. term frequency-inverse document frequency (tf-idf) constitutes the main baseline algorithm in our study. for the computation of tf-idf scores a corpus consisting of 34.5m news articles gathered by emm that span over the first 6 months of 2020 and covering ca. 70 languages was exploited. 2 a maximum of min(20, n/6) keywords with highest tf-idf scores are returned for a news article, where n stands for the total number of tokens in the article. ## conclusions and outlook this paper presented the results of a small comparative study of the performance of some stateof-the-art knowledge-lightweight keyword extraction methods in the context of indexing news articles in various languages with keywords. the best performing method, namely, a combination of yake and kpminer algorithms, obtained f 1 score of 20.1%, 46.6% and 47.2% for the exact, partial and fuzzy matching respectively. since both of these algorithms exploit neither any languagespecific (except stop word lists) nor other external resources like domain-specific corpora, this solution can be easily adapted to the processing of many languages and constitutes a strong baseline for further explorations. the comparison presented in this paper is not exhaustive, other linguistically-lightweight unsupervised approaches could be explored, e.g., the graph-centric approach presented in @xcite , and some post-processing filters to merge redundant keywords going beyond exploiting string similarity metrics, and simultaneously, techniques to improve diversification of the keywords returned. extending the approaches explored in this study, e.g., through use of part-of-speech-based patterns to filter out implausible keywords (e.g., imposing constraints to include only adjectives and nouns as elements of keywords), use of more elaborated graph-based keyword ranking methods (e.g. page rank), integration of semantics (e.g., linking semantic meaning to text sequences through using knowledge bases and semantic networks @xcite @xcite ) would potentially allow to improve the performance. however, these extensions would require significantly more linguistic sophistication, and consequently would be more difficult to port across languages. for matters related to accessing the ground truth dataset created for the sake of carrying out the evaluation presented in this paper please contact the authors.