Extending Neural Keyword Extraction with TF - IDF tagset matching (2021)

Abstract: Keyword extraction is the task of identifying words (or multi-word expressions) that best describe a given document and serve in news portals to link articles of similar topics. In this work, we develop and evaluate our methods on four novel data sets covering less-represented, morphologically-rich languages in European news media industry (Croatian, Estonian, Latvian, and Russian). First, we perform evaluation of two supervised neural transformer-based methods, Transformer-based Neural Tagger for Keyword Identification (TNT-KID) and Bidirectional Encoder Representations from Transformers (BERT) with an additional Bidirectional Long Short-Term Memory Conditional Random Fields (BiLSTM CRF) classification head, and compare them to a baseline Term Frequency - Inverse Document Frequency (TF-IDF) based unsupervised approach. Next, we show that by combining the keywords retrieved by both neural transformer-based methods and extending the final set of keywords with an unsupervised TF-IDF based technique, we can drastically improve the recall of the system, making it appropriate for usage as a recommendation system in the media house environment.

## introduction keywords are words (or multi-word expressions) that best describe the subject of a document, effectively summarise it and can also be used in several document categorization tasks. in online news portals, keywords help with efficient retrieval of articles when needed. similar keywords characterise articles of similar topics, which can help editors to link related articles, journalists to find similar articles and readers to retrieve articles of interest when browsing the portals. for journalists manually assigning tags (keywords) to articles represents a demanding task, and high-quality automated keyword extraction shows to be one of components in news digitalization process that many media houses seek for. the task of keyword extraction can generally be tackled in an unsupervised way, i.e., by relying on frequency based statistical measures @xcite or graph statistics @xcite , or with a supervised keyword extraction tool, which requires a training set of sufficient size and from appropriate domain. while supervised methods tend to work better due to their ability to adapt to a specifics of the syntax, semantics, content, genre and keyword assignment regime of a specific text @xcite , their training for some less resource languages is problematic due to scarcity of large manually annotated resources. for this reason, studies about supervised keyword extraction conducted on less resourced languages are still very rare. to overcome this research gap, in this paper we focus on supervised keyword extraction on three less resourced languages, croatian, latvian, and estonian, and one fairly well resourced language (russian) and conduct experiments on data sets of media partners in the embeddia project foot_0 . the code for the experiments is made available on github under the mit license foot_1 . in media house environments, automatic keyword extraction systems are expected to return a diverse list of keyword candidates (of constant length), which is then inspected by a journalist who manually selects appropriate candidates. while the state-of-the-art supervised approaches in most cases offer good enough precision for this type of usage as a recommendation system, the recall of these systems is nevertheless problematic. supervised systems learn how many keywords should be returned for each news article on the gold standard train set, which generally contains only a small amount of manually approved candidates for each news article. for example, among the datasets used in our experiments (see section 3), the russian train set contains the most (on average 4.44) present keywords (i.e., keywords which appear in the text of the article and can be used for training of the supervised models) per article, while the croatian test set contains only 1.19 keywords per article. this means that for croatian, the model will learn to return around 1.19 keywords for each article, which is not enough. to solve this problem we show that we can improve the recall of the existing supervised keyword extraction system by: the rest of this work is structured as follows: section 2 presents the related work, while section 3 describes the datasets on which we evaluate our method. section 4 describes our proposed method with all corresponding steps. the experiment settings are described in section 5 and the evaluation of the proposed methods is shown in section 6. the conclusions and the proposed further work are presented in section 7. ## related work many different approaches have been developed to tackle the problem of extracting keywords. the early approaches, such as kp-miner (el-beltagy and rafea, 2009) and rake @xcite rely on unsupervised techniques which employ frequency based metrics for extraction of keywords from text. formally, aforementioned approaches search for the words w from vocabulary v that maximize a given metric h for a given text t: @xmath0, t). in these approaches, frequency is of high relevance and it is assumed that the more frequent a given word, the more important the meaning this word carries for a given document. most popular such metrics are the naïve frequency (word count) and the term frequency-inverse document frequency (tf-idf) @xcite . most recent state-of-the-art statistical approaches, such as yake @xcite , also employ frequency based features, but combine them with other features such as casing, position, relatedness to context and dispersion of a specific term in order to derive a final score for each keyword candidate. another line of research models this problem by exploiting concepts from graph theory. approaches, such as textrank @xcite , single rank @xcite , topi-crank @xcite and topical pagerank @xcite build a graph g, i.e., a mathematical construct described by a set of vertexes v and a set of edges e connecting two vertices. in one of the most recent approaches called rakun @xcite , a directed graph is constructed from text, where vertexes v and two words w i , w i+1 are linked if they appear following one another. keywords are ranked by a shortest path-based metric from graph theory -the load centrality. the task of keyword extraction can also be tackled in a supervised way. one of the first supervised approaches was an algorithm named kea @xcite , which uses only tf-idf and the term's position in the text as features for term identification. more recent neural approaches to keyword detection consider the problem as a sequence-tosequence generation task @xcite and employ a generative model for keyword prediction with a recurrent encoder-decoder framework and an attention mechanism capable of detecting keywords in the input text sequence whilst also potentially finding keywords that do not appear in the text. finally, the newest branch of models consider keyword extraction as a sequence labelling task and tackle keyword detection with transformers. @xcite fed contextual embeddings generated by several transformer models (bert @xcite , roberta @xcite , gpt-2 @xcite , etc.) into two types of neural architectures, a bidirectional long shortterm memory network (bilstm) and a bilstm network with an additional conditional random fields layer (bilstm-crf). @xcite on the other hand proposed bert-jointkpe that employs a chunking network to identify phrases and a ranking network to learn their salience in the document. by training bert jointly on the chunking and ranking tasks the model manages to establish balance between the estimation of keyphrase quality and salience. another state-of-the-art transformer based approach is tnt-kid (transformer-based neural tagger for keyword identification) @xcite , which does not rely on pretrained language models such as bert, but rather allows the user to train their own language model on the appropriate domain. the study shows that smaller unlabelled domain specific corpora can be successfully used for unsupervised pretraining, which makes the proposed approach easily transferable to low-resource languages. it also proposes several modifications to the transformer architecture in order to adapt it for a keyword extraction task and improve performance of the model. ## data description we conducted experiments on datasets containing news in four languages; latvian, estonian, russian, and croatian. latvian, estonian and russian datasets contain news from the ekspress group, specifically from estonian ekspress meedia (news in estonian and russian) and from latvian delfi (news in latvian and russian). the dataset statistics are presented in ## methodology the recent supervised neural methods are very precise, but, as was already mentioned in section 1, in same cases they do not return a sufficient number of keywords. this is due to the fact that the methods are trained on the training data with a low number of gold standard keywords (as it can be seen from ## experimental settings we conducted experiments on the datasets described in section 3. we evaluate the following methods and combinations of methods: for tnt-kid, which is the only model that requires language model pretraining, language models were trained on train sets in ## evaluation for evaluation, we employ precision, recall and f1 score. while f1@10 and recall@10 are the most relevant metrics for the media partners, we also report precision@10, precision@5, recall@5 and f1@5. only keywords which appear in a text (present keywords) were used as a gold standard, since we only evaluate approaches for keyword tagging that are not capable of finding keywords which do not appear in the text. lowercasing and lemmatization (stemming in the case of latvian) are performed on both the gold standard and the extracted keywords (keyphrases) during the evaluation. the results of the evaluation on all four languages are listed in results suggest, that neural approaches, tnt-kid and bert+bilstm-crf offer comparable performance on all datasets but nevertheless achieve different results for different languages. tnt-kid outperforms bert-bilstm-crf model according to all the evaluation metrics on the estonian and russian news dataset. it also outperforms all other methods in terms of precision and f1 score. on the other hand, bert+bilstm-crf performs better on the croatian dataset in terms of precision and f1-score. on latvian tnt-kid achieves top results in terms of f1, while bert+bilstm-crf offers better precision. even though the tf-idf tagset matching method performs poorly on its own, we can nevertheless drastically improve the recall@5 and the recall@10 of both neural systems, if we expand the keyword tag sets returned by the neural methods with the tf-idf ranked keywords. the improvement is substantial and consistent for all datasets, but it nevertheless comes at the expanse of the lower precision and f1 score. this is not surprising, since the final expanded keyword set always returns 10 keywords, i.e., much more than the average number of present gold standard keywords in the media partner datasets (see combining keywords returned by tnt-kid and bert + bilstm-crf also consistently improves recall, but again at the expanse of lower precision and f1 score. overall, for all four languages, the best performing method in terms of recall is the tnt-kid & bert + bilstm-crf & tf-idf(tm). ## conclusion and future work in this work, we tested two state-of-the-art neural approaches for keyword extraction, tnt-kid @xcite and bert bilstm-crf @xcite , on three less resourced european languages, estonian, latvian, croatian, as well as on russian. we also proposed a tagset based keyword expansion approach, which drastically improves the recall of the method, making it more suitable for the application in the media house environment. our study is one of the very few studies where supervised keyword extraction models were employed on several less resourced languages. the results suggest that these models perform well on languages other than english and could also be successfully leveraged for keyword extraction on morphologically rich languages. the focus of the study was whether we can improve the recall of the supervised models, in order to make them more useful as recommendation systems in the media house environment. our method manages to increase the number of retrieved keywords, which drastically improves the recall for all languages. for example, by combing all neural methods and the tf-idf based approach, we improve on the recall@10 achieved by the best performing neural model, tnt-kid, by 16.81 percentage points for croatian, 9.70 percentage points for estonian, 9.63 percentage points for russian and 17.12 percentage points for latvian. the resulting method nevertheless offers lower precision, which we will try to improve in the future work. in the future we also plan to perform a qualitative evaluation of our methods by journalists from the media houses. next, we plan to explore how adding background knowledge from knowledge databases -lexical (e.g. wordnet @xcite ) or factual (e.g. wikidata @xcite ) would benefit the aforementioned methods. the assumption is that with the linkage of the text representation and the background knowledge we would achieve a more representative understanding of the articles and the concepts appearing in them, which would result in a more successful keyword extraction. in traditional machine-learning setting a common practice of combining different classifier outputs to a single output is referred to as stacking. we propose further research on this topic by testing combinations of various keyword extraction models. finally, we also plan to further improve our unsupervised tf-idf based keyword extraction method. one way to to do this would be to add the notion of positional encoding, since some of the keywords in the news-media domain often can be found at the beginning of the article and the tf-idf(tm) does not take this into account while applying the weighting on the matched terms.