N-Grams T ext R ank A Novel Domain Keyword Extraction Technique (2020)

Abstract: The rapid growth of the internet has given us a wealth of information and data spread across the web. However, as the data begins to grow we simultaneously face the grave problem of an Information Explosion . An abundance of data can lead to large scale data management problems as well as the loss of the true meaning of the data. In this paper, we present an advanced domain specific keyword extraction algorithm in order to tackle this problem of paramount importance. Our algorithm is based on a modified version of TextRank algorithm - an algorithm based on PageRank to successfully determine the keywords from a domain specific document. Furthermore, this paper proposes a modification to the traditional TextRank algorithm that takes into account bigrams and trigrams and returns results with an extremely high precision. We observe how the precision and f1-score of this model outperforms other models in many domains and the recall can be easily increased by increasing the number of results without affecting the precision. We also discuss about the future work of extending the same algorithm to Indian languages.

## introduction graph based ranking algorithms have proved to be useful for tasks which involve ranking or ordering. this includes important tasks like citation analysis and ranking webpage results. graph based ranking algorithms are used in many key areas even today. pagerank, an algorithm developed by the founders of google, was the primary algorithm used to rank webpage searches until 2018. the fundamental idea behind any graph based ranking algorithm is make use of global knowledge for making local decisions. to determine the importance of a node in a graph, we recursively look at other nodes to gain more information. more recently, the applications of graph based algorithms have extended to other domains as well, including natural language processing. this includes the use of textrank algorithms for summarisation, word sense disambiguation @xcite and keyword extraction tasks. knowledge extracted from the whole text is considered while making local decisions. in this paper we introduce and evaluate an unsupervised approach for the task of domain terminology extraction. we employ the textrank algorithm for this task with a few modifications. taking into account that domain terms are often multi-worded expressions, we consider bigrams and trigrams as nodes in the graph with suitable additional weight to these nodes. furthermore, terms are filtered based on their pos @xcite tags in order to remove excessive domain-less words. ## pre-processing and data for our study we used a dataset that contained a collection of over 800 domain specific documents. the dataset featured documents from 4 distinct domains namely : bio-chemistry, communication, computer science and law. before passing a document through the model, it was crucial to carry out fundamental preprocessing in order to achieve a high standard of results. we first removed non-essential punctuations and tokenized the the document. in addition to the elementary nltk @xcite /spacy stop word list, we curated an additional specific list of common words that we observed added no meaning to our algorithm. pos tagging @xcite was a critical part of our model and was based on the powerful assumption that if a term is domain specific then it is often a noun or a verb, which we made after analyzing the data meticulously. the addition of a pos tagger gave us a significant increase in the f1-score. ## textrank textrank is a graph-based ranking model(like hits @xcite ) for text processing which can be used in order to find the most relevant keywords in a text. textrank is an algorithm based on pagerank which we will explain briefly. pagerank is an algorithm used for computing a ranking for every web page based on the graph of the web and helps in measuring the relative importance of specific web pages. we can take all web pages to be directed graph @xcite . in this graph, a node is a webpage. if webpage a has the link to web page b, it can be represented as a directed edge from a to b. after we construct the whole graph, we can assign weights for web pages by the following formula. textrank is conceptually the same algorithm as pagerank with the difference being that nodes in the graph are words rather than webpages. in order to find relevant keywords, the textrank algorithm constructs this word graph. the graph is constructed by looking which words follow one another. an edge is set up between two words if they are located within a window of a size of our choice, the link gets a higher weight if these 2 words occur more frequently next to each other in the text. as we can see, preprocessing plays a huge part in the textrank algorithm without which the results would easily be skewed towards common stop words and punctuations. ## our implementation in our model(code can be found here foot_0 ), we began the process by passing a document through our preprocessing pipeline. an input document was first split into sentences on the basis of end of sentence punctuation marks and then further tokenized using the spacy tokenizer. stop words were filtered out using the spacy stop words list along with our extensive custom list of domain-less terminology. furthermore, we use the powerful tool of pos tagging in order to filter out irrelevant words and make the computation of our model much quicker. after extensive research we determined that nouns and verbs contained most of the important domain related terminology. we are now left with tokenized sentences of each document. we further extract unigrams, bigrams and trigrams from these documents and take them as seperate nodes in our textrank graph. we initialize our textrank graph with a window-size of 4, which means that 4 words around every n-gram will be considered eligible to have an edge with the n-gram. after extensive trial and error, we observed that using a larger window size significantly increased the execution time of the model without much improvement in results. in some cases, an edge was added between two totally unrelated nodes due to large window size. inspecting equation will reveal that we need to set a damping factor to assign how much relative importance to give the score calculated by the graph. after multiple runs, we achieved the greatest results with a damping factor of 0.85 which gives great importance to the score churned out by the textrank graph. we further set our convergence threshold at 10e-5 for our termination condition. once built, this graph is then used to calculate weights for each node. the weight of a node essentially represents its contribution to the document. we observed that although multi word domain terms are important to the document and contribute significantly, they are usually less frequent. often they can get replaced by pronoun terms as well. in order to counter this neglection of the n-gram nodes, we introduce a novel weighting system to the traditional textrank algorithm. the weights of bigram and trigram nodes are taken as a parameter to our model and our multiplied to the final score returned by the traditional textrank algorithm. this imporves our results astronomically with many key bigrams and trigrams showing up as a result. these weights are useful in the hands of a domain expert who would be able to determine the right weights for each domain in order to get the best results. across the many domains, our law domain results were of extremely high quality and showed that both high precision and recall can be attained by our model. ## results and evaluation the model was tested on 10 domain specific documents of bio-chemistry, communication,computer science and law. we ran three different runs of the model, each with varying bigram and trigram weights. run 1 featured a bigram weight of '1.8' and trigram weight of '1.5'. run 2 featured a bigram weight of '1.8' and trigram weight of '2.5'. run 3 featured a bigram weight of '1.8' and trigram weight of '2.5' along with lemmatization. the precision we got for each domain is very high. although we attained a much lower recall score, this was mostly due to the fact that we returned only 20 results for each document. increasing the number of terms our model would return would drastically increase recall. we further noticed that recall is often document specific. low recall was observed in computer science and bio-chemistry which both featured much longer documents. we came to a conclusion that adjusting the number of results produced by our model to be a function of the length of the document would be a great idea. across the many domains, our law domain results were of extremely high quality and showed that both high precision and recall can be attained by our model. observing our results for various runs, we can see varied results for each domain. we notice that weights for bigrams and trigrams can be changed according to the domain in order to attain the best results. domain knowledge will help in particular, as a domain specialist can identify whether a specific domain will be having more bigrams or trigrams than unigrams so that there weights can be increased. one particularly interesting thing to note was that the longer papers did much better when they were preprocessed with a lemmatizer. in domains like law and communication where we observed that words with same roots are used in different places differently, we see a negative impact of lemmatisation. ## future work we have evaluated the model for english, but the model can be made to work with pretty much any language with very minor modifications. since we are considering the words as nodes and using a graph based approach to assign weights to these nodes, we are not concerned with what the word means or represents. we're only interested in its node weight. this allows our model to work with pretty much any language, given that tools for preprocessing text in that language are available. restrictions can be made on number of letters for a specific language. furthermore, transliteration can help to improve our scores. transliterating the documents can help the results if a script of a language is unknown to the system. it will help to clearly distinguish between similar looking words. further study of bigrams and trigrams weight in a specific language will also help the model. ideally, the model should be able to learn the factor by which the bigram and trigram node weights need to be bumped. these bumping factors will not be same across different domains or even different documents, since it is not a reasonable assumptions that different documents will have a similar distribution of multi word terms.