Biased T ext R ank: Unsupervised Graph-Based Content Extraction (2020)

Abstract: We introduce Biased TextRank, a graph-based content extraction method inspired by the popular TextRank algorithm that ranks text spans according to their importance for language processing tasks and according to their relevance to an input “focus.” Biased TextRank enables focused content extraction for text by modifying the random restarts in the execution of TextRank. The random restart probabilities are assigned based on the relevance of the graph nodes to the focus of the task. We present two applications of Biased TextRank: focused summarization and explanation extraction, and show that our algorithm leads to improved performance on two different datasets by significant ROUGE-N score margins. Much like its predecessor, Biased TextRank is unsupervised, easy to implement and orders of magnitude faster and lighter than current state-of-the-art Natural Language Processing methods for similar tasks.

## introduction content and information extraction are central to many natural language processing (nlp) tasks, from question answering @xcite ) to text summarization @xcite and beyond. while the state-of-the-art solutions for these tasks mainly rely on training neural network architectures on very large datasets, there have been questions around the sustainability of these solutions and their effects on the environment. as highlighted in work by @xcite , training one large transformer-based model produces approximately four times more co 2 emissions than a car in its lifetime. these considerable negative environmental outcomes call for lighter and less resource-intensive alternative methods. textrank @xcite ) is a light-weight unsupervised graph-based content extraction algorithm that was initially designed for summarization and keyword extraction applications. since its introduction, it has been adapted and used in numerous other applications and settings, including opinion mining @xcite , credibility assessment @xcite and lyrics summarization @xcite , among others. most recently, textrank has been included in the latest release of the popular spacy library. 1 there have been online tutorials and updating studies @xcite that demonstrate textrank's relevance years after its initial release. some of the textrank extensions that have been proposed in recent years rely on the idea of personalized (or topic-sensitive) pagerank @xcite and its successor algorithms. for instance, positionrank @xcite changed the textrank rankings to account for the position of candidate words in the input document, and showed that this position-aware algorithm led to improvements in keyword extraction over textrank and over several other baselines. in this paper, we introduce biased textrank, which relies on document representation models and similarity measures that enable capturing meaning closeness between graph nodes and a target (focus) text. while we demonstrate the usefulness of our approach on two applications -focused summarization and explanation extraction -we believe this approach is generalizable to other applications that require content extraction and/or content ranking. the paper makes the following three main contributions: we also perform an ablation study to show the effects of the textrank damping factor and the similarity threshold parameters, providing insight on how biased textrank parameters should be tuned. ## biased textrank biased textrank builds upon the original textrank algorithm, but changes how random restart probabilities are assigned, therefore giving higher likelihood to the nodes that are more relevant to a certain "focus" of the task. ## experiments we conduct two main experiments to explore the ability of biased textrank to perform focused content extraction. ## ablation study to understand how the algorithm parameters affect biased textrank, we carry out an ablation study where we examine how the damping factor and the similarity threshold affect the rankings produced by biased textrank across tasks. the results of the study are presented in figure 1 . the similarity measure (cosine similarity) and the document embedding model (sbert) in this study are fixed. also, while conducting this experiment, we increase the number of selected summary sentences from 20 to 30 to add more variance to our visualizations. we derive the following observations from the ablation study: (1) the damping factor, within suggested ranges found in the literature (0.8 to 0.9), has very limited effect on biased textrank for focused summarization and explanation extraction. (2) with the exception of the democrat focused summaries, the variance of the similarity threshold does not significantly change the outcome of biased textrank. for the democrat focused summarization experiment, a lower similarity threshold, which translates to a more dense graph representation of the document, yields better results. (3) given these results, we recommend setting the damping factor to 0.85 (or anywhere between 0.8 to 0.9) and the similarity threshold around 0.65 to obtain reasonable results. ## discussion & future work in this paper, we showed that biased textrank is a promising method for focused content extraction. we believe that a written description of the focus of a content-extraction task is an intuitive way of operationalizing a corresponding solution. in our experience using biased textrank, we found that choosing the right bias text to direct the focus of content extraction is key in making biased textrank work. like the explanation extraction task in 4.3, sometimes the bias term comes as an input to the algorithm. however that is not always the case, as the focused summarization experiment (among other tasks) require manual selection of the bias term. in those environments it is important to choose biases that best reflect the intention of the task focus and be aware of short-comings of word embeddings. in our experience in the focused summarization experiment, we initially found our republican and democrat summaries to be more similar than desired. as we probed the summaries, we found that although different in essence, the two chosen bias terms were producing similar embedding vectors and we suspected the vagueness and word overlap in both bias terms were among the causes. after selecting more distinct and clear bias terms for each summary flavor, we observed more distinctions and desired properties in produced summaries. we are interested in studying and quantifying the effect of the bias text on the algorithm and develop a deeper understanding of how one should pick the right bias text for their goals. we evaluate the use of biased textrank in focused sentence extraction for english texts only. however, we believe that biased textrank is language-agnostic in the sense that if we have the proper tools to parse and embed non-english documents, the algorithm will be directly applicable. with recent advances in multilingual contextual embedding technologies like laser (which provides embeddings for more than 100 languages), we think it is possible to immediately apply it to languages other than english. in future work we would like to explore the application of biased textrank beyond sentence extraction. for instance the "term-set expansion" task most recently tackled by @xcite , in which an initial seed set of keywords are expanded by similar keywords found in a corpus, could be another application of our algorithm. the task can be modeled as a keyword extraction task similar to the example found in the original textrank paper and the restart probabilities can be assigned based on node proximity to the initial seed set. ## conclusion in this paper, we introduced biased textrank, an unsupervised graph-based algorithm for directed extraction of content from text. biased textrank is unsupervised, fast and resource-efficient, languageagnostic and easy to implement. we demonstrated its effectiveness on two tasks: focused summarization and explanation extraction. for the first task, we collected a dataset of biased interpretations of the u.s. presidential debates by democrat and republican media and showed, through comparative experiments, that a democrat or republican-focused summary of the debates better captures those interpretations than a generic summary. for the second task, our explanation extraction experiments showed that biased textrank improved over the performance of two unsupervised baselines. in addition, we analyzed the effects of the damping factor and similarity threshold parameters on biased textrank through an ablation study and suggested parameter tuning guidelines for the algorithm. although we only demonstrated biased textrank effectiveness in two content extraction tasks, we believe that it can have a variety of natural language processing applications, similar to how textrank has been used to address numerous tasks. with the satisfactory results of biased textrank, we find the approach to be a promising direction towards more sustainable content extraction solutions. the biased textrank code as well as the focused summarization dataset we compiled are publicly available at https://lit.eecs.umich.edu/downloads.html .