H indi History Note Generation with Unsupervised Extractive Summarization (2020)

Abstract: In this work, the task of extractive single document summarization applied to an education setting to generate summaries of chapters from grade 10 Hindi history textbooks is undertaken. Unsupervised approaches to extract summaries are employed and evaluated. TextRank, LexRank, Luhn and KLSum are used to extract summaries. When evaluated intrinsically, Luhn and TextRank summaries have the highest ROUGE scores. When evaluated extrinsically, the effective measure of a summary in answering exam questions, TextRank summaries performs the best.

## introduction our task is to apply text summarization to generate notes for school students where the medium of instruction is hindi. the motivation for this work is that students studying under the indian central board of secondary education (cbse) have a lack of additional resources given their medium of instruction. online resources are limited, with most reference guide material being published in english. given the vast quantities of information that students are made to memorize, we believe that our tool will help provide students with an outline, a text summary, that could serve as both a big picture introduction and a pre-exam study guide. we focus on this task as each year over 18 million students give the grade ten exam. as hindi is a low resource language, we believe that such a tool could help students learn better. from prior research @xcite on comparative text summarization in english and hindi, we see that summarization results vary drastically for different languages and subject matters. while extensive research has been done for summarization techniques in english @xcite @xcite @xcite , directly applying said methods to hindi text performs poorly @xcite . frequency, graph and feature based approaches have been investigated previously to extract summaries from hindi text and have shown to perform well on news documents @xcite . rule based methods @xcite , and improvements to graph based methods incorporating semantic information from the text @xcite perform well for hindi documents from various domains. we wish to address the task of extractive text summarization in hindi as it applies to learning history in an education setting for school students using unsupervised algorithms. the main reason behind choosing unsupervised methods for this task is that these algorithms do not require a dedicated training set annotated by individuals with subject specific knowledge. secondly, employing a supervised approach for a particular domain constrains the portability of the trained model to be applied on different domains. furthermore, the efficiency or goodness of the generated summaries for a particular task rely on accurate and reliable human annotated summaries used for training. to the best of our knowledge, there exists no work that addresses hindi text summarization in the academic domain as a note generating tool for students. this made it difficult to compare our approaches with existing work that deals with different domains of text data. in this work, we investigate unsupervised graph, term frequency and probability based single document summarization methods. our work will build on previous linguistic analyses (for instance, no direct way to identify proper nouns) in hindi @xcite to deal with the nuances of summarizing history written in hindi @xcite . our code is publicly available on github foot_0 . ## materials we used the grade 10 hindi history textbook (ncert, 2018-2019) prescribed by the cbse and published by the national council of educational research and training (ncert) as the dataset. the textbook is available in pdf format and is about 200 pages in length. there are 8 chapters (articles) in the book. each chapter contains around 400 sentences comprising about 18 words each. this amounts to approximately 7200 words per chapter. to evaluate generated summaries, reference summaries of length 75 sentences are manually created for each chapter using the exact sentences from the textbook (extractive summarization). the annotators drafting the reference summaries are proficient in hindi, have studied history at a high school level and are familiar with the course content and exam structure. in order to perform an extrinsic evaluation, we considered questions from the three most recent exam papers, from 2017-2019, and their corresponding rubrics foot_1 . the exams contain 3 types of questions -very short answer questions (1 mark each), short answer questions (3 marks each) and long answer questions (5 marks each) requiring 1, 3 and 5 sentences from the text respectively. there are a total of 35 questions in these exam papers. a sample very short question from the 2017 examination is as follows: a student's response that would score one full point is as follows: ## methodology the basic idea behind the task of extractive summarization is that individual sentences in the source document are scored and ranked to extract the top n sentences as a summary. in this work, four unsupervised methods are investigated to score and rank the input document sentences. the methods used here are klsum @xcite , luhn summarization @xcite , textrank (mihalcea and tarau, 2004) and lexrank @xcite . the summaries generated by these methods are evaluated and compared against each other intrinsically and extrinsically. textrank and lexrank are graph based approaches. klsum utilizes a probabilistic approach. luhn uses a naive ranking algorithm based on word significance. ## results the machine generated summaries are evaluated using intrinsic and extrinsic measures. intrinsic (quantitative) evaluation uses rouge score @xcite which is a recall based metric that compares similar n-grams in generated summaries against the handmade summaries. it is found that rouge based evaluation correlates with human based evaluation in comparing machine generated summaries with ideal summaries @xcite . hence, we consider rouge-1 and rouge-2 scores for this evaluation, which is the percentage of overlapping unigrams and bigrams respectively between the generated and handmade summaries. the main idea of this work is to create a study/revision guide for students to help them understand the study material and do well on exams. hence, the ability to answer exam questions is an indicator of a good summary. the extrinsic (qualitative) evaluation measures how good the summary is in helping students perform well in the history exam. this is carried out by going through the summaries generated by the above mentioned algorithms and making a decision on how many points can be scored on very short and short answer questions given only the sentences in the summary. the scoring is done manually by human evaluators who refer to the examination grading rubric which is available online. length of the summary is an important factor to be considered when generating summaries. the challenge is to balance recall and precision, i.e. to capture as much important information as possible from the whole document while avoiding the inclusion of superfluous information. such a summary with the right length should make for a faster and better learning experience for students. fig. 1 shows the relationship between the length of summary, in sentences, with respect to the shared unigrams with the reference summary. while longer summaries have more overlap, the decrease in slope indicates that a decreasing percentage of added sentences match the reference. thus, 75 sentences was selected as the model summary length. ## discussion when evaluated extrinsically on question answering ability, we see that human generated reference summaries are able to score better on exam questions when compared to machine generated summaries. among the unsupervised approaches, tex-trank scores the most on exam questions, 53.1%. since textrank is able to answer approximately 80% of exam questions that the reference summaries answer, we believe that note generation by textrank provides a good supplementary study tool for students. we observed the impact of hindi on the rouge metric. the presence of stop words, ambiguous pronouns and other commonly used connecting terms in hindi artificially raise the n-gram overlap without adding useful information. for example, consider the two sentences below: the two sentences have completely different meanings, sharing only subject, gandhi in common. the english sentences have two unigrams in common, 'gandhi' and 'the' out of a total of thirteen unique unigrams, approximately a 15% overlap. on the other hand, the hindi sentences have a total of five unigrams in common out of a total of fourteen unique unigrams, approximately a 36% overlap. this aspect of the hindi language, with an abundance of connecting terms, would also raise the rouge metric of sentences which need not convey useful information. we reevaluated the importance of the rouge score for the chosen task. we notice that a good rouge score is not a good indicator of a summary's ability to serve as a study aid. this is evident from the extrinsic evaluation. luhn summarization, which has the highest rouge-1 score (0.74), performs poorly on the question answering task scoring only 38.8%. conversely, klsum having the lowest rouge-1 score (0.39) performs better than luhn summarization extrinsically, obtaining approximately 47%. this relationship between the rouge and exam scores of the summaries can be confirmed by the spearman's rho and kendall's tau coefficients @xcite , which are -0.4 and -0.33 respectively. the negative coefficients indicate a weak correlation between the summary's rouge score and its question answering ability. this shows that, in addition to rouge, it is important to formulate evaluation mechanisms that align with chosen application to evaluate machine generated summaries. we noticed that machine generated summaries have sentences with ambiguous subjects. while the algorithms may identify an important fact, it cannot attribute it to a subject. consider the following sentence: when the machine generated summary contains only the second sentence it is able to answer the question "what caused the school rebellion?" (expelling the girl from school) but cannot identify the subject (school's principal) who carried out the action without the preceding sentence. this is a structure we see often in hindi where one sentence in english corresponds to two in hindi. in the english version of the text, the fact is stated as "the principal, also a colon, expelled her." as the input text documents to the models were not pre-processed, we observed models treating the same entity differently. for instance, the tokens gandhi and gandhi-ji. the addition of an honorific suffix 'ji' results in both terms being treated as different. since the rule of removing the suffix 'ji' applies only to proper nouns, we cannot generalize this as a stemmer rule. we believe that a textrank based summarization tool would prove effective for other subjects whose exam questions test factual knowledge like geography or biology. however, further testing is required before its portability can be validated. also, we believe the project would benefit from an entity recognizer, as a pre-processing step, to solve both ambiguous subjects problems and the ambiguity caused by the honorific suffixes in the summaries. nevertheless, we believe that this project represents a step in the right direction towards providing a note generation tool for students in hindi medium schools.