Abstractive H indi Text Summarization: A Challenge in a Low-Resource Setting (2023)

Abstract: The Internet has led to a surge in text data in Indian languages; hence, text summarization tools have become essential for information retrieval. Due to a lack of data resources, prevailing summarizing systems in Indian languages have been primarily dependent on and derived from English text summarization approaches. Despite Hindi being the most widely spoken language in India, progress in Hindi summarization is being delayed due to the lack of proper labeled datasets. In this preliminary work we address two major challenges in abstractive Hindi text summarization: creating Hindi language summaries and assessing the efficacy of the produced summaries. Since transfer learning (TL) has shown to be effective in low-resource settings, in order to assess the effectiveness of TL-based approach for summarizing Hindi text, we perform a comparative analysis using three encoder-decoder models: attention-based (BASE), multi-level (MED), and TL-based model (RETRAIN). In relation to the second challenge, we introduce the ICE-H evaluation metric based on the ICE metric for assessing English language summaries. The Rouge and ICE-H metrics are used for evaluating the BASE, MED, and RETRAIN models. According to the Rouge results, the RETRAIN model produces slightly better abstracts than the BASE and MED models for 20k and 100k training samples. The ICE-H metric, on the other hand, produces inconclusive results, which may be attributed to the limitations of existing Hindi NLP resources, such as word embeddings and POS taggers.

## introduction automatic text summarization is the process of condensing lengthy text into a concise version that captures and relays important information. the amount of text data created has increased tremendously in the era of the internet, and while statistical approaches were used in earlier summarization methods, deep learning models have emerged as a viable and promising solution. there are two main approaches in text summarization: extractive (ets) and abstractive (ats). to create a summary, ets combines already-written sentences without any alterations while ats involves text generation. essentially, the ats writes its own sentences, similar to how humans summarize text, and is hence preferred over ets @xcite . ats, as it involves text generation, requires massive volumes of data to efficiently train a summarizer. training models for any nlp task is extremely challenging in the low-resource situation, when textual input is either limited, unannotated, inaccessible, or lacking linguistic resources such as lemmatizers, grammar and spelling checkers, part-of-speech taggers, trained word embeddings, etc. with the increase in text data in indian languages due to advent of the internet, text summarization tools for efficiently searching and retrieving information have become imperative. due to a paucity of data resources, prevailing ats systems in indian languages have been primarily dependent on and derived from english text summarization approaches. hindi is, along with english, one of the 22 official languages of india. there are about 615 million native hindi speakers, making it the third most spoken language in the world. a majority of indians use hindi as their main language. the morphologically rich indigenous language is a reservoir of ancient wisdom, composed of folklores, songs, poems, chronicles, etc. @xcite . though hindi is the top-most language used in india, there is a lack of a proper summarization system for hindi text. the prolonged advancements in abstractive hindi summarization are owed to the absence of labeled hindi summarization datasets. even the existing annotated corpora are not sufficiently large enough to train a sequence-to-sequence (seq2seq) @xcite model from scratch. seq2seq models are loaded with massive weights to be optimized during training. when trained on very few data samples, these models fail to optimize the weights and prohibit any learning, whereas, on small to medium-sized corpora, the models often tend to over-fit. apparently, in the scenario when data is scarce for independently training a deep neural network, transfer learning or tl-based approaches have proved to succeed @xcite @xcite @xcite . the primary contributions of this work include: ## results we examine the base, med, and retrain model performances against 5k, 20k, and 100k training samples. for 5k sample training, the models yield incompetent summaries and the rouge scores are computed to be 0.0 for all three aspects, rouge-1, rouge-2, and rouge-l. the rouge scores are observed to show an incline from 5k training samples to 20k training samples ( med: for 20k training samples, the model produces an r1 f1 score of 2.85, r2 f1 score of 0.0, and rl f1 score of 2.85. the results signify a very narrow overlap of words between the original and med-generated summaries, i.e., around 2.8% 1-gram overlap, 0% 2-gram overlap, and 2.8% lcs overlap. for 100k training samples, the model produces an r1 f1 score of 23.5, r2 f1 score of 7.9, and rl f1 score of 22.3. the results signify a significant overlap of words between the original and med-generated summaries, i.e., around 23% 1-gram overlap, 8% 2-gram overlap, and 21% lcs overlap. retrain: for 20k training samples, the retrain model produces an r1 f1 score of 3.32, r2 f1 score of 0.0, and rl f1 score of 3.32. the results signify a very narrow overlap of words between the original summary and retrain-generated summary, i.e., around 3.3% 1-gram overlap, 0% 2-gram overlap, and 3.3% lcs overlap. the retrain model observes around 27% rise in r1 f1 and rl f1 scores compared to the base and med models. for 100k training samples, the model produces an r1 f1 score of 25.5, r2 f1 score of 7.9, and rl f1 score of 23.8. the results signify a significant overlap of words between the original summary and retrain-generated summary, i.e., around 25% 1-gram overlap, 8% 2-gram overlap, and 23% lcs overlap. the model observes around 44% rise in r1 f1 and rl f2 scores compared to the base and med models. ## challenges the low performances relayed by the three models may be a result of one of the following: ## conclusion the primary aim of this preliminary assessment is to conduct experiments to test the effectiveness of three models in the low-resource setting, regular attention-based encoder-decoder (base), multilevel encoder-decoder (med), and tl-based finetuned model (retrain), for performing abstractive hindi text summarization. to test how the size of the training sample effects the output summary, we create randomly sampled instances of sizes 5k, 20k, and 100k from a hindi corpora containing news-headline pairs, for fine-tuning the three models. at 5k, all the models are unable to extract meaningful information from the text and instead produce word repetition sequences that neither make sense nor convey any significant information. as the sample size is increased to 20k, all models start to extract significant words from the text, but the sentences are still poorly constructed. the retrain model, however, is able to predict important words from text but the headline is still incompetent. at 100k, every model begins crafting coherent headlines that convey significant information, where, the retrain strategy produced headlines that were more well-focused and informative. for assessing the quality of the hindi abstracts generated for these random samples, we also propose an evaluation metric (ice-h), the framework of which is borrowed from the ice metric for evaluating english language summaries. we evaluate the respective model's performance using ice-h and report our findings and limitations.