Unsupervised Abstractive Summarization of B engali Text Documents (2021)

Abstract: Abstractive summarization systems generally rely on large collections of document-summary pairs. However, the performance of abstractive systems remains a challenge due to the unavailability of the parallel data for low-resource languages like Bengali. To overcome this problem, we propose a graph-based unsupervised abstractive summarization system in the single-document setting for Bengali text documents, which requires only a Part-Of-Speech (POS) tagger and a pre-trained language model trained on Bengali texts. We also provide a human-annotated dataset with document-summary pairs to evaluate our abstractive model and to support the comparison of future abstractive summarization systems of the Bengali Language. We conduct experiments on this dataset and compare our system with several well-established unsupervised extractive summarization systems. Our unsupervised abstractive summarization model outperforms the baselines without being exposed to any human-annotated reference summaries.

## introduction the process of shortening a large text document with the most relevant information of the source is known as automatic text summarization. a good summary should be coherent, non-redundant, and grammatically readable while retaining the original document's most important contents @xcite . there are two types of summarizations: extractive and abstractive. extractive summarization is about ranking important sentences from the original text. the abstractive method generates human-like sentences using natural language generation techniques. traditionally used abstractive techniques are sentence compression, syntactic reorganization, sentence fusion, and lexical paraphrasing @xcite . compared to extractive, abstractive summary generation is indeed a challenging task. a cluster of sentences uses multi-sentence compression (msc) to summarize into one single sentence originally called sentence fusion @xcite . the success of neural sequence-tosequence (seq2seq) models with attention @xcite provides an effective way for text generation which has been extensively applied in the case of abstractive summarization of english language documents @xcite @xcite @xcite . these models are usually trained with lots of gold summaries, but there is no large-scale human-annotated abstractive summaries available for low-resource language like bengali. in contrast, the unsupervised approach reduces the human effort and cost for collecting and annotating large amount of paired training data. therefore, we choose to create an effective bengali text summarizer with an unsupervised approach. the summary of our contributions: ## related works many researchers have worked on text summarization and introduced different extractive and abstractive methods. nevertheless, very few attempts have been made for bengali text summarization despite bangla being the 7 th most spoken language. 3 das and bandyopadhyay (2010) developed bengali opinion based text summarizer using given topic which can determine the information on sentiments of the original texts. @xcite worked on extractive bengali text summarization using pronoun replacement, sentence ranking with term frequency, numerical figures, and overlapping of title words with the document sentences. unfortunately, the methods are limited to extractive summarization, which ranks some important sentences from the document instead of generating new sentences which is challenging for an extremely low resource language like bengali. moreover, there is no human-annotated dataset to compare abstractive summarization methods of this language. @xcite worked on sentence compression (sc) which has received considerable attention in the nlp community. potential utility for extractive text summarization made sc very popular for single or multi-document summarization @xcite . tex-trank @xcite and lexrank @xcite are graph-based methods for extracting important sentences from a document. @xcite ; @xcite showed a first intermediate step towards abstractive summarization, which compresses original sentences for a summary generation. the word-graph based approaches were first proposed by @xcite , which require only a pos tagger and a list of stopwords. @xcite improved filippova's approach by re-ranking the compression paths according to keyphrases, which resulted in more informative sentences. @xcite developed an unsupervised abstractive summarization system that jointly performs sentence fusion and paraphrasing. ## bensumm model we here describe each of the steps involved in our bengali unsupervised abstractive text summarization model (bensumm) for single document setting. our preprocessing step includes tokenization, removal of stopwords, part-of-speech (pos) tagging, and filtering of punctuation marks. we use the nltk foot_2 and bnlp foot_3 to preprocess each sentence and obtain a more accurate representation of the information. ## experiments this section presents our experimental details for assessing the performance of the proposed ben-summ model. ## conclusion and future work in this paper, we have developed an unsupervised abstractive text summarization system for bengali text documents. we have implemented a graphbased model to fuse multiple related sentences, requiring only a pos tagger and a pre-trained language model. experimental results on our proposed dataset demonstrate the superiority of our approach against strong extractive baselines. we design a bengali document summarization tool to provide both extractive and abstractive summary of a given document. one of the limitations of our model is that it cannot generate new words. in the future, we would like to jointly model multi-sentence compression and paraphrasing in our system.