Unsupervised B engali Text Summarization Using Sentence Embedding and Spectral Clustering (2022)

Abstract: Single document extractive text summarization produces a condensed version of a document by extracting salient sentences from the document. Most significant and diverse information can be obtained from a document by breaking it into topical clusters of sentences. The spectral clustering method is useful in text summarization because it does not assume any fixed shape of the clusters, and the number of clusters can automatically be inferred using the Eigen gap method. In our approach, we have used word embedding-based sentence representation and a spectral clustering algorithm to identify various topics covered in a Bengali document and generate an extractive summary by selecting salient sentences from the identified topics. We have compared our developed Bengali summarization system with several baseline extractive summarization systems. The experimental results show that the proposed approach performs better than some baseline Bengali summarization systems it is compared to.

## introduction with the advancement of search engines, we are flooded with information. this information overload problem affects the proficiency of decision-making of humans. instead of time waste, it also affects the capacity of humans. in today's world where each day technology is changing our daily life, the human brain plays an important role there. so it is unworthy to waste the human brain and time in a negative way. having a crux with relevant information from a long document manually is a very tedious task. text summarization is a very useful solution to this information overload problem. text summarization helps to create a condensed version of a document by selecting sentences with pertinent information from the document. text summarization demands well understanding of the document to create the gist. text summarization can be categorized into two types: extractive and abstractive. extractive summarization aims to generate a summary by selecting textual segments or sentences from the document whereas abstractive summaries are generated from the document by introducing new words or phrase which may not be present in the original document. although abstractive summaries are more human-like than extractive summaries, the state-of-the art abstractive summarization approaches are not good enough in producing an abstract from a longer document. many existing abstractive summarization approaches use two step process. in the first step, an extract is generated. and in the next step, an abstract is generated by reformulating the sentences in the extract @xcite . thus the extractive summarization is useful. moreover, bengali is a resource-scarce language and abstractive summarization requires a large amount of language resources which are not available for bengali language. this motivates us to work on bengali extractive summarization. capturing connectivity among sentences of a document is helpful to group similar sentences and create a condensed extract. sentence clustering is an unsupervised method that groups similar sentences and produces clusters. traditional clustering algorithms though widely used earlier have some pitfalls which are overcome using spectral clustering. spectral clustering emphasizes creating more accurate clusters than traditional clustering algorithms as it does not make assumptions about the shape of the cluster. spectral clustering utilizes the connectivity of data points. if two data points appear side by side but are not connected, spectral clustering will not group them together. the main benefits of using spectral clustering in document segmentation are that the clusters produced by this method do not follow any fixed shape. we assume that the clusters representing topics are non-gaussian. we consider that the spectral clustering algorithm is suitable for segmenting a document into multiple topical clusters where a cluster represents a topical segment that consists of semantically similar sentences appearing in close positional proximity. in our approach, each sentence vector is computed by averaging the word embedding vectors obtained using fasttext foot_0 open source. after obtaining the sentence vector by averaging the word vectors, the position of the sentence is included as an additional dimension of the sentence vector. position information is considered to encourage locally coherent sentences to fall in the same cluster. clusters are ranked according to the average position number of the sentences in the cluster and a summary is created by choosing the most relevant sentences from the ranked clusters. sentence selection from the cluster is also done in an effective way. the efficacy of our approach lies in the effectiveness of spectral clustering in segmenting the document into multiple topical clusters. the approach proposed in this paper differs from the existing approaches @xcite @xcite @xcite @xcite . we use the spectral clustering algorithm to segment a document into multiple topical clusters and create a summary by choosing topic-wise most relevant sentences. on the other hand, the existing approaches @xcite @xcite @xcite @xcite decompose the entire document into a collection of sentences and rank the sentences based on some features to create a summary. so, our proposed approach uses an effective clustering-based method that produces a summary covering all important topics in a document. our paper is set up in the following manner. related work is discussed in section 2. our proposed methodology is explained in section 3. section 4 highlights the dataset used in the approach results and comparison among existing models. section 5 concludes the paper. ## related work in the area of extractive text summarization, the early approaches used various heuristic algorithms to identify important segments from a document. the methods that include features like sentence position, word frequency, and key phrases to extract salient sentences from the document have been presented in @xcite @xcite . most early text summarization algorithms faced the redundancy problem or the diversity problem. so, to deal with these problems and assuring good coverage, clustering of sentences is used @xcite . the idea of employing a clustering algorithm for text summarization was well described in @xcite . this approach used three steps for text summarization: histogram-based clustering algorithm for sentence clustering, ordering of clusters, and extraction of summary-worthy sentences from the clusters to create the summary. in @xcite , a hierarchical agglomerative clustering algorithm was used to create clusters of sentences. to create a summary, sentences were chosen in order from largest to smallest cluster. another clustering-based approach presented in @xcite incorporates cluster-level information in a graph model for ranking sentences. however, the early works @xcite suggest that the performance of clustering-based text summarization heavily depends on the quality of clusters produced. clustering algorithms perform well when we have a clear idea regarding attributes of data points @xcite . clustering based on compactness highlights spatial proximity among data points. for example, agglomerative average link clustering @xcite , k-means @xcite , highlights compactness. the resultant clusters using this algorithm is spherical clusters. modification on k-means was discussed in @xcite which is defined as k-means++. though it uses a better centroid initialization technique for improvements over k-means, still it suffers from some drawbacks because we need to specify the number of clusters to be formed in advance and it assumes a fixed shape of clusters. after investigating different existing clustering algorithms, we can find that spectral clustering is more suitable for our task. it embeds sentences on a low-dimensional eigen space and performs clustering on the data points mapped to the lowdimensional embedding space. it does not assume any fixed shape of the cluster but rather emphasizes graph partitioning @xcite based on connectedness among the vertices representing the data points. so, it is very useful when the shape of cluster is non-convex. nowadays, the spectral clustering algorithm has been used in a wide range of application areas like image clustering @xcite , shape clustering @xcite , motion clustering @xcite and many more. gupta et. al. @xcite presented a spectral clustering-based text summarization approach, which uses textual entailment(te) and spectral clustering (atesc) to calculate sentence connectedness scores. it is used to measure the saliency of a sentence in the input. however, to the best of our knowledge, it is our new attempt to use a spectral clustering algorithm in the bengali text summarization domain. for sentence representation, we have also considered sentence position as a new feature and combined it with the semantic content-based sentence features. the spectral clustering is applied to the sentence vectors to produce multiple clusters where each cluster represents a topical segment of the input document. the final summary is generated by choosing sentences from the ordered clusters using a centrality-based saliency measure @xcite . ## our proposed methodology steps of our proposed system is illustrated in figure 1 . each step of the proposed system is discussed in this section. ## datasets and experimental results since no publicly available dataset is available for bengali text summarization, we have tested our proposed approach on our own dataset consisting of 102 bengali document-summary pairs. the average number of sentences in each document is 40. for evaluation, we have taken 100 words from each summary generated by the system using the proposed approach. we have conducted six experiments to prove the effectiveness of our approach. these experiments include implementations of five state-of-the-art unsupervised methods with which our proposed approach is compared. a brief description of the models implemented by us is given below in this section. model 1: this is our proposed model that uses clustering-based document segmentation for text summarization. model 2: this approach was developed by luhn @xcite . it generates a summary from a document by considering that the sentence containing more frequent words is more important than the sentence containing less frequent words. in this method, stop words are removed before sentence weight calculation. model 3: this approach was developed by @xcite . it uses lexrank, which is a graph-based approach that represents sentences as vertices of a graph and considers the cosine similarity between any two sentences as the weight of the edge between the corresponding vertices. finally, google's page rank algorithm is applied to the graph to rank sentences. model 4: this approach was developed by @xcite . the approach is named "sumbasic" which considers term frequency as the saliency of a term. here probability of a word is calculated based on its frequency and each sentence is assigned a score equal to the average probability of the words contained in the sentence. the sentence with the highest score is selected first in the summary. before selecting the next sentence, the words present in the already selected sentence are penalized by multiplying their probability values by themselves, and the sentences are re-ranked using the newly calculated probability values. after re-ranking the sentences, the sentence with the highest score is selected as the second sentence of the summary. this process is continued until the summary of the desired length is obtained. model 5: this model was developed by @xcite . it is called as textrank. it is also a graph-based approach similar to that used in lexrank @xcite . lexrank used tf*idf-based term weight and cosine similarity value as the edge weight whereas textrank used word overlap-based similarity value as the edge weight. model 6: this is the lead baseline model, where a summary is generated by considering the first 100 words of the input document. this is the baseline defined in duc 2001 and duc 2002 shared tasks on single document summarization. ## conclusion this paper describes a spectral clustering-based method for segmenting a document into multiple topical segments and a summarization method that generates an extractive summary by choosing sentences from the clusters. our proposed summarization approach outperforms several existing baseline summarization approaches. the higher rouge score obtained by the proposed approach proves that the spectral clustering algorithm provides more accurate topical segments of a document if the sentence position is added as an additional dimension to the sentence vector obtained by averaging the word vectors. we have a future plan to use more deep semantic methods for document segmentation and incorporate them into the text summarization process.