CUET _ SSTM at the GEM ’24 Summarization Task: Integration of extractive and abstractive method for long text summarization in S wahili language (2024)

Abstract: Swahili, spoken by around 200 million people primarily in Tanzania and Kenya, has been the focus of our research for the GEM Shared Task at INLG’24 on Underrepresented Language Summarization. We have utilized the XLSUM dataset and have manually summarized 1000 texts from a Swahili news classification dataset. To achieve the desired results, we have tested abstractive summarizers (mT5_multilingual_XLSum, t5-small, mBART-50), and an extractive summarizer (based on PageRank algorithm). But our adopted model consists of an integrated extractive-abstractive model combining the Bert Extractive Summarizer with some abstractive summarizers (t5-small, mBART-50). The integrated model overcome the drawbacks of both the extractive and abstractive summarization system and utilizes the benefit from both of it. Extractive summarizer shorten the paragraphs exceeding 512 tokens, ensuring no important information has been lost before applying the abstractive models. The abstractive summarizer use its pretrained knowledge and ensure to generate context based summary.

## introduction in sub-saharan africa, swahili has been regarded as the most spoken language. it has been serving as the national language of tanzania and kenya and is also widely spoken in uganda, rwanda, burundi, the democratic republic of congo, and comoros. it has been the only african language with an estimated 100 million speakers and has played an important role in east and central africa as a lingua franca (at ohio university swahili language). therefore, summarizing tasks in the swahili language is crucial. summarization in the swahili language has faced challenges because of its rich morphology, multiple dialects, and regional variations (as mentioned in @xcite . these variations have been important in understanding the context essential for producing relevant summaries. no study has proposed a swahilispecific monolingual language model with culturally diverse data, mainly due to swahili being a low-resource language(lrl) with limited data availability @xcite . additionally, there has been limited research on swahili summarization. the primary goal of this paper has been to summarize the swahili texts for the generation, evaluation, and metrics (gem) workshop at the international conference on natural language generation @xcite . the dataset used in this workshop has been introduced by davis, 2020 and consists of swahili news classification texts along with their respective classes. in recent years, automatic text summarization has gained popularity for its ability to summarize text efficiently, quickly, and accurately while maintaining context. it has been categorized into two classes: extractive summarizers and abstractive summarizers @xcite . extractive text summarizers (ets) use mathematical calculations to measure sentence similarity. from this sentence similarity, a similarity matrix is formed which is then converted into a graph. in the graph, sentences are nodes and similarity scores are edges. finally, the summary is constructed from the sentences with the top scores. this can be problematic in some cases. if a text covers multiple topics, like sports and politics, the similarity score diminishes as the topic changes, and thus in the summary, both topics may not be present. moreover, the highest-scoring sentences may cause redundancy. an abstractive text summarizer (ats) focuses on the salient concepts in a text. it not only selects key pieces from the text but also presents these key concepts in a new way, thereby eliminating the redundancy problem often encountered with ets. additionally, ats captures the essence of the text even with multiple topics. however, it is more complex than ets and is typically implemented using lstm, seq2seq model. a limitation of ats is that it can only process up to a limited number of tokens as input and any tokens beyond this limit are truncated. as a result, valuable information may be lost. thus, ats is not fully beneficial for summarizing long texts that contain a large number of tokens. a fusion of extractive and abstractive text summarizers can help by utilizing the strengths of both methods. in many texts of our dataset, the number of tokens has exceeded 512. at first, we have implemented an extractive summarizer that reduced the size of the text beyond 512 tokens, keeping all possibly important information. this slightly summarized text has been summarized again by the abstractive summarizer for further refinement. slightly summarized texts containing fewer than 512 tokens have undergone direct processing by the abstractive summarizer without using the extractive method. this method has enabled the summarization of longer texts and has provided coherent and comprehensive summaries. to achieve our goal, we have manually summarized 1000 texts from the provided swahili news classification dataset. next, we have combined the xlsum dataset with our manually prepared summaries. after that, we evaluated three abstractive summarizers (mt5_multilingual_xlsum, t5small, mbart-50), one extractive summarizer (based on the pagerank algorithm), and two integrated extractive-abstractive summarizes. in the integrated system, we have integrated the bert extractive summarizer with some abstractive summarizers(t5-small, mbart-50). during our comparative analysis, we have trained all the systems on the prepared dataset. the integrated extractive-abstractive summarizer system with the "t5-small" model emerged as the most effective, achieving the highest rouge scores. in the final evalua-tion, this system outpaced gpt-3.5 in the automatic evaluation report in @xcite our core contributions in this work include the manual summarization of 1,000 texts from the provided swahili news classification dataset @xcite . also, we have integrated a bert extractive summarizer and an abstractive summarizer to ensure context-based summarization of the larger texts. detailed information on implementation is available in the github repository linked belowhttps://github.com/ samia2001/cuet_sstm_gem24 . ## related work the effort of enabling computers to automatically generate summaries has been practiced extensively, due to its vast applications in the processing of natural languages. previous works on automatic text summarization can be classified into two categories @xcite . they are extractive summarization and abstractive summarization. one of the pristine approaches in extractive summarization has been found in luhn, 1958. they have taken into account the frequency of words and their relative positions to rank the sentences. graph-based algorithms have been used to introduce faster and more scalable extractive summarization approaches. textrank @xcite and pagerank @xcite are two basic and prominent graph-based extractive summarizers. later on, many other graph-based algorithms have been developed based on these two algorithms. such as lexrank @xcite based on pagerank and topi-crank @xcite , positionrank @xcite based on tex-trank. due to the lack of comprehensibility and rationality of extractive summarization approaches (as mentioned in @xcite , abstractive summarization has been introduced. cnn, rnn, lstm-gru and gan-based approaches have been used frequently @xcite . however, the ultimate improvement in summarization has been done by using transformers. t5 @xcite , bart @xcite etc. transformer architectures have been used to summarize texts. they have multilingual versions such as mt5 @xcite and mbart @xcite which enables summarization in the swahili language. long text summarization has been a drawback of abstractive summarization. to overcome this issue, integration of both the extractive and abstraction have been proposed in @xcite . ## data the given dataset @xcite for this shared task contains a total of 23268 texts that have been collected from bbc news swahili 1 and several other tanzanian news websites. we have not used this dataset because it has been prepared for text classification rather than summarization. as a result, they don't contain a summary which is a must for training the system. manually summarizing and training with such a large dataset would have needed a significant amount of time and resources. we have utilized a different dataset xlsum (introduced in @xcite that contains summaries. this dataset includes 7,898 training, 987 development, and 987 test samples. these samples have been merged, resulting in a total of 9,872 samples. we have also used a custom dataset called swas 2 (swahili summarization) which was taken from the dataset provided by davis, 2020. we have taken the first 1,000 texts and generated summaries with gpt-4 ensuring understandability, compactness, grammaticality, coherence, faithfulness, and saliency. swas dataset and xlsum samples together have yielded a total dataset of 10,872 samples which we have used for training and evaluating our system. the merged dataset has been shuffled. after shuffling, 1,000 samples have been split as the validation set, and the remaining 9,872 samples have been used for training. ## system in the gem 2024 shared task, we participated in subtask 1 of the summarization task, which is an unimodal task. the input text document 1 www.bbc.com/swahili 2 github.com/samia2001/cuet_sstm_gem24 and the generated summary both are in the swahili language. ## conclusion swahili summarization is challenging due to limited resources and no dedicated models. we manually summarized 1,000 texts from a swahili news classification dataset and combined them with xlsum's swahili data. using an extractive-abstract method, we applied a bert-based summarizer for length reduction, followed by an abstractive t5small model. our system outperformed gpt-3.5 in r1 and r2 scores and matched its bertscore, but gpt-3.5 outperformed our model in bart-score, particularly with highly condensed summaries.