Abstractive Text Summarization for I celandic (2023)

Abstract: In this work, we studied methods for automatic abstractive summarization in a low-resource setting using Icelandic text, which is morphologically rich and has limited data compared to languages such as English. We collected and published the first publicly available abstractive summarization dataset for Icelandic and used it for training and evaluation of our models. We found that using multilingual pre-training in this setting led to improved performance, with the multilingual mT5 model consistently outperforming a similar model pre-trained from scratch on Icelandic text only. Additionally, we explored the use of machine translations for fine-tuning data augmentation and found that fine-tuning on the augmented data followed by fine-tuning on Icelandic data improved the results. This work highlights the importance of both high-quality training data and multilingual pre-training in achieving effective abstractive summarization in low-resource languages.

## introduction the task of automatic text summarization has been gaining interest in recent years due to the increasing amount of available information and the need for well-written summaries that preserve key information while being coherent and flowing naturally. two main approaches to automatic text summarization are extractive and abstractive methods. extractive methods compose the summary out of copies of important sections from the original text, whereas abstractive methods rephrase and shorten the text similar to how a human would @xcite . the rise of transformer models @xcite in natural language processing (nlp) has led to great advances in the field, particularly in abstractive summarization @xcite . however, these models often rely on a large amount of text data and computational resources for pre-training. this raises the question of whether low-resource languages can build advanced nlp models for summarization, given the lack of data. we aim to address this question by studying the use of state-of-the-art transformer models for abstractive summarization of icelandic text. we introduce the first publicly available abstractive summarization dataset for icelandic, rúv radio news (rrn), and use it for training and evaluation of the models. with that approach, we aim to study whether state-of-the-art transformer models can be adapted to perform abstractive summarization in a low-resource setting for icelandic text. in order to support future research on abstractive summarization in icelandic, we are sharing our dataset foot_0 and the fine-tuned model foot_1 with the research community. this work is motivated by the increasing demand for automatic text summarization and the challenges of applying machine learning methods to low-resource languages such as icelandic. the study of nlp in low-resource languages is important for language preservation, and this research contributes to this field by providing a dataset for icelandic and evaluating the performance of state-of-the-art transformer models on it. summarization has been claimed to be challenging in low-resource settings @xcite and the potential solution that we base our work on is to apply transfer learning @xcite and data augmentation techniques @xcite . transformers are widely applied through transfer learning, a technique introduced by @xcite where a model trained on one task is fine-tuned or reused as the starting point for a model on a similar or different task. prior to the transfer, the models are generally trained using self-supervision, which allows the models to leverage a large, diverse corpus of unlabeled text data. for generative models, the pre-training objective often involves masking parts of the input sequence and tasking the model with filling in the gaps, as proposed by @xcite for example. @xcite demonstrated with the t5 model that many nlp problems can be treated as text-to-text tasks, allowing for the pre-training of a single encoder-decoder transformer on a diverse set of tasks. additionally, bart models @xcite have been trained to reconstruct a text document that has been corrupted with an arbitrary noising function and have proved to be very effective at tasks such as summarization. the pega-sus model @xcite uses a pre-training objective that closely resembles the summarization task, resulting in a model that adapts faster when fine-tuned on a small number of examples. pre-training language models through selfsupervised learning has achieved impressive results when applied to abstractive summarization tasks. however, obtaining high-quality summarization outcomes can be difficult when there is a scarcity of data for fine-tuning, a common issue encountered with low-resource languages. to tackle this challenge, researchers have turned to transfer learning and data augmentation techniques, which have proven to be effective in various low-resource natural language processing (nlp) tasks @xcite . prior results on abstractive summarization in a low-resource setting serve as good examples of applying such methods @xcite . transfer learning methods have enabled progress in icelandic nlp tasks, such as translation @xcite , question answering @xcite , and named entity recognition @xcite . however, research on icelandic summarization has predominantly concentrated on extractive approaches @xcite @xcite . multilingual models, like xlm-r @xcite and mt5 @xcite , have exhibited promising results across a wide range of nlp tasks and have been particularly advantageous for icelandic tasks @xcite . ## discussion in this study, we investigated techniques for addressing the challenging task of low-resource abstractive summarization for icelandic. we evaluated several well-known approaches and uncovered limitations as well as potential avenues for future work. the main challenge in our study was the lack of sufficient data for training abstractive summarization models for icelandic. we collected a newsdomain abstractive summarization dataset, rúv radio news (rrn), but acknowledge that it is relatively small and may not generalize well to other domains or summarization settings. the collection and processing of rrn were time-consuming due to the inconsistency in the format of the radio transcripts. to aid in future language resource development, publicly funded organizations, such as we evaluated the performance of two gullfaxi models and mt5 on the rrn dataset for abstractive summarization in a low-resource setting, specifically for the icelandic language. mt5 consistently outperformed the gullfaxi models. however, we also observed that pre-training gullfaxi led to better summarization performance when compared to no pre-training. the performance of the multilingual mt5 model can be attributed to the large corpus of multilingual data, including 2.1 million icelandic pages, used for pre-training. it should further model relevance correctness language gullfaxi base (rrn) 74% 8% 42% gullfaxi base (cnn/dailymail) 46% 8% 6% gullfaxi base (cnn/dailymail + rrn) 64% 4% 10% mt5 base (rrn) 84% 46% 54% mt5 base (cnn/dailymail) 80% 42% 44% mt5 base (cnn/dailymail + rrn) 96% 54% 56% we explored using machine translations to augment data for low-resource nlp tasks, specifically abstractive summarization in icelandic. we fine-tuned models on icelandic translations of two large english summarization datasets, cnn/dailymail and xsum, but found the finetuned model did not perform well on the icelandic summarization task, rúv radio news (rrn) and had more grammar mistakes compared to other models. when reviewing the icelandic translations used for data augmentation there are a few things to note. although most of them are easily comprehensible for a native speaker, they tend to be unnatural, use unusual wording, and have the wrong inflection of words. for that reason, we think that exclusively using translated examples for fine-tuning can sometimes lead to worse output texts. we also explored a two-phase fine-tuning approach where we first fine-tuned on translated data and then on rrn. we observed improvements in rouge metrics but a manual inspection revealed better summaries for the mt5 model but worse summaries for the gullfaxi model when compared to using no augmentation. this difference highlights the limitation of using rouge scores as a metric to measure summarization performance. it further highlights the importance of the quality of training data in low-resource settings, as well as the importance of considering the naturalness and grammatical accuracy of machine translations when using them for data augmentation. we investigated the impact of the number of fine-tuning examples on the performance of a lowresource abstractive summarization task by finetuning gullfaxi base with 0, 10, 100, and 1k examples from the rrn dataset. our results showed that even without fine-tuning, gullfaxi base performed better on some tasks than a randomly initialized model fine-tuned on all the rrn training set examples. as we increased the number of fine-tuning examples, gullfaxi base continued to improve, achieving significantly higher scores than the baseline when using the full rrn dataset. this demonstrates the effectiveness of pre-training in a low-resource setting and highlights the potential value of creating small, domain-specific summarization datasets. however, we also observed that the mt5 model was better able to make use of more fine-tuning examples when exceeding a thousand examples. the study has several limitations, including that all models tend to generate summaries that are inconsistent with the source text, which is a common issue for abstractive summarization models, and limits their practical use @xcite . another limitation is that the pre-training objective may encourage the generation of incorrect statements. to address this, the use of reinforcement learning with human feedback, as demonstrated by the instruct gpt model @xcite can be used. additionally, it is worth noting that most state-of-theart models and breakthrough studies in nlp are primarily focused on english-language solutions, and it is unclear to what extent the choice of language impacts the performance of these models when the training budget and amount of training data are fixed. further research comparing the performance of state-of-the-art models across different languages would be necessary to better understand this issue. lastly, we would like to highlight the potential of including the @xmath0, which was deemed out of scope for this work. our evaluation approach may be perceived as a limitation due to its binary nature. however, we intentionally designed it this way to prioritize objectivity, by being stringent about any errors in the model-generated summaries. that said, there are alternative evaluation approaches that could be explored in future research, such as employing continuous rating scales or more nuanced assessment criteria to better capture the intricacies of summary quality. by investigating these alternatives, we can potentially gain a deeper understanding of the strengths and weaknesses of summarization models. our results on data augmentation show that evaluating abstractive summaries is challenging. in this study, we used rouge scores and human evaluation, but rouge has been known to favor lexical similarity, which may not be suitable for abstractive summaries @xcite , particularly in morphologically rich languages like icelandic. the lower rouge scores of icelandic summaries compared to english language studies may be due to the differences in grammar between the two languages. the human evaluation revealed that the summaries tended to be factually inaccurate and had varying levels of grammatical quality. for future evaluations, it could be beneficial to include human-written summaries for comparison. ## conclusion in this work, we explored methods for automatic abstractive summarization in a low-resource setting, specifically in the icelandic language. we collected and published the first publicly available abstractive summarization dataset for icelandic, and used it to train and evaluate state-ofthe-art models. our findings indicate that multilingual pre-training provides significant benefits for this task, as the multilingual mt5 model consistently outperformed a similar capacity pegasus model pre-trained from scratch on icelandic text only. additionally, we found that using machine translations for data augmentation led to higher rouge scores. however, when evaluated manually, the benefits of data augmentation were not consistently observed across models compared to a scenario where models were solely fine-tuned on the rrn dataset. specifically, data augmentation enhanced the quality of the summaries generated by the mt5 model compared to those produced with rrn fine-tuning alone. in contrast, the gullfaxi model's summaries experienced a decrease in quality due to data augmentation, displaying weaker grammar and a higher level of inconsistency compared to the reference text. for future work, we suggest a further collection of abstractive summarization data for icelandic, as well as studying metrics that may be better suited for this language. we also emphasize the benefits of using pre-trained multilingual models, which we expect to apply to other generative tasks and languages. overall, our study highlights the importance of pre-training and the challenges of evaluating abstractive summarization in low-resource settings. the gap sentences are selected using a heuristic approach based on importance criteria. the ideal outcome is that the sentences containing the key information of the text are chosen from the document, but this is not guaranteed by the method. the importance of a sentence within a document is estimated by calculating the rouge1-f1 score between the gap sentence and the rest of the document. in this study, we calculate this score based on the lemmatized sentences as they are given in the igc, due to the inflected nature of the icelandic language. the highest-performing models from the pe-gasus study were obtained by choosing a gap sentence ratio between 15%-45%, varying by task and model. for this study, we mask 20% of the total number of sentences in the original text document.