I ce S um: An I celandic Text Summarization Corpus (2021)

Abstract: Automatic Text Summarization (ATS) is the task of generating concise and fluent summaries from one or more documents. In this paper, we present IceSum, the first Icelandic corpus annotated with human-generated summaries. IceSum consists of 1,000 online news articles and their extractive summaries. We train and evaluate several neural network-based models on this dataset, comparing them against a selection of baseline methods. We find that an encoder-decoder model with a sequence-to-sequence based extractor obtains the best results, outperforming all baseline methods. Furthermore, we evaluate how the size of the training corpus affects the quality of the generated summaries. We release the corpus and the models with an open license.

## introduction due to the increasing number of articles being published online every day, there is a growing need for robust automatic text summarization (ats) systems, which provide readers with a concise and fluent summary of their contents. ats systems are often divided into two main types @xcite . first, based on the number of source documents used to generate a given summary, i.e. either single-document or multi-document summarization. in single document summarization, a single document is used for generating the summary, whereas in multidocument summarization many documents are used as the source for the generated summary. the second type is based on the method used to generate the individual sentences in the summary, i.e. either extractive or abstractive summarization. extractive summaries typically consist of sentencelevel excerpts from the source document(s), and therefore tend to be grammatically correct and fluent. in contrast, abstractive summaries may contain words, phrases and sentences that do not occur in the original text. these summaries may also introduce grammatical errors and contain statements that are inconsistent with the source text. research on ats for icelandic has been limited to the evaluation of simple statistical methods @xcite (described in section 2). furthermore, to our best knowledge, no ats system is currently in use in companies or institutions in iceland. in this paper, we present icesum, a corpus of 1,000 icelandic news articles that can be used to train and evaluate icelandic ats systems. we continue previous work on summarizing icelandic text by evaluating more recently proposed methods for extractive summarization, using neural network-based encoder-decoder models and pretrained language models. we benchmark several single-document ats models on this dataset and compare them against previously published methods. the best performing model obtains a rouge-2 foot_0 @xcite recall score of 71.06. this is the first ats model for icelandic which obtains a better result than the lead baseline method (described in section 4), which obtains a score of 69.14. lemmatization is often employed as a preprocessing step for nlp tasks in icelandic, as it dramatically reduces the size of the vocabulary. although it has been shown to be beneficial for tasks such as named entity recognition @xcite , information extraction @xcite and machine translation @xcite , previous experiments with non-neural network-based models failed to show any improvement for extractive text summarization. we find that the same holds true for neural network-based models. finally, we examine the relationship between the size of the training set and the quality of the generated summaries and find that increasing the size of the corpus would likely lead to significantly better results. we release the corpus foot_1 and the models foot_2 with an open license. the rest of this paper is structured as follows. we discuss related work in section 2 and present the summarization corpus in section 3. the methods are presented in section 4 and the experimental setup in section 5. we present and discuss the evaluation results in section 6, and, finally, we conclude in section 7. ## related work a standard approach to extractive summarization involves allocating a score to each sentence, taking into account certain features, and selecting the most important sentences according to this score. many different approaches have been proposed for this task, including statistical-based methods such as tf-idf @xcite and graphbased methods such as textrank @xcite . other methods include supervised machine learning approaches like support vector machines @xcite , hidden markov models @xcite , conditional random fields @xcite , and genetic algorithms @xcite . these approaches obtain better results than purely statistical or graph-based methods @xcite , but often require some feature engineering or rely on additional language resources, such as wordnet-like databases @xcite , which may not be available for many low or mediumresource languages. the use of neural network-based methods has become commonplace in ats in recent years. one of the advantages of these methods is that the features are normally inferred automatically as opposed to being learnt with the help of hand-crafted feature templates as in feature-engineered systems. @xcite proposed a neural networkbased encoder-decoder model for extractive summarization. in their model, a convolutional neural network (cnn) encoder is used to generate sentence representations which are fed to a recurrent neural network (rnn) encoder that chooses which sentences to extract for the summary. this approach has been improved upon by nallapati ## the corpus our summarization corpus, icesum, consists of 1,000 news articles from mbl.is, an icelandic news site. this corpus is similar in size to manually annotated datasets for other languages, such as the duc-2001 and duc-2002 single document summarization datasets which contain 607 and 657 news texts, respectively @xcite . the goal was originally to assemble around 600 news articles, using the duc-2001 dataset as a model. the summaries were generated by two annotators who are native speakers with a background in general linguistics and icelandic literature. ultimately, the total number of summarized news articles was 1,000, as mentioned above. the articles were split evenly among the two annotators, with each generating a single summary for 500 articles. the articles in icesum span a period of 22 years, published between 1998 and 2019, and the dataset was weighted towards more recent articles. it consists of four news categories: local (50%), world (26%), business (14%) and sports news (10%). the summaries, generated by the two annotators, are extractive, consisting of full sentences or independent clauses from the source text. the majority of the summaries consist of full sentences, i.e. unaltered strings, ending in a full stop. the sentences were carefully selected based on their informative value. the sentences extracted from the text more often than not contained nouns, especially proper nouns, that were of high importance for the context of the summary. if the agent of a sentence was a pronoun, the referent had to be included in earlier sentences in the summary. in this manner, the summaries always needed to be considered as a whole, rather than a series of sentences, functioning as independent entities. in the case of exceptionally long sentences, independent clauses were extracted from the sentence. in these cases, clauses were cut off right before or after a conjunction, so that the extracted clause would make an independent grammatical sentence in a summary. the original goal was to compose summaries of 3-6 sentences for each article. moreover, each summary was meant to contain no more than 50% of the original word count of the article itself. in the case of exceptionally long articles, the total number exceeded the original limit of 6 sentences, resulting in the upper limit of 8 sentences. this resulted in an average of 102 words per summary where the average length of the full articles was 302 words, or roughly three times longer. ## methods we evaluated two types of models. first, three nonmachine learning based models, which we refer to as the baseline models. second, several neural network-based encoder-decoder models. ## experimental setup we used 70% of the corpus for training, 15% for validation and 15% for testing. each set consists of articles from the same time range and contains approximately the same proportion of news categories. for models trained using the nnsum library, we use an averaging encoder, which obtains sentence representations by averaging out word embeddings. we used pre-trained glove embeddings @xcite with 300 dimensions, trained on the icelandic gigaword corpus (igc) @xcite , which contains approximately 1.5 billion tokens. all models are trained for 50 epochs, and we report results obtained on the test set for the model that achieved the highest rouge-2 recall score on the validation set. the transformer model was trained using ice-bert @xcite , which is finetuned using the transformersum library. we use a linear classifier and train for 5 epochs. default settings were used for all experiments, unless otherwise noted. like @xcite , we continue adding sentences to our summary until it contains at least 100 words, and truncate summaries to 100 words when computing rouge scores. we also investigated whether lemmatizing the text improves the quality of the summaries. for lemmatization, we first used abltagger @xcite to assign part-of-speech tags to each token and then nefnir @xcite to lemmatize the text. the tokenizer foot_5 library was used to tokenize the source text. for models trained using nnsum, we use glove embeddings trained on a lemmatized version of the igc. the summarization methods we evaluated extract full sentences from a single document. during training, input sentences are labelled as 1 if they should be extracted and 0 otherwise. as the sentences in the gold summary contain both independent clauses and full sentences, we generated a sentence-level oracle summary for each document, using the same algorithm as @xcite . for a given document, we greedily select the sentences which result in the highest possible rouge-2 score against the gold summary, which are then used to label the training data. we report rouge recall scores, calculated without stemming. ## results the results of the evaluation are summarized in for the first time, we have demonstrated an ats system for icelandic that outperforms baseline methods. although transformer-based models have obtained state-of-the-art performance for extractive summarization @xcite zhong et al., 2020), the transformersum model does not outperform the seq2seq or summarun-ner models in our experiments. this may be due to lack of hyperparameter tuning or the small size of the training set. as shown in to estimate how the rouge score is affected by the size of the training set, we split it into 7 equally sized portions, each containing the same proportion of news categories. figure 1 shows the rouge-2 recall score for the seq2seq model on the test set for a varying number of articles in the training set. notably, the seq2seq model almost matches the rouge-2 recall score of the lead baseline method 7udlqlqjvhwvl]h 528*(uhfdoo with a training set of only 100 news articles. furthermore, the rouge-2 score is still rising at a steady pace with a training set of 700 articles, suggesting that there may be significant benefits to enlarging the size of the corpus. ## conclusion we presented the first icelandic corpus annotated with human-generated summaries and showed that it can be used to to create an ats system that outperforms baseline methods. we also showed that lemmatizing the source text does not result in improved performance. finally, we evaluated how the size of the training corpus affects the quality of the generated summaries. the corpus and models have been released with an open license. for future work, we intend to experiment further with transformer-based models, performing hyperparameter tuning for a selection of transformer models, such as roberta-base and electra-base. we also plan to experiment with abstractive summarization using a much larger, unannotated corpus of icelandic news articles. we also hope to add more summaries to the icesum corpus in the future, and to examine inter-annotator agreement.