DACSA : A large-scale Dataset for Automatic summarization of C atalan and S panish newspaper Articles (2022)

Abstract: The application of supervised methods to automatic summarization requires the availability of adequate corpora consisting of a set of document-summary pairs. As in most Natural Language Processing tasks, the great majority of available datasets for summarization are in English, making it difficult to develop automatic summarization models for other languages. Although Spanish is gradually forming part of some recent summarization corpora, it is not the same for minority languages such as Catalan. In this work, we describe the construction of a corpus of Catalan and Spanish newspapers, the Dataset for Automatic summarization of Catalan and Spanish newspaper Articles (DACSA) corpus. It is a high-quality large-scale corpus that can be used to train summarization models for Catalan and Spanish.We have carried out an analysis of the corpus, both in terms of the style of the summaries and the difficulty of the summarization task. In particular, we have used a set of well-known metrics in the summarization field in order to characterize the corpus. Additionally, for benchmarking purposes, we have evaluated the performances of some extractive and abstractive summarization systems on the DACSA corpus.

## introduction automatic summarization is one of the central problems in natural language processing (nlp). the development of automatic summarization systems is an important issue due to the great amount of information in different formats that is accessible on the web or in other repositories. it is necessary to develop techniques that help us to tackle that huge amount of information. for this reason, there is an increasing interest in the nlp community to develop techniques that allow the users to find, read, understand, or process the documents. in this context, automatic summarization can be an important aid because it provides a condensed version of documents that reduce the time to explore or analyze them. access to large-scale high-quality data is an essential prerequisite for making substantial progress in summarization. the application of supervised methods to automatic summarization, as those based on neural networks, requires the availability of adequate corpora consisting of documentsummary pairs. the construction of large-scale and high-quality corpora for learning neural summarization models is not an easy task. it is necessary a great human effort to generate thousands of manual summaries, or to design new approaches to obtain these summaries in a semiautomatic way. the first important resource for learning corpus-based summarization models was the cnn/dailymail summarization corpus @xcite , originally constructed for the task of passage-based question answering and adapted to the document summarization task. it consists of news stories from cnn and dailymail and contains 312,077 article-summary pairs. afterwards, another english corpus was provided to the research community for summarization purposes, the newsroom corpus @xcite . it consists of 1.3 million article-summary pairs that have been written by the authors and the editors of 38 different major news publications. the corpus was created through a web-scale crawling of over 100 million pages from a set of online publishers by gathering the news and using the summaries provided in the html metadata. the summaries contained in this corpus combine both extractive and abstractive strategies to describe the content of the articles. also in 2018, the xsum corpus @xcite was presented, it is a large scale dataset obtained by harvesting online articles from the british broadcasting corporation (bbc) with onesentence news summary. as in most nlp tasks, the great majority of available datasets for summarization are in english. the lack of this kind of resources for other languages is an encumbrance to modeling that constraints the impact of language technologies on minority language communities. the creation of a largescale indonesian summarization dataset of 215,827 document-summary pairs, has just been published @xcite . recently, some datasets that aim to fill the gap among english and other languages for the automatic summarization task have been proposed: mlsum @xcite , massivesumm @xcite , and xl-summ @xcite . although spanish is the world's second-most spoken native language and is the official language in 21 countries, it has only recently been considered in general domain summarization datasets, as the aforementioned, and in specific domains as in @xcite . the situation is worse for catalan, although it is not an endangered language, it is spoken by 10 million people in spain and other three european countries, it is minority worldwide and is underrepresented or even not considered in summarization corpora. in this work, we describe the construction of a corpus of catalan and spanish newspapers, the dataset for automatic summarization of catalan and spanish newspaper articles (dacsa) corpus. with the aim of building a quality large-scale corpus that could be used to learn automatic summarization neural models for catalan and spanish, we used a strategy inspired by the construction of the newsroom corpus @xcite . we conducted a crawling process on 30 different newspaper websites to extract articles and summaries in a straightforward way. the crawling included from spanish mass media to regional newspapers. in order to obtain the summaries, we took advantage of the highlights and summaries, provided by authors or editors of the articles. to ensure the quality of the dacsa corpus, we perform two subsequent filtering processes on the downloaded articles. the first filter was used to ensure, at least, a minimum length in both the article and the abstract. all the articles or summaries that were considered too short were discarded. obviously, an article or summary too short implies discarding the article-summary pair. the second filter was used to ensure that the summaries were not almost verbatim copies of the first sentences of the articles. to do this, the article-summary pairs in which the overlapping between the summary and the article prefix of the summary length was high were also discarded. this way, we try to avoid a positional bias in the summaries by discarding those samples in which the summary is reduced to select the first sentences of the article. once both filters were applied, we found that some newspaper sources had very few samples, less than 1000 in some cases. to balance the corpus partitions, we decided to remove the sources with few samples from the training, validation, and tests sets. nevertheless, we joined together the samples from those sources to create a special test set, a test set with sources not present in the training process. therefore, the corpus consists of four partitions per language: training, validation, and test sets along with an extra test set. considering all the partitions, the dacsa corpus consists of a set of 725,184 article-summary pairs extracted from 9 different catalan newspaper websites and 2,120,649 articlesummary pairs extracted from 21 different spanish newspaper websites. the dacsa corpus contains articles and summaries about politics, economics, sports, culture and other topics usually addressed in journalistic domains. to our knowledge, the dacsa corpus is the largest summarization dataset for both languages. we have used four well known metrics in the summarization field in order to characterize the corpus. these metrics are: extractive fragment coverage and density @xcite , abstractivityp @xcite , and novel ngrams @xcite . additionally, for benchmarking purposes, we have evaluated the performance of 6 automatic summarization systems on the dacsa corpus. concretely, we have used two unsupervised systems (lead-2 and textrank), an extractive summarization system, shann @xcite , two abstractive summarization systems, mbart @xcite and mt5 @xcite , and one oracle to compute upper bounds of the performance in the dacsa corpus. the dacsa corpus can be requested for research purposes at https://xarrador . dsic.upv.es/resources/dacsa. ## related work the automatic text summarization problem has been addressed in the literature using abstractive, extractive, or mixed approaches. on the one hand, extractive approaches compose summaries by selecting sentences or words directly from the documents @xcite @xcite @xcite @xcite . most of these approaches address a sequential binary sentence classification problem in order to select the most salient sentences of the documents, following different criteria such as negative log likelihood on preselected sentences @xcite @xcite or rouge @xcite rewards in reinforcement learning environments @xcite @xcite . other extractive architectures are based on siamese hierarchical attention networks built in terms of long short term memories and transformer encoders @xcite . these models have been succesfully applied in summarization tasks of spanish newspapers and talk shows @xcite . on the other hand, the abstractive approaches build the summaries by paraphrasing the sentences of the documents @xcite @xcite . the vast majority of existing neural abstractive summarization models are based on encoder-decoder architectures @xcite . finally, there are also mixed approaches that combine extractive and abstractive techniques, performed in a decoupled way or simultaneously inside the models @xcite . due to the recent success of self-supervised learning, the focus of text summarization research has exhibited a gradual shift from extractive techniques to abstractive techniques @xcite @xcite . these kind of objectives allows to pretrain deep architectures (mainly transformers) to learn vast amounts of general linguistic knowledge from large corpora, that can be transferred to downstream tasks by means of finetuning. the most successful model of this type is bert @xcite , that is pretrained with masked language model and next sentence prediction objectives on raw texts from english wikipedia and bookscorpus. based on bert, some architectural improvements have been proposed like roberta @xcite or al-bert @xcite . in some recent works, bert and roberta have been finetuned for extractive summarization @xcite , but, although it boosted the performance of the previous extractive approaches, the pretraining+finetuning philos-ophy has shown to be most effective for abstractive systems. nowadays, the best performing abstractive models are bart @xcite , t5 @xcite and pegasus @xcite , being all of them transformers @xcite pretrained self-supervisedly as denoising sequence to sequence autoencoders. some multilingual variants of these models have been recently proposed, mbart @xcite and mt5 @xcite . both of them were pretrained following a multilingual denoising procedure on large-scale multilingual corpora. on the one hand, the mbart model was pretrained by using a corpus of 25 languages, extracted from the common crawl @xcite ) (cc25). on the other hand, a multilingual variant of the colossal clean crawled corpus @xcite ) was used to pretrain mt5. self-supervised pre-training requires obtaining large amounts of raw data in order to learn good initializations of deep models from denoising objectives. also, the fine-tuning of these architectures in downstream tasks like text summarization implies the availability of adequate corpora consisting of document-summary pairs. as we mention above, the great majority of datasets for summarization are in english: cnn/dailymail, newsroom, xsum @xcite , and so forth. although some multilingual datasets have been recently created, as mlsum, massivesumm, and xl-summ, they do not provide a large enough portion of spanish data and only massivesumm provides a few samples for catalan. it is in this context where we propose to build the dacsa corpus. the most used metrics in the literature to quantify the performance of the models in the summarization task are rouge @xcite and bertscore @xcite ). on the one hand, rouge measures the performance by counting exact matches. on the other hand, bertscore is a more semantic measure which is based on contextual embeddings provided by a bert language model. these metrics are convenient to evaluate the performance, but they do not explicitly measure the abstractivity. measuring the abstractivity of the summaries generated by the models is generally not trivial. in this work, we used a set of metrics as abstractivity indicators to asses the level of abstractivity: extractive fragment coverage and density @xcite , abstractivity p (bommasani and cardie, 2020), and novel n-grams @xcite . additionally, we also used rouge and bertscore to compare the different summarization models. ## building the dacsa corpus the dacsa corpus was collected using a distributed web crawler that captured over 6 million news articles, close to 2 million of articles published in catalan, and more than 4 million written in spanish. the articles were captured from 30 newspapers sources, 9 sources for catalan and 21 sources for spanish. the range of years of publication was between 2010 to 2020. we divided the crawling process into two services. the first service was designed to retrieve the list of articles on the website of the newspapers source; we refer to this service as the urls extractor service. the second one aims to extract the content (article content and summary) of the article; we refer to this service as the content extractor service. the whole crawler was developed with python 3 and javascript (node.js runtime) programming languages. for the configurations (one per source) of the content extractor service, we used css selectors and the library cheerio ( https://cheerio . js.org/). in order to capture the article and summary text, we designed the selectors that captured the visible information that a person would read, avoiding metadata. using visual information instead of metadata is important because we detected that likely the metadata was automatically created by some naive process that could lose information, such as just extracting the first tokens of the article; meanwhile, the visual text is likely complete, readable and coherent. we searched websites of electronic newspapers published in spain, in catalan or spanish languages. to find the addresses of each article, we decided to use the list of news that electronic newspapers usually have on their website. the benefit of using the list of articles provided by these websites, contrary to the common crawling approach of following every link, was that we aimed the articles themselves, and there was no need to identify whether the web page is a news article or other kind of content. thus, from the list of news in that newspapers source, we created two configurations, one for the urls extractor service and another for the content extractor service. we intended dacsa to be a large-scale, high-quality corpus for catalan and spanish. thus, after the massive capture of samples, we defined two requirements that the articles and summaries must satisfy. we first established a threshold in the minimum number of words of the article and the summary, and second, a threshold in the maximum similarity between the summary and the first sentences of the article. on the one hand, we discarded those samples with a short text in the article or the summary. specifically, every sample inside the corpus contains at least 100 words in the article and 10 words in the summary. with this restriction, we ensure that the samples have enough content to generate a summary with a reasonable length. on other hand, we rejected from the corpus those samples in which the summary is generated by simply extracting the first sentences of the article. specifically, we restricted the overlapping between the summary and the starting sentences of the article by using a similarity metric based on the levenshtein distance to quantify the degree of overlapping. the equation (1) presents the definition of this metric. ## dataset after the above processes, the dacsa corpus was built. this corpus provides pairs of news article and its summary from different newspapers for both, the catalan and the spanish languages. regarding the catalan set, there are 725,184 sample pairs from 9 newspapers, regarding the spanish set, the corpus provides 2,120,649 sample pairs from 21 newspapers. the amount of samples by newspapers source is far from being homogeneous. if these distributions would be preserved over the partitions (training, validation, and test set), the models will focus their learning in the predominant newspapers. all the sources excluded were used as a separate test set. this partition allows evaluating the generalization capabilities of the summarization models against unseen newspaper sources. in this work, we refer to the test set with newspapers included in the training set as testi and to the test set that contains newspapers not included in the training set as testni. the statistics of all the sets are shown in tables 1 and 2 . in the appendix a, tables 7 and 8 show the distribution and the average lengths in terms of sentences and words of the articles and their summaries for catalan and spanish sets, detailed by the different newspaper sources. ## analysis of dataset in this section, an analysis of the level of abstractivity of the summaries of the corpus is done. first, the definition of the different measures used in this work is given, and second, we provide the application of these measures to the dacsa corpus. ## summarization models and performance results we evaluate several summarization systems to understand the challenges posed by the dacsa dataset for summarization tasks. we consider both extractive and abstractive models, along with an extractive oracle to show an upper bound of the extractive performance in the corpora. extractive systems: lead-k, textrank (mihalcea and tarau, 2004) and shann @xcite have been evaluated. lead-k is a heuristic that extracts the first k sentences of a text, being especially well suited to summarize newspaper articles. textrank is a graph-based system inspired by pagerank, where nodes represent sentences, and edges measure similarities in terms of shared words. finally, shann is a supervised system based on siamese hierarchical attentional networks. the document sentences are scored using sentencelevel attentions and those with highest scores are extracted to build the summary. as the average number of sentences in the summaries of dacsa is near to two, we extracted two document sentences by using the extractive systems. we built the extractive systems upon code that is available on github @xcite , @xcite . abstractive systems: we considered two representative models with high performance on abstractive summarization, based on encoder-decoder architectures with transformers as backbone: bart and t5. due to there are neither bart nor t5 models pretrained from scratch for the spanish and catalan languages, we finetuned and evaluated their multilingual variants, mbart 1 and mt5 2 . it should be noted that, although both of them considered the spanish language during pretraining, the catalan language is not represented in the case of mbart, as this language is not contained in the cc25 dataset. we built the abstractive systems using the hugginface toolkit @xcite . oracle: we implemented an extractive oracle that aligns each summary sentence with the most similar document sentence using rouge. the aligned document sentences are concatenated to build the oracle summary. in order to evaluate the models, we use rouge and bertscore metrics. rouge-1, rouge-2 and rouge-l are reported to measure lexical overlapping, while bertscore is used to measure semantic similarity. tables 5 and 6 show the performance results of the different models on the catalan and spanish dacsa testi and testni sets in terms of rouge and bertscore metrics. the oracle outperforms the other systems by a large margin. the worse results obtained by the oracle are in the catalan testi, showing that this partition is the most abstractive test partition in the dacsa corpus. generally, extractive systems are worse in the testi than in the testni, which suggests a higher extractivity in testni than in testi. the high results of lead-2, especially in the testni sets, show that there is a positional bias in these sets. ## conclusions languages other than english have a lack of resources for learning models based on deep learning. this is true for endangered languages but it is also true even for those languages that have millions of speakers but are minority worldwide such as catalan. in this work, we describe the construction of a corpus of catalan and spanish newspapers, the dataset for automatic summarization of catalan and spanish newspaper articles (dacsa) corpus. we have included an analysis of the corpus using a set of well-known metrics in the summarization field in order to characterize the corpus. this characterization shows that dacsa provides samples that do not contain a predominance of extractive summaries, and show great diversity regarding their degree of abstractivity. we have also carried out an evaluation of the performance of some extractive and abstractive summarization systems on the dacsa corpus that could be used for benchmarking. to our knowledge, the dacsa corpus is the largest summarization dataset for catalan and spanish languages and is freely available for research purposes.