M assive S umm: a very large-scale, very multilingual, news summarisation dataset (2021)

Abstract: Current research in automatic summarisation is unapologetically anglo-centered–a persistent state-of-affairs, which also predates neural net approaches. High-quality automatic summarisation datasets are notoriously expensive to create, posing a challenge for any language. However, with digitalisation, archiving, and social media advertising of newswire articles, recent work has shown how, with careful methodology application, large-scale datasets can now be simply gathered instead of written. In this paper, we present a large-scale multilingual summarisation dataset containing articles in 92 languages, spread across 28.8 million articles, in more than 35 writing scripts. This is both the largest, most inclusive, existing automatic summarisation dataset, as well as one of the largest, most inclusive, ever published datasets for any NLP task. We present the first investigation on the efficacy of resource building from news platforms in the low-resource language setting. Finally, we provide some first insight on how low-resource language settings impact state-of-the-art automatic summarisation system performance.

## introduction automatic summarisation datasets are generally expensive to create, because they generally involve a human reading a document several times and then crafting a fluent piece of text that captures both the important information of the document and the intention of the resulting summary. each datapoint in such a dataset could take hours to manually create. with digitalisation, archiving, and social media advertising of newswire articles, recent work has shown how, with dedicated time and methodology application, large-scale datasets can now be simply gathered instead of written @xcite . but the method development was carried out over english, and until the research presented here, the method has only been applied to a very limited number of relatively richly-resourced languages @xcite . we have extended the methodology further (section 3) and applied it carefully and widely to generate massivesumm: a very large-scale, very multilingual summarisation dataset of 28.8 million articles, containing data in 92 languages, using more than 35 writing scripts. this is by far both the largest, most inclusive, existing automatic summarisation dataset, as well as one of the largest, most inclusive, ever published datasets for any nlp task. the bulk of this paper outlines the size, diversity and inclusivity of the dataset as an automatic summarisation dataset, as well as simply raw text data in comparison with two other multilingual large-scale widely used datasets in nlp: wikipedia and common crawl (section 4). in light of extending and applying the data acquisition method under the low-resource setting, we identify some unreasonable conditions for language inclusion in automatic summarisation research, which stand to perpetuate a lack of language diversity in system development and therefore unequal access to these tools. we also present some experimental evidence that failure to include a more diverse set of language data in automatic summarisation research can result in only very language specific system design when language agnostic design has been claimed (section 5). ## related work a number of works presenting large-scale datasets for automatic summarisation have been presented in the past couple of years. we survey this work here to provide some research context for mas-sivesumm. the new york times corpus (nyt) consists of 1.8 million articles from the new york times @xcite between 1987 and 2007. the automatic summarisation portion of this dataset consists of 650,000 article-summary pairs, where the summaries are written by library scientists. unlike the rest of the datasets discussed in this section, nyt is created and maintained by the platform that the articles belong to. the cnn/daily mail (cnndm) dataset @xcite is an english language automatically acquired question answering dataset composed of newswire articles and their corresponding highlights from two separate platforms: cnn.com and dailymail.co.uk. the dataset was later converted into a summarisation dataset by concatenating these article highlights into article summaries @xcite . the summarisation dataset consists of 312,000 summary-article pairs. it has become the most broadly used automatically collected english summarisation dataset. with the same methodology as cnndm, @xcite collected the xsum dataset of approximately 230,000 summary-article pairs from the bbc.com news platform. @xcite collected the mlsum dataset for five languages from five corresponding news platforms: french, german, spanish, russian, turkish, catering their platform dependent method to each separate news platform. the resulting datset contains a total of around 1.5 million article-summary language pairs. mlsum was the first large-scale multilingual dataset, but all five of the languages of the dataset were still european, indo-european, and relatively high-resourced within nlp. we note that while, similarly to xsum, massivesumm also contains article-summary pairs from the bbc.com platform, there are two important differences which make for zero overlap between the two datasets: (1) we include no english datapoints in our dataset, and (2) our summaries are not article highlights, but social media article descriptions, as is done for the remaining newswire datasets surveyed here. the newsroom dataset @xcite ) is the first large-scale english dataset generated specifically for automatic summarisation. the key insight into automatically creating this dataset was in observing use of a social media standard, called open graph 1 , by publishers to improve their search engine results. according to this standard, a description of the article contents, used for advertising on social media, should be recorded in the mark-up of the article's web page. the method 1 https://ogp.me/ allowed for scraping news articles from any news outlet, so long as the news outlet upheld the social media standard. hence, by contrast to the method for acquiring the cnndm, newsroom's method was website agnostic, which meant that scraping was no longer constrained to collecting data from specific platforms. @xcite created newsroom by conducting a scrape of news articles from 38 english language news outlets spanning two decades starting from the late 1990s, when news platforms first began digitalising their content widely, to 2017. the dataset contains 1.3 million document summary pairs. varab and schluter (2020) extend, streamline and improve the newsroom methodology to assemble the first automatic summarisation dataset for danish, danewsroom. their work comprises the first non-english website agnostic approach to large-scale article-summary collection, across 19 danish news platforms and resulting in a dataset of 1.1m article-summary pairs. the methodology of this paper is adapted from this extension of the newsroom methodology. related to this, the globalvoices dataset (nguyen and daumé iii, 2019), is an automatic summarisation dataset across 15 languages from one single platform, https:// globalvoices.org . although its original collection is similar to newsroom and danewsroom, the resulting dataset is relatively small with less than 30,000 article-summary pairs across all languages in total, including english. moreover, approximately 800 english summaries are further crowdsourced. the dataset contains purely parallel data and its intended use is for cross-lingual summarisation. massivesumm most likely includes all non-english datapoints scraped for globalvoices, as this was one of the hundreds of its news platform data sources. two further large-scale datasets are not based on newswire. (1) bigpatent @xcite consists of 1.3 million u.s. patent english language abstract-document pairs, written between 1971 and 2018, across nine technological areas, all from the google patents public datasets @xcite . contemporaneously to our work, @xcite developed xl-sum, a summarisation dataset from the bbc news platform. however, their work covers less than a twelfth of the articlesummary pairs: around 1 million across 44 languages and a single news platform, compared with our 12.3 million across 92 languages and 370 news platforms. ## methodology our methodology consists of roughly three parts: (1) manual annotation, (2) automatic collection, and (3) quality control. the first part is unique to the dataset presented here and represents a workintensive annotation process which seeks to ensure both breadth in terms of language inclusivity, quality and consistency of the data. the remaining parts are measured adjustments of the prior extensions of @xcite 's methodology by @xcite . ## reflections on low-resource language automatic summarisation the central datasets for automatic summarisation have consistently been for english. in this section we consider how this focus on english has resulted in limited dataset curation methodology development (section 5.1) and limited automatic summarisation system design (section 5.2). ## concluding remarks in this paper, we presented the most large-scale, most language and linguistically diverse and inclusive dataset for automatic summarisation to date: massivesumm. in acquiring massivesumm, we also acquired one of the most diverse and inclusive sources of raw linguistic data to date. we also provided evidence how focus on anglo-centric data acquisition method development and system development were detrimental to both language inclusion and language agnostic system behaviour.