D a N ewsroom: A Large-scale D anish Summarisation Dataset (2020)

Abstract: Dataset development for automatic summarisation systems is notoriously English-oriented. In this paper we present the first large-scale non-English language dataset specifically curated for automatic summarisation. The document-summary pairs are news articles and manually written summaries in the Danish language. There has previously been no work done to establish a Danish summarisation dataset, nor any published work on the automatic summarisation of Danish. We provide therefore the first automatic summarisation dataset for the Danish language (large-scale or otherwise). To support the comparison of future automatic summarisation systems for Danish, we include system performance on this dataset of strong well-established unsupervised baseline systems, together with an oracle extractive summariser, which is the first account of automatic summarisation system performance for Danish. Finally, we make all code for automatically acquiring the data freely available and make explicit how this technology can easily be adapted in order to acquire automatic summarisation datasets for further languages.

## introduction dataset development for automatic summarisation systems is notoriously english-oriented. this is surprising. on the system user-side, a more feasible access (for example, summaries) to the increasing amounts of digital information informing daily life is of inherent interest to potential users across the globe. at the same time, automatic summarisation provides a challenging nlp test-bed to investigate the limits of deep learning for nlp, and for downstream evaluation of basic core nlp tasks like discourse analysis, co-reference resolution, and other types of parsing @xcite . yet, only very limited datasets exist in languages other than english @xcite . by automatic summarisation dataset we denote a collection of entire documents each paired up with at least one manually written summary; the summaries of such a dataset are intended as a summaries for those documents and not as headlines, or a list of facts or highlights. in fact, until recently central larger-scale automatic summarisation datasets have not included been composed of any summaries. namely, rush et al. @xcite were the first to recast the english gigaword dataset @xcite as a headline-type large-scale summarisation generation dataset. and the cnn/daily mail dataset @xcite , a question answering dataset, was first recast by @xcite as an automatic summarisation dataset. these two datasets have been central to more recent automatic summarisation system development. headline and highlights datasets are are not ideal for the development of summarisation systems, but because of their scale and in the absence of alternatives, they provided a much needed crucial prerequisite for neural system development. the advent of the english language newsroom dataset @xcite -a dataset of 1.3 million english articlesummary pairs that was created by collecting manually writ-ten summaries from news articles provided the first large scale first-class summarisation dataset. to our knowledge, it is also the only existing large-scale automatic summarisation dataset, prior to this paper. with this work, we adopt, extend, and extensively describe an approach to automatically constructing a danish language automatic summarisation dataset. this essentially (1) provides the first danish language automatic summarisation dataset, (2) enables neural system development for danish under a monolingual setting, and establishes (3) the first non-english large-scale automatic summarisation dataset. ## current central datasets for automatic summarisation we now survey the central datasets for automatic summarisation system development and benchmarking. by "central dataset for automatic summarisation", we mean that the dataset (1) is not a specialised type of summarisation exclusive to a particular domain cnn/daily mail. the cnn/daily mail dataset is an automatically generated dataset constructed by crawling cnn.com and dailymail.co.uk. it was originally introduced as a question answering dataset @xcite and comprises articles accompanied by information boxes of a couple of bulleted article highlights. these articles were later converted into a summarisation dataset foot_2 @xcite by considering the bullet points as a description of the article and concatenating the listed facts into a single summary. the summaries have on several accounts been described as being highly extractive @xcite . the dataset contains over 312k articles of mean length 781 words, accompanied by summaries with mean length 56 words. though a useful adaptation, this method for the automatic creation of a new summarisation dataset has two major flaws. first, as we discussed in section 1, bullet-point highlights are not manually written summaries of articles, and are therefore system development over this dataset does not exactly automatic summarisation system development. second, data collection is restricted to news outlets who collect highlights in information boxes within news articles. as such, the data collection strategy doesn't correspond to conventional document structure that is generalised across a wide range of news outlets. newsroom. newsroom @xcite is a largescale dataset created by conducting a scrape of news articles from 38 english language news outlets covering the period 1997-2018. the scrape was enabled by the the internet archive (archive.org), a non-profit organisation which provides a platform for hosting and accessing past published internet content. together with archive.org, this work takes advantage of the use of the semantic web foot_3 and properties of facebook's open graph protocol foot_4 which encouraged online publishers to insert a special metadata summary for each news article. the dataset contains 1.3 million documentsummary pairs, with articles of mean length 659 words and mean summary length 27 words. ## towards a danewsroom we extend the work of newsroom @xcite and use the internet archive foot_5 , a non-profit archiver. specifically, we use the wayback machine foot_6 , a sort of automatic archive system and a product of the internet archive. the wayback machine has automatically and systematically scraped the internet for the past 20 years. as such, the wayback machine provides the history of the web through snapshots and has since collected more than 300 billion websites-all directly accessible through their own online databases. the wayback machine provides an api foot_7 that enables users to query their databases with urls foot_8 . since this historical content is also freely available through the endpoint web.archive.org/web/timestamp/url we are equipped with a method to retrieve web content across time, and in this case, news articles from the past. it is with this procedure that newsroom was collected. though the process of the acquiring urls was not described by @xcite , we provide a reproducible approach with accompanying code for retrieving urls foot_9 here. ## document-summary descriptive measures to explore the quality and extractiveness of summaries with respect to documents, @xcite carried out a series of measurements over extracted fragments: greedy n-gram overlaps between an article body and reference summary: coverage, density and compression. we present the definitions used by grusky et al., and apply these same measures to danewsroom. we then propose using these measures as an automatic tool for identifying high-quality article-summary pairs. let (a, s) be a instance pair of an article @xmath0, a 2 , .., a n ) and a summary @xmath1, s 2 , .., s m ) consisting of tokens a i and s i respectively. and let |a| := n and |s| := m.