The D anish G igaword Corpus (2021)

Abstract: Danish language technology has been hindered by a lack of broad-coverage corpora at the scale modern NLP prefers. This paper describes the Danish Gigaword Corpus, the result of a focused effort to provide a diverse and freely-available one billion word corpus of Danish text. The Danish Gigaword corpus covers a wide array of time periods, domains, speakers’ socio-economic status, and Danish dialects.

## introduction it is hard to develop good general-purpose language processing tools without a corpus that is broadly representative of the target language. further, developing high-performance deep learning models requires hundreds of millions of tokens @xcite . to address this gap for danish, a north germanic/scandinavian language spoken primarily in denmark, we propose an open giga-word corpus. this corpus is free to download and use, thus enabling researchers and organizations to further develop danish nlp without worrying about licensing fees. the corpus is a first necessary step to allow danish speakers to receive the many benefits of the powerful range of nlp technologies. this paper details the danish gigaword corpus (dagw), a billion-word corpus of language across various dimensions, including modality, time, setting, and place. it is tricky to collect such a corpus automatically: automatic language identification tools confound closely related languages, especially danish and bokmål, and are likely to miss important data @xcite . existing representations underperform for danish: the multilingual fasttext embeddings @xcite miss core danish words such as "traels"; multilingual bert lacks sufficient support for the danish vowel "å". 1 to remedy this situation, we propose a danish gigaword corpus. the overriding goals are to create a dataset that is (1) representative, (2) accessible, and (3) a general-purpose corpus for danish. ## background today's nlp is generally data-intensive, meaning that large representative corpora tend to correlate with better models and better processing results. however, large representative corpora are available for only a small set of languages; there are fewer than ten manually-compiled gigaword-scale corpora, for example, and none for danish. several substantial danish text corpora have been compiled during recent decades. clarin-dk offers a variety of individual corpora of varying genres, annotations, and writing times. however, non-commercial licensing restricts corpus usage. some major danish corpora are related to dictionary production, as is the case for the 56 million words korpus-dk available for search at the dictionary site ordnet.dk. 2 leipzig corpora collection assembles danish corpora from the web, news sites, and wikipedia @xcite . the combined size of these corpora is orders of magnitude smaller than the danish gigaword corpus. by themselves, these corpora do not meet the data size needs of modern language models. modern language models like t5 @xcite and gpt2 @xcite ) are texthungry, making automatic corpora construction attractive. massive, monolithic, automatically collected datasets of web content, such as common crawl, support the training of large language models but suffer from quality issues @xcite and bias @xcite . models trained exclusively with such data quickly delve into generating toxic language @xcite . fur- legal social media conversation web wiki & books news other 29.55 25.01 22.91 9.67 8.82 3.92 0.12 thermore, the danish section of common crawl is plagued by significant amounts of non-danish content, in part due to the pervasive confusion between danish and norwegian bokmål by highly multilingual language id classifiers @xcite . datasets derived exclusively from common crawl also have a bias toward webspeak and content from recent years, leaving models built over them sub-optimally prepared to process older danish. the lack of a large and qualitative danish corpus causes danish nlp tools to lag behind equivalent tools for better-resourced languages, and the gap is increasing @xcite @xcite . the first gigaword corpus was the english gigaword @xcite , consisting of roughly one billion (10 9 ) words of english-language newswire text. the content was single-genre, national and global newswire, published between 1994 and 2002. other gigaword corpora emerged later, for french, arabic, chinese, and spanish. even icelandic, a language with just over 360 000 speakers, has a healthy gigaword project @xcite . ## linguistic diversity for a corpus to be useful for a wide range of applications, it must include a wide range of language, mixing domains, speakers, and styles @xcite . failing to do this can lead to severe deficiencies in the data. for example, when nlp work started on social media text, the wall street journal-trained part of speech taggers missed essential words such as "internet" (due to the articles being from the late eighties and early nineties) and "bake", due to their domain. common crawl's undirected collection of content often over-represents some dialects at the expense of other dialects. geowac @xcite uses demographic information to construct english corpora that balance dialects. unfortunately, a demographic-and web-based approach underrepresents danish dialects such as the endangered bornholmsk dialect @xcite , which is almost absent from the web. these deficiencies do not form a solid basis for general-purpose nlp. so the danish gigaword corpus captures and distributes as broad a range of danish language use as possible, explicitly including language from a variety of settings (long-form writing, novels, social media, speeches, spontaneous speech), domains (news, politics, fiction, health, social media, law, finance), time periods (from the 1700s to present day), registers (formal, informal), and dialects (including, e.g., bornholmsk and sønderjysk). ## dataset construction the danish gigaword corpus consists of sections, with each section corresponding to a single source of text. following prior efforts to construct broadcoverage datasets @xcite , sections are selected based on how well they help the corpus' coverage of danish language use over a variety of dimensions, including: time of authorship; speech situation; modality; domain; register; age of utterer; dialect of utterer; socio-economic status of utterer. this is a strong, intentional departure from editions of english gigaword that focused on newswire. achieving some degree of representativeness @xcite requires the inclusion of sources beyond newswire text. we provide an overview of the danish gigaword corpus's content in figure 1 and detail the sections in the danish gigaword corpus follows the definition of genre used by @xcite , grounded in "situationally defined categories", such as a language style recognized by (or used to define) a community, such as news articles, personal letters, or online chat; a domain as a particular topical focus (or set of foci) that are discussed, such as biomedicine, politics, or gaming; and a medium as the means by which communication is conducted, such as writing, online chat, conversation, and so on. there is a natural overlap between medium and speech situations, but the delineation is beyond this work's scope. while the goal of dagw is to cover a range of genres, domains, and media, it is difficult to measure the prevalence of each of these across all danish users, let alone then gather and redistribute this data. therefore, the goal is to cover something of everything that can be feasibly included, without letting any particularly monolithic combination dominate (in contrast to, e.g., the 100% written newswire content of english gigaword v1 or the 100% common crawl content of geowac). not every intersection between genres, domains, and media can be covered, nor represented proportionally, in the first version of this corpus. ## distribution and sustainability as mentioned earlier in this paper and by @xcite ; @xcite , one problem that plagues danish nlp is a lack of large accessible corpora. to address this and maintain strict licensing standards that permit open and free redistribution, danish gigaword corpus is hosted and freely distributed via https://gigaword.dk/ . alternative downloads will be provided through major dataset distribution services at each significant release. dagw is an intrinsically open project. in a bid to improve and uphold its relevance at a broad level, the current group of participants covers academia, industry, and the public sector. however, the dagw project is also volunteer-led and volunteerdriven, which brings intrinsic risk. aside from cross-sector involvement, the dagw project attempts to mitigate that risk through licensing, distribution, membership, community, and data integrity policies. strategically, the corpus strives for an improved balance. the contents in the first release, with this paper, reflect the data that is available in denmark. data that is legally required to be open and unlicensed dominates the corpus, reflecting the current state of text sharing in denmark. we hope that this will become less conservative over time and particularly look forward to further donations of newswire and literature, so that nlp for danish can start to offer danish speakers improved technology. the data is licensed cc-by and cc0, which gives it broad reach and applicability, and makes it easier for stakeholders to join than copyleft or non-commercial licenses, such as gpl or cc-nc, would. it also improves distribution prospects: because of this licensing choice, dagw can be hosted at a third-party research data repository like zenodo or figshare, shifting the responsibility for data hosting and provision to specialized third parties. the dagw project also maintains an open policy, with any qualified stakeholders welcome to join, especially if there is a compatible donation of data. denmark's size helps keep a manageable community. the danish gigaword also fosters community involvement by publishing results -for example, this paper. finally, a small toolkit is included in the project's github repository for automatic validation of any committed data, ensuring content integrity, quality, and uniformity. ## conclusion and future work in denmark, natural language processing is nascent and growing faster and faster. content restrictions and conservative licensing abound. this paper presents the danish gigaword corpus, a unified effort across many institutions and many danish speakers to construct a billion-word corpus representing the language. it aims to be useful to a maximally broad and diverse group of users. the danish gigaword corpus is an active project. there is continuing effort to add sources that enhance the corpus' breadth, including fiction, older works from the 1800s, and newswire. dagw continues past the first billion words, with data always released under creative commons license and freely distributed via https://gigaword.dk/ . we hope that this concrete and significant contribution benefits anyone working with danish nlp or performing other linguistic activities and encourages others to publish language resources openly. journal of machine learning research, 21(140):1-67. anders søgaard, sebastian ebert, jasmijn bastings, and katja filippova. 2021. we need to talk about random splits. in proceedings of the 16th conference of the european chapter of the association for computational linguistics, kiev, ukraine. association for computational linguistics. steinþór steingrímsson, sigrún helgadóttir, eiríkur rögnvaldsson, starkaður barkarson, and jón guðnason. 2018. risamálheild: a very large icelandic text corpus. in proceedings of the eleventh international conference on language resources and evaluation (lrec 2018), miyazaki, japan. european language resources association (elra). kristian tylén, riccardo fusaroli, pernille smith, and jakob arnoldi. 2016. the social route to abstraction. cognitive science.