The CACAPO Dataset: A Multilingual, Multi-Domain Dataset for Neural Pipeline and End-to-End Data-to-Text Generation (2020)

Abstract: This paper describes the CACAPO dataset, built for training both neural pipeline and end-to-end data-to-text language generation systems. The dataset is multilingual (Dutch and English), and contains almost 10,000 sentences from human-written news texts in the sports, weather, stocks, and incidents domain, together with aligned attribute-value paired data. The dataset is unique in that the linguistic variation and indirect ways of expressing data in these texts reflect the challenges of real world NLG tasks.

## introduction the current paper presents the combinations of aligned data-sentences from naturally produced texts (hereafter: cacapo) dataset; a dataset for data-to-text generation (the task of producing adequate, fluent natural language text from nonlinguistic structured data, such as database records, spreadsheets, knowledge graphs, tables, etc., @xcite . the dataset contains sentences from automatically scraped news texts for the sports, weather, stock, and incidents domain in english and dutch, aligned with relevant attributevalue paired data (see figure 1 and appendix a for examples). to our knowledge, this is the first dataset based on 'naturally occurring' humanwritten texts (i.e., texts that were not collected in a task-based setting), that covers various domains, as well as multiple languages. neural natural language generation (nlg) is a promising technique, as neural nlg systems are not bound by any special-purpose mechanisms, and hence are argued to be easily adaptable to other domains and languages @xcite @xcite . yet despite this advantage, it can still be challenging to create a neural nlg system that achieves the same rich and detailed output as a well-designed traditional rule-based pipeline system @xcite @xcite . this is because a large-scale parallel dataset (i.e., a dataset with aligned texts and relevant data) is required for training neural nlg systems, and such datasets are not a common natural occurrence. this limitation is especially persistent in neural pipeline architectures: neural architectures modeled after the 'traditional' pipeline architecture @xcite that sequentially performs tasks related to document planning, sentence planning and linguistic realization @xcite . these architectures require an explicit representation for every intermediate step. the (enriched) webnlg dataset @xcite is presently the only other dataset viable for both end-to-end, as well as neural pipeline architectures. the present paper thus presents a new automatically scraped dataset that can be used for end-toend, as well as neural pipeline architectures. furthermore, it describes a collection process inspired by @xcite , where collection starts with the news reports and attribute-value datapoints are constructed from them, which also enables relatively low-effort extension and adaptation of the current dataset (section 3). characteristics of the dataset are described based on the methodology by perez-beltrachini and @xcite (section 4). finally, a baseline is developed for the dataset using tgen @xcite (section 5). the full dataset is freely available for research purposes upon request, licensed under ausgoal restrictive licence. a 'thin' version of the dataset that contains the annotated data in combination with the urls of the scraped texts and the scraping tools is publicly available via https://github. com/tallchris91/cacapo-dataset , licensed under cc by-nc-sa. ## related work neural data-to-text nlg models have the ability to produce texts without requiring handwritten rules and templates, generating texts in a completely data-driven way. however, neural data-to-text nlg is struggling to overcome two critical bottlenecks, identified by @xcite , that hamper the performance of the models: (1) a data bottleneck, a lack of (high quality, large scale) parallel data-text datasets; and (2) a control bottleneck, which they describe as an inability to control stylistic variation, but can be more broadly described as the inability to systematically control the generation process and the generated output of a neural system. ## statistics we compare the cacapo dataset to the enriched webnlg dataset @xcite , as these datasets are comparable in the sense that both are multilingual, multidomain, and contain explicit intermediate steps that allow for neural pipeline architectures to be employed. however, they are different in the fact that webnlg is constructed using crowdsourcing, while cacapo is constructed using unedited texts 10 https://github.com/proycon/deepfrog scraped from publicly available sources. similar to @xcite we compare the two datasets on size, lexical richness, and sentence complexity. ## baseline system performance tgen, a sequence-to-sequence model using attention @xcite , was used to establish a baseline on the cacapo dataset. 11 the performance of tgen was evaluated on the test data of the cacapo dataset using bleu @xcite , nist @xcite , me-teor @xcite , rouge-l @xcite , cider @xcite and 11 parameters are provided in appendix b. it should be noted that the system is only trained in an end-to-end fashion. bertscore @xcite ( ## conclusion this paper described the cacapo dataset. a multilingual, multi-domain dataset that enables the use of neural pipeline architectures, as well as endto-end architectures. the dataset is comparable in size to the webnlg dataset, and its lexical richness-due to the fact that the texts directly originate from journalistic articles-provides interesting challenges. furthermore, the fact that these texts were derived from 'naturally occurring' texts means that there may be superfluous information, as well as indirect descriptions of the data in the text. this is challenging for nlg systems, as shown by the system performance scores when performing an end-to-end data-to-text task on the dataset using tgen @xcite . however, the dataset closely mirrors real-world scenarios in which companies oftentimes have large amounts of human-written texts that are not purposefully written for nlg applications, accompanied by corresponding data.