G en W iki: A Dataset of 1.3 Million Content-Sharing Text and Graphs for Unsupervised Graph-to-Text Generation (2020)

Abstract: Data collection for the knowledge graph-to-text generation is expensive. As a result, research on unsupervised models has emerged as an active field recently. However, most unsupervised models have to use non-parallel versions of existing small supervised datasets, which largely constrain their potential. In this paper, we propose a large-scale, general-domain dataset, GenWiki. Our unsupervised dataset has 1.3M text and graph examples, respectively. With a human-annotated test set, we provide this new benchmark dataset for future research on unsupervised text generation from knowledge graphs.

## existing data-to-text models and datasets research in nlp can be viewed as a close interplay between models and datasets. traditionally, most methods hand-craft linguistic features to build rule-based systems, so datasets containing hundreds or thousands of samples are adequate. for example, in data-to-text generation, the traditional approach is to hand-craft templates @xcite @xcite . there are also works that abstract the templates and use probabilistic models to learn the rules @xcite . popular datasets for these methods are, for example, the 0.7k robocup dataset @xcite . with the advance of deep learning, models have a large number of parameters, so they have to be fueled by large datasets. for data-to-text generation, many supervised methods use large neural networks. to match with these data-hungry models, researchers devote many efforts to curate supervised datasets. most supervised data-to-text datasets have tens of thousands of data. 4 for instance, webnlg dataset @xcite has 13k valid data-text pairs, 5 weathergov @xcite has 22k weather forecasts, and e2e @xcite has 50k restaurant and hotel descriptions, as shown in recently, there is a rising trend of unsupervised approaches @xcite @xcite . unsupervised data-to-text models @xcite are proposed in response to the lack of data-text pairs in many domains, similar to the emergence of unsupervised machine translation that addresses lack of data low-resource language pairs. however, to match with the active research on unsupervised data-to-text generation, no suitable dataset has been curated. as a result, most previous works have to artificially force parallel datasets to be non-parallel, by separately shuffling the data part and text part of the original dataset, or directly deleting the text part. however, such formulation will make the unsupervised datasets inherit the small size of the supervised datasets, which are limited to only tens of thousands samples. there are several caveats of using small datasets for unsupervised models. (1) limiting the model potential: unsupervised models can be data-hungry. for example, 1.6b german and 2.1b english tokens are fed to unsupervised machine translation models @xcite ; 3.3 billion words are used to pretrain the language model bert @xcite . with abundant data, the potential of unsupervised models is unleashed -impressive performance, based on an enormous number of parameters, such as 12 layers of transformers. so far, no large data-to-text corpus can be used to unveil the potential of unsupervised data-to-text. (2) lack of diversity: as many unsupervised models need to impose strong priors such as grammar and dependency tree @xcite , if a proposed model works well on a specific type of text generation, we cannot validate whether such a model generalizes well. (3) negligence of model efficiency: the goal of unsupervised models is to learn from as many unsupervised data as possible and thus achieve high accuracy. however, the current designs of many unsupervised models do not take efficiency into consideration because of the small datasets they use does not expose the problem of efficiency. hence, to foster research in unsupervised data-to-text learning, we construct a new, large-scale dataset, genwiki, which can satisfy all constraints imposed by existing unsupervised models. the dataset will be introduced in section 3 and 4. ## desiderata although it is easy to collect unsupervised datasets for tasks such as machine translation, it is a different story for data-to-text generation. to match with the design of unsupervised data-to-text models, some strict constraints are required. a common constraint is that the text and knowledge graphs should have the same content distribution. some more specific constraints require, for example, the text corpus to have entity annotations. the reason is that recent unsupervised learning models @xcite use cycle training of two tasks: graph-to-text, and text-to-graph, which is simplified to relation extraction given entities. this simplification requires the unsupervised text corpus to have entity annotations in text. additionally, it is also reasonable to make the relation types in the knowledge graphs a closed set of, for example, 300 relations as in webnlg @xcite . in this way, the relation extraction model can avoid collapsing on unseen relations in the test set, and it is also an easier job for human annotators as they only need to consider a limited-sized, well-defined set of relations. we summarize the requirements of the dataset collection as follows: ## genwiki dataset based on the desiderata outlined in section 3, we will first introduce the construction process of our training set in section 4.1, and test set in section 4.2. we will then analyze the characteristics of the resulting dataset in section 4.3. ## evaluating unsupervised data-to-text models since the main purpose of our genwiki dataset is to serve for future models on unsupervised graph-totext generation, we provide a preliminary set of experiments using previous unsupervised data-to-text models. our results can serve as baselines for future work developed on this resource. ## conclusion in this paper, we constructed a large-scale, general-domain text generation dataset, genwiki. this work is designed for the emerging research on unsupervised models for text generation. our dataset is several magnitudes larger than previous text generation datasets, covers a diverse range of topics, and contains more complicated linguistic structures. it relieves the unsupervised models of the limited sizes of previous datasets, and lays the foundation for more future research on unsupervised text generation models.