Denoising Pre-Training and Data Augmentation Strategies for Enhanced RDF Verbalization with Transformers (2020)

Abstract: The task of verbalization of RDF triples has known a growth in popularity due to the rising ubiquity of Knowledge Bases (KBs). The formalism of RDF triples is a simple and efficient way to store facts at a large scale. However, its abstract representation makes it difficult for humans to interpret. For this purpose, the WebNLG challenge aims at promoting automated RDF-to-text generation. We propose to leverage pre-trainings from augmented data with the Transformer model using a data augmentation strategy. Our experiment results show a minimum relative increases of 3.73%, 126.05% and 88.16% in BLEU score for seen categories, unseen entities and unseen categories respectively over the standard training.

## introduction the purpose of knowledge bases is to store unstructured data into a well-defined and highly structured representation. a commonly used formalism is the resource description framework (rdf) where knowledge takes the form of triplets subject; predicate; object . for instance if we take the triple alan bean; birthdate; 1932 , the subject is alan bean, the predicate/relation/property is birth-date and the object is the date 1932. this triplebased representation allows one to describe complex relations between entities. it is hence frequently used in natural language processing (nlp) applications such as question answering, summarization, recommendation or search engines. although the rdf format is an efficient method to store knowledge, triples are hard to interpret by humans. the automatic translation from sets of triples into human language is however a difficult task. thanks to the webnlg challenge @xcite , this rdf-to-text task attracts now many researchers. this challenge aims at generating a faithful and descriptive textual content from rdf inputs. given the triples alan bean; birth-place; wheeler, texas and alan bean; birth-date; 1932-03-15 , one possible verbalization is "alan bean was born in wheeler, texas on mar 15, 1932.". verbalization of structured data can provide an unambiguous and simple translation from abstract representations. the webnlg dataset is frequently used as a benchmark for information extraction (ie), natural language understanding (nlu) or natural language generation (nlg) tasks. it offers different lexicalization for the same set of triples. despite limited in size, the high-quality of both its textual content and data-to-text mapping significantly stimulated both rdf-to-text, i.e. nlg, and text-to-rdf, i.e. ie and nlu, research directions. for instance, @xcite and @xcite have obtained state-of-the-art performances on the webnlg rdf-to-text task. on the other hand, @xcite considered both nlg and ie objectives within the same learning framework via cycle training. furthermore, the exploration of a new evaluation metric is also investigated on this webnlg dataset, confirming its attractiveness for the research community @xcite . recently, the field of nlp has been significantly impacted by large pre-trained language models and novel learning architectures that improved state-ofthe-art results on several benchmarks @xcite @xcite @xcite . the transformer model @xcite is in most cases at the core of these improvements. the self-attention mechanism of transformer enables a better understanding of underlying dependencies in sentences. however, the aforementioned techniques have not yet been tested on the rdf-to-text task. we therefore propose to combine the transformer model with large pre-trainings to generate more accurate verbalization of the rdf triples. we chose to make use of external corpora during pretraining to reach a better generalization. we build two additional datasets for denoising pre-training and data augmentation purposes. this is motivated by the fact that participants' systems at previous webnlg challenge suffered from high drops in performance when evaluated on new entities and predicates not encountered during the training phase @xcite . thus, by incorporating external corpora, we expect our system to outperform systems solely trained on the webnlg dataset. we evaluate our approach using bleu @xcite , meteor @xcite , chrf++ @xcite and bleurt @xcite . as mentioned in @xcite , bleu is a parameterized metric that may exhibit wild variations. the use of multiple metrics gives better outlooks on the general performance. the reminder of this paper is organized as follows. section 2 gives an overview of existing methods to translate rdfs into their corresponding textual content. after introducing the task and the webnlg datasets in section 3, we introduce our approaches for the challenge in section 4. the different training phases are presented in section 5. we show our results in section 6. finally, we conclude in section 7. ## related work following the traditional nlg pipeline @xcite , early works on factual data verbalization mainly focused on the creation of rules or templates to produce the textual output. @xcite have shown that most of the useful information for the generation is brought by the rich linguistic information of the rdf properties, i.e. predicates. they have noticed that properties can be classified into 6 categories based on their pattern. for instance, some predicates may be a concatenation of a verb and a noun like hasemail, or starting with a verb and finishing with an adjective like hastimeopen. they established handcrafted rules to construct linguistic forms for each category. this technique allows a domain independent verbalization since relying only on the pattern of predicates. on the contrary, @xcite used domain-dependent ontology lexicon to proceed to a fine-grained and specific verbalization of mentioned concepts. although some methods tried to automatically learn those templates @xcite , a major disadvantage of such approaches is the need of handcrafted rules and the poor generalization power. the main downside of kb verbalization is the lack of paired data. in spite of the impressive success of deep learning (dl) methods, few attempts could be made toward rdf-to-text task due to data scarcity. to address this issue, @xcite came up with a new dataset which includes rdfs extracted from dbpedia foot_0 with their corresponding translations in natural language. to further stimulate the development of kb verbalizer, @xcite set up the webnlg challenge 2017, welcoming new techniques for the rdf-to-text task. among the proposed strategies, dl-based models outperformed rules-based systems @xcite . neural approaches relied almost on the attention-based encoder-decoder architecture @xcite . the encoder-decoder framework showed promising results, with some limitations on unseen domains. following webnlg 2017, multiple works were conducted to cope with the witnessed drawbacks of participants' system. @xcite argued that kullback-leibler (kl) divergence should be avoided. minimizing the kl divergence contributes to more diversity in generated samples, at the cost of quality. as a matter of fact, diversity is unnecessary for kbs verbalization since it is constrained on specific entities and relations. therefore, they proposed to optimize the inverse kl divergence to generate higher-quality textual description. another flaw appears in the linearization of input triples. such transformation gets rid of the intrinsic structure of the rdfs graph which may lead to inconsistency or relational mismatch in predicted sentences. to address this question and upon the rising popularity of graph neural networks (gnns), graph-based solutions were presented. @xcite introduced a gtr-lstm triple encoder. the idea is to traverse the graph in a specific manner such that vertices are better ordered to leverage the intra-and inter-triples relationships. more recently, graph neural networks (gnns) have raised a lot of attention to capture dependencies between nodes of a graph @xcite . the message passing scheme offers the ability to better model relations nodes and their neighborhoods. well-suited for the generation from structured data, marcheggiani and perez-beltrachini (2018) made use of a graph convolutional network (gcn) to explicitly encode the input structure of the triples instead of a straightforward linearization. more precisely, they used an encoder-decoder architecture composed of a gcn encoder and a lstm decoder. they further applied a copy mechanism from @xcite to copy words from the input rdf triples. the resulting performances confirmed the benefit of the graph-based encoder. following this conclusion, several papers proposed graph-based encoder as a solution @xcite @xcite . meanwhile, in contrast to the conditional text generation paradigm, @xcite considered the kb verbalization as a text editing task with their facteditor model. text editing consists in applying a sequence of transformations on an input text to produce a particular output. facteditor, a simple bilstm model, creates a verbalization candidate, so-called draft text, from which the edition process starts. three actions can be made on a candidate word: keeping it, dropping it or generating a new one. recent studies also explore the fusion of both nlu and nlg into one unified model. the rdfto-text and text-to-rdf can be seen as the inverse task from one another. inspired by the advancement from computer vision @xcite , @xcite and @xcite provide a cycle training framework to learn both tasks simultaneously. nevertheless, none of the above techniques use large pre-training neither take advantage of the transformer model. we propose in this paper to in-vestigate the effect of massive pre-training coupled with the vanilla transformer for the rdf-to-text task. ## the webnlg challenge in this section we present the webnlg task and datasets. ## proposed approaches we propose to use a transformer model with data augmentation. ## training we suggest in our work to thoroughly combine a pre-training stage before training our transformer model on the webnlg dataset. we describe our learning design in the following subsections. ## experiments we evaluate our models on the webnlg dataset test set. we compare 8 transformers models corresponding to 8 different training settings. ## conclusion we proposed to use the transformer model with data augmentation for the rdf-to-text task on the webnlg dataset. we took advatange of wikipedia to build new datasets for different pre-training strategies. we investigated the effect of denoising autoencoder objective and data augmentation as pre-training approaches. we studied the impact of curriculum learning while fine-tuning our model on webnlg data. further analysis demonstrated that pre-training is highly beneficial for knowledge verbalization. although noisy, our massive data augmentation contributed to generate much accurate textual content. as future work, we will perform a deep analysis of the robustness of our models to unseen entities and categories.