Train Hard, Finetune Easy: Multilingual Denoising for RDF -to-Text Generation (2020)

Abstract: We describe our system for the RDF-to-text generation task of the WebNLG Challenge 2020. We base our approach on the mBART model, which is pre-trained for multilingual denoising. This allows us to use a simple, identical, end-to-end setup for both English and Russian. Requiring minimal taskor languagespecific effort, our model placed in the first third of the leaderboard for English and first or second for Russian on automatic metrics, and it made it into the best or second-best system cluster on human evaluation.

## introduction the landscape of approaches for text generation has evolved since the first edition of the webnlg challenge. self-supervised pre-training objectivessuch as language modelling and text denoisinghave proven efficient for training neural models with excellent surface realization capabilities @xcite . pre-training is used to improve the performance of models on downstream tasks, requiring only a small amount of task-specific data @xcite . pre-trained models can exploit shared representations across languages, following the success of multilingual word embeddings @xcite . although multilingual pre-training (i.e., pre-training on a collection of corpora from multiple languages) may slightly hurt performance for high-resource languages, it allows using the models for crosslingual tasks @xcite . neural architectures for text generation also gave rise to end-to-end approaches, where inputs and outputs are linearized and the task is solved by a single neural sequence-to-sequence model. despite its disproportionate simplicity, this approach can be hard to beat using task-specific, modular approaches @xcite . in our submission, we took advantage of recent advances in pre-trained denoising autoencoders, multilingual representations, and sequenceto-sequence approaches. they enabled us to approach rdf-to-text generation both in english and russian with a simple, identical, end-to-end setup. we finetune the pre-trained mbart model @xcite on the provided training data individually for each language. we feed tokenized and trivially linearized input rdf triples into the model and train it to output ground-truth references. we do not use any additional preprocessing, postprocessing, or other intermediate steps. originally, this approach was just a baseline that we planned to improve. however, the baseline approach yielded results of such quality that we decided to use it for our official webnlg submission. the results of automatic metrics @xcite foot_0 and human evaluation as well as our manual inspections confirmed our expectations. in automatic metrics, our solution placed in the top third of the field (out of 35 submissions) for english and first or second (out of 12 submissions) for russian. in human evaluation, it scored in the best or second-best system cluster. we believe that our approach-with its excessive simplicity-can serve as a benchmark for a trade-off between the output quality and the setup complexity. ## task description the webnlg challenge 2020 (castro-ferreira et al., 2020) foot_1 is the second edition of the shared task in mapping structured data to text. the data contains sets of rdf triples extracted from dbpedia accompanied with verbalizations which were crowdsourced from human annotators. figure 1: our setup is simple: after tokenizing and linearizing the rdf triples, we finetune two separate mbart models for english and russian using provided training data. we submit the unprocessed output from each model. the original challenge @xcite ) included 10 categories in the training data: airport, astronaut, building, city, comicscharacter, food, monument, sportsteam, university, and written-work. each set of triples included several verbalizations to promote lexical variability. webnlg 2020 includes several extensions: ## multilingual denoising denoising autoencoders are trained to take a partially corrupted input and restore the original undistorted input by minimizing the reconstruction error @xcite . on top of regular autoencoders, the model is forced to extract high-level features from the input distribution to filter out the noise. with a suitable noise function, denoising autoencoders can be trained in a self-supervised way on large datasets. bart @xcite ) is a denoising autoencoder with an objective of restoring a corrupted document. the model uses an encoder-decoder architecture: the bi-directional encoder encodes the corrupted input; the left-to-right decoder aims to restore the original, undistorted input. the model can be seen as a generalization of both bert @xcite and gpt-2 @xcite . adopting bart's objective and architecture, mbart @xcite ) is pre-trained on the large-scale cc25 corpus extracted from common crawl, which contains data in 25 languages @xcite . the data is tokenized using a sen-tencepiece model @xcite trained on the training corpus with a vocabulary of 250,000 subword tokens. the noise function of mbart replaces text spans of arbitrary length with a mask token (35% of the words in each instance) and permutes the order of sentences. the model uses the transformer architecture @xcite with 12 layers for the encoder and 12 layers for the decoder (∼680m parameters). ## our submission we formulate the rdf-to-text task as text denoising and train mbart to solve the task individually for each language (see figure 1 ). we use the provided xml webnlg data reader foot_2 to load and linearize the triples. for each triple, we use the flat_triple() method which converts each triple into the following format: note that the constituents of the triple (subject, predicate, object) are only marked positionally, without any extra tags. we use a token not present in the training data (" ") for delimiting individual triples to avoid extending the model vocabulary. we linearize the triples in their default order. the population metro of ciudad ayala is 1777539. ## results we report on webnlg automatic and human evaluation results, as well as our own error analysis. ## discussion our solution benefits from the denoising skills of the pre-trained mbart model, which to a certain extent combines all the tasks of the micro-planning pipeline (lexicalization, aggregation, surface realization, referring expression generation, sentence segmentation). finetuning on task-specific data then mostly helps to specify the task at hand. moreover, multilingual pre-training allows us to use a single architecture for both english and russian. that being said, we note the rdf-to-text task is far from solved. the performance of our model is noticeably lower on categories unseen in training, and it is prone to swapping relations of entities or hallucinating relations. even though the longest examples in the webnlg dataset fit into the model, the length of the input sequence is still limited and the model does not generalize for inputs of arbitrary size. moreover, english and russian are coincidentally the two most represented languages in the mbart pre-training corpora (ca. 300 gb of data each) and the performance of our model would probably be lower with low-resource languages. ## conclusion we presented a simple setup for rdf-to-text generation, consisting of triple linearization and text denosing. with the help of a multilingual pretrained model, this approach is language-agnostic and yields high-quality results with minimal effort. we hope that it will serve as a baseline for more complex approaches to rdf-to-text generation.