NILC at W eb NLG +: Pretrained Sequence-to-Sequence Models on RDF -to-Text Generation (2020)

Abstract: This paper describes the submission by the NILC Computational Linguistics research group of the University of São Paulo/Brazil to the RDF-to-Text task for English at the WebNLG+ challenge. The success of the current pretrained models like BERT or GPT-2 in text-to-text generation tasks is well-known, however, its application/success on data-totext generation has not been well-studied and proven. This way, we explore how good a pretrained model, in particular BART, performs on the data-to-text generation task. The results obtained were worse than the baseline and other systems in almost all automatic measures. However, the human evaluation shows better results for our system. Besides, results suggest that BART may generate paraphrases of reference texts.

## introduction in recent years, the rdf-to-text generation task has gained interest from many researchers @xcite as the rdf language, in which dbpedia is encoded, is widely used within the linked data framework and large scale datasets are encoded in this language. in order to foster the use of the rdf language in this context, the first webnlg challenge was proposed @xcite . this challenge only focused on the text generation task for english and provided a test set that comprised categories included in the training set (seen categories) and categories not included in the training set (unseen categories) to evaluate the ability to generalise of the different approaches. in general, several approaches have been explored in this task and pipeline approaches have shown a better performance than end-to-end approaches for unseen categories but not for seen ones @xcite , leaving the abil-ity to adequately deal with both categories as an open problem. transfer learning has gained relevance in the natural language processing area and pretrained architectures like @xcite or gpt @xcite have outperformed prior state-of-the-art in several tasks and shown a good generalisation ability. even though pretrained models have been widely used in text-to-text generation (such as text simplification and automatic summarisation), this is not the case for data-to-text generation, as the input of this task is generally a graph instead of a text. recently, @xcite fine-tuned gpt-2 for a data-to-text generation task, showing improvements and that current pretrained models can deal with these representations even if the knowledge is not explicitly structured. in this context, this paper presents the system description submitted by the nilc team to the webnlg+ challenge 2020 @xcite . specifically, we fine-tune bart @xcite , a denoising autoencoder for pretraining sequence-to-sequence models, on the rdf-totext generation dataset provided by this task. foot_0 ## webnlg+ challenge the first webnlg challenge @xcite consisted in generating english text from a set of rdf triples extracted from dbpedia. differently from the previous edition, this edition comprises two tasks: ## system description as mentioned, our approach is based on bart. thus, in order to preprocess the input for bart, we linearise the triples by putting all triples (in the form entity, relation and entity/expression) sequentially. we also remove underscores from the entities and expressions, split relations according to uppercase tokens, remove quotes from the expressions and put all in lowercase. for example, the triple "aarhus airport operatingorganisation "aarhus lufthavn as"" is converted into "aarhus airport operating organisation aarhus lufthavn a/s". it is worth noting that we tried other linearisation strategy by putting a dot mark between each preprocessed triple, but this alternative did not lead to improvements. additionally, we train a model to recase the sentences by using the tool provided by moses foot_1 and, finally, we tokenise and convert the sentences to lowercase. we use the large bart model provided by hug-gingface @xcite . we finetune the model for 5 epochs, using a batch size of 16, the adam optimiser with a learning rate of 0.0001, a max length of 100 in the source and target. for the decoding, we use a beam size of 5. for the post-processing, we use the recaser previously trained, normalise the punctuation and detokenise the outputs foot_2 . ## results and discussion the performance of the several proposals at the challenge is computed by using bleu @xcite , meteor (lavie and agarwal, 2007), ter @xcite , chrf++ @xcite , bertscore @xcite , and bleurt (sellam et al., 2020). 4 other explanation is the way as we linearise the rdf triples, as bart could not distinguish the different triples in the input. in recent work, @xcite show similar results to ours on seen categories. besides, they show that performance on rdf-to-text generation is less influenced by the order of the input than other representations. however, a deeper study is necessary to explore the performance on unseen domains. a point to highlight is that even though our results were not good enough, when comparing with approaches that got similar results, we may see that meteor and bleurt produce better results ( each criterion is rated with a single number in the range from "0" (completely disagree) to "100" (completely agree). the scores as they appear for each criterion have been normalised (z-scores) and clustered into groups among which there are no statistically significant differences according to the wilcoxon rank-sum significant test. in general, our system was ranked at 3rd and 4th cluster, except for fluency in which it was ranked at the last cluster. it is worth noting that even though some metrics like bleu or chrf++ showed that our system performed worse than the other ones (upc-poe and orange-nlg teams), results in the human evaluation were opposite and seemed to be more correlated with metrics like meteor or bleurt. furthermore, our system got similar results (without statistical significance differences) to the one proposed by huawei even when the results in automatic evaluation showed a lower performance for our system. all these results reinforce the idea that our proposal could be generating paraphrases in the output. other point to highlight is the result obtained in fluency. we expected that our approach would get better results as it was trained on large corpora and this kind of pretrained models tend to get good performance in terms of fluency. finally, as future work, we plan to evaluate other alternatives for the linearisation process and use multilingual bart @xcite in order to deal with the same task for russian.