Leveraging Large Pretrained Models for W eb NLG 2020 (2020)

Abstract: In this paper, we report experiments on finetuning large pretrained models to realize resource description framework (RDF) triples to natural language. We provide the details of how to build one of the top-ranked English generation models in WebNLG Challenge 2020. We also show that there appears to be considerable potential for reranking to improve the current state of the art both in terms of statistical metrics and model-based metrics. Our human analyses of the generated texts show that for Russian, pretrained models showed some success, both in terms of lexical and morpho-syntactic choices for generation, as well as for content aggregation. Nevertheless, in a number of cases, the model can be unpredictable, both in terms of failure or success. Omissions of the content and hallucinations, which in many cases occurred at the same time, were major problems. By contrast, the models for English showed near perfect performance on the validation set.

## introduction the webnlg challenge has attracted increasing attention since 2017. it aims to promote research on realizing resource description framework (rdf) triples @xcite of various categories in fluent and accurate natural language. the rdf triple sets on which the webnlg dataset is best are from dbpedia @xcite and consist of one or several triples, where each triple contains one subject, one predicate, and one object. the most challenging part of webnlg 2017 is to generate from data containing unseen categories, where many predicates and subject/object entities don't appear in the training data. consequently, for webnlg 2020, we are especially interested in improving realization quality for data from unseen categories. because the size of the training data in the webnlg challenge is quite modest, we think it is important to introduce extra knowledge into the pipeline. considering that large pretrained models have achieved tremendous success in various nlp tasks, including recently for data-to-text nlg @xcite , we investigate using pretrained models to enhance generation from unseen categories, and find they work remarkably well. in addition, we try to further improve the finetuned large pretrained models by using reverse model reranking, which involves reranking the outputs of the beam of the original forward model for a given source input using the perplexity of that input (conditioned on the output) according to the reverse model. as shown in @xcite ; @xcite , techniques similar to the reverse model reranking method we use here, under the guise of rsa or noisy channel models, have shown significant benefits for generation tasks when there is room for improvement. however, in our experiments, reverse model reranking not only provides no benefit but also slightly harms realization quality. by looking at the bleu oracle reranking scores, we see a considerable potential for reranking, so we surmise that the quality of the reverse model must be too low to be useful. this could indicate that it is more difficult for large-scale pretrained models to learn to reconstruct meaning representations than to generate natural language text from them. ## system description data the english dataset contains 13,211 and 1,666 triple sets for training and validation, respectively. the russian dataset contains 5,573 and 789 triple sets for training and validation, respectively. each triple set has two versions, original and modified. we use the modified version because it is a cleaner version. each triple set is also associated rdf: "aarhus airport | cityserved | \"aarhus, denmark\"" mr: the aarhus is the airport of aarhus, denmark. with one to several surface realizations. therefore, we have 35,426 and 4,461 parallel data items for training and validation, respectively, on the english task, and 14,630 and 2,062 ones for training and validation, respectively, on the russian task. preprocessing we linearize each triple set into a sequence with three delimiters, which are subject , predicate , object , as shown in pretrained models the training data of the webnlg challenge is too modest in size to sufficiently train a large neural model to generalize very well. in our preliminary experiments on webnlg 2017 @xcite , we found that models trained from scratch were very bad at generating with unseen categories, even though the outputs with seen categories were mostly acceptable. this is due to the unseen categories frequently introducing unseen predicates and noun phrases, on which models, having never seen these expressions, struggle to consistently produce text from. considering that large pretrained models have achieved tremendous success in many areas, we think they should also be helpful for generating unseen content with small training datasets. to test the conjecture that pretrained models would improve performance on unseen content, we use t5 @xcite for the english task and mbart @xcite for the russian task. these models are chosen because t5 is designed for the monolingual task and mbart is designed for multi-lingual tasks. to be more specific, we use the pretrained t5-large huggingface transformer model @xcite and the mbart-large fairseq model @xcite . fine-tuning 1 considering that both these models tokenize the sentences into subwords and the triple delimiters (e.g. predicate ) are supposed 1 code: https://github.com/znculee/webnlg2020 to be indivisible, we add the three special delimiters to the vocabulary of the pretrained models. 2 the embeddings of these three special delimiters are randomly initialized. after extending the vocabularies, the total parameters to be fine-tuned for t5 and mbart are 737,643,008 and 610,851,840, respectively. the t5 model is fine-tuned using cross entropy loss without label smoothing. the learning rate is constantly 2 × 10 -5 and the batch size is 8 samples. we found that an overly large learning rate and batch size could hurt performance. the optimizer is adam (kingma and ba, 2014) where β 1 = 0.9, β 2 = 0.999, = 1 × 10 -8 , and the weight decay is 0. in our preliminary experiments, we found that l2 normalization did not help, whether we used the weight decay of adam or adamw @xcite . the best checkpoint is selected by validation with patience of 10 training epochs. with this setting, the best checkpoint is at the end of the 7 th epoch. the mbart model is fine-tuned with different hyper-parameters than t5 because we followed the recommended fine-tuning guidelines of the mbart authors. it is optimized according to cross entropy loss with label smoothing of 0.2. the batch size is 2048 tokens. the learning rate is 3 × 10 -5 and the scheduler is polynomial decay with 2500 warmup updates. the optimizer is adam where β 1 = 0.9, β 2 = 0.98, = 1 × 10 -6 . the best checkpoint is selected by validation with patience of 20 training epochs. the best checkpoint is at the end of the 32 nd epoch. ## methodology for analyses analyzing the performance of a model is an important aspect of research pursued with neural networks, because even though their architecture is explicit and transparent, their behavior roughly can be compared to a black box. one of the strategies we relied on in the current work is to analyze success and failure of a model qualitatively by comparing given examples to related proportions of the training data. we tested our russian models against the test set, even though it had no labels. based on our linguistic knowledge and using observations from the training and validation data, we hypothesize a possible text for a given source (meaning representation), which allows us to see whether a model generated a plausible/acceptable text for a given source. we also ran our models for russian against the validation set. we discuss our findings in section 5.1. in addition to russian, we also analyzed the model performance on english, though we used only the validation set. our models worked almost flawlessly on the data given for the validation set, both in terms of content realization and lexical and morpho-syntactic selections. the models exhibit high performance on aggregation of content across sentences and also within the same sentence, using coordinating conjunctions for both multiple noun and multiple verb phrases. ## conclusions for russian, we have seen that in certain cases the models successfully realize meaning representations in text, both with respect to fluency and content, but they fail to do the same in very similar cases. to analyze such cases, we looked into distributions of the relevant data in the training set. while considering distributions provides some insight into the models output, further investigation is needed to explain a number of issues. it is common knowledge that more data means better performance, and our observations are consistent with this view. nonetheless, we observed that the same frequency of two entities or predicates in the training data doesn't determine whether the model consistently produces the text corresponding to such content. it is not impossible augmenting the data would induce more errors; this could make the model more biased towards producing text we would like to avoid. thus it is not straightforward to say whether simple data augmentation would help. pretrained models excel due to their accumulating valuable knowledge about the world (semantic and pragmatic) and the language (lexical and morpho-syntactic). to analyze the performance of a pretrained model that is fine-tuned on some specific data, for a specific task, is not straightforward. the possible sources of success or failure are numerous -it might be the pretrained model is the source of success, but contradictory results could be due to inherent biases in the pretrained model. to conclude, we believe that both the architecture of the model and the data needs to be scrutinized to understand why the model and the data do or do not produce good results. nevertheless, on the whole we find that largescale pretrained models work remarkably well on the webnlg tasks, especially t5 for english. to our surprise though, we find that reverse model reranking does not appear to work well with pretrained models. this could indicate that it is easier for a pretrained model to learn to generate natural language text from meaning representations than the reverse task of reconstructing meaning representations from text. in the future, we aim to investigate better reranking scorers to realize the potential of reranking shown by the bleu oracle. additionally, it would also be interesting to leverage the shape information among the triples and employ constrained decoding @xcite to guarantee the correctness of realizations.