NUIG - DSI at the W eb NLG + challenge: Leveraging Transfer Learning for RDF -to-text generation (2020)

Abstract: This paper describes the system submitted by NUIG-DSI to the WebNLG+ challenge 2020 in the RDF-to-text generation task for the English language. For this challenge, we leverage transfer learning by adopting the T5 model architecture for our submission and fine-tune the model on the WebNLG+ corpus. Our submission ranks among the top five systems for most of the automatic evaluation metrics achieving a BLEU score of 51.74 over all categories with scores of 58.23 and 45.57 across seen and unseen categories respectively.

## introduction the webnlg+ challenge @xcite is concerned with mapping data to text, where the data is in the form of rdf-triples 1 extracted from dbpedia @xcite and the text is a verbalisation of these triples. the rdfto-text generation task focuses on generating a verbalisation in a human language in the output based on a set of rdf-triples in the input. in general, data-to-text generation is concerned with building systems that can produce meaningful texts in a natural language from some underlying non-linguistic representation of information @xcite . traditionally, most applications for datato-text generation have relied on rule-based systems which are designed using a modular pipeline architecture @xcite . however, there has been a shift recently towards end-to-end architectures using neural networks to convert data in the input to text in a natural language in the output. in our submission, we employ an end-to-end approach using the t5 model architecture @xcite which is pre-trained on a large corpus of text scraped from the web. we fine-tune the t5 model on the webnlg+ corpus and explore 1 rdf -resource description framework various pre-training and pre-processing strategies to improve the performance of our system. ## background the webnlg challenge @xcite was created with the goal of producing a common benchmark to compare "microplanners", i.e, generation systems that verbalise non-linguistic content to text in some human language. in 2017, the challenge received a mix of submissions built using template or grammar-based pipeline, statistical machine translation (smt) and neural machine translation (nmt) frameworks. the test set used for final evaluation was split into two subsets, seen and unseen. the first subset contained data from the categories that were also present in the training set while the second included new data from unseen categories that were not present in the training set at all. on the seen categories, the nmt and smt-based systems mostly outperformed the rule-based pipeline sytems in terms of bleu and ter score. however, the scores for the nmt-based systems dropped significantly on the unseen categories while the rule-based systems were able to generalise better on the new and unseen domains. further work by castro ferreira et al. (2019) compared pipeline-based and end-to-end architectures and their findings also suggest that the systems which are trained end-to-end are comparable to pipeline methods on the seen categories but do not generalise to new and unseen domains of data. more recently, @xcite have shown that applying transfer learning using an end-to-end pretrained model such as t5 achieves state-of-the-art results on three benchmark datasets for data-totext generation and performs well even on out-ofdomain inputs in the unseen categories of data. tripleset 200 public square location cleveland cleveland leader frank g. jackson cleveland ispartof ohio t5-prefix translate triple to text: 200 public square location cleveland cleveland leader frank g. jackson cleveland ispartof ohio @xmath0. @xmath1. @xmath2. jackson cleveland is part of ohio lexicalisation frank g jackson is a leader in cleveland, ohio where 200 public square is located. ( @xcite and is pre-trained using unsupervised learning on a large corpus of unlabeled data obtained from the web using the common crawl project. it is trained using a denoising objective, also known as "masked language modelling", where a model is trained to predict missing or otherwise corrupted tokens in the input. the basic idea underlying the t5 model is to treat every text processing problem as a "text-to-text" problem, i.e. taking some form of text in the input and producing new text as the output. for the webnlg+ dataset, this input is not purely textual in the form of sentences but rather a set of rdf-triples. therefore, for modelling the input with the t5 architecture, we linearise the triples in the input into a sequence and effectively treat it as a linear sequence of text. ## dataset the webnlg+ dataset consists of rdf-triples extracted from dbpedia paired with reference text lexicalisations. these lexicalisations contain sequences of one or more short sentences in english and russian, verbalising the data units in the input. the corpus contains rdf-triples from 19 dbpedia categories and is divided into two subsets, seen and unseen. the 16 seen categories are airport, artist, astronaut, athlete, building, celestialbody, city, comicscharacter, company, food, meanoftransportation, monument, politician, sportsteam, university and writtenwork and the three unseen categories are film, musicalwork and scientist. the english corpus contains 16,095 rdf-triples in the input paired with 42,873 lexicalisations in the output. we use the same training, validation and test sets as defined in the challenge, where the training set contains data only from the seen categories and the test set contains data from both seen and unseen categories ( ## methodology the t5 model is pre-trained on a multi-task mixture of supervised and unsupervised tasks where each task is converted into a text-to-text format. this model can then be fine-tuned on a downstream supervised task on some labeled data or further trained in an unsupervised fashion on unlabeled data. in this section, we explore various pre-processing strategies for fine-tuning as well as a pre-training strategy for the t5 model. fine-tuning: for fine-tuning the t5 model, we linearise the triples in the input into a sequence and prepend it with the string, translate triple to text: to all instances in the dataset, similar to the implementation of the original t5 model. we follow the default ordering for the triples when linearising a tripleset. we also incorporate additional tags to mark the subject, predicate and object in each triple @xmath3, <@xmath4. these tags are added as additional special tokens to the model vocabulary. furthermore, for the subject and the object entities in each triple, we add tags for the type of the entity using dbpe- in many instances, the predicate in the rdftriples is made up of two or more tokens joined together using the camelcase convention. we split these multi-word predicates on the camelcase into separate tokens using a regular expression. figure 1 shows an example of a tripleset paired with the modifications mentioned in this section along with the corresponding lexicalisation. the t5 model is then trained in a supervised fashion with the rdf-triples in the input to generate the target lexcalisations in the output. pre-training: the t5 model can additionally be trained on unlabeled data with masked spans of tokens with the objective to predict the missing tokens. in our experiments, we pre-train two models using this strategy on two different corpora. in the first case, we train on the reference lexicalisations in the webnlg+ corpus by randomly corrupting 15% of the tokens in the text and for the second we use a corpus of abstracts from dbpedia. we include abstracts only for the entities which are present in the training set and randomly mask 15% of the tokens in this case too. since we cannot find an abstract for each and every entity in the training set, we ended up with 2,540 abstracts consisting of 398,864 tokens and 50,092 types in total with an average of 157.03 tokens per abstract. after pre-training, we fine-tune on the webnlg+ corpus to predict the target lexicalisation in the output conditioned on the rdf-triples given in the input. ## experimental setup we adopt the webnlg baseline system @xcite as one of the baseline architectures for our experiments, which is a vanilla sequence-tosequence lstm model with attention @xcite where the rdf-triples in the input are linearised as a sequence and the output text is tokenised before training. we use another baseline based on the transformer architecture @xcite similar to the end-to-end architecture setup by castro @xcite . these baseline models are trained using the opennmt library @xcite . we use the default parameters for two baseline models. two hidden layers and 500 units per hidden layer with input feeding @xcite enabled and word embeddings of size 500-dimensions are used for the lstm neural model. dropout is applied with value 0.3 and the lstm model is trained with stochastic gradient descent, starting with a learning rate of 1.0 and learning rate decay enabled. for the transformer model, the encoder-decoder setup contains 6 layers with 512 hidden units. the word embeddings are 512-dimensional and the feed-forward sublayers are 2048-dimensional. each multi-head attention sublayer consists of 8 attention heads. dropout is applied with value 0.1 and the model is trained using adam optimizer (kingma and ba, 2015) for 100,000 steps. we also enable the options for dynamic dictionary and shared vocabulary to allow the model to share tokens between the source and the target side. for the pre-trained t5 model, we follow hug-gingface's @xcite the "small" variant of the t5 architecture for finetuning and additional pre-training in our experiments. the t5-small model contains 6 layers each in the encoder and the decoder with each multi-head attention sublayer consisting of 8 heads. the word-embeddings are 512-dimensional and the feed-forward sublayers are 2048-dimensional. this variant has about 60 million parameters and is faster to train compared to other variants of the t5 architecture. the t5-small variant closely follows the architectural set-up of the baseline transformer model as the two models are roughly equivalent in terms of the structure of the encoder and decoder layers, and the number of parameters. for our final submission, however, we use the "base" variant which consists of 12 layers each in the encoder and the decoder with each multi-head attention sublayer consisting of 12 heads. in this variation, the word-embeddings are 768-dimensional and the dimensionality of the feed-forward sublayers is 3072. this variant has about 220 million parameters. in both variants of the t5 model architecture, weight decay is applied with a value of 0.01 and dropout is applied with a probability of 0.1 for regularisation. fine-tuning as well as additional pre-training is done for 20,000 steps respectively with a batchsize of 32 using the adam optimizer with a learning rate of 0.001. ## results in this section, we report the results of our experiments on the validation set of the webnlg+ corpus. since at the time of writing, we do not have access to the official webnlg+ reference lexicalisations in the test set, to evaluate performance on the unseen categories of data, we treat artist, athlete, celestialbody, company, meanoftransportation and politician as unseen categories and exclude data from these categories from the training set to treat them as new and unseen categories in the validation set. in terms of meteor score, the gains of transfer learning are noticeable only in the unseen categories of data, while for the data in the seen categories, the baseline models are quite competitive and there does not appear to be any significant improvements with the pre-trained model. the addition @xmath5, <@xmath6.4 to 39.5. however, for seen categories, there is a drop of about of 0.9. information about entity types from dbpedia also appears to be useful for the unseen categories, improving the bleu score from 37.4 to 38.9 for the t5+types model. however, it also leads to a performance drop by about 4 points for each metric in the case of seen categories. the t5 model uses sentencepiece @xcite for subword tokenisation to handle unknown and rare tokens, such as the multi-word predicates in this corpus. however, for the small variant of the t5 model architecture, additional unsupervised pre-training on reference lexicalisations (t5+lex in for our final submission to the webnlg+ challenge 2020, we train a "base" variant of the t5 model using data from the entire training set of the webnlg+ corpus. before fine-tuning the t5base model, we split the multi-word predicates and @xmath7, <@xmath8, predicates and objects respectively. ## conclusion in this paper, we presented the description of the system submitted by nuig-dsi to the webnlg+ challenge 2020. we participated in the rdf-totext generation task for the english language using an end-to-end system based on the t5 model architecture. we split the predicates on camelcase and @xmath9, <@xmath10, predicates and objects respectively before fine-tuning the t5 model on the webnlg+ corpus. our submission ranks among the top 5 systems for most of the automatic evaluation metrics, achieving a bleu score of 51.74 over all categories.