NUIG - DSI ’s submission to The GEM Benchmark 2021 (2021)

Abstract: This paper describes the submission by NUIG-DSI to the GEM benchmark 2021. We participate in the modeling shared task where we submit outputs on four datasets for data-to-text generation, namely, DART, WebNLG (en), E2E and CommonGen. We follow an approach similar to the one described in the GEM benchmark paper where we use the pre-trained T5-base model for our submission. We train this model on additional monolingual data where we experiment with different masking strategies specifically focused on masking entities, predicates and concepts as well as a random masking strategy for pre-training. In our results we find that random masking performs the best in terms of automatic evaluation metrics, though the results are not statistically significantly different compared to other masking strategies.

## introduction the gem benchmark @xcite is a living benchmark focusing on generation, evaluation and metrics for a variety of natural language generation tasks including summarization, simplification, dialog and data-to-text generation. in general, the field of natural language generation (nlg) is concerned with automatic generation of human understandable texts, typically from a nonlinguistic or textual representation of information as input @xcite . traditionally, most applications for nlg have relied on rulebased systems designed using a modular pipeline approach @xcite . however, recently approaches based on neutral networks with an encoder-decoder architecture trained in an endto-end fashion have gained popularity. these typically follow the paradigm of pre-training on a large corpus followed by fine-tuning on a task specific dataset and have been shown to achieve state-of-theart results on many natural language tasks @xcite . when evaluated by human annotators, neural models for data-to-text generation have been found to produce fluent text though such models might struggle in terms of data coverage, relevance and correctness where rulebased systems score high @xcite . in our participation in the gem benchmark, we submit outputs for four datasets including dart @xcite , webnlg @xcite , e2e @xcite and commongen @xcite . we use the pre-trained t5-base model architecture @xcite for our submission implemented using the transformers library from hugging face @xcite . we first train on monolingual data before fine-tuning on the task-specific dataset. for dart and webnlg, we use abstracts from dbpedia @xcite for training while for the other two datasets, we use monolingual target-side references for pre-training with a masked language modeling objective. we experiment with different masking strategies where we mask entities and predicates (for dart), meaning representation fields (for e2e) and concepts (for commongen) and compare the results with commonly used approach of random masking. our results suggest that random masking achieves the best scores for automatic evaluation metrics for dart, webnlg and e2e while additional pretraining appears to hurt the performance for com-mongen. ## methodology in this section we define our methodology on the four datasets where we make a submission and subsequently discuss the results based on automatic evaluation metrics defined in the gem benchmark. ## results compared to the baselines described in the gem benchmark @xcite , we observe higher scores in our submissions for automatic metrics on the commongen and dart datasets while scoring lower on the cleaned e2e and webnlg (en) datasets especially on the test and challenge subsets for both e2e and webnlg. ## conclusion we presented a description of the system submitted by nuig-dsi to the gem benchmark 2021. we participated in the modeling shared task and submitted outputs on four datasets for data-to-text generation including dart, webnlg (en), e2e and commongen using the t5-base model. we first trained this model with monolingual data from dbpedia abstracts and target side references before fine-tuning on respective training datasets. additionally we experimented with various masking strategies focusing specifically on masking entities, predicates and concepts as well as a random masking strategy for training. we found random masking to perform the best and submit our final outputs using this approach.