NILC at SR ’20: Exploring Pre-Trained Models in Surface Realisation (2020)

Abstract: This paper describes the submission by the NILC Computational Linguistics research group of the University of S ̃ao Paulo/Brazil to the English Track 2 (closed sub-track) at the Surface Realisation Shared Task 2020. The success of the current pre-trained models like BERT or GPT-2 in several tasks is well-known, however, this is not the case for data-to-text generation tasks and just recently some initiatives focused on it. This way, we explore how a pre-trained model (GPT-2) performs on the UD-to-text generation task. In general, the achieved results were poor, but there are some interesting ideas to explore. Among the learned lessons we may note that it is necessary to study strategies to represent UD inputs and to introduce structural knowledge into these pre-trained models.

## introduction universal dependencies 1 (ud) have gained relevance in the natural language processing (nlp) community. ud treebanks have already proved useful in the development of multilingual applications, becoming an advantage for developers. one of the successful applications of ud is related to data-to-text generation. this may be seen in the two shared-tasks proposed @xcite in which there were several participants. in general, the surface realisation shared task aims to continue with the development of natural language generation methods focused on the surface realisation task. specifically, models in this task must generate sentences from dependency trees (or similar structures) in conll format. this task comprises two tracks: (1) the shallow track, in which word order and word forms are removed from the ud structure, and (2) the deep track, which, in addition to the word order and word forms, functional words, morphological features and other kinds of information are removed or changed in the ud structure to make it more similar to a semantic representation (called deep syntax). diverse approaches have been applied on this shared task in previous editions, being mainly divided in inflection generation and word ordering tasks @xcite . besides, there is a trend to use neural models for the same tasks. recently, pre-trained language models have become standard in a variety of natural language processing (nlp) tasks @xcite @xcite , including sentencelevel classification, sequence tagging and question answering. these models can be pre-trained on large corpora of available unannotated text, and then fine-tuned for specific tasks on smaller amounts of supervised data, relying on the induced language model structure to facilitate generalisation beyond the annotations. these pre-trained models have been widely used in text-to-text generation tasks such as text simplification, automatic summarisation, and machine translation, obtaining good results and outperforming the current state of the art. however, there are few initiatives for the data-to-text generation task. lately, @xcite used gpt-2 for fine-tuning amr-to-text generation task, showing improvements and that current pre-trained models can handle these representations even if the knowledge is not explicitly structured. in this context, this paper presents the description of the system submitted by the nilc team to the track 2 of the surface realisation shared task 2020 (track 2 -sr'20) @xcite . our proposal is an end-to-end approach inspired by the work of @xcite . we explore some strategies to sequentially represent ud structures and to fine-tune gpt-2 @xcite on the pre-processed dataset 2 . ## track dataset -sr'20 the dataset for the track 2 is composed by ud structures and their corresponding sentences. the ud structure is similar to a dependency tree, however, some information are modified: figure 1 and 2 show the conll and graphic representation for the sentence "two of them were being run by 2 officials of the ministry of the interior!". in figure 1 , we may see "idx" and "original id" attributes, where "x" can be a number. this attributes are related to the track 1 and the original ids (positions) of the tokens in the sentence and are removed from the test set. the corresponding source code is available at https://github.com/msobrevillac/ pretrained-amr-to-text finally, the dataset contains subsets from different domains. for english (our target language in this task), there are 4 files in the training and development (dev) set each, and 7 files for the test set from the previous edition @xcite and 1 file for the test set in this edition. ## system description in first edition of this shared task, we submit a system in which we use a data augmentation strategy @xcite to deal with the track 1. however, for this edition, we focus on the track 2 and use only resources allowed by the shared task (closed sub-track). differently from most of the work found in the literature, we propose an end-to-end approach, jointly learning the inflection generation and word ordering tasks. ## results and discussion the automatic performance of the diverse proposals at the shared task is computed by the following measures: in general, morphological relations do not seem to be necessary as the performance improves when these are removed (b in to make the analysis of omitting morphological relations clearer, we may see versions c (without parentheses and morphological relations) and d (without parentheses). both versions contain fewer tokens (in relation to a and b versions) and one may see that disregarding morphological relations produces improvements (results in version c are better than in d). another point to note is that parentheses are the most important tokens as they represent the structure of the input. therefore, removing them from the input leads to a significant drop in the performance (d). furthermore, this drop is bigger than the one obtained by leaving out all the relations (e), showing that parentheses could encode some information about the relations among the nodes. finally, the results of the f version could suggest that, although parentheses encode some information about the relations, there are more information that is not encoded, making the use of relations (in general) necessary. ## conclusion and future work this paper describes the application of a pre-trained model, the gpt-2, to the ud-to-text generation task in the context of the surface realisation shared task. results show that the way as the ud structures are linearised is important for the model in this task. thus, an interesting research line for future work could be investigating other ways to represent/linearise ud structures and to introduce the knowledge about structure in this kind of model. as future work, we also plan to apply this approach to other languages and use a bigger version of gpt-2.