Surface Realization Using Pretrained Language Models (2020)

Abstract: In the context of Natural Language Generation, surface realization is the task of generating the linear form of a text following a given grammar. Surface realization models usually consist of a cascade of complex sub-modules, either rule-based or neural network-based, each responsible for a specific sub-task. In this work, we show that a single encoder-decoder language model can be used in an end-to-end fashion for all sub-tasks of surface realization. The model is designed based on the BART language model that receives a linear representation of unordered and non-inflected tokens in a sentence along with their corresponding Universal Dependency information and produces the linear sequence of inflected tokens along with the missing words. The model was evaluated on the shallow and deep tracks of the 2020 Surface Realization Shared Task (SR’20) using both human and automatic evaluation. The results indicate that despite its simplicity, our model achieves competitive results among all participants in the shared task.

## introduction natural language generation (nlg) models aim to generate fluent human-like texts given structured data. this involves both content planning (selecting the content to communicate) and surface realization (selecting, ordering, and inflecting the actual words) @xcite . this paper focuses on the second sub-task: surface realization (sr). unlike many tasks in natural language processing (nlp), the performance of sr models are still below human performance. in order to fill this gap and encourage more research in this field, several shared tasks in nlg and sr have been proposed. in particular, since 2018, the surface realization shared tasks @xcite @xcite were introduced to provide common-ground datasets for developing and evaluating nlg systems. this year, the task @xcite proposed two tracks in several languages including english: 1) track1: shallow track and 2) track2: deep track. in the shallow track, unordered and lemmatized tokens with universal dependency (ud) structures @xcite were provided to participants and systems were required to reorder and inflect the tokens to produce final sentences. the deep track was similar to the shallow track but functional words and surface-oriented morphological information were not provided and had to be inferred by the systems. therefor, in addition to determining the order and the inflection of tokens, systems participating in the deep track had to guess the omitted words. considering that the input data is in the form of universal dependency (ud) structure, data-to-text models, and graph-to-text models in particular, seem to be the right choice for the task of surface realization. however, in this study, we take a different path to tackle the problem. our proposed model is designed based on text-to-text approaches using a pretrained encoder-decoder language model. more specifically, a bart @xcite language model is used for the task of surface realization for both the shallow and the deep tracks. the proposed approach is an end-to-end model trained on a linearized representation of the graph of the sentences with their corresponding universal dependency information. the results on the english datasets demonstrate the potential of these models for surface realization tasks and in general data-to-text problems. in the following sections, we first review recent and related approaches in surface realization and textto-text models, then in section 3, our proposed model is explained. section 4 describes the results of our model on the surface realization datasets. finally, section 5 discusses conclusion and future work. ## related work surface realization typically involves three tasks: syntactic realization, morphological realization, and orthographic realization @xcite . syntactic realization tries to identify the proper ordering of the input data, whereas morphological and orthographic realization are responsible for word inflections, punctuation, and formatting. several surface realization models presented at the previous surface realization shared task @xcite used a cascade of pointer-based models for syntactic realization followed by another neural network module for morphological and orthographic realization @xcite @xcite . for example, @xcite utilized a graph attention network (gat) @xcite for encoding the input sentences and a pointer decoder @xcite to select the next element from their graph. whereas @xcite used a bidirectional tree-lstm @xcite as the encoder and an lstm @xcite as the decoder and multiple lstm modules for morphological and orthographic realization tasks. these two approaches achieved the highest performance among all participating systems at sr'19. on the other hand, when the text to generate is conditioned on the content provided in the form of graphs, tables, etc then data-to-text generation models are utilized. as a family of data-to-text models, graph-to-text generation tries to generate natural text given its input graph. graph-to-text generation models employ graph encoders to obtain a suitable representation from the input graph. several applications such as text summarization @xcite , question answering @xcite , as well as surface realization @xcite have used these types of generation models. recently, several works have proposed to use language models in the form of text-to-text for what is inherently data-to-text and particularly graph-to-text tasks @xcite @xcite . instead of modeling the node and edges of the graphs, they mapped the graph structure as sequences of words and let the language model encode the graph information. kale (2020) takes advantage of t5 @xcite for data-to-text problems, whereas @xcite and @xcite utilize gpt2 @xcite . following this recent trend of text-to-text models for graph-based problems, we developed an end-toend approach based on a language model for the problem of data-to-text generation. the approach maps the given universal dependency structures to surface forms to tackle all the tasks required for the surface realization. ## model following the success of pretrained language models for data-to-text generation @xcite , we tackled the surface realization problem using a similar approach. bart @xcite ) is an encoder-decoder language model based on transformers @xcite . specifically, bart is a denoising autoencoder model trained on several denoising tasks making it applicable for a variety of downstream nlp tasks. the language model is trained to first encode the input, and then generate the text based on its input representation. the encoder-decoder architecture of bart makes it suitable for our task where the encoder module encodes the graph representation into an embedding space then the decoder generates the inflected form of the input one token at each decoding step. hence, the model performs syntactic, morphological, and orthographic realization all at the same time. in order to take advantage of a pretrained language model, bart in our case, we first need to map the graph structure of the input into a linear representation (i.e. plain text). for this task, given all the universal dependency (ud) information provided by the sr'20 organizers, we considered using only lemma, upos, feats, head, and deprel for each node. figure 1 shows the mapping structure with an example of a linearized node. as figure 1 shows, we concatenate each token with its corresponding features where special tokens <, >, | identify the boundary of each feature and special tokens such as the beginning of @xmath0.e. token). in these mapped representations, instead of the index for the head feature, the actual token of the head is used. as opposed to previous work where tokens are selected among the input (du and black, 2019; mazzei and basile, 2019), the bart decoder generates tokens at each decoding step. as a result, we have no control on the number of tokens that should be generated. this can lead the model to generate extra tokens or ignore some of the input tokens. to alleviate this issue, we considered the tokenized form of the target sentences as the targets of the model. this allows the model to learn to map each node representation in its input into a token in the output and achieve a 1:1 mapping between the number of nodes in the input and the number of output tokens. an alternative would be to train on the detokenized output form but for the sake of automatic evaluation of the shared task, we did not follow this option. ## conclusion in this work, we used a language model (text-to-text) to tackle the graph-to-text surface realization problem. we showed that using a pretrained encoder-decoder language model such as bart, can allow the model to reconstruct the tokens' orders and inflections in an end-to-end fashion without any modification in the network architecture. this shows that the pretrained language model is able to encode the graph information and map this representation into a human readable text. the results of the deep track showed that our proposed model achieved better performance compared with the systems submitted to sr'20 @xcite where the model has to guess the functional words and morphological information. this high performance directly comes from the pretrained language model characteristics of bart. as for future work, we are planning to experiment with different graph linearization schemes. also, we would like to use a multilingual language model such as mbart @xcite on other languages as well as other language models. finally, we would like to evaluate the incorporation of a copy mechanism @xcite in generating long and complex tokens such as urls.