S hape of Synth to Come: W hy We Should Use Synthetic Data for E nglish Surface Realization (2020)

Abstract: The Surface Realization Shared Tasks of 2018 and 2019 were Natural Language Generation shared tasks with the goal of exploring approaches to surface realization from Universal-Dependency-like trees to surface strings for several languages. In the 2018 shared task there was very little difference in the absolute performance of systems trained with and without additional, synthetically created data, and a new rule prohibiting the use of synthetic data was introduced for the 2019 shared task. Contrary to the findings of the 2018 shared task, we show, in experiments on the English 2018 dataset, that the use of synthetic data can have a substantial positive effect – an improvement of almost 8 BLEU points for a previously state-of-the-art system. We analyse the effects of synthetic data, and we argue that its use should be encouraged rather than prohibited so that future research efforts continue to explore systems that can take advantage of such data.

## introduction the shallow task of the recent surface realization (sr) shared tasks @xcite @xcite appears to be a relatively straightforward problem. given a tree of lemmas, a system has to restore the original word order of the sentence and inflect its lemmas, see figure 1 . yet sr systems often struggle, even for a relatively fixed word order language such as english. improved performance would facilitate investigation of more complex versions of the shallow task, such as the deep task in which function words are pruned from the tree, which may be of more practical use in pipeline natural language generation (nlg) systems @xcite ; castro ferreira et al., 2019). in this paper we explore the use of synthetic data for the english shallow task. synthetic data is created by taking an unlabelled sentence, parsing it with an open source universal dependency parser foot_0 and transforming the result into the input representation. unlike in the 2018 shared task, where a system trained with synthetic data performed roughly the same as a system trained on the original dataset @xcite , we find its use leads to a large improvement in performance. the state-of-the-art on the dataset is 72.7 bleu-4 score @xcite ) -our system achieves a similar result of 72.3, which improves to 80.1 with the use of synthetic data. we analyse the ways in which synthetic data helps to improve performance, finding that longer sentences are particularly improved and more exactly correct linearizations are generated overall. although it is common knowledge that machine learning systems typically benefit from more data, this 7.4 point jump in bleu is important and worth emphasizing. the 2019 shared task introduced a new rule which prohibited the use of synthetic data. this was done in order to make the results of different systems more comparable. however, systems designed with smaller datasets in mind might not scale to the use of synthetic data, and an inadvertent consequence of such a rule is that it may produce results which could be misleading for future research directions. for instance, the system which was the clear winner of this year's shared task @xcite used tree-structured long short-term memory (lstm) networks @xcite . in general, tree lstms can be slow and difficult to train. 2 @xcite utilized a variant of the tree lstm in a similar nlg task, converting abstract meaning representation (amr) graphs to text. following the state-of-the-art system @xcite , which used standard lstms, song et al. augmented their training with synthetic data. though their system outperformed konstas et al. at equivalent levels of additional training sentences, it was unable to scale up to the 20 million sentences used by the best konstas et al. system and ultimately did not outperform them. 3 critics of neural nlg approaches foot_3 emphasise that quality and reliability are at the core of production-ready nlg systems. what we are essentially arguing is that if using synthetic data contributes to producing higher quality outputs, then we ought to ensure we are designing systems that can take advantage of synthetic data. ## results in ## conclusion we have argued for the use of synthetic data in english surface realization, justified by the fact that its use gives a significant performance boost on the shallow task, from 72.7 bleu up to 80.1. while this is not yet at the level of reliability needed for neural nlg systems to be used commercially, it is a step in the right direction. assuming the use of synthetic data, more needs to be investigated in order to fully maximize its benefit on performance. future work will look more closely at the choice of corpus, construction details of the synthetic dataset, as well as the tradeoff between training time and accuracy that comes with larger vocabularies. the work described in this paper has focused on english. another avenue of research would be to investigate the role of synthetic data in surface realization in other languages.