Machine Translation Pre-training for Data-to-Text Generation - A Case Study in C zech (2020)

Abstract: While there is a large body of research studying deep learning methods for text generation from structured data, almost all of it focuses purely on English. In this paper, we study the effectiveness of machine translation based pre-training for data-to-text generation in non-English languages. Since the structured data is generally expressed in English, text generation into other languages involves elements of translation, transliteration and copying - elements already encoded in neural machine translation systems. Moreover, since data-to-text corpora are typically small, this task can benefit greatly from pre-training. We conduct experiments on Czech, a morphologically complex language. Results show that machine translation pre-training lets us train endto-end models that significantly improve upon unsupervised pre-training and linguistically informed pipelined neural systems, as judged by automatic metrics and human evaluation. We also show that this approach enjoys several desirable properties, including improved performance in low data scenarios and applicability to low resource languages.

## introduction data-to-text refers to the process of generating accurate and fluent natural language text from structured data such as tables, lists, graphs etc. @xcite for example, consider figure 1 , in the context of a restaurant booking system. the system must take a meaning representation (mr) as input -in this case represented in the form of a dialogue act (inform) and a list of key value pairs related to the restaurant -and generate fluent text that is firmly grounded in the mr. in this work, we focus on generating text in non-english languages and show that it is possible to significantly reduce this accuracy gap by pre-training fully lexicalized models on an nmt figure 1 : generating text from structured data. aligned segments from the structured data and natural language have the same color. task. for an example motivating the use of nmt, consider figure 1 once again. in order to generate semantically correct and natural sounding text in czech (marathi), a data-to-text model would need to learn the following skills: in the case of marathi, which has a different script, there is the additional challenge of transliterating the restaurant name as well. it is unreasonable to expect neural data-to-text models to learn all these skills, especially since the size of most nlg foot_0 datasets is quite small. however, modern neural machine translation systems are already fairly adept at translating, transliterating, copying, inflecting etc. consequently, we hypothesise that the parameters of an nmt model will act as a very strong prior for an nlg model. ## model architecture we use the transformer @xcite based encoder-decoder architecture by casting data-totext as a seq2seq problem, where the structured data is flattened into a plain string consisting of a series of intents and slot key-value pairs. more exotic architectures have been suggested in prior work, but the findings of @xcite show that simple seq2seq models are competitive alternatives, while being simpler to implement. secondly, the transformer architecture is state-of-the art for nmt. thirdly, keeping the pre-train and fine-tune architectures the same allows us to easily transfer knowledge between the two steps by parameter initialization. ## pre-train + fine-tune our modeling approach is simple. we first use a parallel corpus to train a sequence-tosequence transformer based neural machine translation model. next, we fine-tune this nmt model using a data-to-text corpus for a small number of steps. all the model parameters are updated in the fine-tuning process. in practice, we found that a bidirectional model, which can translate from english to the target language and vice-versa, performed slightly better. ## baselines we compare with the following baselines: scratch a baseline where all the parameters are learned from scratch, without any kind of transfer learning. this is a 1-layer transformer model. larger models trained from scratch did not improve performance. unsupervised pre-training baseline monolingual data is generally far easier to obtain than bilingual data, which makes unsupervised pre-training techniques more attractive. interestingly, @xcite and @xcite find that pre-training bert models on a combination of languages can lead to surprisingly effective crosslingual performance on nlu tasks, without using any parallel data. of the myriad unsupervised techniques, we choose the span masking objective employed by t5 @xcite , mass @xcite etc. for our baseline since it has been shown to outperform other alternatives like bert. during pre-training, spans of text are masked in the input sentence and fed to the encoder. the decoder must learn to output the masked spans. tgen is a freely available open-source nlg system based on seq2seq + attention. @xcite create a pipelined system consisting of : a tgen based model that outputs delexicalized text, a classifier that ranks the beam search hypotheses and a language model which does the lexicalization by picking the exact surface form. we denote this combined system, consisting of all 3 components as tgen-sota. it is also currently the state-of-the-art for the data-to-text corpus that we use for downstream evaluation. note that the lexicalization step requires access to lexicon data containing all the morphological forms of words and entities. unlike tgen-sota, our proposed model is trained end-to-end to directly generate lexicalized outputs, which is a much harder task. we also do not rely on any external lexical data. its not realistic to assume that every nlg system is first developed for english. as such, our setting does not assume the existence of a similar dataset in english. therefore, translation based baselines (eg: first running the english model and then translating the output) are not applicable here. ## conclusion in this work we investigated neural machine translation based transfer learning for data-to-text generation in non-english languages. using czech as a target language, we showed that such an approach enables us to learn simple, fully lexicalized end-toend models that outperform competitive baselines. experimental results suggest several desirable properties including improved sample efficiency, robustness to unseen values and potential applications to low resource languages. at the same time, the approach can also be leveraged to improve performance of delexicalized models. studying pre-training on a wide variety of languages, especially those with different scripts, is a direct line of future work. combining unsupervised and translation based pre-training is also a promising avenue and has already shown good results for nlu tasks @xcite .