T ranslate & F ill: I mproving Zero-Shot Multilingual Semantic Parsing with Synthetic Data (2021)

Abstract: While multilingual pretrained language models (LMs) fine-tuned on a single language have shown substantial cross-lingual task transfer capabilities, there is still a wide performance gap in semantic parsing tasks when target language supervision is available. In this paper, we propose a novel Translate-and-Fill (TaF) method to produce silver training data for a multilingual semantic parser. This method simplifies the popular Translate-Align-Project (TAP) pipeline and consists of a sequence-to-sequence filler model that constructs a full parse conditioned on an utterance and a view of the same parse. Our filler is trained on English data only but can accurately complete instances in other languages (i.e., translations of the English training utterances), in a zero-shot fashion. Experimental results on three multilingual semantic parsing datasets show that data augmentation with TaF reaches accuracies competitive with similar systems which rely on traditional alignment techniques.

## introduction semantic parsing is a core task in virtual assistants, popular applications that require accurate natural language understanding (nlu). user utterances are parsed into a structured representation made of intents and slots that is interpreted to initiate an action on the user device. for example, the sentence "set an 8 am alarm" could lead to the following interpretation -create_alarm(@xmath0. as in many nlp tasks, numerous english parsing datasets are available and well studied @xcite @xcite @xcite @xcite . supporting new domains and schemas requires a sizeable data collection effort and while english is receiving the most attention, it is also important to extend nlu to other languages in order to provide users con-sistent experiences across languages. multilingual pretrained language models (lms) are an excellent starting point for enabling cross-lingual transfer in a parser but they are no substitute for using high quality, albeit costly to annotate, training data in the target languages. without such data, we can translate the available annotated examples to other languages and slot annotations can be transferred @xcite . traditionally, annotation transfer requires (i) token alignment models @xcite , which may have been trained on text tokenized differently from the annotated training data, and (ii) label projection logic that can be complex, especially if it includes heuristics for fixing systematic alignment errors, or if nested structures need to be mapped. in this work, we propose an alternative approach to the classical translate-align-project (tap) pipeline: we leverage multilingual pretrained representations and a sequence-to-sequence (seq2seq) model to directly generate the parse of translated examples in a zero-shot fashion. our model is trained on english data only and it is able to reconstruct the full parse while having access to the english utterance and to a signature (or view) of the full parse. at inference, we substitute the english utterance with its translation and our model, pulling content from the latter, is able to construct a high quality silver parse. the main contributions of this paper can be summarized as follows: our work is closely related to two research areas: one way to achieve cross-lingual transfer is by adopting multilingual representations and models pretrained on a large amount of text in different languages. this way, similar languages with overlapping vocabularies at word or subword level can benefit from information sharing. these models can encode the input using words @xcite , characters or subwords @xcite @xcite . with the latter, interesting zero-shot performance (i.e., training on a language and evaluating on a different target language) can be achieved, especially between similar languages @xcite . multilingual representations can be obtained from encoders pretrained on multilingual corpora with tasks such as masked language modeling (mlm), or trained on supervised tasks such as neural machine translation (nmt) @xcite @xcite . after the success of fill-in-the-blankstyle denoising objectives and bert/mbert @xcite , other multilingual encoders achieved a similar level of popularity. these models include xlm @xcite , xlm-r @xcite and a recent multilingual version of t5 @xcite , named mt5 @xcite . t5 based models differ from the others by their seq2seq architecture where both the encoder and the decoder are pretrained with the mlm task. in this work, we leverage the multilinguality and the generative capabilities of mt5 to produce interpretations and create synthetic internationalization (i18n) data for semantic parsing. a second way to improve cross-lingual transfer is data augmentation. typically, annotated data is available in at least one language, and more often than not, this is a high-resource language such as english. nmt is a strong data augmentation baseline, as shown in recent cross-lingual evaluation benchmarks @xcite . nmt can be used to translate training examples from a source to a target language (translate-train), creating training data in the target language. otherwise, it can be used to translate the test data to the language of the trained model (translate-test). while translating works quite well for classification tasks where the label is at instance level, for sequence tagging or parsing tasks the reality is more challenging since the labels are at token level and they have to be transferred from the tokens of the original text to the tokens of its translation. prior work relies on word aligners to establish a match between the tokens of source and translated text, and to transfer the labels @xcite @xcite . alignment methods include unsupervised word alignment @xcite @xcite , the use of attention weights from nmt models @xcite @xcite or computing the similarity between word embeddings @xcite . in this work, we propose an alternative and novel label projection method that leverages the signatures of available parses for internationalization, in the spirit of sketch or template decoding @xcite @xcite ). our method avoids alignment models altogether and leverages multilingual representations and instance labels to generate high quality silver data that can be finally used to train accurate multilingual semantic parsers. in addition and differently from nmt attention-based aligners, our method does not access the internals of neural translation models and therefore has a wider applicability. 3 translate-and-fill (taf) we address the problem of the i18n of semantic parsers when (i) english training data is available and (ii) high quality and cost-effective training data in other languages is needed. we translate english data to a target language using nmt. this leaves us with the problem of mapping original slot annotations to translations. our solution is a novel method that we call translate-and-fill (taf), which replaces the align and project modules of the popular translate-align-project (tap) pipeline while leveraging multilingual pretraining. in our approach, we use two seq2seq models trained differently: one is the usual semantic parser and the other is what we call the filler. figure 1 shows the example instances used to train the semantic parser and our filler, and then to run inference with the latter. the example parse has a create_alarm intent (in:) and a single date_time slot (sl:). we transform a training instance for the semantic parser that maps an utterance to its parse (a) into a training instance for the filler. a filler training instance (b) maps the english utterance concatenated with its parse signature to a full parse (target from a). to obtain the parse signature, we simply remove all the slot values from the parse. the filler must then reproduce the input signature while filling the signature slots with words from the input utterance. we leverage pretrained multilingual seq2seq models (in particular mt5) to train the filler model with only english filler instances. a trained filler can be used to obtain labeled semantic parsing data in other languages, thanks to the cross-lingual transfer capabilities of the pretrained seq2seq model, as well as the slot-filling capabilities gathered from the english training filler instances. we construct an inference example for the filler from the same examples used to train it (b) by simply replacing the english utterance in the input part with its corresponding translation (c). the filler will now reproduce the input signature but fill the slots using words from the translation (d). finally, we create a synthetic i18n instance for training a parser for the target language. the synthetic instance maps the translated utterance to the parse produced by the filler at inference (e). similar to tap, our basic assumption is that the parse structure of a translated sentence does not change. the proposed approach (i) can be applied to any language supported by nmt and by the pretrained seq2seq model; (ii) can handle nested interpretations naturally thanks to the seq2seq formulation; and (iii) since it has access to the interpretation, it can learn label specific projection strategies as opposed to the handcrafted tap projection rules. ## analysis of the filler output in this section, we analyze the output of our filler trained on the english mtop data and run on translated mtop. we use two simple heuristic filters to understand how good the filler is at reproducing the signature provided in the input and how much it suffers from hallucination. therefore we count (i) how many times the input and output signatures differ (ignoring slot orders); (ii) for how many utterances the output slots contain word spans which cannot be found in the input utterance.