Zero-Shot Cross-lingual Semantic Parsing (2022)

Abstract: Recent work in cross-lingual semantic parsing has successfully applied machine translation to localize parsers to new languages. However, these advances assume access to high-quality machine translation systems and word alignment tools. We remove these assumptions and study cross-lingual semantic parsing as a zero-shot problem, without parallel data (i.e., utterance-logical form pairs) for new languages. We propose a multi-task encoder-decoder model to transfer parsing knowledge to additional languages using only English-logical form paired data and in-domain natural language corpora in each new language. Our model encourages language-agnostic encodings by jointly optimizing for logical-form generation with auxiliary objectives designed for cross-lingual latent representation alignment. Our parser performs significantly above translation-based baselines and, in some cases, competes with the supervised upper-bound.

## introduction executable semantic parsing maps a natural language utterance to a logical form (lf) for execution in some knowledge base to return a denotation. the parsing task renders an utterance as a semantically identical, but machine-interpretable, expression grounded in a denotation. the transduction between natural and formal languages has allowed semantic parsers to become critical infrastructure in building human-computer interfaces for question answering, @xcite @xcite , dialog systems @xcite , and robotics @xcite . recent advances in semantic parsing have improved accuracy for neural parsers @xcite @xcite and examined their generalization capabilities with new dataset challenges (zhong et al., 2017; yu 1 our code and data are available at github.com/tomsherborne/zx-parse. list flights from san francisco to pittsburgh? select di sti nct f l i ght _1. f l i ght _i d from? select select select w here * from select; ; ; ; latent representation encoder decoder latent representation figure 1 : accurate cross-lingual semantic parsing requires alignment of latent semantic representations across languages. the encoder generates a representation of the english utterance (blue points) to condition upon during decoding. producing the same logical form from the equivalent chinese utterance requires a similar encoding. however, without alignment, the representation may partially match (purple points) or not at all (red points), leading the decoder to generate an inaccurate, ill-formed query. @xcite , in addition to considering languages other than english @xcite . prior work largely assumes that utterance-logical form training data is parallel in all languages @xcite , or must be created with human translation (susanto and @xcite . this entry barrier to localization for new languages has motivated the exploration of machine translation (mt) as an economical alternative @xcite . however, mt can introduce performance-limiting artifacts and struggle to accurately model native speakers @xcite . additionally, high-quality machine translation is less viable for lower resource languages, further limiting the appeal of mt-based approaches. in this work, we propose a new approach for zero-shot executable semantic parsing. our method maximizes the success of cross-lingual transfer for a parser, trained on english paired data (@xmath0, to accurately generate logical forms from new languages (@xmath1. our goal is to parse utterances in a new language, l, without observing paired training data for this language, suitable machine translation, or bilingual dictionaries between l and english. our critical dependencies are a pre-trained language model and utterancelogical form paired data for a source language (i.e., english). aside from the zero-shot problem which is hard on its own (since paired data is not available for new languages), our semantic parsing challenge is further compounded with the difficulties inherent to structured prediction and the deficiency of copying strategies without gold tokenlevel alignment @xcite . we conceptualize cross-lingual semantic parsing as a latent representation alignment problem. as illustrated in figure 1 , we wish to encode different languages to an overlapping latent space for the decoder to have any chance at generating accurate logical forms. to achieve this, we train a decoder, conditioned upon encodings from a source language (e.g., english), to generate logical forms and simultaneously train encodings of a new language (e.g., chinese) to be maximally similar to english. we hypothesize that if latent representations are aligned from a language-agnostic encoder, one can generate accurate logical forms from a new language without semantic parsing training data and thus eliminate the errors outlined in figure 1 . our approach adopts a multi-task learning paradigm and trains a parser with auxiliary objectives, optimized to converge representations of additional new languages. we encourage languageagnostic representations by jointly optimizing for generating logical forms, reconstructing natural language, and promoting language invariance. our intuition is that auxiliary losses can be exploited to induce similarity in a multi-lingual latent space. the effect of such alignment is that a decoder, trained only on english, can recognize an encoding from another language and generate the relevant logical form. similar multi-task approaches have been successful in spoken-language understanding @xcite ), text simplification @xcite , dependency parsing @xcite , and machine translation @xcite . this work, to our knowledge is the first attempt to devise auxiliary objectives for executable semantic parsing as a zero-shot task. our framework and hypothesis are also sufficiently flexible for application in additional zero-shot sequence transduction tasks. our motivation is to improve parsing for non-english languages with maximal resource efficiency and minimal external dependencies beyond native-speaker utterances. we, therefore, induce a shared multilingual space without resorting to machine translation @xcite and argue that our approach is superior because it (a) nullifies the introduction of translation or word alignment errors and (b) scales to low-resource languages without reliable mt. experimental results on overnight @xcite and a new executable version of multiatis++ show that our parser generates more accurate logical forms with a minimized cross-lingual transfer penalty from english to french (fr), portuguese (pt), spanish (es), german (de), chinese (zh), hindi (hi), and turkish (tr). pre-training has shown to be widely beneficial for a wide range of cross-lingual models @xcite . by virtue of being trained on massive corpora, these models purportedly learn an overlapping cross-lingual latent space @xcite but have also been identified as under-trained for some tasks @xcite , shown poor zero-shot performance, especially for languages dissimilar to english @xcite , and high variance @xcite . semantic parsing most previous work @xcite has focused on multilingual semantic parsing, i.e., learning from multiple natural languages in parallel, largely affirming the benefit of "high-resource" multilingual data and multi-language ensemble training @xcite . @xcite further improved cross-lingual similarity with adversarial language identification across such ensembled training data. code-switching in multilingual parsing has also been explored through mixed-language training datasets @xcite . to adapt a parser to new languages, machine translation has been used as a reasonable proxy for in-language data @xcite . however, machine translation, in either direction can introduce limiting artifacts @xcite with poor generalization due to how "translationese" training data diverges from gold test utterances @xcite . zero-shot parsing has primarily focused on 'cross-domain' challenges to improve generalization across varying query structures and lexicons @xcite or different databases @xcite @xcite . the combination of zero-shot parsing with cross-lingual modeling has also been examined for the ucca formalism @xcite and for task-oriented dialogue systems (see below). dialog modeling cross-lingual transfer has been studied in the context of goal-oriented dialog for the spoken language understanding (slu) tasks of intent classification and slot labeling (i.e., parsing an utterance into a semantic frame identifying the user's intent and its arguments). recently released multilingual datasets like multiatis++ @xcite and mtop @xcite have facilitated the study of zero-shot transfer through the combination of pre-training, machine translation, and word alignment (to project annotations between languages). recent work in this setting @xcite @xcite identifies a penalty for cross-lingual transfer that neither pre-training nor machine translation can fully overcome. ## problem formulation the primary challenge for cross-lingual parsing is learning parameters that can parse an utterance, x, from an unseen test language to an accurate logical form (lf). typically, a parser trained on language l, or multiple languages {l 1 , . . . , l n }, is only capable for these languages and performs poorly outside this set. for a new language, prior approaches require parallel datasets and models @xcite @xcite . in our work, zero-shot parsing refers to parsing utterances in new languages without paired data during training, for some language, l, there exists no pairing of x l to a logical form, y, except for english. 2 this setting also excludes "silver-standard" training pairs created using machine-translation. as these models have ultimately observed some form of utterance-logical form pairs for each new language, we do not consider such approaches here and refer to @xcite as an example of using mt for this task. it might be tempting to approach this problem as a case of fine-tuning a pre-trained (english) decoder for lf generation. problematically, the output target is expressed in a formally defined language (e.g., sql or λ-dcs) which models the semantics of questions very differently to natural language (e.g., without presumption or co-operation; @xcite . formal languages @xcite additionally present artifacts which render fine-tuning challenging such as unfamiliar syntax (e.g., table aliases or explicit recursion) and long output sequences. in practice, we observed finetuning leads to poor performance (e.g., < 1% accuracy on all languages), with the model insisting on hallucinating natural language. this is seemingly at odds with adjacent work in dialog modeling, which has found pre-trained decoders to be beneficial @xcite . however, slu requires learning a lightweight label vocabulary compared to the 200+ tokens required in lfs. additionally, slu typically maintains output sequences of similar size to natural language inputs (with tightly coupled syntactic compositionality between the two), whereas the syntactic and structural demands of lf generation are largely divorced from the input utterance. in our solution, the model is trained to parse from utterance-logical forms pairs only in english. other languages are incorporated using auxiliary objectives and data detailed in section 4. we explore the hypothesis that an overlapping multi-lingual latent space can be learned through auxiliary objectives in tandem with logical form generation (see figure 2 ). our intuition is that introducing these additional losses minimizes cross-lingual variance in latent encoding space by optimizing for language-agnostic representations with high similarity to the source language (i.e., english). our approach minimizes the cross-lingual transfer penalty such that the zeroshot parser predicts logical forms from test inputs regardless of utterance language. by framing the cross-lingual parsing task as a latent representation alignment challenge, we explore a possible upper bound of parsing accuracy without errors from external dependencies. section 6 demonstrates that our zero-shot model, using only english paired data and a small additional corpus, can generate accurate logical forms above translation baselines to compete with fully supervised in-language training. 4 our zero-shot model: zx-parse we adopt a multi-task sequence-to-sequence model @xcite which combines logical form generation with two auxiliary objectives. the first is a language identification discriminator and the second is a reconstruction or translation decoder. an overview of our semantic parser is given in figure 2 ; we describe each component below. generating logical forms predicting logical forms is the primary objective for our model. given an utterance @xmath2, x 2 , . . . , x t ), we wish to generate logical form @xmath3, y 2 , . . . , y m ) representing the same meaning in a machine-executable language. we model this transduction task using an encoder-decoder neural network @xcite based upon the transformer architecture @xcite . the sequence x is encoded to a latent representation @xmath4, z 2 , . . . , z t ) through equation (1) using a stacked self-attention transformer encoder, e, with weights θ e . the conditional probability of the output sequence y is expressed in equation ( 2 ) as each token y i is autoregressively generated based upon z and prior outputs, @xmath5. equation (3) models distribution p (y i |@xmath6, x) using a transformer decoder for logical forms, d lf , with associated weights θ d lf where soft is the softmax function. we predict an output, ŷ, for semantic parsing dataset s @xmath7, y n } n @xmath8, through the encoder and logical form decoder, {e, d lf }. equation ( 4 ) describes the loss objective minimizing the crossentropy between y and ŷ. language prediction our first additional objective encourages language-agnostic representations by reducing the discriminability of the source language, l, from z. equation ( 5 ) defines a language prediction (lp) network to predict l from z using a linear classifier over l training languages: finally, equation ( 7 ) describes the objective function for the lp network: however, we reverse this gradient in the backward pass before the lp network, to encourage the encoder to produce language invariant representations @xcite . the lp network is optimized to discriminate the source language from z, but the encoder is now optimized adversarially against this objective. our intuition is that discouraging language discriminability in z encourages latent representation similarity across languages, and therefore reduces the penalty for cross-lingual transfer. ## experimental setup semantic parsing datasets our experiments examine whether our zero-shot approach generalizes across languages and domains. we evaluate performance on a new version of the atis dataset of travel queries @xcite . we align existing english utterances and sql logical forms from @xcite to the multi-lingual utterances from the multiatis++ dataset for spoken language understanding @xcite . this alignment adds executable sql queries to utterances in chinese (zh), german (de), french (fr), spanish (es), and portuguese (pt). we use the same 4,473/493/448 dataset split for training/validation/test as @xcite . we also add to the test set hindi (hi) and turkish (tr) utterances from @xcite . 3 we can now predict sql from the atis test questions in eight natural languages. the multi-atis++ japanese set was excluded as the utterance alignment to this language was not recoverable. we also examine overnight @xcite , an eight-domain dataset covering basketball, blocks, calendar, housing, publications, recipes, restaurants, and social network domains. overnight comprises 13,682 english utterances paired with λ-dcs logical forms, executable in sempre @xcite , split into 8,754/2,188/2,740 for training/validation/test respectively. this training data exists only in en and we use the zh and de test data from @xcite for multilingual evaluation. given the varying linguistic phenomena across domains (e.g. relative spatial reasoning in blocks or temporal arithmetic in calendar), this dataset presents a harder challenge for cross-lingual transfer. we measure performance with denotation accuracy as all inferred logical forms are executable in some knowledge base. this metric compares the retrieved denotation from the prediction, ŷ, to that from executing the gold-standard logical form. dataset sizes are outlined in appendix a. natural language data for the reconstruction objective, we used the mkqa corpus @xcite , a multi-lingual translation of 10,000 samples from naturalquestions @xcite . this is suitable for our auxiliary objective as the utterances are native-speaker question surface forms, matching our test set while varying in subject. mkqa is also balanced across new languages to limit overexposure bias to one new language. for bi-text, we use the original english and the professionally translated question as a pair. we also report experiments using a sample of crawled data from paracrawl 7.1 @xcite . the sample comprises 10,000 web scraped sentences paired with equivalent english to form bitext. note that these samples are mostly declarative sentences and as such do not match the surface form of our test inputs (i.e., questions) and are also not parallel between sampled languages. we contrast this to mkqa to examine how the style of natural language data influences performance. for atis experiments, we use 60,000 utterances from each source in languages with training data (en, fr, pt, es, de, zh). for overnight, we use 3 misalignment between atis versions result in the test sets containing 442 and 381 utterances for hi and tr respectively. 30,000 utterances in en, de, and zh. ## results our results are outlined to answer four core questions, with additional ablations in appendix b. our findings support the hypothesis that we can minimize the cross-lingual transfer penalty by improving latent alignment with auxiliary objectives. we also examine the latent space directly and find zx-parse learns more similar representations between languages. our parser achieves state-of-theart zero-shot results for all non-english languages in the multiatis++ and overnight benchmarks. ## conclusion we presented a multi-task model for zero-shot cross-lingual semantic parsing which combines logical form generation with auxiliary objectives that require only modest natural language corpora for localization. through aligning latent representations, zx-parse minimizes the error from cross-lingual transfer and improves accuracy across languages unseen during training. although we focused exclusively on executable semantic parsing, our approach is general and potentially relevant for linguistically motivated frameworks such as abstract meaning representation @xcite or discourse representation theory @xcite . in the future, we will investigate a few-shot scenario and study sample efficient cross-lingual transfer by explicitly promoting generalization using techniques such as meta-learning @xcite . @xcite @xcite and overnight @xcite . each example is an utterance paired with a logical form.