Bootstrapping a Crosslingual Semantic Parser (2020)

Abstract: Recent progress in semantic parsing scarcely considers languages other than English but professional translation can be prohibitively expensive. We adapt a semantic parser trained on a single language, such as English, to new languages and multiple domains with minimal annotation. We query if machine translation is an adequate substitute for training data, and extend this to investigate bootstrapping using joint training with English, paraphrasing, and multilingual pre-trained models. We develop a Transformer-based parser combining paraphrases by ensembling attention over multiple encoders and present new versions of ATIS and Overnight in German and Chinese for evaluation. Experimental results indicate that MT can approximate training data in a new language for accurate parsing when augmented with paraphrasing through multiple MT engines. Considering when MT is inadequate, we also find that using our approach achieves parsing accuracy within 2% of complete translation using only 50% of training data.

## introduction semantic parsing is the task of mapping natural language utterances to machine-interpretable expressions such as sql or a logical meaning representation. this has emerged as a key technology for developing natural language interfaces, especially in the context of question answering @xcite @xcite , where a semantically complex question is translated to an executable query to retrieve an answer, or denotation, from a knowledge base. sequence-to-sequence neural networks @xcite are a popular approach to semantic parsing, framing the task as sequence transduction from natural to formal languages @xcite . recent proposals include learning intermediate logic representations @xcite , constrained decoding @xcite @xcite , and graph-based parsing @xcite . given recent interest in semantic parsing and the data requirements of neural methods, it is unsurprising that many challenging datasets have been released in the past decade @xcite @xcite @xcite . however, these widely use english as synonymous for natural language. english is neither linguistically typical @xcite nor the most widely spoken language worldwide @xcite , but is presently the lingua franca of both utterances and knowledge bases in semantic parsing. natural language interfaces intended for international deployment must be adaptable to multiple locales beyond prototypes for english. however, it is uneconomical to create brand new datasets for every new language and domain. in this regard, most previous work has focused on multilingual semantic parsing i.e., learning from multiple natural languages in parallel assuming the availability of multilingual training data. examples of multilingual datasets include geoquery @xcite , atis @xcite and nlmaps @xcite but each is limited to one domain. for larger datasets, professional translation can be prohibitively expensive and require many man-hours from experts and native speakers. @xcite reproduced the public partitions of the spider dataset @xcite into chinese, but this required three expert annotators for verification and agreement. we posit there exists a more efficient strategy for expanding semantic parsing to a new language. in this work, we consider crosslingual semantic parsing, adapting a semantic parser trained on english, to another language. we expand executable semantic parsing to new languages and multiple domains by bootstrapping from in-task english datasets, task-agnostic multilingual resources, and publicly available machine translation (mt) services, in lieu of expert translation of training data. we investigate a core hypothesis that mt can provide a noisy, but reasonable, approximation of training data in a new source language. we further explore the benefit of augmenting noisy mt data using pre-trained models, such as bert @xcite , and multilingual training with english. additionally, we examine approaches to ensembling multiple machine translations as approximate paraphrases. this challenge combines both domain adaptation and localization, as a parser must generalize to the locale-specific style of queries using only noisy examples to learn from. for our evaluation, we present the first multidomain, executable semantic parsing dataset in three languages and an additional locale for a single-domain dataset. specifically, we extend atis @xcite , pairing chinese (zh) utterances from susanto and @xcite to sql queries and create a parallel german (de) humantranslation of the full dataset. following this, we also make available a new version of the multidomain overnight dataset @xcite where only development and test sets are translations from native speakers of chinese and german. this is representative of the real-world scenario where a semantic parser needs to be developed for new languages without gold-standard training data. our contributions can be summarized as follows: (1) new versions of atis @xcite and overnight @xcite for generating executable logical forms from chinese and german utterances; (2) a combined encoder-decoder attention mechanism to ensemble over multiple transformer encoders; (3) a cost-effective methodology for bootstrapping semantic parsers to new languages using minimal new annotation. our proposed method overcomes the paucity of gold-standard training data using pre-trained models, joint training with english, and paraphrasing through mt engines; and (4) an investigation into practical minimum gold-standard translation requirements for a fixed performance penalty when mt is unavailable. ## related work across logical formalisms, there have been several proposals for multilingual semantic parsing which employ multiple natural languages in parallel @xcite @xcite @xcite . @xcite ensemble monolingual parsers to generate a single parse @xmath0. similarly, @xcite propose a polyglot automaton decoder for source-code generation in 45 languages. susanto and lu (2017a) explore a multilingual neural architecture in four languages for geoquery and three languages for atis by extending @xcite with multilingual encoders. other work focuses on multilingual representations for semantic parsing based on universal dependencies @xcite or embeddings of logical forms @xcite . we capitalize on existing semantic parsing datasets to bootstrap from english to another language, and therefore, do not assume that multiple languages are available as parallel input. our work is closest to @xcite , however they explore how to parse both english and german simultaneously using a multilingual corpus. in contrast, we consider english data only as an augmentation to improve parsing in chinese and german and do not use "real" utterances during training. recently, @xcite studied mt for crosslingual entailment, however, our results in section 5 suggest these prior findings may not extend to semantic parsing, owing to the heightened requirement for factual consistency across translations. our work complements recent efforts in crosslingual language understanding such as xnli for entailment @xcite , semantic textual similarity @xcite or the xtreme @xcite and xglue @xcite benchmarks. there has also been interest in parsing into interlingual graphical meaning representations @xcite , spoken language understanding @xcite and λ-calculus expressions @xcite @xcite . in contrast, we focus on logical forms grounded in knowledge-bases and therefore do not consider these approaches further. ## problem formulation throughout this work, we consider the real-world scenario where a typical developer wishes to develop a semantic parser to facilitate question answering from an existing commercial database to noun/adjective ambiguity ("first-class fares" is a noun object) @xcite and overnight @xcite . utterances are translated into chinese and german using both machine translation (l mt ) and crowdsourcing with verification (l h ). we highlight issues with the noisy mt data (underlined and bolded) compared to improved human translations (underlined) for atis. crosslingual semantic parsing presents a unique challenge as an nlu task. it demands the generation of precise utterance semantics, aligned across languages while ensuring an accurate mapping between logical form and the idiomatic syntax of questions in every language under test. in com-parison to nlu classification tasks such as xnli @xcite , our challenge is to preserve and generate meaning, constrained under a noisy mt channel. the misinterpretation of entities, relationships, and relative or numerical expressions can all result in an incorrect parse. lexical translation in mt, however accurate it may be, is insufficient alone to represent queries from native speakers. for example, the english expression "dinner flights" can be directly translated to german as "abendessenflug" [dinner flight], but "flug zur abendszeit" [evening flight] better represents typical german dialogue. this issue further concerns question phrasing. for example, the english query "do you have x?" is often mistranslated to a statement "你有一个x" [you have one x] but typical chinese employs a positive-negative pattern ("有没有一个x?" [have not have one x?]) to query possession. our parser must overcome each of these challenges without access to gold data. ## data we consider two datasets in this work. firstly, we evaluate our hypothesis that mt is an adequate proxy for "real" utterances using atis @xcite . this single-domain dataset contains 5,418 utterances paired with sql queries pertaining to a us flights database. atis was previously translated into chinese by susanto and @xcite for semantic parsing into λ-calculus, whereas we present these chinese utterances aligned with sql queries from @xcite . in addition, we translate atis into german following the methodology described below. we use the split of 4,473/497/448 examples for train/validation/test from @xcite . we also examine the multi-domain overnight dataset @xcite , which contains 13,682 english questions paired with λ-dcs logical forms executable in sempre @xcite . overnight is 2.5× larger than atis, so a complete translation of this dataset would be uneconomical for our case study. as a compromise, we collect human translations in german and chinese only for the test and validation partitions of overnight. we argue that having access to limited translation data better represents the crosslingual transfer required in localizing a parser. we define a fixed development partition of a stratified 20% of the training set for a final split of 8,754/2,188/2,740 for training/validation/testing. note we consider only simplified mandarin chinese for both datasets. crowdsourcing translations the atis and overnight datasets were translated to german and chinese using amazon mechanical turk, following best practices in related work @xcite @xcite . we initially collected three translations per source sentence. submissions were restricted to turkers from germany, austria, and switzerland for german and china, usa, or singapore for chinese. our amt interface barred empty submissions and copying or pasting anywhere within the page. any attempts to bypass these controls triggered a warning message that using mt is prohibited. submissions were rejected if they @xmath1, as were nonsensical or irrelevant submissions. in a second stage, workers cross-checked translations by rating the best translation from each candidate set, including an mt reference, with a rewrite option if no candidate was satisfactory. we collected three judgements per set to extract the best candidate translation. turkers unanimously agreed on a single candidate in 87.8% of the time (across datasets). finally, as a third quality filter, we recruited bilingual native speakers to verify, rewrite, and break ties between all top candidates. annota- ## results and analysis we compare the neural model defined in section 3.1 (seq2seq) to models using each augmentation outlined in section 3.3, a combination thereof, and the back-translation baseline. to the best of our knowledge, we present the first results for executable semantic parsing of atis and overnight in any language other than english. while prior multilingual work using λ-calculus logic is not comparable, we compare to similar results for english in appendix b. atis overnight we now extend our experiments to the multi-domain overnight dataset, wherein we have only utterances from native speakers for evaluation, in the seq2seq model with bert outperforms the baseline by a considerable +28.1% for de and +32.3% for zh, further supporting the notion that an in-language parser is a more suitable strategy for the task. our reference english parser attains an average 79.8% accuracy, incurring a penalty from crosslingual transfer of -14.9% for de and -14.7% for zh with the seq2seq model. similar to atis, we find mt-ensemble as the most performant system, improving over the baseline by +32.5% and +33.7% for de and zh respectively. the best model minimises the crosslingual penalty to -5.2% for de and -8.7% for zh. across both datasets, we find that single augmentations broadly have marginal gain and combining approaches maximizes accuracy. ## conclusions we presented an investigation into bootstrapping a crosslingual semantic parser for chinese and german using only public resources. our contributions include a transformer with attention ensembling and new versions of atis and overnight in chinese and german. our experimental results showed that a) multiple mt systems can be queried to generate paraphrases and combining these with pre-trained representations and joint training with english data can yield competitive parsing accuracy; b) multiple encoders trained with shuffled inputs can outperform a single encoder; c) back-translation can underperform by losing required details in an utterance; and finally d) partial translation can yield @xmath2. our results from paraphrasing and partial translation suggest that exploring semi-supervised and zero-shot parsing techniques is an interesting avenue for future work. john m. zelle and raymond j. mooney. 1996. learning to parse database queries using inductive logic programming. in proceedings of the thirteenth national conference on artificial intelligence -volume 2, aaai'96, pages 1050-1055. sheng zhang, xutai ma, rachel rudinger, kevin duh, and benjamin van durme. 2018. cross-lingual decompositional semantic parsing. in proceedings of the 2018 conference on empirical methods in natural language processing, pages 1664-1675, brussels, belgium. victor zhong, caiming xiong, and richard socher. 2017. seq2sql: generating structured queries from natural language using reinforcement learning. corr, abs/1709.00103. yanyan zou and wei lu. 2018. learning cross-lingual distributed logical representations for semantic parsing. in proceedings of the 56th annual meeting of the association for computational linguistics (volume 2: short papers), pages 673-679, melbourne, australia.