Localizing Open-Ontology QA Semantic Parsers in a Day Using Machine Translation (2020)

Abstract: We propose Semantic Parser Localizer (SPL), a toolkit that leverages Neural Machine Translation (NMT) systems to localize a semantic parser for a new language. Our methodology is to (1) generate training data automatically in the target language by augmenting machine-translated datasets with local entities scraped from public websites, (2) add a few-shot boost of human-translated sentences and train a novel XLMR-LSTM semantic parser, and (3) test the model on natural utterances curated using human translators. We assess the effectiveness of our approach by extending the current capabilities of Schema2QA, a system for English Question Answering (QA) on the open web, to 10 new languages for the restaurants and hotels domains. Our model achieves an overall test accuracy ranging between 61% and 69% for the hotels domain and between 64% and 78% for restaurants domain, which compares favorably to 69% and 80% obtained for English parser trained on gold English data and a few examples from validation set. We show our approach outperforms the previous state-of-the-art methodology by more than 30% for hotels and 40% for restaurants with localized ontologies for the subset of languages tested. Our methodology enables any software developer to add a new language capability to a QA system for a new domain, leveraging machine translation, in less than 24 hours. Our code is released open-source.

## introduction localization is an important step in software or website development for reaching an international audience in their native language. localization is usually done through professional services that can translate text strings quickly into a wide variety of languages. as conversational agents are increasingly used as the new interface, how do we localize them to other languages efficiently? the focus of this paper is on question answering systems that use semantic parsing, where natural language is translated into a formal, executable representation (such as sql). semantic parsing typically requires a large amount of training data, which must be annotated by an expert familiar with both the natural language of interest and the formal language. the cost of acquiring such a dataset is prohibitively expensive. for english, previous work has shown it is possible to bootstrap a semantic parser without massive amount of manual annotation, by using a large, hand-curated grammar of natural language @xcite . this approach is expensive to replicate for all languages, due to the effort and expertise required to build such a grammar. hence, we investigate the question: can we leverage previous work on english semantic parsers for other languages by using machine translation? and in particular, can we do so without requiring experts in each language? the challenge is that a semantic parser localized to a new target language must understand questions using an ontology in the target language. for example, whereas a restaurant guide in new york may answer questions about restaurants near times square, the one in italy should answer questions about restaurants near the "colosseo" or "fontana di trevi" in rome, in italian. in addition, the parser must be able to generalize beyond a fixed set of ontology where sentences refer to entities in the target language that are unseen during training. we propose a methodology that leverages machine translation to localize an english semantic parser to a new target language, where the only labor required is manual translation of a few hundreds of annotated sentences to the target language. neural machine translation sto cercando un posto da " hamburger " vicino a " stagno bosco ". we apply our approach on the restaurants and hotels datasets introduced by @xcite , which contain complex queries on data scraped from major websites. we demonstrate the efficiency of our methodology by creating neural semantic parsers for 10 languages: arabic, german, spanish, persian, finnish, italian, japanese, polish, turkish, chinese. the models can answer complex questions about hotels and restaurants in the respective languages. an example of a query is shown for each language and domain in our contributions include the following: multi-lingual benchmarks previous work has shown it is possible to ask non-experts to annotate large datasets for applications such as natural language inference @xcite and machine reading @xcite , which has led to large cross-lingual benchmarks @xcite . their approach is not suitable for semantic parsing, because it requires experts that know both the formal language and the natural language. semantic parsing semantic parsing is the task of converting natural language utterances into a formal representation of its meaning. previous work on semantic parsing is abundant, with work dating back to the 70s @xcite @xcite . stateof-the-art methods, based on sequence-to-sequence neural networks, require large amounts of manually annotated data @xcite . various methods have been proposed to eliminate manually annotated data for new domains, using synthesis @xcite @xcite , transfer learning @xcite @xcite , or a combination of both @xcite . all these works focus mainly on the english language, and have not been applied to other languages. @xcite investigate cross-lingual transferability of knowledge from a source language to the target language by employing cross-lingual word embedding. they evaluate their approach on the english and german splits of nlmaps dataset @xcite and on a code-switching test set that combines english and german words in the same utterance. however, they found that joint training on english and german training data achieves competitive results compared to training multiple encoders and predicting logical form using a shared decoder. this calls for better training strategies and better use of knowledge the model can potentially learn from the dataset. the closest work to ours is bootstrap @xcite , which explores using public mt systems to generate training data for other languages. they try different training strategies and find that using a shared encoder and training on target language sentences and unmodified logical forms with english entities yields the best result. their evaluation is done on the atis @xcite and overnight @xcite datasets, in german and chinese. these two benchmarks have a very small number of entities. as a result, their method is unsuitable for the open ontology setting, where the semantic parser must detect entities not seen during training. to collect real validation and test utterances, @xcite use a three-staged process to collect data from amazon mechanical turkers (amts). they ask for three translations each per english source sentence with the hypothesis that this will collect at least one adequate translation. we found this approach to be less cost-effective than using professional translators. since this process is done for the test data, it is important for the translations to be verified and have high quality. ## multi-lingual parser generation our goal is to localize an english semantic parser for question answering that operates on an open ontology of localized entities, with no manual annotation and a limited amount of human translation. ## model description this section first describes our translation models, then the neural semantic parser we train with the generated data. ## experiments we have implemented the full spl methodology in the form of a tool. developers can use the spl tool to create a new dataset and semantic parser for their task. we evaluate our models on the schema2qa dataset @xcite , translated to other languages using our tool. we first describe our dataset and then show our tool's accuracy, both without any human-produced training data (zero-shot) and if a small amount of human-created data in the target language is available (few-shot). in our experiments, we measure the logical form exact match (em) accuracy, which considers the result to be correct only if the output matches the gold logical form token by token. we additionally measure the structure match (sm) accuracy, which measures whether the gold and predicted logical forms are identical, ignoring the parameter values. a large difference between exact and structure accuracy indicates that the parameters are poorly handled. we report results on both validation and test sets. we present the results for both restaurants and hotels domain in this paper. our toolkit uses the genie @xcite library for synthesis and data augmentation. our models were implemented using the huggingface @xcite and genienlp foot_1 libraries. ## conclusion this paper presents spl, a toolkit and methodology to extend and localize semantic parsers to a new language with higher accuracy, yet at a fraction of the cost compared to previous methods. spl was incorporated into the schema2qa toolkit @xcite to give it a multilingual capability. spl can be used by any developer to extend their qa system's current capabilities to a new language in less than 24 hours, leveraging professional services to translate the validation data and mature public nmt systems. we found our approach to be effective on a recently proposed qa semantic parsing dataset, which is significantly more challenging than other available multilingual datasets in terms of sentence complexity and ontology size. our generated datasets are automatically annotated using logical forms containing localized entities; we require no human annotations. our model outperforms the previous state-of-the-art methodology by between 30% and 40% depending on the domain and the language. our new datasets and resources are released open-source foot_3 . our methodology enables further investigation and creation of new benchmarks to trigger more research on this topic. detect the language from the input ids without requiring additional language-specific tokens. the decoder is an lstm decoder with attention and a pointer-generator. at each decoding step, the model decides whether to generate a token or copy one from the input context. we preprocess the input sentences by lowercasing all tokens except for entity placeholders such as time_0, date_0, etc. and splitting tokens on white space. the formal code tokens are also split on whitespace, but their casing is preserved. xlm-r uses the sentence piece model to tokenize input words into sub-word pieces. for the decoder, to be able to copy tokens from pretrained xlm-r vocabulary, we perform the same sub-word tokenization of parameter values in the input sentence and in the formal language. the word-pieces are then numericalized using an embedding matrix and fed into a 12-layer pretrained transformer network which outputs contextual representations of each sub-word. the representations are then aggregated using a pooling layer which calculates the final representation of the input sentence: