Multilingual Neural Semantic Parsing for Low-Resourced Languages (2021)

Abstract: Multilingual semantic parsing is a cost-effective method that allows a single model to understand different languages. However, researchers face a great imbalance of availability of training data, with English being resource rich, and other languages having much less data. To tackle the data limitation problem, we propose using machine translation to bootstrap multilingual training data from the more abundant English data. To compensate for the data quality of machine translated training data, we utilize transfer learning from pretrained multilingual encoders to further improve the model. To evaluate our multilingual models on human-written sentences as opposed to machine translated ones, we introduce a new multilingual semantic parsing dataset in English, Italian and Japanese based on the Facebook Task Oriented Parsing (TOP) dataset. We show that joint multilingual training with pretrained encoders substantially outperforms our baselines on the TOP dataset and outperforms the state-of-the-art model on the public NLMaps dataset. We also establish a new baseline for zero-shot learning on the TOP dataset. We find that a semantic parser trained only on English data achieves a zero-shot performance of 44.9% exact-match accuracy on Italian sentences.

## introduction semantic parsing is defined as the task of parsing a natural language sentence into a logical form that represents its meaning. the logical form, or sometimes called the meaning representation language (mrl) expression, can be executed against a knowledge base to extract information; therefore, semantic parsing often finds its application in question answering, code generation, information retrieval, etc. due to its wide range of applications, semantic parsing has drawn a lot of research interest. among them, neural semantic parsing methods have gained popularity in recent years due to their good results @xcite . neural semantic parsing often formulates the task as a machine translation problem and uses neural networks to translate the sentences into mrl expressions. multilingual neural semantic parsing is a costeffective method that allows a single model to understand different languages. however, similar to other machine-learning based approaches, neural semantic parsing requires large amounts of training data. to understand texts in different languages, semantic parsing models need training data for each target language. unfortunately, researchers face a great imbalance of availability of training data for semantic parsing: while we have lots of data in english, the data in non-english languages is often scarce. although there is a growing number of datasets published for semantic parsing in english, very few datasets are available in other languages. moreover, manually annotating data for semantic parsing is difficult and time-consuming, as it requires a lot of training and effort for annotators to write mrls. instead of manually annotating semantic parsing data in low-resourced languages, can we bootstrap training data for multilingual semantic parsing from the more abundant english data? in this paper, we aim to tackle the data limitation problem for multilingual semantic parsing with machine translation. we machine translate english sentences into target non-english languages and make use of the alignment information in the english mrl to create mrl annotations in other languages (see section 3). we then describe our methods to build multilingual semantic parsing models on the machine translated training data (see @xcite . to train the multilingual semantic parser, we mix the training data from all languages together and train a model from scratch (see section 4.1). we base our neural semantic parser on the sequence-to-sequence model with pointer mecha-nism @xcite , where both the natural language question and the target mrl are treated as sequences of tokens and the parser learns from the training data a mapping to translate questions into mrls. the machine translation-based data generation method allows us to easily extend english data to other languages. however, the quality of the bootstrapped training data is constrained by the accuracy of the machine translation model and other components of the generation method, such as alignment. to mitigate the problem of data quality of the machine translated training data, we make use of transfer learning with pretrained multilingual encoders to further improve the multilingual semantic parsing model (see section 4.2). to evaluate the model performance on sentences written by human as opposed to machine translated ones, we introduce a new multilingual semantic parsing dataset based on the facebook task oriented parsing (top) dataset @xcite . we compare our method against several baselines, including monolingual models and a popular technique in literature that relies on translating the utterances and using an english model to understand them (see section 4.3). we report the experimental results and our analysis in section 5. to show that our multilingual semantic parsing models also work with human-generated training data and to compare them against previous work, we report the performance of our models on the public multilingual nlmaps dataset in section 5.3. apart from bootstrapping training data, zeroshot learning is also a technique that allows a multilingual model to generalize to low-resourced languages. we study how the multilingual semantic parsers with pretrained encoders can generalize to other languages in a zero-shot scenario (see section 5.4). our main contributions are as follows: the dataset is available for download at: due to the research interest in semantic parsing, many public datasets have been made available for english semantic parsing, ranging from small datasets that contain only a few hundred or a few thousand examples, such as geoquery @xcite and atis @xcite , to larger datasets with tens of thousands of questionlogical form pairs, such as wikisql @xcite and overnight @xcite . multilingual semantic parsing, however, has only begun to draw research attention in more recent years. therefore, very few datasets have been published for semantic parsing in non-english languages. so far, almost all of the multilingual semantic parsing datasets are manually translated from their english versions. due to the cost of manual translation, they are limited to small datasets. for example, @xcite translated the geo-query dataset into german, greek, and thai. susanto and lu (2017) translated the atis dataset into indonesian and chinese. @xcite created the nlmaps dataset which contains around 2,400 queries to a geographic database in english. the authors translated the queries into german but kept the mrl annotation the same as that for english. apart from the semantic parsing datasets for question answering, there are some multilingual datasets with other logical form representations, such as multilingual graphquestions with graphs as the meaning representation @xcite , parallel meaning bank with drt (discourse representation theory) representation @xcite , and multilingual amr test set with abstract meaning representation @xcite . the logical form representation in these datasets are very different from the mrls used for question answering and thus cannot be easily harnessed by many semantic parsers. among the limited literature on multilingual semantic parsing, several different methods have been proposed. the first attempts on multilingual semantic parsing @xcite use statistical/neural machine translation methods to translate non-english questions into english and rely on using an english semantic parser to parse all the utterances. annotation projection is an alternative technique to deal with the lack of multilingual data. it maps the annotation from one language to another using word alignment. it has been applied to many nlp applications, including pos tagging @xcite , role-labeling @xcite , semantic ccg parsing @xcite , and amr parsing @xcite . in addition, susanto and lu (2017) approached multilingual semantic parsing with a multi-task learning technique. they used separate encoders to encode sentences in different languages and used a shared decoder to predict the mrl. @xcite used cross-lingual word embeddings in a sequenceto-sequence model. they observed that using crosslingual word embeddings improves the results on both english and german over their baseline models on the nlmaps dataset. they also compared training a model with a single encoder on multilingual data against training with separate encoders for each language and found that keeping separate encoders actually harms semantic parsing accuracy. based on their observation, we will use a single encoder for multiple languages in our experiments. ## multilingual semantic parsing data to tackle the data scarcity problem for multilingual semantic parsing, we aim to utilize machine translation to automatically generate training data from the more abundant english data for other languages. in this section, we introduce the english semantic parsing dataset we are using and describe our strategy to bootstrap training data for multilingual semantic parsing. ## conclusion in this paper, we describe our method to build multilingual semantic parsing models when the multilingual data is limited. we introduce a new multilingual semantic parsing dataset in english, italian and japanese based on the public top dataset, with training and validation data automatically generated from english and 8k test data manually translated. the multilingual top test set is so far the largest dataset for multilingual semantic parsing, which will be useful for future research. by leveraging joint multilingual training and transfer learn-ing from pretrained encoders, our semantic parsing models outperform several baselines on the top dataset and the state-of-the-art on the nlmaps dataset. we show that semantic parsing models with pretrained multilingual encoders can generalize from english to italian with 44.9% zero-shot accuracy. however, we find that there is a gap between italian and japanese semantic parsing with our method. in future work, we plan to improve our models with both language-invariant and languagespecific encodings and apply our method to more languages.