The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning (2023)

Abstract: Multilingual semantic parsing aims to leverage the knowledge from the high-resource languages to improve low-resource semantic parsing, yet commonly suffers from the data imbalance problem. Prior works propose to utilize the translations by either humans or machines to alleviate such issues. However, human translations are expensive, while machine translations are cheap but prone to error and bias. In this work, we propose an active learning approach that exploits the strengths of both human and machine translations by iteratively adding small batches of human translations into the machine-translated training set. Besides, we propose novel aggregated acquisition criteria that help our active learning method select utterances to be manually translated. Our experiments demonstrate that an ideal utterance selection can significantly reduce the error and bias in the translated data, resulting in higher parser accuracies than the parsers merely trained on the machine-translated data.

## introduction multilingual semantic parsing allows a single model to convert natural language utterances from multiple languages into logical forms (lfs). due to its wide applications in various research areas, e.g. multilingual question answering and multilingual virtual assistant, multilingual semantic parsing has drawn more attention recently @xcite @xcite . training a multilingual semantic parser (msp) requires training data from all target languages. however, there is a severe imbalance of data availability among languages for current multilingual semantic parsing research. the utterances in most current semantic parsing datasets are in english, while non-english data is scarce. to overcome the data imbalance issue, prior studies translate utterances in the msp datasets from high-resource languages (e.g. english) to the target low-resource languages of interest by either human translators @xcite @xcite or automatic machine translation (mt) @xcite . unfortunately, human translation (ht), though effective, is cost-intensive and timeconsuming. while the cost of mts is much lower than that of hts, the low quality of the machinetranslated utterances severely weakens the performance of the msps in the target languages. we observe that the quality of mts is lower than that of hts, mainly due to translation bias and errors. first, mt systems are likely to be influenced by algorithmic bias. hence, the outputs of mt systems are generally less lexically and morphologically diverse than human translations @xcite . so, there is a lexical distribution discrepancy between the machine-translated and the human-generated utterances. second, mt systems are prone to generate translations with errors @xcite . prior study @xcite demonstrates that adding only a small portion of humantranslated data into the complete set of machinetranslated training data significantly improves the msp performance on the test set of the target language. given this observation, we propose a novel annotation strategy based on active learning (al) that benefits from both human translations and automatic machine translations (hat). it initially machine-translates all utterances in training sets from the high-resource languages to target languages. then, for each iteration, hat selects a subset of utterances from the original training set to be translated by human translators, followed by adding the ht data to the mt training data. the multilingual parser is trained on the combination of both types of translated data. we further investigate how hat can select utterances whose hts maximally benefit the parser performance. we assume the performance improve-ment is ascribed to the less biased and erroneous training set in a mixture of the mt and ht data. we have found that resolving the bias and error issues for the translations of the most semantically diversified and representative utterances improves the parser performance to the greatest extent. given this assumption, we provide an aggregated acquisition function that scores the utterances on how much their hts can mitigate the bias and error issues for learning the multilingual parsers (abe). it aggregates four individual acquisition functions, two of which measure the error and bias degree for the translations of the source utterances. the other two encourage the selection of the most representative and semantically diversified utterances. our key contributions are as follows: active learning. al is to select the most valuable unlabeled instances to be annotated in order to maximize the model's performance and hence reduce the annotation cost for data-hungry machine learning models. al has been used to mt @xcite , sequence labelling @xcite ), text classification @xcite , and semantic parsing @xcite @xcite . following most deep learning al methods @xcite @xcite , our work also adopts a pool-based query strategy, which means we sample batches from a large pool of unlabelled data instead of evaluating examples one by one from an incoming stream. among all the al for semantic parsing works, @xcite is the one most similar to ours, which selects utterances to be translated. however, they do not utilize mt systems. ## multilingual semantic parsing with automatic machine translation an msp is a parametric model p θ (y|x) that maps a natural language utterance @xmath0, where @xmath1. the standard training objective for a multilingual parser is, ## combining human and automatic translations with active learning to mitigate the negative effect of translation bias and error in the mt data, we propose hat, which introduces extra human supervision to machine supervision when training the msps. two major intuitions motivate our training approach: our hat al setting considers only the bilingual scenario. one of the languages is in high-resource, and the other one is in low-resource. however, it is easy to extend our method to more than two languages. we assume access to a well-trained black-box multilingual mt system, g mt (•), and a semantic parsing training set that includes utterances in a high-resource language l s (e.g. english) paired with lfs, with utterances in high and low-resource languages, respectively. each utterance x s in d s is translated into the utterance @xmath2, @xmath3, y i )} n @xmath4. the goal of our al method is to select an optimal set of utterances from the training data in the source language, @xmath5, and ask human translators to translate them into the target language, denoted by @xmath6, for training a semantic parser on the union of dt and dt . the selection criterion is based on the acquisi-tion functions that score the source utterances. following the conventional batch al setting @xcite , there are q selection rounds. at the qth round, al selects utterances with a budget size of k q . the detailed hat al procedure iteratively performs the following steps as in algorithm. 1. algorithm 1: hat procedure input :initial training set d 0 = ds ∪ dt, source utterance pool ds, budget size kq, number of selection rounds q, human annotators g ht (•) output :a well-trained multilingual parser p θ (y|x) # train the initial parser on the initial data update θ of p θ (y|x) with ∇ θ l(θ) on d 0 evaluate p θ (y|x) on ts and tt estimate the acquisition function ϕ(•) d0 @xmath7, kq) dq @xmath8 ## experiments datasets. we evaluate our al method for msp on two datasets, geoquery (susanto and lu, 2017) and nlmap @xcite with multilingual human-translated versions. geo-query includes 600 utterances-lf pairs as the training set and 280 pairs as the test set. nlmap includes 1500 training examples and 880 test examples. in our work, we consider english as the resourcerich source language and use google translate system foot_0 to translate all english utterances in geo-query into german (de), thai (th), greek (el) and the ones in nlmap into german, respectively. the al methods actively sample english utterances, the hts of which are obtained from the multilingual geoquery and nlmap. active learning setting. the hat active learning procedure performs five rounds of query, which accumulatively samples 1%, 2%, 4%, 8% and 16% of total english utterances in geoquery and nlmap. we only perform five rounds as we found the performance of the multilingual parser is saturated after sampling 16% of examples with most acquisition functions. base parser. we use bert-lstm as our multilingual parser @xcite . it is a seq2seq model with the copy mechanism @xcite that applies multilingual bert-base @xcite as the encoder and lstm @xcite as the decoder. baselines. we compare abe with eight acquisition baselines and an oracle baseline. 1. random randomly selects english utterances in each round. evaluation. we evaluate the al methods by measuring the accuracy of the msp, the bias of the training set, and the semantic discrepancy rate between the selected utterances and their bts. ## conclusion we have tackled the problem of data imbalance when adapting an msp to a low-resource language. we presented methods to efficiently collect a small amount of human-translated data to reduce bias and error in the training data, assuming a realistic scenario with an mt system and budget constraints for human annotation. our experiments show that by manually translating only 16% of the dataset, the parser trained on this mixed data outperforms parsers trained solely on machine-translated data and performs similarly to the parser trained on a complete human-translated set.