Active Learning for Multilingual Semantic Parser (2023)

Abstract: Current multilingual semantic parsing (MSP) datasets are almost all collected by translating the utterances in the existing datasets from the resource-rich language to the target language. However, manual translation is costly. To reduce the translation effort, this paper proposes the first active learning procedure for MSP (AL-MSP). AL-MSP selects only a subset from the existing datasets to be translated. We also propose a novel selection method that prioritizes the examples diversifying the logical form structures with more lexical choices, and a novel hyperparameter tuning method that needs no extra annotation cost. Our experiments show that AL-MSP significantly reduces translation costs with ideal selection methods. Our selection method with proper hyperparameters yields better parsing performance than the other baselines on two multilingual datasets.

## introduction multilingual semantic parsing converts multilingual natural language utterances into logical forms (lfs) using a single model. however, there is a severe data imbalance among the msp datasets. currently, most semantic parsing datasets are in english, while only a limited number of non-english datasets exist. to tackle the data imbalance issue, almost all current efforts build msp datasets by translating utterances in the existing datasets from the resource-rich language (e.g. english) into other languages @xcite . however, manual translation is slow and laborious. in such cases, active learning is an excellent solution to lower the translation cost. active learning (al) is a family of methods that collects training data when the annotation budgets are limited @xcite . our work proposes the first active learning approach for msp. compared to translating the full dataset, al-msp aims to select only a subset from the ex-isting dataset to be translated, which significantly reduces the translation cost. we further study which examples al-msp should select to optimize multilingual parsing performance. @xcite demonstrated that a training set with diverse lf structures significantly enhances compositional generalization of the parsers. furthermore, our experiments show that the examples with lfs aligned with more diversified lexical variants in the training set considerably improve the performance of multilingual parsing during al. motivated by both, we propose a novel strategy for selecting the instances which include diversified lf structures with more lexical choices. our selection method yields better parsing performance than the other baselines. by translating just 32% of all examples, the parser achieves comparable performance on multilingual geo-query and nlmap as translating full datasets. prior works obtain the hyperparameters of the al methods by either copying configurations from comparable settings or tuning the hyperparameters on the seed evaluation data @xcite . however, the former method is not suitable as our al setting is unique, whereas the second method requires extra annotation costs. in this work, we provide a cost-free method for our al scenario for obtaining optimal hyperparameters. our contributions are i) the first active learning procedure for msp that reduces the translation effort, ii) an approach that selects examples for getting superior parsing performance, and iii) a hyperparameter tuning method for the selection that does not incur any extra annotation costs. the model is trained on the utterance-lf pairs {(x i , y i )} n @xmath0. ## active learning for msp al-msp considers only a bilingual scenario for the proof of concept, while extending our al method to more than two languages is easy. the goal of al-msp is to minimize the human effort in translating utterances while the semantic parser can still achieve a certain level of performance on the bilingual test sets. starting from a semantic parser initially trained on the dataset d @xmath1, y i )} n @xmath2, al-msp selects k q examples @xmath3, y i )} kq @xmath4, followed by manually translating the utterances in ds into a target language t, denoted by @xmath5, where @xmath6, y i )} @xcite . each example e @xmath7, y) is featurized by extracting all the atoms and compounds in the lf tree τ y , followed by calculating the tf-idf @xcite ) value for each atom and compound. incremental k-means considers each example of d l s as a fixed clustering centroids and estimates k q new cluster centroids. for each of the k q new clusters, we select one example closest to the centroid. such selection strategy is reformulated as selecting k q examples with the highest acquisition scores one by one at each iteration: ## experiments datasets. we experiment with multilingual geo-query and nlmap. geoquery utterances are in english (en), german (de), thai (th), and greek (el); nlmap utterances are in english and german. neither corpora include a development set, so we use 20% of the training sets of geo-query and nlmap in each language as the development sets for tuning the hyperparameters. to simulate al process, we consider english as the resource-rich language and others as the target languages. after the examples are selected from the english datasets, we reveal their translations in the target languages and add them to the training sets. al setting. we perform six iterations, accumulatively selecting 1%, 2%, 4%, 8%, 16% and 32% of examples from english geoquery and nlmap. baselines. we compare four selection baselines and the oracle setting: i) random picks english utterances randomly to be translated, ii) s2s (fw) @xcite selects examples with the lowest parser confidence on their lfs, iii) csse @xcite selects the most rep-resentative and diversified utterances for machine translation, iv) max compound @xcite selects examples that diversify the atoms and compounds in the lfs, v) oracle trains the parser on the full bilingual training set. evaluation. we adopt the exact match accuracy of lfs for all the experiments. we only report the parser accuracy on the target languages as we found the influence of new data is negligible to the parser accuracy on english data (see appendix a.2). base parser. we employ bert-lstm @xcite as our multilingual parser. please see appendix a.1 for its detailed description. ## conclusion we conducted the first in-depth empirical study to investigate active learning for multilingual semantic parsing. in addition, we proposed a method to select examples that maximize msp performance and a cost-free hyperparameter tuning method. our experiments showed that our method with the proper hyperparameters selects better examples than the other baselines. our al procedure with the ideal example selection significantly reduced the translation effort for the data collection of msp.