Meta-Learning a Cross-lingual Manifold for Semantic Parsing (2023)

Abstract: Localizing a semantic parser to support new languages requires effective cross-lingual generalization. Recent work has found success with machine-translation or zero-shot methods, although these approaches can struggle to model how native speakers ask questions. We consider how to effectively leverage minimal annotated examples in new languages for few-shot cross-lingual semantic parsing. We introduce a first-order meta-learning algorithm to train a semantic parser with maximal sample efficiency during cross-lingual transfer. Our algorithm uses high-resource languages to train the parser and simultaneously optimizes for cross-lingual generalization to lower-resource languages. Results across six languages on ATIS demonstrate that our combination of generalization steps yields accurate semantic parsers sampling ≤10% of source training data in each new language. Our approach also trains a competitive model on Spider using English with generalization to Chinese similarly sampling ≤10% of training data.1

## introduction a semantic parser maps natural language (nl) utterances to logical forms (lf) or executable programs in some machine-readable language (e.g., sql). recent improvement in the capability of semantic parsers has focused on domain transfer within english @xcite , compositional generalization @xcite @xcite , and, more recently, cross-lingual methods @xcite @xcite . within cross-lingual semantic parsing, there has been an effort to bootstrap parsers with min-imal data to avoid the cost and labor required to support new languages. recent proposals include using machine translation to approximate training data for supervised learning @xcite @xcite and zero-shot models, which engineer crosslingual similarity with auxiliary losses @xcite @xcite . these shortcuts bypass costly data annotation but present limitations such as ''translationese'' artifacts from machine translation @xcite or undesirable domain shift @xcite . however, annotating a minimally sized data sample can potentially overcome these limitations while incurring significantly reduced costs compared to full dataset translation @xcite . we argue that a few-shot approach is more realistic for an engineer motivated to support additional languages for a database-as one can rapidly retrieve a high-quality sample of translations and combine these with existing supported languages (i.e., english). beyond semantic parsing, cross-lingual few-shot approaches have also succeeded at leveraging a small number of annotations within a variety of tasks @xcite including natural language inference, paraphrase identification, part-of-speech-tagging, and named-entity recognition. recently, the application of meta-learning to domain generalization has further demonstrated capability for models to adapt to new domains with small samples @xcite @xcite . in this work, we synthesize these directions into a meta-learning algorithm for cross-lingual semantic parsing. our approach explicitly optimizes for cross-lingual generalization using fewer training samples per new language without performance degradation. we also require minimal computational overhead beyond standard gradientdescent training and no external dependencies beyond in-task data and a pre-trained encoder. our algorithm, cross-lingual generalization reptile (xg-reptile) unifies two-stage meta-learning into a single process and outperforms prior and constituent methods on all languages, given identical data constraints. the proposed algorithm is still model-agnostic and applicable to more tasks requiring sample-efficient cross-lingual transfer. our innovation is the combination of both intra-task and inter-language steps to jointly learn the parsing task and optimal cross-lingual transfer. specifically, we interleave learning the overall task from a high-resource language and learning cross-lingual transfer from a minimal sample of a lower-resource language. results on atis @xcite in six languages (english, french, portuguese, spanish, german, chinese) and spider @xcite in two languages (english, chinese) demonstrate our proposal works in both single-and cross-domain environments. our contributions are as follows: our approach yields more accurate parsing in a few-shot scenario and demands 10× fewer samples than prior methods. in this vein, the reptile algorithm @xcite transforms the higher-order gradient approach into k successive first-order steps. reptile-based training approximates a solution manifold across tasks (i.e., a high-density parameter sub-region biased for strong cross-task likelihood), then similarly followed by rapid fine-tuning. by learning an optimal initialization, meta-learning proves useful for low-resource adaptation by minimizing the data required for out-of-domain tuning on new tasks. @xcite also demonstrate the utility of reptile to improve single-task performance. we build on this to examine single-task cross-lingual transfer using the manifold learned with reptile. ## methodology we combine two meta-learning techniques for cross-lingual semantic parsing. the first is the reptile algorithm outlined in section 2. reptile optimizes for dense likelihood regions within the parameters (i.e., a solution manifold) through promoting inter-batch generalization @xcite . standard reptile iteratively optimizes the manifold for an improved initialization across objectives. rapid fine-tuning yields the final task-specific model. the second technique is the first-order approximation of dg-maml @xcite . this single-stage process optimizes for domain generalization by simulating ''source'' and ''target'' batches from different domains to explicitly optimize for crossbatch generalization. our algorithm, xg-reptile, combines these paradigms to optimize a target loss with the overall learning ''direction'' derived as the optimal manifold learned via reptile. this trains an accurate parser demonstrating sample-efficient cross-lingual transfer within an efficient single-stage learning process. (1) run k iterations of gradient descent over k support batches to learn φ k , (2) compute ∇ macro , the difference between φ k and φ 1 , (3) find the loss on the target batch using φ k , and (4) compute the final gradient update from ∇ macro and the target loss. ## experimental design we evaluate xg-reptile against several comparison systems across multiple languages. where possible, we re-implement existing models and use identical data splits to isolate the contribution of our training algorithm. ## results and analysis we contrast xg-reptile to baselines for atis in ## conclusion we propose xg-reptile, a meta-learning algorithm for few-shot cross-lingual generalization in semantic parsing. xg-reptile is able to better utilize fewer samples to learn an economical multilingual semantic parser with minimal cost and improved sample efficiency. compared to adjacent training algorithms and zero-shot approaches, we obtain more accurate and consistent logical forms across languages similar and dissimilar to english. results on atis show clear benefit across many languages and results on spider demonstrate that xg-reptile is effective in a challenging cross-lingual and cross-database scenario. we focus our study on semantic parsing, however, this algorithm could be beneficial in other low-resource cross-lingual tasks. in future work we plan to examine how to better align entities in low-resource languages to further improve parsing accuracy.