Optimal Transport Posterior Alignment for Cross-lingual Semantic Parsing (2023)

Abstract: Cross-lingual semantic parsing transfers parsing capability from a high-resource language (e.g., English) to low-resource languages with scarce training data. Previous work has primarily considered silver-standard data augmentation or zero-shot methods; exploiting few-shot gold data is comparatively unexplored. We propose a new approach to cross-lingual semantic parsing by explicitly minimizing cross-lingual divergence between probabilistic latent variables using Optimal Transport. We demonstrate how this direct guidance improves parsing from natural languages using fewer examples and less training. We evaluate our method on two datasets, MTOP and MultiATIS++SQL, establishing state-of-the-art results under a few-shot cross-lingual regime. Ablation studies further reveal that our method improves performance even without parallel input translations. In addition, we show that our model better captures cross-lingual structure in the latent space to improve semantic representation similarity.1

## introduction semantic parsing maps natural language utterances to logical form (lf) representations of meaning. as an interface between human-and computer-readable languages, semantic parsers are a critical component in various natural language understanding (nlu) pipelines, including assistant technologies @xcite , knowledge base question answering @xcite , and code generation @xcite . recent advances in semantic parsing have led to improved reasoning over challenging questions @xcite and accurate generation of complex queries @xcite , however, most prior work has focused on english @xcite . expanding, or localizing, an english-trained model to additional languages is challenging for several reasons. there is typically little labeled data in the target languages due to high annotation costs. cross-lingual parsers must also be sensitive to how different languages refer to entities or model abstract and mathematical relationships @xcite . transfer between dissimilar languages can also degrade in multilingual models with insufficient capacity @xcite . previous strategies for resource-efficient localization include generating ''silver-standard'' training data through machine-translation @xcite or prompting large language models @xcite . alternatively, zero-shot models use ''gold-standard'' external corpora for auxiliary tasks @xcite and few-shot models maximize sample-efficiency using meta-learning @xcite . we argue that previous work encourages cross-lingual transfer through implicit alignment only via minimizing silver-standard data perplexity, multi-task ensembling, or constraining gradients. we instead propose to localize an encoderdecoder semantic parser by explicitly inducing cross-lingual alignment between representations. we present minotaur (minimizing optimal transport distance for alignment under representations)a method for cross-lingual semantic parsing which explicitly minimizes distances between probabilistic latent variables to reduce representation divergence across languages (figure 1 ). minotaur leverages optimal transport theory @xcite to measure and minimize this divergence between english and target languages during episodic few-shot learning. our hypothesis is that explicit alignment between latent variables can improve knowledge transfer between languages without requiring additional annotations or lexical alignment. we evaluate this hypothesis in a few-shot cross-lingual regime and study how many examples in languages beyond english are needed for ''good'' performance. our technique allows us to precisely measure, and minimize, the cross-lingual transfer gap between languages. this yields both sample-efficient training and establishes leading performance for few-shot cross-lingual transfer on two datasets. we focus our evaluation on semantic parsing but minotaur can be applied directly to a wide range of other tasks. our contributions are as follows: in all cases, cross-lingual semantic parsing demands fine-grained semantic understanding for successful transfer across languages. multilingual pre-training @xcite has the potential to unlock certain understanding capabilities but is often insufficient. previous methods resort to expensive dataset translation @xcite or attempt to mitigate data paucity by creating ''silver'' standard data through machine translation @xcite @xcite or prompting @xcite . however, methods that rely on synthetic data creation are yet to produce cross-lingual parsing equitable to using goldstandard professional translation. zero-shot methods bypass the need for indomain data augmentation using multi-task objectives which incorporate gold-standard data for external tasks such as language modeling or dependency parsing @xcite @xcite . few-shot approaches which leverage a small number of annotations have shown promise in various tasks @xcite including semantic parsing. sherborne and lapata (2023) propose a first-order meta-learning algorithm to train a semantic parser capable of sample-efficient cross-lingual transfer. our work is most similar to recent studies on cross-lingual alignment for classification tasks @xcite and spoken-language understanding using token-and slot-level annotations between parallel inputs @xcite . while similar in motivation, we contrast in our exploration of latent variables with parametric alignment for a closed-form solution to cross-lingual transfer. additionally, our method does not require fine-grained word and phrase alignment annotations, instead inducing alignment in the continuous latent space. alignment and optimal transport optimal transport (ot; @xcite minimizes the cost of mapping from one distribution (e.g., utterances) to another (e.g., logical forms) through some joint distribution with conditional independence (monge, 1781), i.e., a latent variable conditional on samples from one input domain. ot in nlp has mainly used sinkhorn distances to measure the divergence between non-parametric discrete distributions as an online minimization subproblem @xcite . cross-lingual approaches to ot have been proposed for embedding alignment @xcite , bilingual lexicon induction @xcite , and summarization (nguyen and luu, 2022). our method is similar to recent proposals for crosslingual retrieval using variational or ot-oriented representation alignment @xcite . @xcite consider a ''continuous'' perspective on ot using the wasserstein auto-encoder @xcite as a language model which respects geometric input characteristics within the latent space. our parametric formulation allows this continuous approach to ot, similar to the wae model. while monolingual prior work in semantic parsing has identified that latent structure can benefit the semantic parsing task @xcite , it does not consider whether it can inform transfer between languages. to the best of our knowledge, we are the first to consider the continuous form of ot for cross-lingual transfer in a sequence-to-sequence task. we formulate the parsing task as a transportation problem in section 3 and describe how this framework gives rise to explicit cross-lingual alignment in section 4. ## conclusion we propose minotaur, a method for few-shot cross-lingual semantic parsing leveraging optimal transport for knowledge transfer between languages. minotaur uses a multi-level posterior alignment signal to enable sample-efficient semantic parsing of languages with few annotated examples. we identify how minotaur aligns individual and aggregate representations to bootstrap parsing capability from english to multiple target languages. our method is robust to different choices of alignment metrics and does not mandate parallel data for effective cross-lingual transfer. in addition, minotaur learns more semantically distributed and language-agnostic latent representations with verifiably improved semantic similarity, indicating its potential application to improve cross-lingual generalization in a wide range of other tasks.