DAMP : Doubly Aligned Multilingual Parser for Task-Oriented Dialogue (2023)

Abstract: Modern virtual assistants use internal semantic parsing engines to convert user utterances to actionable commands. However, prior work has demonstrated multilingual models are less robust for semantic parsing compared to other tasks. In global markets such as India and Latin America, robust multilingual semantic parsing is critical as codeswitching between languages is prevalent for bilingual users. In this work we dramatically improve the zero-shot performance of a multilingual and codeswitched semantic parsing system using two stages of multilingual alignment. First, we show that contrastive alignment pretraining improves both English performance and transfer efficiency. We then introduce a constrained optimization approach for hyperparameter-free adversarial alignment during finetuning. Our Doubly Aligned Multilingual Parser (DAMP) improves mBERT transfer performance by 3x, 6x, and 81x on the Spanglish, Hinglish and Multilingual Task Oriented Parsing benchmarks respectively and outperforms XLM-R and mT5-Large using 3.2x fewer parameters.

## introduction task-oriented dialogue systems are the backbone of virtual assistants, an increasingly common direct interaction between users and natural language processing (nlp) technology. semantic parsing converts unstructured text to structured representations grounded in task actions. due to the conversational nature of the interaction between users and task-oriented dialogue systems, speakers often use casual register with regional variation. such variation is an essential challenge for the inclusiveness and reach of virtual assistants which aim to serve a global and diverse userbase @xcite . work partially done during an internship at google. 1 we release code for our constrained optimization technique on github and finetuned t5 models on huggingface. in this work, we are motivated by a common form of variation for bilingual speakers @xcite : codeswitching. codeswitching occurs in two forms which both affect task-oriented dialogue. inter-sentential codeswitching is when multilingual users make whole requests in different languages within a single dialogue: play all rap music on my itunes toca toda la música rap en mi itunes intra-sentential codeswitching appears when the user switches languages during a single query: play toda la rap music en mi itunes both forms are used by bilingual speakers @xcite and cause location, language preference, and even language identification to be unreliable mechanisms for routing requests to an appropriate monolingual system @xcite . this makes zero-shot codeswitching performance an aspect of system robustness instead of a way to reduce annotation costs. however, zero-shot structured prediction and parsing is still a challenge for state-of-the-art multilingual models @xcite , highlighting the need for improved methods beyond scale to achieve this goal. fortunately, as a fundamental property of the task, these linguistically diverse inputs are grounded in a shared semantic output space. each of the above outputs corresponds to: this grounded and shared output space makes explicit alignment across languages especially attractive as a mechanism for cross-lingual transfer. we propose using both contrastive alignment pretraining and a novel constrained adversarial finetuning method to perform double alignment, shown in figure 1 . our doubly aligned multilingual parser (damp) achieves strong zero-shot performance on both multilingual (inter-sentential) and intra-sentential codeswitched data, making it a robust model for bilingual users without harming english performance. we contribute the following: ## related work multilingual language model alignment massively multilingual transformers (mmts) @xcite @xcite have become the de-facto basis for multilingual nlp and are effective at intrasentential codeswitching as well @xcite . while prior work has studied explicit alignment of individual embeddings @xcite , mmts appear to implicitly perform alignment within their hidden states @xcite . mmts are remarkably robust for multilingual and intra-sentential codeswitching benchmarks @xcite @xcite . however, the gap between performance on the training language and zero-shot targets is larger in task-oriented parsing benchmarks @xcite @xcite , similar to the large discrepancy for other syntactically intensive tasks @xcite . our work applies the pretraining regime from @xcite , which adds multiple explicit alignment objectives to traditional mmt pretraining. we show that this technique is effective both for semantic parsing, a new task, and intra-sentential codeswitching, a new linguistic domain. ## methods we utilize two separate stages of alignment to improve zero-shot transfer in damp. during pretraining, we use contrastive learning to improve alignment amongst pretrained representations. during finetuning, we add double alignment through domain adversarial training using a binary language discriminator and a constrained optimization approach. we apply these improvements to the en-coder of a pointer-generator network that copies and generates tags to produce a parse. ## alignment analysis we analyze how well our alignment goals are met using two methods in figure 1 . first, we use a twodimensional projection of the resulting encoder embeddings to provide a visual intuition for alignment. then, we provide a more reliable quantitatively evaluate alignment using a post-hoc linear probe. ## conclusions in this work, we introduce a doubly aligned multilingual parser (damp), a semantic parsing training regime that uses contrastive alignment pretraining and adversarial alignment during fine-tuning with a novel constrained optimization approach. we demonstrate that both of these stages of alignment benefit transfer learning in semantic parsing to both inter-sentential (multilingual) and intrasentential codemixed data, outperforming both similarly sized and larger models. we analyze the effects of damp, comparing our proposed alignment method broadly to prior both adversarial techniques and regularization baselines, and its generalizability, with applications to pretrained decoders. finally, we interpret the impacts of both stages of alignment through qualitative improvement analysis and quantitative probing. importantly, damp shows that alignment in both pretraining and finetuning can outperform larger models pretrained on more data. this offers an orthogonal improvement to the current scaling paradigm, supporting the idea that current multilingual models underutilize available bitext @xcite . in cases where bitext is unavailable, our work shows that alignment still possible via adversarial procedures. by releasing our simplified constrained optimization approach for multilingual adversarial alignment, we aim to simplify and improve the application of such approaches for future work.