Code-Switching Can be Better Aligners: Advancing Cross-Lingual SLU through Representation-Level and Prediction-Level Alignment (2024)

Abstract: Zero-shot cross-lingual spoken language understanding (SLU) can promote the globalization application of dialog systems, which has attracted increasing attention. While current code-switching based cross-lingual SLU frameworks have shown promising results, they (i) predominantly utilize contrastive objectives to model hard alignment, which may disrupt the inherent structure within sentences of each language; and (ii) focus optimization objectives solely on the original sentences, neglecting the relation between original sentences and code-switched sentences, which may hinder contextualized embeddings from further alignment. In this paper, we propose a novel framework dubbed REPE (short for Representation-Level and Prediction-Level Alignment), which leverages both code-switched and original sentences to achieve multi-level alignment. Specifically, REPE introduces optimal transport to facilitate soft alignment between the representations of code-switched and original sentences, thereby preserving structural integrity as much as possible. Moreover, REPE adopts multi-view learning to enforce consistency regularization between the prediction of the two sentences, aligning them into a more refined language-invariant space. Based on this, we further incorporate a self-distillation layer to boost the robustness of REPE. Extensive experiments on two benchmarks across ten languages demonstrate the superiority of the proposed REPE framework.

## introduction spoken language understanding (slu) serves as a fundamental component in dialog systems, which involves two tasks: intent detection to classify the intent of user utterances and slot filling to extract useful semantic concepts @xcite @xcite . recently, massive efforts based on the joint training paradigm (xing * corresponding author @xcite @xcite @xcite have shown superior performance in english. nonetheless, the dependency on extensive labeled training data constrains their applicability to low-resource languages with little or no training data @xcite , thus hindering the globalization application of dialog systems. towards this goal, zero-shot crosslingual slu gains increasing attention. due to the unavailability of low-resource languages @xcite , codeswitching @xcite has been developed to reduce the dependency on machine translation. technically, it employs bilingual dictionaries to randomly select some words in the sentence to be replaced by their counterparts in other languages. in line with this, numerous zero-shot cross-lingual slu methods have been proposed @xcite @xcite , yielding promising results. among them, @xcite incorporated contrastive learning to achieve fine-grained cross-lingual transfer. based on this, @xcite further proposed a multi-level contrastive learning framework for explicit alignment of utterance-slot-word structure. recently, @xcite integrated with auxiliary task and curriculum learning, obtaining state-of-the-art (sota) performance. despite the promising progress, we discover existing methods suffer from two main issues: (i) existing methods @xcite employed token-to-token hard contrastive learning objectives to model explicit alignment, potentially disrupting the inherent structural information of sentences, such as inherent phrases or collocations specific to certain languages. (ii) they primarily focus on optimizing objectives based on original sentences, while the correlation between original sentences and code-switched counterparts is ignored, which may lead to the loss of some interactive information and hinder contextualized embeddings from further alignment. in this paper, we propose a novel framework dubbed repe to tackle the above two issues. for the first issue, we resort to optimal transport (ot) @xcite to adaptively model the alignment between the representations of original sentence and code-switched counterpart. in contrast to token-to-token hard contrastive learning, our repe adaptively considers contextual representations through the alignment matrix, preserving the syntactic structure as much as possible. for the second issue, we construct two views from the multilingual pre-trained model (mplm): the prediction of original and code-switched sentences. by employing multi-view learning @xcite , we seek to establish concordance between these two views by minimizing the kullback-leibler @xcite ) (kl) divergence, which encourages similar words across different languages to align into a shared latent space. to improve the robustness of the model and prevent over-confidence, we further introduce a self-distillation layer which minimizes kl divergence between the current prediction and the previous one. experimental results on two benchmarks across ten languages demonstrate that our proposed repe significantly outperforms previous methods and achieves new sota performance, and further analysis verifies the advantages of our repe. ## method this section introduces the repe for zeroshot cross-lingual spoken language understanding (slu), which comprises representation-level alignment ( §2.2), prediction-level alignment ( §2.3) and self-distillation ( §2.4). figure 1 shows the overview of the proposed repe framework. ## experiments we show the details of the datasets and implementation settings in appendix §a.1 and §a.2. ## method analysis we further provide insights into the effectiveness of our model by comparing different ot solutions and the potential of leveraging complementary per