Enhancing Code-Switching for Cross-lingual SLU : A Unified View of Semantic and Grammatical Coherence (2023)

Abstract: Despite the success of spoken language understanding (SLU) in high-resource languages, achieving similar performance in low-resource settings, such as zero-shot scenarios, remains challenging due to limited labeled training data. To improve zero-shot cross-lingual SLU, recent studies have explored code-switched sentences containing tokens from multiple languages. However, vanilla code-switched sentences often lack semantic and grammatical coherence. We ascribe this lack to two issues: (1) randomly replacing code-switched tokens with equal probability and (2) disregarding token-level dependency within each language. To tackle these issues, in this paper, we propose a novel method termed SoGo, for zero-shot cross-lingual SLU. First, we use a saliency-based substitution approach to extract keywords as substitution options. Then, we introduce a novel token-level alignment strategy that considers the similarity between the context and the code-switched tokens, ensuring grammatical coherence in code-switched sentences. Extensive experiments and analyses demonstrate the superior performance of SoGo across nine languages on MultiATIS++.

## introduction acting as the interface between users and machines, spoken language understanding (slu) is crucial in task-oriented dialogue systems @xcite @xcite . while joint training models have led to significant advancements, most existing models rely on annotated training data, limiting their scalability to low-resource languages. to this end, zero-shot cross-lingual slu gains increasing attention. existing works have shown the effectiveness of multilingual bert (mbert) in multilingual corpus pre-training, enabling promising zero-shot † equal contribution. * corresponding author. cross-lingual slu @xcite @xcite . therein, @xcite extended this concept to a code-switched setting, aligning the source language with multiple target languages. @xcite incorporate contrastive learning to achieve fine-grained crosslingual transfer. based on this, @xcite further propose a multi-level contrastive learning framework for explicit alignment of utterance-slotword structure in cross-lingual slu. however, most above-mentioned models adopt code-switching methods, which often lack both semantic coherence and grammatical coherence. this can be attributed to two main issues: (1) current approaches overlook the importance of individual words in a sentence, as they are randomly replaced with equal probability, potentially introducing unnecessary translation burden and noise that affects semantic coherence. (2) code-switched sentences comprising tokens from multiple languages may lack grammatical coherence, as token-level dependencies within each language are disregarded. to address these issues, we propose a novel framework dubbed sogo for enhancing codeswitching. concretely, we first employ a saliencybased substitution approach to identify keywords with high saliency scores. these keywords are then replaced with their target language equivalents using bilingual dictionaries, to generate codeswitched sentences. notably, unlike the approach by @xcite , our sogo framework requires only a single training process for multiple target languages. then, we devise a novel token-level alignment strategy to bridge the gap between original context and code-switched tokens. by mapping code-switched tokens back to the original context, sogo minimizes the impact of token substitutions on token-level coherence, thus maintaining consistent contextual space within each language. experimental results on multiatis++ @xcite demonstrate that employing code-switching shared [cls] like funny movie [sep] … ! ! "#@xmath0 ! & "#$ … ! ! '"( ! % '"( ! & '"( [cls] like きげき 电影 [sep] encoder layer 1 encoder layer 1 encoder layer 6 encoder layer 6 … encoder layer 12 encoder layer 12 … … … token-level alignment mbert mbert gradient-based substitution ℒ ! (b) (a) id *: like funny movie … english training corpus … id 1: how much does the american airlines … id 2: could you tell me the cheapest fare … local saliency score 0.54 1.27 0.89 like funny movie global saliency score inverse document frequency expensive 1.52 cheapest 1.47 lowest 1.44 … cos dictionary expensive teuer cheapest さいやす lowest le plus bas … ℒ " mean mean we perform token-level alignment in lower six layers of mbert to enhance the grammatical coherence of the code-switched sentence. with sogo leads to significant performance improvement compared to baseline models. furthermore, we demonstrate the compatibility of sogo with other existing slu structures. ## method we describe problem formulation ( §2.1) and backbone model ( §2.2) for zero-shot cross-lingual slu first, before describing sogo ( §2.3). the main architecture of sogo is illustrated in figure 1 . ## conclusion we proposed a novel semantics-coherent and grammar-coherent method (sogo) to enhance code-switching for zero-shot cross-lingual slu. specifically, we utilized saliency-based substitution to replace keywords for better semantic coherence in the shared space. besides, we introduced an alignment strategy to map the code-switched tokens to original contexts, which solves the grammatical incoherence of code-switching. experiments on multiatis++ showed that code-switching based methods with sogo obtained competitive performance compared to counterpart baselines.