GL - CL e F : A Global–Local Contrastive Learning Framework for Cross-lingual Spoken Language Understanding (2022)

Abstract: Due to high data demands of current methods, attention to zero-shot cross-lingual spoken language understanding (SLU) has grown, as such approaches greatly reduce human annotation effort. However, existing models solely rely on shared parameters, which can only perform implicit alignment across languages. We present Global-Local Contrastive Learning Framework (GL-CLeF) to address this shortcoming. Specifically, we employ contrastive learning, leveraging bilingual dictionaries to construct multilingual views of the same utterance, then encourage their representations to be more similar than negative example pairs, which achieves to explicitly align representations of similar sentences across languages. In addition, a key step in GL-CLeF is a proposed Local and Global component, which achieves a fine-grained cross-lingual transfer (i.e., sentence-level Local intent transfer, token-level Local slot transfer, and semantic-level Global transfer across intent and slot). Experiments on MultiATIS++ show that GL-CLeF achieves the best performance and successfully pulls representations of similar sentences across languages closer.

## introduction spoken language understanding (slu) is a critical component in task-oriented dialogue systems @xcite . it usually includes two sub-tasks: intent detection to identify users' intents and slot filling to extract semantic constituents from the user's query. with the advent of deep neural network methods, slu has met with remarkable success. however, existing slu models rely on large amounts of annotated data, which makes it hard to scale to lowresource languages that lack large amounts of labeled data. to address this shortcoming, zero-shot to solve the aforementioned challenges, we propose a global-local contrastive learning framework (gl-clef) for zero-shot crosslingual slu. for the first challenge, as shown in figure 1 (b), the key insight in gl-clef is to explicitly ensure that representations of similar sentences across languages are pulled closer together via contrastive learning (cl). specifically, we leverage bilingual dictionaries to generate multi-lingual code-switched data pairs, which can be regarded as cross-lingual views with the same meaning. with the use of cl, our model is able to learn to distinguish the code-switched utterance of an input sentence from a set of negative examples, and thus encourages representations of similar sentences between source language and target language closer. for the second challenge, slu requires accomplishing tasks at two different levels: token-level slot filling and sentence-level intent detection. as such, simply leveraging ordinary sentence-level contrastive learning is ineffective for fine-grained knowledge transfer in token-level slot filling. therefore, we first introduce a local module in gl-clef to learn different granularity alignment representations (i.e., sentence-level local intent cl and token-level local slot cl). to be specific, sentence-level local intent cl and token-level local slot cl are introduced for aligning similar sentence and token representations across different languages for intent detection and slot filling, respectively. in addition, we further argue that slot and intent are highly correlated and have similar semantic meanings in a sentence. this phenomenon can serve as a signal for self-supervised alignment across intent and slots. therefore, a global module named semantic-level global intent-slot cl is further proposed to bring the representations of multilingual bert multilingual bert multilingual bert watch [cls] sports movie [sep] 观看 [cls] 运动 电影 [sep] 見る [cls] スポーツ 映画 [sep] slu out (i) english (ii) chinese (iii) japanese slot and intents within a sentence closer together. we conduct experiments on multiatis++ @xcite , which includes nine different languages. our experiments show that gl-clef achieves state-of-the-art results of 54.09% sentence accuracy, outperforming the previous best by 10.06% on average. besides, extensive analysis experiments demonstrate that gl-clef has successfully reduced the representation gap between different languages. to facilitate further research, codes are publicly available at https://github.com/ lightchen233/gl-clef . ## background we first describe traditional slu before the specifics of zero-shot cross-lingual version of slu. traditional slu in task-oriented dialogue. slu in task-oriented dialogue contains two subtasks: intent detection and slot filling. since the two tasks of intent detection and slot filling are highly correlated, it is common to adopt a joint model that can capture shared knowledge. we follow the formalism from @xcite , formulated as (o i , o s ) = f (x), where f is the trained model. zero-shot cross-lingual slu. this means that a slu model is trained in a source language, e.g., english (cf. figure 2 (a) ) and directly applied to other target languages (cf. figure 2 (b) ). ## model we describe the general approach to general slu task first, before describing our gl-clef model which explicitly uses contrastive learning to explicitly achieve cross-lingual alignment. the main architecture of gl-clef is illustrated in figure 3 . ## experiments we use the latest multilingual benchmark dataset of multiatis++ @xcite ) which consists of 9 languages including english (en), spanish (es), portuguese (pt), german (de), french (fr), chinese (zh), japanese (ja), hindi (hi), and turkish (tr). ## related work traditional spoken language understanding. since slot filling and intent detection are two correlated tasks, traditional slu approaches mainly explore a joint model for capturing shared knowledge across the two tasks. specifically, zhang and wang (2016 zero-shot cross-lingual spoken language understanding. traditional slu has largely been limited to high-resource languages. to solve this problem, zero-shot cross-lingual slu has gained increasing attention. recently, cross-lingual contextualized embeddings have achieved promising results (e.g., mbert @xcite ). many works target improving mbert at the pre-training stage @xcite @xcite @xcite @xcite . compared with their work, our focus is on enhancing mbert at the fine-tuning stage. in recent years, related work also considers aligning representations between source and target languages during fine-tuning, eschewing the need for an extra pre-training process. specifically, @xcite propose code-mixing to construct training sentences that consist of both source and target phrases for implicitly fine-tuning mbert. qin et al. (2020) further propose a multi-lingual code-switching data augmentation to better align a source language and all target languages. in contrast to their work, our framework consider aligning similar representation across languages explicitly via a contrastive learning framework. in addition, in gl-clef, we propose a multi-resolution loss to encourage fine-grained knowledge transfer for token-level slot filling. contrastive learning. contrastive learning is now commonplace in nlp tasks. @xcite adopt multiple sentence-level augmentation strategies to learn a noise-invariant sentence representation. fang and xie (2020) apply back translation to create augmentations of original sentences for training transformer models. @xcite propose contrastive learning with semantically negative examples (cline) to improve the robustness under semantically adversarial attack. inspired by the success of cl, we utilize contrastive learning to explicitly align similar representations across source language and target language. ## conclusion we introduced a global-local contrastive learning (cl) framework (gl-clef) to explicitly align representations across languages for zero-shot crosslingual slu. besides, the proposed local cl module and global cl module achieves to learn different granularity alignment (i.e., sentence-level local intent alignment, token-level local slot alignment, semantic-level global intent-slot alignment). experiments on multiatis++ show that gl-clef obtains best performance and extensive analysis indicate gl-clef successfully pulls closer the representations of similar sentence across languages.