Cyclical Contrastive Learning Based on Geodesic for Zero-shot Cross-lingual Spoken Language Understanding (2024)

Abstract: Owing to the scarcity of labeled training data, Spoken Language Understanding (SLU) is still a challenging task in low-resource languages. Therefore, zero-shot cross-lingual SLU attracts more and more attention. Contrastive learning is widely applied to explicitly align representations of similar sentences across different languages. However, the vanilla contrastive learning method may face two problems in zero-shot cross-lingual SLU: (1) the consistency between different languages is neglected; (2) each utterance has two different kinds of SLU labels, i.e. slot and intent, the utterances with one different label are also pushed away without any discrimination, which limits the performance. In this paper, we propose Cyclical Contrastive Learning based on Geodesic (CCLG), which introduces cyclical contrastive learning to achieve the consistency between different languages and leverages geodesic to measure the similarity to construct the positive pairs and negative pairs. Experimental results demonstrate that our proposed framework achieves the new state-of-the-art performance on MultiATIS++ and MTOP datasets, and the model analysis further verifies that CCLG can effectively transfer knowledge between different languages.

## introduction spoken language understanding (slu) holds the central position in the task-oriented dialogue systems @xcite @xcite @xcite . the primary objective of slu task is to comprehend and extract relevant information from user utterances. this capability enables the system to discern the user's current objective and generate appropriate responses. slu comprises two critical sub-tasks: intent detection, which focuses on identifying users' intentions, and slot filling, which entails extracting semantic elements from user queries @xcite ., * corresponding author. however, the effectiveness of traditional slu models is intrinsically linked to the availability of extensive annotated data, which poses challenges in the scalability. this challenge is particularly evident in the case of low-resource languages, where the lack of substantial labeled datasets exacerbates scalability issues, hindering the seamless deployment and advancement of slu models. with the demand for language processing solutions extending across various diverse linguistic landscapes, the necessity for scalable slu models that can operate effectively in resource-constrained environments becomes increasingly critical. to tackle these constraints, the concept of zeroshot cross-lingual slu generalization has emerged as a central focus of interest and investigation. recently, mbert @xcite has demonstrated significant advancements in zero-shot crosslingual slu. building upon this work, @xcite first introduces an attention-informed mixedlanguage training approach for cross-lingual slu. in addition, the exploration of multilingual codeswitched settings has been extended by @xcite , which entails aligning a source language with target languages. gl-clef @xcite employs contrastive learning, leveraging bilingual dictionaries to construct multilingual views of the same utterance, then encouraging their representations to be more similar than those negative example pairs. laj-mcl @xcite proposes to model the utterance-slot-word structure using a multi-level contrastive learning framework to facilitate explicit alignment, further enhancing performance. fc-mtlf @xcite points out the deficiencies in conventional code-switching methods @xcite and introduces an auxiliary multilingual neural machine translation task to facilitate knowledge transfer across different languages. although existing zero-shot cross-lingual slu methods have made promising strides by contrastive learning, we identify two main issues: in this paper, we propose cyclical contrastive learning based on geodesic (cclg) to solve these two problems. for the first problem, we introduce two consistency losses, including the cross-lingual consistency loss and the intra-lingual consistency loss, aiming to boost the consistency between different languages. for the second problem, we abandon the previous approach of directly employing code-switching to construct positive samples and negative samples in contrastive learning. instead, we utilize geodesic to reconstruct positive and negative samples and employ geodesic-based similarity instead of the traditional similarity metrics, thereby facilitating the learning of representations. we conduct experiments on multiatis++ @xcite and mtop @xcite , covering nine and six different languages, respectively. the experimental results show that our framework can outperform previous cross-lingual slu baselines. further model analysis also indicates that our method can transfer knowledge from high-resource languages to low-resource languages. in summary, our work makes three-fold contributions: ## related works the related works are introduced from zero-shot cross-lingual slu and contrastive learning. ## method in this section, we first introduce the generic slu module (sec. 4.1) and the previous paradigm of utilizing contrastive learning to enhance zero-shot cross-lingual slu (sec. 4.2). then, we introduce the components of our proposed approach, including cyclical contrastive learning (sec. 4.3) and geodesic (sec. 4.4). finally, we introduce the final training objective (sec. 4.5). the overview of our approach is demonstrated in figure 1 . ## conclusion in this paper, we propose a novel framework cclg for zero-shot cross-lingual spoken language understanding (slu), which utilizes cyclical contrastive learning to achieve the consistency across different languages and applies geodesic to construct positive samples and negative samples in contrastive learning. experiments on the multiatis++ dataset and the mtop dataset show that our cclg outperforms the previous best model and achieves a new state-of-the-art performance. further analysis also demonstrates that our method could indeed transfer knowledge between different languages effectively.