Label-aware Multi-level Contrastive Learning for Cross-lingual Spoken Language Understanding (2022)

Abstract: Despite the great success of spoken language understanding (SLU) in high-resource languages, it remains challenging in low-resource languages mainly due to the lack of labeled training data. The recent multilingual code-switching approach achieves better alignments of model representations across languages by constructing a mixed-language context in zero-shot cross-lingual SLU. However, current code-switching methods are limited to implicit alignment and disregard the inherent semantic structure in SLU, i.e., the hierarchical inclusion of utterances, slots and words. In this paper, we propose to model the utterance-slot-word structure by a multi-level contrastive learning framework at the utterance, slot and word levels to facilitate explicit alignment. Novel code-switching schemes are introduced to generate hard negative examples for our contrastive learning framework. Furthermore, we develop a label-aware joint model leveraging label semantics to enhance the implicit alignment and feed to contrastive learning. Our experimental results show that our proposed methods significantly improve the performance compared with the strong baselines on two zero-shot cross-lingual SLU benchmark datasets.

## introduction spoken language understanding (slu) is a critical component of goal-oriented dialogue systems, which consists of two subtasks: intent detection and slot filling @xcite . recently, massive efforts based on the joint training paradigm @xcite have shown superior performance in english. however, the majority of them require large amounts of labeled training data, which limits the scalability to low-resource languages with little or no training data. zeroshot cross-lingual approaches have arisen to tackle 1 work is done during internship at microsoft stca. 2 corresponding authors. ## methodology figure 2 illustrates our method. on the left part, the proposed label-aware joint model (lajoint) transfers the semantics of the language-invariant slot set across languages. on the right part, we develop a systematic approach to multi-level contrastive learning (mcl) by novel code-switching schemes. the full framework is called laj-mcl. ## conclusion in this paper, we propose a novel label-aware joint model (lajoint) with a multi-level contrastive learning framework (mcl) for zero-shot crosslingual slu. the former leverages the languageinvariant slot set to transfer knowledge across languages and the latter exploits the semantic structure of slu and develops contrastive learning based on novel code-switching schemes for explicit alignment. the results of extensive experiments demon-strate the effectiveness of our methods.