Cross-lingual Spoken Language Understanding with Regularized Representation Alignment (2020)

Abstract: Despite the promising results of current cross-lingual models for spoken language understanding systems, they still suffer from imperfect cross-lingual representation alignments between the source and target languages, which makes the performance sub-optimal. To cope with this issue, we propose a regularization approach to further align word-level and sentence-level representations across languages without any external resource. First, we regularize the representation of user utterances based on their corresponding labels. Second, we regularize the latent variable model (Liu et al., 2019) by leveraging adversarial training to disentangle the latent variables. Experiments on the cross-lingual spoken language understanding task show that our model outperforms current state-of-the-art methods in both few-shot and zero-shot scenarios, and our model, trained on a few-shot setting with only 3% of the target language training data, achieves comparable performance to the supervised training with all the training data.

## introduction data-driven neural-based supervised training approaches have shown effectiveness in spoken language understanding (slu) systems @xcite @xcite . however, collecting large amounts of high-quality training data is not only expensive but also timeconsuming, which makes these approaches not scalable to low-resource languages due to the scarcity of training data. cross-lingual adaptation has naturally arisen to cope with this issue, which leverages the training data in rich-resource source languages and minimizes the requirement of training data in low-resource target languages. in general, there are two challenges in crosslingual adaptation. first, the imperfect alignment of word-level representations between the source and target language limits the adaptation performance. second, even though we assume that the word-level alignment is perfect, the sentence-level alignment is still imperfect owing to grammatical and syntactical variances across languages. therefore, we emphasize that cross-lingual methods should focus on the alignments of word-level and sentence-level representations, and increase the robustness for inherent imperfect alignments. in this paper, we concentrate on the cross-lingual slu task (as illustrated in figure 1 ), and we consider both few-shot and zero-shot scenarios. to improve the quality of cross-lingual alignment, we first propose a label regularization (lr) method, which utilizes the slot label sequences to regularize the utterance representations. we hypothesize that if the slot label sequences of user utterances are close to each other, these user utterances should have similar meanings. hence, we regularize the distance of utterance representations based on the corresponding representations of label sequences to further improve the cross-lingual alignments. then, we extend the latent variable model (lvm) proposed by @xcite . the lvm generates a gaussian distribution instead of a feature vector for each token, which improves the adaptation robustness. however, there are no additional constraints on generating distributions, making the latent variables easily entangled for different slot labels. to handle this issue, we leverage adversarial training to regularize the lvm (alvm). we train a linear layer to fit latent variables to a uniform distribution over slot types. then, we optimize the latent variables to fool the trained linear layer to output the correct slot type (one hot vector). in this way, latent variables of different slot types are encouraged to disentangle from each other, leading to a better alignment of cross-lingual representations. the contributions of our work are summarized as follows: ## methodology our model architecture and proposed methods are depicted in figure 2 , and combine label regularization (lr) and the adversarial latent variable model (alvm) to conduct the intent detection and slot filling. in the few-shot setting, the input user utterances are in both the source and target languages, while in the zero-shot setting, the user utterances are only in the source language. note that both the source and target languages contain only one language. ## conclusion current cross-lingual slu models still suffer from imperfect cross-lingual alignments between the source and target languages. in this paper, we propose label regularization (lr) and the adversarial latent variable model (alvm) to regularize and further align the word-level and sentence-level representations across languages without utilizing any additional bilingual resources. experiments on the cross-lingual slu task illustrate that our model achieves a remarkable performance boost compared to the strong baselines in both zero-shot and few-shot scenarios, and our model has a robust adaptation ability to unrelated target languages in the few-shot scenario. in addition, visualization for latent variables further proves that our approaches are effective at improving the alignment of crosslingual representations.