Saliency-based Multi-View Mixed Language Training for Zero-shot Cross-lingual Classification (2021)

Abstract: Recent multilingual pre-trained models, like XLM-RoBERTa (XLM-R), have been demonstrated effective in many cross-lingual tasks. However, there are still gaps between the contextualized representations of similar words in different languages. To solve this problem, we propose a novel framework named Multi-View Mixed Language Training (MVMLT), which leverages code-switched data with multi-view learning to fine-tune XLM-R. MVMLT uses gradient-based saliency to extract keywords which are the most relevant to downstream tasks and replaces them with the corresponding words in the target language dynamically. Furthermore, MVMLT utilizes multi-view learning to encourage contextualized embeddings to align into a more refined language-invariant space. Extensive experiments with four languages show that our model achieves state-of-the-art results on zero-shot cross-lingual sentiment classification and dialogue state tracking tasks, demonstrating the effectiveness of our proposed model.

## introduction due to the availability of large labeled datasets and parallel corpus, neural network models have achieved remarkable performance on a variety of natural language processing (nlp) tasks. however, generally large-scale training data with high quality is only available in a few languages. artificially collecting or translating training data for different languages could be time-consuming and expensive, which will inevitably create a massive performance gap between high-resource language models (e.g., english and french) and low-resource language models (e.g., swahili and urdu). cross-lingual transfer learning (cltl) aims at bridging this gap by transferring the learned knowl-edge from a resource-rich language (source) to a resource-lean language (target) @xcite . the main idea of cltl is to learn a shared language-invariant feature space for both languages, so that a model trained on the source language could be applied to the target language directly. recently, cross-lingual contextualized embedding methods such as multilingual bert (mbert) @xcite , xlm @xcite , and xlm-roberta (xlm-r) @xcite have achieved state-of-the-art results on a variety of zeroshot cross-lingual tasks. however, those bertstyle transformer @xcite architectures, training cross-lingual embeddings from selfsupervised masked language modelling with monolingual corpus, may not well capture the semantic similarity of subwords across different languages. in order to alleviate inconsistent contextualized representations within different languages, some supervised cross-lingual signals have been introduced in prior work @xcite , e.g., bilingual dictionaries and parallel corpora. @xcite propose a data augmentation framework called code-switching or mix language training, which chooses a set of words randomly and replaces them with the corresponding words in a different language. for example, "i 喜欢 this 电影 so much" 2 is a code-switched sentence. they only use a bilingual dictionary to generate code-switched data to fine-tune mbert, which encourage model to align representations between different languages. nevertheless, there are two main problems in this method: (1) the importance of different words in a document is ignored, since they just replace words with the same probability randomly. replacing some unimportant words will increase the burden of translation and even introduce noise that impairs the sentence semantic coherence; (2) they only use code-switched corpus to fine-tune mbert, while the relation between original sentences and code-switched sentences is ignored completely, which may leads to the loss of some interactive information and hinder contextualized embeddings from further alignment. to address the issues mentioned above, we propose a new framework named multi-view mixed language training (mvmlt), which leverages code-switched data with multi-view learning for zero-shot cross-lingual transfer. mvmlt first uses gradient-based saliency method to find keywords with high saliency scores in downstream tasks (section 3.1). for example, in cross-lingual sentiment classification tasks, some words with sentiment information (e.g., "excellent", "interesting" and "boring") should have higher saliency scores than background words (e.g., "the", "a" and "what"). relying on a bilingual dictionary, we replace these keywords with their corresponding words in the target language to generate code-switched data (section 3.2). these code-switched keywords are the essential part for effective cross-lingual transfer, because they intersect with different languages and allow the shared encoder to learn some direct tying of meaning across different languages. therefore, selecting the most task-related keywords by saliency detection facilitates cross-lingual performance for providing a strong tie across different languages. furthermore, mvmlt acquires comprehensive cross-lingual information from different perspectives and explores the consistency of multiple views by means of multi-view learning @xcite . specifically, mvmlt constructs two views from the multilingual pre-trained model, i.e., xlm-r: in summary, our main contributions are as follows: ## methodology the architecture of our model is illustrated in figure1, which consists of three components: ## experiments we evaluate the effectiveness of our proposed method on zero-shot cross-lingual dialog state tracking and sentiment classification tasks in four languages. in details, english is the source language, and the target languages are german, italian, french and japanese, respectively. ## conclusion in this paper, we propose multi-view mixed language training (mvmlt), a novel zero-shot cross-lingual transfer framework. our approach utilizes gradient-based saliency to replace a few task-related words with target language, which is used for fine-tuning on downstream tasks. besides, we introduce multi-view learning to construct a language-invariant feature space. experiments show that our model achieves state-of-theart results on cross-lingual sentiment classification and dialogue state tracking tasks. in the future, we will investigate the effectiveness of our approach in multi-lingual setting and apply our model to more tasks.