Margin-aware Unsupervised Domain Adaptation for Cross-lingual Text Labeling (2020)

Abstract: Unsupervised domain adaptation addresses the problem of leveraging labeled data in a source domain to learn a well-performing model in a target domain where labels are unavailable. In this paper, we improve upon a recent theoretical work (Zhang et al., 2019b) and adopt the Margin Disparity Discrepancy (MDD) unsupervised domain adaptation algorithm to solve the cross-lingual text labeling problems. Experiments on cross-lingual document classification and NER demonstrate the proposed domain adaptation approach advances the state-of-the-art results by a large margin. Specifically, we improve MDD by efficiently optimizing the margin loss on the source domain via Virtual Adversarial Training (VAT). This bridges the gap between theory and the loss function used in the original work Zhang et al.(2019b), and thereby significantly boosts the performance. Our numerical results also indicate that VAT can remarkably improve the generalization performance of both domains for various domain adaptation approaches.

## introduction unsupervised domain adaptation provides an appealing solution to many applications where direct access to a massive amount of labeled data is prohibitive or very costly @xcite @xcite . for example, we often have sufficient labeled data for english, while very limited or even no labeled data are available for many other languages. successfully transferring knowledge learned from the english domain to other languages is of great interest in solving many tasks in natural language processing. many recent successes in unsupervised domain adaptation have been achieved by learning domain invariant features that are simultaneously being discriminative to the task in the source domain @xcite @xcite . following this line, @xcite propose a language-adversarial training approach for cross-lingual document classification and ner. they leverage the benefit of contextualized word embeddings by using multilingual bert @xcite as the feature generator, and adopt the gan framework @xcite to align the features from the two domains. @xcite show significant improvement over the baseline where the pretrained multilingual bert is finetuned on the english data alone and testing on the same tasks in other languages. however, @xcite , as well as the works mentioned above, are inspired by the pioneering work of @xcite , which only rigorously studies domain adaptation in the setting of binary classification; there is a lack of theoretical guarantees when it comes to multiclass classification. in this work, we are instead motivated by a recent work @xcite that focuses on the theoretical analysis of unsupervised domain adaption for multiclass classification and provides explicit guidance for algorithm design. instead of training a discriminator that predicts if the representations are from the source domain or the target domain @xcite @xcite , @xcite proposes to optimize an auxiliary classifier which, together with the classifier, minimizes the discrepancy between the two domains via adversarial training. we apply this approach to cross-lingual text labeling tasks, which, as demonstrated in section 4, outperforms @xcite by a large margin. to the best of our knowledge, we are the first to apply the novel theoretical findings of @xcite for unsupervised domain adaptation in nlp. another contribution of our work lies in identifying the gap between theory and the actual loss function being used in @xcite . specifically, @xcite use the cross-entropy loss as a proxy to optimize the classification margin loss on the source domain, whereas the crossentropy loss often leads to poor margins @xcite . to tackle this problem, we augment the cross-entropy loss with virtual adversarial training (vat) @xcite . as shown in @xcite , the local consistency regularization introduced by vat is capable of promoting large classification margin by optimizing the classification boundary error. this is further demonstrated in section 4 that the incorporation of vat leads to remarkable improvement over @xcite . although the pretrained language models @xcite @xcite have provided a good foundation for many downstream tasks, to leverage them for unsupervised domain adaptation, we need to tackle the potential overfitting problem, especially when we only have limited labeled data in the source domain but can require many training iterations to minimize the domain discrepancy. as shown in section 4, vat can efficiently prevent overfitting in the source domain, and hence significantly improve the generalization in the target domain. this matches the theoretical insights @xcite that the generalization of the target domain can be boosted as a consequence of the improvement in the source domain. ## related work inspired by a pioneering work @xcite , there has been a surge of interest in learning domain invariant representations @xcite @xcite @xcite for unsupervised domain adaptation. at a high level, these methods leverage deep neural networks (dnns) to learn rich representations, and adopt adversarial training @xcite to promote the emergence of domain invariant representations that are simultaneously being discriminative to the predictor learned in the source domain. in the mostly related work, @xcite apply such strategy to multilingual document classification and ner, see figure 1a . although @xcite have achieved remarkable improvements over the baseline, the underlying theory @xcite is only applicable to binary classification with restrictive 0-1 loss. there is a lack of theoretical understanding of @xcite when it comes to multiclass classification with more general loss functions. in a recent theory work, @xcite extend the previous theories to the multiclass classification setting. instead of training an additional discriminator, @xcite proposes to train an auxiliary classifier that shares the same structure as the classifier. the discrepancy between the two domains is optimized by playing the minimax game between the two classifiers. as illustrated in figure 1b , we improve upon this new strategy to address two cross-lingual classification tasks, where the pretrained multilingual bert is used as the feature generator followed by two identical classifiers with different parameters. numerical results in section 4 demonstrate that our proposed approach outperforms both @xcite and @xcite by a large margin. another line of relevant work is the consistency regularization technique used to force the model output to remain unchanged under input perturbations. as observed in the literature, it can promote large classification margin and significantly improve the performance in semi-supervised learning @xcite @xcite @xcite . to effectively optimize the classification margin loss proposed by @xcite , we augment its original objective with virtual adversarial training @xcite . by doing so, we bridge the gap between the theory and the loss function used in @xcite , which in turn yields remarkable improvement on the generalization performance of both domains. ## model we formalize the unsupervised domain adaptation as follows. let @xmath0, . . . , k} denote the input and output space of the model, respectively. we consider two domains s, @xmath1, which are referred to as the source domain and the target domain correspondingly. our ultimate goal is to learn a well-performing classifier on the target domain, while labels are only available for the source domain. let ψ : r @xmath2, which we use to transform the minimization of the domain discrepancy from the data space to the representation space. let f, f : r @xmath3, respectively. note that, for a scoring function, e.g., f , the outputs of each dimension indicate the prediction confidence. hence, given an input example x, the prediction is followed as: let σ denote the softmax function, i.e., following @xcite , we choose the standard cross-entropy loss for the classification task in the source domain, on par with the classification loss, we need to optimize the domain discrepancy between the two domains. before that, we first introduce the measurements we used to quantify the discrepancy between f and f on each domain. let y s , y t denote the predictions given by the classifier f (see equation (1)), then as we can see, both d s (f , f ) and d t (f , f ) are increasing functions of the difference between the auxiliary classifier f and the classifier f , i.e., they both increase when the output of f at the class predicted by f has lower confidence. following @xcite , the domain discrepancy is then approximated as, in other words, given a specific classifier f , the domain discrepancy is induced by f as the maximal difference between the disparities of f and f on the two domains. here γ is proposed by @xcite to promote convergence of the optimization of the domain discrepancy. given γ > 1 and no restrictions on f , @xcite prove that the global minimum of the discrepancy defined in (4) is achieved when ψ(s) = ψ(t ). intuitively, solving the inner maximization requires finding a f that can maximally differ from f on the target domain while staying close to f on the source domain. minimizing the domain discrepancy naturally induces minimax optimization. the main objective thereby can be formulated as, ## numerical results we evaluate the performance of the proposed approach on two different nlp tasks: text classifica-tion, where we use the mldoc corpus @xcite ; and named entity recognition, where we use the conll 2002/2003 ner corpus @xcite . we compare our regularized mdd approach against both @xcite and the baseline. for the baseline, we train the model on the english corpus only, while evaluating on the corpus of the other languages. we also do an ablation study to demonstrate vat can yield remarkable performance boost for all three approaches evaluated in this section. we implement all three approaches in pytorch @xcite with the huggingface library @xcite . we use the pretrained cased multilingual bert @xcite as the initialization for the feature extractor, which is followed by a linear classifier of size 768 × k with k indicating the number of classes. we train an additional linear discriminator with size 786 × 2 for @xcite , and an auxiliary classifier with the same size of the primary classifier, i.e., 768 × k, for mdd. we use the adam optimizer (kingma and ba, 2015) with batch size of 24 for all approaches. we use a constant learning rate η @xmath4, and use the learning rates η ψ , η f and η d to optimize the feature extractor, the auxiliary classifier (mdd), and the discriminator @xcite correspondingly. ## conclusion in this paper, we followed the novel theoretical findings of @xcite , and applied the margin disparity discrepancy (mdd) based unsupervised domain adaptation approach to address the cross-lingual text labeling problems. we demonstrated that mdd can generally outperform the current state-of-the-art model @xcite by a large margin. we further improve mdd by identifying the gap between theory and the actual loss function being used in the original work @xcite . we resolve the problem by using virtual adversarial training (vat) @xcite , which, as demonstrated by our numerical results, leads to remarkable improvement over @xcite . we attribute this to the fact that vat is capable of promoting large classification margin by optimizing the classification boundary error @xcite . this also explains why vat can generally boost the generalization of the source domain for all three approaches explored in this paper, which in turn leads to the generalization improvement on the target domain. the remarkable improvement achieved by vat also motivates us to explore more sophisticated regularization to further improve the performance of various unsupervised domain adaptation approaches. one promising direction is replacing the vat with adversarial training, which, as proven in @xcite , yields a reliable classifier that is robust to adversarial attacks in the source domain. to successfully transferring the robustness from the source domain to target domain is of great interest for both theory and practical applications. we leave this as future work.