Improving Cross-lingual Text Classification with Zero-shot Instance-Weighting (2021)

Abstract: Cross-lingual text classification (CLTC) is a challenging task made even harder still due to the lack of labeled data in low-resource languages. In this paper, we propose zero-shot instance-weighting, a general model-agnostic zero-shot learning framework for improving CLTC by leveraging source instance weighting. It adds a module on top of pre-trained language models for similarity computation of instance weights, thus aligning each source instance to the target language. During training, the framework utilizes gradient descent that is weighted by instance weights to update parameters. We evaluate this framework over seven target languages on three fundamental tasks and show its effectiveness and extensibility, by improving on F1 score up to 4% in single-source transfer and 8% in multi-source transfer. To the best of our knowledge, our method is the first to apply instance weighting in zero-shot CLTC. It is simple yet effective and easily extensible into multi-source transfer.

## introduction natural language processing (nlp) has largely benefited from recent advances in deep learning and large-scale labeled data. unfortunately, such labeled corpora are not available for all languages. cross-lingual transfer learning is one way to spread the success from high-resource to low-resource languages. cross-lingual text classification (cltc) @xcite can learn a classifier in a low-resource target language by transferring from a resource-rich source language @xcite . previous work has learned a classifier in the target language using a very small sample of labeled target instances or external corpora of unlabeled instances @xcite . in addition, other resources that may be utilized to achieve the same include, but are not limited to, parallel corpora of unlabeled instances in the target language @xcite . in this work, we address the most challenging setting, zero-shot cltc @xcite , where no resource in the target language is given. among the many methods for transfer learning that have been successfully employed in nlp @xcite @xcite , instance (re-) weighting is perhaps one of the oldest and most well known @xcite . it is best illustrated when we are given access to a few target labeled instances (few-shot learning). for example, both @xcite and @xcite learn a classifier iteratively by assigning weights to each instance in the source training data. while @xcite assigns weights to both source and target instances, @xcite pre-trains a classifier on the source training data and then re-weights the target labeled instances. crucially, the weights are set to be a function of the error between the prediction made for the instance by the current classifier and the instance's gold label. in a few-shot case, it is easy to see the appeal of re-weighting target language instances, since an instance that incurs a higher prediction loss can be given a larger weight, so as to improve the classifier. but in a zero-shot case, it seems impossible to compute instance weights based on prediction loss. in this work, we make it possible to assign such weights on instances in zero-shot cltc. to the best of our knowledge, this is the first attempt to apply such a method to nlp tasks. our contributions are two-fold: first, we introduce zero-shot instance-weighting, a simple but effective, and extensible framework to enable instance weighted transfer learning for zero-shot cltc. second, we evaluate on three cross-lingual classification tasks in seven different languages. results show that it improves f1 score by up to 4% in single-source transfer and 8% in multi-source transfer, identifying a promising direction for utilizing knowledge from unlabeled data. ## proposed method we illustrate the zero-shot cltc framework in figure 1 . the source and target language inputs are x s and x t respectively, during training, only the source label y s is available and the task is to predict the target label y t . we first apply the pre-trained model as an encoder to encode the inputs, the encoded representations are denoted by h s and h t . the figure illustrates four instances for each language in the mini-batch. then there is an instance weighting module to assign weights to source language instances by considering the hidden representations h s and h t . note that these layers are shared. we train the task layer and fine-tune the pre-trained language model layers. ## evaluation we test on three tasks: opinion target extraction, document classification, and sentiment classification foot_2 . english is the source language for all the experiments. we evaluate four settings: 1) direct adaptation with mbert-base (mbert), 2) mbert with instance weighting (mbert+iw), 3) direct adaption of xlmr-base (xlmr), and 4) xlmr with instance weighting (xlmr+iw). opinion target extraction we choose semeval 2016 workshop task 5 @xcite for opinion target extraction. it includes restaurant reviews in five languages foot_3 : english, spanish (es), dutch (nl), russian (ru) and turkish (tr). given a sentence as input, one needs to classify each token into one of the three classes according to the iob scheme. the training and testing size varies from 144 to 3,655. we compare against a list of models. @xcite and @xcite are supervised and require extra corpora or resources to train. agerri and rigau exploits additional resources like unlabeled corpora. jebbara and cimiano (2019) applies multi-source (including the target) languages to train a classifier using cross-lingual embeddings and evaluates in a zeroshot manner. we summarize the results in cross-lingual document classification we conduct cross-lingual document classification task on the mldoc dataset @xcite . it is a set of news articles with balanced class priors in eight languages; each language has 1,000 training documents and 4,000 test documents, and splits into four classes. we select a strong baseline @xcite , which applies pre-trained multicca word embeddings @xcite and then trained in a supervised way. another baseline is a zero-shot method proposed by @xcite , which applies a single bilstm encoder with a shared vocabulary among all languages, and a decoder trained with parallel corpora. @xcite apply mbert as a zero-shot language transfer. results among the three tasks, both base models achieve competitive results for all languages thanks to the choice of pre-trained models. instance weighting produces consistent improvements over the base models for nearly all target languages. especially, in while our approach is model-agnostic, when the base model or the embedding improves, instance weighting will still help, as we can see the improved results obtained by switching from mbert to xlmr. again, the framework is simple but effective given these observations. most importantly, it requires no additional external data and is easily adaptable into any deep models. ## discussion multi-source expansion studies show that multilingual transfer outperforms bilingual transfer @xcite . we run an experiment on the opinion extraction task to illustrate how our approach can be easily extended to enable multi-source transfer, (see case study intuitively, we should focus on the source instances with a smaller difference with target language, because they contain more common features with the target language. thus, if we let those instances contribute more, it is possible that the model may perform better on the target language. as an example, language score content label english instance 2 0.5056 ...i liked the book. kaplan has consistently been one of my favorite authors (atlantic monthly) his theme is consistent: many nation states are not really nation states... kaplan had great hope for the future of iran as they struggle with theocracy... pos english instance 1 0.3647 one start , for some very acurate dramatic and terrorific facts about the ebola, but very weak regarding origin of the virus, very unconvincing about possible "theories". sound more like that old music of desinformation, he almost blame another monkey for the ebola... neg french origin: ...ce livre est exceptionnel..la construction du livre est superbe, l'écriture magique... pos translation: ...this book is outstanding..the construction of book is superb, magical writing ... ## conclusion we proposed instance weighting for cltc and evaluated on 3 fundamental tasks. the benefits of our approach include simplicity and effectiveness by ensuring wide applicability across nlp tasks, extensibility by involving multiple source languages and effectiveness by outperforming a variety of baselines significantly. in the future, we plan to evaluate on more tasks such as natural language inference @xcite and abstract meaning representation @xcite .