Cross-Lingual Text Classification with Minimal Resources by Transferring a Sparse Teacher (2020)

Abstract: Cross-lingual text classification alleviates the need for manually labeled documents in a target language by leveraging labeled documents from other languages. Existing approaches for transferring supervision across languages require expensive cross-lingual resources, such as parallel corpora, while less expensive cross-lingual representation learning approaches train classifiers without target labeled documents. In this work, we propose a cross-lingual teacher-student method, CLTS, that generates “weak” supervision in the target language using minimal cross-lingual resources, in the form of a small number of word translations. Given a limited translation budget, CLTS extracts and transfers only the most important task-specific seed words across languages and initializes a teacher classifier based on the translated seed words. Then, CLTS iteratively trains a more powerful student that also exploits the context of the seed words in unlabeled target documents and outperforms the teacher. CLTS is simple and surprisingly effective in 18 diverse languages: by transferring just 20 seed words, even a bag-of-words logistic regression student outperforms state-of-the-art cross-lingual methods (e.g., based on multilingual BERT). Moreover, CLTS can accommodate any type of student classifier: leveraging a monolingual BERT student leads to further improvements and outperforms even more expensive approaches by up to 12% in accuracy. Finally, CLTS addresses emerging tasks in low-resource languages using just a small number of word translations.

## introduction the main bottleneck in using supervised learning for multilingual document classification is the high cost of obtaining labeled documents for all of the target languages. to address this issue in a target language l t , we consider a cross-lingual text classification approach that requires labeled documents only in a source language l s and not in l t . existing approaches for transferring supervision across languages rely on large parallel corpora or machine translation systems, which are expensive to obtain and are not available for many languages. 1 to scale beyond high-resource languages, multilingual systems have to reduce the cross-lingual requirements and operate under a limited budget of cross-lingual resources. such systems typically ignore target-language supervision, and rely on feature representations that bridge languages, such as cross-lingual word embeddings @xcite or multilingual transformer models @xcite . this general approach is less expensive but has a key limitation: by not considering labeled documents in l t , it may fail to capture predictive patterns that are specific to l t . its performance is thus sensitive to the quality of pre-aligned features @xcite . in this work, we show how to obtain weak supervision for training accurate classifiers in l t without using manually labeled documents in l t or expensive document translations. we propose a novel approach for cross-lingual text classification that transfers weak supervision from l s to l t using minimal cross-lingual resources: we only require a small number of task-specific keywords, or seed words, to be translated from l s to l t . our core idea is that the most indicative seed words in l s often translate to words that are also indicative in l t . for instance, the word "wonderful" in english indicates positive sentiment, and so does its translation "magnifique" in french. thus, given a limited budget for word translations (e.g., from a bilingual speaker), only the most important seed words should be prioritized to transfer task-specific information from l s to l t . having access only to limited cross-lingual resources creates important challenges, which we address with a novel cross-lingual teacher-student method, clts (see figure 1 ). efficient transfer of supervision across languages: as a first contribution, we present a method for cross-lingual transfer in low-resource settings with a limited word translation budget. clts extracts the most important seed words using the translation budget as a sparsity-inducing regularizer when training a classifier in l s . then, it transfers seed words and the classifier's weights across languages, and initializes a teacher classifier in l t that uses the translated seed words. effective training of classifiers without using any labeled target documents: the teacher, as described above, predicts meaningful probabilities only for documents that contain translated seed words. because translations can induce errors and the translation budget is limited, the translated seed words may be noisy and not comprehensive for the task at hand. as a second contribution, we extend the "weakly-supervised co-training" method of @xcite to our cross-lingual setting for training a stronger student classifier using the teacher and unlabeled-only target documents. the student outperforms the teacher across all languages by 59.6%. robust performance across languages and tasks: as a third contribution, we empirically show the benefits of generating weak supervision in 18 diverse languages and 4 document classification tasks. with as few as 20 seed-word translations and a bag-of-words logistic regression student, clts outperforms state-of-the-art methods relying on more complex multilingual models, such as multilingual bert, across most languages. using a monolingual bert student leads to further improvements and outperforms even more expensive approaches (figure 2 ). clts does not require cross-lingual resources such as parallel corpora, machine translation systems, or pre-trained multilingual language models, which makes it applicable in low-resource settings. as a preliminary exploration, we address medical emergency situation detection in uyghur and sinhalese with just 50 translated seed words per language, which could be easily obtained from bilingual speakers. the rest of this paper is organized as follows. section 2 reviews related work. section 3 defines our problem of focus. section 4 presents our crosslingual teacher-student approach. foot_1 section 5 describes our experimental setup and results. finally, section 6 concludes and suggests future work. ## related work relevant work spans cross-lingual text classification and knowledge distillation. ## problem definition consider a source language l s , a target language l t , and a classification task with k predefined classes of interest @xmath0, . . . , k} (e.g., sentiment categories). labeled documents d @xmath1, y i )} n @xmath2, where y @xmath3. only unlabeled documents d @xmath4, where each target document x t i is a sequence of words from the target vocabulary v t . we assume that there is no significant shift in the conditional distribution of labels given documents across languages. furthermore, we assume a limited translation budget, so that up to b words can be translated from l s to l t . our goal is to use the labeled source documents d s , the unlabeled target documents d t , and the translations of no more than b source words to train a classifier that, given an unseen test document x t i in the target language l t , predicts the corresponding label y @xmath5. ## cross-lingual teacher-student we now describe our cross-lingual teacher-student method, clts, for cross-lingual text classification. given a limited budget of b translations, clts extracts only the b most important seed words in l s (section 4.1). then, clts transfers the seed words and their weights from l s to l t , to initialize a classifier in l t (section 4.2). using this classifier as a teacher, clts trains a student that predicts labels using both seed words and their context in target documents (section 4.3). ## experiments we now evaluate clts for several cross-lingual text classification tasks in various languages. ## conclusions and future work we presented a cross-lingual text classification method, clts, that efficiently transfers weak supervision across languages using minimal crosslingual resources. clts extracts and transfers just a small number of task-specific seed words, and creates a teacher that provides weak supervision for training a more powerful student in the target language. we present extensive experiments on 4 classification tasks and 18 diverse languages, including low-resource languages. our results show that even a simple student outperforms the teacher and previous state-of-the-art approaches with more complex models and more expensive resources, highlighting the promise of generating weak supervision in the target language. in future work, we plan to extend clts for handling cross-domain distribution shift (ziser and reichart, 2018) and multiple source languages @xcite . it would also be interesting to combine clts with available cross-lingual models, and extend clts for more tasks, such as cross-lingual named entity recognition @xcite , by considering teacher architectures beyond bag-of-seed-words. north american chapter of the association for computational linguistics: human language technologies. xin dong and gerard de melo. 2019. a robust selflearning framework for cross-lingual text classification. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing. julian eisenschlos, sebastian ruder, piotr czapla, marcin kadras, sylvain gugger, and jeremy howard. 2019. multifit: efficient multi-lingual language model fine-tuning. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing. manaal faruqui and chris dyer. 2014. improving vector space word representations using multilingual correlation. in proceedings of the 14th conference of the european chapter of the association for computational linguistics. goran glavaš, robert litschko, sebastian ruder, and ivan vulić. 2019. how to (properly) evaluate crosslingual word embeddings: on strong baselines, comparative analyses, and some misconceptions. in proceedings of the 57th annual meeting of the association for computational linguistics. alfio gliozzo and carlo strapparava. 2006. exploiting comparable corpora and bilingual dictionaries for cross-language text categorization. in proceedings of the 21st international conference on computational linguistics and the 44th annual meeting of the association for computational linguistics. association for computational linguistics. sebastian ruder, ivan vulić, and anders søgaard. 2019. a survey of cross-lingual word embedding models. journal of artificial intelligence research, 65:569-631. mohammad salameh, saif mohammad, and svetlana kiritchenko. 2015. sentiment after translation: a case-study on arabic social media posts. in proceedings of the 2015 conference of the north american chapter of the association for computational linguistics: human language technologies. holger schwenk and xian li. 2018. a corpus for multilingual document classification in eight languages. in proceedings of the eleventh international conference on language resources and evaluation. lei shi, rada mihalcea, and mingjun tian. 2010. cross language text classification by model translation and semi-supervised learning. in proceedings of the 2010 conference on empirical methods in natural language processing. association for computational linguistics. jasdeep singh, bryan mccann, richard socher, and caiming xiong. 2019. bert is not an interlingua and the bias of tokenization. in proceedings of the 2nd workshop on deep learning approaches for low-resource nlp. anders søgaard, sebastian ruder, and ivan vulić. 2018. on the limitations of unsupervised bilingual dictionary induction. in proceedings of the 56th annual meeting of the association for computational linguistics. stephanie strassel and jennifer tracey. 2016. lorelei language packs: data, tools, and resources for technology development in low resource languages. in proceedings of the tenth international conference on language resources and evaluation. oscar täckström, ryan mcdonald, and jakob uszkoreit. 2012. cross-lingual word clusters for direct transfer of linguistic structure. in proceedings of the 2012 conference of the north american chapter of the association for computational linguistics: human language technologies. association for computational linguistics. ivan vulić, goran glavaš, roi reichart, and anna korhonen. 2019. do we really need fully unsupervised cross-lingual embeddings? in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing. yogarshi vyas and marine carpuat. 2019. weakly supervised cross-lingual semantic relation classification via knowledge distillation. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing. xiaojun wan. 2008. using bilingual knowledge and ensemble techniques for unsupervised chinese sentiment analysis. in proceedings of the 2008 conference on empirical methods in natural language processing. xiaojun wan. 2009. co-training for cross-lingual sentiment classification. in proceedings of the joint conference of the 47th annual meeting of the acl and the 4th international joint conference on natural language processing of the afnlp. association for computational linguistics. thomas wolf, lysandre debut, victor sanh, julien chaumond, clement delangue, anthony moi, pierric cistac, tim rault, r'emi louf, morgan funtowicz, and jamie brew. 2019. huggingface's transformers: state-of-the-art natural language processing. arxiv, abs/1910.03771. shijie wu and mark dredze. 2019. beto, bentz, becas: the surprising cross-lingual effectiveness of bert. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing. jiateng xie, zhilin yang, graham neubig, noah a smith, and jaime g carbonell. 2018. neural crosslingual named entity recognition with minimal resources. in proceedings of the 2018 conference on empirical methods in natural language processing. ruochen xu and yiming yang. 2017. cross-lingual distillation for text classification. in proceedings of the 55th annual meeting of the association for computational linguistics. michelle yuan, mozhi zhang, benjamin van durme, leah findlater, and jordan boyd-graber. 2020. interactive refinement of cross-lingual word embeddings. in proceedings of the 2020 conference on empirical methods in natural language processing. xinjie zhou, xiaojun wan, and jianguo xiao. 2016. cross-lingual sentiment classification with bilingual document representation learning. in proceedings of the 54th annual meeting of the association for computational linguistics. yftah ziser and roi reichart. 2018. deep pivot-based modeling for cross-language cross-domain transfer with minimal guidance. in proceedings of the 2018 conference on empirical methods in natural language processing.