F ree T ransfer- X : Safe and Label-Free Cross-Lingual Transfer from Off-the-Shelf Models (2022)

Abstract: Cross-lingual transfer (CLT) is of various applications. However, labeled cross-lingual corpus is expensive or even inaccessible, especially in the fields where labels are private, such as diagnostic results of symptoms in medicine and user profiles in business. Nevertheless, there are off-the-shelf models in these sensitive fields. Instead of pursuing the original labels, a workaround for CLT is to transfer knowledge from the off-the-shelf models without labels. To this end, we define a novel CLT problem named FreeTransfer-X that aims to achieve knowledge transfer from the off-the-shelf models in rich-resource languages. To address the problem, we propose a 2-step knowledge distillation (KD, Hinton et al., 2015) framework based on multilingual pre-trained language models (mPLM). The significant improvement over strong neural machine translation (NMT) baselines demonstrates the effectiveness of the proposed method. In addition to reducing annotation cost and protecting private labels, the proposed method is compatible with different networks and easy to be deployed. Finally, a range of analyses indicate the great potential of the proposed method.

## introduction cross-lingual transfer (clt) is a critical topic for natural language processing due to the data imbalance between languages. while models of richresource languages (e.g. english) have been applied on various real-world tasks, the progress on poor-resource languages lags behind. clt researches enable the knowledge transfer from the rich-resource languages to the poor-resource languages. although the application of clt is valuable, data labels are expensive or even inaccessible in private and sensitive domains, such as medicine and business. for example, the diagnostic results of a user's symptoms are private and a company's internal description of users are confidential. since short of labels for clt, even though there are excellent applications in rich-resource languages, it is difficult to benefit the people using poor-resource languages. previous clt researches have not well studied how to leverage knowledge of rich-resource languages without labels. to define and tackle this problem will benefit both the community and the industry. in order to reduce the demand of labels, existing works mainly fall into two paradigms as follows. one paradigm focuses on learning languageagnostic representation and model parameters. clt is realized by either aligning parameters of monolingual models or sharing parameters among different languages @xcite @xcite . the objective is to build a unified representation, which is used by downstream tasks, for all the languages. in this paradigm, although the demand of labels is reduced, it still requires a certain number of labels to adapt the model to a particular language and task. besides, models in this paradigm are usually large-scale transformers @xcite based on mplms, which limits their deployment in real-world. another paradigm is to leverage machine translation (mt) systems to generate training or testing pseudo-corpus for a specific language @xcite . for simplicity, we take english as the rich-resource languages in this paper. translate-train translates annotated training corpus from english to other languages. gold labels are directly applied to the translated data. although labels in poor-resource languages are not required, gold labels in english are still necessary. on the contrary, translate-test translates testing corpus from poor-resource languages to english. this method can directly leverage off-the-shelf english models, but it runs a 2pass inference which highly limits its efficiency. both the two clt paradigms mentioned above require language-specific and task-specific labels, except for the 2-pass translate-test. the demand of labels highly limits the reuse of the english knowledge in private and sensitive domains. then a question comes up: is it possible to perform clt totally without labels? in this paper, we define a novel problem: safe and label-free cross-lingual transfer from off-theshelf models (freetransfer-x). the freetransfer-x asks researchers to achieve clt only with off-theshelf english models but any labels, as formally defined in section 2.1. to the best knowledge of the authors, it's the first time that the freetransfer-x is clearly defined. to address the freetransfer-x, we propose a 2-step knowledge distillation (kd, @xcite framework based on mplm, as shown in figure 1 . given an off-the-shelf model θ src in the source language (e.g. english), first we take θ src as the teacher and an mplm model θ m src as the student, then train θ m src on unlabeled corpus d src . second, we take θ m tgt as the teacher and train a student θ tgt on unlabeled corpus d tgt . this cross-lingual transfer framework is label-free and applicable for any model architecture. experimental results demonstrate the effectiveness of the proposed framework on both sentence classification and sequence tagging. in short, the major contributions of this work include: freetransfer-x is defined. the freetransfer-x asks researchers to achieve clt from off-the-shelf models without using labels. it reduces the labeling cost and protects the labels in private domains such as medicine and business. ## further analysis in order to explore the potential of the proposed framework, we analyze it in more details. for simplicity, experiments in this section are conducted only on the multiatis++ sentence classification task. ## conclusions in this paper, we define a novel cross-lingual transfer (clt) problem -freetransfer-x, especially for clt in private scenarios such as medical and business. the freetransfer-x is defined to transfer knowledge from off-the-shelf models in rich-resource languages to poor-resource languages, without labeled corpora.