Can Cross-Lingual Transferability of Multilingual Transformers Be Activated Without End-Task Data? (2023)

Abstract: Pretrained multilingual Transformers have achieved great success in cross-lingual transfer learning. Current methods typically activate the cross-lingual transferability of multilingual Transformers by fine-tuning them on end-task data. However, the methods cannot perform cross-lingual transfer when end-task data are unavailable. In this work, we explore whether the cross-lingual transferability can be activated without end-task data. We propose a cross-lingual transfer method, named PlugIn-X. PlugIn-X disassembles monolingual and multilingual Transformers into sub-modules, and reassembles them to be the multilingual end-task model. After representation adaptation, PlugIn-X finally performs cross-lingual transfer in a plug-and-play style. Experimental results show that PlugIn-X successfully activates the cross-lingual transferability of multilingual Transformers without accessing end-task data. Moreover, we analyze how the cross-model representation alignment affects the cross-lingual transferability.

## introduction annotated data is crucial for learning natural language processing (nlp) models, but they are mostly only available in high-resource languages, typically in english, making nlp applications hard to access in other languages. this motivates the studies on cross-lingual transfer, which aims to transfer knowledge from a source language to other languages. cross-lingual transfer has greatly pushed the state of the art on nlp tasks in a wide range of languages @xcite @xcite . advances in cross-lingual transfer can be substantially attributed to the cross-lingual transferability discovered in pretrained. multilingual transformers @xcite . pretrained on large-scale multilingual text data, the multilingual transformers perform crosslingual transfer surprisingly well on a wide range of tasks by simply fine-tuning them @xcite @xcite . based on this finding, follow-up studies further improve the transfer performance in two aspects, by (1) designing pretraining tasks and pretraining multilingual models with better cross-lingual transferability @xcite , or (2) developing fine-tuning methods with reduced cross-lingual representation discrepancy @xcite . current methods typically activate the transferability of multilingual transformers by fine-tuning them on end-task data. however, they cannot perform cross-lingual transfer when end-task data are unavailable. it is common that some publicly available models are trained with non-public in-house data. in this situation, one can access an alreadytrained end-task model but cannot access the inhouse end-task data due to privacy policies or other legal issues. as a consequence, current methods cannot perform cross-lingual transfer for such models because of the lack of end-task data. in this work, we study the research question: whether the cross-lingual transferability of multilingual transformers can be activated without end-task data? we focus on the situation that we can access an already-trained monolingual end-task model but cannot access the in-house end-task data, and we would like to perform cross-lingual transfer for the model. to achieve this, we propose a cross-lingual transfer method named plugin-x. plugin-x disassembles the monolingual end-task model and multilingual models, and reassembles them into the multilingual end-task model. with cross-model representation adaptation, plugin-x finally performs cross-lingual transfer in a plugand-play style. to answer the research question, we conduct experiments on the cross-lingual transfer on the natu-ral language inference and the extractive question answering tasks. in the experiments, the multilingual model only sees unlabeled raw english text, so the performance of the reassembled model indicates whether the cross-lingual transferability is activated. experimental results show that plugin-x successfully transfers the already-trained monolingual end-task models to other languages. moreover, we analyze how the cross-model representation alignment affects the cross-lingual transferability of multilingual transformers, and discuss the benefits of our work. our contributions are summarized as follows: ## related work cross-lingual transfer aims to transfer knowledge from a source language to target languages. early work on cross-lingual transfer focuses on learning cross-lingual word embeddings (clwe; @xcite with shared task modules upon the embeddings, which has been applied to document classification @xcite , sequence labeling @xcite , dialogue systems (schuster et al., 2019), etc. follow-up studies design algorithms to better align the word embedding spaces @xcite or relax the bilingual supervision of lexicons and parallel sentences @xcite . later studies introduce sentence-level alignment objectives and obtain better results @xcite . most recently, fine-tuning pretrained language models (plm; @xcite @xcite ) have become the mainstream approach to cross-lingual transfer. benefiting from large-scale pretraining, pretrained multilingual language models are shown to be of cross-lingual transferability without explicit constraints @xcite . based on this finding, much effort has been made to improve transferability via (1) pretraining new multilingual language models @xcite @xcite , or (2) introducing extra supervision such as translated data to the fine-tuning procedure @xcite @xcite . plm-based methods have pushed the state of the art of the cross-lingual transfer on a wide range of tasks @xcite @xcite . ## methods in this section, we first describe the problem definition. then, we present how plugin-x performs cross-lingual transfer with model reassembling and representation adaptation. ## discussion transferability activation to answer our research question, we have conducted experiments on cross-lingual transfer under the public-modelin-house-data (pmid) setting. our experimental results in section 4.2 and section 4.3 show that plugin-x successfully activates the cross-lingual transferability of multilingual transformers without using the in-house end-task data. notice that our goal is to answer the research question, rather than develop a state-of-the-art algorithm for the common cross-lingual transfer setting. transferability quantification it is difficult to quantify cross-lingual transferability because the results are non-comparable and the compared models typically have different performances in the source language. we propose to transfer an alreadytrained end-task model to other languages. as the end-task model is stationary, the transfer gap is only dependent on cross-lingual transferability. therefore, we recommend that the models to be evaluated should transfer the same end-task model to obtain comparable transferability scores. ## conclusion in this paper, we have investigated whether the cross-lingual transferability of multilingual transformers can be activated without end-task data. we present a new problem setting of cross-lingual transfer, the public-model-in-house-data (pmid) setting. to achieve cross-lingual transfer under pmid, we propose plugin-x, which reassembles the monolingual end-task model and multilingual models as a multilingual end-task model. our results show that plugin-x successfully activates the cross-lingual transferability of multilingual transformers without accessing the in-house end-task data. for future work, we would like to study the research question on more types of models such as large language models @xcite .