On the Role of Parallel Data in Cross-lingual Transfer Learning (2023)

Abstract: While prior work has established that the use of parallel data is conducive for cross-lingual learning, it is unclear if the improvements come from the data itself, or if it is the modeling of parallel interactions that matters. Exploring this, we examine the usage of unsupervised machine translation to generate synthetic parallel data, and compare it to supervised machine translation and gold parallel data. We find that even model generated parallel data can be useful for downstream tasks, in both a general setting (continued pretraining) as well as the task-specific setting (translate-train), although our best results are still obtained using real parallel data. Our findings suggest that existing multilingual models do not exploit the full potential of monolingual data, and prompt the community to reconsider the traditional categorization of cross-lingual learning approaches.

## introduction multilingual models have been shown to generalize across languages in a zero-shot fashion @xcite @xcite . these models are usually pretrained on concatenated monolingual corpora in multiple languages using some form of language modeling or denoising objective. the models are then finetuned using labeled downstream data in the source language (typically english), which makes them capable of generalizing to the target language thanks to the aligned representations learned at pretraining. while this paradigm does not require any data in the target language other than the monolingual pretraining corpus, prior work has reported improved results by incorporating parallel data into the pipeline, either at pretraining or finetuning time. our experiments on natural language inference (nli), question answering (qa) and named entity recognition (ner) show that the explicit modeling of parallel interactions is indeed beneficial, demonstrating that existing pretraining and finetuning methods do not exploit the full potential of monolingual data. however, our best results are obtained using real parallel data-either directly or indirectly through supervised mt-showing that there is also some inherent value on it. in the light of these results, we argue that the traditional categorization of cross-lingual transfer approaches into zero-shot, translate-train and translate-test (figure 1a ) falls short at capturing the required detail for a fair comparison across different approaches. given this, we encourage further research on understanding what the contribution of monolingual and parallel data is, and how to best leverage them (directly or indirectly through mt, and at different parts of the pipeline), which requires thinking beyond the boundaries of the existing categorization (figure 1b ). ## reconsidering the categorization of cross-lingual learning approaches as illustrated in figure 1a , approaches to crosslingual learning have traditionally been classified into 3 categories: zero-shot (finetune a multilingual model on english and zero-shot transfer into the target language), translate-train (translate the english training data into the target languages through mt and finetune a multilingual model), and translatetest (translate the test set into english and run inference using a monolingual model). this distinction is primarily based on which stage of the pipeline mt is incorporated into. while relevant from a practical perspective, we believe that, if taken in a rigid manner, such a framework can hinder addressing the more fundamental question of what the contribution of each data source is, and how to best leverage each of them. more concretely, as shown in figure 1b , there are different data types that one can use (monolingual source corpora, monolingual target corpora and parallel corpora, in addition to downstream data), which can be incorporated at different stages of the pipeline (pretraining, finetuning, testing) and via different procedures (directly or indirectly through mt). we argue that research in cross-lingual learning should aim to understand how the variants in each dimension as well the interactions between them impact downstream performance, which can require thinking beyond the boundaries of the 3 conventional categories. for instance, our variant using unsupervised mt to translate the downstream training data would fall within the definition of translate-train. however, this approach is more comparable to zero-shot in that it only uses monolingual data, and it would be unfair to compare it to conventional translate-train systems that rely on parallel data to train the mt system. ## related work prior work has explored the extent to which monolingual pretraining relies on knowledge transfer from unlabeled corpora by using synthetic data @xcite or downstream data @xcite instead, and similar ideas have also been explored in computer vision @xcite . however, to the best of our knowledge, we are first to examine if cross-lingual learning also relies on knowledge transfer from parallel data. our use of synthetic parallel corpora is also connected with back-translation, which is widely used in mt @xcite . however, conventional mt systems are trained on parallel data, and backtranslation is usually motivated as a way to leverage additional (monolingual) data. in contrast, our unsupervised mt variant does not use any additional data compared to regular pretraining. ## conclusions in this work, we show that even model-generated parallel data can be useful for cross-lingual learning-greatly expanding the possibilities for multilingual models to improve their performance by taking advantage of their own machine translation capabilities. given this, we advocate for investigating the optimal way to leverage monolingual and/or parallel data for cross-lingual learning, which might require thinking beyond the boundaries of the conventional zero-shot, translate-train and translate-test categories.