T 3 L : Translate-and-Test Transfer Learning for Cross-Lingual Text Classification (2023)

Abstract: Cross-lingual text classification leverages text classifiers trained in a high-resource language to perform text classification in other languages with no or minimal fine-tuning (zero/ few-shots cross-lingual transfer). Nowadays, cross-lingual text classifiers are typically built on large-scale, multilingual language models (LMs) pretrained on a variety of languages of interest. However, the performance of these models varies significantly across languages and classification tasks, suggesting that the superposition of the language modelling and classification tasks is not always effective. For this reason, in this paper we propose revisiting the classic “translate-and-test” pipeline to neatly separate the translation and classification stages. The proposed approach couples 1) a neural machine translator translating from the targeted language to a high-resource language, with 2) a text classifier trained in the high-resource language, but the neural machine translator generates “soft” translations to permit end-to-end backpropagation during fine-tuning of the pipeline. Extensive experiments have been carried out over three cross-lingual text classification datasets (XNLI, MLDoc, and MultiEURLEX), with the results showing that the proposed approach has significantly improved performance over a competitive baseline.

## introduction pretrained language models (lms) have become ubiquitous in natural language processing (nlp) in recent years @xcite @xcite @xcite . these large, transformer-based @xcite deep networks are often pretrained over hundreds of gigabytes or even terabytes of textual data in an unsupervised man-ner using masked language model training objectives. after pretraining, lms are typically fine-tuned with supervised datasets for a variety of downstream tasks (e.g., text classification, natural language generation, question answering), regularly achieving state-of-the-art results. multilingual lms @xcite @xcite )-lms trained with monolingual corpora from multiple languages (e.g., 25, 50, 100)-are typically used to extend the downstream tasks to multilingual and crosslingual scenarios. in their cross-lingual application, multilingual lms are fine-tuned for a given downstream task using training data in a highresource language (e.g., english), and then used for inference in another, less-resourced target language, either with no additional fine-tuning (zeroshot) or minimal fine-tuning (few-shot) in the target language. however, the cross-lingual performance of multilingual lms tends to vary significantly across languages and downstream tasks, fundamentally because of the deeply uneven training resources and the intrinsic linguistic differences @xcite . while it is possible, in principle, to expand the pretraining with additional monolingual data in the target language, the computational requirements are often remarkably expensive. on top of that, many low-resource languages do not avail of sufficient amounts of monolingual data in the first place @xcite . more so, recent research has shown that multilingual lms can suffer from the curse of multilinguality @xcite , which means that at a parity of model capacity, training an lm with too many languages may cause language interference and degrade its cross-lingual performance. in addition, there is room to speculate that the superposition of a downstream task to the multilingual embedding in the same encoder may lead to further interference. for these reasons, in this paper we propose approaching cross-lingual text classification by combining explicit translation from the target language to a high-resource language (i.e., english) with classification in the high-resource language. this approach is known as translate-and-test in the literature @xcite , and in recent years it has mainly been regarded as a mere baseline for comparison. however, it offers principled advantages: 1) the possibility to reuse the wealth of existing resources for machine translation, i.e., pretrained mt models, accumulated over decades of research and publicly shared in repositories such as huggingface foot_0 ; and 2) exploiting robust text classifiers trained in highresource languages. on the other hand, a cascaded approach such as translate-and-test is admittedly vulnerable to errors from both the translation stage and the possible semantic mismatch between the lm employed for translation and that used for classification. for this reason, we also propose fine-tuning the two modules end-to-end with a few-shot classification training data in the target language. to this aim, the proposed approachnamed t3l for translation-and-test transfer learning-generates soft translations (jauregi @xcite to ensure the continuity of the network and allow backpropagation end-toend. 2 our experimental results show that the joint fine-tuning has improved the coupling of the two models and significantly contributed to the final classification accuracy. in addition, unlike lm pretraining, which is extremely computationallyintensive, fine-tuning this model has been easily manageable and has been carried out with a single gpu for all datasets. overall, the main contributions of our paper are: ## related work cross-lingual text classification dates back to at least the seminal work of @xcite . their approach was based on machine translation from the targeted language to english, but was restricted to a selected terminology per class since full-document translation at the time was still very expensive and inaccurate. while much of the early work kept on being based on explicit or latent (e.g., @xcite machine translation, the advent of word embeddings paved the way for approaches based on multilingual representations of text, which leveraged distributional similarities across languages @xcite . eventually, the rise of the masked language models @xcite @xcite made it possible to train the multilingual representations from monolingual data alone, relieving the need for parallel corpora. currently, the state of the art of cross-lingual text classification is held by multilingual lms such as mbert @xcite , mbart @xcite , xlm-r @xcite , and mt5 @xcite . cross-lingual benchmarks such as xtreme @xcite and xglue @xcite have helped established a reference for text classification and other nlp tasks over many languages. generally, the models that are the highest in the leaderboards have been trained with masked language models, electra-style models that use additional discriminative pretraining tasks @xcite , and models with cross-attention modules that build language interdependencies explicitly @xcite , all typically using hundreds of gigabytes or terabytes of pretraining data (e.g., 2.5 tb of common crawl data for xlm-r). however, such large, pretrained multilingual models still suffer from significant performance limitations, particularly in low-resource languages. for example, @xcite have shown that a basic transformer directly trained in 11 lowresource african languages has neatly outperformed larger models such as mbert and xlm-r in text classification, even in languages present in the larger models. other work has addressed the curse of multilinguality @xcite by adding language-specific adapter modules @xcite or reparametrizing the transformer around the highest-resource language @xcite . however, significant performance differences between languages still persist. the extensive availability of resources for machine translation such as datasets and pretrained multilingual mt models (e.g., many-to-many, many-to-one) justifies its continued exploration for cross-lingual text classification @xcite @xcite . a well-practised approach is translate-and-train, which first translates all the classification training data from the high-resource language (e.g., english) to the target language using an existing machine translation model, and then trains a classifier in the target language. in a sense, it could be objected that such an approach is not really ''cross-lingual'', in that the classification is performed by a model directly trained in the target language, making it more akin to a monolingual classifier trained with silver standard corpora. however, its main limitation lies in its inflexibility, since performing t classification tasks over l languages would require training t l classification models. a more flexible alternative is offered by translate-and-test, where the text from the target language is first translated to the high-resource language, and then classified using a classifier trained in the high-resource language. by comparison, this approach is genuinely cross-lingual and only requires training t classification models in total. however, both approaches have proved able to achieve competitive results, provided that the machine translation step is sufficiently accurate. our work follows the translate-and-test approach, aiming to jointly fine-tune the machine translator and the classifier over few shots of the target classification task. this is in a similar vein with other works @xcite that have fine-tuned a machine translator with reinforcement learning using the output of the downstream task classifier as reward. in this way, the machine translator can learn to adjust its translations optimally for the classification in the target language. however, where available, differentiable objectives are generally preferable to reinforcement learning due to the high variance and instability of the latter. for this reason, in our approach, we leverage soft translations (i.e., predictions of differentiable token embeddings rather than tokens) from the machine translation module to retain differentiability through the entire network and fine-tune the classifier and the translator jointly end-to-end. ## t3l in this section, we formally describe the proposed approach, t3l, which essentially consists of two language models-a machine translation language model (mt-lm) and a text classification language model (tc-lm)-connected by a sequence of expected embeddings. figure 1 shows a detailed diagram of the model architecture. the mt-lm takes a sequence of tokens in the target language, {x 1 , . . . , x n }, as input, and translates it into a sequence of tokens, {t 1 , . . . , t m }, in the high-resource language (i.e., a language that has enough resources to build a strong classifier). at each decoding step j, given the input sequence and the previous decoded token, t j-1 , the lm generates a vector p j that represents the probability distribution over the words in the vocabulary, v : with p v j the probability of the v-th token in the vocabulary, and v the vocabulary size. to generate the probability distribution vector, p j , we use the standard softmax, but other approaches could be explored in alternative, including sparsemax @xcite and the gumbel-softmax @xcite , to respectively encourage sparsity or controllable diversity in the probability vector. in addition, different decoding strategies such as beam search and sampling can be employed. as the next step, the generated predictions are passed in input to the tc-lm model, yet ensuring that the whole network remains differentiable end-to-end. to this aim, the sequence of probability vectors, {p 1 , . . . , p m }, generated by the mt-lm is used to ''mix'' the token embedding layer, e, of the tc-lm to obtain a sequence of expected embeddings, {ē 1 . . . ēj . . . ēm }: finally, the tc-lm classifier predicts a text class given the sequence of expected embeddings: ## results in general, these results show that the proposed approach, based on the translate-and-test pipeline and soft translations, has been able to improve the performance over the lm baseline for many languages even in the zero-shot configuration. in addition, the end-to-end fine-tuning has helped further increase the accuracy in all tested cases. ## discussion and limitations our results have shown that t3l is a competitive approach for cross-lingual text classification, particularly for low-resource languages. in broad terms, the proposed approach does not impose onerous training or inference requirements. the number of trained parameters is approximately the same as for a single language model. in all the experiments, we have been able to leverage an existing, pretrained translator without the need for dedicated training, with the exception of those languages that were not included in the pretraining corpora. the experiments have also shown that increasing the translation accuracy can have a positive impact on the classification accuracy, and that this can be achieved with parallel corpora of relatively small size. this seems a very attractive alternative to trying to improve the classification accuracy by increasing the already very large monolingual training corpora used for training the lm baseline. in addition, the same translation module can be shared by an unlimited number of different downstream tasks. it is also worth noting that an approach such as t3l is not confined to text classification tasks, but could instead be easily extended to perform natural language generation tasks. nevertheless, t3l also has its shortcomings. the main limitation that we identify is the vocabulary constraint described in section 3.1. since the output of the translator module is a sequence of embeddings rather than a detokenized string, the vocabulary of the classification module needs to be the same as that of the translator. as such, we believe that it would be very useful to remove or mollify this constraint in future work, and for this we plan to explore differentiable alignment another intrinsic limitation of translate-and-test approaches is their higher inference time, mainly due to the sequential decoding introduced by the translation stage. to exemplify the relative slowdown, ## conclusion this paper has presented t3l, a novel approach for cross-lingual text classification built upon the translate-and-test strategy and allowing end-toend fine-tuning thanks to its use of soft translations. extensive experimental results over three benchmark cross-lingual datasets have shown that t3l is a performing approach that has worked particularly well over low-resource languages and has outperformed a completely comparable cross-lingual lm baseline in the vast majority of cases. while a model such as t3l certainly introduces some overheads compared to a conventional cross-lingual lm in terms of memory requirements and inference speed, we hope that its attractive classification performance may spark a renewed interest in the translate-and-test pipeline.