Using Machine Translation to Augment Multilingual Classification (2024)

Abstract: An all-too-present bottleneck for text classification model development is the need to annotate training data and this need is multiplied for multilingual classifiers. Fortunately, contemporary machine translation models are both easily accessible and have dependable translation quality, making it possible to translate labeled training data from one language into another. Here, we explore the effects of using machine translation to fine-tune a multilingual model for a classification task across multiple languages. We also investigate the benefits of using a novel technique, originally proposed in the field of image captioning, to account for potential negative effects of tuning models on translated data. We show that translated data are of sufficient quality to tune multilingual classifiers and that this novel loss technique is able to offer some improvement over models tuned without it.

## introduction one of the most common uses of machine learning for natural language processing (nlp) is the classification of text into one of multiple mutuallyinclusive or mutually-exclusive labels. recently, generative llms, such as palm @xcite and chatgpt @xcite have shown exciting and impressive capabilities to do zero-or few-shot prompting, classify text given only a few examples for the task across a variety of languages. nevertheless, it is still the case that the highest performing and most efficient means to classify text is the use of a bespoke classifier trained with hundreds or thousands labeled examples @xcite , particularly when the task requires a level of human-like subjectivity or general reasoning ability @xcite . to this end, finding or creating a corpus of labeled examples is a necessary step in the creation of any classifier. for high-resource languages like english, which have many existing labeled corpora available and large populations of annotators on crowd-sourced workers such as amazon mechanical turk, the challenge of creating or finding training and evaluation data can be costly, but not prohibitively so. yet, for lower-resourced languages which lack existing annotated corpora and have smaller or even non-existent populations on these large annotation platforms, acquiring the required training data can prove to be much more difficult. moreover, if the model is intended to be able to perform the same classification across multiple languages, the time and effort required to annotate training data becomes multiplicative. fortunately, classification is not alone in the applications of machine learning in nlp. machine translation (mt) has seen major improvements in recent years @xcite , accelerated by the adoption of the transformer architecture @xcite . to date, several options for high quality machine translation currently exist, between api services and open-source models. mt api services, such as google translate, have become nearly ubiquitous, provide high quality translations, while still being relatively inexpensive. in fact, in one experiment, translating data using google translate into english and using existing english-trained classifier models outperformed certain models trained on the original language directly @xcite . in addition to mt api services, several open-source translation models are easily available, such as the multilingual m2m100 model @xcite ), nllb200 model @xcite or the over 1400 models trained by the university of helsinki @xcite , with many of these models have performance that approaches or exceeds that of mt apis @xcite . with this in mind, it may be the case that translating an existing, labeled dataset with one of the aforementioned mt options is a feasible alternative to creating a novel dataset directly in that language. this has several benefits. firstly, it avoids the problem of existing corpora or annotation options not existing for the language in question. secondly, it minimizes the data needed for multilingual models and allows annotations for one language to serve another. here, we ask if it is possible to use mt to train a multilingual model, given only original, annotated data for a single language. of course, the potential benefits of using mt to train a multilingual model are still affected by the old machine learning adage: garbage in, garbage out. even the best translations, either human or machine, will lose some of the information of the original language, which will inevitably lead to dropped performance for a model trained on the translated examples. fortunately, the problem of training models using semantically similar but imperfect pairs of data is not unique to the task at hand and there is a growing body of research which may provide some benefit. in particular, image captioning is a task to generate the ideal natural language text caption for an image and these captioning models must learn to represent semantically related data from very different modalities similarly, i.e., text and images @xcite . in this way, image captioning is somewhat analogous to the task of training on translated data, where we want to have semantically identical text from different languages predicted to have the same labels. as a result, we ask in addition whether some of the model training techniques used in image captioning models can lead to improved performance for multilingual models trained using mt data. ## related work this work is by no means the first to suggest the usage of machine translation to create or augment datasets for lower resourced languages. @xcite and @xcite augmented chinese language corpora with annotated data translated from english to improve the performance of a chinese-language sentiment analysis model. on the other hand, @xcite and @xcite used existing api translation services to translate annotated data from english into lower-resourced languages and trained classifiers solely on these translated data, finding that classifiers trained on translated data were fairly accurate but did see drops in performance, likely due to the effects of imperfect translations of the training data. it should be noted that training a model from scratch is not the only means to create an accurate classifier, particularly for lower-resourced languages. large multilingual transformer models such as m-bert @xcite , xlm-roberta @xcite or gpt-3 @xcite have been shown to have the ability to generalize from one language to the other, i.e., train in one language and improve test performance in another language, @xcite , but benefits of this vary on the languages in question, with languages that share closer genealogical origin or structural similarities benefiting more from inter-language transfer. regardless, training a model with examples of a particular language dependably yields the best classifier for new data in that language. nevertheless, to date there has been no investigation of how fine-tuning large multilingual transformer models on translated data affects final performance compared to simple interlanguage transfer. moreover, previous work to train models using translated data employed a naive approach, treating translated data as if it were no different than original, untranslated data which annotated itself. in this work, we investigate both how multilingual transformer models trained on translated data perform compared to interlanguage transfer and explore a means to mitigate imperfect translation quality when creating these training datasets. ## image captioning and image-text contrastive loss image-text contrastive (itc) loss is a technique used when training multimodal models to caption images with natural language descriptions @xcite . for example, blip @xcite is a image-captioning model that was trained with a mix of human-and artificially-annotated images where itc loss was integral to the models ability to learn from noisy, artificially-annotated data. itc loss, then, has been shown to mitigate negative effects of both noise and different modalities for multimodal models. at an intuitional level, these captioning models decompose text and images into a shared embedding space and itc loss seeks to penalize cases where related image-text pairs are dissimilar in this shared embedding space. in other words, itc looks seeks to bring semantically related items from disparate modalities closer in a shared embedded space and has empirically improved image-captioning models, with little impact on training time or resources. training multilingual classification models with translated data bears a similarity to captioning, though rather than have semantically related examples from different modalities, there are semantically parallel data in different languages. that being the case, we will be a slightly modified form of itc loss, namely original-translated contrastive (otc) loss, to enforce similarity within a batch between data from the original language and its translated counterpart. like itc loss, otc loss penalizes a transformer model for dissimilar embedding representations for translated pairs. one way to think of it is that this loss encourages the model to embed sentences with the same meaning identically, regardless of language. in detail, we implement otc loss as follows. we begin by deriving a probability of each original/translated pairing in a training minibatch, p o2t and p t2o , that is, which original examples pairs with which translated example and vice versa. here, s(t, o) is a similarity function between the original, untranslated data and the translated examples in a minibatch. we compute s(t, o) by first extracting and normalizing the embedding for the initial [cls] token after the final attention head of the encoder stack in m-bert, computing a pairwise dot product for all possible pairs of original and translated data and dividing by τ , which is a learnable parameter. we then apply the softmax function as a way to represent the likelihood of each original/translated match. ideally, each correct original/translated pair will have the most similar embeddings, resulting in a value close to 1 after softmax. as a final step, we compute the cross-entropy between the result of the previous step and a target vector which encodes the correct original/translated pairs, weighting this by a hyperparameter, α otc . following blip @xcite , we set α @xmath0.4 for all runs. ## results in these experiments, we asked two simple questions: 1) how feasible is it to tune a multilingual transformer model on translated data and 2) does the inclusion of otc loss improve model performance for languages where only translated training data was used. in answer to the first, for each of the six languages in the original dataset, models finetuned with translated data showed higher f1-micro scores 3 on the held-out test set, compared to models trained with only original data for a single language (see that said, it is clear that the use of translated training data does improve model performance, even if the trained model only sees translated examples for that language. it should also be noted that due to the batching and sampling strategy used here, models trained with translated data saw far fewer examples of each language where they only saw translated data. that is, because each original review was paired with a single translated example out of five possible translated, these models were exposed to roughly one fifth of the data for translated languages and still saw a sizable boost in performance. moving on to the effect of otc loss, when averaging across all runs where a language in the testing set was only represented by translated data, otc loss shows an improvement over models trained without it for all languages. to ensure that the results here were in fact statistically significant, we fit a linear mixed-effect model to predict final model f1-micro for a language, given the hyperparameters of a particular tuning run. mixed-effect models are able to accurately evaluate the contribution of different fixedeffect independent variables, e.g., whether otc was used when training a particular model, on dependent variables, e.g., the final accuracy of the trained model, all the while being robust to expected random variance between trials, e.g., because of random initialization and batching, some deep learning models score higher than others with identical hyperparameters (see @xcite , @xcite for more). this statistical model was fit to predict perlanguage test f1-micro, given a random effect of each model run and three fixed effects: i) the tested language, ii) the identity of the single original language and iii) whether otc loss was added. otc was found to have a significant, positive effect (@xmath1.036, std.@xmath2.017, for all model details see 2), indicating that even after taking into consideration differences between languages and random variance for each multilingual model, the inclusion of otc loss did yield an improved final model f1-micro. ## discussion and future directions we investigated the feasibility of using translated text to fine-tune a multilingual transformer model, as well as any potential gains by utilizing a novel application of deep learning technique to improve performance. we found that models trained using only translated data for a language do show a noticeable improvement over baselines, though as expected, there was still a performance drop from using original, untranslated data for that language. we also found that slight further gains can be achieved by the use of otc loss, suggesting that training the model in such a way where it is sensitive to potential data issues improves its ability to generalize. granted, this is a very open problem and results of using translated data to tune a multilingual classifier will vary highly depending on the quality of mt model used, architecture of the classifier being tuned and the type of classification being modeled. nevertheless, the results here are exciting for multiple reasons. firstly, as suggested by previous works @xcite , as an example), mt is useful tool for language-specific dataset creation when creating a dataset for that language directly may prove difficult. in this case, we showed that m-bert models tuned on translated examples showed large gains over simple multilingual transfer during training. this is particularly interesting given that for each translated language, the model was only given a fraction of samples compared to the original language due to the 1:1 ratio of original and translated data. a future direction for this work may be to adjust this ratio or the number of languages in the dataset to investigate how this affects model training. secondly, the use of otc loss was shown to lead to a small, but robust boost to performance. this suggests that methods of mitigating the natural effects of translation have a potential to bridge the gap, so to speak, between models trained on translated data and on datasets in the target language directly. particularly relevant, chinese, which is linguistically dissimilar from the majority of languages in the set used here, showed a mixed ability to benefit from training with other languages, but a clearer improvement using otc loss. this may suggest that otc loss is able to mitigate structural differences between languages and a future direction for this may be to explore exactly how otc loss affects individual examples and how other noise-reduction techniques may lead to further gains in model performance. putting this together, this is an indication that mt-augmented datasets stand as a good first step for developing multilingual classification models. given that mt can quickly and efficiently expand an annotated dataset from one language into another and that translated dataset is of sufficient quality to improve over basic interlingual transfer, this technique has great potential to expanding classification tasks to new languages quickly. in addition, otc loss may be able to slightly but significantly increase the quality of these models with no additional data. all in all, we are confident that the use of mt augmentation is an exciting and interesting topic for future exploration.