Improving Similar Language Translation With Transfer Learning (2021)

Abstract: We investigate transfer learning based on pre-trained neural machine translation models to translate between (low-resource) similar languages. This work is part of our contribution to the WMT 2021 Similar Languages Translation Shared Task where we submitted models for different language pairs, including French-Bambara, Spanish-Catalan, and Spanish-Portuguese in both directions. Our models for Catalan-Spanish (82.79 BLEU)and Portuguese-Spanish (87.11 BLEU) rank top 1 in the official shared task evaluation, and we are the only team to submit models for the French-Bambara pairs.

## introduction we present the findings from our participation in the wmt 2021 similar language translation shared task 2021, which focused on translation between similar language pairs in low-resource settings. the similar languages task focuses on building machine translation (mt) systems for translation between pairs of similar languages, without english as a pivot language. similarity between languages interacts with mt quality, usually positively @xcite . languages described as similar usually share certain levels of mutual intelligibility. depending on the level of closeness, certain languages may share orthography, lexical, syntactic, and/or semantic structures which may facilitate translation between pairs. however, (a) scarcity of parallel data that can be used for training mt models remains a bottleneck. even high-resource pairs can suffer from (b) low-data quality. that is, available data is not always guaranteed to be actual bitext with target standing as a translation of source. in fact, some open resources such as opus @xcite can suffer from noise such as when the source and target sentences belong to the same language. in this work, we tackle both (a) scarcity and (b) low-data quality. for a, we use simple knowledge transfer from already trained mt models to the downstream pair. for b, we use a simple procedure of language identification to remove noisy bitext where both the source and target are detected to be the same language or where source or target is identified as a different language from what it is expected to be. the models we develop are for spanish to catalan (es-ca), catalan to spanish (ca-es), spanish to portuguese (pt-es), portuguese to spanish (pt-es), french to bambara (fr-bm), and bambara to french (bm-fr) language pairs foot_0 . whenever possible, we choose an available mt model trained with the same source and target languages as our pair of interest. in cases where no such a model exists, we pick a model with either the source or the target language as our intended pair (section 5). to show the utility of our transfer learning approach to the problem, we also train on one pair from scratch (which we treat as a baseline). we experiment with tokenized (primary models) and untokenized (contrastive models) settings and compared the settings with models developed by fine-tuning pre-trained models as well as models trained from scratch. our experiments show that the tokenized settings perform better than the untokenized settings for all language pairs. the model fine-tuned on top of the pre-trained mt model has higher performance than our baseline model from the first epoch compared with the model trained from scratch (for six epochs). our models for the ca-es and pt-es language pairs achieve top 1 rank in the offical shared task results, with 82.96 and 47.71 bleu scores respectively. in addition, we are the only team that submitted for the rest of the language pairs (i.e., es-pt, fr-bm, and br-fr). the rest of our paper is organized as follows: we discuss related work in section 2. we describe the data and pre-processing in section 3. next, we describe the data cleaning process in section 4. in section 5, we describe the models we developed for this task and we discuss the various experiments we perform. we also describe the architectures of the models we developed. then we discuss the evaluation criteria in section 6. evaluation is done on both the validation and test sets. in section 7 we perform error analysis on the output of our models for some language pairs to determine the types of errors the models make. we conclude with discussion of the insights we gained from the shared task in section 8. ## related work in recent times, there has been an increase of research interest in low-resource mt scenarios @xcite . nmt models, specifically those based on the transformer architecture, have been shown to perform well when translating between similar languages (przystupa and abdul-mageed, 2019; @xcite @xcite , low resource scenarios @xcite , and in contexts not involving english @xcite . furthermore, pre-training techniques have been successful for many nlp tasks @xcite including @xcite . self-supervised pretraining acquires general knowledge from a large amount of unlabeled monolingual or multilingual data to improve the training process of downstream tasks @xcite . the pre-trained model acquires some syntactic and semantic knowledge which can be transferred as initialized parameters to improve nmt models and translation quality @xcite @xcite . the intuitive justification for using pre-trained models is that the embedding space becomes more consistent, with semantically similar words closer together. the knowledge from pre-trained language models (lms) can be used to initialize the nmt model before training it on parallel data. however, there are certain limitations for mt tasks. first, lms cannot be easily fine-tuned for mt tasks. second, there is a discrepancy between pre-training objectives for lms and the training objective in mt. existing pre-training approaches such as mbart rely on autoencoding objectives to pre-train the models, which are different from mt. furthermore, lms learn to reconstruct all source tokens with some noises, while nmt learns to translate most source tokens and copy only a few of them. lm pre-training is said to copy about 65% of tokens, while nmt training needs to copy less than 10% @xcite . the unexpected knowledge/bias can be therefore propagated to the nmt model via pre-training, which may result in nmt models mistakenly copying source tokens to the target side @xcite . for instance, because copying behaviours can be learned, a source word such as "shoe" may be copied to the target by pretraining based nmt models instead of providing a translation. therefore, fine-tuning mt models on pre-trained lms still do not achieve adequate improvements. in order to address the difference in training objectives that using pre-trained language models results in, we use pre-trained mt models to initialize our models. this is still a type of transfer learning. following the justification for pre-trained models, we hypothesize that two linguistically similar languages will share closer semantic and syntactic relationships. this is based on the assumption that the more similar the source and target languages, the more similar the syntax and semantic properties and the higher the gains from using pre-trained models will be. we now introduce our data. ## data for our experiments, we use parallel data from opus @xcite . our data are from the following language pairs spanish and catalan, spanish and portuguese, and french and bambara. we use data in the two directions from each of these three pairs. details about our data is in ## data cleaning & analysis we perform data cleaning on the es-ca, ca-es, es-pt, and pt-es language pairs. we do not clean the french and bambara pairs because we had very few training sentences for these. for cleaning, we run the langid tool @xcite on the concatenation of the source and target and remove sentences that are not identified as belonging to one or both of the language pair. in ## evaluation we evaluated our models on both the dev and test sets. we used the checkpoint with the best bleu score as evaluated on dev as our best model. we used a beam size of four during evaluation on both dev and test and evaluated on de-tokenized data. evaluation on dev set. we report the results on the dev sets for each language pair in evaluation on test set. our test set performance was evaluated by the shared task organizers using bleu @xcite , ribes, and ter @xcite . we report the scores in ## effect of language similarity in order to gain some insight into the interference of similarity between languages of a given pair, we performed an analysis based on levenstein distance that allows us to identify the percentage of cognates shared between the languages. we then compared system output to the reference sentences, trying to quantify how much the system is able to translate cognates correctly (in this case the correct translation will have the same cognate word in the target as it is in the source). we performed this analysis for one language pair: ca-es and found that the model learned the cognates correctly up to 80% of the time. ## conclusion we describe our contribution to the wmt2021 similar languages translation shared task. we develop models for es-ca, ca-es, es-pt, pt-es, fr-bm, and bm-fr and show the improvement our models make with tokenized data when compared to untokenized data. we also show the utility of transfer learning based on fine-tuning nmt pretrained models. future work can investigate how the choice of pre-trained models affects the downstream tasks.