Tencent AI Lab - Shanghai Jiao Tong University Low-Resource Translation System for the WMT 22 Translation Task (2022)

Abstract: This paper describes Tencent AI Lab - Shanghai Jiao Tong University (TAL-SJTU) Low-Resource Translation systems for the WMT22 shared task. We participate in the general translation task on English-Livonian.Our system is based on M2M100 with novel techniques that adapt it to the target language pair.(1) Cross-model word embedding alignment: inspired by cross-lingual word embedding alignment, we successfully transfer a pre-trained word embedding to M2M100, enabling it to support Livonian.(2) Gradual adaptation strategy: we exploit Estonian and Latvian as auxiliary languages for many-to-many translation training and then adapt to English-Livonian.(3) Data augmentation: to enlarge the parallel data for English-Livonian, we construct pseudo-parallel data with Estonian and Latvian as pivot languages.(4) Fine-tuning: to make the most of all available data, we fine-tune the model with the validation set and online back-translation, further boosting the performance. In model evaluation: (1) We find that previous work underestimated the translation performance of Livonian due to inconsistent Unicode normalization, which may cause a discrepancy of up to 14.9 BLEU score.(2) In addition to the standard validation set, we also employ round-trip BLEU to evaluate the models, which we find more appropriate for this task. Finally, our unconstrained system achieves BLEU scores of 17.0 and 30.4 for English to/from Livonian.

## introduction this paper introduces our submissions to the wmt22 general machine translation task. last year, tencent ai lab participated in two translation tasks: news @xcite and biomedical * work was done when zhiwei he was interning at tencent ai lab. we use m2m100 1.2b foot_0 @xcite as the pre-trained model which is a massive multilingual translation model that supports any pair of 100 languages foot_1 and shows promising performance for low-resource translation. to adapt it to en-liv, the first thing to do is enabling it to support liv. a common approach is to expand the vocabulary and the word embedding matrix to contain the extra tokens. however, the incoming embeddings must be randomly initialized @xcite , which leads to inconsistency with the original embeddings and increases training difficulty. fortunately, @xcite has released a translation model for en-liv called liv4ever-mt 4 . inspired by supervised crosslingual word embedding alignment @xcite , we propose cross-model word embedding alignment (cmea) that learns a linear transformation between the embedding matrices of two models. therefore, the incoming embeddings can be extracted from liv4ever-mt and transformed to m2m100's word embedding space rather than random initialization. in terms of model training, we adopt a gradual adaptation strategy. the overall training process is shown in figure 1 . following @xcite , we also use estonian (et) and latvian (lv) as auxiliary languages. liv has been influenced by et and lv for centuries. there are about 800 et loanwords and 2,000 lv loanwords in liv @xcite . therefore, we first add et and lv for manyto-many translation training, resulting in a 4-lingual translation model. we then augment the en-liv data with forward and backward translations using et and lv as the pivot languages. finally, we combine all the authentic and synthetic data to retrain the model, followed by a few steps of fine-tuning with the validation set and online back-translation. in terms of model evaluation, we find that the data set provided by @xcite suffers from inconsistent unicode normalization. this inconsistency is reflected in using two or more encodings for the same character, which leads to inconsistent encoding between model hypothesis and reference foot_3 and thus inaccurate evaluation. in our experiments, normalizing the character encoding can bring an average improvement of +2.5 bleu on the liv4ever foot_4 test set (see appendix a) and up to +14.9 bleu on a subset from a specific source. in addition to the standard validation set, we also employ round-trip bleu to evaluate our models, which is an effective unsupervised criterion @xcite and reduces the demand for the parallel corpus. @xcite have found that in the scope of neural machine translation, roundtrip translation quality correlates consistently with forward translation quality. we consider round-trip bleu a better evaluation method for this task. the reasons for this are threefold: more data, more general domain, and the same original language as the wmt22 en-liv test set. this paper is structured as follows: section 2 describes the data statistics and processing methods. then we present our evaluation methods in section 3. our translation system and ablation study are detailed in section 4, followed by the final results. finally, we conclude the paper in section 5. ## model evaluation this section describes our methods for model evaluation. specifically, we explain the unicode inconsistency problem in the liv4ever data set and the resulting underestimation of model performance. in addition, we introduce round-trip bleu as the more appropriate way for this competition. @xcite collected the liv4ever data set and built liv4ever-mt, the first machine translation model for livonian. we find that the liv4ever data set does not use consistent unicode normalization, resulting in different encodings for the same character. this did not lead to any training problem in @xcite because sentencepiece does nfkc 12 normalization by default. however, when computing sacrebleu 13 , the encoding of model output and the reference will be inconsistent, resulting in inaccurate evaluation. ## system and ablation study in this section, we describe our system in this competition and provide a comprehensive ablation study of the key components. ## conclusion this paper presents the tencent ai lab -shanghai jiao tong university (tal-sjtu) low-resource translation systems for the wmt22 shared task. we start from the m2m100 1.2b model and investigate techniques to adapt it to english⇔livonian. we propose cross-model word embedding alignment that transfer the embeddings of liv4ever-mt to m2m100, enabling it to support livonian. then, estonian and latvian are involved in model training and synthetic data generation as auxiliary and pivot languages. we further fine-tune the model with validation set and online back-translation followed by rule-based post-processing. in model evaluation, we correct the inaccurate evaluation of livonian due to inconsistent unicode normalization and use round-trip bleu as an alternative to the standard validation set.