Vega- MT : The JD Explore Academy Machine Translation System for WMT 22 (2022)

Abstract: We describe the JD Explore Academy’s submission of the WMT 2022 shared general translation task. We participated in all high-resource tracks and one medium-resource track, including Chinese-English, German-English, Czech-English, Russian-English, and Japanese-English. We push the limit of our previous work – bidirectional training for translation by scaling up two main factors, i.e. language pairs and model sizes, namely the Vega-MT system. As for language pairs, we scale the “bidirectional” up to the “multidirectional” settings, covering all participating languages, to exploit the common knowledge across languages, and transfer them to the downstream bilingual tasks. As for model sizes, we scale the Transformer-Big up to the extremely large model that owns nearly 4.7 Billion parameters, to fully enhance the model capacity for our Vega-MT. Also, we adopt the data augmentation strategies, e.g. cycle translation for monolingual data, and bidirectional self-training for bilingual and monolingual data, to comprehensively exploit the bilingual and monolingual data. To adapt our Vega-MT to the general domain test set, generalization tuning is designed. Based on the official automatic scores of constrained systems, in terms of the sacreBLEU shown in Figure-1, we got the 1st place on Zh-En (33.5), En-Zh (49.7), De-En (33.7), En-De (37.8), Cs-En (54.9), En-Cs (41.4) and En-Ru (32.7), 2nd place on Ru-En (45.1) and Ja-En (25.6), and 3rd place on En-Ja(41.5), respectively; W.R.T the COMET, we got the 1st place on Zh-En (45.1), En-Zh (61.7), De-En (58.0), En-De (63.2), Cs-En (74.7), Ru-En (64.9), En-Ru (69.6) and En-Ja (65.1), 2nd place on En-Cs (95.3) and Ja-En (40.6), respectively. Models will be released to facilitate the MT community through GitHub and OmniForce Platform.

## introduction in this year's wmt general translation task, our vega-mt translation team participated in 10 shared tasks, including chinese↔english (zh↔en), german↔english (de↔en), czech↔english (cs↔en), russian↔english (ru↔en), and japanese↔english (ja↔en). we use the same model architectures, data strategies and corresponding techniques for all tasks. we aim to leverage the cross-lingual knowledge through pretraining (pt) to improve the highresource downstream bilingual tasks. although recent works @xcite @xcite attempt to leverage sequence-to-sequence pt for neural machine translation (nmt; @xcite @xcite ) by using a large amount of unlabeled (i.e. monolingual) data, @xcite show that it usually fails to achieve notable gains (sometimes, even worse) on resource-rich nmt on par with their random-initialization counterpart, which is consistent with our preliminary experiments. ding et al. (2021d) show that bidirectional pretrained model as initialization for downstream bilingual tasks could consistently achieve significantly better performance. it is natural to assume that scaling the "bidirectional" to the "multidirectional" setting with {1) multilingual pretraining and 2) large enough model capacity} could benefit the downstream resource-rich bilingual translations. @xcite and @xcite also provide empirical evidences to support our motivation of supervised multilingual pretraining. different from @xcite that explores the effectiveness of multilingual training, we show that further tuning on the bilingual downstream task provide more in-domain knowledge and thus could gain better translation quality. compared with @xcite , our model do not require any alignment information during pretraining, which will consume more extra time and computation resources, making our strategy flexible to be applied to any language. for model frameworks in §2.1, we tried autoregressive neural machine translation, including transformer-big and -xl @xcite , and non-autoregressive translation models @xcite , where the transformer-xl is employed as the foundation model and autoregressive big and non-autoregressive models are used during augmenting. for the core training strategy of our vega-mt, we cast the multilingual pretraining as foundation models in §2.2, including multi-directional pretraining ( §2.2.1) and specific-directional finetun-ing ( §2.2.2). for data augmentation strategies, we employ cycle translation ( §2.3.1) and bidirectional self-training ( §2.3.2) for both monolingual and parallel data. in or-m base m big m xl #stack 6 6 24 #hidden size 512 1024 2048 #ffn size 2048 4096 16384 #heads 8 16 32 the subsequent paper is designed as follows. we introduce the major approaches we used in section 2. in section 3, we provide the data description. we also present the experimental settings and results in section 4. conclusions are described in section 5. ## data preparation we participated in translation of all highresource tracks and one medium-resource track, including chinese↔english (zh↔en), german↔english (de↔en), czech↔english (cs↔en), russian↔english (ru↔en), and japanese↔english (ja↔en). in this section, we take the en↔zh translation as example and describe how to prepare the training data. the setting is the same for other language pairs. we use all available parallel corpus for en↔zh § , including paracrawl v9, news commentary v16, wiki titles v3, un parallel corpus v1.0, ccmt corpus, wikimatrix and backtranslated news. for monolingual data, we randomly sample from "news crawl" and "common crawl". the final corpus statistics are presented in to improve the quality of parallel data, we further propose to filter the low-quality samples. first, we remove the sentence pair which is predicted as wrong language with fasttext @xcite . second, we replace unicode punctuation and also normalize punctuation with mosesdecoder. we also remove duplicate sentence pairs and filter out sentences with illegal characters. for length, we remove sentences longer than 250 words and with a source/target length ratio exceeding 3. ## experiments settings we use the extremely large transformer (m xl ) for all tasks and transformer-big (m big ) for bilingual baselines. for m big , we empirically adopt large batch strategy @xcite (i.e. 458k tokens/batch) to optimize the performance. the learning rate warms up to 1 × 10 -7 for 10k steps, and then decays for 70k steps with the cosine schedule. for regularization, we tune the dropout rate from [0.1, 0.2, 0.3] based on validation performance, and apply weight decay with 0.01 and label smoothing with ϵ = 0.1. we use adam optimizer @xcite to train models. we evaluate the performance on an ensemble of last 10 checkpoints to avoid stochasticity. for the main model m xl , we adopt 1m tokens/batch to optimize the performance both in multilingual pretraining and bilingual finetuning. we set 0.1 as the label smoothing ratio, 4000 as warm-up steps, and 1e-3 as the learning rate. we optimize vega-mt with @xcite . we use 100k updates for multi-directional pretraining, 40k updates for each specific-directional finetuning. for all parallel data will be used in the multidirectional pt stage, and during specificdirectional ft, corresponding bilingual data augmented by bidirectional self-training are utilized. each sentence are jointly tokenized in to sub-word units with sentencepiece @xcite , which is trained on all concatenated multilingual parallel data for transformer-xl with merge operation 120k at the pretraining stage, and during finetuning stage, is trained on corresponding bilingual data with merge operation 60k for english and 75k for other languages. and for each baseline with transformer-big, the joint bilingual vocab size is 80k. during pretraining, we select the sample with temperature-based method (@xmath0, e.g. japanese. we grid-search the beam size within the range of [3,4,5,..,8] on validation set for each translation task. all models are trained on 32 dgx-superpod a100 gpus for about two weeks pre-training and five days fine-tuning. ## conclusion this paper presents the jd explore academy machine translation system vega-mt for wmt 2022 shared tasks on general machine translation. we investigate various frameworks, including autoregressive and non-autoregressive transformer with base, big and xl settings, respectively, to build strong baseline models. then we push the limit of bidirectional training by scaling up two main factors, i.e. language pairs and model scales, to develop the powerful foundation vega-mt model. also, the popular data augmentation methods, e.g. cycle translation and bidirectional self-training, are combined to improve their performance. we carefully design the generalization adaptation strategies to further improve the multidomain performance. among all participated constrained systems, our vega-mt won 7 champions, 2 runners-up, and 1 third place w.r.t sacrebleu. and according to the comet, we won 8 champions and 2 runners-up.