NICT -2 Translation System at WAT -2021: Applying a Pretrained Multilingual Encoder-Decoder Model to Low-resource Language Pairs (2021)

Abstract: In this paper, we present the NICT system (NICT-2) submitted to the NICT-SAP shared task at the 8th Workshop on Asian Translation (WAT-2021). A feature of our system is that we used a pretrained multilingual BART (Bidirectional and Auto-Regressive Transformer; mBART) model. Because publicly available models do not support some languages in the NICT-SAP task, we added these languages to the mBART model and then trained it using monolingual corpora extracted from Wikipedia. We fine-tuned the expanded mBART model using the parallel corpora specified by the NICT-SAP task. The BLEU scores greatly improved in comparison with those of systems without the pretrained model, including the additional languages.

## introduction in this paper, we present the nict system (nict-2) that we submitted to the nict-sap shared task at the 8th workshop on asian translation (wat-2021) @xcite . because the nict-sap task expects to perform translations with little parallel data, we developed a system to improve translation quality by applying the following models and techniques. pretrained model: an encoder-decoder model pretrained using huge monolingual corpora was used. we used a multilingual bidirectional auto-regressive transformer (mbart) (i.e., multilingual sequence-to-sequence denoising autoencoder @xcite ) model, which supports 25 languages. because it includes english and hindi, but does not include indonesian, malay, and thai, we expanded it to include the unsupported languages and additionally pretrained it on these five languages. 1 multilingual models: we tested multilingual models trained using multiple parallel corpora to increase resources for training. ## nict-sap shared task the nict-sap shared task was to translate text between english and four languages, that is, hindi (hi), indonesian (id), malay (ms), and thai (th), for which the amount of data in parallel corpora is relatively low. the task contained two domains. the data in the asian language translation (alt) domain @xcite consisted of translations obtained from wikinews. the alt is not supported by either the mbart model or mbart-50 models. therefore, we applied additional pretraining to the mbart model. the data in the it domain consisted of translations of software documents. the wat organizers provided the development and test sets @xcite . for the training set, we obtained gnome, kde, and ubuntu sub-corpora from the opus corpus @xcite . therefore, the domains for the training and dev/test sets were not identical. the data sizes are shown in ## mbart model in this section, we briefly review the pretrained mbart model @xcite . the mbart model is a multilingual model of bidirectional and auto-regressive transformers (bart; @xcite ). the model is based on the encoder-decoder transformer @xcite , in which the decoder uses an autoregressive method (figure 1 ). two tasks of bart are trained in the mbart model. one is the token masking task, which restores masked tokens in input sentences. the other is the sentence permutation task, which predicts the original order of permuted sentences. both tasks learn using monolingual corpora. to build multilingual models based on bart, mbart supplies language tags (as special tokens) at the tail of the encoder input and head of the decoder input. using these language tags, a mbart language #sentences #tokens english (en) 7,000,000 (*1) 174m hindi (hi) 1,968,984 51m indonesian (id) 6,997,907 151m malay (ms) 2,723,230 57m thai (th) 2,233,566 (*2) 60m the published pretrained mbart model foot_2 consists of a 12-layer encoder and decoder with a model dimension of 1,024 on 16 heads. this model was trained on 25 languages in the common crawl corpus @xcite . of the languages for the nict-sap task, english and hindi are supported by the published mbart model, but indonesian, malay, and thai are not supported. the tokenizer for the mbart model uses bytepair encoding @xcite of the sen-tencepiece model @xcite foot_3 . the vocabulary size is 250k subwords. ## experiments the models and methods described above were fine-tuned and tested using the hyperparameters in tables 5 and 6 show the official bleu scores @xcite for the test set in the alt and it domains, respectively. similar results were obtained on the development sets, but they were omitted in this paper. we submitted the results using the pretrained mbart model, which were good on the development sets, on average. when we focus on indonesian, malay, and thai, which were not supported in the original mbart model, the bleu scores of the submitted results were increased over 8 points from the baseline results in the alt domain. we conclude that language expansion and additional pretraining were effective for translating new languages. for verification, we checked sentences in the test sets and the corpus for the pretrained model (c.f., the multilingual models were effective under low-resource conditions because the size of parallel data increased during training. however, they were ineffective if the models had learned sufficiently in advance, like pretrained models. ## conclusions in this paper, we presented the nict-2 system submitted to the nict-sap task at wat-2021. a feature of our system is that it uses the mbart pretrained model. because the published pretrained model does not support indonesian, malay, and thai, we expanded it to support the above languages using additional training on the wikipedia corpus. consequently, the expanded mbart model improved the bleu scores, regardless of whether multilingual models or domain adaptation methods were applied.