KYB General Machine Translation Systems for WMT 22 (2022)

Abstract: We here describe our neural machine translation system for general machine translation shared task in WMT 2022. Our systems are based on the Transformer (Vaswani et al., 2017) with base settings. We explore the high-efficiency model training strategies, aimed to train a model with high-accuracy by using small model and a reasonable amount of data. We performed fine-tuning and ensembling with N-best ranking in English to/from Japanese directions. We found that fine-tuning by filtered JParaCrawl data set leads to better translations for both of direction in English to/from Japanese models. In English to Japanese direction model, ensembling and N-best ranking of 10 different checkpoints improved translations. By comparing with other online translation service, we found that our model achieved a great translation quality.

## introduction we participated in the japanese to/from english translation for the general machine translation shared task of wmt 2022. japanese →english is one of the challenging language pairs for machine translation since their differences are large in both vocabulary and grammatical structure. recent advances in neural machine translation models have greatly promoted the development of the community. the transformer is the current key model and most recent participants are using a bigsetting transformer model to improve the quality of translations. however, developing a more efficient model is also important. we here use a smaller model and limited computation resources to pursue high-quality translation models. our systems are based on the transformer model with base settings, and the models are trained on the parallel corpus of japanese and english @xcite . we compared the quality of translations by using fine-tuning with several datasets. also, we tested several different hyperparameters of the training to find suitable values for the task. after the fine-tuning, we tried to perform ensembling of multiple results from the model to earn a better-quality translation in the english to/from japanese model. here we describe the details of our systems. ## data selection and preprocessing we select a suitable parallel corpus for model finetuning. we compare wmt provided dataset (which contained 7 different sources including the jparacrawl dataset), kftt (kyoto free translation task data set, @xcite , the jparacrawl dataset (ver 2) and so on. we performed fine-tuning for these datasets and found that the model trained on the jparacrawl dataset achieved better performance. we used a test data set made from wmt provided data and compare model performances by bleu score. the score of the no fine-tuned model was 37.21, kftt finetuned model was 14.87 and jparacrawl fine-tuned model was 44.09. therefore, we decided to use jparacrawl as our fine-tuning dataset finally. we also consider that jparacrawl has a reasonable amount of data for our high-efficiency training strategies. before we use the dataset, we check the corpus data to clean up. the jparacrawl dataset contains over 10 million sentence pairs which were constructed by broadly crawling the web and automatically aligning. therefore, there were noise and low-quality translations. we filtered low quality translation pairs and made a better translation dataset for fine-tuning. we also find that there were some contaminations of non-japanese languages (e.g., korean, chinese) in the japanese data. we also remove these pairs from the dataset. ## tokenization we perform the tokenization procedure using the sentencepiece toolkit 1 which provides us with a segmented sentence as tokens. in japanese and some other languages like chinese, words were not separated by spaces, therefore, tokenization needs to detect divided positions to separate each token. for japanese, tokenization can be performed by a lattice-based tokenizer like mecab 2 . a latticebased tokenizer performs tokenization based on a dictionary and if the contents of the dictionary cover whole words in data, it provides highly accurate tokenization. however, in the development of machine translation using neural network mechanisms, more efficient tokenization methods like byte-pair-encoding (bpe) were proposed @xcite . sentencepiece was developed based on these methods and provides more efficient tokenization for the nmt @xcite . sentencepiece is especially effective for languages not using spaces to separate words, has agglutinating morphology, and contains many compound words. using sentencepiece helps extract subwords within compound words and create a more robust tokenizer. sentencepiece was used again to detokenize by removing the meta symbols from the output translation. for preprocessing the data, we have used the sentencepiece model, in which the vocabulary size is set to 32,000, and sentences whose length exceeded 250 subwords are removed from the training data. ## model training we train our nmt models with the fairseq foot_0 toolkit. the models are based on transformer @xcite with base settings. we use an encoder/decoder with six layers. we set their embedding size to 512, and their feed-forward embedding size to 2048. we use eight attention heads for both the encoder and the decoder. we used dropout with a probability of 0.3. as an optimizer, we used adam with α = 0.001, β1 = 0.9, 1 https://github.com/google/sentencepi ece 2 https://taku910.github.io/mecab/ and β2 = 0.98. we used a root-square decay learning rate schedule with a linear warmup of 4000 steps. we clipped gradients to avoid exceeding their norm 1.0 to stabilize the training. for the base settings, each mini-batch contained about 5,000 tokens (subwords), and we accumulated the gradients of 64 mini-batches for updates. we trained the model with 24,000 iterations, saved the model parameters every 200 iterations, and averaged the last eight models. to achieve maximum performance with the latest gpus, we use mixed-precision training. when decoding, we used a beam search with a size of six as the default condition and length normalization by dividing the scores by their lengths. we test other parameters of a beam search in the model of @xmath0, 3, 4, and 10) and found that @xmath1. we also compared models output by scarableu @xcite . our models are trained on the google cloud platform's compute engine with 2-t4 gpus. model training generally took approximately 3.5 hours. we train our models in mixed precision to save costs without compromising on the accuracy. ## model ensembling and n-best reranking for @xmath2 after we fine-tuned our base model, we performed model ensembling with n-best reranking @xcite . for n-best reranking, we have created a script by referring to a script by xu song foot_1 , bert-as- ## https://github.com/yoheikikuta/bertjapanese we also check the translation result of the test set released by wmt2022. the dataset consists of 2037 english sentences and there were no reference sentences of japanese. therefore, we cannot calculate bleu score here. alternatively, we calculate perplexity foot_3 (ppl), by using bertjapanese model 5 , which is explained in the model ensembling section. ppl is a metric of a language model and lower values mean better. we also check the translation quality by the human evaluation of a japanese native speaker. the average of the ppl of our model was lower than deepl ( the results above ( ## conclusions we explored the high-efficiency model training strategies with a small model and a reasonable amount of data. our systems are based on the transformer with a base setting. in our experiments, we found that data cleaning, model averaging, model ensembling, beam search, finetuning, parameter-tuning, and post-processing are useful techniques to train a high-quality model. finally, we compared the translation results between our model and the online translation service, we found that our model achieved better translation quality. our experiments suggested that exploring more efficient training strategies with a smaller model, a reasonable amount of data, and limited computational resources is promising to achieve a high-quality translation model. the quality of translations is better for our model based on the confirmation of a native speaker.