To Optimize, or Not to Optimize, That Is the Question: T el U - KU Models for WMT 21 Large-Scale Multilingual Machine Translation (2021)

Abstract: We describe TelU-KU models of large-scale multilingual machine translation for five Southeast Asian languages: Javanese, Indonesian, Malay, Tagalog, Tamil, and English. We explore a variation of hyperparameters of flores101_mm100_175M model using random search with 10% of datasets to improve BLEU scores of all thirty language pairs. We submitted two models, TelU-KU-175M and TelU-KU- 175M_HPO, with average BLEU scores of 12.46 and 13.19, respectively. Our models show improvement in most language pairs after optimizing the hyperparameters. We also identified three language pairs that obtained a BLEU score of more than 15 while using less than 70 sentences of the training dataset: Indonesian-Tagalog, Tagalog-Indonesian, and Malay-Tagalog.

## introduction this paper describes our participation in the wmt21 shared task of large-scale multilingual machine translation. specifically, we chose small track #2, which involves thirty language pairs, and used the neural machine translation (nmt) method. we call our models telu-ku (telkom university -kumamoto university) as we use our university name in our submissions. nmt has been widely used in machine translation research for many languages. currently nmt has become the state of the art of machine translation with a large number of parallel corpus @xcite @xcite . meanwhile, for low resources cases, the nmt tends to give poor translation results @xcite @xcite . in order to get better translation results of nmt for low resources languages, some approaches are applied, such as using a large number of monolingual corpora @xcite , applying transfer learning approach to share lexical and sentence level representation @xcite , using sub-word representation @xcite , and hyperparameter optimization (hpo) @xcite . hpo is an important part of building an nmt system in many real-world applications. in other words, selecting effective hyperparameters is critical to building a strong nmt system. however, in many cases, hyperparameters are often set manually based on intuition and heuristics mechanisms, tedious and error-prone processes that can lead to unreliable experimental results and poor performance of shared tasks or production systems. this is because hpo requires rigorous testing and resources, which makes it a high-cost process. to deal with this problem, tablelookup has been proposed as a benchmark procedure @xcite . their study provides evaluation protocols and a benchmark dataset for comparing the hpo methods. moreover, like other nmt models, transformers require setting various hyperparameters, but researchers often use default parameters, even when their data conditions differ substantially from the data conditions previously used to determine those default values. in low-resource languages cases, the performance of the transformer is highly dependent on the hyperparameter settings @xcite . the experimental results show that the best-suited combination of hyperparameters and regularization methods can produce substantial improvement for low-resource languages data. on the other side, grid search and manual search are the most frequently used strategies for hpo. however, according to the experiment, random search is actually better than grid search in several conditions @xcite . random searches are actually better suited to running on a cluster of computers than grid searches when a group of computers fails. random search also allows the experimenter to change the "resolution" on the fly. in addition, they have advantages in high-dimensional searching spaces. in this work, we experimented with two models, that is, telu-ku-175m and telu-ku-175m_hpo. both models are based on a pre-trained model of flores101_mm100_175m. the telu-ku-175m is our model that manually fine-tuning the hyperparameters, whereas the telu-ku-175m_hpo is based on hyperparameter optimization. we used a random search method while using 10% of datasets to find the best hyperparameter optimization. in addition, we also included the m2m-100 175m model to compare with our results. this model uses the same pre-trained model as ours but without fine-tuning. fine-tuning is a common practice in nlp to train a pretrained model for several epochs on a downstream dataset and has proven to improve performance. our experimental results show improvement in most language pairs after optimizing the hyperparameters. the telu-ku-175m is able to improves the average bleu scores by 0.35-0.59 over m2m-100 175m. meanwhile, the telu-ku-175m_hpo improve the scores by 1.08-1.41 over the baseline. we also identified three language pairs that obtained a bleu score of more than 15 while using less than 70 sentences of the training dataset: id-tl, tl-id, and ms-tl. this paper is organized as follows. section 2 explains the experiment. section 3 shows the obtained results. section 4 discusses the effect of hpo and the multilingual model. section 5 provides the conclusion and future direction of this work. ## experiments in this section, we first describe languages overview of the southeast asian language. then, we discuss data and preprocessing. finally, we discuss the model and architecture of our model submission. ## results we evaluate the generated texts of our models using the sentence-piece bleu (spbleu). the spbleu uses a sentencepiece tokenizer with 256k tokens, and then the bleu score is computed on the sentencepiece tokenized text. ## discussion this section discusses the effect of hpo and nmt of the multilingual model against our models' evaluation results. ## conclusion we described our team submission for wmt21. our results show improvement in most language pairs after optimizing the hyperparameters. furthermore, we also found three language pairs that obtained a bleu score of more than 15 while using less than 70 sentences of the training dataset. in this study, we used 30 language pairs that are considered as lowresource language and mostly have the same language family. this result indicated that low-resource language with the same language family should be considered in the nmt of the multilingual model. as future work, we plan to use a more sophisticated optimization algorithm, specifically informed searches such as bayesian optimization or evolutionary algorithm. additionally, we want to try other percentages of the optimized dataset to see the effect of the number of training data on the performance. we also plan to use a specific tokenizer for tamil, e.g., indic nlp library @xcite , inltk @xcite . the tamil language needs a particular pre-processing due to its writing system that differs from other languages. last, we plan to clean the dataset in pre-processing steps, considering that the dataset used in this work is noisy. we expect this will maximize the attention mechanism in the nmt of a multilingual model. therefore, our model could produce better translation results.