NECTEC ’s Participation in WAT -2021 (2021)

Abstract: In this paper, we report the experimental results of Machine Translation models conducted by a NECTEC team for the translation tasks of WAT-2021. Basically, our models are based on neural methods for both directions of English-Myanmar and Myanmar-English language pairs. Most of the existing Neural Machine Translation (NMT) models mainly focus on the conversion of sequential data and do not directly use syntactic information. However, we conduct multi-source neural machine translation (NMT) models using the multilingual corpora such as string data corpus, tree data corpus, or POS-tagged data corpus. The multi-source translation is an approach to exploit multiple inputs (e.g. in two different formats) to increase translation accuracy. The RNN-based encoder-decoder model with attention mechanism and transformer architectures have been carried out for our experiment. The experimental results showed that the proposed models of RNN-based architecture outperform the baseline model for English-to-Myanmar translation task, and the multi-source and shared-multi-source transformer models yield better translation results than the baseline.

## introduction machine translation (mt) is a quick and very effective way to communicate one language to another. mt consists of the automatic translation of human languages by using computers. the first machine translation systems were rule-based built only using linguistic information. the translation rules were manually created by experts. although the rules are well defined, this process is very expensive and cannot translate well for all domains and languages. currently, many researchers had successfully built the most popular machine translations such as smt (statistical machine translation) and nmt (neural machine translation) for various languages instead of rule-based translation. nmt has become the state-of-the-art approach compared to the previously dominant phrase-based statistical machine translation (smt) approaches. however, the existing nmt models do not directly use syntactic information. therefore, we propose tree-tostring and pos-to-string nmt systems by the multi-source translation models. we conducted these multi-source translation models with myanmar-english and english-myanmar in both directions. the multi-source translation models conducted in our experiments are based on the multi-source and shared-multisource approaches of the previous research work @xcite . figure 1 and figure 2 show the architecture of multi-source translation models. for doing the training processes of proposed models by the transformer and s2s architectures, word-level segmentation and tree-format on the english corpus side and syllable-level segmentation on the myanmar corpus side are applied in english-to-myanmar translation. in addition, we used the syllable-level segmentation and pos-tagged word on the myanmar corpus side, and word-level segmentation on the english side for conducting the myanmarto-english translation. in this paper, section 2 will describe our mt systems. the experimental setup will be proposed in section 3. in section 4, the results of our experiments will be reported, and section 5 will present the error analysis on translated outputs. finally, section 6 will conclude the report. figure 2: the basic idea of multi-source translation model for myanmar-to-english translation 2 system description in this section, we describe the methodology used in our experiments for this paper. to build nmt systems, we chose the marian framework foot_0 (junczys-dowmunt et al., 2018) with the architectures of transformer and rnn based encoder-decoder model with attention mechanism (s2s). marian is a selfcontained neural machine translation toolkit focus on efficiency and research. this framework, the reimplementation of nematus (sennrich et al., 2017), is an efficient neural machine translation framework written in pure c++ with minimal dependencies. the main features of marian are pure c++ implementation, one engine for gpu/cpu training and decoding, fast multi-gpu training and batched translation on gpu/cpu, minimal dependencies on external software (cuda or mkl, and boost), the static compilation (i.e., compile once, copy the binary and use anywhere), and permissive opensource mit license. there are several model types supported by the marian framework. among them, we used transformer, multitransformer, shared-multi-transformer, s2s (rnn-based encoder-decoder model with attention mechanism), multi-s2s, and shared-multi-s2s models for our experiment. in our experiments, two baseline models (transformer and rnn based attention: s2s) are used for the translation tasks of englishto-myanmar and myanmar-to-english. for the first translation task, the baseline models take single input of english tree data {tree-en} and produce the output of myanmar string {my}. the multi-transformer, shared-multitransformer, multi-s2s, and shared-multi-s2s models use two inputs of english string data and tree data {en, tree-en} and produce the output of myanmar string {my}. for the second translation task, the input of myanmar pos data {pos-my} is taken by the baseline models and produces the output of english string {en}. the multi-source and shared multi-source models take two inputs of myanmar sting data and myanmar pos data {my, pos-my} and yield the output of english string {en}. the baseline models, the multi-source and shared-multi-source models do the same action as the first translation task with different inputs and outputs. ## evaluation results our systems are evaluated on the alt test set and the evaluation results are shown in the bleu score measures the precision of ngram (overall @xmath0. intuitively, the bleu score measures the adequacy of the translation and a larger bleu score indicates a better translation quality. ribes is an automatic evaluation metric based on rank correlation coefficients modified with precision and special care is paid to the word order of the translation results. the ribes score is suitable for distance language pairs such as myanmar and english. larger ribes scores indicate better translation quality. am-fm is a two-dimensional automatic evaluation metric for machine translation, which is used to evaluate the machine translation systems. the evaluation metric designed to address independently the semantic and syntactic aspects of the translation. the larger the amfm scores, the better the trans for the first architecture (i.e., transformer) in the first part of the table, the shared-multi-transformer model achieves higher bleu scores (+1.18) than the baseline transformer model. furthermore, the multi-transformer model performs better than the baseline transformer in terms of amfm scores. however, ribes scores of multi-transformer and shared-multitransformer models are lower than the baseline transformer model. for the second architecture (i.e., s2s or rnn-based attention), the multi-s2s model outperforms the baseline s2s model and shared-multi-s2s in terms of bleu and amfm scores. the sharedmulti-s2s model provides better ribes scores (0.626460). the highest bleu scores (13.90) of the shared-multi-transformer model and the highest amfm scores (0.654780) of the multitransformer model are produced by the first architecture while the highest ribes scores (0.625476) are achieved by the multi-s2s model of the second architecture. myanmar-to-english translation results are shown in the second part of the ## error analysis for both english-to-myanmar and myanmarto-english translation models, we analyzed the translated outputs by using word error rate foot_9 . for doing the error analysis, we used sclite (score speech recognition system output) program from the nist scoring toolkit sctk foot_10 version 2.4.10 for making dynamic programming based alignments between reference (ref) and hypothesis (hyp) and calculation of wer. the wer formula can be described as the following equation: for the first architecture of english-to-myanmar translation, the baseline transformer model gives lower wer scores (81.3%) than the multi-transformer and shared-multi-transformer models. however, in the second architecture, the sharedmulti-s2s model provides lower wer scores (82.5%) compared with the baseline (s2s) and multi-s2s models. in myanmar-to-english translation, the multi-transformer and sharedmulti-transformer models yield greater wer scores (90.0% and 88.2%) than the baseline transformer model of the first architecture. in addition, the multi-s2s and shared-multi-s2s model also give higher wer scores (91.8% and 86.0%) than the baseline s2s models (85.1%). due to the higher wer scores in myanmar-to-english translation models, the multi-transformer and sharedmulti-transformer models couldn't provide better translation results than the baseline transformer model, and the multi-s2s and shared-multi-s2s models couldn't also yield the improvements than the baseline s2s model. after we analyzed the confusion pairs of english-to-myanmar and myanmar-to-english translation models in detail, we found that most of the confusion pairs in the translations are caused by (1) the nature of the myanmar language (written or speaking form), (2) the incorrect word segmentation or data cleaning errors of english language, (3) the myanmar language with no articles (i.e., a, an, and the), and (4) the different nature and language gaps of myanmar and english languages. the top 10 confusion pairs of englishto-myanmar and myanmar-to-english translations of the model transformer are shown in all of the confusion pairs in the first column are caused by the nature of the myanmar language. for example, in myanmar written or speaking form, the word "သည ("is" in english)" are the same as the word "တယ ("is" in english)". moreover, the words "၏ ("of or 's" in english)" and "ရ ("of or 's" in english)" in the possessive place and the words "မ ("plural form" in english)" and " တ ("plural form" in english)" are the same meanings. in other words, these hypotheses are synonyms of the reference words. in the third column of the for instance, for the myanmar sentence "သ က အတန ထ မ အ တ ဆ က င သ ဖစ တယ ။", they translate this sentence to the english sentence "he is the most clever student of the class.". in this case, they misused the word "of" instead of the word "in". the correct english sentence is "he is the most clever student in the class." for another example of myanmar sentence "စ ပ က သစ သ ဖင ပ လ ပ ထ တယ ။", they translate to english sentence "the table is made with wooden." with the misused of the word "with" instead of "of". the correct sentence for this example is "the table is made of wooden." when the prepositions "in, of, and with" are combined with the main verbs, the prepositions "in and of" and "with and of" have generally same meanings in myanmar language. these may cause the myanmar-to-english translation models hard to learn well during the training processes compared with the english-to-myanmar translation models. ## conclusion in this system description for wat-2021, we submitted our nmt systems with two architectures such as transformer and rnn with attention. we evaluated our proposed models in both directions of myanmar-english and english-myanmar translations at wat-2021. in this paper, for english to myanmar translation, multi-source and shared-multi-source models outperform the baseline models in terms of bleu, ribes, and amfm scores. in the myanmar-to-english translation task, the proposed models could not provide better translation quality than the baselines. the top 10 frequent errors in the model's hypothesis could be clearly found from our error analysis. for examples, the confusion pairs of "သည → တယ 371", "မ → တ 63", "၏ → ရ 33", and so on. moreover, our study also made a contribution to the fact that if these errors can be cleaned up, the translation performance of the shared task will improve. in the future, we intend to apply post-editing techniques in myanmar to english translation to improve the translation quality. furthermore, we intend to extend string-to-tree and string-to-pos translation approaches for under-resourced languages such as myanmar and thai.