Neural Machine Translation for translating into C roatian and S erbian (2020)

Abstract: In this work, we systematically investigate different set-ups for training of neural machine translation (NMT) systems for translation into Croatian and Serbian, two closely related South Slavic languages. We explore English and German as source languages, different sizes and types of training corpora, as well as bilingual and multilingual systems. We also explore translation of English IMDb user movie reviews, a domain/genre where only monolingual data are available. First, our results confirm that multilingual systems with joint target languages perform better. Furthermore, translation performance from English is much better than from German, partly because German is morphologically more complex and partly because the corpus consists mostly of parallel human translations instead of original text and its human translation. The translation from German should be further investigated systematically. For translating user reviews, creating synthetic in-domain parallel data through back- and forward-translation and adding them to a small out-of-domain parallel corpus can yield performance comparable with a system trained on a full out-of-domain corpus. However, it is still not clear what is the optimal size of synthetic in-domain data, especially for forward-translated data where the target language is machine translated. More detailed research including manual evaluation and analysis is needed in this direction.

## introduction whereas south slavic languages are generally less supported and investigated in natural language processing, they have been explored in the field of machine translation (mt). nevertheless, a large part of the work deals with the previous state-of-the-art approach, namely phrase-based statistical machine translation (pbsmt) @xcite @xcite @xcite @xcite , while much less work can be found about the new state-of-the-art, neural machine translation (nmt) @xcite . in this work, we focus on nmt into croatian and serbian, two very closely related south slavic languages. we explore two source languages, english and german. the main goals of our research are to explore two source languages, each of them with different sizes and types of training corpora, as well as to test our systems on translating english user reviews, a challenging domain/genre where no parallel training data are available. for these purposes, we train bilingual and multilingual nmt systems on different publicly available parallel training corpora. for translating english user reviews, we also explore different types of synthetic parallel in-domain data @xcite @xcite , which is a widely used practice in nmt. we explored two types of synthetic data: back-translated (bt) and forward-translated (ft). bt data consist of in-domain target language data and their machine translations into english, whereas ft data consist of english data and their machine translations into serbian and croatian. all our experiments were carried out on publicly available data sets. for all our systems, we present three distinct automatic mt evaluation scores, training corpus size, percentage of the particular target language in the training corpus, as well as training time. in order to get a broader picture about the current state of nmt into these two languages, we also present automatic scores for mt outputs foot_0 of two on-line systems: google translate 2 and amazon translate. 3 ## nmt systems all our systems are based on the transformer architecture @xcite and built using the sockeye implementation @xcite . the systems operate on sub-word units generated by byte-pair encoding (bpe) @xcite . we set the number of bpe merge operations at 32000 both for the source and for the target language texts. we do not use shared vocabularies between the source (english, german) and the target (serbian, croatian) languages because they are distinct. for multilingual systems, on the other hand, we build a joint vocabulary for the two target languages (serbian and croatian) because they are very similar. these systems are built using the same technique as @xcite and @xcite , namely adding a target language label "sr" or "hr" to each source sentence. all the systems have transformer architecture with 6 layers for both the encoder and decoder, model size of 512, feed forward size of 2048, and 8 attention heads. for training, we use adam optimiser (kingma and ba, 2015), initial learning rate of 0.0002, and batch size of 4096 (sub)words. validation perplexity is calculated after every 4000 batches (at so-called "checkpoints"), and if this perplexity does not improve after 20 checkpoints, the training stops. for set-ups with less than two million segments, 8 following the recommendations for low-resource settings in @xcite , we changed the following parameters: training stops after 10 checkpoints instead of 20, initial learning rate is 0.0001 instead of 0.0002, and checkpoint interval is 100 batches instead of 4000. ## results we evaluate our systems using the following three automatic overall evaluation scores: sacrebleu @xcite , chrf @xcite and character @xcite . the bleu score is based on word ngram (n in range from 1 to 4) precision and brevity penalty which should replace recall. the chrf score is based on character n-gram matching (n in range from 1 to 6) instead of word n-gram matching. it is f-score which weights recall two times more than precision. the character score is based on edit distance which takes into account not only substitutions, insertions and deletions, but also word sequence reorderings and character sequences in unmatched words. we use the bleu score because of the long tradition of using it for mt evaluation despite well-known faults, and the two character level scores because they are shown to correlate much better with human assessments @xcite . recently, the chrf score is recommended as a replacement for bleu @xcite . in addition to the automatic mt evaluation scores, for each of the systems we report the size of the training corpus, the percentage of particular target language data in this corpus, as well as the training time in terms of days. ## summary and outlook this work presents a systematic investigation of different set-ups for training nmt systems for translation into serbian and croatian, two closely related south slavic languages. we explore english and german as source languages, different sizes and types of training corpora, as well as bilingual and multilingual systems. we also explore translation of english imdb user movie reviews, a domain/genre where only monolingual data are available. our results confirm that multilingual systems with joint target languages perform better. the performance of translation from english is generally much better than from german, partly because german is morphologically more complex and partly because the corpus consists mostly of parallel human translations instead of original text and its human translation. more research should be carried out on translation from german (as well as on more languages other than english) in order to better understand the poten- for translating user reviews, creating synthetic in-domain parallel data through back-and forwardtranslation and adding them to a small out-of-domain parallel corpus can yield performance comparable with a system trained on a full out-of-domain corpus. our experiments still leave some important questions open, such as the impact of the size of synthetic ft data and impact of performance of mt system which generated this data. therefore, more detailed research including manual evaluation and analysis of translated reviews is needed in this direction. apart from this, removing misaligned segments by log-probabilities provided by mt systems should be investigated systematically, by comparing different nmt systems for cleaning together with different thresholds for log-probabilities.