Neural Machine Translation between similar S outh- S lavic languages (2020)

Abstract: This paper describes the ADAPT-DCU machine translation systems built for the WMT 2020 shared task on Similar Language Translation. We explored several set-ups for NMT for Croatian–Slovenian and Serbian–Slovenian language pairs in both translation directions. Our experiments focus on different amounts and types of training data: we first apply basic filtering on the OpenSubtitles training corpora, then we perform additional cleaning of remaining misaligned segments based on character n-gram matching. Finally, we make use of additional monolingual data by creating synthetic parallel data through back-translation. Automatic evaluation shows that multilingual systems with joint Serbian and Croatian data are better than bilingual, as well as that character-based cleaning leads to improved scores while using less data. The results also confirm once more that adding back-translated data further improves the performance, especially when the synthetic data is similar to the desired domain of the development and test set. This, however, might come at a price of prolonged training time, especially for multitarget systems.

## introduction machine translation (mt) between closely related languages is, in principle, less challenging than translation between distantly related languages, but it is still far from being solved. while mt between closely related south-western slavic languages, croatian, slovenian and serbian based on the rule-based (rbmt) and the phrase-based (pb-smt) approaches has been investigated in the last years @xcite @xcite @xcite , to the best of our knowledge, the new state-of-the-art neural machine translation (nmt) has not been investigated yet for these languages. in this work, we first compare bilingual and multilingual systems in order to determine whether joining serbian and croatian data is useful. afterwards, we investigate additional cleaning of remaining misaligned segments by using character n-gram matching scores @xcite . the beauty of the method for similar languages is that it can be applied directly to the given training corpus providing matching scores for each pair of the source-target segments. for distant languages, translation of one side of the training corpus would be required. finally, we make use of monolingual data in each of the three languages by creating additional synthetic parallel training sets via back-translation @xcite @xcite . ## language properties common properties all three languages, croatian, serbian and slovenian, belong to the south-western slavic branch. as slavic languages, they have a very rich inflectional morphology for all word classes: six cases and three genders for all nouns, pronouns, adjectives and determiners. for verbs, person and many tenses are expressed by the suffix so that the subject pronoun is often omitted. there are two verb aspects, so that many verbs have perfective and imperfective form(s) depending on the duration of the described action. as for syntax, all three languages have quite a free word order, and neither language uses articles, either definite or indefinite. in addition to this, multiple negation is always used. croatian and serbian croatian and serbian exhibit a large overlap in vocabulary and a strong morpho-syntactic similarity so that the speakers can understand each other without difficulties. nev-ertheless, there is a number of small but notable and also frequently occurring differences between them. the largest differences between the two languages are in vocabulary: some words are completely different, some however differ only by one or two letters. apart from lexical differences, there are also structural differences mainly concerning verbs: modal verb constructions, future tense, as well as conditional. slovenian even though slovenian is very closely related to croatian and serbian, and the languages share a large degree of mutual intelligibility, a number of croatian/serbian speakers may have difficulties with slovenian and the other way round. the nature of the lexical differences is similar to the one between croatian and serbian, namely a number of words is completely different and a number only differs by one or two letters. however, the amount of different words is much larger. in addition to that, the set of overlapping words includes a number of false friends (e.g. brati means to pluck in croatian and serbian but to read in slovenian). the amount of grammatical differences is also larger and includes local word order, verb mood and/or tense formation, question structure, usage of cases, structural properties for certain conjunctions, as well as some other structural differences. another important difference is the slovenian dual grammatical number which refers to two entities (apart from singular for one and plural for more than two). it requires additional set of pronouns, as well as additional sets for noun, adjective and verb inflexion rules not existing either in croatian or in serbian. ## data for training, we used publicly available opus 1 parallel corpora @xcite indicated by the workshop organisers. opensubtitles is indicated for all translation directions. for croatian-slovenian, other corpora are indicated too, but they are either not sentence-aligned (jw300 ) or are extremely noisy (dgt, multiparacrawl ). therefore, we decided to use only opensubtitles for all translation directions. it is worth noting that the organisers also indicated the setimes news parallel croatian-serbian corpus. developing an additional croatian-serbian mt system for converting serbian data into croatian and vice versa was shown to be helpful for the pbsmt approach @xcite . however, our preliminary experiments in this direction indicated that this technique is not helpful for the nmt approach. the original parallel data were filtered in order to eliminate noisy parts: too long segments (more than 100 words), segment pairs with disproportional sentence lengths, segments with more than 1/3 of non-alphanumeric characters, as well as duplicate segment pairs were removed. the statistics of the remaining subtitles together with the development and test sets is shown in ## mt systems all our systems are built using the sockeye implementation @xcite of the transformer architecture @xcite . the systems operate on sub-word units generated by byte-pair encoding (bpe) @xcite . we set the number of bpe merging operations at 32000. we use shared vocabularies between the languages because they are similar. multilingual systems are built using the same technique as @xcite and @xcite , namely adding a target language label "sr" or "hr" to each source sentence. we investigated the following set-ups: the four bilingual systems, @xmath0, @xmath1, @xmath2, are trained separately for each language pair and each translation direction on about 11m parallel segments. the multisource system hr+@xmath3, thus resulting in 20.2m parallel segments. the multitarget system @xmath4.2m segments with target language identificators "sr" and "hr" added to the source side. ## results we evaluate our systems using the following three automatic overall evaluation scores: sacre-bleu @xcite , chrf @xcite and char-acter @xcite . the bleu score is used because of the long tradition. the two character level scores are shown to correlate much better with human assessments @xcite , especially for morphologically rich languages. in addition, the chrf score is recommended as a replacement for bleu in a recent detailed study encompassing a number of automatic mt metrics @xcite . in addition to the automatic mt evaluation scores, for each of the systems we report the size of the training corpus and the training time. results on the development set for the systems trained on opensubtitles, it can be seen that for each translation direction, multilingual systems yield better automatic scores than bilingual systems at the cost of slightly prolonged training time (from about 3 days to 3-4 days). therefore we choose the two multilingual systems hr+@xmath5. the chrf cleaning of opensubtitles reduces the size of the corpus and the training time while slightly improving automatic scores. the reduction in time is slightly smaller for the multitarget translation from slovenian (down to 2-3 days) than for the multisource translation into slovenian (down to less than 2 days). adding the back-translated data from wac improves the automatic scores for more than 10 points for multisource translation (into slovenian) and for 5 to 10 points for multitarget translation (from slovenian). this could be expected, especially since the monolingual data was chosen to be similar to the development data. nevertheless, this large improvement comes at a price. although the increase of the corpus is not very large, from 10.8m to 14.8m, the training time increases to (more than) 3 days. it can be noted that for some set-ups, the multitarget system needs more training time. the probable reason is the diversity of the target part of the training corpus -the system has to deal with two target languages, and when synthetic data is added, also with two different domains/genres for each of them. results on the test set based on the results on the development set, we submitted the outputs of the systems with backtranslated data (hr+@xmath6, @xmath7. the outputs of the systems trained on cleaned data (hr+@xmath8, @xmath9, and the outputs of the baseline multilingual systems (hr+@xmath10, @xmath11. the test sets were not at all translated by the initial bilingual systems, therefore the results are not available. it can be seen that the tendencies for the test set are almost the same as for the development set. the only difference is the larger improvement obtained by cleaning opensubtitles with the chrf scores. further detailed analysis involving manual inspection is needed to better understand this difference. ## summary and outlook this work investigates different set-ups for training nmt systems for translation between three closely related south-slavic languages: slovenian on one side, and serbian and croatian on the other side. we explore different sizes and types of training corpora, as well as bilingual and multilingual systems. our results show that for all translation directions, multilingual systems with joint croatian and serbian data perform better than bilingual systems. the results also show that cleaning misaligned segments using character n-gram matching (chrf score) represents a fast and useful method for closely related languages, which improved the evaluation scores while reducing corpus size and training time. finally, we confirm that adding backtranslated synthetic data, which is the usual practice in neural machine translation, can yield large improvements of evaluation scores also for these languages. nevertheless, for multitarget translation, it might result in a prolonged training time due to increased variety of the target language side. future work should include more genres and domains, as well as detailed analysis of errors and problems in order to further improve the performance of nmt between south slavic languages.