Neural Machine Translation of B asque (2018)

Abstract: We describe the first experimental results in neural machine translation for Basque. As a synthetic language featuring agglutinative morphology, an extended case system, complex verbal morphology and relatively free word order, Basque presents a large number of challenging characteristics for machine translation in general, and for data-driven approaches such as attentionbased encoder-decoder models in particular. We present our results on a large range of experiments in Basque-Spanish translation, comparing several neural machine translation system variants with both rule-based and statistical machine translation systems. We demonstrate that significant gains can be obtained with a neural network approach for this challenging language pair, and describe optimal configurations in terms of word segmentation and decoding parameters, measured against test sets that feature multiple references to account for word order variability.

## introduction neural machine translation (nmt) is fast becoming the dominant paradigm in both academic research and commercial exploitation, as evidenced in particular by large machine translation providers turning to nmt for their production engines @xcite and nmt systems achieving the best results in most cases on standard shared tasks datasets @xcite . c 2018 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. sequence-to-sequence neural networks have proved effective in modelling translation phenomena @xcite . in particular, attentional encoder-decoder models @xcite have become a default nmt architecture, with other architectural variants explored in recent work @xcite . these models have already been applied to a wide range of languages, initially on the most studied european languages and recently to a larger set of cases that includes morphologically rich languages @xcite . in this article we explore the applicability of neural machine translation to basque, a language with noteworthy characteristics that may represent a challenge for encoder-decoder approaches with attention mechanisms. first, basque is a synthetic language that features agglutinative morphology, i.e. where words can be formed via morphemic sequences, and a large number of case affixes that mark ergativity, datives, different types of locatives and genitives, instrumentality, comitativity or causality, among others. verbal morphology is also relatively rare, displaying complex forms that include subject, direct object, indirect object and allocutive agreement markers, with number, tense and aspect being marked as well. this kind of rich morphology raises difficulties in terms of word representations and drastically increases data sparseness issues. a detailed description of basque grammar can be found in @xcite . secondly, although phrases in this language present a rather fixed inner order, as exemplified for instance by the regular structure of noun phrases, 1 at the sentential level the ordering is rela-tively free. syntactically, order is essentially determined in terms of focus and topic. although different orderings mostly reflect underlying variations according to these notions, for translation between basque and languages with more rigid syntax the end-result is higher variability in terms of sentential input and output. such variations may represent an additional challenge for nmt models that manage input information via learned attentional distributions and generate translations via decoding processes based on the previously generated element and beam searches. finally, basque is a low-resourced language, with few publicly available parallel corpora. this is a third challenge for data-driven approaches in general, and nmt in particular as it usually requires larger training resources than statistical machine translation @xcite . to explore these challenges, we built several large neural machine translation models for generic basque-spanish translation, and compare their results with those obtained with rule-based and statistical phrase-based systems @xcite . our exploration of nmt variants for this language pair mainly focuses on different sub-word representations, obtained via either linguistically-motivated or frequency-based word segmentations, and on different data exploitation methods. we measure the impact of ordering variations partly via manually-created multiple references and also evaluate the impact of tuning the decoding process in terms of length and coverage along the lines of @xcite . the paper is organised as follows: section 2 describes related work in machine translation for basque and other morphologically-rich languages; section 3 presents the parallel corpora collected for the basque-spanish language pair; section 4 describes the different translation models used for the experiments presented in section 5; finally, section 6 draws conclusion from this work. ## related work morphologically rich languages, a large denomination which includes synthetic languages where words are formed via productive morphological affixation, have been extensively studied in the machine translation literature. in statistical machine translation (smt) in particular @xcite tion of determiners to the rightmost constituent in the noun phrase. several approaches address morphological variation via character-based translation @xcite @xcite . a case study along these lines for languages with rich morphology is @xcite , who implement a character-to-character nmt system augmented with a re-scoring model. they report improvements for finnish-english translations but not for turkish-english, although the latter result might be due to lack of sufficient training data. other approaches tackle this issue via word segmentation into sub-words. byte pair encoding (bpe) @xcite has become a popular segmentation method where infrequent words are segmented according to character pair frequencies. alternatives include the use of morphological analysers such as morfessor @xcite or chipmunk @xcite . @xcite compare the use of these three segmentation alternatives for turkish-english, obtaining better results with chipmunk and morfes-sor than with bpe. in @xcite , both supervised and unsupervised morphological segmentation are shown to outperform bpe for turkish to english neural machine translation. morphological decomposition has also been performed with tools tailored for the task, as is the case in (sánchez-cartagena and toral, 2016), who report improvements using the rule-based morphological segmentation provided by omorfi @xcite for english-finnish translation. finally, hybrid techniques have also been applied, as in @xcite who built a character/word hybrid nmt system where translation is performed mostly at the word level and the character component is consulted for rare words. their results for english to czech demonstrate that their character models can successfully learn to generate well-formed words for a highly-inflected language. this approach has been notably applied to english-finnish by @xcite , who also include bpe segmentation in a system that ranked as the top contribution in the wmt2017 shared task for english-finnish. the challenges of machine translation of basque have been addressed in different frameworks. an example-based data-driven system was thus developed by @xcite and a rule-based approach was used to develop the matxin system for spanish to basque translation @xcite ; both systems are compared in @xcite . in @xcite , a hybrid architecture is presented, combining rulebased and phrase-based statistical machine translation approaches. their hybrid system resulted in significant improvements over both individual approaches. in the next sections, we provide the first description of a large-scale nmt system for the basque-spanish language pair. ## corpora to build representative translation models for the basque-spanish language pair, parallel corpora were collected and prepared from three different sources: professional translations in different domains, bilingual web pages, and comparable data in the news domain. ## models in the next subsections, we describe the different nmt models for basque-spanish that were built using the corpora described in the previous section, as well as the considered baseline systems. ## experiments in this section we first describe the experimental settings and system variants, then present and discuss the results. ## conclusions we presented the first results in neural machine translation for basque, a synthetic language with an extended case system, complex verbal morphology and relatively free word order. the characteristics of the language made it an interesting test case for nmt and we showed that significant gains could be obtained with a neural network approach, when compared to both rule-based and statistical machine translation systems. several variants were prepared in terms of both corpora and models, to determine the optimal configurations for generic machine translation in basque-spanish. the impact of noisy datasets when training nmt systems was confirmed in our experiments, and we showed the improvements achievable with a simple filtering of length difference outliers. also coming from our results were the gains resulting from fine-grained morphological analysis on the source side, although byte pair encoding was shown to be a robust method overall for this language pair. the presented results were com-puted on different complementary test set, providing a strong confirmation of the importance of multiple references in general, and for the evaluation of basque translation in particular. neural machine translation has been successfully applied to a large range of languages and scenarios, with recent work centred on languages with rich morphology. this work contributes to this line of research, demonstrating the significant improvements obtained with nmt on a language which features a wide range of properties that represent a challenge for past and current approaches to machine translation.