A multi-source approach for B reton– F rench hybrid machine translation (2020)

Abstract: Corpus-based approaches to machine translation (MT) have difficulties when the amount of parallel corpora to use for training is scarce, especially if the languages involved in the translation are highly inflected. This problem can be addressed from different perspectives, including data augmentation, transfer learning, and the use of additional resources, such as those used in rule-based MT. This paper focuses on the hybridisation of rule-based MT and neural MT for the Breton–French under-resourced language pair in an attempt to study to what extent the rule-based MT resources help improve the translation quality of the neural MT system for this particular under-resourced language pair. We combine both translation approaches in a multi-source neural MT architecture and find out that, even though the rule-based system has a low performance according to automatic evaluation metrics, using it leads to improved translation quality.

## introduction corpus-based approaches to machine translation (mt), such as neural mt (nmt), struggle when the size of the available parallel corpora for a given language pair is scarce @xcite . even though the problem can be partially mitigated with accurate hyper-parameter tuning @xcite , taking advantage of additional resources can help to further improve the quality of the system. monolingual texts in both languages can be leveraged with the help of back-translation @xcite to generate synthetic c 2020 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. parallel corpora. it is also possible to use only monolingual corpora and follow an unsupervised nmt approach @xcite . parallel corpora from related language pairs can also be leveraged thanks to multilingual nmt @xcite and other forms of transfer learning @xcite . in addition to the use of corpora, linguistic resources can also be used to improve nmt. if morphological analysers or syntactic parsers are available, they can be used to build a richer representation of the words being translated @xcite . even full rule-based mt (rbmt) systems can be combined with nmt in order to build hybrid systems @xcite . in this work, we focus on an under-resourced language pair: breton-french, and study mechanisms to build a hybrid system by combining nmt with the breton-french system built with the apertium rbmt platform @xcite . we aim at producing sentences that combine knowledge extracted from the parallel corpus and from the rbmt system. hence, we go beyond approaches that simply choose the best system (either rbmt or nmt) for each input sentence (see below). we use multi-source nmt and formalise the problem of combining both sources of knowledge as an automatic post-editing @xcite problem. in this way, we are able to explore different ways of generating the rbmt output, using different resources, to study which resources are more useful for the hybrid approach. the rest of the paper is organised as follows. the remainder of this section lists previous works related to the hybridisation of rbmt and corpusbased systems, including approaches for integrating external bilingual segments into nmt. section 2 then explains the resources available for breton-french and the challenges of translating between breton and french. section 3 describes the hybrid architecture chosen. section 4 presents the experiments carried out and discusses the results obtained. the paper ends with some concluding remarks. hybrid systems combining rule-based and corpus-based approaches. the creation of hybrid systems combining rbmt and statistical mt (smt) has been explored by many authors. the most relevant approach for this work @xcite enlarged the training corpus of an smt system with 116,500 sentence pairs made up of all possible inflected breton forms and their inflected french translations as present in an earlier version of the apertium breton-french system we are using. @xcite followed a similar approach for other language pairs. more sophisticated approaches @xcite @xcite involve modifying the smt architecture. concerning the combination of rbmt and nmt, a relevant line of research involves choosing the best output (either rbmt or nmt) for each source sentence. for instance, @xcite propose training an automatic classifier for this task and use some features to help predict how difficult is the source sentence for each system: for instance, the degree of morphological and syntactic ambiguity is useful to estimate how difficult is the sentence for the rbmt system, while the token frequency on the training corpus can help to assess how difficult it is for the nmt system. similarly, @xcite use confidence scores computed for each system to choose the best alternative for each source sentence. @xcite experimented with the integration of rbmt bilingual dictionaries and syntactic parsers into nmt without success. finally, the multi-source architecture studied in this paper has been preliminary explored by @xcite . the main differences with this work are: i) they did not study the impact of the different components of the rbmt system; and ii) they did not perform a hyper-parameter search, which could explain the poor performance of their transformer systems. in addition, we conduct an automatic analysis of the errors produced by our hybrid approach. ## breton-french machine translation the breton language (brezhoneg in breton) is a celtic language of the brittonic group that is spoken in the west of brittany (breizh izel or "lower brittany") in france, and the main language with which it has contact is french, the only official language; in fact, breton, spoken by about 200,000 people, has virtually no legal recognition in france. resources for breton: programs like firefox, google applications and some microsoft programs have been localized and there is a 70,000-page breton wikipedia. there is little software dedicated to breton; most of it free/open-source, such as the apertium mt system and the languagetool spelling and grammar checker. this software and services such as the freelang online dictionary foot_0 are based on linguistic resources such as morphological analyzers, monolingual and bilingual dictionaries. as for bilingual text corpora, today opus foot_1 contains about 400,000 sentence pairs, most of them very specialized, in the field of computer science. the apertium breton-french system: the apertium platform foot_2 contains an mt system designed to allow french-speaking readers to access written breton content (gisting). foot_3 this mt system @xcite , the only one in the world for breton, was released in may 2009 as the result of the joint efforts of the ofis ar brezhoneg, 5 the spanish company prompsit language engineering, and the universitat d'alacant and is based on the apertium platform @xcite . dictionary development started with the free dictionaries for breton in lexilogos. 6 development of the apertium breton-french mt system slowly continues. the quality of the french generated is not suitable for publishing, but may be used to get a rough idea of the meaning of a breton text. automatic inference of translation rules for breton-french: there have been attempts to improve the apertium breton-french system in an unsupervised way. in particular, @xcite proposed an algorithm for the automatic inference of shallow-transfer rules from small parallel corpora and existing rbmt dictionaries.the result of applying the algorithm to the apertium breton-french system using just the parallel data prepared by @xcite was a set of rules whose quality, as measured by automatic mt evaluation metrics, was close to the existing hand-crafted ones. ## system architecture we propose combining the explicit linguistic knowledge encoded in the breton-french apertium system with the implicit knowledge encoded in a parallel corpus by means of multi-source nmt @xcite . given a source-language (sl) sentence to be translated, our proposed architecture proceeds as follows (see figure 1 ): first, the sl sentence is translated with the rbmt system; then the original sl sentence and its rbmt translation are passed as inputs to the multi-source nmt system, which produces the final translation. at training time, the sl side of the parallel sentences in the training corpus is translated with apertium to obtain a "trilingual" parallel corpus. as it is common practice, the multi-source system works on byte-pair-encoding (bpe) sub-word units @xcite obtained from both inputs and the output together. with this architecture, we expect the nmt system to learn to translate from the sl text with help from the rbmt output. it could also be seen the other way round: the nmt system postedits the good and use it improperly as if it were correct @xcite ). 5 now ofis publik ar brezhoneg 6 https://www.lexilogos.com/breton_ dictionnaire.htm this work has been funded by european union's horizon 2020 research and innovation programme under grant agreement no 825299 (gourmet project)  joint sl-tl bpe applied after segmentation conclusions transformer rnn + rbmt weighted ensembling  transfer learning, multilingual mt and backtranslation allowed us to take advantage of additional data  multi-source mt is useful to mix rbmt and nmt, but transformer multi-source architectures need to be thoroughly evaluated en-kk data augmentation multi-source nmt fr translation br text fr text apertium s, uine ithout ta* d data nd system bleu chrf++ single transformer 9.57 39.76 single rnn 8.43 37.24 single rnn +apertium 8.68 37.99 constrained submission 9.97 40.28 unconstrained submission 9.90 40.31 chrf++ 27.80 39.93 39.67 42.87 42.86 44.46 44.99 45.24 automatic evaluation of submissions on newstest2019 • constrained: 2 transformer + 2 rnn checkpoints • unconstrained: 2 transformer + 2 hybrid checkpoints : 2 nd rbmt output with the help of the sl sentence. in fact, this architecture has been successfully applied for automatic post-editing (junczys-dowmunt and grundkiewicz, 2018). the apertium architecture as well as the multisource nmt architecture used in our experiments are described in the remainder of this section. ## experiments and results for the experiments we used the following corpora available at opus: foot_6 tatoeba, gnome, ofispublik, kde4, wikimedia, ubuntu and opensubtitles. for development and testing we used the same portions of the ofispublik corpus used by @xcite , the rest of corpora, after de-duplication, were used for training. in order to determine the appropriate amount of bpe operations and hyper-parameter values to be used for the two models we proceed as follows: first we tried with 5,000, 10,000, 20,000, and 30,000 bpe operations with a baseline system not using any apertium data. when doing so the rest of hyper-parameters were set to the values recommended by @xcite for the recurrent model and by @xcite for the base transformer model, respectively, except for the model size which was set to 512. training stopped after 5 validations without any perplexity improvement on the development corpus; validations were performed every 1,000 mini-batches; each minibatch contained 8,000 tokens. the best results were obtained with 20,000 bpe operations for the recurrent model and 5,000 for the transformer. we then performed a grid search to find the appropriate hyper-parameters for each model. the hyper-parameters tried for the recurrent model are: we used the same value for both so as not to explore the cartesian product. cell depth is defined as the number of gru transitions in the deep transition architecture proposed by miceli barone et al. (2017, sec. 4.2). the hyper-parameters tried for the transformer model are: the best results for the recurrent model were obtained with an embedding size of 512 and encoder and decoder cell depths of 2. for the transformer, the best results were obtained with 4 attention heads, model size of 512 and 4 encoder and decoder layers. these hyper-parameters are the ones used for the rest of experiments reported. an explanation of the different reference systems follows: as regards the different ways of exploiting the linguistic resources in apertium, we generated the additional input translation provided to the multisource nmt system with the same rbmt configurations used as reference systems (see above) as well as a word-for-word translation obtained using exactly the same bilingual dictionary we used for the base+dic nmt reference system. as this dictionary contains multi-word lexical units, we translated word for word in a left-to-right, longestmatch fashion so that the bilingual entry covering the longest sequence of tokens is selected when there is more than one possibility. when the bilingual dictionary contained more than one translation per source word, they were all included in the output separated by a special token. this happened to 495 source words in the training corpus. the results in hjerson works on the surface form and lemma of the words in the reference translations and mt outputs. the lemmas used were obtained with the standfordnlp lemmatiser @xcite . we computed the relative difference in the num-ber of hjerson errors in the test set between the multi-source nmt systems and the base nmt system; 11 a positive value means that the multi-source system made more errors than the base nmt system. for the recurrent architecture, the addition of expanded dictionaries to the bilingual training corpus does not significantly alter the number of errors. one possible explanation could be that the potential gains of introducing more lexical knowledge in the system are neutralised by the presence of single-word sentences in the training corpus, that could harm the fluency of the generated sentences. multi-source nmt systems, on the contrary, tend to make fewer lexical errors than the base nmt system. this happens for three out the four multisource systems, where the system with hand-crafted rules is the only one in which the reduction in lexical errors is not statistically significant. neither automatically inferred nor hand-crafted transfer rules cause a statistically significant impact in the amount of inflection errors, and both of them make reordering errors increase. the multi-source system without transfer rules is the best performing system according to automatic evaluation metrics because it is the one that brings the largest reduction in lexical errors, which constitute the most frequent error category (see ## concluding remarks this paper focused on the hybridisation of rbmt and nmt for the breton-french under-resourced language pair. the aim of the paper is to study to what extent the resources from the apertium rbmt system help the nmt system to improve its output. we combined both translation approaches in a multi-source nmt architecture and explore the use of different resources in the apertium breton-french system to generate the rbmt translation to be used as an additional input. despite the low performance of the rbmt system, the hybrid system is able to outperform a pure nmt baseline. the best translation performance is achieved with a hybrid system whose rbmt subsystem contains no transfer rules at all but takes advantage of the breton morphological analyser and part-of-speech tagger, the french generator and post-generator and the bilingual dictionary. the fact that the use of no transfer rules provides the best results while the rbmt system using no transfer rules, when evaluated in isolation, performs worse than the rest of rbmt configurations may seem contradictory. however, the automatic error analysis revealed that the hybrid systems using