Alignment verification to improve NMT translation towards highly inflectional languages with limited resources (2021)

Abstract: The present article discusses how to improve translation quality when using limited training data to translate towards morphologically rich languages. The starting point is a neural MT system, used to train translation models, using solely publicly available parallel data. An initial analysis of the translation output has shown that quality is sub-optimal, due mainly to an insufficient amount of training data. To improve translation quality, a hybridized solution is proposed, using an ensemble of relatively simple NMT systems trained with different metrics, combined with an open source module, designed for a low-resource MT system. Experimental results of the proposed hybridized method with multiple independent test sets achieve improvements over (i) both the best individual NMT and (ii) the standard ensemble system provided in the Marian-NMT system. Improvements over Marian-NMT are in many cases statistically significant. Finally, a qualitative analysis of translation results indicates a greater robustness for the hybridized method.

## introduction the state of the art in mt involves corpus-based systems developed with machine-learning methods. these methods learn from corpora the models needed for translation. a key strength of this approach is that the system is adapted specifically towards the data it is trained with. for many years, the most successful data-driven approaches were phrase-based and syntax-based statistical mt (smt; @xcite . however, lately neural mt (nmt) based on the encoderdecoder architecture and the concept of attention @xcite has become very popular. indeed, since 2015, in mt shared tasks @xcite @xcite most top-performing systems have been nmt systems. this trend is confirmed in the most recent mt shared task @xcite , where 80% of participating systems are of nmt type. though nmt represents the state of the art for mt, specific weaknesses have been reported: nmt requires very large amounts of parallel data, measured in millions of parallel sentences. this is reflected by the separate studies carried out for mt with limited resources, which includes initiatives such as lorelei foot_0 . in the case of morphologically-rich languages, the requirements for parallel corpora are further exacerbated. proposed approaches for translating towards lowresource and morphologically-rich languages have included transfer learning @xcite as well as multilingual and multi-way nmt @xcite . in this paper, an effort to improve the translation quality is presented, when translating towards a morphologically-rich language, while reducing the training time. this approach combines the output of multiple nmt systems with an nlp module developed for an example-based mt paradigm, resulting in a hybridized solution. the latter module is fast and runs independently of its original mt system and thus the computational complexity of the proposed hybrid solution is not substantially increased over the base nmt system. the idea of combining multiple mt models to produce a higher performing mt system has been studied extensively in the area of mt. for instance in the recent shared task @xcite more than 20 entries consist of ensembles of multiple nmt systems. ensembles of weaker nmt systems of the same general architecture have been proposed by @xcite to train a higher performing nmt system. in addition ensembles of factored nmt models have been proposed for automatic post-editing and quality estimation (for example @xcite . this base nmt system is described in section 2. the training data used is reported in section 3. the proposed hybridization is presented in section 4, whilst the improvements attained are presented in section 5. future developments are discussed in section 6. ## overview of the base nmt system since nmt systems have achieved the highest translation quality in recent evaluation contests, the marian-nmt package (junczys-dowmunt et al., 2018) is adopted for experimentation here. marian-nmt development was funded by the european commission to consolidate nmt research and incorporates the most recent advances in nmt. its code is optimized to reduce the cpu/gpu time required to complete the simulations of nmt systems. for creating nmt systems, three of the models provided by marian-nmt were chosen, termed as the "transformer", "amun" and "s2s" models. the "transformer" model has been based on the work of @xcite and uses a simple structure incorporating attention mechanisms and dispensing with recurrence to implement a fast nmt system. the other two models are more conventional, using a recurrent neural network to implement the translation. the "amun" model follows the approach of @xcite , employing a recurrent neural network but allowing the model to automatically search for wider ranges of the source language (sl) to connect with the target language side (tl) words. finally, "s2s" implements a recurrent neural network-based encoder-decoder model with attention mechanism, using the architecture proposed in @xcite . hereafter, the three models are identified via the names used within marian-nmt, which are also used in evaluations (cf. @xcite . the main configuration parameters used for each model are depicted in regarding the main nmt parameters, all recurrent networks comprise 1,024 units in the hidden layer, an encoder depth of 6 layers and an embedding size of 512. all cells used both in the encoder and decoder side are gated recurrent units (gru). the transformer dimension is set to 2,048. to reduce the lexicon size, a total of 85,000 merge operations are allowed using the bpe (byte pair encoding) method proposed in @xcite , this being the default setting for marian-nmt applications. initially, the three marian-nmt models are trained to provide the base nmt systems. typically, for a single-gpu system (equipped with an nvidia titan xp gtx1080 gpu card driven by an intel i-9700k cpu), 24 hours are required for training the transformer, 130 hours for amun and 308 hours for s2s. this is equivalent to a ratio of 1:5:12 to train the respective systems. ## experimental set-up the experiments aim to improve the translation accuracy of an nmt system, taking into account limited training data and constrained computing resources. in order to investigate translation into a lesser-used and highly inflectional language, we common to all 3 models layer-normalization yes exponential smoothing yes beam-size 6 normalize 0.6 early-stopping 5 transformer-specific transformer-dropout 0.1 transformer-dropout-attention 0.1 transformer-dropout-ffn 0.1 amun and s2s-specific dropout-rnn 0.2 dropout-src 0.1 dropout-trg 0.1 the largest part of the europarl corpus and the entire dgt-acquis corpus are used to train the nmt system. three small portions of the europarl corpus have been reserved for test and validation purposes. more specifically, two independent sets of approx. 3,000 europarl sentences each are excluded, to ensure that the nmt evaluation is unbiased. in the present experiment, one of these sets is used for in-training validation. the other set is reserved to allow additional cross-evaluation of experiments in the future, without invalidating the previously trained models. finally, a sample of 1,000 sentences from europarl (testset2) is retained to provide an unseen in-domain test set. another independent test set was drawn from the presemt project resources, comprising 200 sentences which have not been used to either train an mt model or create any resources used herewith (denoted as testset1). a preliminary analysis of the nmt outputs has shown that translations are commendably fluent, though errors are evident. a sample of amun translations is shown in figure 1 . in sentence #1, the term "αμερικανοί" (transl. "americans") is erroneously used as a translation of the terms "american", "european", and "japanese monopolies". similarly, in sentence #2, the phrase "η καταπολέμηση της φτώχειας" (meaning "the reduction of poverty") is used to translate semantically diverse phrases, including "genetically modified organisms", and "the negative social effects of unbridled, unregulated globalization". repetition is a widely reported weakness of nmt systems, most frequently attributed to insufficient training data. an additional problem concerns the translation of rare words (i.e. words with low frequency in the corpus), due to the limited vocabulary that nmt systems can directly handle. this is especially severe when translating towards languages with complex morphology, which increases the effective vocabulary size. for example the word "ostensibly" is translated into greek as "ostenfigher" (ungrammatical). similarly the word "room" is translated as "δωματείο" instead of the correct "δωμάτιο" (meaning room), whilst the word "indistinct" is translated as "άχωρος" which is not a valid greek word. αnother issue is that entire phrases present in the source text may be omitted in the translation. for instance the sentence "businesses have undertaken the education" is translated by a transformer nmt as "h εκπαίδευση έχει αναλάβει , [meaning "education has undertaken"]. hence, the subject "business" has been deleted. ## experimental results to determine the quality of the nmt-based translations (amun-, s2s-and transformer-based models), two widely used mt evaluation metrics are utilised, namely bleu @xcite and nist @xcite . to calculate both these metrics, the mt-eval package (version 13a) is used. for pam, the presemt bilingual lexicon from greek to english is used, which contains approx. 8,000 lemmas and 40,000 greek-english token pairs. this lexicon is from the same domain as testset1 and is thus out-of-domain for testset2, providing a more limited coverage for this testset. two different types of experiments are possible, depending on whether the ensemble comprises multiple nmt architectures, or only one type of architecture. the first experiment reported here involves nmt ensembles that all share the same architecture, but are optimized with different criteria. the second type of experiment studies ensembles which consist of systems with different architectures, to investigate if their combination results in a better translation quality. ## conclusions and future work this article has studied the creation of translation systems towards highly inflectional languages, when the amount of in-domain training data is limited. emphasis has been placed on improving the translation accuracy of nmt models that can be trained more rapidly and cost-effectively (in terms of cpu processing power) and rendering this performance comparable to that of more complex models. the marian-nmt package has been chosen as the starting point to create nmt models for the english to greek language pair. using only publicly available text corpora, the nmt models produce commendably fluent translations. identified errors in the nmt translations are typical of a lack of training data. a hybrid methodology has been proposed that samples an ensemble of nmt models to select the final translation, chosen by a module calculating the alignment level between the input sentence and each translation. this module was developed for resource-poor mt systems. the proposed hybrid approach has resulted in higher bleu and nist scores, compared to those of single nmt models. improvements are in many cases statistically significant even over the ensemble system provided within the marian-nmt package, indicating the promising nature of the hybrid approach. also, the translation process is found to be more robust, giving more consistent translations in comparison to the marian-nmt ensemble system, which occasionally omits large portions of the input text from the translation. one of the advantages of the proposed method is that it is general-purpose and does not rely on the use of ensembles of neural mt systems with a specific architecture. instead, it can be used to combine the results of different types of neural mt systems, or mt systems that belong to different paradigms, or even to combine human translations. in addition the proposed method can be used to clean up a corpus of parallel sentences or several such corpora, by removing sentence pairs for which the source and target-language texts do not have a high degree of parallelism. similarly, the proposed method may be used to filter a corpus consisting of original text and its mt-derived translation, to produce a parallel corpus for training of other mt systems, fulfilling a role similar to that proposed by @xcite . one point for future research is how effective a filtering system based on pam would be, in comparison to already proposed systems. future work involves some relatively simple activities that can be imminently implemented, such as releasing the modified version of pam for experimentation by interested parties. another short term activity involves using the proposed method with sacrebleu instead of the bleu and nist metrics provided by mt-eval. future experiments will investigate the effectiveness of this hybrid approach for other language pairs. one area of interest would be to determine the effectiveness of the pam-based method when very limited dictionaries are available as well as the limitations when the accuracy of the parser used is relatively low. all these represent issues for the future. it is also planned to study the approach using systematic optimisation of the pam parameters, to identify in more detail configurations that produce more accurate translations. another possibility is to use pam to detect sub-sentential parts of the translated sentences with particularly poor alignments between input and translation and seek better translations of only these specific parts. another direction is to investigate more extensively cases where the translation is not sufficiently close to the input sentence. then, comparisons to other low-scored translations are more difficult and result in a reduced level of confidence of the chosen translation. such a line of study will evaluate more thoroughly the robustness of the proposed method.