Neural Machine Translation with Inflected Lexicon (2021)

Abstract: The paper presents experiments in neural machine translation with lexical constraints into a morphologically rich language. In particular and we introduce a method and based on constrained decoding and which handles the inflected forms of lexical entries and does not require any modification to the training data or model architecture. To evaluate its effectiveness and we carry out experiments in two different scenarios: general and domain-specific. We compare our method with baseline translation and i.e. translation without lexical constraints and in terms of translation speed and translation quality. To evaluate how well the method handles the constraints and we propose new evaluation metrics which take into account the presence and placement and duplication and inflectional correctness of lexical terms in the output sentence.

## introduction the incorporation of an inflected lexicon into neural machine translation (nmt) enables system developers to adapt the translation to specific domains, and users to adjust translations of phrases generated by the translation system. phrase-based statistical machine translation (pb-smt; @xcite provided control over system output, e.g. by using a domain-specific lexicon. the shift from phrase tables in pb-smt to a continuous-valued representation of text in nmt has made it more difficult to incorporate lexical constraints into the translation process. the task of integrating the lexicon and a neural translator is even more challenging for highly morphological languages, when the lexical items should be correctly inflected in the output text. we carry out experiments for translation with inflected lexical constraints. as the target language of the translation we choose polish, whose inflection is typical of the slavic languages. the number of declination cases is six, and the verbal groups are inflected by tense, number, and person. in terms of correct inflection of the output, translation from english to polish seems to be a more challenging task than translation in the other direction. unlike in some preceding experiments, we require that the lexicon may be modified after the model training has been completed. we believe that in post-editing mode users expect the translation engine to immediately mirror their adjustments to the lexicon. ## related work one of the first papers that addressed the incorporation of a lexicon into an nmt system was @xcite . the authors noticed that nmt systems tend to produce unexpected output for low-frequency words (such as names of countries). the solution proposed there consisted in designing probability lexicons and combining them with probabilities calculated by an nmt model. let us note that the motivation for that research was the avoidance of major translation errors, rather than domain adaptation. @xcite introduced the concept of a constrained beam search (cbs) in the task of picture captioning. the proposed algorithm forces the inclusion of selected tag words in the output. the solution makes it possible to apply, in the caption, words that were never present in the training data. the method yields the desired results provided that these out-of-vocabulary tags are based on "ground truth", such as labels obtained by reliable object detectors. the application of cbs for lexical interference in the process of neural text generation was investigated in @xcite . in the decoding phase, the beam is limited only to hypotheses, which include predefined phrases or words. the algorithm called the grid beam search (gbs) may be used for various text-generation tasks where auxiliary knowledge is expected to be incorporated into the text output. if applied to translation, the solution searches for lexical items in the source text and, in positive cases, imposes the presence of their equivalents on the beam. @xcite pointed out a danger in the cbs method resulting from the lack of correspondence between constraints and the source words they cover -the placement of the constraint translation in the output may not be correct. to avoid this undesirable effect, the authors "employ alignment information between target-side constraints and their corresponding source words." the downside of the above algorithms is their complexity: exponential (cbs) or linear (gbs) in the number of constraints. @xcite introduce an improvement of the gbs algorithm, called dynamic beam allocation (dba), which divides the fixed-size beam into "banks": sets of hypotheses that satisfy the same number of constraints. the algorithm depends only on the sentence length and the beam size, being independent of the number of constraints. @xcite notice that the use of positive (specific tokens must be present in the output) or negative (specific tokens must not be generated) constraints may be useful in rewriting tasks other than translation. rewriting (see e.g. @xcite consists in generating an output sentence in the same language and similar in meaning to the input. examples of such tasks are paraphrasing, question answering and natural language inference. @xcite regard it as crucial to focus on complexity issues to speed up the process of constrained text generation. they develop a "vectorized dba algorithm with trie representation", which speeds up the computations fivefold compared with the standard dba algorithm. further complexity improvements to constrained nmt are suggested in @xcite . they apply the idea of so-called "code-switching", which consists in injecting the target terms to the source side of the training data. the idea is similar to that of using placeholder tags to stand for rare names @xcite or named entities @xcite . the difference is that the direct translations of terms are placed in the source text instead of tags. the output text is then left untouched. the authors claim that the idea improves translation because it "does not hurt unconstrained words." we believe, however, that in some (not rare) cases the replacement of the constrained word(s) should have an impact on the choice of unconstrained words. @xcite apply the idea of "code-switching" in two different scenarios. depending on the experimental setup the target terms are placed either beside or in place of their source equivalents. the code-switching method is faster than the previous implementations based on constrained decoding (the presence of constraints need not be verified in the beam). the downside is that it requires interference with the training data. @xcite verify the efficiency of the code-switching method in an industrial sce-nario. they inject the terminology of the sap company into two translation pairs, english-german and english-russian, and provide both automatic and human evaluation. from our point of view, the english-russian case is more interesting because it addresses the problem of inflected forms of lexical constraints. there are two questions of interest to us: we could not find answers to the above questions in the paper. therefore, we investigated other solutions, such as the levenshtein transformer, introduced in @xcite . the method uses "dual policy learning", which consists in using two adversary policies during learning: when training one policy, the output from its adversary at the previous iteration is used as input. in the levenshtein transformer the two policies are deletion and insertion of a token in the generated text. the idea is supposed to resemble human intelligence, which sometimes chooses to delete an item from the text intended as output. in @xcite the levenshtein transformer was used to incorporate lexical constraints in nmt. the idea seemed more appealing to us than code-switching because it does not interfere with the training procedure. however, our initial experiments with the methodology did not succeed -the inflected forms of lexicon entries were not generated correctly. finally, we decided to carry out our experiments with the base transformer model, as introduced by @xcite , and design an algorithm that handles inflected forms of lexical constraints based on the gbs algorithm. ## experiments the purpose of our experiments was to find an efficient solution that applies lexical constraints in interactive-mode translation into a morphologically rich language. to be more specific, we aimed to develop a method that would satisfy the following conditions: ## conclusions we have examined a new approach to terminology translation into a morphologically rich language with the use of lexicons. we verified that our method, based on constrained decoding, enables the selection of accurate inflected forms of lexical constraints. the method yields an increase in the bleu metric score provided that appropriate lexical variants of terms are present in the lexicon and the input sentence context is consistent with the lexicon entries. the cost of the algorithm is a decrease in the translation speed. we proposed new metrics for the evaluation of terminology translations: placement rate, duplication rate and inflection rate. the manual evaluation results show that our method ensures terminological adequacy and consistency when translating into a morphologically rich language in domain-specific scenarios.