Towards Precise Lexicon Integration in Neural Machine Translation (2021)

Abstract: Terminological consistency is an essential requirement for industrial translation. High-quality, hand-crafted terminologies contain entries in their nominal forms. Integrating such a terminology into machine translation is not a trivial task. The MT system must be able to disambiguate homographs on the source side and choose the correct wordform on the target side. In this work, we propose a simple but effective method for homograph disambiguation and a method of wordform selection by introducing multi-choice lexical constraints. We also propose a metric to measure the terminological consistency of the translation. Our results have a significant improvement over the current SOTA in terms of terminological consistency without any loss of the BLEU score. All the code used in this work will be published as open-source.

## motivation the importance of consistent terminology has long been discussed by translation experts @xcite @xcite @xcite . terminological standardisation is a critical task for technical and nontechnical industrial translation. patents, technical manuals, and medical instructions rely on consistent usage of technical terminology. but also nontechnical news releases, marketing texts, promotion materials, legal and financial documents need to adhere to the same terminology. @xcite correctly points out that many large companies have their own terminologies that should be used in all texts. such terminologies prescribe the correct usage of terms and provide not only a list of words that are to be used but also a list of their synonyms that should not be used by writers and translators (so-called negative terms). @xcite describe such terminology for an automotive company and its usage in detail. not adhering to these rules can be not only confusing for a reader but can also lead to serious legal and financial consequences if it is proven that damage was caused by the ambiguity of the instructions. morphologically rich languages also pose a very practical problem for terminology integration: terminological entries are provided in their nominative singular form @xcite . the sota approaches rely on the assumption that the terminological entry can be found as is in the translated text. this is not the case for slavic languages (e.g. russian), for which finding the correct wordform on the target side is a key challenge for the terminology integration. morphologically poor languages (e.g. english), on the contrary, pose a very different challenge. homographs appear in such languages not only due to polysemy and homonymy but also due to poor derivational morphology (e.g. a report vs. to report), thus, becoming a very common phenomenon. @xcite show that sota neural machine translation (nmt) fails to resolve homography efficiently. despite being a known issue, the problem has received very little attention from the research community, and we are currently not aware of any prior work that would explicitly address the problem of homographs in the context of terminology integration into machine translation. this paper focuses on the following issues: resolving homographs when the source language is morphologically poor, choosing the right wordform in the morphologically rich target language, and evaluating terminological consistency in the resulting translation. we show that our approach for homograph disambiguation and morphologically flexible lexical constraints significantly improves terminological consistency as compared to the current sota. ## related work previous work can be roughly divided into two groups: approaches that integrate lexicon during inference and approaches that integrate lexicon during training. a constrained decoding approach that has established itself as the sota in the past two years is @xcite . they proposed the dynamic beam allocation (dba) strategy, which decreased the decoding time complexity to constant time in respect to the number of lexical constraints. the proposed algorithm aims to allocate banks dynamically, prioritising the beams that satisfy the most constraints. this algorithm only allows incorporating a single wordform of a constraint, as @xcite discussed in their work. this is a notable disadvantage of this approach as it assumes an unrealistic precondition that the provided lexical constraints will be correctly inflected. this condition cannot be satisfied when translating into a morphologically rich language. on the contrary, training time approaches are more flexible in selecting the inflected forms. the sota in-training approaches tune a transformer model @xcite towards producing translations that are biased towards an external lexicon. @xcite proposed a simple way to copy target side terms into source sentences. likewise, @xcite suggested a source sentence modification method by replacing/appending target side terms using additional source factors. nevertheless, these methods are only encouraging the model to use predefined target terms, whereas constrained decoding methods are enforcing terms' usage. thus, it can be argued that in-training approaches are inferior to the constrained decoding methods in terms of straightforward terminology integration and, indeed, @xcite report the terminology usage rate 6-9% less than the constrained decoding method. to ensure the appearance of terms in the output, @xcite use placeholders with the help of morphosyntactic annotations. even though the approach is effective for choosing a correctly inflected form, it depends on the availability and performance of morphological analysers both in source and target languages. while all the aforementioned approaches have succeeded in improving the terminological consistency of translations, they essentially rely on a supervised selection of terminological entries. in other words, they assume that the homographs have already been resolved and a correct wordform is provided. once the discussed approaches are set on a trial under realistic conditions, translation quality deteriorates. word sense disambiguation is meanwhile a well-researched nlp task, and current stateof-the-art approaches can efficiently resolve homographs @xcite but due to being time-consuming, are not applicable during translation inference. ## approach the approach consists of two major steps. on the source side of the morphologically poor language, it solves the problem of frequent homographs by applying a homograph disambiguator. on the target side of the morphologically rich language, it ensures that the translated term is correctly inflected. ## evaluation all the models in our experiments were trained in the sockeye 6 toolkit @xcite . the models that incorporate 6-layer, 8-head transformer architecture are trained 50 epochs on the training corpus (10,402,336 bilingual sentences after preprocessing). we modified the sockeye toolkit to add the multi-choice lexical constraints algorithm and are going to publish the extension as an opensource. for translation quality evaluation, we report bleu score @xcite using sacrebleu (post, 2018), 7 after detokenising the translations. following @xcite , @xcite , we also report the terminology usage rate to evaluate terminological consistency. ## conclusion we have presented an approach for terminology integration into a neural machine translation from a morphologically poor into a morphologically rich language. our work makes the following contributions: we propose a solution to the problem of rich morphology in the target language by presenting multi-choice lexical constraints and show that our combined approach (mlc+bert) has a significantly 8 better f-score than all the other models.