Towards Accurate Translation via Semantically Appropriate Application of Lexical Constraints (2023)

Abstract: Lexically-constrained NMT (LNMT) aims to incorporate user-provided terminology into translations. Despite its practical advantages, existing work has not evaluated LNMT models under challenging real-world conditions. In this paper, we focus on two important but understudied issues that lie in the current evaluation process of LNMT studies. The model needs to cope with challenging lexical constraints that are “homographs” or “unseen” during training. To this end, we first design a homograph disambiguation module to differentiate the meanings of homographs. Moreover, we propose PLUMCOT which integrates contextually rich information about unseen lexical constraints from pre-trained language models and strengthens a copy mechanism of the pointer network via direct supervision of a copying score. We also release HOLLY, an evaluation benchmark for assessing the ability of model to cope with “homographic” and “unseen” lexical constraints. Experiments on HOLLY and the previous test setup show the effectiveness of our method. The effects of PLUMCOT are shown to be remarkable in “unseen” constraints. Our dataset is available at https://github.com/papago-lab/HOLLY-benchmark .

## introduction lexically-constrained neural machine translation (lnmt) is a task that aims to incorporate pre-specified words or phrases into translations @xcite @xcite @xcite @xcite . it plays a crucial role in a variety of real-world applications where it is required to translate pre-defined source terms into accurate target terms, such as domain adaptation leveraging domain-specific or user-provided terminology. for example, as shown in case a of despite its practicality, previous studies on lnmt have not evaluated their performances under challenging real-world conditions. in this paper, we focus on two important but understudied issues that lie in the current evaluation process of the previous lnmt studies. semantics of lexical constraints must be considered. in previous work, at test time, lexical constraints are automatically identified from the source sentences by going through an automatic string-matching process @xcite @xcite . for example, in case b of unseen lexical constraints need to be examined. one desideratum of lnmt systems is their robustness to handle "unseen" lexical constraints, thereby responding to random, potentially neologistic, or technical terms that users might bring up. however, translation covid-19 extra has been the top-selling imported drink in the u.s. since 1998. corona in previous studies, a significant portion of the lexical constraints is exposed during training. @xcite demonstrated the overlapped ratio of lexical constraints between the training and evaluation data (35.6% on average). @xcite also raises the issue of the high frequency of lexical constraints for test sets appearing in the training data. when lexical constraints are included in the training examples, we find that a well-optimized vanilla transformer @xcite already satisfies lexical constraints by merely learning the alignment between the source and target terms cooccurring in the parallel training sentences. 2 this presents difficulties in identifying whether the presence of target terms in the output is attributed to the learned alignment, or the proposed components in previous studies. therefore, it is important to control lexical constraints not exposed during training to examine the model's ability to cope with "unseen" lexical constraints. as a response, we present a test benchmark for evaluating the lnmt models under these two critical issues. our benchmark is specifically crafted not only to evaluate the performance of lnmt models but also to assess its ability to discern whether given lexical constraints are semantically appropriate or not. to the best of our knowledge, we are the first to release a hand-curated highquality test benchmark for lnmt. concurrently, we suggest a pipeline that allows researchers in lnmt communities to simulate realistic test conditions that consider the homograph issue and assign "unseen" lexical constraints. to this end, we propose a two-stage framework to deal with these issues. we first develop a homograph disambiguation module that determines whether lnmt models should apply a given lexi2 we observe that the vanilla transformer achieves a 66.67% copy success rate. cal constraint by evaluating its semantic appropriateness. further, we propose an lnmt model that integrates provided lexical constraints more effectively by learning when and how to apply these lexical constraints. our contributions are summarized as follows: ## holly benchmark here, we introduce holly (homograph disambiguation evaluation for lexicallly constrained nmt), a novel benchmark for evaluating lnmt systems in two circumstances; either the assigned lexical constraints are semantically appropriate or not, as illustrated in ## methodology our methodology for semantically appropriate application of lexical constraints consists of two stages. initially, we propose a homograph disambiguation module that can differentiate the semantics of lexical constraints. this module determines whether lnmt models should incorporate a lexical constraint or not. subsequently, lnmt models, plumcot in our case, perform the translation, either with or without the given lexical constraints. ## experiments on the holly benchmark in this section, we report the performance of our methodology when tested on the holly benchmark. in section 4.1, we evaluate the performance of our homograph disambiguation module in determining the semantic appropriateness of a lexical constraint. in section 4.2, we assess the performance of lnmt models using positive examples from the holly benchmark under conventional settings. subsequently, we investigate the potential advantages that the homograph disambiguation module might bring when applied to the negative examples from the holly benchmark. ## conclusions in this paper, we investigate two unexplored issues in lnmt and propose a new benchmark named holly. to address the homograph issue of the source terms, we built a homograph disambiguation module to infer the exact meaning of the source terms. we confirm that our homograph disambiguation module alleviates mistranslation led by semantically inappropriate lexical constraints. plumcot is also proposed to improve lnmt by using the rich information of plm and ameliorating its copy mechanism via direct supervision of a copying score. experiments on our holly benchmark show that plumcot significantly outperforms existing baselines in terms of bleu and csr.