Integrating Vectorized Lexical Constraints for Neural Machine Translation (2022)

Abstract: Lexically constrained neural machine translation (NMT), which controls the generation of NMT models with pre-specified constraints, is important in many practical scenarios. Due to the representation gap between discrete constraints and continuous vectors in NMT models, most existing works choose to construct synthetic data or modify the decoding algorithm to impose lexical constraints, treating the NMT model as a black box. In this work, we propose to open this black box by directly integrating the constraints into NMT models. Specifically, we vectorize source and target constraints into continuous keys and values, which can be utilized by the attention modules of NMT models. The proposed integration method is based on the assumption that the correspondence between keys and values in attention modules is naturally suitable for modeling constraint pairs. Experimental results show that our method consistently outperforms several representative baselines on four language pairs, demonstrating the superiority of integrating vectorized lexical constraints.

## introduction controlling the lexical choice of the translation is important in a wide range of settings, such as interactive machine translation @xcite , entity translation @xcite , and translation in safety-critical domains @xcite . however, different from the case of statistical machine translation @xcite , it is non-trivial to directly integrate discrete lexical constraints into neural machine translation (nmt) models @xcite , whose hidden states are all continuous vectors that are difficult for humans to understand. in accordance with this problem, one branch of studies directs its attention to designing adwe omit queries for simplicity. blue and green squares denote the continuous representation of the source sentence and the constraints, respectively. the provided constraints are "@xmath0. to make nmt models better learn from and cope with lexical constraints, we propose to lever-age attention modules @xcite to explicitly integrate vectorized lexical constraints. as illustrated in figure 1 , we use vectorized source constraints as additional keys and vectorized target constraints as additional values. intuitively, the additional keys are used to estimate the relevance between the current query and the source phrases while the additional values are used to integrate the information of the target phrases. in this way, each revised attention is aware of the guidance to translate which source phrase into what target phrase. experiments show that our method can significantly improve the ability of nmt models to translate with constraints, indicating that the correspondence between attention keys and values is suitable for modeling constraint pairs. inspired by recent progress in controlled text generation @xcite , we also introduce a plug-in to the output layer that can further improve the success rate of generating constrained tokens. we conduct experiments on four language pairs and find that our model can consistently outperform several representative baselines. ## neural machine translation training the goal of machine translation is to translate a source-language sentence @xmath1. . . x |x| into a target-language sentence @xmath2. . . y |y| . we use p (y|x; θ) to denote an nmt model @xcite parameterized by θ. modern nmt models are usually trained by maximum likelihood estimation @xcite , where the log-likelihood is defined as inference the inference of nmt models can be divided into two sub-processes: these two sub-processes are performed alternatively until reaching the maximum length or generating the end-of-sentence token. ## approach this section explains how we integrate lexical constraints into nmt models. section 3.1 illustrates the way we encode discrete constraints into continuous vectors, section 3.2 details how we integrate the vectorized constraints into nmt models, and section 3.3 describes our training strategy. ## conclusion in this work, we propose to vectorize and integrate lexical constraints into nmt models. our basic idea is to use the correspondence between keys and values in attention modules to model constraint pairs. experiments show that our approach can outperform several representative baselines across four different translation directions. in the future, we plan to vectorize other attributes, such as the topic, the style, and the sentiment, to better control the generation of nmt models.