Lexically Constrained Neural Machine Translation with L evenshtein Transformer (2020)

Abstract: This paper proposes a simple and effective algorithm for incorporating lexical constraints in neural machine translation. Previous work either required re-training existing models with the lexical constraints or incorporating them during beam search decoding with significantly higher computational overheads. Leveraging the flexibility and speed of a recently proposed Levenshtein Transformer model (Gu et al., 2019), our method injects terminology constraints at inference time without any impact on decoding speed. Our method does not require any modification to the training procedure and can be easily applied at runtime with custom dictionaries. Experiments on English-German WMT datasets show that our approach improves an unconstrained baseline and previous approaches.

## introduction neural machine translation (nmt) systems can generate higher-quality translations than phrasebased mt systems, but they come at the cost of losing control over how translations are generated. without the explicit link between the source and the target vocabulary, enforcing specific terminological translation in domain-specific settings becomes painfully difficult for nmt systems. consider an example where we have a chinese-english nmt system trained for the e-commerce domain, and there is no prior knowledge of the brand name "红 米" in the training data, the system would translate the input term literally as "red (红) rice (米)" instead of "redmi". in such scenarios, machine translation users often maintain in-domain dictionaries to ensure that specific information is translated accurately and consistently. a line of previous work that tried to address this problem required re-training the nmt models with lexical constraints, either by a placeholder mecha-nism @xcite or via code-mixed training @xcite . however, they do not reliably guarantee the presence of the constraints at test time. another approach focused on constrained beam search decoding @xcite @xcite . although the latter approach has higher control over the target constraint terms, they significantly slow down the decoding. different from the existing line of work, we invoke lexical constraints using a non-autoregressive approach. 1 to do this, we use levenshtein transformer (levt) @xcite , an edit-based generation model that performs deletion and insertion operations during inference iteratively. levt achieves substantially higher inference speed compared to beam search without affecting quality. we add a constraint insertion step in levt decoding to seamlessly decode the target language sequence while adhering to specific lexical constraints, achieving the same speed as standard levt decoding. ## related work previous approaches integrated lexical constraints in nmt either via constrained training or decoding. @xcite replaced entities with placeholders that remained unchanged during translation and placed them back in a post-processing step. @xcite trained a transformer @xcite model by augmenting the data to include the constraint target phrases in the source sentence. @xcite proposed a similar idea and additionally used factored training. other approaches proposed enforcement of lexical constraints during inference with various improvements to constraint-aware beam search, such as grid beam search @xcite , dynamic beam allocation @xcite , and its optimized vectorized version @xcite . @xcite built finite-state acceptors to integrate constraints in a multi-stack decoder. these lexically-constrained decoding approaches rely on autoregressive inference that generates one target token at a time, which makes it difficult to parallelize the decoder and monotonically increases decoding time. while being mostly effective at forcing the inclusion of pre-specified terms in the output, these approaches further slow down the beam search process. @xcite reported 3× slow down compared to standard beam search. non-autoregressive neural machine translation (nat) @xcite attempts to move away from the conventional autoregressive decoding. such a direction enables parallelization during sequence generation that results in lower inference latency. recent nat approaches treat inference as an iterative refinement process, first proposed by @xcite . following this direction, it is intuitive to perform decoding using "edit" operations, such as insertion @xcite or both insertion and deletion (levt, @xcite ). the levt model has been shown to outperform existing refinement-based models, such as @xcite and performs comparably to autoregressive transformer models. our method integrates lexical constraints in nat decoding utilizing the flexibility, speed, and performance of levt. ## levenshtein transformer levenshtein transformer (levt) @xcite has an encoder-decoder framework based on transformer architecture @xcite with multi-headed self-attention and feed-forward networks. unlike token generation in a typical transformer model, levt decoder models a markov decision process (mdp) that iteratively refines the generated tokens by alternating between the insertion and deletion operations. after embedding the source input through a transformer encoder block, the levt decoder follows the mdp formulation for each sequence at the k-th iteration y @xmath0, y 2 , ..., y n ), where y 1 and y n are the start (<@xmath1. the decoder then generates y k+1 by performing deletion and insertion operations via three classifiers that run sequentially: constraint insertion placeholder classifier token @xmath2. </@xmath3. deletion classifier, which predicts for each token position whether they should be "kept" or "deleted", 2. placeholder classifier, which predicts the number of tokens to be inserted between every two consecutive tokens and then inserts the corresponding number of placeholder [plh] tokens, 3. token classifier, which predicts for each [plh] token an actual target token. each prediction is conditioned on the source text and the current target text. the same transformer decoder block is shared among the three classifiers. decoding stops when the current target text does not change, or a maximum number of refinement iterations has been reached. the levt model is trained using sequence-level knowledge distillation @xcite ) from a transformer teacher whose beam search output is used as ground truth during training. we refer the readers to @xcite for a detailed description of the levt model and training routine. ## incorporating lexical constraints for sequence generation, the levt decoder typically starts the first iteration of the decoding process with only the sentence boundary tokens y 0 = <@xmath4. to incorporate lexical constraints, we populate the y 0 sequence before the first deletion operation with the target constraints, as shown in figure 1 . the initial target sequence will pass through the deletion, placeholder, and insertion classifiers sequentially, and the modified sequence will be refined for several iterations. the decoding steps are explained in detail below. constraint insertion more formally, given a list of m target constraints c 1 , c 2 , ..., c m , where each constraint c i is possibly a multi-token phrase deletion operation next, y 0 passes through the deletion classifier to decide which w i j token to remove. if the deletion operation is allowed on the constraint tokens, the presence of each constraint in the final output is not guaranteed, especially when the supplied constraints are out of context for the decoder. to mitigate this problem, we optionally disallow the deletion operation on the constraint tokens by introducing a constraint mask to indicate the positions of constraint tokens in the sequence. we forcefully set the deletion classifier prediction for all positions in this mask to "keep". the positions in this mask are re-computed accordingly after each deletion and insertion operation. insertion operation finally, the y 0 passes through the placeholder classifier to predict the number of tokens to be inserted and generate the corresponding number of [plh] tokens and the token classifier assigns an actual target token for every [plh] token. each constraint may contain multiple tokens, and the [plh] tokens may be inserted between the tokens from the same constraint. to prevent this from happening and to keep each constraint intact, we optionally prohibit inserting [plh] within a multi-token constraint by constraining 0 to the number of such placeholders. in figure 1 , our constraint insertion is executed at the first pass, and subsequent iterations start from deletion (indicated by a loop in the figure). we note that this step happens only at inference; during training, the original levt training routine is carried out without the constraint insertion. ## experiments we extend the fairseq 2 @xcite form lexically-constrained decoding. all transformer blocks in our levt model follow the base configuration that contains 6 layers with 8 attention heads each, with a model size d @xmath5; the source and target embeddings share the same vocabulary. the levt model is trained using knowledge distillation routine using transformer base output released by @xcite . we leave more experimental details in the appendix. ## conclusion we proposed a non-autoregressive decoding approach to integrate lexical constraints for nmt. our constraint insertion step is simple and we have empirically validated its effectiveness. the approach demonstrated control over constraint terms in target translations while being able to decode as fast as a baseline levenshtein transformer model, which achieves significantly higher decoding speed than traditional beam search. 6 in addition to the terminological lexical constraints discussed in this work, future work can potentially modify insertion or selection operations to handle target translations of multiple forms; this can potentially disambiguate the morphological variants of the lexical constraints.