Restricted or Not: A General Training Framework for Neural Machine Translation (2022)

Abstract: Restricted machine translation incorporates human prior knowledge into translation. It restricts the flexibility of the translation to satisfy the demands of translation in specific scenarios. Existing work typically imposes constraints on beam search decoding. Although this can satisfy the requirements overall, it usually requires a larger beam size and far longer decoding time than unrestricted translation, which limits the concurrent processing ability of the translation model in deployment, and thus its practicality. In this paper, we propose a general training framework that allows a model to simultaneously support both unrestricted and restricted translation by adopting an additional auxiliary training process without constraining the decoding process. This maintains the benefits of restricted translation but greatly reduces the extra time overhead of constrained decoding, thus improving its practicality. The effectiveness of our proposed training framework is demonstrated by experiments on both original (WAT21 En \leftrightarrow Ja) and simulated (WMT14 En \rightarrow De and En \rightarrow Fr) restricted translation benchmarks.

## introduction neural machine translation (nmt) has recently entered use because of rapid improvements in its performance @xcite @xcite . the translation mechanism of an nmt model is a black box because it is a special deep neural network model, which means that translation generation is uncontrollable @xcite @xcite . although uncontrollable (or unguaranteed) translation can satisfy basic requirements, it is unacceptable in some formal scenarios, particularly for key numbers, time, and proper nouns. to address this concern, the restricted translation task has been proposed @xcite @xcite @xcite . this restricts translation by forcing the inclusion of prespecified words and phrases in the generation output, which enables explicit control over the system output. lexically constrained (or guided) decoding (cd) @xcite , a modification of beam search, has commonly been used in recent restricted translation studies. although cd is a reasonable option for restricted translation, its slow decoding limits the practicality of restricted translation. therefore, we propose a novel training framework for restricted translation that requires only minor changes to the ordinary translation model, to address the inconvenience of the decoding time overhead caused by additional constraints. in this framework, restricted machine translation is achieved by the model structure instead of the cd. specifically, we perform translation in two modes in the training framework: end-to-end translation and restricted translation, and reuse the selfattention and cross-attention in the decoder of the translation model. to make the restricted translation training mode adapt to the training data situation with only parallel sentences available, we propose the sampled constraints as concentration (scc) training approach. in this approach, we sample the target sequence to simulate the constraint words and impose additional penalties on the loss of these sampled words. because the restricted translation is embedded with the model structure and training objective in the translation model trained with our framework, restricted translation is performed without cd. consequently, the inference speed is substantially increased, which greatly improves the practicality of restricted translation. experimental re-sults show that our end-to-end translation model can achieve approximately the same performance as the end-to-end translation baseline; moreover, although it only requires unconstrained decoding, it can achieve performance competitive or even superior with that of the baseline with cd. ## our training framework our training framework comprises two training subprocesses: end-to-end translation and restricted translation. recent restricted translation studies have focused mainly on the decoding phase, but we set out to integrate restricted translation into the training phase, which makes the motivation of our work completely different from that of previous studies. our implementation is based on the existing mainstream transformer nmt baseline; however, because the training method is independent of the baseline, our training framework can easily be generalized to other nmt models and language generation tasks. due to space limitations, please refer to appendix a.2 for training details. ## empirical evaluation and analysis our method was evaluated on the aspec @xcite en↔ja benchmark and the wmt14 @xmath0. the constrained words for the aspec en↔ja test set were provided by the wat21 restricted translation shared task and, for wmt14 @xmath1, we followed previous work by adopting random sampling to extract the constraints. we chose two typical transformer model settings as our baseline: transformer-base and transformer-big, both of which are consistent with @xcite . during training, we set α = 0.15 and γ = 1.0. for a fair comparison, the beam size was set to 10 and the batch size was fixed at 64. we reported multibleu scores in our experiments and calculated them using the moses script. for en, de, and fr, we use the default tokenizer provided by moses @xcite , and for ja, we adopted mecab foot_0 for word segmentation. in the evaluation of wat21 en↔ja, we also reported a consistency metric -the exact match (em) score -according to the wat21 official instructions. this score is the ratio of sentences in the whole corpus that exactly match the given constraints. for the em score evaluation, we use lowercase hypotheses and constraints, then use character-level sequence matching (including whitespaces) for each constraint in en, while for ja, we use characterlevel sequence matching (including whitespaces) for each constraint without preprocessing. please see appendix a.3 for more preparation details. ## related work lexically constrained (or guided) decoding (cd), a modification of beam search, has commonly been used in recent restricted translation studies. specifically, some prespecified words or phrases are forced in translation choice. however, although these approaches can theoretically achieve the goal of restricted translation, existing methods are very expensive in terms of decoding time; this limits the practicality of cd. starting from @xcite , in which cd was introduced and utilized in nmt, attempts have been made to reduce the time overhead of cd by the use of dynamic beam allocation. although the time complexity is formally consistent with that of general beam search, it remains too inefficient to be used on a large scale @xcite . @xcite further extended cd and improved the throughput of restricted translation systems by using batching in vectorized dynamic beam allocation. although these efforts have improved the practicality of restricted translation, the decoding speed is still far less than that of ordinary decoding. ## conclusion in this paper, we proposed novel training and decoding methods for restricted translation that do not use cd. furthermore, we established a general training framework. with our framework, endto-end translation and restricted translation can be implemented in the same model. compared to using cd in the end-to-end translation model, we achieved better translation results, as well as smaller beam size and consistently higher decoding speed. we evaluated the framework on multiple benchmarks, and demonstrated the performance advantages of restricted translation. using our training framework and decoding method, restricted translation can overcome the limitation of its extremely slow decoding speed and become practical.