Input Augmentation Improves Constrained Beam Search for Neural Machine Translation: NTT at WAT 2021 (2021)

Abstract: This paper describes our systems that were submitted to the restricted translation task at WAT 2021. In this task, the systems are required to output translated sentences that contain all given word constraints. Our system combined input augmentation and constrained beam search algorithms. Through experiments, we found that this combination significantly improves translation accuracy and can save inference time while containing all the constraints in the output. For both En->Ja and Ja->En, our systems obtained the best evaluation performances in automatic and human evaluation.

## introduction this year, we participated in the restricted translation task at wat 2021 @xcite , in which we were asked to control a model so that the translation output would contain specified terms. although the recent neural machine translation (nmt) model achieves excellent performance, controlling its output is still a challenging task. figure 1 shows an overview of the task. each sentence includes the target words (constraints) that must be contained in the output. we believe this task reflects a critical function, especially in practical applications. for example, users may want to control the translation of technical terms or proper nouns. several works have tried to control the nmt outputs, and these works can be divided into two categories: hard and soft methods. the hard lexically constrained method guarantees that all the target words are in the output. current works achieve this by modifying the beam search algorithm to find the hypothesis that contains all of the target words @xcite figure 1 : overview of the restricted translation task @xcite . the hard method guarantees all constraints are satisfied, but its translation performance is sometimes lower than the conventional nmt. this is because it requires all given target words to be contained in the decoding step, which may disrupt the model inference. the soft lexically constrained method, on the other hand, does not guarantee that all target words are contained in the output. these methods usually modify or augment the input of the nmt model and try to output the given target words without changing the decoding algorithm @xcite . its decoding speed is usually faster than the hard method, but some of the constraints may not be satisfied. our submission aims to contain all of the specified target words with high translation accuracy. to achieve this goal, we applied both input augmentation and constrained beam search algorithms. to the best of our knowledge, this is the first work that combines these two methods. through experiments, we found that this combination achieves quite high translation performance while containing all target words in the output and saving inference time. we submitted the systems to the english-to-japanese (@xmath0, and we were ranked first in both language pairs in terms of bleu scores and human evaluations. in the restricted translation task, lists of target words are provided to represent word restrictions, and systems are required to output translations that contain all of the target words in each list. here, the problem of translation with word constraints can be defined as the performance of systems in this task is evaluated through two metrics: for the final ranking, the combined score of the above metrics is calculated as follows: ## official results ## analysis figure 3 shows the example translation of the baseline and leca with lexically constrained decoding. underlines in figure 3 show the terms that match the constraints. obviously, the baseline model generated the same term repeatedly and failed to translate while all of the constraints were satisfied. the baseline model appears to struggle with generating the constraint "superconductivity single phase autotransformer." one likely reason for this is that the baseline model generated a phrase that was quite similar to the constraint in the early phase (marked with a wavy line in figure 3 ), and thus the model considered the constraint as translated. in contrast, leca+lcd successfully translated the sentence with the constraints. we believe this is because the leca model correctly gives higher scores to the constraint phrases compared to the baselines, helping to generate a sentence with constraints. figure 4 shows the bleu scores of @xmath7. as mentioned in §4.2, the beam size of base+lcd needs to be larger than 60 to successfully generate all of the constraints. in contrast, leca+lcd can generate all of the constraints and improve the translation accuracy even when their beam size is quite small. this result indicates that the output of leca is helpful for lcd to score the candidates and that leca can save inference time. susanto et al. (2020) proposed nonautoregressive nmt for lexically constrained source 分路巻線のみに補助巻線を持つ超電導単相単巻変圧器を試作した。 reference superconductivity single phase auto-transformer with auxiliary winding only at the shunt winding was produced experimentally. constraints shunt winding, auxiliary winding, superconductivity single phase auto-transformer ## conclusion this paper described the systems that were submitted to the wat 2021 restricted translation task. we submitted systems for both @xmath8, and both of our systems won the best translation accuracy as assessed by bleu, the consistency score, and human evaluations. we also confirmed that the data augmentation method makes lexically constrained decoding more effective and, furthermore, that combining data augmentation and constrained decoding significantly improves translation accuracy. matt post and david vilar. 2018. fast lexically constrained decoding with dynamic beam allocation for neural machine translation. in proceedings of the 2018 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long papers), pages 1314-1324, new orleans, louisiana. association for computational linguistics. keisuke sakaguchi and benjamin van durme. 2018. efficient online scalar annotation with bounded support. in proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: long papers), pages 208-218, melbourne, australia. association for computational linguistics. toshinori sato. 2015. neologism dictionary based on the language resources on the web for mecab.