Neighbors Are Not Strangers: Improving Non-Autoregressive Translation under Low-Frequency Lexical Constraints (2022)

Abstract: Lexically constrained neural machine translation (NMT) draws much industrial attention for its practical usage in specific domains. However, current autoregressive approaches suffer from high latency. In this paper, we focus on non-autoregressive translation (NAT) for this problem for its efficiency advantage. We identify that current constrained NAT models, which are based on iterative editing, do not handle low-frequency constraints well. To this end, we propose a plug-in algorithm for this line of work, i.e., Aligned Constrained Training (ACT), which alleviates this problem by familiarizing the model with the source-side context of the constraints. Experiments on the general and domain datasets show that our model improves over the backbone constrained NAT model in constraint preservation and translation quality, especially for rare constraints.

## introduction despite the success of neural machine translation (nmt) @xcite @xcite , real applications usually require the precise (if not exact) translation of specific terms. one popular solution is to incorporate dictionaries of pre-defined terminologies as lexical constraints to ensure the correct translation of terms, which has been demonstrated to be effective in many areas such as domain adaptation, interactive translation, etc. previous methods on lexically constrained translation are mainly built upon autoregressive translation (at) models, imposing constraints at inference-time @xcite ## proposed approach the findings and assumptions discussed above motivate us to propose a plug-in algorithm for lexically constrained nat models, i.e., aligned constrained training (act). act is designed based on two major ideas: 1) constrained training: bridging the discrepancy between training and constrained inference; 2) alignment prompting: helping the model understand the context of the constraints. ## analysis 6.1 self-constrained translation revisited as a direct response to our motivation in this paper, we revisit the ablation study of self-constrained nat in §3.3 with the proposed act algorithm. same as before, we build self-constraints from each target sentence and sort them by frequency. as shown in figure 3(a) , different from constrained levt that suffers from drop#2 ( §3.3), act managed to handle this scenario pretty well. following the motivations given in §3.3, when constraints become rarer, act successfully breaks the trade-off with better understanding of the provided contextual information. what if the self-constraints are sorted based on tf-idf? we also study the importance of different words in a sentence via tf-idf by forcing them as constraints. as results in figure 3(b) show, we have very similar observations from frequencybased self-constraints at figure 3(a) , and the gap between levt and levt + act is even higher as tf-idf score reaches the highest. ## conclusion in this work, we propose a plug-in algorithm (act) to improve lexically constrained nonautoregressive translation, especially under lowfrequency constraints. act bridges the gap between training and constrained inference and prompts the context information of the constraints to the constrained nat model. experiments show that act improves translation quality and term preservation over the backbone nat model levenshtein transformer. further analyses show that the findings are consistent over constraints varied from frequency, tf-idf, and lengths. in the future, we will explore the application of this approach to more languages. we also encourage future research to explore a new paradigm of constrained nat methods beyond editing-based iterative nat.