NICT ’s Neural Machine Translation Systems for the WAT 21 Restricted Translation Task (2021)

Abstract: This paper describes our system (Team ID: nictrb) for participating in the WAT’21 restricted machine translation task. In our submitted system, we designed a new training approach for restricted machine translation. By sampling from the translation target, we can solve the problem that ordinary training data does not have a restricted vocabulary. With the further help of constrained decoding in the inference phase, we achieved better results than the baseline, confirming the effectiveness of our solution. In addition, we also tried the vanilla and sparse Transformer as the backbone network of the model, as well as model ensembling, which further improved the final translation performance.

## introduction the performance of machine translation has been greatly improved since it entered the era of neural machine translation (nmt) @xcite @xcite . different from traditional statistical machine translation (smt) @xcite , nmt models are trained end-to-end with contextualized representations to alleviate the locality problem and dense representations to mitigate the sparsity issue. the incorporation of novel structures such as cnn @xcite and transformer @xcite into nmt has brought the performance one step closer to practical translation. though nmt can more effectively exploit large parallel corpora, the performance is still insufficient to meet the requirements in some special translation scenarios. the end-to-end nmt models remove many approaches in the smt paradigm for manually guiding the translation process. one attractiveness of the smt method is that it provides explicit control over translation output, which is effective in a variety of translation settings, including interactive machine translation @xcite and domain adaptation @xcite , which is also crucial for the practical application of nmt. since there is still a need for manual interventions for the new nmt paradigm, much effort is spent in studying how to incorporate this explicit control into the end-to-end neural translation @xcite . among these efforts, constrained decoding (cd) has gained a lot of attention in this research field, which is a modification to commonly adopted beam search in ordinary nmt models. @xcite proposed grid beam search, which expands beam search to include pre-specified lexical constraints. @xcite used constrained beam search to force the inclusion of restricted words in the output, and employed fixed pre-trained word embeddings to facilitate vocabulary expansion to unseen words in training. while these works accomplish the goal of explicit translation control, the time complexity of their decoding algorithm and resultant decoding speed falls short of the expectations. the complexity of grid beam search and constrained beam search is linear and exponential to the number of constraints, respectively. these algorithms are thus too inefficient to be practical for large-scale use. to alleviate the shortcomings in constrained decoding, @xcite proposed a new constrained decoding algorithm with a claimed complexity of o(1) in the number of constraints -dynamic beam allocation which allocates the slots in a fixed-size beam. however, their approach still processes sentence constraints sequentially rather than batch processing, limiting the gpu's parallel processing capabilities. based on @xcite , a vectorized dynamic beam allocation approach was proposed in @xcite , which which vector-izes the dynamic beam allocation for batching and thus leading to improvement in throughput with parallelization. based on @xcite , @xcite proposed a vectorized dynamic beam allocation approach, which vectorizes the dynamic beam allocation for batching, resulting in increased throughput with parallelization. constrained decoding is a very general method for incorporating additional translation knowledge into the output without modifying the model parameters or training data. however, the model's prediction distribution can be skewed during the decoding process with hard constraints, resulting in poor translation results. when the model is exposed to the restricted translation paradigm during training, the gap between training and inference can be reduced, potentially improving performance. therefore, in this paper, we propose a training method of sampled constraints as concentration (scc). in this method, training data is the same as the ordinary nmt; only minor modifications on the loss calculation are required to adapt the model to restricted translation. in our submission to wat'21 @xcite restricted translation task, we chose transformer @xcite as our baseline because of its high performance and scalability. although there are some variants, our previous experiments have shown there are not too many approaches that can be both concise and effective. at the same time, though multi-head self-attention in transformer can model extremely long dependencies, deep layer attention tends to overconcentrate on a single token, resulting in inadequate use of local information and difficulty representing long sequences. to address this disadvantage, we employ the prime transformer @xcite with a multi-scale sparse attention mechanism as a second baseline. the models in the two architectures are ensembled to improve the overall results. our final system uses a combination of the scc training method and the constrained decoding of @xcite , which makes our system leverages soft constrained (inside the model) and benefit from hard restrictions (external decoding). ## our system in this section, we describe the methods used in our system in detail. our system is made up of four components: the transformer model, the sparse transformer model, the scc training ap-proach, and the constrained decoding algorithm. in translation, given the source input sequence @xmath0, w 2 , ..., w m }, its target translation is @xmath1, y 2 , ..., y n }, the parameter of the nmt model is θ, then the probability form of the translation process can be written as: ## results ## conclusion in this paper, we present our nmt systems for wat21 restricted translation shared tasks in english ↔ english. by integrating the following techniques: sparse transformer, sampled constraints as concentration, and lexically constrained decoding, our final system achieves substantial improvement over baseline systems which show the effectiveness of our approaches.