B i- S im C ut: A Simple Strategy for Boosting Neural Machine Translation (2022)

Abstract: We introduce Bi-SimCut: a simple but effective training strategy to boost neural machine translation (NMT) performance. It consists of two procedures: bidirectional pretraining and unidirectional finetuning. Both procedures utilize SimCut, a simple regularization method that forces the consistency between the output distributions of the original and the cutoff sentence pairs. Without leveraging extra dataset via back-translation or integrating large-scale pretrained model, Bi-SimCut achieves strong translation performance across five translation benchmarks (data sizes range from 160K to 20.2M): BLEU scores of 31.16 for \texttt{en}\rightarrow\texttt{de} and 38.37 for \texttt{de}\rightarrow\texttt{en} on the IWSLT14 dataset, 30.78 for \texttt{en}\rightarrow\texttt{de} and 35.15 for \texttt{de}\rightarrow\texttt{en} on the WMT14 dataset, and 27.17 for \texttt{zh}\rightarrow\texttt{en} on the WMT17 dataset. SimCut is not a new method, but a version of Cutoff (Shen et al., 2020) simplified and adapted for NMT, and it could be considered as a perturbation-based method. Given the universality and simplicity of Bi-SimCut and SimCut, we believe they can serve as strong baselines for future NMT research.

## introduction the state of the art in machine translation has been dramatically improved over the past decade thanks to the neural machine translation (nmt) @xcite , and transformer-based models @xcite often deliver state-of-the-art (sota) translation performance with large-scale corpora @xcite . along with the development in the nmt field, consistency training @xcite has been widely adopted and shown great promise to improve nmt performance. it simply regularizes the nmt model predictions to be invariant to either small perturbations applied to the inputs @xcite and hidden states @xcite or the model randomness and variance existed in the training procedure @xcite . specifically, @xcite introduce a set of cutoff data augmentation methods and utilize jensen-shannon (js) divergence loss to force the consistency between the output distributions of the original and the cutoff augmented samples in the training procedure. despite its impressive performance, finding the proper values for the four additional hyper-parameters introduced in cutoff augmentation seems to be tedious and time-consuming if there are limited resources available, which hinders its practical value in the nmt field. in this paper, our main goal is to provide a simple, easy-to-reproduce, but tough-to-beat strategy for training nmt models. inspired by cutoff augmentation @xcite and virtual adversarial regularization @xcite for nmt, we firstly introduce a simple yet effective regularization method named simcut. technically, simcut is not a new method and can be viewed as a simplified version of token cutoff proposed in @xcite . we show that bidirectional backpropagation in kullback-leibler (kl) regularization plays a key role in improving nmt performance. we also regard simcut as a perturbation-based method and discuss its robustness to the noisy inputs. at last, motivated by bidirectional training @xcite in nmt, we present bi-simcut, a two-stage training strategy consisting of bidirectional pretraining and unidirectional finetuning equipped with simcut regularization. the contributions of this paper can be summarized as follows: ## datasets and baseline settings in this section, we describe the datasets used in experiments as well as the model configurations. for fair comparisons, we keep our experimental settings consistent with previous works. datasets we initially consider a low-resource (iwslt14 en↔de) scenario and then show further experiments in standard (wmt14 en↔de) and high (wmt17 @xmath0. the detailed information of the datasets are summarized in settings we implement our approach on top of the transformer @xcite . we apply a transformer with 6 encoder and decoder layers, 4 attention heads, embedding size 512, and ffn layer dimension 1024. we apply cross-entropy loss with label smoothing rate 0.1 and set max tokens per batch to be 4096. we use adam optimizer with beta (0.9, 0.98), 4000 warmup updates, and inverse square root learning rate scheduler with initial learning rates 5e -4 . we use dropout rate 0.3 and beam search decoding with beam size 5 and length penalty 1.0. we apply the same training configurations in both pretraining and finetuning stages which will be discussed in the following sections. we use multi-bleu.pl foot_1 for bleu @xcite evaluation. we train all models until convergence on a single nvidia tesla v100 gpu. all reported bleu scores are from a single model. for all the experiments below, we select the saved model state with the best validation performance. ## bi-simcut in this section, we formally propose bidirectional pretrain and unidirectional finetune with simple cutoff regularization (bi-simcut), a simple but effective training strategy that can greatly enhance the generalization of the nmt model. bi-simcut consists of a simple cutoff regularization and a two-phase pretraining and finetuning strategy. we introduce the details of each part below. ## standard resource scenario we here investigate the performance of bi-simcut on the larger translation benchmark compared with the iwslt14 benchmark. ## high resource scenario to investigate the performance of bi-simcut on the distant language pairs which naturally do not share dictionaries, we here discuss the effectiveness of bi-simcut on the chinese-english translation task. ## related work adversarial perturbation it is well known that neural networks are sensitive to noisy inputs, and adversarial perturbations are firstly discussed in the filed of image processing @xcite . simcut could be regarded as a perturbation-based method for the robustness research. in the field of natural language processing, miyato et al. (2017) consider adversarial perturbations in the embedding space and show its effectiveness on the text classification tasks. for the nmt tasks, sato et al. (2019) and wang et al. (2019) apply adversarial perturbations in the embedding space during training of the encoderdecoder nmt model. cheng et al. (2019) leverage adversarial perturbations and generate adversarial consistency training besides perturbationbased methods, our approach also highly relates to a few works of model-level and data-level consistency training in the nmt field. among them, the most representative methods are r-drop @xcite and cutoff @xcite . r-drop studies the intrinsic randomness in the nmt model and regularizes the nmt model by utilizing the output consistency between two dropout sub-models with the same inputs. cutoff considers consistency training from a data perspective by regularizing the inconsistency between the original sentence pair and the augmented samples with part of the information within the input sentence pair being dropped. note that cutoff takes the dropout submodels into account during the training procedure as well. we want to emphasize that simcut is not a new method, but a version of cutoff simplified and adapted for the nmt tasks. ## conclusion in this paper, we propose bi-simcut: a simple but effective two-stage training strategy to improve nmt performance. bi-simcut consists of bidirectional pretraining and unidirectional finetuning procedures equipped with simcut regularization for improving the generality of the nmt model. experiments on low (iwslt14 en↔de), standard (wmt14 en↔de), and high (wmt17 @xmath2. given the universality and simplicity of bi-simcut and sim-cut, we believe: 1) simcut could be regarded as a perturbation-based method, and it could be used as a strong baseline for the robustness research. 2) bi-simcut outperforms many complicated methods which incorporate large-scaled pretrained models or sophisticated mechanisms, and it could be used as a strong baseline for future nmt research. we hope researchers of perturbations and nmt could use simcut and bi-simcut as strong baselines to make the usefulness and effectiveness of their proposed methods clear. for future work, we will explore the effectiveness of simcut and bi-simcut on more sequence learning tasks, such as multilingual machine translation, domain adaptation, text classification, natural language understanding, etc.