Bi-SimCut: A Simple Strategy for Boosting Neural Machine Translation
Abstract
We introduce Bi-SimCut: a simple but effective training strategy to boost neural machine translation (NMT) performance.
It consists of two procedures: bidirectional pretraining and unidirectional finetuning.
Both procedures utilize SimCut, a simple regularization method that forces the consistency between the output distributions of the original and the cutoff sentence pairs.
Without leveraging extra dataset via back-translation or integrating large-scale pretrained model, Bi-SimCut achieves strong translation performance across five translation benchmarks (data sizes range from 160K to 20.2M): BLEU scores of 31 .
16 for en → de and 38 .
37 for de → en on the IWSLT dataset, 30 . 78 for en → de and 35 .
15 for de → en on the WMT14 dataset, and 27 .for
zh → en on the WMT17 dataset.
SimCut is not a new method, but a version of Cutoff (Shen et al., 2020) simplified and adapted for NMT, and it could be considered as a perturbation-based method.
Given the universality and simplicity of SimCut and Bi-SimCut, we believe they can serve as strong baselines for future NMT research.
Introduction
The state of the art in machine translation has been dramatically improved over the past decade thanks to the neural machine translation (NMT)
(Wu et al., 2016), and Transformer-based models (Vaswani et al., 2017) often deliver state-of-the-art (SOTA) translation performance with large-scale corpora (Ott et al., 2018).
Along with the development in the NMT field, consistency training (Bachman et al., 2014) has been widely adopted and shown great promise to improve NMT performance.
It simply regularizes the NMT model predictions to be invariant to either small perturbations applied to the inputs (Sato et al., 2019;
Shen et al., 2020) and hidden states (Chen et al., 2021) or the model randomness and variance existed in the training procedure (Liang et al., 2021).
Specifically, Shen et al.
(2020) introduce a set of cutoff data augmentation methods and utilize
Jensen-Shannon (JS) divergence loss to force the consistency between the output distributions of the original and the cutoff augmented samples in the training procedure.
Despite its impressive performance, finding the proper values for the four additional hyper-parameters introduced in cutoff augmentation seems to be tedious and time-consuming if there are limited resources available, which hinders its practical value in the NMT field.
In this paper, our main goal is to provide a simple, easy-to-reproduce, but tough-to-beat strategy for training NMT models.
Inspired by cutoff augmentation (Shen et al., 2020) and virtual adversarial regularization (Sato et al., 2019) for NMT, we firstly introduce a simple yet effective regularization method named SimCut.
Technically, SimCut is not a new method and can be viewed as a simplified version of Token Cutoff proposed in Shen et al.
We show that bidirectional backpropagation in Kullback-Leibler (KL) regularization plays a key role in improving NMT performance.
We also regard SimCut as a perturbation-based method and discuss its robustness to the noisy inputs.
At last, motivated by bidirectional training (Ding et al., 2021) in NMT, we present Bi-SimCut, a two-stage training strategy consisting of bidirectional pretraining and unidirectional finetuning equipped with SimCut regularization.
The contributions of this paper can be summarized as follows:
• We propose a simple but effective regularization method,
SimCut, for improving the generalization of NMT models.
SimCut could be regarded as a perturbation-based method and serves as a strong baseline for the approaches of robustness.
We also show the compatibility of SimCut with the pretrained language models such as mBART (Liu et al., 2020).
• We propose Bi-SimCut, a training strategy for NMT that consists of bidirectional pretraining and unidirectional finetuning with SimCut regularization.
• Our experimental results show that NMT training with Bi-SimCut achieves significant improvements over the Transformer model on five translation benchmarks (data sizes range from 160K to 20.2M), and outperforms the current SOTA method BiBERT (Xu et al., 2021) on several benchmarks.
Background 2. Neural Machine Translation
The NMT model refers to a neural network with an encoder-decoder architecture, which receives a sentence as input and returns a corresponding translated sentence as output.
Assume x = x 1 , ..., x I and y =
y 1 , ..., y J that correspond to the source and target sentences with lengths I and J respectively.
Note that y J denotes the special end-of-sentence symbol ⟨ eos ⟩ .
The encoder first maps a source sentence x into a sequence of word embeddings e ( x ) =
e ( x 1 ) , ..., e ( x I ) , where e ( x ) ∈ R d × I , and d is the embedding dimension.
The word embeddings are then encoded to the corresponding hidden representations h .
Similarly, the decoder maps a shifted copy of the target sentence y , i.e., ⟨ bos ⟩ , y 1 , ..., y J − 1 , into a sequence of word embeddings e ( y ) =
e ( ⟨ bos ⟩ ) , e ( y 1 ) , ..., e ( y J − 1 ) , where ⟨ bos ⟩ denotes a special beginning-of-sentence symbol, and e ( y ) ∈
R d × J .
The decoder then acts as a conditional language model that operates on the word embeddings e ( y ) and the hidden representations h learned by the encoder.
Given a parallel corpus S = { x i , y i } |S| i =1 , the standard training objective is to minimize the empirical risk:
L ce ( θ ) =
E (
[Equation]
(1) where ℓ denotes the cross-entropy loss, θ is a set of model parameters, f ( x , y ; θ ) is a sequence of probability predictions, i.e., f j ( x , y ; θ )
=
P ( y | x , y <
j ; θ ) , (2) and ¨ y is a sequence of one-hot label vectors for y .
2. Cutoff Augmentation
Shen et al.
(2020) introduce a set of cutoff methods which augments the training by creating the partial views of the original sentence pairs and propose Token Cutoff for the machine translation task.
Given a sentence pair ( x , y ) , N cutoff samples { x i cut , y i cut } Ni =1 are constructed by randomly setting the word embeddings of x 1 , ..., x I and y 1 , ..., y J to be zero with a cutoff probability p cut .
For each sentence pair, the training objective of Token Cutoff is then defined as:
[Equation]
where
L
[Equation]
in which KL ( ·∥· ) denotes the Kullback-Leibler (KL) divergence of two distributions, and α and β are the scalar hyper-parameters that balance L ce ( θ ) ,
L cut ( θ ) and L kl ( θ ) .
Datasets and Baseline Settings
In this section, we describe the datasets used in experiments as well as the model configurations.
For fair comparisons, we keep our experimental settings consistent with previous works
. Datasets
We initially consider a low-resource (IWSLT14 en ↔ de ) scenario and then show further experiments in standard (WMT14 en ↔ de ) and high (WMT17 zh → en ) resource scenarios in Sections 5 and 6.
The detailed information of the datasets are summarized in Table 1.
We here conduct experiments on the IWSLT14 EnglishGerman dataset 1 , which has 160K parallel bilingual
IWSLT
WMT en ↔ de en ↔ de zh → en train   valid
[Table]
[Table]
sentence pairs.
Following the common practice, we lowercase all words in the dataset.
We build a shared dictionary with 10K byte-pair-encoding (BPE) (Sennrich et al., 2016) types.
Settings
We implement our approach on top of the Transformer (Vaswani et al., 2017).
We apply a Transformer with 6 encoder and decoder layers, 4 attention heads, embedding size 512, and FFN layer dimension 1024.
We apply cross-entropy loss with label smoothing rate 0 .
1 and set max tokens per batch to be 4096 .
We use Adam optimizer with Beta (0 . 9 , 0 . 98) , 4000 warmup updates, and inverse square root learning rate scheduler with initial learning rates 5 e − 4 .
We use dropout rate 0 .and
beam search decoding with beam size 5 and length penalty 1 . 0 .
We apply the same training configurations in both pretraining and finetuning stages which will be discussed in the following sections.
We use multi-bleu.pl 2 for BLEU (Papineni et al., 2002) evaluation.
We train all models until convergence on a single NVIDIA Tesla V100 GPU.
All reported BLEU scores are from a single model.
For all the experiments below, we select the saved model state with the best validation performance.
Bi-SimCut
In this section, we formally propose Bidirectional
Pretrain and Unidirectional Finetune with Simple
Cutoff Regularization (Bi-SimCut), a simple but effective training strategy that can greatly enhance the generalization of the NMT model.
Bi-SimCut consists of a simple cutoff regularization and a two-phase pretraining and finetuning strategy.
We introduce the details of each part below.
SimCut: A
Simple Cutoff Regularization for NMT
Despite the impressive performance reported in
Shen et al.
(2020), finding the proper hyperparameters ( p cut , α, β, N ) in Token Cutoff seems to be tedious and time-consuming if there are limited resources available, which hinders its practical value in the NMT community.
To reduce the burden in hyper-parameter searching, we propose
SimCut, a simple regularization method that forces the consistency between the output distributions of the original sentence pairs and the cutoff samples.
Our problem formulation is motivated by Virtual Adversarial Training (VAT), where Sato et al.
(2019) introduces a KL-based adversarial regularization that forces the output distribution of the samples with adversarial perturbations δ x ∈ R d
×
I and δ y ∈ R
d × J to be consistent with that of the original samples:
[Equation]
Instead of generating perturbed samples by gradient-based adversarial methods, for each sentence pair ( x , y ) , we only generate one cutoff sample ( x cut , y cut ) by following the same cutoff strategy used in Token Cutoff.
For each sentence pair, the training objective of SimCut is defined as:
[Equation]
where
[Equation]
There are only two hyper-parameters α and p cut in SimCut, which greatly simplifies the hyperparameter searching step in Token Cutoff.
Note that VAT only allows the gradient to be backpropagated through the right-hand side of the KL divergence term, while the gradient is designed to be backpropagated through both sides of the KL regularization in SimCut.
We can see that the constraints introduced by L cut ( θ ) and L kl ( θ ) in (3) still implicitly hold in (8):
• L cut ( θ ) in Token Cutoff is designed to guarantee that the output of the cutoff sample should close to the ground-truth to some extent.
In SimCut, L ce ( θ ) requires the outputs of the original sample close to the ground-truth, and L simkl ( θ ) requires the output distributions of the cutoff sample close to that of the original sample.
The constraint introduced by L cut ( θ ) then implicitly holds.
• L kl ( θ ) in Token Cutoff is designed to guarantee that the output distributions of the original sample and N different cutoff samples
[Table]
should be consistent with each other.
In SimCut, L simkl ( θ ) guarantees the consistency between the output distributions of the original and cutoff samples.
Even though SimCut only generates one cutoff sample at each time, different cutoff samples of the same sentence pair will be considered in different training epochs.
Such constraint raised by L kl ( θ ) still implicitly holds.
4. Analysis on SimCut 4.2. How Does the Simplification Affect Performance?
We here investigate whether our simplification on
Token Cutoff hurts its performance on machine translation tasks.
We compare SimCut with Token
Cutoff, VAT, and R-Drop (Liang et al., 2021), a strong regularization baseline that forces the output distributions of different sub-models generated by dropout to be consistent with each other.
Table 2 shows that SimCut achieves superior or comparable performance over VAT, R-Drop, and Token
Cutoff, which clearly shows the effectiveness of our method.
To further compare SimCut with other strong baselines in terms of training cost, we summarize the validation BLEU score along the training time on IWSLT14 de → en translation task in
Table 3.
From the table, we can see that the BLEU score of SimCut continuously increases in the first 1500 minutes.
The results on VAT are consistent with the previous studies on adversarial overfitting, i.e., virtual adversarial training easily suffering from overfitting (Rice et al., 2020).
Though
SimCut needs more training time to converge, the final NMT model is much better than the baseline.
For the detailed training cost for each epoch, Token
Cutoff costs about 148 seconds per epoch, while
SimCut costs about 128 seconds per epoch.
Note that the training cost of Token Cutoff is greatly influenced by the hyper-parameter N .
We set N to be 1 in our experiments.
With the increasing of
N , the training time of Token Cutoff will be much longer.
Due to the tedious and time-consuming hyper-parameter searching in Token Cutoff, we will not include its results in the following sections and show the results of SimCut directly.
4.2. How Does the Bidirectional Backpropagation Affect Performance?
Even though the problem formulation of SimCut is similar to that of VAT, one key difference is that the gradients are allowed to be backpropagated bidirectionally in the KL regularization in SimCut.
We here investigate the impact of the bidirectional backpropagation in the regularization term on the
NMT performance.
Table 4 shows the translation results of VAT and SimCut with or without bidirectional backpropagation.
We can see that both VAT and SimCut benefit from the bidirectional gradient backpropagation in the KL regularization.
4.2. Performance on Perturbed Inputs
Given the similar problem formulations of VAT and SimCut, it is natural to regard cutoff operation as a special perturbation and consider SimCut as a perturbation-based method.
We here investigate the robustness of NMT models on the perturbed inputs.
As discussed in Takase and Kiyono (2021), simple techniques such as word replacement and word drop can achieve comparable performance to sophisticated perturbations.
We hence include them as baselines to show the effectiveness of our method as follows:
• UniRep : Word replacement approach constructs a new sequence whose tokens are randomly replaced with sampled tokens.
For each token in the source sentence x , we sample ˆ x i uniformly from the source vocabulary, and use it for the new sequence x ′ with probability 1 − p ′ :
x ′ i
= (cid:40) x
i , with probability p ′ , ˆ x i
, with probability
[Equation]
p ′ .
[Equation]
We construct y ′ from the target sentence y in the same manner.
Following the curriculum learning strategy used in Bengio et al.
(2015), we adjust p ′ with the inverse sigmoid decay:
[Equation]
where q and k are hyper-parameters.
p ′ t decreases to q from 1, depending on the training epoch num-
[Table] [Table]
[Table]
ber t .
We use p ′ t as p ′ in epoch t .
We set q and k to be 0 .
9 and 25 respectively in the experiments.
• WordDrop :
Word drop randomly applies the zero vector instead of the word embedding e ( x i ) or e ( y i ) for the input token x i or y i (Gal and
Ghahramani, 2016).
For each token in both source and target sentences, we keep the original embedding with the probability β and set it to be the zero vector otherwise.
We set β to be 0 .
9 in the experiments.
We construct noisy inputs by randomly replacing words in the source sentences based on a predefined probability.
If the probability is 0 . 0 , we use the original source sentence.
If the probability is 1 . 0 , we use completely different sentences as source sentences.
We set the probability to be 0 . 00 , 0 . 01 , 0 . 05 , and 0 .
10 in our experiments.
We randomly replace each word in the source sentence with a word uniformly sampled from the vocabulary.
We apply this procedure to IWSLT14 de → en test set.
Table 5 shows the BLEU scores of each method on the perturbed test set.
Note that the
BLEU scores are calculated against the original reference sentences.
We can see that all methods improve the robustness of the NMT model, and
SimCut achieves the best performance among all the methods on both the clean and perturbed test sets.
The performance results indicate that SimCut could be considered as a strong baseline for the perturbation-based method for the NMT model.
As shown in Table 6, the baseline model completely ignores the translation of “in spielen (in games)” due to the replacement of “denken (think)” with “festgelegten (determined)” in the source sentence.
In contrast, our model successfully captures the translation of “in spielen” under the noisy input.
This result shows that our model is more robust to small perturbations in an authentic context.
4.2. Effects of α and p cut
We here investigate the impact of the scalar hyperparameters α and p cut in SimCut.
α is a penalty parameter that controls the regularization strength in our optimization problem.
p cut controls the percentage of the cutoff perturbations in SimCut.
We here vary α and p cut in { 1 , 2 , 3 , 4 , 5 } and { 0 . 00 , 0 .
15 , 0 . 20 } respectively and conduct the experiments on the IWSLT14 de → en dataset.
Note that SimCut is simplified to R-Drop approximately when p cut = 0 . 00 .
The test BLEU scores are reported in Figure 1.
By checking model performance under different combinations of α and p cut , we have the following observations: 1)
A too small α (e.g., 1 ) cannot achieve as good performance as larger α
(e.g., 3 ), indicating a certain de-
Input wir denken (festgelegten), dass wir in der realität nicht so gut sind wie in spielen.
Reference we feel that we are not as good in reality as we are in games.
Vaswani et al.
(2017) on Input
we think we’re not as good in reality as we are in games.
on Noisy Input we realized that we weren’t as good as we were in real life.
SimCut on Input we think in reality, we’re not as good as we do in games.
on Noisy Input we realized that we’re not as good in reality as we are in games.
gree of regularization strength during NMT model training is conducive to generalization.
Meanwhile, an overwhelming regularization ( α = 5 ) is not plausible for learning NMT models.
When α = 3 , the best performance is achieved when p cut = 0 . 05 , and p cut = 0 . 00 performs suboptimal among all selected probabilities.
Such an observation demonstrates that the cutoff perturbation in SimCut can effectively promote the generalization compared with R-Drop.
Is SimCut Compatible
with
the
Pretrained Language Model?
The multilingual sequence-to-sequence pretrained language models (Song et al., 2019;
Liu et al., 2020; Xue et al., 2021) have shown impressive performance on machine translation tasks, where the pretrained models generally learn the knowledge from the large-scale monolingual data.
It is interesting to investigate whether SimCut can gain performance improvement based on the pretrained language model.
We adopt mBART (Liu
Method de → en
Transformer 32.
mBART 38.
mBART with SimCut
et al., 2020) as the backbone model, which is a sequence-to-sequence denoising auto-encoder pretrained on CC25 Corpus 3 .
We conduct experiments on IWSLT14 de → en dataset and only remove the duplicated sentence pairs following mBART (Tang et al., 2021) in the data preprocessing step.
The source and target sentences are jointly tokenized into sub-word units with the 250K SentencePiece (Kudo and Richardson, 2018) vocabulary of mBART.
We use case-sensitive sacreBLEU (Post, 2018) to evaluate the translation quality, and the methods applied in the experiments are as follows:
• Transformer:
The Transformer model is randomly initialized and trained from scratch.
We utilize the same model and training configurations discussed in
Section
• mBART: The Transformer model is directly finetuned from mBART.
We utilize the default training configurations of mBART.
• mBART with SimCut: The Transformer model is finetuned from mBART with SimCut regularization.
We utilize the default training configurations of mBART.
From Table 7 we can see that SimCut could further improve the translation performance of mBART,
[Table]
which again shows the effectiveness and universality of our method.
4. Training Strategy: Bidirectional Pretrain and Unidirectional Finetune
Bidirectional pretraining is shown to be very effective to improve the translation performance of the unidirectional NMT system (Ding et al., 2021;
Xu et al., 2021).
The main idea is to pretrain a bidirectional NMT model at first and use it as the initialization to finetune a unidirectional NMT model.
Assume we want to train an NMT model for “English → German”, we first reconstruct the training sentence pairs to “English + German → German + English”, where the training dataset is doubled.
We then firstly train a bidirectional NMT model with the new training sentence pairs:
[Equation]
and finetune the model with “English → German” direction.
We follow the same training strategy in Ding et al.
and apply SimCut regularization to both pretraining and finetuning procedures.
Table 8 shows that bidirectional pretraining and unidirectional finetuning strategy with SimCut regularization could achieve superior performance compared with strong baseline such as R-Drop.
Comparison with Existing Methods
We summarize the recent results of several existing works on IWSLT14 en ↔ de benchmark in Table 9.
The existing methods vary from different aspects, including Virtual Adversarial Training (Sato et al., 2019), Mixed Tokenization for NMT (Wu et al., 2020), Unified Dropout for the Transformer model (Wu et al., 2021), Regularized Dropout (Liang et al.,
[Table]
2021), and BiBERT (Xu et al., 2021).
We can see that our approach achieves an improvement of 2 .
BLEU score over Vaswani et al.
and surpass the current SOTA method BiBERT that incorporates large-scale pretrained model, stochastic layer selection, and bidirectional pretraining.
Given the simplicity of Bi-SimCut, we believe it could be considered as a strong baseline for the NMT task.
Standard Resource Scenario
We here investigate the performance of Bi-SimCut on the larger translation benchmark compared with the IWSLT14 benchmark.
5. Dataset Description and Model Configuration
For the standard resource scenario, we evaluate
NMT models on the WMT14 English-German dataset, which contains 4.5M parallel sentence pairs.
We combine newstest2012 and newstest as the validation set and use newstest2014 as the test set.
We collect the pre-processed data from
Xu et al.
’s release 4 , where a shared dictionary with 52K BPE types is built.
We apply a standard Transformer Big model with 6 encoder and decoder layers, 16 attention heads, embedding size 1024, and FFN layer dimension 4096.
We apply cross-entropy loss with label smoothing rate 0 .
1 and set max tokens per batch to be 4096 .
We use Adam optimizer with Beta (0 . 9 , 0 . 98) ,warmup updates, and inverse square root learning rate scheduler with initial learning rates 1 e − 3 .
We decrease the learning rate to 5 e − 4 in the finetuning stage.
We select the dropout rate from 0 .
3 , 0 . 2 , and 0 .
1 based on the validation performance.
We
[Table]
use beam search decoding with beam size 4 and length penalty 0 .
We train all models until convergence on 8 NVIDIA Tesla V100 GPUs.
All reported BLEU scores are from a single model.
5. Results
We report test BLEU scores of all comparison methods and our approach on the WMT14 dataset in
Table 10.
With Bi-SimCut bidirectional pretraining and unidirectional finetuning procedures, our
NMT model achieves strong or SOTA BLEU scores on en → de and de → en translation benchmarks.
During the NMT training process, we fix p cut to be 0 . 05 and tune the hyper-parameter α in both
R-Drop and SimCut based on the performance on the validation set.
Note that the BLEU scores of
R-Drop are lower than that reported in Liang et al.
Such gap might be due to the different prepossessing steps used in Liang et al.
(2021) and Xu et al.
It is worth mentioning that Bi-SimCut outperforms BiBERT on de → en direction even though BiBERT incorporates bidirectional pretraining, large-scale pretrained contextualized embeddings, and stochastic layer selection mechanism.
High Resource Scenario
To investigate the performance of Bi-SimCut on the distant language pairs which naturally do not share dictionaries, we here discuss the effectiveness of
Bi-SimCut on the Chinese-English translation task.
6. Dataset Description and Model Configuration
For the high resource scenario, we evaluate NMT models on the WMT17 Chinese-English dataset, which consists of 20.2M training sentence pairs,
[Table]
and we use newsdev2017 as the validation set and newstest2017 as the test set.
We firstly build the source and target vocabularies with 32K BPE types separately and treat them as separated or joined dictionaries in our experiments.
We apply the same Transformer Big model and training configurations used in the WMT14 experiments.
We use beam search decoding with beam size 5 and length penalty 1 .
We train all models until convergence on 8 NVIDIA Tesla V100 GPUs.
All reported BLEU scores are from a single model.
6. Results
We report test BLEU scores of the baselines and our approach on the WMT17 dataset in Table 11.
Note that share means the embedding matrices for encoder input, decoder input and decoder output are all shared.
The NMT models with separated dictionaries perform slightly better than those with the shared dictionary.
We can see that our approach significantly improves the translation performance.
In particular, Bi-SimCut achieves more than
. 6 BLEU score improvement over Vaswani
et al.
(2017), showing the effectiveness and univer- sality of our approach on the distant language pair in the NMT task.
Related Work Adversarial Perturbation
It is well known that neural networks are sensitive to noisy inputs, and adversarial perturbations are firstly discussed in the filed of image processing (Szegedy et al., 2014;
Goodfellow et al., 2015).
SimCut could be regarded as a perturbation-based method for the robustness research.
In the field of natural language processing, Miyato et al.
(2017) consider adversarial perturbations in the embedding space and show its effectiveness on the text classification tasks.
For the NMT tasks, Sato et al.
(2019) and Wang et al.
(2019) apply adversarial perturbations in the embedding space during training of the encoderdecoder NMT model.
Cheng et al.
(2019) leverage adversarial perturbations and generate adversarial examples by replacing words in both source and target sentences.
They introduce two additional language models for both sides and a candidate word selection mechanism for replacing words in the sentence pairs.
Takase and Kiyono (2021) compare perturbations for the NMT model in view of computational time and show that simple perturbations are sufficiently effective compared with complicated adversarial perturbations.
Consistency Training
Besides perturbationbased methods, our approach also highly relates to a few works of model-level and data-level consistency training in the NMT field.
Among them, the most representative methods are R-Drop
(Liang et al., 2021) and Cutoff (Shen et al., 2020).
R-Drop studies the intrinsic randomness in the NMT model and regularizes the NMT model by utilizing the output consistency between two dropout sub-models with the same inputs.
Cutoff considers consistency training from a data perspective by regularizing the inconsistency between the original sentence pair and the augmented samples with part of the information within the input sentence pair being dropped.
Note that Cutoff takes the dropout submodels into account during the training procedure as well.
We want to emphasize that SimCut is not a new method, but a version of Cutoff simplified and adapted for the NMT tasks.
Conclusion
In this paper, we propose Bi-SimCut: a simple but effective two-stage training strategy to improve
NMT performance.
Bi-SimCut consists of bidirectional pretraining and unidirectional finetuning procedures equipped with SimCut regularization for improving the generality of the NMT model.
Experiments on low (IWSLT14 en ↔ de ), standard (WMT14 en ↔ de ), and high (WMT17 zh → en ) resource translation benchmarks demonstrate BiSimCut and SimCut’s capabilities to improve translation performance and robustness.
Given the universality and simplicity of Bi-SimCut and SimCut, we believe: 1) SimCut could be regarded as a perturbation-based method, and it could be used as a strong baseline for the robustness research.
2) BiSimCut outperforms many complicated methods which incorporate large-scaled pretrained models or sophisticated mechanisms, and it could be used as a strong baseline for future NMT research.
We hope researchers of perturbations and NMT could use SimCut and Bi-SimCut as strong baselines to make the usefulness and effectiveness of their proposed methods clear.
For future work, we will explore the effectiveness of SimCut and Bi-SimCut on more sequence learning tasks, such as multilingual machine translation, domain adaptation, text classification, natural language understanding, etc.
Acknowledgements
We would like to thank the anonymous reviewers for their insightful comments.