Single Model Ensemble for Subword Regularized Models in Low-Resource Machine Translation (2022)

Abstract: Subword regularizations use multiple subword segmentations during training to improve the robustness of neural machine translation models. In previous subword regularizations, we use multiple segmentations in the training process but use only one segmentation in the inference. In this study, we propose an inference strategy to address this discrepancy. The proposed strategy approximates the marginalized likelihood by using multiple segmentations including the most plausible segmentation and several sampled segmentations. Because the proposed strategy aggregates predictions from several segmentations, we can regard it as a single model ensemble that does not require any additional cost for training. Experimental results show that the proposed strategy improves the performance of models trained with subword regularization in low-resource machine translation tasks.

## introduction subword regularizations are the technique to make a model robust to segmentation errors by using multiple subword segmentations instead of only the most plausible segmentation during the training process @xcite . previous studies demonstrated that subword regularizations improve the performance of lstmbased encoder-decoders and transformers in various machine translation datasets, especially in lowresource settings @xcite . however, previous subword regularizations contain the discrepancy between the training and inference. in the training process, we stochastically re-segment a given sequence into subwords based on statistics such as the uni-gram language model @xcite . thus, we use multiple segmentations for each input sequence. in contrast, we use only the most plausible segmentation in the inference phase. we expect that we can improve the performance by solving this discrepancy. to solve this discrepancy, we propose an inference strategy that uses multiple subword segmentations. we construct multiple subword segmentations for an input in the same manner as that in the training process, and then aggregate the predictions from each segmentation. therefore, our proposed inference strategy can be regarded as a single model ensemble using multiple segmentations. figure 1 illustrates the overview of previous methods and our proposed inference strategy. we conduct experiments on several machine translation datasets. experimental results show that the proposed strategy improves the performance of a subword regularized model without any additional costs in the training procedure when the subword regularization significantly contributes to the performance, i.e., in low-resource settings. moreover, we indicate that our strategy can be combined with a widely used model ensemble technique. ## subword regularization our proposed strategy is based on a model trained with subword regularization. thus, we briefly describe subword regularization in this section. @xcite proposed subword regularization to improve the robustness of a neural machine translation model. let x and y be the source and target sentences, @xmath0, ..., x s ) and @xmath1, ..., y t ) be the most plausible subword segmentations corresponding to x and y . in the vanilla training strategy, i.e., without subword regularization, we train the parameters of a neural machine translation model θ to maximize the following log-likelihood: training inference vanilla we see the … wir sehe n das … encoder decoder we se e the … encoder decoder subword regularization we se e the … encoder decoder we se e the … w ir se hen das … encoder decoder w e see t h e … w e se e th e … w ir sehen da s … wir s e hen das … we se e the … w ir se hen das … encoder decoder w e see t h e … w e se e th e … w ir sehen da s … wir s e hen das … we se e the … encoder decoder w e see t h e … w e se e th e … proposed inference strategy in contrast, subword regularization uses multiple subword segmentations during training. let p (x |x) and p (y |y ) be segmentation probabilities for sequences x and y , respectively. we optimize the parameters θ with the following marginalized likelihood in subword regularization: we sample x i and y j for every mini-batch during training to yield a good approximation. in the inference phase, we input the most plausible segmentation x and search a sequence y * that maximizes the log-likelihood log p (y|x; θ). in other words, we input one segmentation to the model foot_0 even though we use multiple segmentations during training. 3 proposed method ## performance in enough training data section 4 shows the results in low-resource settings but previous studies reported that subword regularizations can improve the performance if we use sufficient training data. thus, we investigate the performance of subword regularization and proposed strategy by varying the size of training data. we used the wmt 2016 english-to-german training dataset, which is widely used in previous studies @xcite @xcite . this dataset contains 4.5m sentence pairs, that are more than 25 times as many as iwslt datasets. we conducted pre-processing in the same manner as that in @xcite . we trained the transformer (base) model in @xcite . for subword regularization, we set α = 0.5 in the same as @xcite . we evaluated bleu scores on newstest2013, which is widely used as a valid data. figure 2 shows bleu scores of each method for each training data size. this figure indicates that the model trained with subword regularization outperformed vanilla in all training data sizes but the improvement decreased in accordance with the increase in the training data. the proposed strategy slightly improved the performance from subword regularization for the small training data but the improvement also decreased as the training data increased. when we used the entire training data (4.5m translation pairs), the bleu score of the proposed strategy was identical to that of subword regularization. this result implies that the impact of the proposed strategy on the performance is small when the improvement by subword regularization is small. in other words, the proposed strategy is effective especially in low-resource settings because subword regularization probably provides much improvement in low-resource settings. however, we emphasize that the proposed strategy has no negative effect on the bleu score for sufficient training data fortunately. ## related work in this study, we proposed the inference strategy to mitigate the discrepancy between the training and inference in subword regularizations. in experiments, we focused the subword regularization proposed by @xcite but we can apply the proposed inference strategy to variants of the subword regularization such as bpe dropout @xcite and compositional word replacement @xcite . @xcite reported that simple perturbations such as word dropout are effective in a large amount of training data. thus, we might improve the performance of the model trained with such simple perturbations if we use multiple inputs constructed by the same perturbation during the inference. we focused on an input of a neural encoderdecoder. in contrast, @xcite focused on internal layers. for neural network methods, we often apply the dropout during the training but do not use it in the inference. @xcite proposed the variational inference to mitigate this gap on the dropout. as described in section 1, our proposed inference strategy can be regarded as a single model ensemble. huang et al. (2017) and kuwabara et al. (2020) also proposed single model ensemble methods. huang et al. (2017) proposed the snapshot ensemble that uses multiple models in the middle of the training. kuwabara et al. (2020) used pseudo-tags and predefined distinct vectors to obtain multiple models virtually during the training of a single model. since these methods are orthogonal to ours, we can combine our proposed strategy. ## conclusion we proposed an inference strategy to address the discrepancy between the training and inference in subword regularizations. our proposed strategy uses multiple subword segmentations as inputs to approximate the marginalized likelihood used as the objective function during training. the proposed strategy improved the performance of the model trained with subword regularization in cases where subword regularization provided the significant improvement, i.e., in low-resource settings. moreover, the proposed strategy outperformed the n-best decoding strategy @xcite . experimental results show that our proposed strategy has no negative effect on the bleu score even if the improvement by subword regularization is small. because the proposed inference strategy does not require any additional training cost, we encourage using the strategy to highlight the potential of models trained with subword regularization.