Two Heads are Better than One? Verification of Ensemble Effect in Neural Machine Translation (2021)

Abstract: In the field of natural language processing, ensembles are broadly known to be effective in improving performance. This paper analyzes how ensemble of neural machine translation (NMT) models affect performance improvement by designing various experimental setups (i.e., intra-, inter-ensemble, and non-convergence ensemble). To an in-depth examination, we analyze each ensemble method with respect to several aspects such as different attention models and vocab strategies. Experimental results show that ensembling is not always resulting in performance increases and give noteworthy negative findings.

## introduction ensemble is a technique for obtaining accurate predictions by combining the predictions of several models. in neural machine translation (nmt), ensembles are most closely related to vocabulary (vocab). in particular, by aggregating the prediction results of multiple models, the ensemble averages the probability values over the vocab of the softmax layer @xcite . most existing studies on ensembling for nmt focus on improving the performance of shared tasks. for example, in wmt's shared task, almost every participating team applied the ensemble technique to improve performance @xcite @xcite . however, in most cases, only experimental results that improved performance by applying the ensemble technique are introduced; in-depth comparative analysis is rarely conducted @xcite @xcite . in this study, we attempt to investigate three main aspects regarding ensembles for machine translation. first, we investigate the ensemble effect when using various vocab strategies and different attention models. for the vocab that plays the most important role in the machine translation ensemble, three dif-ferent experimental conditions-independent vocab, share vocab, and share embedding-are applied to two different attention networks @xcite . second, we investigate which among intraensemble and inter-ensemble is more effective for performance improvement. notably, intraensemble is an ensemble of identical models, while inter-ensemble represents an ensemble between models that follow different network structures. third, we analyze the effect of the nonconverging model on ensemble performance. most existing studies create an ensemble using only those models that have been fitted. however, we perform in-depth comparative analysis experiments, raising the question of whether the nonconverging model has only negative effects. ## conclusion most researchers consider it common sense that ensembles are better; however, few studies have conducted any type of close verification. in this study, we perform various tests based on three experimental designs related to the ensemble technique, and demonstrate its negative aspects. thus, we provide insights into the positives and negatives of ensembling for machine translation. in the future, we plan to conduct expanded experiments based on different language pairs.