Optimizing Transformer for Low-Resource Neural Machine Translation (2020)

Abstract: Language pairs with limited amounts of parallel data, also known as low-resource languages, remain a challenge for neural machine translation. While the Transformer model has achieved significant improvements for many language pairs and has become the de facto mainstream architecture, its capability under low-resource conditions has not been fully investigated yet. Our experiments on different subsets of the IWSLT14 training data show that the effectiveness of Transformer under low-resource conditions is highly dependent on the hyper-parameter settings. Our experiments show that using an optimized Transformer for low-resource conditions improves the translation quality up to 7.3 BLEU points compared to using the Transformer default settings.

## introduction despite the success of neural machine translation (nmt) @xcite @xcite , for the vast majority of language pairs for which only limited amounts of training data exist (a.k.a. low-resource languages), the performance of nmt systems is relatively poor @xcite . most approaches focus on exploiting additional data to address this problem @xcite @xcite . however, @xcite show that a well-optimized nmt system can perform relatively well under low-resource data conditions. unfortunately, their results are confined to a recurrent nmt architecture @xcite , and it is not clear to what extent these findings also hold for the nowadays much more commonly used transformer architecture @xcite . like all nmt models, transformer requires setting various hyper-parameters but researchers often stick to the default parameters, even when their data conditions differ substantially from the original data conditions used to determine those default values @xcite . in this paper, we explore to what extent hyper-parameter optimization, which has been applied successfully to recurrent nmt models for low-resource translation, is also beneficial for the transformer model. we show that with the appropriate settings, including the number of bpe merge operations, attention heads, and layers up to the degree of dropout and label smoothing, translation performance can be increased substantially, even for data sets with as little as 5k sentence pairs. our experiments on different corpus sizes, ranging from 5k to 165k sentence pairs, show the importance of choosing the optimal settings with respect to data size. ## hyper-parameter exploration in this section, we first discuss the importance of choosing an appropriate degree of subword segmentation before we describe the other optimal hyper-parameter settings. vocabulary representation. in order to improve the translation of rare words, word segmentation approaches such as byte-pair-encoding (bpe) @xcite have become standard practice in nmt. this is especially true for language pairs with small amounts of data where rare words are a common phenomenon. @xcite show that reducing the number of bpe merge operations can result in substantial improvements of up to 5 bleu points for a recurrent nmt model. it is natural to assume that reducing the bpe vocabulary is similarly effective for transfomer. architecture tuning. a current observation in neural networks, and in particular in transformer architectures, is that increasing the number of model parameters improves performance @xcite . however, those findings are mostly obtained for scenarios with ample training data and it is not clear if they are directly applicable to low-resource conditions. while @xcite show that using fewer transformer layers improves the quality of low-resource nmt, we expand our exploration towards the effects of using a narrow and shallow transformer by reducing i) the number of layers in both the encoder and decoder, ii) the number of attention heads, iii) feed-forward layer dimension (d ff ), and iv) embedding dimensions (d model ). regularization. following @xcite , we analyze the impact of regularization by applying dropouts to various transformer components @xcite . in addition to regular dropout which is applied to the output of each sub-layer (feed-forward and self-attention) and after adding the positional embedding in both encoder and decoder @xcite , we employ attention dropout after the softmax for self-attention and also activation dropout inside the feed-forward sub-layers. moreover, we drop entire layers using layer dropout @xcite . we further drop words in the embedding matrix using discrete word dropout @xcite . we also experiment with larger label-smoothing factors @xcite . ## conclusion in this paper, we study the effects of hyper-parameter settings for the transformer architecture under various low-resource data conditions. while our findings are largely in line with previous work @xcite for rnn-based models, we show that very effective optimizations for rnn-based models such as reducing the number of bpe merge operations or using small batch sizes are less effective or even hurt performance. our experiments show that a proper combination of transformer configurations combined with regularization techniques results in substantial improvements over a transformer system with default settings for all low-resource data sizes. however, under extremely low-resource conditions an optimized rnn model still outperforms transformer.