Multi-Unit Transformers for Neural Machine Translation (2020)

Abstract: Transformer models achieve remarkable success in Neural Machine Translation. Many efforts have been devoted to deepening the Transformer by stacking several units (i.e., a combination of Multihead Attentions and FFN) in a cascade, while the investigation over multiple parallel units draws little attention. In this paper, we propose the Multi-Unit Transformer (MUTE) , which aim to promote the expressiveness of the Transformer by introducing diverse and complementary units. Specifically, we use several parallel units and show that modeling with multiple units improves model performance and introduces diversity. Further, to better leverage the advantage of the multi-unit setting, we design biased module and sequential dependency that guide and encourage complementariness among different units. Experimental results on three machine translation tasks, the NIST Chinese-to-English, WMT’14 English-to-German and WMT’18 Chinese-to-English, show that the MUTE models significantly outperform the Transformer-Base, by up to +1.52, +1.90 and +1.10 BLEU points, with only a mild drop in inference speed (about 3.1%). In addition, our methods also surpass the Transformer-Big model, with only 54% of its parameters. These results demonstrate the effectiveness of the MUTE, as well as its efficiency in both the inference process and parameter usage.

## introduction transformer based models @xcite have been proven to be very effective in building the state-of-the-art neural machine translation (nmt) systems via neural networks and attention mechanism @xcite . following the standard sequence-to-sequence architecture, transformer models consist of two essential components, namely the encoder and decoder, which rely on stacking several identical layers, i.e., multihead attentions and positionwise feed-forward network. multihead attentions and position-wise feedforward network, together as a basic unit, plays an essential role in the success of transformer models. some researchers @xcite propose to improve the model capacity by stacking this basic unit many times, i.e., deep transformers, and achieve promising results. nevertheless, as an orthogonal direction, investigation over multiple parallel units draws little attention. compared with single unit models, multiple parallel unit layout is more expressive to capture complex information flow @xcite @xcite in two aspects. first, this multiple-unit layout boosts the model by its varied feature space composition and different attentions over inputs. with this diversity, multi-unit models advance in expressiveness. second, for the multiunit setting, one unit could mitigate the deficiency of other units and compose a more expressive network, in a complementary way. in this paper, we propose the multi-unit transformers (mute), which aim to promote the expressiveness of transformer models by introducing diverse and complementary parallel units. merely combining multiple identical units in parallel improves model capability and diversity by its varied feature compositions. furthermore, inspired by the well-studied bagging @xcite and gradient boosting algorithms @xcite in the machine learning field, we design biased units with a sequential dependency to further boost model performance. specifically, with the help of a module named bias module, we apply different kinds of noises to form biased inputs for corre-sponding units. by doing so, we explicitly establish information gaps among units and guide them to learn from each other. moreover, to better leverage the power of complementariness, we introduce sequential ordering into the multi-unit setting, and force each unit to learn the residual of its preceding accumulation. we evaluate our methods on three widely used neural machine translation datasets, nist chinese-english, wmt'14 english-german and wmt'18 chinese-english. experimental results show that our multi-unit model yields an improvement of +1.52, +1.90 and +1.10 bleu points, over the baseline model (transformer-base) for three tasks with different sizes, respectively. our model even outperforms the transformer-big on the wmt'14 english-german by 0.7 bleu points with only 54% of parameters. moreover, as an interesting side effect, our model only introduces mild inference speed decrease (about 3.1%) compared with the transformer-base model, and is faster than the transformer-big model. the contributions of this paper are threefold: ## transformer architecture the transformer architecture @xcite for neural machine translation (nmt) generally adopts the standard encoder-decoder paradigm. in contrast to rnn architectures, the transformer stacks several identical self-attention based layers instead of recurrent units for better parallelization. specifically, given an input sequence @xmath0, x 2 , • • • , x n } in source language (e.g., english), the model is asked to predict its corre-sponding translation @xmath1, y 2 , • • • , y m } in target language (e.g., german). encoder. digging into the details of the model, a transformer encoder consists of n e stacked layers, where each layer consists of two sub-layers, a multihead self-attention sub-layer and a position-wise feed-forward network (ffn) sub-layer. decoder. the decoder follows a similar architecture, with an additional multihead cross-attention sub-layer for each of n d decoder layers. here, we omit layer norms among sub-layers for simplicity. we take the bundle of cascading subcomponents (i.e., attention modules and ffn) as a unit, and refer to the original transformer and its variants with such cascade units as single-unit transformer. for ease of reading, we refer to a single unit of encoder and decoder as f e and f d in the following sections. then, the probability p (y |x) is produced with another softmax layer on top of decoder outputs, ## experimental settings in this section, we elaborate our experimental setup on three widely-studied neural machine translation tasks, nist chinese-english (zh-en), wmt'14 english-german (en-de) and wmt'18 chinese-english. ## results through experiments, we first evaluate our model performance (section 5.1 and 5.2). then, we analyze how each part of our model works (section 5.4 and 5.6). finally, we conduct experiments to further understand the behavior of our models (sec- system #param. bleu ∆ existing nmt systems vaswani et al. (2017) (base) 81m 27.3 ref. bapna et al. (2018) (base, 16l) 137m 28.0 -wang et al. (2019a) (base, 30l) 137m 29.3 -vaswani et al. (2017) (big) 251m 28.4 -chen et al. (2018) (big) 379m 28.5 -he et al. (2018) (big) *251m 29.0 -shaw et al. (2018) (big) *251m 29.2 -ott et al. (2018) (big) 251m 29.3 -existing multi-unit style nmt systems ahmed et al. (2017) (base) *81m 28.4 -li et al. (2019) (base) 118m 28.5 -li et al. (2018) (base) *81m 28.5 -baseline nmt systems transformer (base) 81m 27.4 ref. transformer + relative (base) 81m 28.2 +0.8 transformer + relative (big) 251m 28.6 +1.2 our nmt systems mute (base, 4 units) 130m 28.8 † † +1.4 + bias (base, 4 units) 130m 29.1 † † +1.7 + bias + seq. (base, 4 units) 130m 29.3 † † +1.9 ## related work recently transformer-based models @xcite @xcite become the de facto methods in neural machine translation, owing to high parallelism and large model capacity. some researchers devise new modules to improve the transformer model, including combining the transformer unit with convolution networks @xcite @xcite , improving the self-attention architecture @xcite @xcite , and deepening the transformer architecture by dense connections @xcite . since our multi-unit framework makes no limitation about its unit, these models can be easily integrated into our multi-unit framework. there are also some works utilizing the power of multiple modules to capture complex feature representations in nmt. @xcite use a vast network and a sparse gated function to select from multiple experts (i.e., mlps). @xcite train a weighted transformer by replacing the multi-head attention by self-attention branches. nevertheless, these models ignore the modeling of relations among different modules. then, some multihead attention variants @xcite introduce modeling of diversity or interaction among heads. however, complementariness is not taken into account in their approaches. our mute models differ from their methods in two aspects. first, we use a powerful unit with a strong performance in diversity (section 5.6). second, we explicitly model the complementariness with bias module and sequential dependency. ## conclusion in this paper, we propose multi-unit transformers for nmt to improve the expressiveness by introducing diverse and complementary units. in addition, we propose two novel techniques, namely bias module and sequential dependency to further improve the diversity and complementariness among