Efficient Inference For Neural Machine Translation (2020)

Abstract: Large Transformer models have achieved state-of-the-art results in neural machine translation and have become standard in the field. In this work, we look for the optimal combination of known techniques to optimize inference speed without sacrificing translation quality. We conduct an empirical study that stacks various approaches and demonstrates that combination of replacing decoder self-attention with simplified recurrent units, adopting a deep encoder and a shallow decoder architecture and multi-head attention pruning can achieve up to 109% and 84% speedup on CPU and GPU respectively and reduce the number of parameters by 25% while maintaining the same translation quality in terms of BLEU.

## introduction and related work transformer models @xcite have outperformed previously used rnn models and traditional statistical mt techniques. this improvement, though, comes at the cost of higher computation complexity. the decoder computation often remains the bottleneck due to its autoregressive nature, large depth and self-attention structure. there has been a recent trend towards making the models larger and ensembling multiple models to achieve the best possible translation quality @xcite . leading solutions on common benchmarks @xcite usually use an ensemble of transformer big models, which combined can have more than 1 billion parameters. previous works suggest replacing the expensive self-attention layer in the decoder with simpler alternatives like the average attention network (aan) @xcite , simple recurrent unit (sru) @xcite and simpler simple recurrent unit (ssru) @xcite . aan is ## efficient inference for neural machine translation this section presents the proposed efficient inference architecture for neural machine translation. first, we outline the overall procedure of building an efficient inference architecture. then, we detail each step in the process. first, we use sequence-level knowledge distillation @xcite to transfer knowledge from a strong teacher model to a smaller student model. this approach allows the student model to learn from a simpler target distribution and therefore enables us to use a simpler architecture. then, to simplify the decoder of the student model, the self-attention mechanism is replaced by lightweight recurrent units @xcite , and the feed-forward network is removed. to further reduce the decoder computation, we adopt the deep encoder, shallow decoder architecture @xcite . lastly, we prune redundant attention heads through l 0 regularization @xcite . each architecture modification is performed by retraining the student model. figure 1 shows the proposed efficient transformer architecture. ## experiments we use the transformer base model @xcite trained on teacher decoded data as our baseline. all the described methods are stacked on top of this baseline model. following @xcite , we use 4 million bitext from the wmt'14 english-german news translation task. all sentences are encoded with 32k subword units using sentence-piece @xcite . we report bleu on the newstest2014 in all the experiments and use newstest2015 for the final evaluation in section 3.4 all experiments are implemented in fairseq @xcite . the configuration of teacher-student training follows the settings in @xcite . we use an effective batch size of 458k words and 16 gpus for training. adam optimizer is applied with β = (0.9, 0.98). we use label smoothing with ε = 0.1, inverse square root learning rate schedule with 2500 warmup steps and peak learning rate of 0.0007. the models are trained with 50k updates except for the models with pruning, where additional fine-tuning with 100-150k updates is applied. we use a beam size of 5 during inference. we evaluate the inference speed with batch size of 128 sentences on gpu, batch size 1 on cpu and report speed in words per second (wps), averaged over 10 decoding runs. hardware: we evaluate our performance on 1 gpu (nvidia tesla v100-sxm2-32gb) and 1 core cpu (intel xeon e5-2640 v4 @ 2.40ghz) ## conclusion in this paper we explored the combination of techniques aimed at improving inference speed which lead to the discovery of a very efficient architecture. the best architecture has a deep 12-layer encoder, and a shallow decoder with only one single lightweight recurrent unit layer and one encoderdecoder attention mechanism. 75% of the encoder heads were pruned giving rise to a model with 25% fewer parameters than the baseline transformer. in terms of inference speed, the proposed architecture is 84% faster on a gpu, and 109% faster on a cpu.