E dinburgh’s Submissions to the 2020 Machine Translation Efficiency Task (2020)

Abstract: We participated in all tracks of the Workshop on Neural Generation and Translation 2020 Efficiency Shared Task: single-core CPU, multi-core CPU, and GPU. At the model level, we use teacher-student training with a variety of student sizes, tie embeddings and sometimes layers, use the Simpler Simple Recurrent Unit, and introduce head pruning. On GPUs, we used 16-bit floating-point tensor cores. On CPUs, we customized 8-bit quantization and multiple processes with affinity for the multi-core setting. To reduce model size, we experimented with 4-bit log quantization but use floats at runtime. In the shared task, most of our submissions were Pareto optimal with respect the trade-off between time and quality.

## introduction this paper describes the university of edinburgh's submissions to the workshop on neural generation and translation (wngt) 2020 efficiency shared task 1 using the marian machine translation toolkit @xcite . the task has gpu, single-core cpu, and multi-core cpu tracks. our submissions focus on the tradeoff between translation quality and speed; we also address model size after submission. starting from an ensemble of 4 transformer-big teacher models, we trained a variety of student configurations and on top of that sometimes pruned transformer heads. for the decoding process, we explored the use of lower precision gemm for both our cpu and gpu submissions. small models appear to be more sensitive to quantization than large models. most of our single-cpu submissions had a memory leak, which also impacted speed; we report results before and after fixing the leak. the task measures quality approximated by bleu @xcite , speed, model size, docker image size, and memory consumption of a machine translation system from english to german for the wmt 2019 data condition @xcite . we did not optimize docker image size (using stock ubuntu) or memory consumption (preferring large batches for speed). the task intentionally did not specify a test set until after submissions were made. this was later revealed to be the average of bleu from wmt test sets from 2010 through 2019, inclusive. however, the 2012 test set was excluded because it contains english sentences longer than 100 words and participants were promised input would be at most 100 words. we refer to the task's metric as wmt1*. all bleu scores are reported using sacrebleu. 2 the cpu tracks used an intel xeon platinum 8275cl while the gpu track used an nvidia t4. for speed, the official input has 1 million lines of text with 15,048,961 space-separated words. ## teacher-student training following junczys-dowmunt et al. (2018b) and @xcite , all our optimized models are students created using interpolated sequence-level knowledge distillation @xcite , and trained on data generated from a teacher system. teacher we used the sentence-level english-german system from microsoft's constrained submission to the wmt'19 news translation task (junczys-dowmunt, 2019). it is an ensemble of four deep transformer-big models @xcite , each with 12 blocks of layers in encoder and decoder, model size of 1024, filter size of 4096, and 8 transformer heads. 3 the ensemble achieved 42.5 bleu on the official wmt19 test set when decoded with beam size of 8. we refer the reader to the original paper for more details on how this system has been built. ## attention pruning attention is one of the most expensive operations in the transformer architecture, yet many of the heads can be pruned after training @xcite . moreover, the lottery ticket hypothesis (frankle and carbin, 2018) and subsequent work on pruning optimisation @xcite suggests that pruning is less damaging during training rather than after training. hence we combine these two ideas to prune attention heads during training. since we are starting from a relatively optimized model (tiny in we try removing @xmath0, dubbing these steady and pushy in system names, respectively. since the algorithm usually picks one head from each layer, the final architecture differs. for example, removing 6 heads per iteration results in a monotonic attention distribution across the 6 encoder layers. for submissions, we pruned 36 of the 48 heads; as an additional experiment we tried removing 42 of the 48 heads. the final attention distribution, size and bleu scores for those models are presented in considering that our students perform better on newer testsets, the pruning results show that it is possible to remove at least 75% of self-attention heads in an encoder with an average 0.4 bleu loss. with harsher pruning, the model with even num-bers of heads performs better than the one missing any from the first two layers. this indicates that, in extreme cases, it is better to have at least one head per layer than none. since the dimension of each head was small (256 / 8 = 32), pruning has not reduced the overall size of the models drastically. the speed-up is about 10% on cpu with 75% encoder heads removed. in terms of on gpu, our best pruned model gains 15% speed-up w.r.t. words per second (wps) losing 0.1 bleu in comparison to an unpruned model (tab. 4). ## cpu optimizations for our cpu optimization we build upon last year submission @xcite . we use the same lexical shortlist, but we extend the usage of 8bit integer quantized gemm operations to also cover the shortlisted output layer in order to have faster computation and even smaller model size. ## gpu systems this year, we did not implement any gpu-specific optimizations and focused on comparing the performance of student architectures, developed for cpu decoding, on the gpu. we made 4 submissions to the gpu track. the results for all student models, averaged across 3 runs are reported in the average speed-up from decoding in 16-bit floats is 21%, depending on the model architecture. the larger the model size, the larger speed improvement, with as high as 56% improvement for the large student model, through 32% for base, and only 13-18% for tiny models. this is with barely any change in bleu, lower @xmath1.1. models with pruned transformer heads are faster than the original tiny model by 15% on gpu, but decrease the accuracy by 0.1-0.5 bleu on the wmt19 test set. on this relatively small data set, we notice a small translation speed decrease of up to 2% from using lexical shortlists. running concurrent streams on a single gpu did not yield significant improvements for us. ## results and discussion all submissions and select experiments are depicted in figure 1 . we explored a variety of ways to optimize the trade-off between quality, speed, and model size. we use an ensemble of 4 transformer-big teacher models to train a number of different student configurations. smaller student models are faster to decode, but also further degrade the performance compared to the ensemble of teachers. furthermore, we apply gradual transformer head pruning to the student models. while pruning the number of heads does not reduce the number of parameters significantly, it has a major impact on the computational cost and is beneficial for increasing translation speed, at a small penalty in bleu score. on the software side, we experiment with a number of methods that reduce the precision for the gemm operations. for our gpu submissions, we decode using 16-bit floats and for cpu ones we use 8-bit integers. we note that the smaller (in terms of number of parameters) the model is, the more impacted quality is by quantization, and the bigger the model is, the larger the speed increase is. we found that fine tuning with a quantized gemm can recover some of the quality loss from quantization. we also experimented with logarithmic 4-bit model compression, which did not yield increased translation speed due to hardware, but produced the smallest model sizes.