Compressing Neural Machine Translation Models with 4-bit Precision (2020)

Abstract: Neural Machine Translation (NMT) is resource-intensive. We design a quantization procedure to compress fit NMT models better for devices with limited hardware capability. We use logarithmic quantization, instead of the more commonly used fixed-point quantization, based on the empirical fact that parameters distribution is not uniform. We find that biases do not take a lot of memory and show that biases can be left uncompressed to improve the overall quality without affecting the compression rate. We also propose to use an error-feedback mechanism during retraining, to preserve the compressed model as a stale gradient. We empirically show that NMT models based on Transformer or RNN architecture can be compressed up to 4-bit precision without any noticeable quality degradation. Models can be compressed up to binary precision, albeit with lower quality. RNN architecture seems to be more robust towards compression, compared to the Transformer.

## introduction neural machine translation (nmt) is resourcedemanding. current state-of-the-art architectures, such as the transformer @xcite or deep @xcite are typically hundreds of megabytes in size. in a client-based translation system, these large models must be deployed locally, thus consuming network bandwidth for distributing the model, and disk space for storing the model. model quantisation has been widely studied as a way to compress model size and increase the inference speed. however, most of this work has focused on convolution neural networks for computer vision tasks @xcite @xcite @xcite . as such, research on model quantisation for nmt tasks remains limited. we find that the model can be compressed at up to 4-bit precision without sacrificing quality. we first explore the use of logarithmic-based quantisation over fixed-point quantisation @xcite based on the empirical findings that parameter distribution is not uniform, but instead concentrated near zero @xcite . the magnitude of a parameter also varies across layers; therefore, we propose an improved method of scaling the quantization centres. we also notice that biases do not quantise very well. however, since biases do not consume a noticeable amount of memory, they can be left unquantised. lastly, we explore the significance of re-training in the model compression scenario. we adopt an error feedback mechanism @xcite to preserve the quantisation error rather than discarding it at every update during re-training. ## related work a considerable amount of research on model quantisation has been performed in the area of computer vision with convolutional neural networks; however, research on model quantisation in the field of neural machine translation is far more limited. therefore, we will also refer to work on neural models for image processing in this section, where appropriate. @xcite quantised the model and activation to binary on a cnn network for various image classification tasks. the binary network achieved near state-of-the-art quality on several easier tasks such as mnist and cifar-10 but achieved sub-par performance on the more challenging imagenet dataset (losing over 20% accuracy with quantised googlenet). @xcite later reported that with 6-bit fixed-point quantisation, googlenet "only" lost 5% accuracy. @xcite used different bit precisions on various cnn layers, achieving over 20% compression on the cifar-10 task. since the model's parameters are highly concentrated near zero, @xcite opted for logarithmic quantisation. they report an improvement in preserving model accuracy over linear quantisation while achieving the same model compression rate. they also reported negligible accuracy degradation when compressing vgg16 with 3-bit logarithmic quantisation, whereas 3-bit fixed-point quantisation suffered a 6% accuracy drop. @xcite compress an lstm-based architecture for language modelling to 4-bits without any quality degradation but had to scale the hidden layer size by 3. @xcite pruned an nmt model by removing any weight values lower than a certain threshold. they achieve 80% model sparsity without any quality degradation. a relevant work with respect to our purposes is the submission of @xcite to the shared task on efficient neural machine translation in 2018. this submission applied an 8-bit linear quantisation for nmt models without any noticeable deterioration in translation quality. similarly, @xcite proposed the use of 8-bit matrix multiplication to increase the cpu inference speed of an nmt system. ## conclusion we compress the model size in neural machine translation to approximately 7.9x smaller than 32bit floats by using a 4-bit logarithmic quantisation. bias terms can be left uncompressed without significantly affecting the compression rate. we also find that re-training after quantisation is necessary to restore the model's performance. matrix multiplication can further be quantised, although quality is sacrificed. unfortunately, 4bit dot products found in matrix multiplication are slow because current hardware does not natively support the necessary 4-bit instructions.