Towards Fine-tuning Pre-trained Language Models with Integer Forward and Backward Propagation (2023)

Abstract: The large number of parameters of some prominent language models, such as BERT, makes their fine-tuning on downstream tasks computationally intensive and energy hungry. Previously researchers were focused on lower bit-width integer data types for the forward propagation of language models to save memory and computation. As for the backward propagation, however, only 16-bit floating-point data type has been used for the fine-tuning of BERT.In this work, we use integer arithmetic for both forward and back propagation in the fine-tuning of BERT.We study the effects of varying the integer bit-width on the model’s metric performance. Our integer fine-tuning uses integer arithmetic to perform forward propagation and gradient computation of linear, layer-norm, and embedding layers of BERT.We fine-tune BERT using our integer training method on SQuAD v1.1 and SQuAD v2., and GLUE benchmark. We demonstrate that metric performance of fine-tuning 16-bit integer BERT matches both 16-bit and 32-bit floating-point baselines. Furthermore, using the faster and more memory efficient 8-bit integer data type, integer fine-tuning of BERT loses an average of 3.1 points compared to the FP32 baseline.

## introduction over the past few years, integration of attention mechanisms into deep learning models led to the creation of transformer based models. bert @xcite is a prominent transformer based language model which has shown state-of-the-art performance in natural language processing (nlp) tasks. bert requires high memory and computational resources due to its large number of parameters. having large number of parameters incurs challenges for inference, training, and also fine-tuning * equal contribution. of this model. moreover, the training phase i.e. pretraining and fine-tuning, involves more operations compared to the inference. more specifically, the training phase includes gradient computation and weight update that make the training more computationally intensive. one method of reducing the computational complexity of deep learning models is to represent their parameters and activations in low bit-width data types. this reduces the memory footprint of the model and enables more efficient computations. for instance, figure 1 shows that low-bit integer data types have higher throughput and better energy consumption compared to floating-point. previous research attempts at integer quantization of transformer based language models were only focused on forward propagation and the gradient computation were kept in 32-bit floating-point data type (fp32) @xcite @xcite . furthermore, earlier efforts for using low bitwidth data types for gradient computation of transformer based language models has only been limited to . this method, known as mixed precision training @xcite , uses fp16 data type to represent weights, activations and gradients while using fp32 for the weight update. here we present an integer fine-tuning method for transformer based language models such as bert. unlike previous works, we use integer data types for both forward propagation and gradient computation during the fine-tuning of bert. moreover, we use the dynamic fixed-point format to represent floating-point numbers as integers. our integer mapping strategy can be used alongside floating-point numbers in fine-tuning and inference similar to mixed precision training. in our proposed strategy, the arithmetic of all the compute intensive layers for both forward and back propagation are performed using integer arithmetic while other components of the model, such as nonlinear functions and the weight updates are kept in fp32. we use integer versions of compute intensive layers such as linear, normalization (layer-norm), and embedding layers. we study the effect of various bit-widths of the integer input activation and show that increasing the bit-width of the fixed-point mapping function improves the convergence behaviour of the model. this enables us to find the minimum bit-width required for integer fine-tuning of bert. our fine-tuning experiments show that 16-bit integer bert is able to match the metric performance of mixed precision fp16 and fp32 methods. we also further reduce the bit-widths and show that integer fine-tuning of bert with 8-bit integer weights and 12-bit integer activations has a score drop of 3.1 compared to the original performance. to summarize, this paper makes the following contributions: the rest of this paper is structured as follows. section 2 briefly discusses previous works in which low bit-width data types are used for inference and training of deep learning models. section 3 provides details of our integer fine-tuning method, including the representation mapping functions and integer-only layers. the convergence behaviour of the dynamic fixed-point mapping is studied in section 4 by providing empirical observations and theoretical analysis. the fine-tuning experiments on various integer and floating-point setups are presented in section 5. finally, section 6 concludes the ideas proposed in this work. ## related works in this section we discuss the previous works that use low bit-width data types in transformer based language models. these works could be categorized into two major groups. in the first group, called low-bit inference, the low bit-width data types are used only in the forward propagation phase to improve computational complexity and reduce memory usage during the inference. in the second group, also known as low-bit training, lower bit-width data types are used for both the forward and back propagation phases. ## conclusion we proposed an integer fine-tuning method for transformer based language models using dynamic fixed-point format. we used dynamic fixed-point data type to represent parameters, input activations and gradients in integer values. as a result, our fine-tuning method uses integer arithmetic for the forward and back propagation of compute intensive layers such as linear, layer-norm and embedding layers of bert model. furthermore, we studied that increasing the bit-width of the dynamic fixedpoint format reduces the variance of the mapping function and thus, improves the convergence of our integer fine-tuning method. we conduct fine-tuning experiments on glue benchmark and squad datasets to compare the metric performance of our integer bert with fp16 mixed precision and fp32 fine-tuning methods. our experiments show that the 16-bit integer fine-tuning is able to achieve the same metric performance as the fp16 mixed precision fine-tuning method. in addition, fine-tuning bert with lower bit-width data types, i.e. 8-bit integer, maintains an average drop of metric score within 3.1 points of the fp16 setup.