B inary BERT : Pushing the Limit of BERT Quantization (2021)

Abstract: The rapid development of large pre-trained language models has greatly increased the demand for model compression techniques, among which quantization is a popular solution. In this paper, we propose BinaryBERT, which pushes BERT quantization to the limit by weight binarization. We find that a binary BERT is hard to be trained directly than a ternary counterpart due to its complex and irregular loss landscape. Therefore, we propose ternary weight splitting, which initializes BinaryBERT by equivalently splitting from a half-sized ternary network. The binary model thus inherits the good performance of the ternary one, and can be further enhanced by fine-tuning the new architecture after splitting. Empirical results show that our BinaryBERT has only a slight performance drop compared with the full-precision model while being 24x smaller, achieving the state-of-the-art compression results on the GLUE and SQuAD benchmarks. Code will be released.

## introduction recent pre-trained language models have achieved remarkable performance improvement in various natural language tasks @xcite . however, the improvement generally comes at the cost of increasing model size and computation, which limits the deployment of these huge pre-trained language models to edge devices. various methods have been recently proposed to compress these models, such as knowledge distillation @xcite @xcite , pruning @xcite , low-rank approximation @xcite , weightsharing @xcite @xcite , dynamic networks with adaptive depth and/or width @xcite @xcite , and quantization (zafrir et al., 2019; @xcite @xcite . among all these model compression approaches, quantization is a popular solution as it does not require designing a smaller model architecture. instead, it compresses the model by replacing each 32-bit floating-point parameter with a low-bit fixedpoint representation. existing attempts try to quantize pre-trained models @xcite @xcite to even as low as ternary values (2-bit) with minor performance drop @xcite . however, none of them achieves the binarization (1-bit). as the limit of quantization, weight binarization could bring at most 32× reduction in model size and replace most floating-point multiplications with additions. moreover, quantizing activations to 8-bit or 4-bit further replaces the floating-point addition with int8 and int4 addition, decreasing the energy burden and the area usage on chips @xcite . in this paper, we explore to binarize bert parameters with quantized activations, pushing bert quantization to the limit. we find that directly training a binary network is rather challenging. according to figure 1 , there is a sharp performance drop when reducing weight bit-width from 2-bit to 1-bit, compared to other bit configurations. to explore the challenges of binarization, we analyze the loss landscapes of models under different precisions both qualitatively and quantitatively. it is found that while the full-precision and ternary (2bit) models enjoy relatively flat and smooth loss surfaces, the binary model suffers from a rather steep and complex landscape, which poses great challenges to the optimization. motivated by the above empirical observations, we propose ternary weight splitting, which takes the ternary model as a proxy to bridge the gap between the binary and full-precision models. specifically, ternary weight splitting equivalently converts both the quantized and latent full-precision weights in a well-trained ternary model to initialize binarybert. therefore, binarybert retains the good performance of the ternary model, and can be further refined on the new architecture. while neuron splitting is previously studied @xcite for full-precision network, our ternary weight splitting is much more complex due to the additional equivalence requirement of quantized weights. furthermore, the proposed bi-narybert also supports adaptive splitting. it can adaptively perform splitting on the most important ternary modules while leaving the rest as binary, based on efficiency constraints such as model size or floating-point operations (flops). therefore, our approach allows flexible sizes of binary models for various edge devices' demands. empirical results show that binarybert split from a half-width ternary network is much better than a directly-trained binary model with the original width. on the glue and squad benchmarks, our binarybert has only a slight performance drop compared to the full-precision bert-base model, while being 24× smaller. moreover, bi-narybert with the proposed importance-based adaptive splitting also outperforms other splitting criteria across a variety of model sizes. ## difficulty in training binary bert in this section, we show that it is challenging to train a binary bert with conventional binarization approaches directly. before diving into details, we first review the necessary backgrounds. we follow the standard quantization-aware training procedure @xcite . specifically, given weight @xmath0.k.a latent full-precision weights), each forward propagation quantizes it to ŵ = q(w) by some quantization function q(•), and then computes the loss ℓ( ŵ) at ŵ. during back propagation, we use ∇ℓ( ŵ) to update latent fullprecision weights w due to the non-differentiability of q(•), which is known as the straight-through estimator @xcite . recent ternarybert @xcite follows ternary-weight-network (twn) @xcite to quantize the elements in w to three values {±α, 0}. to avoid confusion, we use superscript t and b for the latent full-precision weights and quantized weights in ternary and binary models, respectively. specifically, twn ternarizes each element w t i in the ternary weight w t as binarization. binarization is first proposed in @xcite and has been extensively studied in the academia @xcite @xcite . as a representative work, binary-weight-network (bwn) @xcite binarizes w b elementwisely with a scaling parameter α as follows: despite the appealing properties of network binarization, we show that it is non-trivial to obtain a binary bert with these binarization approaches. ## experiments in this section, we empirically verify our proposed approach on the glue @xcite and squad @xcite benchmarks. we first introduce the experimental setup in section 4.1, and then present the main experimental results on both benchmarks in section 4.2. we compare with other state-of-the-arts in section 4.3, and finally provide more discussions on the proposed methods in section 4.4. code is available at https://github.com/huawei-noah/ pretrained-language-model/tree/master/ binarybert . ## conclusion in this paper, we propose binarybert, pushing bert quantization to the limit. as a result of the steep and complex loss landscape, we find directly training a binarybert is hard with a large performance drop. we thus propose a ternary weight splitting that splits a trained ternary bert to initialize binarybert, followed by fine-tuning for further refinement. our approach also supports adaptive splitting that can tailor the size of binary-bert based on the edge device constraints. empirical results show that our approach significantly outperforms vanilla binary training, achieving stateof-the-art performance on bert compression. a. fan, e. grave, and a. joulin. 2019. reducing transformer depth on demand with structured dropout. in international conference on learning representations. a. fan, p. stock, b. graham, e. grave, r. gribonval, h. jegou, and a. joulin. 2020. training with quantization noise for extreme model compression. preprint arxiv:2004.07320. h. habi, r. jennings, and a. netzer. 2020. hmq: hardware friendly mixed precision quantization block for cnns. in european conference on computer vision, pages 448-463. y. hao, l. dong, f. wei, and k. xu. 2019. visualizing and understanding the effectiveness of bert. in conference on empirical methods in natural language processing. l. hou, z. huang, l. shang, x. jiang, x. chen, and q. liu. 2020. dynabert: dynamic bert with adaptive width and depth. in advances in neural information processing systems. l. hou and j. t. kwok. 2018. loss-aware weight quantization of deep networks. in international conference on learning representations. l. hou, yao q., and j. t. kwok. 2017. loss-aware binarization of deep networks. in international conference on learning representations. z. huang, l hou, l. shang, x. jiang, x. chen, and q. liu. 2021. ghostbert: generate more features with cheap operations for bert. in annual meeting of the association for computational linguistics. i. hubara, m. courbariaux, d. soudry, r. el-yaniv, and y. bengio. 2016. binarized neural networks. in advances in neural information processing systems. x. jiao, y. yin, l. shang, x. jiang, x. chen, l. li, f. wang, and q. liu. 2020. tinybert: distilling bert for natural language understanding. in findings of empirical methods in natural language processing. d. kim, k singh, and j. choi. 2020. learning architectures for binary networks. in european conference on computer vision, pages 575-591. h. kim, k. kim, j. kim, and j. kim. 2019. binaryduo: reducing gradient mismatch in binary activation network by coupling binary activations. in international conference on learning representations. z. lan, m. chen, s. goodman, k. gimpel, p. sharma, and r. soricut. 2020. albert: a lite bert for selfsupervised learning of language representations. in international conference on learning representations. f. li, b. zhang, and b. liu. 2016. ternary weight networks. preprint arxiv:1605.04711. h. li, z. xu, g. taylor, c. studer, and t. goldstein. 2018. visualizing the loss landscape of neural nets. in advances in neural information processing systems. y. li, x. dong, and w. wang. 2020a. additive powersof-two quantization: a non-uniform discretization for neural networks. in international conference on learning representations. y. li, w. wang, h. bai, r. gong, x. dong, and f. yu. 2020b. efficient bitwidth search for practical mixed precision neural network. preprint arxiv:2003.07577. h. liu, k. simonyan, and y. yang. 2019. darts: differentiable architecture search. in international conference on learning representations.