Partially-Random Initialization: A Smoking Gun for Binarization Hypothesis of BERT (2022)

Abstract: In the past few years, pre-trained BERT has become one of the most popular deep-learning language models due to their remarkable performance in natural language processing (NLP) tasks. However, the superior performance of BERT comes at the cost of high computational and memory complexity, hindering its envisioned widespread deployment in edge devices with limited computing resources. Binarization can alleviate these limitations by reducing storage requirements and improving computing performance. However, obtaining a comparable accuracy performance for binary BERT w.r.t. its full-precision counterpart is still a difficult task. We observe that direct binarization of pre-trained BERT provides a poor initialization during the fine-tuning phase, making the model incapable of achieving a decent accuracy on downstream tasks. Based on this observation, we put forward the following hypothesis : partially randomly-initialized BERT with binary weights and activations can reach to a decent accuracy performance by distilling knowledge from the its full-precision counterpart. We show that BERT with pre-trained embedding layer and randomly-initialized encoder is a smoking gun for this hypothesis. We identify the smoking gun through a series of experiments and show that it yields a new set of state-of-the-art results on the GLUE and SQuAD benchmarks.

## introduction in the past few years, pre-trained language models, particularity @xcite , have shown a great success across various natural language processing (nlp) tasks. as such, it is envisioned that bert will be widely deployed in edge devices with limited computing resources in the near future. such a widespread deployment, however, requires a significant decrease in both memory storage and computational complexity of bert. therefore, model compression techniques, such as pruning @xcite , quantization @xcite , parameter sharing @xcite and distillation @xcite , have been an active field of studies among practitioners to meet the required computing constraints of edge devices. among the aforementioned compression methods, quantization is considered as an effective edge computing approach that can reduce the model size by up to 32× and replace expensive floating-point multiplications with simpler fixed-point counterparts. however, quantization of bert comes at the cost of a severe performance degradation due to the low bit-width representation @xcite . to compensate for the performance drop of quantization in bert, knowledge distillation (kd) is used an auxiliary optimization approach, which encourages the quantized bert to mimic the behavior of its full-precision counterpart. extreme 1-bit quantization, which is commonly referred to as binary representation, is the ideal scenario for edge computing due to the significant decrease in the model size and the computation gain from the bit-wise operations. however, binary representation of weights and activations in bert while obtaining a comparable performance is still a challenging task even when the binarization process is assisted with kd. in fact, binarization of activations in bert accounts for most of the performance drop in nlp tasks @xcite . therefore, previous works mainly focused on binarization of weights in the encoder and word embedding layer of bert @xcite . in this paper, we first investigate the root cause of the performance drop in bert with binary weights and activations. our finding shows that the binarization of pre-trained bert completely undermines the good initialization point obtained from pretraining, leading to its poor generalization on unseen data. moreover, the binary representation con-straint significantly reduces the learning capacity of bert, making kd incapable of finding the right optimization direction during the fine-tuning stage. motivated by these observations, we then propose a hypothesis stating that introducing some degrees of freedom to the binarized network suffices to fully distill knowledge from its full-precision counterpart using kd. we introduce such a freedom by randomly initializing some parts of the binarized network, allowing it to mainly focus on minimizing the total loss with less concern about preserving information from pre-training. we finally search for a smoking gun of this hypothesis and identify it as binary bert whose weights are initialized randomly except for its embedding weights which are initialized from pre-training. a smoking gun is an undeniable piece of empirical evidence that proves our hypothesis. we refer to the binary bert obtained from the smoking gun as sgbert. we empirically support the proposed hypothesis and the importance of this fortuitous initialization of sgbert through a series of experiments. the empirical results show that our sgbert outperforms existing binary bert models and achieves a new set of state-of-the-art results on the glue and squad benchmarks. more specifically, our sgbert outperforms bibert @xcite by 15% in terms of the average accuracy of the glue benchmark. the accuracy performance and the hardware benefits (i.e., the binary representation and the bit-wise operations) of sgbert show its great potential for adoption in edge computing. ## our binary bert architecture bert architecture is consisted of three main parts: embedding layer, encoder and classifier. the encoder module itself contains two main components: multi-head attention (mha) module and feedforward network (ffn). we binarize the weights of the aforementioned components using the sign function as follows: activations in bert are also binarized in a similar manner to weights using the sign function in the forward path and its gradient is computed using the straight-through estimator (ste) @xcite in the backward propagation, i.e., given the unipolar input a, the bipolar input b, and the result of multiplication as c, the least significant bit of c is equal to a and the most significant bit of c is equal to a • not(b) where • and not denote the bit-wise an d and logical complement operations, respectively. figure 1 shows the details of the attention computations in the binary bert. the rest of the encoder module contains linear layers which are binarized using eq. ( 1 ) and eq. ( 2 ). ## knowledge distillation in this section, we describe the kd method used in this paper. in general, we use kd for the embedding output, hidden states and logits. ## the proposed binarization method the unsupervised pre-training on enormous unlabeled data is the key factor in the remarkable success of bert in the context of few-shot learning across various nlp tasks. in fact, the pre-training process provides a good initial point for bert that leads to a better generalization on unseen data of downstream tasks. the fine-tuning procedure of bert is performed by adjusting the pre-training weights to minimize the loss of downstream tasks. it is worth mentioning that these adjustments are usually small such that even fine-tuned parameters of bert on a specific task can be used as an initialization point for another task while still achieving a decent performance. for instance, since the good performance of bert relies on its parameters obtained from pre-training, binarization has been always integrated into the finetuning stage in previous studies @xcite @xcite . however, fully binarization of bert (i.e., binarization of its weights and activations) using this approach results in a huge performance drop @xcite . we attribute this issue to disruption of the pretraining initialization incurred by fully binarization of bert. in other words, the fully binary model cannot hold on to its information from pre-training with binary weights and activations. for instance, the binary bert in during the past few years, distillation-in isolation-has been used as a compression technique where a small student model (which is in full-precision) is trained to mimic the behavior of a larger teacher model. in this compression method, the student model can have a similar structure @xcite @xcite to or a different architecture @xcite ) from its teacher model. for instance, in bert-pkd @xcite , the student model only differs in the number layers from its teacher model (i.e., bert) and kd was used to distill knowledge from the bert's intermediate layer during the finetuning phase. when the student model's structure is similar to its teacher model, it is a common approach to initialize the student model's parameters from the pre-trained bert (e.g., bert-pkd). on the other hand, when the student model's structure is different from its teacher model, its parameters are initialized either partially from the pre-trained bert or randomly (e.g., adabert @xcite ). in either case (i.e., having a similar or different structure), the student model can acquire sufficient knowledge from its teacher model to perform well on downstream tasks. based on these distillation works, we put forward the binarization hypothesis. ## experiments in this section, we perform empirical experiments to verify the effectiveness of the identified smoking gun as the supporting example for our proposed hypothesis on the glue @xcite and squad @xcite benchmarks. we first define our experimental setup. we then present our main experimental results on the aforementioned benchmarks and compare them with stateof-the-art quantized models. finally, we conduct an ablation study to further discuss the importance of initialization when binarizing bert. ## conclusion in this paper, we showed that fully binarization of bert initialized with parameters from pre-training generalizes poorly. based on this observation, we put forward a hypothesis stating that partiallyrandom initialization of bert leads to a better generalization in the context of binarization. we identified the best initialization approach for training binary bert is to initialize the embedding layer's parameters from pre-training while randomly initializing the remaining modules (i.e., the mha, ffn and classifier modules). we referred to this fortuitous initialization as the smoking gun that supports our hypothesis. given the smoking gun, we managed to outperform existing fully binary models and achieve a new set of state-of-the-art results on both glue and squad v2.0 benchmarks.