G host BERT : Generate More Features with Cheap Operations for BERT (2021)

Abstract: Transformer-based pre-trained language models like BERT, though powerful in many tasks, are expensive in both memory and computation, due to their large number of parameters. Previous works show that some parameters in these models can be pruned away without severe accuracy drop. However, these redundant features contribute to a comprehensive understanding of the training data and removing them weakens the model’s representation ability. In this paper, we propose GhostBERT, which generates more features with very cheap operations from the remaining features. In this way, GhostBERT has similar memory and computational cost as the pruned model, but enjoys much larger representation power. The proposed ghost module can also be applied to unpruned BERT models to enhance their performance with negligible additional parameters and computation. Empirical results on the GLUE benchmark on three backbone models (i.e., BERT, RoBERTa and ELECTRA) verify the efficacy of our proposed method.

## introduction recently, there is a surge of research interests in compressing the transformer-based pre-trained language models like bert into smaller ones using various compression methods, i.e., knowledge distillation @xcite @xcite , pruning @xcite , low-rank approximation @xcite , weight-sharing @xcite , dynamic networks with adaptive depth and/or width @xcite @xcite , and quantization @xcite @xcite . previous works show that there are some redundant features in the bert model, and unimportant attention heads or neurons can be pruned away without severe accuracy degradation @xcite . however, for computer vision (cv) tasks, it is shown in @xcite that redundant features in convolutional neural networks also contribute positively to the performance, and using cheap linear operations to generate more ghost feature maps enhances the performance with few additional parameters. on the other hand, it is shown in @xcite @xcite ) that many attention maps in pre-trained language models exhibit typical positional patterns, e.g., diagonal or vertical, which can be easily generated from other similar ones using operations like convolution. based on the above two aspects, in this paper, we propose to use cheap ghost modules on top of the remaining important attention heads and neurons to generate more features, so as to compensate for the pruned ones. considering that the convolution operation (1) encodes local context dependency, as a complement of the global self-attention in transformer models @xcite ; and (2) can generate some bert features like positional attention maps from similar others, in this work, we propose to use the efficient 1-dimensional depthwise separable convolution @xcite as the basic operation in the ghost module. to ensure the generated ghost features have similar scales or ghost features original features dwconv [cls] the the mat [sep] cat sat on [cls] the the mat [sep] cat sat on n d (pruned) mha (pruned) ffn input g-mha softmax g-ffn (a) adding ghost modules {g f,h } f,m @xmath0,@xmath1. dwconv [cls] the the mat [sep] cat sat on [cls] the the mat [sep] as the original ones, we use a softmax function to normalize the convolution kernel. afterwards, we fine-tune the parameters in both the bert backbone model and the added ghost modules. note that the ghost modules are not necessarily applied to pruned models. they can also be directly applied to pre-trained language models for better performance while with negligible additional parameters and floating-point operations (flops). figure 1 summarizes the average accuracy versus parameter size and flops on the glue benchmark, where adding ghost modules to both the unpruned (@xmath2. more experiments on the glue benchmark show that with only 0.4% more parameters and 0.9% more flops, the proposed ghost modules improve the average accuracy of bert-base, roberta-base and electra-small by 0.9, 0.6, 2.4 points, respectively. when applying ghost modules to small or pruned models, the resultant models outperform other bert compression methods. ## approach in this section, we first introduce where to add ghost modules in a bert model (section 2.1), and then discuss the components and optimization details of the ghost module (section 2.2). ## experiment in this section, we show the efficacy of the proposed method with (pruned) bert @xcite , roberta @xcite and elec-tra @xcite ) as backbone models. ## discussion in this section, we discuss about different choices of which type of convolution to use in the ghost module (section 4.1), and where to posit the ghost modules in a bert model (section 4.2). ## conclusion in this paper, we propose ghostbert to generate more features in pre-trained model with cheap operations. we use the softmax-normalized 1-dimensional convolutions as ghost modules and add them to the output of the mha and ffn of each transformer layer. empirical results on bert, roberta and electra demonstrate that adding the proposed ghost modules enhances the representation power and boosts the performance of the original model by supplying more features.