Adder Encoder for Pre-trained Language Model (2023)

Abstract: “BERT, a pre-trained language model entirely based on attention, has proven to be highly per-formant for many natural language understanding tasks. However, pre-trained language mod-els (PLMs) are often computationally expensive and can hardly be implemented with limitedresources. To reduce energy burden, we introduce adder operations into the Transformer en-coder and propose a novel AdderBERT with powerful representation capability. Moreover, weadopt mapping-based distillation to further improve its energy efficiency with an assured perfor-mance. Empirical results demonstrate that AddderBERT6 achieves highly competitive perfor-mance against that of its teacher BERTBASE on the GLUE benchmark while obtaining a 4.9xreduction in energy consumption.”

## introduction the last five years have seen great success achieved by large-scale pre-trained language models, such as bert @xcite , electra @xcite , and gpt3 @xcite . by modeling long-distance dependencies based on self-attention, they can learn powerful language representations from the unlabeled corpus. while these models lead to significant improvement on many downstream tasks (eg., the glue benchmark @xcite ), the growing computation costs have impaired their deployment, especially on limited-resource devices such as mobile phones, ar glasses, and smartwatch. since attending to all tokens yields a complexity of o(n 2 ) with respect to sequence length, prior works aim to investigate efficient transformers with lower complexity. @xcite replaces dot-product attention with one using locality-sensitive hashing. @xcite decomposes the original attention into multiple smaller ones by linear projections. however, they can only solve the problem partway, for the consumption except self-attention has not been changed in the encoder. various attempts also focus on model compression techniques, including quantization @xcite , weights pruning @xcite , and knowledge distillation (kd) @xcite . as one of the most popular methods, kd aims to transfer knowledge from a large teacher network to a small student network, employed by distilbert @xcite , bert-pkd @xcite , tinybert @xcite , and fastbert @xcite . beyond these methods, @xcite proposed adder neural network (addernet), which replaced massive multiplications with cheaper additions to reduce computation costs, and achieved better performance on the imagenet dataset compared to cnns. then researchers attempt to build efficient deep-learning models based on addernet for computer vision tasks @xcite . inspired by this, it is an interesting idea to investigate the feasibility of replacing multiplications with additions in pre-trained language models like bert. in this paper, we first present adderbert, a pre-trained model consisting of several adder encoders, in which key modules including multi-head attention and feed-forward network are implemented with cheaper adder operations. as shown in figure 1 , it also has a unique mapping-based distillation that could make it to be more energy-efficient with an assured performance. finally, we conduct full experiments on the glue benchmark. empirical results demonstrate that our method can achieve comparable performance with the baselines in much lower energy consumption. the contributions are summarized as follows: ## preliminary in this section, we revisit the related works including addernet and knowledge distillation. we are motivated by them to design adderbert. ## method this section describes our proposed adderbert as well as its training method. concisely, adderbert implements the encoder block with cheaper adder operations, and it takes advantage of mapping-based distillation to be better in performance and efficient in energy. ## experiment in this section, we verify the effectiveness of adderbert on three tasks with different model settings. ## conclusion in this paper, we propose an energy-efficient version of bert, called adderbert. specifically, adder-bert consists of several adder encoders implemented by cheap addition operations but has a powerful representation capacity. it adopts a unique mapping-based distillation method to narrow the gap in feature distribution between the teacher and student model. empirical results on the glue benchmark demonstrate that our method can achieve highly competitive performance to the teacher bert base while reducing energy consumption significantly.