H ybrid BERT - Making BERT Pretraining More Efficient Through Hybrid Mixture of Attention Mechanisms (2024)

Abstract: Pretrained transformer-based language models have produced state-of-the-art performance in most natural language understanding tasks. These models undergo two stages of training: pretraining on a huge corpus of data and fine-tuning on a specific downstream task. The pretraining phase is extremely compute-intensive and requires several high-performance computing devices like GPUs and several days or even months of training, but it is crucial for the model to capture global knowledge and also has a significant impact on the fine-tuning task. This is a major roadblock for researchers without access to sophisticated computing resources. To overcome this challenge, we propose two novel hybrid architectures called HybridBERT (HBERT), which combine self-attention and additive attention mechanisms together with sub-layer normalization. We introduce a computing budget to the pretraining phase, limiting the training time and usage to a single GPU. We show that HBERT attains twice the pretraining accuracy of a vanilla-BERT baseline. We also evaluate our proposed models on two downstream tasks, where we outperform BERT-base while accelerating inference. Moreover, we study the effect of weight initialization with a limited pretraining budget. The code and models are publicly available at: www.github.com/gokulsg/HBERT/.

## introduction the last few years have witnessed ground-breaking research on pretrained transformer-based language models. these large language models usually follow a two-stage training process: the initial pretraining stage for learning global knowledge using large text collections and the later fine-tuning stage for adapting the learned knowledge to a specific task. pretraining is the most crucial and most computationally expensive phase and often requires modern computing devices like gpus or tpus and several weeks or even months. pretraining is thus a major roadblock for researchers who do not have access to sophisticated computing resources. for example, the bert model @xcite was pretrained on 16 tpus for 4 days. such modern computing devices cannot be easily accessed by individual researchers, which limits the freedom of researchers to explore other architectures for a given task and is a hurdle to the development of highly optimized neural architectures. following the scaling laws @xcite , researchers tried to improve the performance by increasing the model size, data volume and training time. this resulted in extremely huge models with several billion parameters. even fine-tuning such enormous language models with a limited compute power is extremely challenging and it is one of the main reasons that motivated researchers to explore alternative approaches to make the best use of the existing pretrained models rather than training from scratch. though approaches like prompt tuning @xcite or adapters @xcite provide competitive results, they restrict any architectural modifications to the underlying pretrained model. making the pretraining process more computationally inexpensive would motivate researchers to explore other architectures. in our work, we try to address this problem by imposing restrictions on computational devices and training time during the pretraining stage of a bert model. we also introduce two new models which we call hybridbert (hbert) and use -for the first time, to the best of our knowledge -a hybrid mixture of self-attention and additive attention together with sub-layer normalization. we show that on a limited time budget of 1 or 2 days, our usage of additive attention and sub-layer normalization increases both the pretraining performance and downstream performance of a reference vanilla bert model on two tasks, namely intent classification and emotion recognition. with our work, we aim to provide a means to researchers without access to large computing devices to pretrain their own high-performance language models. making pretraining more efficient and not dependent on sophisticated computing resources also helps to save cost and lower the emission of co 2 , thus proving to be more environmentally friendly. ## related work the bert model @xcite was pretrained on 16 tpus for 4 days. the equivalent time on 8 nvidia v100 gpus will be 11 days. there has been recent work on introducing restrictions on the bert pretraining: @xcite tried to train bert-large for a single day using 8 v100 gpus. later work @xcite reduced the gpu usage from 8 to 1, still training the model for a single day. inspired by these works, we restrict the pretraining to a single gpu, utilizing one of the most commonly used gpus for pretraining. after the success of transformer-based models on natural language understanding tasks, the field of efficient attention mechanisms came into the spotlight. several works @xcite tried to reduce the quadratic complexity of the self-attention mechanism. few approaches @xcite used hashing techniques to accelerate the self-attention computation. approximating the self-attention mechanisms by low-rank matrices @xcite is also an active research direction. with a linear time complexity, additive attention @xcite proves to be an efficient alternative to self-attention, which is why we employ it in this work. layer normalization is a lightweight component in the bert architecture that can influence the learning capabilities of the model. while the conventional bert-based models use post-layer normalization, the decoder-based model @xcite and vision transformers @xcite show improvements using prelayer normalization. earlier work @xcite suggested pre-layer normalization to be more beneficial during computing resource crunch. recent work @xcite tries to generalize layer normalization across different models by introducing sub-layer normalization. sub-layer normalization has been shown to improve the performance of models on various tasks in the text, speech, and vision domains, which is why we decided to employ it for our work. ## conclusion in this work, we altered the bert architecture by combining self-attention and additive attention and employing sub-layer normalization. our experiments show that on a limited compute budget, our architecture outperforms vanilla bert both during pretraining and fine-tuning. in the future, we would like to study the effect of knowledge distillation from larger teacher models during fine-tuning. also, we want to study the effect of hybrid architecture on decoder-based models. compressing our hybrid models using other model compression approaches is another research direction.