T opic BERT for Energy Efficient Document Classification (2020)

Abstract: Prior research notes that BERT’s computational cost grows quadratically with sequence length thus leading to longer training times, higher GPU memory constraints and carbon emissions. While recent work seeks to address these scalability issues at pre-training, these issues are also prominent in fine-tuning especially for long sequence tasks like document classification. Our work thus focuses on optimizing the computational cost of fine-tuning for document classification. We achieve this by complementary learning of both topic and language models in a unified framework, named TopicBERT. This significantly reduces the number of self-attention operations – a main performance bottleneck. Consequently, our model achieves a 1.4x ( 40%) speedup with 40% reduction in CO2 emission while retaining 99.9% performance over 5 datasets.

## introduction natural language processing (nlp) has recently witnessed a series of breakthroughs by the evolution of large-scale language models (lm) such as elmo @xcite , bert @xcite , roberta @xcite , @xcite etc. due to improved capabilities for language understanding @xcite . however this massive increase in model size comes at the expense of very high computational costs: longer training time, high gpu/tpu memory constraints, adversely high carbon footprints, and unaffordable invoices for small-scale enterprises. figure 1 shows the computational cost (training time: millisecond/batch; co 2 emission, and gpu memory usage) of bert all of which grow quadratically with sequence length (n). we note that this *equal contribution co 2 bert pre-training (nas) (strubell et al., 2019) 626k bert fine-training (@xmath0 recent studies address the excessive computational cost of large language models (lms) in the pre-training stage using two main compression techniques: (a) pruning @xcite ) by reducing model complexity, and (b) knowledge distillation @xcite @xcite which a student model (compact model) is trained to reproduce a teacher (large model) leveraging the teacher's knowledge. finally, in order to process long sequences, @xcite and @xcite investigate simple approaches of truncating or partitioning them into smaller sequences, e.g., to fit within 512 token limit of bert for classification; however, such partitioning leads to a loss of discriminative cross-partition information and is still computationally inefficient. in our work, we address this limitation by learning a complementary representation of text using topic models (tm)) @xcite @xcite . because topic models are bag-of-words based models, they are more computationally efficient than large scale language models that are contextual. our work thus leverages this computational efficiency of tms for efficient and scalable fine-tuning for bert in document classification. specifically our contributions(1) complementary fine-tuning: we present a novel framework: topicbert, i.e., topic-aware bert that leverages the advantages of both neural network-based tm and transformer-based bert to achieve an improved document-level understanding. we report gains in document classification task with full selfattention mechanism and topical information. (2) efficient fine-tuning: topicbert offers an efficient fine-tuning of bert for long sequences by reducing the number of self-attention operations and jointly learning with tm. we achieve a 1.4x ( 40%) speedup while retaining 99.9% of classification performance over 5 datasets. our approaches are model agnostic, therefore we extend bert and distilbert models. code in available at https: //github.com/yatinchaudhary/topicbert. carbon footprint (co 2 ) estimation: we follow @xcite and use ml co 2 impact calculator 1 to estimate the carbon footprint (co 2 ) of our experiments using the following equation: for ## conclusion we have presented two novel architectures: top-icbert and topicdistilbert for an improved and efficient (fine-tuning time/co 2 ) document classification, leveraging complementary learning of topic (nvdm) and language (bert) models. iulia turc, ming-wei chang, kenton lee, and kristina toutanova. 2019. well-read students learn better: the impact of student initialization on knowledge distillation. corr, abs/1908.08962. qizhe xie, zihang dai, eduard hovy, minh-thang luong, and quoc v le. 2019. unsupervised data augmentation for consistency training. zhilin yang, zihang dai, yiming yang, jaime g. carbonell, ruslan salakhutdinov, and quoc v. le. 2019. xlnet: generalized autoregressive pretraining for language understanding. in advances in neural information processing systems 32: annual conference on neural information processing systems 2019, neurips 2019, 8-14 december 2019, vancouver, bc, canada, pages 5754-5764. a supplementary material a.1 co 2 : carbon footprint estimation for