DACT - BERT : Differentiable Adaptive Computation Time for an Efficient BERT Inference (2022)

Abstract: Large-scale pre-trained language models have shown remarkable results in diverse NLP applications. However, these performance gains have been accompanied by a significant increase in computation time and model size, stressing the need to develop new or complementary strategies to increase the efficiency of these models. This paper proposes DACT-BERT, a differentiable adaptive computation time strategy for BERT-like models. DACT-BERT adds an adaptive computational mechanism to BERT’s regular processing pipeline, which controls the number of Transformer blocks that need to be executed at inference time. By doing this, the model learns to combine the most appropriate intermediate representations for the task at hand. Our experiments demonstrate that our approach, when compared to the baselines, excels on a reduced computational regime and is competitive in other less restrictive ones. Code available at https://github.com/ceyzaguirre4/dact_bert .

## introduction the use of pre-trained language models based on large-scale transformers @xcite has gained popularity after the release of bert @xcite . the usual pipeline consists of finetuning bert by adapting and retraining its classification head to meet the requirements of a specific nlp task. unfortunately, the benefits of using a powerful model are also accompanied by a highly demanding computational load. in effect, current pre-trained language models such as bert have millions of parameters, making them computationally intensive both during training and inference. while high accuracy is usually the ultimate goal, computational efficiency is also desirable. the use of a demanding model not only causes longer processing times and limits applicability to lowend devices, but it also has major implications * work done at pontificia universidad católica de chile. in terms of the environmental impact of ai technologies @xcite . as an example, @xcite provides an estimation of the carbon footprint of several large nlp models, including bert, concluding that they are becoming unfriendly to the environment. recent works have shown that behind bert's immense capacity, there is considerable redundancy and over-parametrization @xcite . consequently, others works have explored strategies to develop efficient and compact versions of bert. one such strategy known as dynamic transformers consists of providing bert with an adaptive mechanism to control how many transformers blocks are used @xcite @xcite . in this paper, we present dact-bert, an alternative to current dynamic transformers that uses an adaptive computation time (act) mechanism @xcite to control the complexity of the processing pipeline of bert. this mechanism controls the number of transformer blocks executed at inference time by using additional classifiers. this allows resulting models to take advantage of the information already encoded in intermediate layers without the need to run all layers. specifically, our model integrates dact, a fully differentiable variant of the adaptive computation module @xcite ) that allows us to train a halting neuron after each transformer block. this neuron indicates the confidence the model has on returning the correct answer after executing said block. we use the dact algorithm to determine when the answer stabilizes in a given output using the halting neuron and halt once it is sure running more blocks cannot change the output. ## related work several architectures have been designed to avoid overcomputing in transformer-based models. according to @xcite , there are two groups. ## dact-bert: differentiable adaptive computation time for bert dynamic early stopping methods use a proxy of model confidence to decide when it is safe to cut computation. in this work our signaling module, dact, approximates this gating mechanism with a soft variant that allows our model to independently learn the confidence function. this mechanism can then be used to detect when stable results are obtained, allowing for the reduction of the total number of steps necessary for a given prediction. the original formulation of dact (eyzaguirre and soto, 2020) applies this module to recurrent models. in our case, we adapt the formulation to the case of transformer based architectures, mainly bert. ## conclusions this work explored the value of using the dact algorithm with pre-trained transformer architectures. this results in a fully differentiable model that explicitly learns how many transformers blocks it needs to perform a specific task. our results show that our approach, dact-bert, outperforms the current dynamic transformer architectures in several tasks when significantly reducing computation.