  In recent times, BERT-based models have been extremely successful in solving
a variety of natural language processing (NLP) tasks such as reading
comprehension, natural language inference, sentiment analysis, etc. All
BERT-based architectures have a self-attention block followed by a block of
intermediate layers as the basic building component. However, a strong
justification for the inclusion of these intermediate layers remains missing in
the literature. In this work we investigate the importance of intermediate
layers on the overall network performance of downstream tasks. We show that
reducing the number of intermediate layers and modifying the architecture for
BERT-Base results in minimal loss in fine-tuning accuracy for downstream tasks
while decreasing the number of parameters and training time of the model.
Additionally, we use the central kernel alignment (CKA) similarity metric and
probing classifiers to demonstrate that removing intermediate layers has little
impact on the learned self-attention representations.
