Although Bidirectional Encoder Representations from Transformers (BERT) have achieved tremendous success in many natural language processing (NLP) tasks, it remains a black box , so much previous work has tried to lift the veil of BERT and understand the functionality of each layer . In this paper, we found that removing or duplicating most layers in BERT would not change their outputs . This fact remains true across a wide variety of BERT-based models  . Based on this observation, we propose a quite simple method to boost the performance of BERT. By duplicating some layers in the BERT-based models to make it deeper (no extra training required in this step), they obtain better performance in the down-stream tasks after fine-tuning.
Although Bidirectional Encoder Representations from Transformers (BERT) have achieved tremendous success in many natural language processing (NLP) tasks, it remains a black box . A variety of previous works have tried to lift the veil of BERT and understand each layer's functionality . In this paper, we found that surprisingly the output layer of BERT can reconstruct the input sentence by directly taking each layer of BERT as input, even though the output layer has never seen the input other than the final hidden layer . This fact remains true across a wide variety of BERT-based models , even when some layers are duplicated . Based on this observation, we propose a quite simple method to boost the performance of BERT. By duplicating some layers in the BERT-based models to make it deeper (no extra training required in this step), they obtain better performance in the downstream tasks after fine-tuning.
R
, so much previous work has
. A variety of previous works have
R
the functionality of each layer
each layer's functionality
R
removing or duplicating most layers in BERT would not change their outputs
surprisingly the output layer of BERT can reconstruct the input sentence by directly taking each layer of BERT as input, even though the output layer has never seen the input other than the final hidden layer
A
, even when some layers are duplicated
R
down-stream
downstream