As the excessive pre-training cost arouses the need to improve efficiency , considerable efforts have been made to train BERT progressively--start from an inferior but low-cost model and gradually  increase the computational complexity. Our objective is to help advance the understanding of such Transformer growth and discover principles that guide progressive training. First, we find that similar to network architecture selection , Transformer growth also favors compound scaling. Specifically, while existing methods only conduct network growth in a single dimension, we observe that it is beneficial to use compound growth operators and balance multiple dimensions (e.g., depth, width, and input length of the model). Moreover, we explore alternative growth operators in each dimension via controlled comparison to give practical guidance for operator selection . In light of our analyses, the proposed method CompoundGrow speeds up BERT pre-training by 73.6\% and 82.2\% for the base and large models respectively  while achieving comparable performances . Code will be released for reproduction and future studies.
Due to the excessive cost of large-scale language model pre-training , considerable efforts have been made to train BERT progressively -- start from an inferior but low-cost model and gradually grow the model to increase the computational complexity. Our objective is to advance the understanding of Transformer growth and discover principles that guide progressive training. First, we find that similar to network architecture search , Transformer growth also favors compound scaling. Specifically, while existing methods only conduct network growth in a single dimension, we observe that it is beneficial to use compound growth operators and balance multiple dimensions (e.g., depth, width, and input length of the model). Moreover, we explore alternative growth operators in each dimension via controlled comparison to give operator selection practical guidance . In light of our analyses, the proposed method speeds up BERT pre-training by 73.6\% and 82.2\% for the base and large models respectively , while achieving comparable performances 
R
As the excessive pre-training cost arouses the need to improve efficiency
Due to the excessive cost of large-scale language model pre-training
R
progressively--start
progressively -- start
A
grow the model to
D
help
D
such
R
selection
search
R
practical guidance for operator selection
operator selection practical guidance
D
CompoundGrow
A
,
D
. Code will be released for reproduction and future studies.