On the Transformer Growth for Progressive BERT Training (2021)

Abstract: As the excessive pre-training cost arouses the need to improve efficiency, considerable efforts have been made to train BERT progressively–start from an inferior but low-cost model and gradually increase the computational complexity. Our objective is to help advance the understanding of such Transformer growth and discover principles that guide progressive training. First, we find that similar to network architecture selection, Transformer growth also favors compound scaling. Specifically, while existing methods only conduct network growth in a single dimension, we observe that it is beneficial to use compound growth operators and balance multiple dimensions (e.g., depth, width, and input length of the model). Moreover, we explore alternative growth operators in each dimension via controlled comparison to give practical guidance for operator selection. In light of our analyses, the proposed method CompoundGrow speeds up BERT pre-training by 73.6% and 82.2% for the base and large models respectively while achieving comparable performances.

## introduction thanks to the rapid increase of computing power, large-scale pre-training has been breaking the glass ceiling for natural language processing tasks @xcite @xcite @xcite . however, with great power comes great challenges: the required excessive computational consumption significantly impedes the efficient iteration of both research exploration and industrial application. to lower the training cost, many attempts have been made to conduct progressive training, which starts from training an inferior but low-cost model, and gradually increases its resource consumption @xcite . as elaborated in section 5, two components are typically needed for designing such progressive training algorithms-the growth scheduler and the growth operator @xcite . the former controls when to conduct network growth, and the latter controls how to perform network growth. here, our objectives are to better understand growth operators with a focus on transformer models @xcite , and specifically, to help design better progressive algorithms for bert pre-training @xcite . specifically, we recognize the importance of using compound growth operators in our study, which balance different model dimensions (e.g., number of layers, the hidden size, and the input sequence length). regarding previous efforts made on transformer growth, they mainly focus on one single model dimension: either the length @xcite or the depth @xcite . in this work, however, we find that compound effect plays a vital role in growing a model to different capacities, just like its importance in deciding network architectures under specific budgets @xcite . here, we show that growing a transformer from both dimensions leads to better performance with less training cost, which verifies our intuition and shows the potential of using compound growth operators in progressive bert training. further, we explore the potential choices of growth operators on each dimension. we conduct controlled experiments and comprehensive analyses to compare various available solutions. these analyses further guide the design of effective compound growth operators. specifically, we observe that, on the length dimension, embedding pooling is more effective than directly truncating sentences. compound effect. existing progressive training methods only focus on one model dimension. for example, gong et al. (2019) conduct transformer growth by gradually increasing the network depth. devlin et al. (2019) use shorter input sequence length at early stages. however, as studies in network architecture search have revealed (tan and le, 2019), growth operators that balance different model dimensions can achieve better performance than single-dimensional operators under the same budget. note that our objective (equation 1 ) is close to the objective of efficientnet @xcite , which aims to find the optimal network architecture by maximizing the model accuracy for a given resource budget: the optimal value of the objective function (equation 1 ) is bounded by the feasible set of the growth operator. empirical verification. for empirical verification, we compare existing single-dimensional growth operators in model depth and length with the corresponding compound operator that balances both dimensions. for all three compared growth operators, their configurations are adjusted to make sure they have the same model after growth, and their low-cost models have empirically comparable training costs. as to the training, we first train the low-cost model for 100/300/500/700k steps, and then grow the model to a standard bert-base model for another 300k steps training. for models trained with different steps/growth operators, we compare their performance after finetuning on mnli, squad v1.1, and squad v2.0 respectively. as figure 1 shows, across different settings (columns) and metrics (rows), the compound operator consistently outperforms or at least achieves comparable results with single-dimensional operators. the observation meets our intuition: to achieve same speedup, the compound method can distribute the reduction on training cost to different dimensions, and achieve better performance. ## explore possible growth operators after verifying the importance of compound growing, we conduct more analysis to provide guidance for growth operator design. ## related work progressive training was originally proposed to improve training stability, which starts from an efficient and small model and gradually increase the model capacity @xcite . recent study leverages this paradigm to accelerate model training. for example, multi-level residual network @xcite explores the possibility of augmenting network depth in a dynamic system of view and transforms each layer into two subsequent layers. autogrow @xcite attempts to automate the discover of proper depth to achieve near-optimal performance on different datasets. lipgrow @xcite proposes a learning algorithm with an automatic growing scheduler for convolution nets. at the same time, many studies have been conducted on the model growing operators. network morphism @xcite manages to grow a layer to multiple layers with the represented function intact. net2net @xcite is a successful application to transfer knowledge to a wider network with function-preserving initializa-tion. similar ideas can be discovered in many network architectures, including progressive growing of gan @xcite and adaptive computation time @xcite . as large-scale pre-training keeps advancing the state-of-the-art @xcite , their overwhelming computational consumption becomes the major burden towards further developing more powerful models @xcite . preliminary application of progressive training has been made on transformer pre-training. @xcite designs two-stage training with a reduced sequence length for the first 90% of updates. @xcite stack shallow model trained weights to initialize a deeper model, which grows the bertbase model on the depth dimension and achieves 25% shorter training time. ## conclusion in this work we empirically verify the importance of balancing different dimensions in transformer growth and propose compound growth operators, which integrates operators for more than one dimension. moreover, we conduct controlled experiments on various design choices of growth operators, which provides a practical guidance to algorithm design. our final model speeds up the training of the bert-base and bert-large models by 73.6% and 82.2% in walltime respectively while achieving comparable performance.