How Many Layers and Why? A n Analysis of the Model Depth in Transformers (2021)

Abstract: In this study, we investigate the role of the multiple layers in deep transformer models. We design a variant of Albert that dynamically adapts the number of layers for each token of the input. The key specificity of Albert is that weights are tied across layers. Therefore, the stack of encoder layers iteratively repeats the application of the same transformation function on the input. We interpret the repetition of this application as an iterative process where the token contextualized representations are progressively refined. We analyze this process at the token level during pre-training, fine-tuning, and inference. We show that tokens do not require the same amount of iterations and that difficult or crucial tokens for the task are subject to more iterations.

## introduction transformers are admittedly over-parametrized @xcite @xcite . yet the role of this over-parametrization is not well understood. in particular, transformers consist of a fixed number of stacked layers, which are suspected to be highly redundant @xcite and to cause over-fitting @xcite . in this paper we provide a study on the role of the multiple layers traditionally used. the mechanism of transformer layers is often compared to intuitive nlp pipelines @xcite . starting with the lower layers encoding surface information, middle layers encoding syntax and higher layers encoding semantics @xcite . transformers progressively refine the features, which become more fine-grained at each iteration @xcite . however, albert @xcite highlights that it is possible to tie weights across layers and repeat the application of the same function. consequently, we hypothesize that it is the number of layer applications that gradually abstracts the surface information into semantic knowledge. to better study the transformation of token representations across layers, we propose a variant of albert. our model implements the key specificity of weights tying across layers but also dynamically adapts the number of layers applied to each token. since all layers share the same weight, we refer to the application of the layer to the hidden states as an iteration. after reviewing the related work (section 2), we detail the model and the training methodology in section 3. in particular, we encourage our model to be parsimonious and limit the total number of iterations performed on each token. in section 4, we analyze iterations of the model during pre-training, fine-tuning and inference. ## related work adapting the transformer depth is an active subject of research. in particular, deep transformer models are suspected to struggle to adapt to different levels of difficulty. while large models correctly predict difficult examples, they over-calculate simpler inputs @xcite . this issue can be addressed using early-stopping: some samples might be sufficiently simple to classify using intermediate features. some models couple a classifier to each layer @xcite @xcite . after each layer, given the classifier output, the model either immediately returns the output or passes the sample to the next layer. exiting too late may even have negative impacts due to the network "over-thinking" the input @xcite . ongoing research also refines the application of layers at the token level. @xcite build sentence embeddings by combining token representations from distinct layers. @xcite and @xcite successfully use dynamic layers depth at the token level for full transformers (encoder-decoder). however, to the best of our knowledge, our attempt is the first to apply such mechanism to encoder only transformers and to provide an analysis of the process. ## method in this section, we detail the model architecture, illustrated in figure 1 , and pre-training procedure. ## experiments we now analyze our iterative model properties during pre-training (section 4.1) and fine-tuning (section 4.2). we start by describing the setup for each of the subtasks. ## conclusion we investigated the role of the layers in deep transformers. we designed an original model that progressively transforms each token through a dynamic number of iterations. we analyzed the distribution of these iterations during pre-training and confirmed the results obtained by analyzing the distribution of attention across bert layers, particularly the specific behavior played by special tokens. moreover, we observed that key tokens for the prediction task benefit from more iterations. we confirmed this observation during fine-tuning, where the tokens with a large number of iterations are also suspected to be key for achieving the task. our experiments provide a new interpretation path for the role of layers in deep transformer models. rather than extracting some specific features at each stage, layers could be interpreted as the iteration from an iterative and convergence process. we hope that this can help to better understand the convergence mechanisms for transformers models, reduce the computational footprint or provide new regularization methods.