Effects of Parameter Norm Growth During Transformer Training: Inductive Bias from Gradient Descent (2021)

Abstract: The capacity of neural networks like the widely adopted transformer is known to be very high. Evidence is emerging that they learn successfully due to inductive bias in the training routine, typically a variant of gradient descent (GD). To better understand this bias, we study the tendency for transformer parameters to grow in magnitude ( \ell_2 norm) during training, and its implications for the emergent representations within self attention layers. Empirically, we document norm growth in the training of transformer language models, including T5 during its pretraining. As the parameters grow in magnitude, we prove that the network approximates a discretized network with saturated activation functions. Such “saturated” networks are known to have a reduced capacity compared to the full network family that can be described in terms of formal languages and automata. Our results suggest saturation is a new characterization of an inductive bias implicit in GD of particular interest for NLP. We leverage the emergent discrete structure in a saturated transformer to analyze the role of different attention heads, finding that some focus locally on a small number of positions, while other heads compute global averages, allowing counting. We believe understanding the interplay between these two capabilities may shed further light on the structure of computation within large transformers.

## introduction transformer-based models @xcite like bert @xcite , xlnet @xcite , roberta @xcite , and t5 @xcite have pushed the state of the art on an impressive array of nlp tasks. overparameterized transformers are known to be unversal approximators @xcite , suggesting their generalization performance ought to rely on useful biases or constraints imposed by the learning algorithm. despite various attempts to study these biases in transformers @xcite , it remains an interesting open question what they are, or even how to characterize them in a way relevant to the domain of language. in this work, we take the perspective that thoroughly understanding the dynamics of gradient descent (gd) might clarify the linguistic biases of transformers, and the types of representations they acquire. we start by making a potentially surprising empirical observation ( §3): the parameter ℓ 2 norm grows proportional to √ t (where t is the timestep) during the training of t5 @xcite and other transformers. we refer to the phenomenon of growing parameter norm during training as norm growth. previous work has analyzed norm growth in simplified classes of feedforward networks @xcite , but, to our knowledge, it has not been thoroughly demonstrated or studied in the more complicated and practical setting of transformers. our main contribution is analyzing the effect of norm growth on the representations within the transformer ( §4), which control the network's grammatical generalization. with some light assumptions, we prove that any network where the parameter norm diverges during training approaches a saturated network @xcite : a restricted network variant whose discretized representations are understandable in terms of formal languages and automata. empirically, we find that internal representations of pretrained transformers approximate their saturated counterparts, but for randomly initialized transformers, they do not. this suggests that the norm growth implicit in training guides transformers to approximate saturated networks, justifying studying the latter @xcite as a way to analyze the linguistic biases of nlp architectures and the structure of their representations. past work @xcite reveals that saturation permits two useful types of attention heads within a transformer: one that locally targets a small number of positions, and one that attends uniformly over the full sequence, enabling an "average" operation. empirically, we find that both of these head types emerge in trained transformer language models. these capabilities reveal how the transformer can process various formal languages, and could also suggest how it might represent the structure of natural language. combined, our theoretical and empirical results shed light on the linguistic inductive biases imbued in the transformer architecture by gd, and could serve as a tool to analyze transformers, visualize them, and improve their performance. finally, we discuss potential causes of norm growth in §5. we prove transformers are approximately homogeneous @xcite , a property that has been extensively studied in deep learning theory. with some simplifying assumptions, we then show how homogeneity might explain the √ t growth observed for t5. foot_0 ## norm growth in transformers we start with the observation that the parameter ℓ 2 norm grows during training for practical transformer language models. we first consider the parameter norm of 104 historical checkpoints from t5-base @xcite pretraining, a 220m parameter model, which was trained using the adafactor optimizer @xcite . further details are in §a. fig. 1 shows that the t5 norm follows a √ t trend, where t is time in training steps. the top right of fig. 1 breaks down the growth trend by layer. generally, the norm grows more quickly in later layers than in earlier ones, although always at a rate proportional to √ t. 5 next, in the bottom row of fig. 1 , we plot the cosine similarity between each parameter checkpoint θ t+1 and its predecessor θ t . this rapidly approaches 1, suggesting the "direction" of the parameters (θ t / θ t ) converges. the trend in directional convergence looks similar across layers. we also train smaller transformer language models with 38m parameters on wikitext-2 @xcite and the penn treebank (ptb; @xcite . we consider two variants of the transformer: pre-norm and post-norm, which vary in the relative order of layer normalization and residual connections (cf. @xcite . every model exhibits norm growth over training. 6 combined, these results provide evidence that the parameter norm of transformers tends to grow over the course of training. in the remainder of this paper, we will discuss the implications of this phenomenon for the linguistic biases of transformers, and then discuss potential causes of the trend rooted in the optimization dynamics. ## explanation for norm growth we have documented norm growth in t5 and other transformers ( §3) and showed how it induces partial saturation in their representations ( §4). this section points towards an understanding of why the parameter norm grows over the course of training, grounded in results about norm growth from deep learning theory. we do not analyze specific optimizers directly; instead, we analyze norm growth within simplified models of training dynamics taken from the literature. we then evaluate how these candidate dynamics models fit t5's training. ## conclusion we empirically found that θ t grows ∝ √ t during t5 pretraining-a fact that may be caused by the approximate homogeneity of the transformer architecture. we proved that norm growth induces saturation, and then showed empirically that t5 and other large transformers become approximately saturated through their pretraining. examining highly saturated transformer language models, we found the attention heads largely split between two distinct behaviors that can be roughly interpreted as argmax and mean operations. while we lack a precise formal characterization of "semi-saturated" transformers, we conjecture their capacity resembles that of the saturated models. thus, we believe further analyzing the capabilities of saturated attention may clarify the linguistic biases that emerge in transformers through training, and the mechanisms they use to represent linguistic structure.