Is the Number of Trainable Parameters All That Actually Matters?
Abstract
Recent work has identiﬁed simple empirical scaling laws for language models, linking compute budget, dataset size, model size, and autoregressive modeling loss.
The validity of these simple power laws across orders of magnitude in model scale provides compelling evidence that larger models are also more capable models.
However, scaling up models under the constraints of hardware and infrastructure is no easy feat, and rapidly becomes a hard and expensive engineering problem.
We investigate ways to tentatively cheat scaling laws, and train larger models for cheaper.
We emulate an increase in effective parameters, using efﬁcient approximations: either by doping the models with frozen random parameters, or by using fast structured transforms in place of dense linear layers.
We ﬁnd that the scaling relationship between test loss and compute depends only on the actual number of trainable parameters; scaling laws cannot be deceived by spurious parameters.
Introduction
Predictably linking model and dataset size with generalization error is a long-standing open question.
Discrepancies between classical bias-variance trade-off models and modern practices have been identiﬁed [1], with phenomena such as deep double descent [2] providing glimpses into a deeper understanding.
However, actionable insights for machine learning practitioners have remained elusive.
Recently, simple and general empirical scaling laws for deep learning models have been uncovered.
Starting with ﬁrst relationships for modern convolutional networks [3], guidelines informing optimal architectures–such as EfﬁcientNet [4]–have been derived.
Importantly, these laws can extrapolate model performance across scale, motivating the training of increasingly large and capable models [5].
We focus on seminal work on large generative language models [6], establishing and using scaling laws to predict the computational requirements for a speciﬁc performance level–as pioneered by [7].
This work showed that the performance of an auto-regressive language model follows a remarkably simple relationship, linking model and dataset size, compute budget, and modeling loss across many orders of magnitude in scale.
This make it possible to answer questions central to the efﬁcient training of extreme-scale models, such as: (1) which model size achieves the best loss given a ﬁxed compute budget ; or (2) how many samples are necessary to train a model of a given size optimally .
Unfortunately, these empirical laws also show that training models much larger than current ones comes at a prohibitive cost.
State-of-the-art language models such as GPT-3 have required several thousand PF-days to train [8], and model size has plateaued since then [9, 10].
Proposed distributed sharding methods for the training of such extreme-scale models only bridge the memory gap [11], and do not fundamentally change the required compute budget.
Pending drastic hardware improvements, it is unlikely we will see a 100-1,000x increase in model size as quickly as we have seen in the past.
Accordingly, to enable sustained growth in model size and capabilities, we attempt to cheat scaling laws.
To do so, we propose to fudge the effective number of parameters in the model.
Instead of using only expensive fully trainable parameters, we explore two classes of efﬁcient parameters motivated by potential efﬁcient hardware implementations: either by doping the models with cheaper and simpler frozen parameters, or by implementing structured transforms in place of dense linear layers.
We study these additions under the guise of scaling laws.
Frozen parameters Transformer models with ﬁxed random parameters have previously been studied in Reservoir Transformers [12], where some layers of a RoBERTa model [13] are initialized randomly and then never updated.
Using an area under convergence curve metric, they ﬁnd this approach to lead to more efﬁcient training.
This is motivated by a broader line of work around random projections in machine learning, with applications in random kernel methods [14], unsupervised feature learning [15], and reservoir computing [16].
In a language model, frozen parameters are one third cheaper than fully trainable ones: assuming the backward pass is normally twice the cost in
FLOP of the forward (backpropagation + update) [6], the update step is skipped for frozen parameters.
Furthermore, multiplication with random matrices can be ofﬂoaded to specialized co-processors [17].
Structured parameters Structured efﬁcient linear layer parametrizations have been proposed to accelerate fully-connected layers [18, 19, 20].
They leverage transforms with an efﬁcient O ( n log n ) implementation, such as the Hadamard Transform, Discrete Fourier Transform or the Discrete Cosine
Transform.
These structured layers have far fewer trainable parameters (on the order of n ) but still emulate an operation equivalent to n 2 parameters.
We initially hypothesize that the scaling laws will depend on the the emulated number of parameters n 2 rather than n , reducing the memory needs and compute budget for training the model, while maintaining identical modelling loss.
Contributions We study random and structured parameters in large auto-regressive language models, and show that scaling laws cannot be cheated using spurious parameters.
More speciﬁcally, we ﬁnd that in both cases, the empirical scaling laws obtained depend only on the real number of trainable parameters.
Neither the frozen or structured parameters make pre-training more efﬁcient.
Methods
All experiments are based on the training on a language modeling task of a decoder-only autoregressive Transformer based on the GPT architecture [21].
Our only signiﬁcant deviation from the architecture is the use of rotary embeddings [22], which enable increased performance.
We performed initial pathﬁnding experiments without rotary embeddings and obtained similar results.
Architecture details for the four model sizes considered are available in the supplementary.
The model is trained on French data obtained using a Common Crawl dump ﬁltered by CCNet [23].
We use byte-level byte-pair encoding, with a vocabulary of 50,262 tokens.
Our dataset containsbillion tokens, and is similar to the setup of FlauBERT [24].
All validation and test losses are obtained on the fr-wiki dataset, made of 0.5 billion tokens obtained from a dump of French Wikipedia.
All experiments were run in a distributed setup with four NVIDIA V100-16GB GPUs per node, on the Jean Zay supercomputer.
We use a sharded data parallel strategy for training the larger models, similar to ZeRO-3 [25], and implemented with FairScale [26].
Structured transforms are implemented as optimized CUDA kernels and compiled in PyTorch extensions for best performance.
We estimate the compute budget in PF-day using the formula C =
NBS
[6], where N is the number of trainable non-embedding parameters, B is the number of tokens per batch, S is the number of parameter updates, and 6 is the factor to account for the forward and backward passes.
We veriﬁed our experimental setup, and in particular the choice of a French dataset in place of an English one, by reproducing the scaling law ﬁt of [6] and obtaining a similar exponent – as shown in Figure 1. [Figure]
o
[Figure]
N=100k N=396k
N=991k N=9.5M 3 .
22 · C − 0 .
Similar to [12] we replace the decoder layers–consisting of self-attention followed by a multilayer perceptron (MLP)–with a single frozen MLP. Indeed, we observed that keeping the entire frozen decoder layer lead to similar performances as keeping only the frozen MLP. The latter, which involves random matrix multiplications could more easily be done on a specialized co-processor.
In order to approximately preserve the same total number of parameters, we increase the hidden dimension of the feedforward network from 4 × d model to 6 × d model .
By training models with ﬁxed parameters over multiple orders of magnitude, we establish the scaling laws both for the models with ﬁxed random parameters and the baseline models.
3.2 Results
The cost of the random parameters depends on the choice of hardware, as well as any potential algorithm used to approximate them.
To be able to compare the models with frozen random parameter with the baseline models, we plot the scaling laws for different costs of the random parameters.
Figure 2 shows scaling laws for random parameters costing two thirds as much as trainable parameters, which corresponds to skipping the parameter update step and which is effectively achieved by our implementation.
The baseline models (dotted lines) signiﬁcantly outperform the models with random parameters (solid lines).
Figure 3 shows the scaling behaviour assuming the best-case scenario of free random parameters: doped models reproduce the training curves of a model with the same amount of trainable parameters and no frozen parameters.
We observe the same behavior on different scales
+ I − I − I
+ I
+ I + I −
I − I
+ I − I + I − I + I
+ I + I
+ I of models. In other words, the frozen parameters have no effect at all on the loss-compute curve of these models, and are effectively useless for pre-training.  Structured Efﬁcient Linear Layers Since frozen layers do not contribute to the loss-compute curve, we turn our attention to structured layers.
Structured layers are a natural midpoint between fully frozen and fully trainable layers, implementing a smaller number of structured parameters, typically only linearly increasing with layer width, whereas fully-connected layers will increase quadratically. Importantly, and in contrast with sparse layers, structured layers can be efﬁciently implemented in modern hardware.
4.1 Methods
Random projections can be efﬁciently approximated by alternating diagonal matrices and Fast
Hadamard transformations using FastFood [28], as shown in Figure 4.
A vector can then be projected with time complexity O ( n log n ) , signiﬁcantly speeding up the original O ( n 2 ) operation.
Using the approach of [19], we allow for some adaptability of the frozen layers by training the diagonal matrices.
This operation dramatically reduces the number of parameters from O ( n 2 ) down to O ( n ) .
Since in a language model, n ∼ 1000 , we approximate the cost of the structured operation to zero.
We also investigate using a combination of block diagonal matrices and block Hadamard matrices, as in Figure 5.
The block diagonal matrices can transform nearby features, while the block Hadamard matrix introduce long-range correlations.
The parameter reduction of such a structured MLP is given by the number of blocks, and is tunable.
We set the number of blocks to 4 , maintaining a relatively large number of trainable parameters in the MLP, with a cost per parameter identical to dense layers.
4.2 Results
In Figure 6, the FastFood structured model is supposed to emulate a Small model, but actually performs like an XSmall with a similar number of trainable parameters.
Furthermore, in Figure 7, the behavior of the block structured model closely resembles that of a model with as many trainable parameters.
Accordingly, structured models behave almost identically to fully-trainable models with similar number of trainable parameters, even though both the number of layers and the width of the model is different.
The number of spurious parameters does not matter, and only the real one does.
Conclusion and outlooks
We showed that random parameters and structured transforms do not provide an advantage over dense fully trainable layers in the loss-compute landscape of autoregressive models.
Remarkably, it appears that the scaling law for these models reliably depends only on the actual number of trainable parameters.
Unfortunately, scaling laws are not easily cheated by using spurious parameters.
Our investigation points to interesting questions: is the number of trainable parameters all that actually matters?
And is this true for any model architecture or speciﬁc to autoregressive language models?
Acknowledgments and Disclosure of Funding
We thank Gustave Pariente and the LightOn team for useful feedback and support. This work was granted access to the HPC/AI resources of IDRIS under the allocation 2021-A0101012429 made by GENCI. Speciﬁcally, we thank the team of the Jean Zay supercomputer for their support.
This project has received funding from the European Union’s Horizon 2020 research and innovation program under the Marie Skłodowska-Curie grant agreement
No A Appendix We include all the hyperparameter and architectural details for the baselines (Table 1), the doped models (Table 2, and the structured models (Table 3).
The doped models are built by alternating trainable transformer and frozen MLP layers, so that the ﬁrst and last layers of the models are always trainable.
[Table]