Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling? (2023)

Abstract: There have been a lot of interest in the scaling properties of Transformer models. However, not much has been done on the front of investigating the effect of scaling properties of different inductive biases and model architectures. Do model architectures scale differently? If so, how does inductive bias affect scaling behaviour? How does this influence upstream (pretraining) and downstream (transfer)? This paper conducts a systematic study of scaling behaviour of ten diverse model architectures such as Transformers, Switch Transformers, Universal Transformers, Dynamic convolutions, Performers, and recently proposed MLP-Mixers. Via extensive experiments, we show that (1) architecture is an indeed an important consideration when performing scaling and (2) the best performing model can fluctuate at different scales. We believe that the findings outlined in this work has significant implications to how model architectures are currently evaluated in the community.

## introduction there have been a lot recent interest in the scaling properties of transformer models @xcite @xcite @xcite . however, not much is understood about the scaling properties of different inductive biases imposed by model architectures. improvements at a a specific scale (compute, size etc) are often assumed to transfer to different scales and compute regions @xcite @xcite and new research is often presented in a point-wise fashion with respect to scale. in short, it is not uncommon for new methods to be presented with data points at very specific or limited compute regions (e.g., base size). we believe that understanding the interaction between architecture and scaling laws is crucial as designing models that perform well at diverse scales will likely have significant impact. this paper is an attempt to understand the effect of inductive bias (architecture) on scaling laws of language models. to this end, we pre-train and finetune over ten diverse model architectures across multiple compute region and scales (e.g., from 15m to 40 billion parameters). in total, we pre-train and finetune over 100 different models of different architectures and sizes and present insights and challenges at scaling these ten diverse architectures. we consider a broad spectrum of models in our extensive experiments. concretely, we consider several well-established transformer variants @xcite such as evolved transformer @xcite , universal transformers @xcite and switch transformers @xcite . we also consider lightweight models such as albert @xcite and/or efficient transformers @xcite such as performer @xcite and funnel transformers @xcite . in our comparison, we are also interested in finding out if general improvements to the transformer architectures such as mixture-of-softmax @xcite and/or gated linear units @xcite influence the scaling behaviour of models. finally, we also evaluate models outside the family of transformers including lightweight convolutions @xcite , dynamic convolutions @xcite and the recently proposed mlp-mixers @xcite . figure 1 illustrates an overview about the experiments we run. we also note that scaling these models is not as straightforward as it seems, i.e., there are intricate details of scale that are intertwined with architectural choices which we study in detail in this paper. for example, a distinct feature of universal transformers (and albert) is parameter sharing. hence, compared with standard transformers, this architectural choice significantly warps the scaling behaviour not only with respect to performance but also amongst compute metrics such as flops, speed and number of parameters @xcite . conversely, models such as switch transformers are on the other end of the spectrum with an uncommon relationship between flops and number of parameters, i.e., they have high parameter to flops ratio. this difficulty makes navigating this landscape challenging. ## methods this section outlines our experimental setup. ## experiment setup our setup, along with all models, are implemented in mesh tensorflow @xcite , a library with similar interface to tensorflow but enables distributed model parallelism across multiple workers. for fair comparison, all models are pretrained for 2 19 steps on the english c4 corpus optimized using an inverse square root learning rate with adafactor @xcite . all models use the same sentencepiece tokenizer @xcite ) containing 32k subwords. this closely follows the setup in the t5 paper @xcite . finetuning is performed for 100k steps on a mixture of glue @xcite , superglue @xcite and squad @xcite . we evaluate on both upstream (pre-training) validation perplexity as well as downstream transfer for nlu tasks (glue + super-glue + squad) after fine-tuning. we pretrain and finetune our models with 16 tpu-v3 chips with data parallelism. all large models have a model parallelism of 2 and xl models have a model parallelism of 8. ## main results we report the main results of this paper in ## epilogue and conclusion in this paper, we conducted extensive experiments, pretraining and finetuning of up to 100 models ranging from 10 well-established transformer and non-transformer architectures. we showed that different model architectures can have different scaling behaviours and models performing well in one compute region (or model size) may not do identically well in another compute region. we also showed that model architectures may do well on upstream perplexity but fail to transfer to downstream tasks. hence, practitioners should be cautious about developing architectures that not only scale well with respect to the upstream perplexity, but also based on downstream performance. while we certainly do not expect researchers to always report model performance across all scales (especially large-scale), we believe that it is good to keep in mind that architectures can perform quite differently at different compute regions. hence, this might be a good dimension to consider when designing new inductive biases. as such, performing evaluation at a certain compute region may be insufficient to capture the full picture. it is also good to consider if different inductive biases will result in different extents of emergent capabilities @xcite . we also showed that different model architectures may react differently to different scaling protocols, reaffirming that comparing and benchmarking these models can be very challenging @xcite . when it comes to scaling large models, we show that novel inductive biases can be indeed quite risky which might explain why most state-of-the-art large language models @xcite @xcite are based on relatively vanilla architectures. our advice is to be cautious when staking an expensive run on an architecture that drastically modifies the attention mechanism. finally, we acknowledge that not every practitioner or researcher would require models that are able to scale to billions of parameters. in that case, inductive biases that are tailored to small or low compute will be sufficient.