Original: S CALE E FFICIENTLY : I NSIGHTS FROM P RE - TRAINING AND F INE - TUNING T RANSFORMERS
Revised: S CALE E FFICIENTLY : I NSIGHTS FROM P RE - TRAINING AND F INE - TUNING T RANSFORMERS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: There remain many open questions pertaining to the scaling behaviour of Transformer architectures.
Revised: There remain many open questions pertaining to the scaling behaviour of Transformer architectures.

Original: These scaling decisions and ﬁndings can be critical, as training runs often come with an associated computational cost which have both ﬁnancial and/or environmental impact.
Revised: These scaling decisions and ﬁndings can be critical, as training runs often come with an associated computational cost which have both ﬁnancial and/or environmental impact.

Original: The goal of this paper is to present scaling insights from pretraining and ﬁnetuning Transformers.
Revised: The goal of this paper is to present scaling insights from pretraining and ﬁnetuning Transformers.

Original: While Kaplan et al.
Revised: While Kaplan et al.

Original: (2020) presents a comprehensive study of the scaling behaviour of Transformer language models, the scope is only on the upstream (pretraining) loss.
Revised: (2020) presents a comprehensive study of the scaling behaviour of Transformer language models, the scope is only on the upstream (pretraining) loss.

Original: Therefore, it is still unclear if these set of ﬁndings transfer to downstream task within the context of the pretrain-ﬁnetune paradigm.
Revised: Therefore, it is still unclear if these set of ﬁndings transfer to downstream task within the context of the pretrain-ﬁnetune paradigm.

Original: The key ﬁndings of this paper are as follows: (1) we show that aside from only the model size, model shape matters for downstream ﬁne-tuning, (2) scaling protocols operate differently at different compute regions, (3) widely adopted T5-base and T5-large sizes are Pareto-inefﬁcient.
Revised: The key ﬁndings of this paper are as follows: (1) we show that aside from only the model size, model shape matters for downstream ﬁne-tuning, (2) scaling protocols operate differently at different compute regions, (3) widely adopted T5-base and T5-large sizes are Pareto-inefﬁcient.

Original: To this end, we present improved scaling protocols whereby our redesigned models achieve similar downstream ﬁne-tuning quality while having 50% fewer parameters and training 40% faster compared to the widely adopted T5-base model.
Revised: To this end, we present improved scaling protocols whereby our redesigned models achieve similar downstream ﬁne-tuning quality while having 50% fewer parameters and training 40% faster compared to the widely adopted T5-base model.

Original: Wewill publicly release over 100 pretrained checkpoints of different T5 conﬁgurations to facilitate future research and analysis.
Revised: We publicly release over 100 pretrained checkpoints of different T5 conﬁgurations to facilitate future research and analysis.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Training Transformers incurs both ﬁnancial and environmental costs (Schwartz et al., 2019; Patterson et al., 2021).
Revised: Training Transformers incurs both ﬁnancial and environmental costs (Schwartz et al., 2019; Patterson et al., 2021).

Original: To this end, researchers and practitioners often have to work around ﬁxed compute budgets and ﬁgure out the best ways to train their models.
Revised: To this end, researchers and practitioners often have to work around ﬁxed compute budgets and ﬁgure out the best ways to train their models.

Original: In lieu of the rising computation demand for training state-of-the-art Transformer (Vaswani et al., 2017;
Revised: In lieu of the rising computation demand for training state-of-the-art Transformer (Vaswani et al., 2017;

Original: Devlin et al., 2018; Raffel et al., 2019;
Revised: Devlin et al., 2018; Raffel et al., 2019;

Original: Brown et al., 2020; Fedus et al., 2021) models, the goal of this paper is to present insights and lessons from scaling Transformers and making them efﬁcient and effective for transfer learning on downstream tasks.
Revised: Brown et al., 2020; Fedus et al., 2021) models, the goal of this paper is to present insights and lessons from scaling Transformers and making them efﬁcient and effective for transfer learning on downstream tasks.

Original: Despite the insights offered in scaling laws research (Kaplan et al., 2020; Hernandez et al., 2021) there remain unresolved questions: Should one follow ﬁxed scaling ratios?
Revised: Despite the insights offered in scaling laws research (Kaplan et al., 2020; Hernandez et al., 2021) there remain unresolved questions: Should one follow ﬁxed scaling ratios?

Original: If not, should one scale by depth?
Revised: If not, should one scale by depth?

Original: Or by width?
Revised: Or by width?

Original: Will scaling experiments on upstream pre-training generalize for downstream transfer?
Revised: Will scaling experiments on upstream pre-training generalize for downstream transfer?

Original: Do scaling protocols for small models generalize to larger models?
Revised: Do scaling protocols for small models generalize to larger models?

Original: Are scaling behaviours similar in all compute regions?
Revised: Are scaling behaviours similar in all compute regions?

Original: We hope the insights presented in this paper can be useful to both practitioners and researchers in informing their scaling decisions.
Revised: We hope the insights presented in this paper can be useful to both practitioners and researchers in informing their scaling decisions.

Original: Neural scaling laws (Kaplan et al., 2020) is a common resource that many look to for advice on scaling Transformer architectures.
Revised: Neural scaling laws (Kaplan et al., 2020) is a common resource that many look to for advice on scaling Transformer architectures.

Original: However, this paper limited its scope to an exhaustive study of upstream cross entropy on language modeling tasks.
Revised: However, this paper limited its scope to an exhaustive study of upstream cross entropy on language modeling tasks.

Original: It is furthermore unclear if ﬁndings from (Kaplan et al., 2020) will transfer to downstream applications.
Revised: It is furthermore unclear if ﬁndings from (Kaplan et al., 2020) will transfer to downstream applications.

Original: Speciﬁcally, Kaplan et al.
Revised: Speciﬁcally, Kaplan et al.

Original: (2020) proposed that the performance of a Transformer language model strongly depends on model size and only weakly on its shape .
Revised: (2020) proposed that the performance of a Transformer language model strongly depends on model size and only weakly on its shape .

Original: They also argue that many model conﬁgurations with the same number of parameters perform similarly regardless of architectural details.
Revised: They also argue that many model conﬁgurations with the same number of parameters perform similarly regardless of architectural details.

Original: Our work empirically conﬁrms this on upstream training but ﬁnds a distinct discrepancy when considering practical downstream performance – a key insight that we believe is highly important.
Revised: Our work empirically conﬁrms this on upstream training but ﬁnds a distinct discrepancy when considering practical downstream performance – a key insight that we believe is highly important.

Original: To this end, we conduct extensive experiments involving pre-training and ﬁne-tuning overtransformer conﬁgurations ranging from 5 M to 30 B parameters.
Revised: To this end, we conduct extensive experiments involving pre-training and ﬁne-tuning overtransformer conﬁgurations ranging from 5 M to 30 B parameters.

Original: To the best of our knowledge, this is the largest empirical study of practical scaling of transformer to date that considers both upstream and practical downstream transfer.
Revised: To the best of our knowledge, this is the largest empirical study of practical scaling of transformer to date that considers both upstream and practical downstream transfer.

Original: While there have been many proposed scaling protocols for
Revised: While there have been many proposed scaling protocols for

Original: ConvNets (Tan and Le, 2019; Bello et al., 2021), there is still limited advice on scaling of transformer architectures, apart from (Kaplan et al., 2020; Li et al.,2020; Levine et al., 2020).
Revised: ConvNets (Tan and Le, 2019; Bello et al., 2021), there is still limited advice on scaling of transformer architectures, apart from (Kaplan et al., 2020; Li et al., 2020).

Original: Hence, the key goal of this paper is to distill our experiences and insights with scaling Transformer architectures and share them with the broader community.
Revised: Hence, the key goal of this paper is to distill our experiences and insights with scaling Transformer architectures and share them with the broader community.

Revised: Contributions

Original: The overall ﬁndings and insights of the paper can be summarized as follows:
Revised: The overall ﬁndings and insights of the paper can be summarized as follows:

Original: • We ﬁnd that scaling laws may differ in upstream and downstream setups.
Revised: • We ﬁnd that scaling laws may differ in upstream and downstream setups.

Original: Speciﬁcally, contrary to Kaplan et al.
Revised: Speciﬁcally, contrary to Kaplan et al.

Original: (2020), we ﬁnd that downstream performance strongly depends on shape and not only on model size.
Revised: (2020), we ﬁnd that downstream performance strongly depends on shape and not only on model size.

Original: Hence, pretraining performance may not necessarily transfer to downstream applications.
Revised: Hence, pretraining performance may not necessarily transfer to downstream applications.

Original: (Figure 1).
Revised: (Figure 1).

Original: • Our ﬁndings show that pre-training perplexity can often be a deceiving indicator of downstream quality and therefore model building based on upstream perplexity can be challenging.
Revised: • Our ﬁndings show that pre-training perplexity can often be a deceiving indicator of downstream quality and therefore model building based on upstream perplexity can be challenging.

Original: Scaling laws can differ substantially when considering metrics on actual downstream ﬁnetuning.
Revised: Scaling laws can differ substantially when considering metrics on actual downstream ﬁnetuning.

Original: (Figure 1) •
Revised: (Figure 1) •

Original: Given that empirical scaling laws differ when considering quality on the downstream, our work investigates the pareto-frontier of transformer conﬁgurations in this setup.
Revised: Given that empirical scaling laws differ when considering quality on the downstream, our work investigates the pareto-frontier of transformer conﬁgurations in this setup.

Original: We ﬁnd that the canonical model conﬁgurations such as T5-Base and T5-Large sizes (Raffel et al., 2019) are relatively inefﬁcient (Figure 2).
Revised: We ﬁnd that the canonical model conﬁgurations such as T5-Base and T5-Large sizes (Raffel et al., 2019) are relatively inefﬁcient (Figure 2).

Original: Note that these sizes are based off the canonical BERT (Devlin et al., 2018) base and large sizes.
Revised: Note that these sizes are based off the canonical BERT (Devlin et al., 2018) base and large sizes.

Original: • We ﬁnd that scaling strategies differ at different compute regions, i.e., applying same strategies at different compute regions (small vs large) has a different effect on model quality.
Revised: • We ﬁnd that scaling strategies differ at different compute regions, i.e., applying same strategies at different compute regions (small vs large) has a different effect on model quality.

Original: This has practical implications since ﬁnding strategies at small scale might not necessarily transfer or generalize to higher compute regions (section 4.2).
Revised: This has practical implications since ﬁnding strategies at small scale might not necessarily transfer or generalize to higher compute regions (section 4.2).

Original: • After extensive empirical exploration of the pareto-frontier of transformer models, we propose a simple but effective scaling strategy which we call the DeepNarrow strategy .
Revised: • After extensive empirical exploration of the pareto-frontier of transformer models, we propose a simple but effective scaling strategy which we call the DeepNarrow strategy .

Original: We show that we are able to obtain model quality on par or better than canonical model sizes (e.g., base) with 50% less parameters and being 40% faster.
Revised: We show that we are able to obtain model quality on par or better than canonical model sizes (e.g., base) with 50% less parameters and being 40% faster.

Original: While we highlight the limitations of this strategy, we also show that this DeepNarrow strategy is applicable to all model sizes.
Revised: While we highlight the limitations of this strategy, we also show that this DeepNarrow strategy is applicable to all model sizes.

Original: (Table 4).
Revised: (Table 4).

Original: • To consider how generalized these scaling strategies are, we conduct additional experiments on Vision Transformers (ViT; Dosovitskiy et al., 2020) to verify them in the vision domain.
Revised: • To consider how generalized these scaling strategies are, we conduct additional experiments on Vision Transformers (ViT; Dosovitskiy et al., 2020) to verify them in the vision domain.

Original: Moreover, on top of the 17 GLUE (Wang et al., 2018) / SuperGLUE (Wang et al., 2019) and SQuAD (Rajpurkar et al., 2016) tasks we employed in our extensive study, we verify our ﬁndings via additional downstream experiments across 12 diverse language tasks (section A.1).
Revised: Moreover, on top of the 17 GLUE (Wang et al., 2018) / SuperGLUE (Wang et al., 2019) and SQuAD (Rajpurkar et al., 2016) tasks we employed in our extensive study, we verify our ﬁndings via additional downstream experiments across 12 diverse language tasks (section .2).

Original: • Wewill release (1) the pre-trained checkpoints for our T5 models with improved scaling protocols and (2) all 100+ model checkpoints, including intermediate training checkpoints to the research community.
Revised: • We release (1) the pre-trained checkpoints for our T5 models with improved scaling protocols and (2) all 100+ model checkpoints, including intermediate training checkpoints to the research community.

Original: We believe that this is a treasure trove of data to study the behaviour of large LM pretraining and ﬁnetuning especially pertaining to scaling laws.
Revised: We believe that this is a treasure trove of data to study the behaviour of large LM pretraining and ﬁnetuning especially pertaining to scaling laws.

Original: The checkpoints and code will be released at anonymous_link_withheld .
Revised: The checkpoints and code will be released at https://github.com/google-research/ google-research/tree/master/scaling_transformers . The checkpoints are now publicly available at our Google Cloud Bucket gs://scenic-bucket/ scaling_explorer/scaling_explorer . More recently, these checkpoints are also now available on Huggingface https://huggingface.co/models?other= deep-narrow .

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Transformers (Vaswani et al., 2017) have become ubiquitous in the modern deep learning stack and have seen widespread impact across not only language (Devlin et al., 2018; Raffel et al., 2019; Brown et al., 2020) but also computer vision (Dosovitskiy et al., 2020; Arnab et al., 2021), reinforcement  learning (Parisotto et al., 2020) and computational biology (Senior et al., 2020).
Revised: Transformers (Vaswani et al., 2017) have become ubiquitous in the modern deep learning stack and have seen widespread impact across not only language (Devlin et al., 2018; Raffel et al., 2019; Brown et al., 2020) but also computer vision (Dosovitskiy et al., 2020; Arnab et al., 2021), reinforcement [Table] [Table] learning (Parisotto et al., 2020) and computational biology (Senior et al., 2020).

Original: To this end, discovering empirical scaling laws of these models is a research area that has garnered considerable interest (Kaplan et al., 2020; Henighan et al., 2020; Hernandez et al., 2021; Bahri et al., 2021; Levine et al., 2020).
Revised: To this end, discovering empirical scaling laws of these models is a research area that has garnered considerable interest (Kaplan et al., 2020; Henighan et al., 2020; Hernandez et al., 2021; Bahri et al., 2021).

Original: Discovering empirical scaling laws that govern neural language model scaling has been a recent subject of keen interest (Kaplan et al., 2020; Hernandez et al., 2021; Bahri et al., 2021).
Revised: Discovering empirical scaling laws that govern neural language model scaling has been a recent subject of keen interest (Kaplan et al., 2020; Hernandez et al., 2021; Bahri et al., 2021).

Original: Many of these works present scaling laws across a variety of axis such as model size, compute and/or dataset size.
Revised: Many of these works present scaling laws across a variety of axis such as model size, compute and/or dataset size.

Original: It is worth to note that many of these works evaluate on autoregressive language modeling and use cross entropy loss to measure performance (Kaplan et al., 2020; Hernandez et al., 2021).There
Revised: It is worth to note that many of these works evaluate on autoregressive language modeling and use cross entropy loss to measure performance (Kaplan et al., 2020; Hernandez et al., 2021).

Original: [Table] [Table] are a multitude of interesting ﬁndings presented (Kaplan et al., 2020) whereby the authors show that performance (loss) scales as a power-law with model size and dataset size.
Revised: There are a multitude of interesting ﬁndings presented (Kaplan et al., 2020) whereby the authors show that performance (loss) scales as a power-law with model size and dataset size.

Original: However, one notable claim is that architectural details (e.g., network depth and width) have minimal effects.
Revised: However, one notable claim is that architectural details (e.g., network depth and width) have minimal effects.

Original: Subsequently,
Revised: Subsequently,

Original: Hernandez et al.
Revised: Hernandez et al.

Original: (2021) builds upon the work of Kaplan et al.
Revised: (2021) builds upon the work of Kaplan et al.

Original: (2020), evaluating scaling laws for ‘transfer’.
Revised: (2020), evaluating scaling laws for ‘transfer’.

Original: To this end, the authors study the effect of dataset scaling on unsupervised transfer learning and ﬁnetuning.
Revised: To this end, the authors study the effect of dataset scaling on unsupervised transfer learning and ﬁnetuning.

Original: That said, the experiments of Hernandez et al.
Revised: That said, the experiments of Hernandez et al.

Original: (2021) are mainly targeted at dataset transfer between two different distributions (language and code) and make the same assumptions as
Revised: (2021) are mainly targeted at dataset transfer between two different distributions (language and code) and make the same assumptions as

Original: Kaplan et al.
Revised: Kaplan et al.

Original: (2020) about model scaling.
Revised: (2020) about model scaling.

Original: In a similar vein, Henighan et al.
Revised: In a similar vein, Henighan et al.

Original: (2020) studied empirical scaling laws for different domains for generative modeling in vision, video and multimodal setups.
Revised: (2020) studied empirical scaling laws for different domains for generative modeling in vision, video and multimodal setups.

Original: There have been increasing demand for training and scaling Transformers (Shoeybi et al., 2019; Raffel et al., 2019; Fedus et al., 2021; Conneau et al., 2019; Lin et al., 2021).
Revised: There have been increasing demand for training and scaling Transformers (Shoeybi et al., 2019; Raffel et al., 2019; Fedus et al., 2021; Conneau et al., 2019; Lin et al., 2021).

Original: Despite the beneﬁts from improved performance, there are ﬁnancial considerations and environmental costs (Schwartz et al., 2019; Patterson et al., 2021) to training these models.
Revised: Despite the beneﬁts from improved performance, there are ﬁnancial considerations and environmental costs (Schwartz et al., 2019; Patterson et al., 2021) to training these models.

Original: Given that every moment spent on hardware accelerators is a cost incurring activity, we believe that research in distilling practical scaling insights and recommendations to be highly crucial (Li et al., 2020; Kaplan et al., 2020; Bello et al., 2021).
Revised: Given that every moment spent on hardware accelerators is a cost incurring activity, we believe that research in distilling practical scaling insights and recommendations to be highly crucial (Li et al., 2020; Kaplan et al., 2020; Bello et al., 2021).

Original: Notably, the research problem of making transformers efﬁcient have also been tackled from an extensive number of angles such as (but not limited to) distillation (Hinton et al., 2015), compression (Zafrir et al., 2019), parameter sharing (Lan et al., 2019; Tay et al., 2019; Zhang et al., 2021), efﬁcient attention (Tay et al., 2020c; Kitaev et al., 2020; Choromanski et al., 2020; Tay et al., 2020b; Ainslie et al., 2020; Jaegle et al., 2021), architecture search (So et al., 2019), alternative non Transformer-based architectures (Tolstikhin et al., 2021; Tay et al., 2021; 2020a; Lee-Thorp et al., 2021).
Revised: Notably, the research problem of making transformers efﬁcient have also been tackled from an extensive number of angles such as (but not limited to) distillation (Hinton et al., 2015), compression (Zafrir et al., 2019), parameter sharing (Lan et al., 2019; Tay et al., 2019; Zhang et al., 2021), efﬁcient attention (Tay et al., 2020c; Kitaev et al., 2020; Choromanski et al., 2020; Tay et al., 2020b; Ainslie et al., 2020; Jaegle et al., 2021), architecture search (So et al., 2019), alternative non Transformer-based architectures (Tolstikhin et al., 2021; Tay et al., 2021a; 2020a; Lee-Thorp et al., 2021).

Original: With so much extensive research into novel techniques to improving the efﬁciency of transformers, it is surprising that the standard conﬁgurations (e.g., base, large) of transformers in BERT (Devlin et al., 2018) or
Revised: With so much extensive research into novel techniques to improving the efﬁciency of transformers, it is surprising that the standard conﬁgurations (e.g., base, large) of transformers in BERT (Devlin et al., 2018) or

Original: T5 (Raffel et al., 2019) have not been rethought.
Revised: T5 (Raffel et al., 2019) have not been rethought.

Original:  M ETHODS
Revised:  M ETHODS

Original: This section describes our main experimental setup.
Revised: This section describes our main experimental setup.

Original: Architecture We study a Transformer encoder-decoder architecture that uses relative attention based of the T5 architecture (Raffel et al., 2019).
Revised: Architecture We study a Transformer encoder-decoder architecture that uses relative attention based of the T5 architecture (Raffel et al., 2019).

Original: The choice of adopting Seq2Seq architectures (Sutskever et al., 2014) is mainly due to their universality and ability to both subsume encoder (Bert-like) and decoder (language) models within an identical framework.
Revised: The choice of adopting Seq2Seq architec- tures (Sutskever et al., 2014) is mainly due to their universality and ability to both subsume encoder (Bert-like) and decoder (language) models within an identical framework.

Original: Moreover, the universality of Seq2Seq architectures also allow us to ﬁne-tune across a broad range of tasks.
Revised: Moreover, the universality of Seq2Seq architectures also allow us to ﬁne-tune across a broad range of tasks.

Original: Our implementation and experiments are performed in Mesh Tensorﬂow 1 (Shazeer et al., 2018) using the default T library 2 .
Revised: Our implementation and experiments are performed in Mesh Tensorﬂow 1 (Shazeer et al., 2018) using the default T library 2 .

Original: Model Conﬁgurations
Revised: Model Conﬁgurations

Original: We ﬁrst deﬁne eight Transformer sizes, i.e., tiny, mini, small, base, large,
Revised: We ﬁrst deﬁne eight Transformer sizes, i.e., tiny, mini, small, base, large,

Original: XL, XXL and XXXL.
Revised: XL, XXL and XXXL.

Original: The small, base, large, XL and XXL corresponds to the canonical T5 sizes that are released in Raffel et al.
Revised: The small, base, large, XL and XXL corresponds to the canonical T5 sizes that are released in Raffel et al.

Original: We use three other sizes as starting points, e.g., tiny and mini since there is a lack of representation of transformers at lower compute regions.
Revised: We use three other sizes as starting points, e.g., tiny and mini since there is a lack of representation of transformers at lower compute regions.

Original: Pretraining We pretrain on the Colossal Cleaned Common Crawl Corpus (C4; Raffel et al., 2019).
Revised: Pretraining We pretrain on the Colossal Cleaned Common Crawl Corpus (C4; Raffel et al., 2019).

Original: We pre-train encoder-decoder models using the span-based masked language modeling (MLM) objective (Fedus et al., 2018; Devlin et al., 2018).
Revised: We pre-train encoder-decoder models using the span-based masked language modeling (MLM) objective (Fedus et al., 2018; Devlin et al., 2018).

Original: We pretrain all our models for 2 19 steps using 16 TPU-v3 chips.
Revised: We pretrain all our models for 2 19 steps using 16 TPU-v3 chips.

Original: For larger models, we run our models with 64 TPU-v3 chips.
Revised: For larger models, we run our models with 64 TPU-v3 chips.

Original: We use 2 19 steps since majority of the experiments in (Raffel et al., 2019) were conducted in the same fashion.
Revised: We use 2 19 steps since majority of the experiments in (Raffel et al., 2019) were conducted in the same fashion.

Original: We would also like to emphasize that the ofﬁcial released T5 checkpoints were pretrained on 1 T tokens ( million steps with a batch size of 2048).
Revised: We would also like to emphasize that the ofﬁcial released T5 checkpoints were pretrained on 1 T tokens ( million steps with a batch size of 2048).

Original: Given that this extended long pretraining setup is infeasible given the number of experiments we would have to run, we opt to follow the standard ablation setup in (Raffel et al., 2019) which pretrains on more manageable number of tokens.
Revised: Given that this extended long pretraining setup is infeasible given the number of experiments we would have to run, we opt to follow the standard ablation setup in (Raffel et al., 2019) which pretrains on more manageable number of tokens.

Original: Downstream Tasks We consider a myriad of downstream tasks.
Revised: Downstream Tasks We consider a myriad of downstream tasks.

Original: In total, we considertasks.
Revised: In total, we considertasks.

Original: We ﬁnetune on a mixture of GLUE (Wang et al., 2018), SuperGLUE (Wang et al., 2019),
Revised: We ﬁnetune on a mixture of GLUE (Wang et al., 2018), SuperGLUE (Wang et al., 2019),

Original: SQuAD (Rajpurkar et al., 2016) for the key downstream experiment results and report aggregate
Revised: SQuAD (Rajpurkar et al., 2016) for the key downstream experiment results and report aggregate

Original: GLUE/SuperGLUE scores.
Revised: GLUE/SuperGLUE scores.

Original: We believe that an aggregate of 17 tasks in natural language understanding that conmprises of both high-resource and low-resource tasks gives us a good overview of a model’s downstream performance.
Revised: We believe that an aggregate of 17 tasks in natural language understanding that conmprises of both high-resource and low-resource tasks gives us a good overview of a model’s downstream performance.

Original: Finetuning is typically performed with 16 TPU-v3 chips.
Revised: Finetuning is typically performed with 16 TPU-v3 chips.

Original: Notation for Scaling Operators For the remainder of the paper, we use a shortform code for each scaling operator applied on a standard transformer size.
Revised: Notation for Scaling Operators For the remainder of the paper, we use a shortform code for each scaling operator applied on a standard transformer size.

Original: For example, NL32-SM refers to scaling small (SM) transformers to 32 layers (NL32).
Revised: For example, NL32-SM refers to scaling small (SM) transformers to 32 layers (NL32).

Original: We use EL,DL to represent scaling encoder and decoder layers independently, KV to represent scaling each key-value size, DM to represent scaling d model .
Revised: We use EL,DL to represent scaling encoder and decoder layers independently, KV to represent scaling each key-value size, DM to represent scaling d model .

Original: NH to represent modifying the number of heads and FF to represent scaling d FF .
Revised: NH to represent modifying the number of heads and FF to represent scaling d FF .

Original: The initial/standard model sizes are tiny, mini, small, base, large, XL, XXL and XXXL.
Revised: The initial/standard model sizes are tiny, mini, small, base, large, XL, XXL and XXXL.

Original: This is described in Table 2.
Revised: This is described in Table 2.

Original: Convention With the exception of Figure 1, all charts are plotted with FLOPS as the main compute metric.
Revised: Convention With the exception of Figure 1, all charts are plotted with FLOPS as the main compute metric.

Original: We use number of params for Figure 1 to align with Kaplan et al.
Revised: We use number of params for Figure 1 to align with Kaplan et al.

Original: All of the downstream results are plot with SuperGLUE accuracy (Wang et al., 2019) as the Y-axis.
Revised: All of the downstream results are plot with SuperGLUE accuracy (Wang et al., 2019) as the Y-axis.

Original: Due to the lack of space, we report charts/plots of other metrics (params of speed) and other tasks (GLUE or
Revised: Due to the lack of space, we report charts/plots of other metrics (params of speed) and other tasks (GLUE or

Original: SQuAD) in the supplementary material.
Revised: SQuAD) in the supplementary material.

Original: All parameter counts also include the embedding parameters.
Revised: All parameter counts also include the embedding parameters.

Original: We re-emphasize that it is critical to take into account multiple facets of efﬁciency and therefore report all three key metrics (FLOPs, throughput/speed and parameter count) in the supplementary material.
Revised: We re-emphasize that it is critical to take into account multiple facets of efﬁciency and therefore report all three key metrics (FLOPs, throughput/speed and parameter count) in the supplementary material.

Original: Model and Data Parallelism By default, our models are trained across multiple workers via data parallelism.
Revised: Model and Data Parallelism By default, our models are trained across multiple workers via data parallelism.

Original: As per convention in the T5 paper, our larger models use the default model parallelism.
Revised: As per convention in the T5 paper, our larger models use the default model parallelism.

Original: Speciﬁcally, this is set to 2 for large models, 8 for XL models and 32 for XXL models.
Revised: Speciﬁcally, this is set to 2 for large models, 8 for XL models and 32 for XXL models.

Original:  A NALYSIS AND R ESULTS
Revised:  A NALYSIS AND R ESULTS

Original: This section presents our overall ﬁndings and key results.
Revised: This section presents our overall ﬁndings and key results.

Original: 4. M ODEL S HAPE M ATTERS

Original: We extend the results of Kaplan et al.

Original: (2020) to ﬁne-tuning and present model shape dependence not highlighted in Hernandez et al.

Original: Kaplan et al.

Original: (2020) studies pre-training (upstream) and concludes that performance depends only weakly on model shape, but strongly on model size.

Original: Hernandez et al.

Original: (2021) extends this work to measure an effective data transfer measure when pretraining and then ﬁne-tuning on a Python dataset.

Original: However, this work does not consider details of model shape, and instead focused on the relative predictability with model scale alone.

Original: Our work stands in contrasts since we ﬁnd that model shape matters considerably for downstream ﬁne-tuned performance.

Original: Figure 1 shows compute-performance scatter plots for pre-training (left) and ﬁne-tuning

Original: 2.7e+8 5.4e+ 1.1e+ 2.1e+ 4.3e+ 8.6e+9 1.7e+10 Params -1.
Revised: 2.7e+8 5.4e+ 1.1e+ 2.1e+ 4.3e+ 8.6e+9 1.7e+10 Params -1.

Original: N e g a t
Revised: N e g a t

Original: i v e L o gP e r p l e x i t y
Revised: i v e L o gP e r p l e x i t y

Original: Large
Revised: Large

Original: XL
Revised: XL

Original: NL
Revised: NL

Original: NL32NL
Revised: NL32NL

Original: NL32-LG NL8-XL
Revised: [Figure] LG

Original: NL32-XL NL6-XXL NL8-XXL
Revised: [Figure] XL [Figure] XXL

Original: NL12-XXL NL6-XXXL NL12-XXXL
Revised: [Figure] XXL

Original: [Figure]
Revised: [Figure]

Original: 2.7e+8 5.4e+ 1.1e+ 2.1e+ 4.3e+ 8.6e+9 1.7e+10 Params
Revised: 2.7e+8 5.4e [Figure] 1.1e+ 2.1e+ 4.3e+ 8.6e+9 1.7e+ [Figure]

Original: S u p e r
Revised: S u p e r

Original: G l u e
Revised: G l u e

Original: A cc u r a c y
Revised: A cc u r a c y

Original: Large
Revised: Large

Original: XL
Revised: XL

Original: NL NL
Revised: NL NL

Original: NL36 NL32-LG
Revised: NL36 NL32-LG

Original: NL8-XL
Revised: NL8-XL

Original: NL32-XL NL6-XXL
Revised: NL32-XL NL

Original: NL8-XXL
Revised: [Figure] XXL

Original: NL
Revised: NL

Original: [Figure]
Revised: [Figure]

Original: XXL
Revised: XXL

Original: NL
Revised: NL

Original: [Figure]
Revised: [Figure]

Original: XXXL
Revised: XXL

Revised: NL

Revised: [Figure]

Original: NL
Revised: NL

Original: [Figure]
Revised: [Figure]

Revised: 4. M ODEL S HAPE M ATTERS

Original: [Table]
Revised: We extend the results of Kaplan et al. (2020) to ﬁne-tuning and present model shape dependence not highlighted in Hernandez et al. Kaplan et al. (2020) studies pre-training (upstream) and concludes that performance depends only weakly on model shape, but strongly on model size.

Revised: Hernandez et al.

Revised: (2021) extends this work to measure an effective data transfer measure when pretraining and then ﬁne-tuning on a Python dataset.

Revised: However, this work does not consider details of model shape, and instead focused on the relative predictability with model scale alone.

Revised: Our work stands in contrasts since we ﬁnd that model shape matters considerably for downstream ﬁne-tuned performance.

Revised: Figure 1 shows compute-performance scatter plots for pre-training (left) and ﬁne-tuning (right) over a dozen Transformers.

Revised: The models considered are sampled diversely within a two-order of magnitude band of model scales.

Original:  conﬁgurations such as XXXL (33B), XXL (11B), XL (3B) and LG (750M) parameters but have their depths/lengths modiﬁed.
Revised: We adjust the model shape primarily through depth variations, starting with conﬁgurations such as XXXL (33B), XXL (11B), XL (3B) and LG (750M) parameters but have their depths/lengths modiﬁed.

Original: From Figure 1 reveals a strong correlation of the upstream performance with model size, corroborating the neural scaling laws of Kaplan et al.
Revised: From Figure 1 reveals a strong correlation of the upstream performance with model size, corroborating the neural scaling laws of Kaplan et al.

Original: But the strong pre-training correlation largely vanishes when ﬁne-tuning these models on SuperGLUE (Wang et al., 2019).
Revised: But the strong pre-training correlation largely vanishes when ﬁne-tuning these models on SuperGLUE (Wang et al., 2019).

Original: While we conﬁrm the ﬁndings of Kaplan et al.
Revised: While we conﬁrm the ﬁndings of Kaplan et al.

Original: (2020) that performance scales strongly dependent on model size but weakly on model shape, we ﬁnd that model shape (such as depth-scaling) is highly important for downstream transfer – a dimension that is not considered in Kaplan et al.
Revised: (2020) that performance scales strongly dependent on model size but weakly on model shape, we ﬁnd that model shape (such as depth-scaling) is highly important for downstream transfer – a dimension that is not considered in Kaplan et al.

Original: As a substantiating point and additional context to Figure 1, we also show via a counter-example that pretraining perplexity is not indicative of transfer performance, i.e., we explicitly show that a case (in
Revised: As a substantiating point and additional context to Figure 1, we also show via a counter-example that pretraining perplexity is not indicative of transfer performance, i.e., we explicitly show that a case (in

Original: Table 3) where a model can have outstanding pre-training perplexity but substantially undeliver when it comes to downstream performance.
Revised: Table 3) where a model can have outstanding pre-training perplexity but substantially undeliver when it comes to downstream performance.

Original: Appendix A presents more evidence on this ﬁnding by ﬁtting a power law curve pre-training and ﬁne-tuninng performance versus parameter size.

Original: To the best of our knowledge, while this has been mentioned implicitly in several existing works (Narang et al., 2021), this is the ﬁrst work explicitly shows this point.
Revised: To the best of our knowledge, while this has been mentioned implicitly in several existing works (Narang et al., 2021), this is the ﬁrst work explicitly shows this point.

Original: Zooming in Versus Zooming Out Here, one may argue that a general trend (even on downstream) may still exist if we zoom out and cover a very wide range of model sizes (e.g., very tiny to very large).
Revised: Zooming in Versus Zooming Out Here, one may argue that a general trend (even on downstream) may still exist if we zoom out and cover a very wide range of model sizes (e.g., very tiny to very large).

Original: A tiny model is not likely to outperform a very large model no matter how well-conﬁgured it might be.
Revised: A tiny model is not likely to outperform a very large model no matter how well-conﬁgured it might be.

Original: Our purpose here is not to contradict this general trend but to distinguish between both arguments.
Revised: Our purpose here is not to contradict this general trend but to distinguish between both arguments.

Original: We argue that, in practical setups, comparisons between models and scaling decisions are often made when zooming-in and our pairwise comparisons above are not on largely differentmodels, rather those that are on the same neighborhood in the size (close in the x-axis). Thus, what we claim is that when you zoom in, which is what happen in practice, it is not uncommon to see 1.1e+ 2.2e+ 4.4e+ 8.8e [Table]
Revised: We argue that, in practical setups, comparisons between models and scaling decisions are often made when zooming-in and our pairwise comparisons above are not on largely different

Original: 1.8e

Original: [Table]
Revised: [Table]

Original: S u p e r G l u e A cc u r a c y

Original: Small

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure] [Table] a [Table]
Revised: models, rather those that are on the same neighborhood in the size close

Original: y [Table] [Figure] cases similar to the models in Table 3 where taking the upstream perplexity into account may lead to a sub-optimal choice.
Revised: Thus, what we claim is that when you zoom in, which is what happen in practice, it is not uncommon to see cases similar to the models in Table 3 where taking the upstream perplexity into account may lead to a sub-optimal choice.

Original: It is also worth to mention that zoom-ing in on upstream returns very different trends compared to zoom-ing in on downstream results.
Revised: It is also worth to mention that zoom-ing in on upstream returns very different trends compared to zoom-ing in on downstream results.

Original: 4. S CALING BEHAVIOUR AT DIFFERENT COMPUTE REGIONS IS DIFFERENT
Revised: 4. S CALING BEHAVIOUR AT DIFFERENT COMPUTE REGIONS IS DIFFERENT

Original: In this section, we evaluate how each scaling hyperparameter and model shape inﬂuences a model’s position on the compute-performance chart.
Revised: In this section, we evaluate how each scaling hyperparameter and model shape inﬂuences a model’s position on the compute-performance chart.

Original: Figure 2 shows three plots which varying different scaling knobs.
Revised: Figure 2 shows three plots which varying different scaling knobs.

Original: Given three starting points, small, base and large, we scale the starting points across different knobs.
Revised: Given three starting points, small, base and large, we scale the starting points across different knobs.

Original: It is clear from Figure 2 that the effect of applying different scaling operators is very different across different compute regions.
Revised: It is clear from Figure 2 that the effect of applying different scaling operators is very different across different compute regions.

Original: We also observe that the Pareto-boundary is also very different at different compute regions.
Revised: We also observe that the Pareto-boundary is also very different at different compute regions.

Original: The implications of this ﬁnding is nontrivial, since this effectively means that ﬁnding improvements at a (plausibly) smaller scale in hopes that it will generalize at large scale might not be an effective strategy.
Revised: The implications of this ﬁnding is nontrivial, since this effectively means that ﬁnding improvements at a (plausibly) smaller scale in hopes that it will generalize at large scale might not be an effective strategy.

Original: This corroborates recent ﬁndings of Bello et al.
Revised: This corroborates recent ﬁndings of Bello et al.

Original: (2021) which studied scaling properties of ImageNet models.
Revised: (2021) which studied scaling properties of ImageNet models.

Original: Their paper demonstrates that the compound scaling rules (Tan and Le, 2019) derived in a small-scale regime, lead to Pareto-inefﬁcient models when then extrapolated to larger scales.
Revised: Their paper demonstrates that the compound scaling rules (Tan and Le, 2019) derived in a small-scale regime, lead to Pareto-inefﬁcient models when then extrapolated to larger scales.

Original: 4. N OT ALL S CALING S TRATEGIES AND M ODEL S HAPES ARE C REATED E QUAL
Revised: 4. N OT ALL S CALING S TRATEGIES AND M ODEL S HAPES ARE C REATED E QUAL

Original: From Figure 2, we can also observe that different scaling strategies results in very different outcome.
Revised: From Figure 2, we can also observe that different scaling strategies results in very different outcome.

Original: A common pattern amongst all three model sizes is that the NL operator has strong inﬂuence on the
Revised: A common pattern amongst all three model sizes is that the NL operator has strong inﬂuence on the

Original: Pareto-frontier.
Revised: Pareto-frontier.

Original: On the other hand, settings such as KV (varying d kv ) seem to result in models that are less Pareto-efﬁcient.
Revised: On the other hand, settings such as KV (varying d kv ) seem to result in models that are less Pareto-efﬁcient.

Original: We notice mixed signals from varying d model .
Revised: We notice mixed signals from varying d model .

Original: In this case, scaling down results in a model on the pareto-frontier, but scaling up, i.e., DM 2 K results in a highly pareto-inefﬁcient model.
Revised: In this case, scaling down results in a model on the pareto-frontier, but scaling up, i.e., DM 2 K results in a highly pareto-inefﬁcient model.

Original: When compared to scaling up, NL is a substantially superior option to d model .
Revised: When compared to scaling up, NL is a substantially superior option to d model .

Original: We describe scaling recommendations in subsequent sections.
Revised: We describe scaling recommendations in subsequent sections.

Original: Overall, it does seem like d kv and N H does not inﬂuence the Pareto frontier as much as other knobs. 
Revised: Overall, it does seem like d kv and N H does not inﬂuence the Pareto frontier as much as other knobs. [Figure] [Figure] [Table]

Revised: [Figure]

Revised: [Table]

Original: Effect of scaling different knobs Figure 3 illustrates the effect of scaling different knobs on the compute-performance boundary.
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: Effect of scaling different knobs Figure 3 illustrates the effect of scaling different knobs on the compute-performance boundary.

Original: It becomes clear that not all strategies inﬂuence the boundary with the same impact.
Revised: It becomes clear that not all strategies inﬂuence the boundary with the same impact.

Original: For example, NL has the biggest impact while NH does not inﬂuence the model’s position on the graph much.
Revised: For example, NL has the biggest impact while NH does not inﬂuence the model’s position on the graph much.

Original: Finally, we also note that the effect of scaling on upstream and downstream is quite different.
Revised: Finally, we also note that the effect of scaling on upstream and downstream is quite different.

Original: For example, FF2K is clearly better option than the canonical base model in downstream but not upstream.
Revised: For example, FF2K is clearly better option than the canonical base model in downstream but not upstream.

Original: Scaling Depth vs Width In Figure 3, we also note that the NL scaling operator (depth) has generally impact on the Pareto-boundary as compared to width ( FF ).
Revised: Scaling Depth vs Width In Figure 3, we also note that the NL scaling operator (depth) has generally impact on the Pareto-boundary as compared to width ( FF ).

Original: For instance, we can see that
Revised: For instance, we can see that

Original: FF12K (in Figure 3 is clearly not Pareto-optimal, being outclassed by conﬁgurations such as NL16- [Figure] [Figure] y
Revised: FF12K (in Figure 3 is clearly not Pareto-optimal, being outclassed by conﬁgurations such as NL16SM, EL12-SM.

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table] [Figure] SM, EL12-SM. Likewise in the base setup, FF9K and F12K are less Pareto-efﬁcient as compared to
Revised: Likewise in the base setup, FF9K and F12K are less Pareto-efﬁcient as compared to

Original: NL16 and NL24.
Revised: NL16 and NL24.

Original: 4. S CALING R ECOMMENDATIONS
Revised: 4. S CALING R ECOMMENDATIONS

Original: We generally recommend a DeepNarrow strategy where the model’s depth is preferentially increased before considering any other forms of uniform scaling across other dimensions.
Revised: We generally recommend a DeepNarrow strategy where the model’s depth is preferentially increasedbefore considering any other forms of uniform scaling across other dimensions.

Original: This is largely due to how much depth inﬂuences the Pareto-frontier as shown in earlier sections of the paper.
Revised: This is largely due to how much depth inﬂuences the Pareto-frontier as shown in earlier sections of the paper.

Original: Speciﬁcally, a tall small (deep and narrow) model is generally more efﬁcient compared to the base model.
Revised: Speciﬁcally, a tall small (deep and narrow) model is generally more efﬁcient compared to the base model.

Original: Likewise, a tall base model might also generally more efﬁcient compared to a large model.
Revised: Likewise, a tall base model might also generally more efﬁcient compared to a large model.

Original: We generally ﬁnd that, regardless of size, even if absolute performance might increase as we continue to stack layers, the relative gain of Pareto-efﬁciency diminishes as we increase the layers, converging at 32 tolayers.
Revised: We generally ﬁnd that, regardless of size, even if absolute performance might increase as we continue to stack layers, the relative gain of Pareto-efﬁciency diminishes as we increase the layers, converging at 32 tolayers.

Original: Finally, we note that our notion of efﬁciency here relates to any one compute dimension, i.e., params, FLOPs or throughput (speed).
Revised: Finally, we note that our notion of efﬁciency here relates to any one compute dimension, i.e., params, FLOPs or throughput (speed).

Original: We report all three key 3 efﬁciency metrics (number of params,
Revised: We report all three key 4 efﬁciency metrics (number of params,

Revised: [Table]

Original: FLOPS and speed) and leave this decision to the practitioner to decide which compute dimension to consider.
Revised: FLOPS and speed) and leave this decision to the practitioner to decide which compute dimension to consider.

Original: Efﬁcient Alternatives to T5-Base/
Revised: Efﬁcient Alternatives to T5-Base/

Original: Large/XL/XXL Table 4 describes this phenomena in which we list efﬁcient alternatives to the canonical model sizes using the DeepNarrow strategy.
Revised: Large/XL/XXL Table 4 describes this phenomena in which we list efﬁcient alternatives to the canonical model sizes using the DeepNarrow strategy.

Original: Note that this list is not exhaustive.
Revised: Note that this list is not exhaustive.

Original: Firstly, we ﬁnd that signiﬁcantly increasing the depth of the small model does substantially better in terms of the compute-performance trade-off and may result in pareto-efﬁcient models.
Revised: Firstly, we ﬁnd that signiﬁcantly increasing the depth of the small model does substantially better in terms of the compute-performance trade-off and may result in pareto-efﬁcient models.

Original: The Small 16L model achieves comparable performance to Base while being 40% faster, cost 50% less parameters and has only about 63 .
Revised: The Small 16L model achieves comparable performance to Base while being 40% faster, cost 50% less parameters and has only about 63 .

Original: 1% of total FLOPs.
Revised: 1% of total FLOPs.

Original: Alternatively, the Small 24L model has 87% of FLOPs of the base model, similar speed (steps/s), and only 16% parameter savings and yet outperforms Base on all three downstream tasks.
Revised: Alternatively, the Small 24L model has 87% of FLOPs of the base model, similar speed (steps/s), and only 16% parameter savings and yet outperforms Base on all three downstream tasks.

Original: Meanwhile the canonical large model can be outperformed by a base model of 36 layers with 16% parameter saving and lower ﬂops cost.
Revised: Meanwhile the canonical large model can be outperformed by a base model of 36 layers with 16% parameter saving and lower ﬂops cost.

Original: The
Revised: The

Original: Large 36L model is only 37% of the parameter cost of the XL model and yet outperforms the XL[Table] model on all three downstream tasks.
Revised: Large 36L model is only 37% of the parameter cost of the XL model and yet outperforms the XL model on all three downstream tasks.

Original: Finally, the XL 32L model is only a third the size of the XXL model, approximately consume 44% the number of FLOPs of the XXL model and is about 3 times faster on the same hardware.
Revised: Finally, the XL 32L model is only a third the size of the XXL model, approximately consume 44% the number of FLOPs of the XXL model and is about 3 times faster on the same hardware.

Original: The Limits of Depth vs Width We note an obvious limitation with our advice.
Revised: The Limits of Depth vs Width We note an obvious limitation with our advice.

Original: Scaling depth has an obvious limiter, i.e., they are non-parallelizable across different machines or devices and every computation has to always wait for the previous layer.
Revised: Scaling depth has an obvious limiter, i.e., they are non-parallelizable across different machines or devices and every computation has to always wait for the previous layer.

Original: This is unlike width, which can be easily parallelizable over thousands or hundreds of thousands of devices.
Revised: This is unlike width, which can be easily parallelizable over thousands or hundreds of thousands of devices.

Original: Within the limitation of scaling to 64 workers with a model parallelism of 32 , we still ﬁnd that scaling depth can still improve the
Revised: Within the limitation of scaling to 64 workers with a model parallelism of 32 , we still ﬁnd that scaling depth can still improve the

Original: Pareto-efﬁciency of models.
Revised: Pareto-efﬁciency of models.

Original: From our experiments, from Table 4, we see that the efﬁcient small
Revised: From our experiments, from Table 4, we see that the efﬁcient small

Original: DeepNarrow models (e.g., Small 16L etc) are still much faster than the base models.
Revised: DeepNarrow models (e.g., Small 16L etc) are still much faster than the base models.

Original: Things get tricky as we approach larger models where model parallelism makes it difﬁcult to compare the utility between wide and deep models.
Revised: Things get tricky as we approach larger models where model parallelism makes it difﬁcult to compare the utility between wide and deep models.

Original: To this end, we believe the proposed scaling protocol holds within a certain hardware limit.
Revised: To this end, we believe the proposed scaling protocol holds within a certain hardware limit.

Original: Scaling to extreme number of machines via model parallelism (of width) is out of scope of this paper.
Revised: Scaling to extreme number of machines via model parallelism (of width) is out of scope of this paper.

Original: Another potential drawback to depth-scaling is that this may inﬂuence the stability of training these models (due to vanishing gradients).
Revised: Another potential drawback to depth-scaling is that this may inﬂuence the stability of training these models (due to vanishing gradients).

Original: However, we did not observe this in our experiments with T5 models.
Revised: However, we did not observe this in our experiments with T5 models.

Original: [-2.2e+12 4.4e+12 8.8e+
-]
Revised: [-2.2e+12 4.4e+12 8.8e+
-]

Original: 1.8e+13 3.5e+13 7.0e+13 FLOPS -2.

Original: N e g a t

Original: i v e L

Original: o g - P

Original: e r p l [Figure] Relationship of Model Depth with Paretofrontier Figure 4 shows the performance of scaling small, base and large models by depth.
Revised: Relationship of Model Depth with Pareto-frontier Figure 2 shows the performance of scaling small, base and large models by depth.

Original: It is clear that the small model (green) dominates the Pareto-frontier initially but slowly tapers off at a certain point.
Revised: It is clear that the small model (green) dominates the Paretofrontier initially but slowly tapers off at a certain point.

Original: Here, we note that the depth-scaled small model is more Pareto-efﬁcient compared to the base model.
Revised: Here, we note that the depth-scaled small model is more Pareto-efﬁcient compared to the base model.

Original: After the Pareto-efﬁciency of the small  model tapers off, the base model (red line) becomes Pareto-efﬁcient.
Revised: After the Pareto-efﬁciency of the small [Table] [Figure] model tapers off, the base model (red line) becomes Pareto-efﬁcient.

Original: Similarly, this tapers off and the large model becomes Pareto-efﬁcient.
Revised: Similarly, this tapers off and the large model becomes Pareto-efﬁcient.

Original: On the depth-width interplay We also note that(Levine et al., 2020) proposed scaling laws speciﬁc to the depth-width interplay and proposed an optimal depth-width ﬁt.
Revised: In this paper, we present several important ﬁndings pertaining to training and practical usage of efﬁcient transformer architectures. Speciﬁcally, we discovered that scaling laws differ in upstream and downstream.

Original: While our overall messaging is similar that the current models are depth inefﬁcient (not depth-scaled enough), a substantially more optimistic perspective on depth scaling was presented in (Levine et al., 2020). Even with wide networks (e.g., d ff > 16 k ) ) we still ﬁnd a limit to depth-scaling these wide XL models.
Revised: Contrary to prior work (Kaplan et al., 2020), we emphasize the importance of model shape for ensuring strong downstream performance. Next, we also discovered that scaling happens rather differently at different compute regions and scaling a small model would behave differently from scaling a large model.

Original: The optimal ﬁt predicted in (Levine et al., 2020) for d ff = 13 .
Revised: We highlight that this has implications since model development in a certain compute region could potentially not transfer to another compute region. We go on to show that not all model knobs are created equal, i.e., some knobs (such as depth) has a larger impact on the

Original: 5 K is 80 layers.
Revised: Pareto-frontier.

Original: On the contrary, we ﬁnd it hard to scale XL models beyond 32 layers (or 64 layers considering EncoderDecoder models). Our earlier point on scaling laws differing in compute regions may also show that it may be intrinsically difﬁcult to extrapolate inter-plays between conﬁgurations.
Revised: Finally, we propose a simple but highly effective improved scaling protocol.

Original: To this point, our perspective that width scaling becomes way more important after a certain depth threshold aligns with (Levine et al., 2020).
Revised: With this strategy, we obtain models with similar downstream ﬁnetuning quality while having 50% fewer parameters and/or training 40% faster.

Original: 4. T RANSFERABILITY OF R ESULTS TO V ISION T RANSFORMERS (V I T)
Revised: A PPENDIX . T RANSFERABILITY OF R ESULTS TO V ISION T RANSFORMERS (V I T)

Original: Following our language experiments, and as per the advice of (Narang et al., 2021), in order to examine our DeepNarrow scaling strategy in another domain and check if the observations extends to the cases where Transformers are applied on other modalities, we pre-retrained several different conﬁgurations of Vision Transformer (ViT; Dosovitskiy et al., 2020) and evaluated on downstream few-shot image recognition task.
Revised: Following our language experiments, and as per the advice of (Narang et al., 2021), in order to examine our DeepNarrow scaling strategy in another domain and check if the observations extends to the cases where Transformers are applied on other modalities, we pre-retrained several different conﬁgurations of Vision Transformer (ViT; Dosovitskiy et al., 2020) and evaluated on downstream few-shot image recognition task.

Original: We focused on investigating the pareto-efﬁciency of DeepNarrow small models compared to base models.
Revised: We focused on investigating the pareto-efﬁciency of DeepNarrow small models compared to base models.

Revised: We follow the exact same setup as Dosovitskiy et al.

Revised: and pre-train ViT on the JFT dataset (Sun et al., 2017) with 18k classes and 303M images, for 7 epochs.

Revised: We evaluate our model on ImageNet 10-shot classiﬁcation.

Revised: In our experiments, we use the patch size of 32 × 32 .

Original: [Table]
Revised: [Table]

Original: Results Table 5 report results on ViT experiments.
Revised: Results Table 5 report results on ViT experiments.

Original: When considering the number of trainable parameters or FLOPs, we observe that DeepNarrow scaling of the ViT-S model achieves better Pareto efﬁciency compared to the ViT-B model.
Revised: When considering the number of trainable parameters or FLOPs, we observe that DeepNarrow scaling of the ViT-S model achieves better Pareto efﬁciency compared to the ViT-B model.

Original: Notably, when L = 24 , the model achieves better few-shot accuracy with 15% less parameters, 11% less FLOPs and achieves +1 .
Revised: Notably, when L = 24 , the model achieves better few-shot accuracy with 15% less parameters, 11% less FLOPs and achieves +1 .

Original: 4% percentage improvement in accuracy.
Revised: 4% percentage improvement in accuracy.

Original: With respect to the step per seconds (speed), given ViT-S L =24 or ViT-S L =28 add to the sequential operations in depth, they become a bit slower than ViT-B.
Revised: With respect to the step per seconds (speed), given ViT-S L =24 or ViT-S L =28 add to the sequential operations in depth, they become a bit slower than ViT-B.

Original: However, we consider 6 .
Revised: However, we consider 6 .

Original: 11 s and 6 .
Revised: 11 s and 6 .

Original: 74 s to be reasonably within the same ballpark.
Revised: 74 s to be reasonably within the same ballpark.

Original: In short, the ViT-S L =24 is still a compelling alternative.C ONCLUSION
Revised: In short, the ViT-S L =24 is still a compelling alternative.

Original: In this paper, we present several important ﬁndings pertaining to training and practical usage of efﬁcient transformer architectures.

Original: Speciﬁcally, we discovered that scaling laws differ in upstream and downstream.

Original: Contrary to prior work (Kaplan et al., 2020), we emphasize the importance of model shape for ensuring strong downstream performance.

Original: Next, we also discovered that scaling happens rather differently at different compute regions and scaling a small model would behave differently from scaling a large model.

Original: We highlight that this has implications since model development in a certain compute region could potentially not transfer to another compute region.

Original: We go on to show that not all model knobs are created equal, i.e., some knobs (such as depth) has a larger impact on the

Original: Pareto-frontier.

Original: Finally, we propose a simple but highly effective improved scaling protocol.

Original: With this strategy, we obtain models with similar downstream ﬁnetuning quality while having 50% fewer parameters and/or training 40% faster.

Original: R EPRODUCIBILITY S TATEMENT

Original: All code required to run experiments in this paper has been already open sourced.

Original: We will not mention the link here for anonymity reasons.

Original: We will publicly (and have obtained approval) release all checkpoints trained in this paper in Q4 2021.

Original: E THICS S TATEMENT

Original: Training large Transformer models can be ﬁnancially and environmentally costly.

Original: Our work performs an extensive search of Transformer architectures so that the community wouldn’t have to.

Original: We believe our computational investment and scaling recommendation would have an overall net positive effect, i e., we would save some compute costs for many practitioners and researchers.

Original: Given the widespread adoption of Transformer models across the world, we believe this would make a positive ‘roi’ at the end

Original: A PPENDIX A M ODELING SCALING TRENDS BY FITTING POWER LAW

Original: When studying the scaling behavior of models, several previous work observed a power-law relationship between the empirical performance and different scaling factors, like the number of parameters (Kaplan et al., 2020).

Original: Here, we also ﬁt a curve following power law to model the scaling trend for data points from Figure 1.

Original: Inspired by recent work on scaling law (Kaplan et al., 2020; Hernandez et al., 2021; Abnar et al., 2021), we use the following function for ﬁtting the power-low:

Original: [Equation]

Original: where e refers to the error (e.g., loss for upstream and 1 − accuracy for downstream), N refers to the number of parameters, N c and α are constants and e ∞ is the irreducible error.

Original: [Figure]

Original: In this work, we mainly focused on how the scaling behavior of Transformers can be different with respect to the upstream (pre training) versus downstream (

Original: -tuning performances.

Original: Figure 5 shows the power-law curve ﬁtted to the data points from Figure 1.

Original: We cannot directly compare the values of the function we ﬁt for upstream and downstream, since the performance metrics are completely different.

Original: However, we can see a much higher ﬁtting error when modeling the downstream performance compared to modeling the upstream performance.

Original: This shows that unlike pre-training performance, ﬁne-tuning performance is not easily predictable based on parameter size, and once again, this indicates that the shape of models plays an important role on how it performs on the target task.

Original: A. H OW TRANSFERABLE ARE THESE RESULTS ACROSS MULTIPLE DIVERSE NLP TASKS ?
Revised: . H OW TRANSFERABLE ARE THESE RESULTS ACROSS MULTIPLE DIVERSE NLP TASKS ?

Original: To verify that our scaling protocols transfer to tasks outside of the 17 tasks explored in (GLUE/SuperGLUE and SQuaD), we run additional experiments on a myriad of diverse NLP tasks.
Revised: To verify that our scaling protocols transfer to tasks outside of the 17 tasks explored in (GLUE/SuperGLUE and SQuaD), we run additional experiments on a myriad of diverse NLP tasks.

Original: Verifying whether the ﬁndings generalize outside GLUE and SuperGLUE is important since we do not want to fall prey to the benchmark lottery problem (Dehghani et al., 2021).
Revised: Verifying whether the ﬁndings generalize outside GLUE and SuperGLUE is important since we do not want to fall prey to the benchmark lottery problem (Dehghani et al., 2021).

Original: As such, the purpose of this additional experiment is to verify if our results are universal enough for a general recommendation.
Revised: As such, the purpose of this additional experiment is to verify if our results are universal enough for a general recommendation.

Original: Setup In this experiment, we compare the base T5 transformer with the efﬁcient 24 layer small model using the DeepNarrow strategy, which has 14% less parameters and 10% less FLOPS compared to the T5 base model.
Revised: Setup In this experiment, we compare the base T5 transformer with the efﬁcient 24 layer small model using the DeepNarrow strategy, which has 14% less parameters and 10% less FLOPS compared to the T5 base model.

Original: This ﬁnetuning protocol uses a constant learning rate of 10 − 3 and a batch size of 128 for all tasks.
Revised: This ﬁnetuning protocol uses a constant learning rate of 10 − 3 and a batch size of 128 for all tasks.

Original: Note that we used the same pretrained models as earlier sections that produced ﬁnetuned results on SuperGLUE and GLUE.
Revised: Note that we used the same pretrained models as earlier sections that produced ﬁnetuned results on SuperGLUE and GLUE.

Original:  We conduct experiments on a total of 12 extra tasks, i.e., 6 tasks of Rainbow (Lourie et al., 2021) which contains commonsense reasoning tasks,generation/summarization tasks (XSum (Narayan et al., 2018), CNN/Dailymail (See et al., 2017) and MultiNews (Fabbri et al., 2019)), along with 3 text classiﬁcation tasks (civil comments (Borkan et al., 2019), wiki toxicity (Wulczyn et al., 2017) and IMDb reviews (Maas et al., 2011)).
Revised: Diverse NLP tasks We conduct experiments on a total of 12 extra tasks, i.e., 6 tasks of Rainbow (Lourie et al., 2021) which contains commonsense reasoning tasks, 3 generation/summarization tasks (XSum (Narayan et al., 2018), CNN/Dailymail (See et al., 2017) and MultiNews (Fabbri et al., 2019)), along with 3 text classiﬁcation tasks (civil comments (Borkan et al., 2019), wiki toxicity (Wulczyn et al., 2017) and IMDb reviews (Maas et al., 2011)).

Original: For the Rainbow suite, we co-train all tasks in (Lourie et al., 2021) and report peak validation results. 
Revised: For the Rainbow suite, we co-train all tasks in (Lourie et al., 2021) and report peak validation results. [Table] [Table]

Original:  Results Our ﬁndings show that DeepNarrow outperforms  base for 11 out of 12 tasks considered. The average performance of base on all 12 tasks is 63.2 while DeepNarrow performs at 63.5.
Revised: [Table] Results on other tasks Table 6, Table 7 and Table 8 reports results on these 12 tasks. On alladditional diverse NLP tasks considered , we show that the Pareto-efﬁcient alternative outperforms or ties with the base T5 model on 11 out of 12 tasks where 4 of them are ties.

Original: Note that DeepNarrow has 10% less FLOPs and has 14% less parameters.
Revised: It is good to bear in mind that this is a model which is effectively smaller in parameters and has 10% less FLOPs.

Original: [Table]
Revised: [Table]

Original: B S UPPLEMENTARY R ESULTS

Original: [Figure]
Revised: [Table]

Original: [Figure] [Table] 
Revised: [Table] [Figure]

Original: [Table]
Revised: [Table]

Original: S u p e r G l u e A cc u r a c y
Revised: u p e r G l u e A cc u r a c y

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: G lu e A cc u r a c y
Revised: G l A cc u r a c y

Original: [Figure] [Table] 
Revised: [Table] [Figure]

Revised: XL

Original: [Table]
Revised: [Figure]

Original: u a d A cc u r a c y
Revised: NL

Original: [Figure]
Revised: [Figure]

Revised: NL6-

Original: [Table]
Revised: [Figure]

Original: NL12-XXL NL6-XXXL
Revised: NL [Figure]

Original: NL12-XXXL 1.8e+13 3.5e+ 7.0e+ 1.4e+14 2.8e+14 FLOPS 85.
Revised: XXXL 1.8e+13 3.5e+ 7.0e+ 1.4e+14 2.8e+14 FLOPS 83.

Original: S q u a d A cc u r a c y
Revised: G l u e A cc u r a c y

Original: Large
Revised: Large

Original: XL
Revised: XL

Original: NL NL
Revised: NL NL

Original: NL36 NL32-LG
Revised: NL NL32-LG

Original: NL8-XL
Revised: NL8-XL

Original: NL32-XL
Revised: NL32-XL

Original: NL6-XXL
Revised: NL6-

Original: NL8-XXL NL12-XXL NL6-XXXL
Revised: [Table] -XXL

Original: [-NL12-XXXL 2.0e3.0e-1 4.0e-1 5.0e-16.0e-17.0e-
-]
Revised: [-NL12-XXXL 2.0e3.0e-1 4.0e-1 5.0e-16.0e-17.0e-
-]NL12- [Table] [Table] [Table]

Original: 9.0e-1 1/Throughput 85.
Revised: /Throughput [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Original: S q u a d A cc u r a c y
Revised: G l u e A cc u r a c y

Original: Large
Revised: Large

Original: XL
Revised: XL

Original: NL NL
Revised: NL NL

Original: NL36NL32-LG

Original: NL8-XL NL32-XL NL6-XXL NL8-XXL
Revised: [Table]

Original: NL12-XXL NL6-XXXL NL12-XXXL [Figure]
Revised: [Table] XL

Original: [Table]
Revised: [Table]

Original: N e g a t i v e L o g - P e r p l e x i t y
Revised: XL

Original: [Table]
Revised: [Table]

Original: [Figure] [Table]
Revised: [Table] [Table]

Original: t i v
Revised: [Table] [Figure] [Figure] [Figure]

Original: [Table]
Revised: [Figure]

Original: - P
Revised: [Figure] [Figure] u [Figure]

Revised: r

Revised: [Figure]

Revised: y

Revised: [Figure]

Revised: 1.8e+

Revised: [Figure]

Revised: 5e+ 7.0e

Revised: [Figure]

Revised: 1.4e

Revised: [Figure]

Revised: S q u a d A cc u r a c y

Revised: Large

Revised: XL

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Table]

Revised: g a t i v

Revised: [Table]

Revised: P

Revised: [Table]

Original: e r p l
Revised: r p l

Original: [Table]
Revised: [Table]

Revised: x i

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: .0e

Revised: [Table]

Revised: 2 5.0e

Original: [Table]
Revised: [Table]

Revised: 6.0e

Revised: [Table]

Revised: 2 7.0e

Original: [Table]
Revised: [Table]

Revised: 28.0e-29.0e-21.0e-1 1.2e-1 1/Throughput -2.

Original: [Table]
Revised: [Table]

Original: u p e r G lu e A cc u r a c y
Revised: N e g a t i v e L o g - P e r p l e x i t y

Original: [Figure] [Table] 
Revised: [Table] [Figure]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Figure] [Table] [Figure]
Revised: 1.7e+ 3.4e+ 6.7e+ 1.3e+ 2.7e+8 Params

Original: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Figure] [Table] [Figure]
Revised: S u p e r G l u e A cc u r a c y

Original: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Figure] [Table] [Figure]
Revised: 1e+12 2.2e+ 4.4e+

Original: [Table] [Figure] [Table] [Figure]
Revised: 8.8e+12 1.8e

Original: [Table]
Revised: [Table]

Original: [-.1e+
 2.2e+ 4.4e+ 8.8e+12 1.8e+13-] FLOPS
Revised: [-.1e+
 2.2e+ 4.4e+ 8.8e+12 1.8e+13-]13 FLOPS

Original: S u p e r G l u e A cc u r a c y
Revised: S u p e r G l u e A cc u r a c y

Original: Small FF1K-SM
Revised: Small [Figure]

Original: FF3K-SM
Revised: -SM [Figure]

Original: FF6K-SMFF9K-SM FF12K-SM
Revised: [Figure] -SMFF9K-SM [Figure]

Original:  EL2-SM EL4-SM
Revised: -SM EL2-SM EL4-

Original: EL8-SM
Revised: [Figure] [Figure]

Original: EL12-SM
Revised: EL12-

Original: DL2-SM
Revised: [Figure] [Figure] [Figure] [Figure]

Original: DL4-SM
Revised: [Figure] [Table] [Figure] [Figure]

Original: DL8-SM DL12-SM
Revised: [Table] [Figure] [Table]

Original: NL2-SMNL4-SM
Revised: [Figure] [Table] [Figure] [Table]

Original: NL8-SM
Revised: [Figure]

Original: NL16-SM NL24-SM
Revised: [Table] [Figure] [Table]

Original: KV16-SM
Revised: [Figure] [Table] [Figure] [Table]

Original: KV32-SM
Revised: [Figure]

Original: KV128-SM KV256-SM
Revised: [Table] [Figure] [Table]

Original: DM256-SM DM768-SM
Revised: [Figure] DM2K [Figure]

Original: DM1K-SM
Revised: [Table]

Original: DM2K-SM 4.0e-2 5.0e-2 6.0e-2 7.0e-2 8.0e-29.0e-21.0e-1 1.2e-1 1/Throughput
Revised: -2 5.0e-2 6.0e-2 7.0e-2 8.0e-29.0e-21.0e-1 1.2e-1 1/Throughput

Original: S u p e r G l u e A cc u r a c y
Revised: S u p e r G l u e A cc u r a c y

Original: SmallFF1K-SM FF3K-SM FF6K-SMFF9K-SM FF12K-SM
Revised: Small

Original: EL2-SM EL4-SM EL8-SM EL12-SM
Revised: [Figure]

Original: DL2-SM DL4-SM DL8-SM DL12-SM
Revised: [Figure]

Original: NL2-SMNL4-SM NL8-SM NL16-SM NL24-SM
Revised: [Figure]

Original: KV16-SM KV32-SM KV128-SM KV256-SM
Revised: EL2-

Original: DM256-SM DM768-SM DM1K-SM DM2K-SM
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [-+ 3.4e+ 6.7e+ 1.3e+ 2.7e+8 Params
-]
Revised: [-+ 3.4e+ 6.7e+ 1.3e+ 2.7e+8 Params
-]

Original: [Table] [Figure] [Table]
Revised: [Figure] [Figure]

Original: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Table]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Table] [Figure]
Revised: [Table] [Table]

Original: [Table] [Figure]
Revised: [Table] [Table]

Original: {+Params
+}
Revised: {+Params
+}

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: e g a t i v e L o g - P e rp l e x i t y
Revised: r

Original: [Figure]
Revised: [Figure]

Original: N
Revised: c

Original: [Figure]
Revised: [Figure]

Original: g

Original: [Figure]
Revised: [Figure]

Original:  t i v eL o g
Revised: g a t i v e

Original: [Figure]

Original:  P e r p [Figure] i t [Figure]
Revised: L o g - P e r p l e x i t y

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: u
Revised: u

Original: [Figure]
Revised: [Figure]

Original: r G l ue A cc u r a c y
Revised: r G l u

Original: [Figure]
Revised: [Figure]

Original: S u p e r G l u e A cc u r a c y
Revised: A c c u r a c y

Original: Base FF1K

Original: FF2K [Table]
Revised: [Figure]

Original: FF12K

Original: DM DM DM1K DM2K
Revised: c c u r

Original: NL [Table]
Revised: [Figure]

Original: [Table] [Table] G l u A cc u r a c y
Revised: c y

Original: [Figure]
Revised: [Figure]

Original: [Table] [Figure] [Table] [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table][Figure] [Figure] d
Revised: [Table]

Original: A cc
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Figure]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original:  a c 
Revised: [Table] [Figure] S q u a d A cc u r a c y

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: u p e r G l u e A cc u r a c y
Revised: u p e r G l u e A cc u r a c y

Original: [Table]
Revised: [Table]

Original: [Figure] [Table] [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: G l u e A cc u r a c y
Revised: cc u r a c y

Original: [Table]
Revised: [Table]

Original: [Figure] [Table] 
Revised: [Table] [Figure]

Original: [Figure][Table] [Table] [Figure]
Revised: [Figure]