Revisiting Neural Scaling Laws in Language and Vision
Anonymous Author(s)
AfﬁliationAddress email
Abstract
The remarkable progress in deep learning in recent years is largely driven byimprovements in scale, where bigger models are trained on larger datasets forlonger schedules.
To predict the beneﬁt of scale empirically, we argue for a morerigorous methodology based on the extrapolation loss, instead of reporting the bestﬁtting (interpolating) parameters.
We then present a recipe for estimating scalinglaw parameters reliably from learning curves.
We demonstrate that it extrapolatesmore accurately than previous methods in a wide range of architecture familiesacross several domains, including image classiﬁcation, neural machine translation(NMT) and language modeling, in addition to tasks from the BIG-Bench evaluationbenchmark.
Finally, we release a benchmark dataset comprising of 90 evaluationtasks to facilitate research in this domain.
Introduction
Scale has led to innovative research in both the vision domain [10, 14, 26, 35, 40] and the naturallanguage processing (NLP) [8, 12] domain.
Recent work has found that scaling up the data size [34],the model size [26, 35], the training schedule [5, 39] or all of them together [8, 40] often lead toimproved performance.
More importantly, scaling up the data size and the model size together canbetter utilize the compute resources.
Scaling laws have been properly studied in several works,e.g. [3, 18–20, 23], and it has been found that the performance f ( x ) (e.g. excess loss) often follows apower law f ( x ) ∼ βx c as one varies a dimension of interest x , such as the data or the model size.
While theoretical arguments alone seldom predict scaling law parameters in modern neural architectures [2, 21, 32], it has been observed that the beneﬁt of scale could be predicted empirically[3, 4, 9, 17, 18, 20, 22, 23, 28, 30, 31].
The general approach is to acquire a learning curve , i.e. acollection of samples ( x, f ( x )) , where x is a dimension of interest such as the training data size whilef ( x ) is a measure of performance, such as the validation loss.
After that, scaling law parameters areestimated, e.g. by computing the best-ﬁtting values of β and c in the model f ( x ) = βx c .
Given theestimated scaling law parameters, one can then extrapolate by predicting the performance f ( x ) forlarge values of x .
Such learning curve extrapolation has found many applications, of which four seem to be moreprominent.
First, it offers a tool for understanding deep neural networks; e.g. how the architectureand data distribution impact scaling behaviors [1–3, 18, 20, 23, 30, 32].
Second, it has been usedfor sample size planning, particularly in data-scarce domains such as medicine [4, 9, 17, 28].
Third,it can reduce the environmental footprint of experimentation by terminating experiments early andaccelerating hyper-parameter search [13, 20, 22].
Forth, it has been applied in neural architecturesearch (NAS) [15, 20, 25].
In addition, learning curve prediction offers a different methodology forcomparing performance; e.g. instead of comparing accuracy on a single dataset, one can also examinethe full (hypothetical) scaling curves.
101010 9 Upstream Examples 0.
10 - s h
o t v a li d a t i o n e rr o r
M 101010 9 Upstream Examples 0.
M 101010 9 Upstream Examples 0. M 101010 9 Upstream Examples 0.
M4 (Ours)
UsedNew
[Figure] [Figure]
[Figure] [Figure]
A v e r a g e
R a n k i n g
[Figure]
perform well in not one
However, in order to achieve such beneﬁts in practice, it is imperative that scaling laws extrapolateaccurately instead of merely interpolating the learning curve.
To our knowledge, a validation ofthis sort based on extrapolation is often lacking in the literature and previous works have generallyreported the best-ﬁtting (interpolating) parameters.
We demonstrate why this can be misleading in
Section 4, where we illustrate how the scaling exponent c that extrapolates best can be quite differentfrom the exponent that best ﬁts the given (ﬁnite) learning curve.
In addition, we propose an estimatorfor scaling laws denoted M 4 , which extrapolates more accurately than previous methods as shownin Figure 1.
We validate the proposed estimator in several domains, including image classiﬁcation,neural machine translation, language modeling, and other related tasks.
Our contributions are:
We argue in Section 4 for a more rigorous methodology to validate scaling law parametersbased on extrapolation, instead of only reporting the best-ﬁtting (interpolating) parameters.2.
We propose a recipe in Section 3 to estimate scaling laws reliably from learning curves.
Thenew estimator is veriﬁed across several domains: image classiﬁcation, neural machine trans-
lation (NMT), language modeling
and tasks from BIG-Bench evaluation benchmark [6].
We use the proposed recipe to study the impact of the neural architecture’s type and size onscaling exponents.
We release a benchmark dataset consisting of 90 tasks to accelerate research in scaling laws.
Related work
Power law scaling in deep neural architectures has been veriﬁed in a wide range of domains, includingimage classiﬁcation [2, 20, 32, 40], language modeling [20, 23, 32], NMT [3, 18–20], and speechrecognition [20].
To explain this theoretically, at least for data scaling, several works have arguedfor a power law behavior under various contexts.
For instance, in the universal learning setting [7]under the realizable case with a 0-1 misclassiﬁcation loss, power law scaling emerges with exponentc = 1 if the hypothesis space has an inﬁnite Littlestone tree but not an inﬁnite VC-Littlestone tree [7].
Another argument for the exponent c = 1 can be made in the non-realizable setting if the chosen lossis sufﬁciently smooth and the model size is limited by deriving the variance of the empirical solutionaround its population limit [2].
A more relevant setting for deep neural networks is to assume that themodel size is effectively inﬁnite and the loss is Lipschitz continuous (e.g. continuous loss in bounded domains).
Under the latter assumptions, it has been argued that the scaling exponent would satisfyc = O (1 /d ) where d is the intrinsic dimension of the data manifold [2, 21, 32].
This is consistentwith the fact that scaling exponents often satisfy c (cid:28) 1 and that the exponent seems to be independentof the neural network architecture size as long as the architecture is sufﬁciently large [20, 23, 32].
Writing x for the dimension of interest (e.g. data size) and ε x for the error/loss of the model as afunction of x , three function classes have been used in the literature to model the performance ε x as afunction of x while capturing its expected power law behavior:
M 1 : The simplest model assumes a power law throughout the domain of x : ε x = βx c .
Thishas been used, for example, to estimate the required sample size in healthcare [9], neuralmachine translation (NMT) [19], and language models [23], among others [20, 22, 32].
M 2 : To capture saturating performance for large x (i.e. when the Bayes optimal risk is boundedaway from zero), a parameter ε ∞ is added: ε x − ε ∞ =
βx c .
This is, perhaps, the mostcommonly used model in the literature; see for instance [1, 13, 17, 19, 20, 22, 28, 30, 31].
M 3 : A different parameterization has been recently used in NMT [3]: ε x = β ( x − 1 + γ ) − c .
Variants of this approach were used previously in studying, for example, scaling lawsin vision transformers [40], accelerating hyper-parameter optimization [13], and (moregenerally) in learning curve prediction [25].
In this work, we introduce a fourth estimator M 4 and verify experimentally that it outperforms theabove methods in terms of its extrapolation capability in several domains, as summarized in Figure 1.
We describe M 4 and discuss its rationale in Section 3.
The Scaling Law Estimator M 4
Motivation.
The function class M 2 , in which it is assumed that ε x = ε ∞ + βx c , captures (bydeﬁnition) what it means for the excess risk to follow a power law.
Hence, a question naturally arises:do we need any other function classes to estimate the scaling law parameters ε ∞ , β and c ?
To see why using M 2 can occasionally fail, consider the following simple classiﬁcation problemwhose optimal Bayes risk is known.
Suppose that the instances x ∈ S d − 1 are generated uniformlyat random from the unit sphere S d − 1 = { x ∈ R d : || x || 2 = 1 } .
In addition, let the (ground-truth)labeling function be given by:
[Equation]
with probability
[Equation]
(cid:63) ,
[Equation]
) , otherwise , for some ﬁxed w (cid:63) ∈ S d − 1 .
If a classiﬁer is trained using, for example, logistic regression, themisclassiﬁcation error rate ε x of the learning algorithm as a function of the data size x would typicallyundergo three stages as illustrated in Figure 2 [20].
First, we have saturating performance for smallsample sizes shown on the left, in which the trained model does not perform much better than randomguessing.
Second, we have a transitional stage in which the performance of the model improvesquickly
but it does not constitute a power law yet.
Third, we have the ﬁnal power law regime in whichthe excess risk ε x − ε ∞ ﬁts a power law curve.
Let D
= { ( x, ε x ) } be the learning curve and write D τ = { ( x, ε x ) : x ≥ τ } ⊆ D 0 for the restrictionof the learning curve to x ≥ τ .
To extrapolate from a learning curve, we train each of the four models
M 1 , M 2 , M 3 , and M 4 on the learning curve after applying a cutoff τ to mitigate the effect of smalldata samples.
Then, we plot the excess risk ε x − ε (cid:63) ∞ predicted by each model, where ε (cid:63) ∞ =
δ isthe (ground-truth) Bayes risk.
Since ε (cid:63) ∞ is known exactly, an accurate model that extrapolates wellwould produce a linear curve in each plot.
As shown in Figure 2, M 2 is accurate only when the dataresides entirely in the power law regime (rightmost ﬁgure), whereas M 4 works well in all cases.
Derivation.
The function class M 4 arises from several natural requirements.
First, we would likeour function class to be sigmoid-like so that it fails only gracefully when the data deviates from theexpected power law behavior; e.g. to avoid failures like that of M 1 and M 2 in Figure 2(left).
Second,we would like our function class F to reduce to power law functions f ( x ) → ε ∞ + βx c as x → ∞ .
More precisely, we require that:
[Equation]
[Figure]
Excess
Risk
[Figure]
where instances x ∈ R d are drawn uniformly at random from the surface of the unit sphere and the label is binary y ∈ {− 1 , +1 } with noise rate
δ (see Section 3).
In this experiment, d = 100 and δ = 0 . 2 .
In each ﬁgure, only the data sizes that exceed the indicated cutoff value are used to estimate the scaling law parameters.
M 2 is accurate only when the data resides entirely in the power law regime (rightmost ﬁgure), whereas M 4 works well in all cases.
To reiterate, this is because power law behavior has been empirically veriﬁed in a wide range ofdomains (see Section 2).
Third, we would like our function class F to be expressive enough tocontain all of the functions in M 2 , i.e. M 2 ⊂ M 4 , so that using M 4 becomes equivalent to using
M 2 when the observed learning curve resides entirely in the power law regime.
If we take the ﬁrst requirement above on the shape of the function, a general approach to achieve thisis to write the performance as a convex combination of the form:
[Equation]
for some function γ ( x ) that satisﬁes lim x →∞ γ ( x ) = 0 and lim x → 0 γ ( x ) = ∞ .
Here, ε ∞ is thepredicted limiting performance when x → ∞ while ε 0 is the performance at the random-guessinglevel.
To meet the second requirement, we set γ ( x ) =
βx c for some learnable parameters β > 0 andc < 0 .
Rearranging the terms yields ( ε x − ε ∞ ) · ( ε 0 − ε x ) − 1 = γ ( x ) .
=
βx c .
Finally, we introduce alearnable parameter α > 0 to meet our ﬁnal requirement:
[Equation]
With α = 0 , our function class M 4 reduces to M 2 as required.
By differentiating both sides of theequation above and noting that c < 0 , we deduce that ε x remains a monotone decreasing function ofx for all α ≥ 0 as expected.
In addition, by rearranging terms and using both the binomial theoremand the Lagrange series inversion theorem, we have the following asymptotic expansion for theexcess loss ε x − ε ∞ as x → ∞ :
[Equation]
This is a second-order expansion on the power law term βx c .
Choosing α > 0 allows the algorithmto handle measurements that deviate from the expected power law behavior; i.e. when the learningcurve does not entirely fall into the asymptotic power law regime.
The parameters to be ﬁtted here are α ≥ 0 , β > 0 , c < 0 and ε ∞ > 0 .
The parameter ε 0 correspondsto the value of the loss at the random-guessing level and can be either ﬁxed or optimized.
We ﬁx ε 0in our evaluation to be equal to the loss at the random-guessing level, although we observe similarresults when it is optimized.
In all four methods, scaling law parameters are optimized by minimizingthe square-log loss as discussed in Appendix A.1, which is similar to the approach used in [28].
Validating Scaling Laws using the Extrapolation Error
A common approach in the literature for estimating scaling law parameters is to assume a parametricmodel, e.g. M 2 , and reporting its best-ﬁtting parameters to an empirical learning curve (see forexample the prior works discussed in Section 2).
Afterwards, patterns are reported about the behaviorof the scaling law parameters; e.g. how the exponent varies with the architecture size.
We argue,next, for a more rigorous methodology based on the extrapolation loss, instead of only reporting thebest-ﬁtting (interpolating) parameters.
Speciﬁcally, choices of scaling law parameters that achieve
a
[Figure]
small interpolation error do not necessarily achieve a small extrapolation error so they may not, infact, be valid estimates of scaling law parameters.
Scaling law parameters should be validated bymeasuring how well they extrapolate .
To see why a validation of this sort matters, consider the following example.
If we pretrain a visiontransformer ViT/B/16 [14] on subsets of JFT-300M (a proprietary dataset with 300M examples and18k classes [34]) using the Adam optimizer [24] 1 , and evaluate the 10-shot error rate on ImageNetILSRCV2012 [11], we obtain the learning curve shown in Figure 3(left, in green).
Evidently, powerlaw emerges; i.e. ImageNet 10-shot error rate (shown in green) follows a linear curve on a log-logplot.
Hence, one might estimate, for example the scaling exponent c using least squares.
However, consider now the family of curves shown in Figure 3(left), all corresponding to M 2 butwith scaling exponents that vary from about c = − 0 .
24 to c = − 0 .
4 (while ﬁtting the parametersβ and ε ∞ ).
Note that all ﬁve curves overlap with each other signiﬁcantly.
Choosing the best ﬁttingparameters on the learning curve would favor a scaling exponent of c = − 0 .
24 as shown in Figure3(right).
However, if we validate the parameters by evaluating how well they extrapolate (i.e. howwell they predict performance when the number of seen examples x (cid:29) 10 9 ), a different pictureemerges.
We observe that a more accurate estimate of the scaling exponent is c = − 0 .
This isshown in Figure 3(left) and in Figure 3(right) by measuring the extrapolation loss.
Here, validation ismeasured using the root mean square error (RMSE) to the log-loss:
[Equation]
in which x is uniform over the set [10 9 , 2 × 10 9 ] , where ˆ ε x is the predicted loss while ε x is the actual.
We apply the logarithm so that we penalize the relative error and, hence, assign equal importance toall error scales (both large and small) 2 .
In summary, scaling law parameters that give the best ﬁt on the learning curve (i.e. lowest interpolationloss) do not generally extrapolate best.
When using extrapolation loss instead, different scaling lawparameters emerge.
In this work, we use extrapolation to evaluate the quality of scaling law estimators.
Experiments
We provide an empirical evaluation of the four scaling law estimators in several domains, includingimage classiﬁcation (72 tasks), neural machine translation (5 tasks), language modeling (5 tasks),and other language-related evaluations (10 tasks).
The dataset for neural machine translation isavailable at [3].
The code and dataset for the remaining tasks used in this evaluation are made publiclyavailable to facilitate further research in this domain 3 .
In all experiments, we divide the learning
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
curve into two splits: (1) one split used for training the scaling law estimators, and (2) one split usedfor evaluating extrapolation performance.
We set the cutoff between the two splits to be equal to where max is the maximum value of x .
We
5.1 Image Classiﬁcation
Architectures and Tasks.
We use three families of architectures: (1) big-transfer residual neuralnetworks (BiT)
[26], (2) vision transformers (ViT) [14], and (3) MLP mixers (MiX) [37].
For eachfamily, we have two models of different sizes as shown in Table 1 in order to assess the impact of thesize of the architecture on the scaling parameters.
We pretrain each architecture on JFT-300M [34]and evaluate the few-shot accuracy downstream on four datasets: (1) ImageNet [11], (2) Birds 200[38], (3) [-CIFAR
 [27],and-]CIFAR100 [27], and (4) Caltech101 [16].
For each downstream dataset, we report 5-shot,10-shot and 25-shot few-shot accuracy.
It results in a total of 72 tasks for each combination ofarchitecture, dataset, and few-shot metric.
Following [14, 26], we removed duplicate pre-trainingexamples between upstream JFT-300M dataset and all the downstream train and test sets.
Bootstrapped Examples.
In the few-shot image classiﬁcation setting under the transfer learningsetup, overﬁtting can occur if the upstream dataset is small, where training beyond a particular numberof steps would reduce the downstream validation accuracy .
This is demonstrated in Figure 4, wherewe pretrain on subsets of JFT-300M (upstream) and evaluate ImageNet 10-shot error (downstream).
Nevertheless, we observe that prior to reaching peak performance, training examples behave as ifthey were fresh samples.
This observation generalizes the bootstrapping phenomenon observed in[29], where it showed that training examples behave as fresh samples prior to convergence, whichwould be equivalent to our observation if no overﬁtting occurs.
Throughout the sequel, we refer tothe examples seen during training prior to peak performance as “bootstrapped examples" and usetheir number as the independent variable x when evaluating the scaling law estimators in this section.
Figure 5 illustrates how well each of the the four scaling law estimators M 1 , M 2 , M 3 , and M 4 canextrapolate from a given learning curve.
The complete set of ﬁgures is provided in Appendix A.3.
We observe that M 4 extrapolates better than other methods and produces learning curves thatapproximate the empirical results more faithfully.
As shown in Figure 1, M 4 outperforms the othermethods in more than 70% of the tasks in this domain.
M
M M
M
[Figure] [Figure]
Top row is for M 2 (nearly identical in this case to M 3 ) while the bottom row is for M 4 .
We observe that M 4 suggests estimates of the scaling exponents that are larger (in absolute magnitude) than in previous methods.
Also, larger architectures within the same family have more favorable exponents.
Impact of the Architecture.
Figure 6 plots the scaling exponent c in each architecture when thedownstream task is n -shot accuracy on ImageNet.
We observe that within each family of models,larger models have more favorable scaling exponents.
In addition, M 4 yields estimates of the scalingexponents that are larger in absolute magnitude than in other methods.
Figure 7 shows that suchdifferences in scaling exponents show up indeed in the slopes of the learning curves as expected.
5.2 Neural Machine Translation (NMT)
Next, we evaluate the scaling law estimators on NMT.
We use the setup studied in [3], in whichmodels are trained with the per-token cross-entropy loss using Adafactor optimizer [33] with abatch-size of 500K tokens and a dropout rate of 0.1 [3].
We use the encoder-decoder transformermodels 6L6L, 28L6L and 6L28L, where 28L6L means that the architecture has 28 encoders and 6decoders.
We also use the two architectures: decoder-only with language modeling loss (D/LM) andthe transformer-encoder with LSTM decoder (TE/LSTM).
These correspond to the architectures usedin Figure 1 in [3].
In all cases, performance is measured using log-perplexity on a hold-out dataset.
To evaluate the accuracy of the scaling law estimator, we ﬁt its parameters on the given learningcurve (for up to 256M sentence pairs) and use it to predict the log-perplexity when the architecture istrained on 512M sentence pairs.
Because the learning curves contain few points, we only evaluateon the 512M sentence pairs.
Table 2 displays the RMSE of each estimator.
Clearly, M 4 performsbetter than the other methods as summarized in Figure 1, which is consistent with the earlier resultsin image classiﬁcation.
[Figure] [Figure]
[Table]
[Figure] [Figure]
[Table]
5.3 Language Modeling
Next, we evaluate the scaling law estimators in language modeling, where the goal is to predictthe next token.
We use the LaMDA architecture used in [36], which is a decoder-only transformerlanguage model.
Five model sizes are used, ranging from 10 7 to 10 9 model parameters.
In eachmodel, we rescale validation loss to the unit interval [0 , 1] .
Figure 8 and Table 2 summarize theresults.
We observe that M 4 and M 2 perform best, with M 4 tending to perform better.
As statedearlier, M 4 becomes equivalent to M 2 when the learning curve resides entirely in the power lawregime, hence the similar performance.
Figure 9 displays the scaling exponents c predicted by eachestimator as a function of the architecture size.
We observe that M 1 and M 3 produce estimates ofc that are small in absolute magnitude.
However, in M 2 and M 4 , the scaling exponent is close to− 1 / 3 and decreases (in absolute magnitude) for larger models.
5.4 Scalable Tasks from the BIG-Bench Evaluation Benchmark
Finally, we evaluate the scaling law estimators on language tasks from the BIG-Bench collaborative benchmark [6].
Here, we pretrain a 262M-parameter decoder-only transformer (middlearchitecture in Figure 8) on language modeling and evaluate its 1-shot and 2-shot capabilities inﬁve language-related tasks.
We choose the ﬁve tasks that exhibit the highest learnability fromthe benchmark (i.e. improvement in performance when pretrained on language modeling, see [6]for details).
The ﬁve tasks are: linguistic_mappings, qa_wikidata, unit_conversion,mult_data_wrangling and date_understanding .
We use the benchmark’s preferred metrics inall cases, which is either “multiple choice grade" or “exact string match" depending on the task.
Table3 and Figure 1 summarize the results.
In this evaluation, both M 3 and M 4 perform best and equallywell.
In addition, we observe that M 1 and M 2 perform equally well and consistently worse than theother methods.
One possible reason is that the learning curves are quite noisy (see Appendix A.2).
Discussion
The remarkable progress in deep learning in recent years is largely driven by improvements in scale,where bigger models are trained on larger datasets for longer training schedules.
Several worksobserve that the beneﬁt of scale can be predicted empirically by extrapolating from learning curvesand this has found important applications, such as in sample size planning and neural architecturesearch.
However, to achieve such beneﬁts in practice, it is imperative that scaling laws extrapolateaccurately.
We demonstrate that scaling parameters that yield the best ﬁt to the learning curve do notgenerally extrapolate best, thereby challenging their use as valid estimate of scaling law parameters.
Hence, we argue for a more rigorous validation of scaling law parameters based on the extrapolationloss.
In addition, we present a recipe for estimating scaling law parameters that extrapolates moreaccurately than in previous works, which we verify in several state-of-the-art architecture across awide range of domains.
To facilitate research in this domain, we also release a benchmark datasetcomprising of 90 evaluation tasks.
We believe that the proposed scaling law estimator can be utilized,for example, to accelerate neural architecture search (NAS), which we plan to study in future work.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[N/A] We are aware of many appli-
cations using learning curve extrapolation, and plan to apply our new recipe to new
applications (e.g. NAS) to land the impact.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
We
provide a study of scaling laws and an improved estimator of scaling law parameters.
We do not anticipate any negative societal impacts of this work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] We plan to
release the code and a benchmark dataset to facilitate research in this direction.
Some
of the datasets used in our experiments
are proprietary and cannot be released, such
as JFT-300M. We also include experiments on publicly available datasets, such as
Big-Bench for reproducibility.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] We describe the data, training procedure and architectures in
details.
See Section 5.
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] Please see Tables 3 and 2.
(d) Did you include the total amount of compute and the type of resources used (e
g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] All experiments are executed in
Tensor Processing Units (TPUs).
See Section 5.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cite all thedatasets and architectures we use.
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We plan to release a code and a benchmark dataset.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content
[N/A
a
Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
[Table]
you describe any potential participant risks, with links to Institutional Review
[Table]
Board
approvals, if applicable?
[N
[Table]
Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
[Table]