To update or not to update? Neurons at equilibrium in deep models
Abstract
Recent advances in deep learning optimization showed that, with some a-posterioriinformation on fully-trained models, it is possible to match the same performanceby simply training a subset of their parameters.
Such a discovery has a broad impactfrom theory to applications, driving the research towards methods to identify theminimum subset of parameters to train without look-ahead information exploitation.
However, the methods proposed do not match the state-of-the-art performance, andrely on unstructured sparsely connected models.
In this work we shift our focus from the single parameters to the behavior of thewhole neuron, exploiting the concept of neuronal equilibrium (NEq).
When aneuron is in a configuration at equilibrium (meaning that it has learned a specificinput-output relationship), we can halt its update; on the contrary, when a neuron isat non-equilibrium, we let its state evolve towards an equilibrium state, updatingits parameters.
The proposed approach has been tested on different state-of-theart learning strategies and tasks, validating NEq and observing that the neuronalequilibrium depends on the specific learning setup.
Introduction
In recent years, deep learning has become a staple solution to different tasks, such as computervision, bio-informatics, speech recognition, and many more.
Unfortunately, modern neural networkarchitectures are becoming more and more “expensive”: performing simple inferences requires a lotof computational resources, and training the models even more.
Such cost poses several challengesfor the research community: the training of a network model is associated to large carbon footprintsand the commercialization of of AI research (especially for edge devices) is hindered by the resourcerequirements of the models Strubell et al.
For several years now, many works in literature have shown that is possible to shrink both the size andresource requirements, mainly via quantization Yang et al.
[2020], Jin et al.
[2022] and pruning Wanget al.
[2020], Tang et al.
[2020], Tanaka et al.
In particular, pruning techniques are ableto drastically reduce the number of operations needed to perform inference, without affecting theoverall performance of the model.
Pruning targets the reduction of parameters in the model, whichhas advantages after training; unfortunately, it is unable to reduce the cost of the training cycle, andon the contrary it requires, in general, iterations of few-shot pruning, followed by fine-tuning Leeet al.
[2019], Tartaglione et al.
A recent work shows the existence of sub-graphs in the whole deep learning model which, whentrained in isolation, match the performance of the whole model Frankle and Carbin [2019].
Thisopens the road towards a whole field of research, where many approaches are proposed to find these parameters a-priori: this shows a potential environmental impact, considering that a smaller model istrained, leading to a lower energy consumption Strubell et al.
Indeed, when training a neuralnetwork, the back-propagation procedure and the weights update lead to the larger part of FLOPs(compared to forward propagation) Plaut et al.
[1986], Baydin et al.
Researchers started toexperiment pruning early in the training or even before the training starts in the hope of training arestricted number of parameters.
Unfortunately, training a sparse network with standard optimizersleads to subpar results Evci et al.
[2019] or the final result does not differ much from magnitudepruning at the end of the training Frankle et al.
The causes for such behaviors are still amatter of debate among the community 2019, 2021.
In this work we shift the focus from the single parameter to the whole neuron, and we propose
NEq, an approach to evaluate whether a given neuron is at equilibrium with respect to the learningdynamics.
If the neuron is at such state, its parameters have already reached a target configurationand do not require further update.
Unlike many other recent approaches, NEq disables entire neurons(hence, in a structured way), does not require prior knowledge on the specific task (like the samemodel already at convergency) and automatically self-adapts to the specific learning policy deployed.
Unlike pruning techniques, NEq does not remove the neurons’ contribution to the output; instead, itonly prevents unnecessary updates to their weights: as a result, we reduce the number of operationsperformed by the back-propagation algorithm and the optimizer.
The rest of the paper is organized as follows.
In Sec.
2 we discuss the related literature; Sec.
3presents the concept of neuronal equilibrium and how to evaluate it; Sec. 4 presents the experimentalvalidation inclusive of an ablation study and Sec.
5 draws the conclusions.
Related works
It is broadly acknowledged that the typically-deployed deep learning models on the state-of-the-artscenarios are over-parametrized Mhaskar and Poggio [2016], Brutzkus et al.
This ignites twolines of research: reducing the size of these models (with pruning algorithms), or saving computationalresources at training time.
While the first has been broadly explored, the latter suffered a stalemateuntil a recent work suggested its feasibility Frankle and Carbin [2019].
Pruning strategies.
Attempts to reduce the number of parameters from learned models date backto 1989, when Mozer and Smolensky proposed skeletonization , a technique to identify less relevantneurons in a trained model and to remove them 1989.
This was accomplished thanks to the evaluationof an error function penalty from a pre-trained model.
In the same years, LeCun et al.
also proposeda work where the information from the second order derivative of the error function is leveraged torank the parameters of the trained model on a saliency-like basis LeCun et al.
In the lastdecade, thanks to the broad availability of computational resources, pruning approaches gained-backpopularity, with approaches including the exploitation of dropout Molchanov et al.
[2017], Gomezet al.
[2019], sensitivity-based approaches Lee et al.
[2018], Tartaglione et al.
[2018], relaxation of ℓ 0regularization Louizos et al.
[2017], Sanh et al.
[2020] and optimization of auxiliary parameters Xiaoet al.
Despite leading to very compact deep models, the demanded training complexityis frequently very high, as many of the most well-known approaches rely on iterative fine-tuning(or few-shot pruning) approaches.
Hence, the training complexity for these techniques is largerthan training a vanilla model (which in many cases is used as initialization).
Much computationalcomplexity could be saved if structured pruning is applied before training the model itself (namely,entire neurons/channels are pruned).
Some works on pruning have suggested the possibility ofre-allocating previously pruned parameters Tresp et al.
[1996] or filters
He et al.
[2018] in order tolearn new functions and minimize the loss, towards higher generalization capability Blalock et al.[2020].
The lottery ticket hypothesis.
In a recent paper, Frankle and Carbin provided empirical evidence that,at initialization, there exists a sub-graph of the original deep learning model such that, when trained inisolation, it can match the performance of the complete model Frankle and Carbin [2019].
Indeed, theauthors claim that the parameters in the sub-graph have “won at the lottery of initialization”.
From avery practical perspective, this means that all the parameters not in the selected sub-graph are, de-facto,pruned from the model, not requiring any gradient computation.
This empirical evidence potentially
[Figure]
a t e d a nd no r m a li
ze
d c on ca t e n a t e d a nd no r m a li
ze
d opens the road to the development of algorithms for pruning at the initialization. Unfortunately, atsuch point, two obstacles are yet to be tackled.
From one side, Frankle and Carbin [2019] uses aniterative procedure to identify the sub-graph: it simply proves its existence, but does not provide amethod to find it directly at initialization.
From another side, it focuses on un-structured sub-graphs,meaning that parameters are treated in isolation: is it possible to find structured sub-graphs (or inother words, removing entire nodes and not just arcs)?
Beyond the lottery ticket hypothesis.
Many works of the last two years have received inspirationfrom the lottery ticket hypothesis, in the quest of determining the early lottery tickets.
This is,however, a tough task to solve: Frankle et al.
[2020] shows that there is a region, at the very earlystages of learning, where the lottery tickets identified with iterative pruning are “not stable” (meaningthat tickets extracted at different moments of this early stages are essentially different).
This suggeststhat, in the very first epochs, the neural network evolves in very different states, making the problem ofa-priori identifying winning tickets hard.
This is also endorsed by other works, including Morcos et al.[2019], Malach et al.
On the other hand, other approaches reduce the overall complexity of theiterative training by drawing early-bird tickets You et al.
[2019] (meaning that they learn the lotterytickets when the model have not yet reached full convergence), even reducing the training data Zhanget al.
[2021] or moving the first steps towards structurally-sparse winning tickets, yet still at iterativefashion, applying similar concept as Frankle and Carbin to entire neurons and channels Chen et al.[2022].
With NEq we learn the important lessons related to the lottery ticket hypothesis, targeting thereduction of computational complexity at training time, without exploiting knowledge on pre-trainingmodels, or rewinding.
Hence, we do not target the achievement of sparse architectures, but we aim atdetermining when a whole neuron requires to be updated or when the computation of the gradient forits parameters is not necessary.
As such, the comparison with the other presented approaches will bein general unfair, as they require a much greater computational complexity for training because ofun-structured sparsity (which introduces an overhead in the representation of the tensors) and theiterative strategies.
Furthermore, we will observe the possibility, along the training process, that someneurons, already kept in a “frozen” state, might unfreeze, requiring additional update steps.
Althoughresources re-allocation has been exploited before Tresp et al.
He et al.
[2018], our unfreezingis different as it involves learning of a specific target function by the neuron, and not learning newones through their re-allocation.
In the next section the notion of neuronal equilibrium will bepresented, as well as the strategy to determine which neurons will require gradient computation.
Neurons at equilibrium
In this section we will treat the problem of determining when a given neuron, along the learningdynamics, finds itself at equilibrium.
Towards this end, we define y ti,ξ as the output of the i -th neuron when the input ξ is fed to the whole model trained after t epochs.
Given a set of inputs ξ ∈ Ξ val(where Ξ val is the validation set), it is possible to compare each n -th element y ti,n,ξ with y t − 1 i,n,ξ , forthe same model’s input: what changes are the parameters of the model.
Fig.
1 provides an overviewover the nomenclature used: in the rest of the section we will see how to determine when a neuron isat equilibrium.
3.1 Neuronal equilibrium
In this section we are interested in evaluating when the relationship between the input of the modeland the output of the i -th neuron is modified.
When this happens, the neuron is at non-equilibrium ,meaning that its learned function, in the whole picture (or in other words, taking into account theevolution of the neurons in the previous layers as well), is still “evolving”.
We are interested inidentifying the scenarios where the neuron is at equilibrium at the net of the interactions with theother neurons.
In order to assess it, let us define the cosine similarity between all the outputs of thei -th neuron at time t and at time t − 1 for the whole validation set Ξ val as
[Equation]
Here we can determine that, when ϕ i = 1 , the i -th neuron produces the same (eventually scaled)output between the evaluation at time t and at time t − 1 for the same input ξ of the model.
We saythe i -th neuron reaches equilibrium when we have
[Equation]
where k ∈ [ − 1; +1] is some constant value.
We can have the following scenarios:
• k = 1 .
In this case, the two outputs are perfectly correlated, meaning that the relationship
bounding the input of the whole model ξ and the output of the specific i -th neuron is
maintained.
• k ∈ (0; 1) .
The outputs correlate, but we are in presence of an oscillatory behavior (in thesense that the cosine similarity varies by a constant value between consecutive evaluation).This effect can be caused by stochastic effects like high learning rate/regularization, smallbatch-size or combination of them.
• k ∈ [ − 1; 0] .
Also in this case we are in the presence of an oscillatory behavior, but theoutputs are anti-correlated or de-correlated.
Here follows the evaluation framework to determine the arrival to an equilibrium state for the i -thneuron.
3.2 Neuron dynamics evaluation
In order to assess the convergency to equilibrium for (2), it is important to evaluate the variation ofthe similarities ϕ ti over time.
Towards this end, let us introduce the variation of similarities
[Equation]
According to the analysis in Sec. 3.1, in this case we say we reach equilibrium when ∆ ϕ ti → 0 .
Hence, it is useful to keep track of the recent evolution over the similarity scores in the model:towards this end, we can introduce the velocity of the similarity variations:
[Equation]
where µ eq is the momentum coefficient.
We can rewrite (4) making the similarity scores explicit,obtaining
[Equation]
where ( · ) m indicates power of m .
If we assume ϕ t ∈ [0; 1] ∀ t (which is the case of ReLU-activatedneurons), in order to prevent (5) from exploding, we need to set µ eq ∈ [0; 0 . 5] .
We can extend thisdimensioning for the momentum coefficient to any layer assuming that the neurons in the trainedmodel will reach at most equilibrium with de-correlated outputs.
3.3 Selection of trainable neurons at non-equilibrium
In order to evaluate when a given neuron has reached equilibrium, exploiting (2), we can say that thei -th neuron is at equilibrium when it is able to satisfy
[Equation]
It is important to notice that, once (6) is satisfied for a certain t , in case something changes in thelearning dynamics (for example, the learning rate is re-scaled), it is possible that there exists somet ′ >
t such that the constraint is not satisfied anymore.
When this happens, it means that the neuron isdriven towards new states , and is not anymore at equilibrium.
Hence, it requires to be updated again.
3.4 Overall training scheme
[Figure]
The overall training scheme is summarized in Fig.
The model is trained for one epoch, after whichneurons at equilibrium are identified.
We split this in two phases: in the first (neuron dynamicsevaluation), the velocity of the similarities is evaluated according to (4), while in the second (trainableneurons at non-equilibrium) the hidden neurons at non-equilibrium, which will be trained for thenext epoch, are identified according to (6).
For the first epoch, all the neurons are considered atnon-equilibrium by default.
The evaluation of neurons at equilibrium is agnostic with respect to thegeneral training strategy, which can include arbitrary re-scaling for the learning rate/hyper-parametersor the most common optimizers.
In the next section, we will test this procedure on very differentarchitectures, tasks and learning strategies.
Experiments
In this section we report the experiments supporting the approach as presented in Sec. 3.4.
First, wewill perform an ablation study, analyzing single contributions for the introduced hyper-parametersand providing an overview over neuronal equilibrium along the training process; then, we will testthe proposed technique on state-of-the-art network architectures, datasets and learning policies.
Allexperiments were performed using 8 NVIDIA A40 GPUs and the source code uses PyTorch 1.10. [Figure] a
[Figure]
n i n g R a t e
[Figure] [Figure]
[Figure] [Figure]
B p r o p .
F L O P s p e r i t e r .
× 100 .0 .1 .1 .2 .
U pd a t e d N e u r o n s ×
A cc u r a c y
[Figure]
i n
[Figure]
F
[Figure]
p e r i t e r
[Figure]
[Figure]
[Figure]
U pd a t e d N e u r o n s ×
[Figure]
A cc u r a c y
[Figure]
4.1 Ablation study
We performed our ablation study training a ResNet-32 He et al.
[2015] model on CIFAR10 Krizhevsky et al..
Unless differently specified, following the hyper-parameters setup of Zagoruykoand Komodakis [2016], the model is trained using SGD as optimizer, with a starting learning rateη = 0 . 1 and momentum µ opt = 0 .
9 and a weight decay of 5 × 10 − 4 for 250 epochs.
The learningrate is decayed by a factor of 10 after 100 and 150 epochs, using the formulation as in (4), with∥ Ξ val ∥ 0 = 50 , µ eq = 0 .
5 , and ε = 0 .
4.1.1 SGD vs Adam
To show that our technique automatically self-adapts to the training policy, we compare the evolutionof the FLOPs required for a back-propagation step and the amount of updated neurons of two differenttraining of the ResNet-32: one using the SGD optimizer with µ opt = 0 . 9 , and the other using the Adam optimizer.
For Adam we leave the hyper-parameters to their default values ( η = 0 .
001 ,β 1 = 0 .
9 , and β 2 = 0 .
and use the same weight decay as for SGD ( 5 × 10 − 4 ).
Fig.
3 showsthe trends for the two training procedures.
We can see that in the first phase of the train, where η ishigh, the amount of the trained neurons (and the FLOPs required for the backward pass) is higher.
This is related to the general lack of equilibrium in the neurons of the network: at high learning rates,the configuration of the neurons’ parameters is subjected to a high stochastic noise.
As the trainprogresses, and the network progresses towards its final configuration, less and less neurons need tobe updated.
Noticeably, Adam drives the neurons towards equilibrium faster as expected; however,in simple tasks like the considered one, converges to lower accuracy scores ( 92 .
96% for SGD and92.01% for Adam).
Furthermore, at the first learning rate decay (epoch 100 ), for SGD the numberof updated neurons first decreases and then increases, such phenomenon is not present in the Adamcase.
This is explained by the different working principle of the two optimizers: SGD explores thesolution space looking for large minima, searching for configurations which disallow equilibrium inhigh learning rate regimes.
4.1.2 Distribution of ϕ & choice of µ eq
Looking at different values for µ eq in Fig.
4, we observe for all the values a convergence to similaraccuracy.
Despite without warranty from the theory, we tested a very large value for the momentum
[Figure]
r i t
[Figure]
[Figure]
cc u r a c y µ eq = 0 .µ
eq = 0 .
[Figure]
µ eq = 0 .
[Figure]
coefficient ( 0 . 9 ): the convergence of v t ∆ ϕ
i shows that the neurons are in general in a very correlatedcase of equilibrium, with very high values for k in (2), which is also empirically observed in Fig.
5a.
However, including a very large value for µ eq maintains memory of very old variations, producingsub-optimal reduction in terms of FLOPs reduction.
We find that a good compromise, supported bythe findings as in Sec. 3.2, is to set µ eq to 0 .
Fig.
5 reports the distribution for the velocities for ϕ ,∆ ϕ
[Table]
and v ∆ ϕ , observing that the average converges to a specific k for each of the three learning ratesused.
4.1.3 Impact of the validation set size and ε
Tab. 1b provides empirical evaluation of the impact on the performance and on the FLOPs varyingthe validation set size.
We indeed observe not a significant impact on the performance of the model
[Table]
[Table]
varying it.
Interestingly
[Table]
the approach produces extremely good results even for extremely lowcardinality for the validation set (down to even a single image): this can be explained by the presenceof convolutional layers (the only fully-connected layer is the output layer, excluded by default) whicheven with little images produce high-dimensionality output in every neuron (Fig. 1) and by thehomogeneity of the considered dataset.
Investigating the impact of ε , instead, we find for very highvalues of ε a drop in performance, identifying a good compromise for classification tasks to 0 .
4.2 Main experiments
In this section we show the results of the proposed method.
For our experiments we focused ondifferent state-of-the-art architectures trained on standard classification and semantic segmentationdatasets.
All the learning policies used are borrowed by other works and are un-optimized in order totest the adaptability of NEq.
ResNet-32 trained on CIFAR-10 .
The training spans 250 epochs, using SGD as optimizer withmomentum µ opt = 0 .
9 , weight decay 5 × 10 − 4 and initial learning rate η = 0 .
1 , reduced by afactor of 10 after 100 and 150 epochs.
To evaluate the neuronal equilibrium we used a ∥ Ξ val ∥ 0 of 50images, µ eq = 0 . 5 , and ε = 0 . 001 .
We used a batch size of 100 images during training.
ResNet-18 trained on ImageNet-1K Krizhevsky et al.
This model was trained with SGDas optimizer for 90 epochs, with η = 0 . 1 , reduced by a factor of 10 every 30 epochs, µ opt = 0 .
9 andweight decay 10 − 4 using a batch size of 128 .
We used a ∥ Ξ val ∥ 0 of 1.2k images, µ eq = 0 .
5 , andε = 0 .
Swin Transformer Liu et al.
[2021] (Swin-B) trained on ImageNet-1K .
To test our technique onmore modern models and training policies, we used the Swin-B architecture.
Here we trained the model starting from a pre-trained checkpoint trained on ImageNet-21K, following the official GitHubrepository 2 released under the MIT License.
We used a ∥ Ξ val ∥ 0 of 1.2k images, µ eq = 0 .
5 , andε = 0 .
DeepLabv3 Chen et al.
[2017] trained on COCO Lin et al.
Other than classificationtasks of varying complexity, we tested our procedure on a semantic segmentation problem.
For thisexperiment we used DeepLabv3 with a ResNet-50 backbone and the COCO dataset.
To train thenetwork, we followed the state-of-the-art procedure defined in PyTorch 3 .
We evaluated the neuronalequilibrium using a ∥ Ξ val ∥ 0 of 320 images, µ eq = 0 .
5 , and ε = 0 .
4.3 Discussion
The results (average over five different runs) are reported in Tab. 2.
For each experiment we compareour technique with a “stochastic” approach.
Namely, we randomly disallow, at every epoch andwith probability p , the update of a given neuron.
We test on three different probabilities: 0 . 2 , 0 .
5and a probability which is as close as possible with the average over the one achieved by NEqindicated with “*”.
To evaluate the effectiveness of the proposed procedure we focus on the averagecomputational complexity of the back-propagation for a single update iteration (expressed in FLOPs),and the network generalization capabilities at the end of the training.
In all the considered scenarios,it is possible to observe a reduction of FLOPs with very marginal or no performance drop for NEq.
When compared to the stochastic approach, with fixed probabilities, the amount of saved computationis similar in all the considered scenarios, but the loss in performance varies, depending on the specificarchitecture/dataset.
On the contrary, NEq remains consistent in performance, self-adapting to thespecific setup and saving the largest FLOPs for the given performance.
Furthermore, testing thestochastic approach with the same FLOPs saving (hence, even letting that information leaking infavor of the stochastic approach), the performance loss is lower.
Limitations.
The current approach analyzes the behavior of an entire neuron.
However, empiricalexperiments show that there could be further improvements considering ensembles of neurons.
Forexample, Fig.
5 shows the average value for the similarities close to a constant but many neuronsare still away from the convergence value, meaning that these neurons, at isolation, are still notat equilibrium: is the scenario changing when considering the dynamics of groups of neurons?
Furthermore, in order to validate the adaptability of NEq to the most popular training schemes, nooptimization on the hyper-parameters for the training procedure has been performed (as it is out ofscope for our evaluation).
However, higher savings in computational complexity are possible tuningthe training strategy as well.
In such a direction, prospectively, it will be of interest the design ofmore efficient learning strategies which keep into account the concept of neuronal equilibrium.
Conclusions
The work by Frankle and Carbin [2019] showed the existence of sub-graphs in deep models which,when trained in isolation, are able to match the original performance of the whole model.
Findingthese sub-graphs is, however, a complex task, as in the first stages of the learning the model itselfis at non-equilibrium.
Identifying these with a dynamic strategy, without requiring a posterior overthe whole training process, is a crucial task to be solved, towards computational resources saving.
Differently from the vast majority of the literature which focuses on the identification of sub-graphswithout any concrete computational saving (as they rely on iterative or roll-back algorithms), we haveintroduced the knowledge of neuronal equilibrium, looking for entire structures of the deep model atequilibrium, not requiring further optimization and gradient computation, which self-adapts to veryspecific experimental setups on very different learning scenarios.
This work opens the doors towardsa deeper understanding on the deep neural network’s learning dynamics and to the development ofnew training strategies exploiting this knowledge.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes] The theory is presented in Sec.
3 and a deeper understanding of the single term’s contribution is presented in Sec. 4.1; also experiments onpopular architectures trained with very different learning policies are performed andpresented in Sec.
4.2 and Sec. 4.3. b)
Did you describe the limitations of your work?
[Yes] Limitations are discussed in thespecific paragraph of Sec. 4.3.
c) Did you discuss any potential negative societal impacts of your work?
[N/ A
] The workshows that there are learning regimes where not all the neurons need to be updated inorder to achieve a target performance.
Considering that this approach is not entailedfor specific applications, no negative societal impact can be found.
d) Have you read the ethics review guidelines and ensured that your paper conforms tothem
Yes
If you are including theoretical results
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] All the
assumptions for the µ eq estimations are stated in Sec. 3.2.
(b) Did you include complete proofs of all theoretical results?
[Yes] Proofs and additional results can be found in the supplementary material.
If you ran experiments...
(a) Did you include the code, data, and
instructions
needed
to
reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes Code is included in the supplementary material, as well as instructions on how to use it.
(b) Did you specify all the training details (e g., data splits, hyperparameters,
how they
were chosen)?
[Yes] Please see Sec.
The selection of the hyper-parameters adopted is taken from the referenced sources
(c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[Yes] In the whole ablation study in Sec. 4.1 all the standard deviations/error bars are reported.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Beginning of Sec. 4. [Table]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a)
If your work uses existing assets, did you cite the creators?
[Yes] All the existing
[Table] datasets, models and code used is presented in Sec. 4 and credit is given to the respectivecreators. [Table]
(b) Did you mention the license of the assets?
[Yes] Where open source code is used in [Table]
Sec. 4 the license is mentioned.
[Table] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
[Table]
The code including the proposed technique is included in the supplementary material.
[Table] (d) Did you discuss whether and how consent was obtained from people whose data you’re [Table] using/curating?
[N/A]
[Table] (e) Did you discuss whether the data you are using/curating contains personally identifiable [Table]
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table] (a) Did you include the full text of instructions given to participants and screenshots, if [Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A] [Table]