The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon
Anonymous Author(s)
AffiliationAddress email
Abstract
The grokking phenomenon as reported by Power et al.
[13] refers to a regime wherea long period of overfitting is followed by a seemingly sudden transition to perfectgeneralization.
In this paper, we attempt to reveal the underpinnings of Grokkingvia a series of empirical studies.
Specifically, we uncover an optimization anomalyplaguing adaptive optimizers at extremely late stages of training, referred to asthe Slingshot Mechanism .
A prominent artifact of the Slingshot Mechanism canbe measured by the cyclic phase transitions between stable and unstable trainingregimes, and can be easily monitored by the cyclic behavior of the norm of thelast layers weights.
We empirically observe that without explicit regularization,
Grokking as reported in [13] almost exclusively happens at the onset of Slingshots ,and is absent without it.
While common and easily reproduced in more generalsettings, the Slingshot Mechanism does not follow from any known optimizationtheories that we are aware of, and can be easily overlooked without an in depthexamination.
Our work points to a surprising and useful inductive bias of adaptivegradient optimizers at late stages of training, calling for a revised theoreticalanalysis of their origin.
[Figure]
Introduction
Recently, the grokking phenomenon was proposed by [13], in the context of studying the optimizationand generalization aspects in small, algorithmically generated datasets.
Specifically, grokking refers to a sudden transition from chance level validation accuracy to perfect generalization, long past thepoint of perfect training accuracy, i.e., Terminal Phase of Training (TPT).
This curious behaviorcontradicts the common belief of early stopping in the overfitting regimes, and calls for furtherunderstandings of the generalization behavior of deep neural networks.
In the literature, it has been suggested that in some scenarios, marginal improvements in validationaccuracy appears in TPT, which seem to directly support grokking .
For example, it has been shownin [14] that gradient descent on logistic regression problems converges to the maximum marginsolution, a result that has been since extended to cover a wider setting [11, 17].
A key finding in [14]shows that when training on linearly separable data with gradient descent using logistic regression,the classifier’s margin slowly improves at a rate of O ( 1 log t ) , while the weight norm of the predictorlayer grows at a rate of O ( t ) , where t is the number of training steps.
While specified for gradientdescent, Wang et al.
[17] showed that similar results also hold for adaptive optimizers (such as Adamand RMSProp).
Taking these results into consideration, one could reasonably hypothesise that deepnonlinear networks could benefit from longer training time, even after achieving zero errors on thetraining set.
In this paper, we provide in depth empirical analyses to the mechanism behind grokking .
We find thatthe phenomenology of grokking differs from those predicted by [14] in several key aspects.
To beconcrete, we find that grokking occurs during the onset of another intriguing phenomenon directlyrelated to adaptive gradient methods (see Algorithm 1 for a generic description of adaptive gradientmethods).
In particular, leveraging the basic setup in [13], we make the following observations:
During the TPT, training exhibits a cyclic behaviour between stable and unstable regimes.
Aprominent artifact of this behaviour can be seen in the norm of a model’s last layer weights, whichexhibits a cyclical behavior with distinct, sharp phase transitions that alternate between rapid growthand plateaus over the course of training.2.
The norm grows rapidly sometime after the model has perfect classification accuracy on trainingdata.
A sharp phase transition then occurs when the model missclassifies training samples.
Thisphase change is accompanied by a sudden spike in training loss, and a plateau in the norm growth ofthe final classification layer.3.
The features (pre-classification layer) show rapid evolution as the weight norm transitions fromrapid growth to a growth plateau, and change relatively little at the norm growth phase.
Phase transitions between norm growth and norm plateau phases are typically accompanied by asudden bump in generalization as measured by classification accuracy on a validation set, as observedin a dramatic fashion in [13].
It is empirically observed that grokking as reported in [13] almost exclusively happens at the onset
of Slingshots ,
and is absent without it.
We denote the observations above as the Slingshot Effect , which is defined to be the full cycle startingfrom the norm growth phase, and ending in the norm plateau phase.
And empirically, a single trainingrun typically exhibits multiple Slingshot Effects.
Moreover, while grokking as described in [13]might be data dependent, we find that the Slingshot Mechanism is pervasive, and can be easilyreproduced in multiple scenarios, encompassing a variety of models (Transformers and MLPs) anddatasets (both vision, algorithmic and synthetic datasets).
Since we only observe Slingshot Effectswhen training classification models with adaptive optimizers, our work can be seen as empiricallycharacterizing an implicit bias of such optimizers.
Finally, while our observations and conclusionshold for most variants of adaptive gradient methods, we focus on Adam in the main paper, andrelegate all experiments with additional optimizers to the appendix.
Input: X 1 ∈ F , step size µ , sequence of functions { ϕ t , ψ t } Tt =1 , ϵ ∈ R + Output: Fitted α .
1 for t = 1 ..., T do g t = ∇ f t ( x t ) .
m t =
ϕ t ( g 1 , ..., g t ) and V t =
ψ t ( g 1 , ..., g t ) .
x t +1 = x t µm ϵ
1.1 Implications of Our Findings
The findings in this paper have both theoretical and practical implications that go beyond characterizing Grokking.
A prominent feature of the Slingshot Mechanism is the repeating phase shifts betweenstable and unstable training regimes, where the unstable phase is characterized by extremely largegradients, and spiking training loss.
Furthermore, we find that learning at late stages of training havea cyclic property, where non trivial feature adaptation only takes place at the onset of a phase shift.
From a theoretical perspective, this is contradictory to common assumptions made in the literature ofconvergence of adaptive optimizers, which typically require L smooth cost functions, and boundedstochastic gradients, either in the L 2 or L ∞ norm, decreasing step sizes and stable convergence[18, 1, 2].
From the apparent generalization benefits of Slingshot Effects, we cast doubt on the abilityof current working theories to explain the Slingshot Mechanism.
Practically, our work presents additional evidence for the growing body of work indicating theimportance of the TPT stage of training for optimal performance [6, 13, 12].
In an era where the sheer size of models are quickly becoming out of reach for most practitioners,our work suggest focusing on improved methods to prevent excessive norm growth either implicitlythrough Slingshot Effects or through other forms of explicit regularization or normalization.
Related Work
The Slingshot Mechanism we uncover here is reminiscent of the catapult mechanism described in
Lewkowycz et al.
Lewkowycz et al.
[9] show that loss of a model trained via gradient descent withan appropriately large learning rate shows a non-monotonic behavior —the loss initially increasesand starts decreasing once the model "catapults" to a region of lower curvature —early in training.
However, the catapult phenomenon differs from Slingshot Effects in several key aspects.
The catapultmechanism is observed with vanilla or stochastic gradient descent unlike the Slingshot Mechanismthat is seen with adaptive optimizers including Adam [7] and RMSProp [15].
Furthermore, thecatapult phenomenon relates to a large initial learning rate, and does not exhibit a repeating cyclicbehavior.
More intriguingly, Slingshot Effects only emerge late in training, typically long after themodel reaches perfect accuracy on the training data.
Cohen et al.
[3] describe a "progressive sharpening" phenomenon in which the maximum eigenvalueof the loss Hessian increases and reaches a value that is at equal to or slightly larger than 2 /η whereη is the learning rate.
This "progressive sharpening" phenomenon leads to model to enter a regime
Cohen et al.
[3] call Edge of Stability where-in the model shows non-monotonic training loss behaviorover short time spans.
Edge of Stability is similar to the Slingshot Mechanism in that it is shown tooccur later on in training.
However, Edge of Stability is shown for full-batch gradient descent whilewe observe Slingshot Mechanism with adaptive optimizers, primarily Adam [7] or AdamW [10].
As noted above, the Slingshot Mechanism emerges late in training, typically longer after the modelreaches perfect accuracy and has low loss on training data.
The benefits of continuing to traininga model in this regime has been theoretically studied in several works including [14, 11].
Soudryet al.
[14] show that training a linear model on separable data with gradient using the logisticloss function leads to a max-margin solution.
Furthermore Soudry et al.
[14] prove that the lossdecreases at a rate of O ( 1 t ) while the margin increases much slower O ( 1 log t ) , where t is the numberof training steps.
Soudry et al.
[14] also note that the weight norm of the predictor layer increasesat a logarithmic rate, i.e., O (log( t )) .
Lyu and Li [11] generalize the above results to homogeneousneural networks trained with exponential-type loss function and show that loss decreases at a rate of
O (1 /t (log( t )) 2 − 2 /L ) .
This is, where L is defined as the order of the homogenous neural network.
Although these results indeed prove the benefits of training models, their analyses are limitedto gradient descent.
Moreover, the analyses developed by Soudry et al [14] do not predict anyphenomenon that resembles the Slingshot Mechanism.
Wang et al.
[17] show that homogenous neuralnetworks trained with RMSProp [15] or Adam without momentum [17] do converge in direction tothe max-margin solution. However, none of these papers can explain the Slingshot Mechanism andspecifically the cyclical behavior of the norm of the last layer weights
[Figure]
The Slingshot Mechanism
3.1 Experimental Setup
We use the training setup studied by Power et al.
[13] in the main paper as a working example toillustrate the Slingshot Mechanism.
In this setup, we train decoder-only Transformers [16] on amodular division dataset [13] of the form a ÷ b = c , where a , b and c are discrete symbols and ÷refers to division modulo p for some prime number p , split into training and validation sets.
Thetask consists of calculating c given a and b .
The algorithmic operations and details of the datasetsconsidered in our experiments are described in
Appendix B. The Transformer consists of 2 layers,of width 128 and 4 attention heads with approximately 450K trainable parameters and is optimizedby Adam [7, 10].
For these experiments we set learning rate to 0.001, weight decay to 0, β 1 = 0 .
9 ,β 2 = 0 .
98 , ϵ = 10 − 8 , linear learning rate warmup for the first 10 steps and minibatch size to 512which are in line with the hyperparameters considered in [13].
Figure 2 shows the metrics of interest that we record on training and validation samples for modulardivision dataset.
Specifically, we measure 1) train loss ; 2) train accuracy ; 3) validation loss ; 4)validation accuracy ; 5) last layer norm : denoting the norm of the classification layer’s weights and 6)feature change : the relative change of features of the l-th layer ( h l ) after the t-th gradient update step∥ h lt +1 − h lt ∥ ∥ h lt ∥ .
We observe from Figure 2b that the model is able to reach high training accuracy aroundstep 300 while validation accuracy starts improving after 10 5 steps as seen in Figure 2d.
Power etal.
[13] originally showed this phenomenon and refer to it as grokking.
We observe that while thevalidation accuracy does not exhibit any change until much later in training, the validation loss shownin Figure 2c exhibits a double descent behavior with an initial decrease, then a growth before rapidlydecreasing to zero.
Seemingly, some of these observations can be explained by the arguments in [14] and their extensionsto adaptive optimizers [17].
Namely, at the point of reaching perfect classification of the training set,the cross-entropy (CE) loss by design pressures the classification layer to grow in norm at relativelyfast rate.
Simultaneously, the implicit bias of the optimizer coupled with the CE loss, pushes thedirection of the classification layer to coincide with that of the maximum margin classifier, albeit at amuch slower rate.
These insights motivate us to measure the classifier’s last layer norm during training.
We observe in
Figure 2a that once classification reaches perfect accuracy on the training set, the classification layernorm exhibits a distinct cyclic behavior, alternating between rapid growth and plateau, with a sharpphase transition between phases.
Simultaneously, the training loss retains a low value in periods ofrapid norm growth, and then wildly fluctuating in periods of norm plateau.
Figure 2e and Figure 2fshows the evolution of the relative change in features output by each layer in the Transformer.
Weobserve that the feature maps are not updated much during the norm growth phase.
However, at thephase transition, we observe that the feature maps receive a rapid update, which suggests that theinternal representation of the model is updating.
Is Slingshot a general phenomenon?
In an attempt to ascertain the generality of Slingshot
Effects as an optimization artifact, we run similar experiments with additional architectures, datasets,optimizers, and hyperparameters.
We use all algorithmic datasets as proposed in [13], as well asfrequently used vision benchmarks such as CIFAR-10 [8], and even synthetic Gaussian dataset.
Forarchitectures, we use Transformers, MLPs and deep linear models (see figure 1).
We find abundantevidence of Slingshot Effects in all of our experiments with Adam, AdamW and RMSProp.
Weare unable to observe Slingshot Effects with Adagrad [5] and also with stochastic gradient descent(SGD) or SGD with momentum, pointing to the generality of the mechanism across architectures anddatasets.
We refer the reader to Appendix A for the full, detailed description of the experiments.
Why does Slingshot happen?
We hypothesize that the norm growth continues until the curvatureof the loss surface becomes large, effectively “flinging" the weights to a different region in parameterspace as small gradient directions get amplified, reminiscent of the mechanics of a slingshot flinging aprojectile.
We attempt to quantify how far a model is flung by measuring the cosine distance betweena checkpoint during optimization and initial parameters.
Specifically, we divide the model parametersinto representation (pre-classifier) parameters and classifier (last layer) parameters and calculate howfar these parameters have moved from initialization.
We show that checkpoints collected after amodel experiences Slingshot have a larger representation cosine distance.
We defer the reader to theappendix for further details
[Figure]
[Figure]
By design, adaptive optimizers adapt the learning rate on a per parameter basis.
In toy, convexscenarios, the ϵ parameter provably determines whether the algorithm will converge stably.
To illustrate this, we take inspiration from [3], and consider a quadratic cost function
L ( A, B, C ) =
12 x ⊤ Ax + B ⊤ x + C,
A ∈ R d × d , x, B ∈ R d , C ∈ R , where we assume A is symmetric and positivedefinite.
Note that the global minimum of this cost is given by x ⋆ = −
A − 1 B .
The gradient ofthis cost with respect to x is given by g
=
Ax + B .
Consider optimizing the cost with adaptiveoptimization steps of the simple form x t +1 = x
t − µ g | g | + ϵ = x
t − µ Ax t + B |
Ax t + B | + ϵ where µ is a learningrate, and the division and absolute operations are taken element wise.
Starting from some x 0 , theerror e t
= x t − x ⋆ evolves according to:
[Equation]
Note that the condition ∥
A ∥ s < 2 ϵµ where ∥ · ∥ s denotes the spectral norm, implies that the mapping
M t is a contraction for all values of t , and hence convergence to the global optimum is guaranteed(This is in contrast to gradient descent, where the requirement is ∥
A ∥ s < 2 µ ).
Note that the choiceof ϵ crucially controls the requirement on the curvature of the cost, represented by the the spectrumof A in this case.
In other words, the smaller ϵ , the more restrictive the requirements on the topeigenvalue of A .
In [3], it was observed that full batch gradient descent increases the spectral normof the Hessian to its maximum allowed value.
We therefore hypothesize that for deep networks, asmall value for ϵ requires convergence to a low curvature local minimum, causing a Slingshot Effectwhen this does not occur.
Moreover, we may reasonably predict that increasing the value of ϵ wouldlift the restriction on the curvature, and with it evidence of Slingshot Effects.
Figure 3 shows evidence consistent with the hypothesis that Slingshot Effects occur in the vicinity ofhigh loss curvature, by measuring the local loss surface curvature along the optimization trajectory.
Let H t denote the local Hessian matrix of the loss, and u t the parameter update at time t given theoptimization algorithm of choice.
We use the local curvature along the trajectory of the optimizer,given by 1 ∥ u t ∥ 2 u ⊤ t H t u t , as a curvature measure.
Across the arithmetic datasets from [13], wheneverthe last layer weight norm plateaus, the curvature measure momentarily peaks and settles back down.
Varying ϵ
We next observe from Figure 2a that the training loss value also spikes up around thetime step when the weight norm transitions from growth to plateau.
A low training loss value suggeststhat the gradients (and their moments) used as inputs to the optimizer are small, which in turn cancause the ϵ hyperparameter value to play a role in calculating updates.
Our hypothesis here is that the
Slingshot Effect should eventually disappear with a sufficiently large ϵ .
To confirm this hypothesis,we run an experiment where we vary ϵ while retaining the rest of the setup described in the previoussection.
Figure 4 shows the results for various values of ϵ considered in this experiment.
We first observe thatthe number of Slingshot Effect cycles is higher for smaller values of ϵ .
Secondly, smaller values of ϵcause grokking to appear at an earlier time step when compared to larger values.
More intriguingly,models that show signs of grokking also experience Slingshot Effects while models that do notexperience Slingshot Effects do not show any signs of grokking.
Lastly, the model trained with thelargest ϵ = 10 − 5 shows no sign of generalization even after receiving 500K updates.
3.2 Effects on Generalization
In order to understand the relationship between Slingshot Effects and neural networks generalization,we experiment with various models and datasets.
We observe that models that exhibit Slingshot tendto generalize better, which suggests the benefit of training models for a long time with Adam [7] and
AdamW [10].
More surprisingly, we observe that Slingshots and grokking tend to come in tandem.
Transformers with algorithmic datasets We follow the setting in Power et al.
[13] and generateseveral datasets that represent algorithmic operations and consider several training and validationsplits.
This dataset creation approach is consistent with the methodology used to demonstrategrokking [13].
The Transformer is trained with AdamW [10] with a learning rate of 0.001, weightdecay set to 0, and with learning rate warmup for 500K steps.
We consider ϵ of AdamW as ahyperparameter in this experiment.
Figure 5 summarizes the results for this experiment where thex-axis indicates the algorithmic operation followed by the training data split size.
As can be seenin Figure 5, Slingshot Effects are seen with lower values of ϵ and disappear with higher values of ϵ
[Figure]
training loss vs epochs
[Figure]
[Figure] [Figure]
last layer norm 0.
l a s t l a y e r n o r m
[Figure]
(l)
[Figure]
which confirms the observations made in Section 3 with modular division dataset.
In addition, modelsthat exhibit Slingshot Effects and grokking (shown in green) tend to generalize better than modelsthat do not experience Slingshot Effects and grokking (shown in red).
ViT with CIFAR-10 For further validation of Slingshot Effects and generalization, we train a
Vision Transformer (ViT) [4] on CIFAR-10 [8].
The ViT consists of 12 layers, width 384 and12 attention heads trained on fixed subsets of CIFAR-10 dataset [8].
The ViT model describedabove is trained with 10K, 20K, 30K, 40K and 50K (full dataset) training samples.
We train themodels with the following learning rates: 0 . 0001 , 0 . 00031 and 0 .
001 and with a linear learning ratewarmup for the 1 epoch of optimization.
We consider multiple learning rates to study the impact ofthis hyperparameter on Slingshot taking inspiration from [13] where the authors report observing
[Figure]
grokking over a narrow range of learning rates .
Figure 6 shows a plot of the highest test accuracy fora set of hyperparameters (learning rate, number of training samples) as a function of the number oftraining samples from which we make the following observations.
The best test accuracy for a givenset of hyperparameters is typically achieved after Slingshot phase begins during optimization.
Thecheckpoints that achieve the highest test accuracy are labeled as "post-slingshot" and shown in greenin Figure 6.
While post-Slingshot checkpoints seem to enjoy higher test accuracy, there are certaincombinations of hyperparameters that lead to models that show better test accuracy prior to the startof the first Slingshot phase.
We label these points as "pre-slingshot" (shown in blue) in Figure 6.
Theabove observations appear to be consistent with our finding that training long periods of time maylead to better generalization seen with grokking datasets [13].
Non-Transformer Models We conduct experiments with MLPs on synthetic data where thesynthetic data is a low dimensional embedding projected to higher dimensions via random projections.
This design choice is critical with showing the existence of the Slingshot Effect with syntheticallygenerated data.
We find that using low dimensional data does not lead to any Slingshots.
With thisdataset, we show that generalization occurs late in training with Adam.
Specifically, we tune ϵ in
Adam and show that the optimizer is highly sensitive to this hyperparameter.
These observations areconsistent with the behavior reported above with Transformers and on algorithmic datasets as wellas standard vision benchmark such as CIFAR-10.
We refer the reader to Appendix ?? for completedescription and details of these experiments.
3.3 Drawbacks and Limitations
While the Slingshot Mechanism exposes an interesting implicit bias of Adam that often promotesgeneralization, due to its arresting of the norm growth and ensuing feature learning, it also leads tosome training instability and prolonged training time.
In the Appendix we show that it is possible toachieve similar levels of generalization with Adam on the modular division dataset [13] using thesame Transformer setup as above, while maintaining stable learning, in regimes that do not showa clear Slingshot Effect.
First we employ weight decay, which causes the training loss values toconverge to a higher value than the unregularized model.
In this regime the model does not becomeunstable, but instead regularization leads to comparable generalization, and much more quickly.
However, it is important to tune the regularization strength appropriately.
Similarly, we find that it is
[Figure]
possible to normalize the features and weights using the following scheme to explicitly control normgrowth: w = w ∥ w ∥ , f ( x ) =
f ( x ) ∥ f ( x ) ∥ , where w and f ( x ) are the weights and inputs to the classificationlayer respectively, the norm used above is the L 2 norm, and x is the input to the neural network.
Thisscheme also results in stable training and similar levels of generalization.
In all cases the effects relyon keeping the weight norms from growing uncontrollably, which may be the most important factorfor improving generalization.
These results suggest that while the Slingshot Mechanism may be aninteresting self-correcting scheme for controlling norm growth, there are likely more efficient waysto leverage adaptive optimizers to similar levels of generalization without requiring the instabilitythat is a hallmark of the Slingshot effect.
Finally, we lack a satisfactory theoretical explanation for the Slingshot Mechanism, and henceremoved all attempts at a more rigorous mathematical definition, which we feel would only serve as adistraction.
Conclusion
We have empirically shown that optimizing deep networks with cross entropy loss and adaptiveoptimizers produces the Slingshot Mechanism, a curious optimization anomaly unlike anythingdescribed in the literature.
We have provided ample evidence that Slingshot Effects can be observedwith different neural architectures and datasets.
Furthermore, we find that Grokking [13] almostalways occurs in the presence of Slingshot Effects and associated regions of instability in the Terminal
Phase of Training (TPT).
These results in their pure form absent explicit regularization, reveal anintriguing inductive bias of adaptive gradient optimizers that becomes salient in the TPT, characterizedby cyclic stepwise effects on the optimization trajectory.
These effects often promote generalizationin ways that differ from non-adaptive optimizers like SGD, and warrant further study to be ableto harness efficiently.
There are open question remaining to be answered, for instance 1) What’sthe causal factor of the plateau of weight norm growth? 2)
Are there better ways of promotinggeneralization without relying on this accidental training instability?
Answering these questions w illallow us to decouple optimization and regularization, and ultimately to control and improve themindependently.
Societal Impact
This is a fundamental work in Deep Learning, it will impact the society via its effects on relevantmodels and applications.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[Yes]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[Yes]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[Yes]