SInGE: Sparsity via Integrated Gradients Estimation of Neuron Relevance
Abstract
The leap in performance in state-of-the-art computer vision methods is attributed to the development of deep neural networks.
However it often comes at a computational price which may hinder their deployment.
To alleviate this limitation, structured pruning is a well known technique which consists in removing channels, neurons or ﬁlters, and is commonly applied in order to produce more compact models.
In most cases, the computations to remove are selected based on a relative importance criterion.
At the same time, the need for explainable predictive models has risen tremendously and motivated the development of robust attribution methods that highlight the relative importance of pixels of an input image or feature map.
In this work, we discuss the limitations of existing pruning heuristics, among which magnitude and gradient-based methods.
We draw inspiration from attribution methods to design a novel integrated gradient pruning criterion, in which the relevance of each neuron is deﬁned as the integral of the gradient variation on a path towards this neuron removal.
Furthermore, we propose an entwined DNN pruning and ﬁne-tuning ﬂowchart to better preserve DNN accuracy while removing parameters.
We show through extensive validation on several datasets, architectures as well as pruning scenarios that the proposed method, dubbed SInGE , signiﬁcantly outperforms existing state-of-the-art DNN pruning methods.
Introduction
Deep neural networks (DNNs) are ubiquitous in modern solutions for most computer vision problems such as image classiﬁcation [1], object detection [2] and semantic segmentation [3].
However, this performance was achieved at the price of high computational requirements and memory foot-print.
As such, over-parameterization [4] is a common trait of well performing DNNs that may hinder their deployment on mobile and embedded devices.
Furthermore, in the case of deployment on a cloud environment, latency and energy consumption are of paramount importance.
Consequently, compression and acceleration techniques aim at tackling the issue of DNN deployment.
Among these methods, pruning approaches consist in removing individual weights ( unstructured pruning) or entire computational blocks, such as neurons channels or ﬁlters ( structured pruning) [5, 6, 7, 8].
The sparsity induced by pruning reduces both the computational cost and the memory foot-print of neural networks.
To do so, there exists a wide variety of heuristics behind such pruning techniques.
A few examples are: pruning at initialization [9], grid search [10, 11], magnitude-based [12] or redundancy based [7, 13] approaches.
Among such heuristics, magnitude-based pruning remains the favoured one [14, 15, 16].
It consists in deﬁning a metric to assess the relevance of each neuron in the network, with the goal to remove the least important ones while still preserving the predictive function as much as possible.
An important limitation of these methods lies in the choice of this importance criterion: magnitude-based criteria [17] do not take into account the whole computations performed in the network (e.g. within the other layers) and gradient-based [18] criteria are intrinsically local within the neighborhood of a current value or set thereof: from this perspective, setting a value abruptly to zero might break this locality property.
To craft a better criterion, we borrow ideas from the ﬁeld of DNN attribution [19].
These methods aim at understanding the behavior of a neural network, i.e. , in the case of a computer vision model, by providing visual cues of the most relevant regions in a image for the prediction of a network.
Tools developed to explain individual predictions are also often called visual explanation techniques [20, 21, 22, 23, 24].
One example of such model is the Integrated Gradient method [24] that consists in deﬁning the contribution of each input by the inﬂuence of marginal local changes in the input on the ﬁnal prediction.
This provides a ﬁne-grained evaluation of the importance of each pixel of the image (alternatively, of an intermediate feature map) in the ﬁnal decision.
Our work is based on the idea that DNN pruning and attribution methods share an important notion, namely that they both rely on the deﬁnition of an importance metric to compare several variables of a multidimensional prediction system: for pruning, to remove the least important DNN parameters , and, for attribution, to highlight the most important pixels .
With this in mind, we propose to adapt the integrated gradient method for pruning purposes.
Speciﬁcally, for each parameter (or set thereof, if we consider structured sparsity), we deﬁne its importance as an integral of the product between the norm of this weight and its attribution along a path between this weight value and a baseline (zero) value.
By doing so, we avoid pathological cases which less sophisticated gradient-based methods are subjected to such as weights that can be reduced but not zeroed-out without harming the accuracy of the model.
Furthermore, we embed the proposed integrated gradient method within a re-training framework to maximize the accuracy of the pruned DNN.
We name our method SInGE , standing for
S parsity via In tegrated G radients E stimation of neuron relevance.
In short, the contributions of this paper are the following:
• We discuss the limitations of existing pruning heuristics, among which magnitude and gradient based methods.
We draw inspiration from attribution methods to design an integrated gradient criterion for estimating the importance of each DNN weight.
• We entwine the updates of the importance measurement within the ﬁne-tuning ﬂowchart to preserve the better DNN accuracy while pruning.
• The proposed approach, dubbed SInGE, achieves superior accuracy v
s. pruning ratio on every tested dataset and architecture, compared with recent state-of-the-art approaches.
Related Work 2.1 Pruning
Pruning methods are often classiﬁed as either structured [25, 26, 27, 10, 28] (ﬁlters, channels or neurons are removed) or unstructured [15, 29, 30, 31] (single scalar weight values are set to zero).
In practice, the former offers straightforward implementation for inference and immediate runtime beneﬁts but at the price of a lower number of parameters removed.
For instance, in GDP [32], weights are pruned with a learned gate that zeroes-out some channels for easier pruning post-training.
In CCP [33], sparsity is achieved by evaluating the inter-channel dependency and the joint impact of pruned and preserved channels on the ﬁnal loss function.
In HAP [34], authors replace less sensitive channels based on the trace of the Hessian of predictive function with respect to the weights.
Generally speaking, these methods rely on deﬁning a criterion to estimate and compare the importance of weights in the networks, and remove the least important such candidates.
A limitation of these methods is that the proposed criteria are usually only relevant within the neighborhood of the current value for a considered weight, which can be problematic since abruptly setting this weight value might violate this locality principle.
In this work, we address this limitation by borrowing ideas from the DNN attribution ﬁeld.
2.2 Attribution
Attribution methods, also referred to as visual explanation methods [20, 21, 22, 23, 24] measure the importance of each input feature on the prediction.
Their use was motivated by the need for explainable models [19] as well as constrained learning [35].
We can classify attribution as either occlusion-based or gradient-based.
The latter usually offers satisfactory results at a much lower
| | ∇μ w ln | | | | ∇μ 2 w ln | | | | w ln | | | | μ w ln | | | | ∇ w ln | | | | μ 2 w ln | || | ∇μ w ln | | | | ∇μ 2 w ln | | | | w ln | | | | μ w ln | | | | ∇ w ln | | | | μ 2 w ln | | (a) (b) (c) | | ∇μ w ln | | | | ∇μ 2 w ln | | | | w ln | | | | μ w ln | | | | ∇ w ln | | | | μ 2 w ln | | computational cost.
Considering that most DNNs for classiﬁcation are derivable, Grad-CAM [36] computes the gradients of the predictive function with respect to feature maps and weights these gradients by the features.
The resulting importance maps are then processed by a ReLU function to extract the features that should be increased in order to increase the value of the target class.
Another gradient-based attribution of interest is Integrated-Gradients [24].
In this work, Sundararajan et al.
propose to sum the gradients of the predictive function with respect to the feature maps over a uniform path in the feature space between feature at hand and a reference point.
The resulting attribution maps are usually sharper than maps obtained by Grad-CAM.
In the proposed method, we draw inspiration from these methods as we propose to integrate the (local) evolution of the pruning criteria throughout a path going from the current weight value down to a baseline (zero) value.
This way, we can smoothly bring the most irrelevant weights down to zero even using intrinsically local criteria such as gradients or gradients per weight norm products.
Methodology
Let F : D (cid:55)→ R n o be a feed forward neural network deﬁned over a domain D ⊂ R n i (e.g. the training dataset in most instances) and an output space R n o .
The operation performed by a layer f l , for l ∈ { 1 , . . .
, L } , is deﬁned by the corresponding weight tensor W l ∈ A n l − 1 × n l where A is simply
R in the case of fully-connected layers and R k × k in the case of a k × k convolutional layer.
For the sake of simplicity, we assume in what follows that A = R , i.e. we remove neurons as represented by their weight vectors.
3.1 Simple baseline pruning criteria
One major component of pruning methods lies in the deﬁnition of an importance measurement for each neuron.
The most straightforward such criterion is based on the magnitude of the weight vectors.
In such a case, the importance criterion C L p based on the L p norm (cid:107) · (cid:107) p , is deﬁned as:
[Equation]
where W nl is the n th column of W l , corresponding to the weight values associated with the n th neuron of layer f l .
The transformation C L p operates layer per layer and independently of the rest of network
F and the domain D .
Intuitively, C L p assumes that the least important neurons are the smallest in norm because such neurons have a lower impact on the predictions of F .
Such a simple criterion however face limitations: consider for instance the two ﬁrst neurons (a) and (b) depicted in Figure 1 by the purple stars in two-dimensional spaces as function of their magnitude and gradient norm respectively denoted (cid:107) W nl (cid:107) and (cid:107)∇ W nl (cid:107) , for simplicity.
However, we can clearly see how these local measurements provide a wrong evaluation of the cost of pruning these neurons.
In such a case, pruning according to C L p will remove neuron (a) regardless of the fact that the predictive function
F will be very sensitive to small modiﬁcation of this neuron, as indicated by the large value of its gradients.
This is however not the case with the gradient-based pruning criterion C ∇ p deﬁned as:
[Equation]
[Equation] ) where ∇ W nl F ( X ∈ D ) is the gradient of F with respect to W l , evaluated on X a sample from D .
Intuitively, the latter measurement puts more emphasis on neurons that can be modiﬁed without directly altering the predictive function F .
However, a neuron may have a low gradient norm and still strongly contribute to the predictive function, e.g. in the case where the weight is large as in the case of neuron (b) on Figure 1.
To handle this, the norm × gradient criterion C L p ×∇ p straightforwardly combines the best of both worlds:
[Equation]
3.2 Integrating gradients towards neuron removal
The importance criterion C L p ×∇ p in Equation (3) faces another kind of limitation.
due to the local nature of gradient information: if we consider neuron (b) on Figure 1, this neuron may initially ( i.e. within a neighborhood of the purple star) have a low gradient norm or even low magnitude per gradient norm product.
However, the gradient becomes larger as we bring this value down to 0.
This is due to the fact that ∇ W nl F ( X ∈ D ) only holds within a neighborhood of W nl current value, and abruptly setting this neuron weights to zero may very well violate this locality principle.
Thus, inspired from attribution methods, we propose a more global integrated gradient criterion.
Formally, for neuron n of a layer, l , we deﬁne I nl as the following integral:
[Equation]
Intuitively, we measure the cost of progressively decaying the weights of neuron n and integrating the gradient norm throughout.
In practice, we approximate I nl with the following Riemann integral:
[Equation]
where µ ∈ ]0; 1[ denotes an update rate parameter.
C IG p approximates ( I nl ) n ∈{ 1 ,...,n l } up to a multiplicative constant.
Practically, this criterion measures the cost (as expressed by its gradients) of progressively bringing W n l down to 0 by S successive multiplication with the update rate parameter µ : the higher µ , the more precise the integration at the expanse of increasing number of computations
S .
Also note that, similarly to Equation 2, we can get rid of the weight magnitude term in Equationto obtain criterion C SG p , based on the sum of gradient norms.
Explicitly, we get C SG p : ( W l , F, D ) (cid:55)→ (cid:16)(cid:80)
[Equation]
[Equation]
l } .
In the case depicted on Figure 1, we will prune neuron (c) as its gradient quickly diminishes as its magnitude becomes lower, despite high initial values for both magnitude and gradient.
Thus, the proposed integrated gradients criterion C IG p allows to take into account both the magnitude of a neuron’s weights and the sensitivity of the predictive function w.r.t.
small (local) variations of these weights.
Furthermore, it measures the cost of removing this neuron by smoothly decaying it, re-estimating the gradient value at each step, hence preserving the local nature of gradients.
3.3 Entwining neuron pruning and ﬁne-tuning
In order to preserve the accuracy of the network F , we alternate between removing the neurons and ﬁne-tuning the pruned network using classical stochastic optimization updates.
More speciﬁcally, given ρ a global pruning target for the whole network F , we deﬁne layer-wise pruning objectives
( ρ l ) l ∈{ 1 ,...,L } such that (cid:80) Ll =
ρ l × Ω( W l ) = ρ × Ω( F ) where Ω( W l ) and Ω( F ) denote the number of parameters in W l and F , respectively.
Similarly to [13], we tested several strategies for the per-layer pruning rates and kept their per-block strategy.
Then, we sequentially prune each layer, starting from the ﬁrst one, by ﬁrst evaluating the relevance of each neuron ( C IG p ( W l , F, D )) n ∈{ 1 ,...,n l } (with parameter µ ) in layer l .
We then rank the neurons by ascending numbers of importance and select the ﬁrst, least important one.
Notice at this point that if we remove neuron n we have to recompute the criterion C IG p for all other neurons: in fact, during the ﬁrst pass, the gradients ∇ µ s W nl F ( X ∈ D ) were computed with W nl (cid:54) =
and are bound to be altered with the removal of neuron n , thus affecting the order of the n l − 1 remaining neuron importance.
Last but not least, once layer l is pruned, we perform O ﬁnetuning steps (which corresponds to O gradient descent optimization steps) to retain the network accuracy.
This method, dubbed SInGE for S parsity via In tegrated G radients E stimation of neuron importance, is summarized in Algorithm 1.
Algorithm 1 SInGE Algorithm
Require: neural network F , hyper-parameters : O , µ and ( ρ l ) l ∈{ 1 ,...,L } and dataset D for l ∈ { 1 , . . .
, L } do while pruning_rate ( W l ) ≤ ρ l do (cid:46) wait until we reach the goal evaluate M ← C IG p ( W l , F, D ) (cid:46) magnitude estimation
ﬁnd n = arg min { M } (cid:46) ﬁnd the neuron to prune set W nl ←(cid:46)
the pruning is performed here for o ∈ { 1 , . . .
, O } do
Empirically, as we show through a variety of experiments that the proposed integrated gradients-based neuron pruning, along with efﬁcient entwined ﬁne-tuning allows to achieve superior accuracy vs. pruning rate trade-offs, as compared to existing methods.
Experiments
First, we introduce our experimental setup, including the datasets and architectures as well as the implementation details to ensure reproducibility of the results.
Second, we validate our approach on Cifar10 dataset by showing the interest of the proposed integrated gradient criterion, as well as the entwined pruning and ﬁne-tuning scheme.
We also compare our results with existing approaches on Cifar10.
Last but not least, we demonstrate the superior performance of our SInGE method on several architectures on ImageNet compared with state-of-the-art approaches for both structured and unstructured pruning.
4.1 Experimental setup
Datasets and Architectures: we evaluate our models on the two de facto standard datasets for architecture compression, i.e. Cifar10 [37] and ImageNet [38].
We use the standard evaluation metrics for pruning, i.e. the % of removed parameters as well as the % of removed Floating-point operations (FLOPs).
We apply our approach on ResNet 56 ([1] with 852K parameters and accuracies 93 .
46% ) on Cifar10 and ResNet 50 ([1] with 25M parameters and 76 .
17 accuracy on ImageNet), as well as MobileNet v2 [39] backbone on ImageNet with 71 .
80 accuracy and 3.4M parameters.
Implementation Details: our implementations are based on tensorﬂow and numpy python libraries.
We measured the different pruning criteria using random batches X of 64 training images for both
Cifar10 and ImageNet and ﬁne-tuned the pruned models with batches of size 128 and 64 for Cifar and ImageNet, respectively.
The number of optimization steps varies from 1 k to 5 k on Cifar and from 5 k to 50 k on ImageNet, while the original models were trained with batches of sizeand stochastic gradient descent of 78 k and 2 m steps on Cifar10 and ImageNet, respectively.
All experiments were performed on NVidia V100 GPU.
We evaluate our approach both for structured and unstructured pruning: for the former, we use µ = 0 .
9 and µ = 0 .
95 for ImageNet and Cifar10,
[Table]
respectively.
For unstructured pruning
[Table]
we use µ = 0 .
8 for ImageNet.
In all experiments we performed batch-normalization folding from [40] and measured the pruning ratio using the same metric as SOSP [41].
4.2 Empirical Validation
Pruning Criterion Validation: In Figure 2, we illustrate the evolution of (cid:13)(cid:13) ∇ µ s W nl F ( X ∈ D ) (cid:13)(cid:13) p (y axis) as the magnitude (cid:107) µ s W nl (cid:107) p (x axis) is brought to 0.
This observation conﬁrms the limitations of gradient-based criteria pinpointed in Section 3.2: as the neuron magnitude is progressively decayed, the gradient norm (e.g. yellow curves on both plots, as well as red on the left and blue one on the right plot) for these neurons rise signiﬁcantly, making these neurons bad choices for removal despite low initial gradient values.
This empirical observation suggests that intuition behind the proposed criterion C IG p is valid.
Table 1 draws a comparison between the different criteria introduced in
Section 3, applied to prune a ResNet 56 on Cifar10.
More speciﬁcally, given a percentage of removed operations (or equivalently, percentage of removed parameters), we compare the resulting accuracy
[Table] [Table]
without ﬁne-tuning.
We observe similar trends for the 3 pruning targets: First, using euclidean norm performs slightly better than L 1 : thus, we set p = 2 in what follows.
Second, using gradient instead of magnitude-based criterion allows to signiﬁcantly improve the accuracy given a pruning target.
Third, the magnitude × gradient criterion C L 2 ×∇ 2 allows to better preserve the accuracy by combining the best of both worlds: for instance, with 85% parameters removed, applying C L 2 ×∇ 2 increases the accuracy by 51 .
97 points compared with C ∇ 2 .
However, those simple criteria face limitations particularly in the high pruning rate regime ( 90% parameters removed), where the accuracy of the pruned network falls to chance level.
Conversely, the proposed integrated gradient-based criteria
C SG 2 and, a fortiori C IG 2 allows to preserve high accuracies in such a case.
Overall, C IG 2 is the best criterion, allowing to preserve near full accuracy with both 85% and 85% removed parameters, and 85 .
38% accuracy with 90% removed parameters, outperforming the second best method by 8 .points.
For this reason, we will use this criterion ( C IG 2 ) in the following experiments.
As the pruning rate increases, the cost of removing a neuron increases and any ill-advised selection of neuron to remove has a growing impact on the accuracy.
Consequently, the standard deviation increases as the pruning rate increases this is due to the network expressivity going down.
Fine-tuning Protocol Validation: Table
validates our entwined pruning and ﬁne-tuning approach with different pruning targets and number of ﬁne-tuning steps.
Speciﬁcally, for a given total number of ﬁne-tuning step, we either perform all these steps at once post-pruning or alternatively spread them evenly after pruning each layer in an entwined fashion, as described in Section 3.3.
First, we observe that simply increasing the number of ﬁne-tuning steps vastly improves the accuracy of the pruned model, particularly in the high % removed parameters regime.
Moreover, entwining pruning and ﬁne-tuning performs consistently better than ﬁne tuning after pruning.
This suggests that recovering the accuracy is an easier task when performed frequently over small modiﬁcations rather than once over a signiﬁcant modiﬁcation.
Comparison with state-of-the-art approaches on Cifar10: our approach relies on removing the least important neurons, as indicated by criterion C IG 2 .
We compare with similar recent approaches such as LP [42] and DPF [29] as well as other heuristics such as training neural networks in order to separate neurons for easier pruning post-training (HAP [34], GDP [32]) or similarity removal (RED [13] or LDI [31]).
We report the results in Table 3 for two accuracy set-ups: lossless pruning (accuracy identical to the baseline model) and lossy pruning ( ≈ 2 points of accuracy drop).
The proposed SInGE method signiﬁcantly outperforms other existing methods by achieving 1 . 3% higher pruning rate in the lossy setup and a considerable 8 . 1% improvement in lossless pruning rate.
As such, it bridges the gap with unstructured methods such as LP [42] and DPF [29].
This demonstrates the quality of the proposed method.
[Table] [Table]
4.3 Performance on ImageNet
Structured Pruning:
Table 4 summarizes results obtained by current state-of-the-art approaches in structured pruning.
For clarity we divided these results in the low (< 50% parameters removed, where the methods are often lossless) and high pruning regime (> 50% parameters removed with signiﬁcant accuracy loss).
In the low pruning regime, the proposed SInGE method manages to remove slightly more than 50% parameters ( 57 .
35% FLOPS) with nearly no accuracy loss, which signiﬁcantly improves over existing approaches.
Second, in the high pruning regime, other methods such as OTO [43] and HAP [34] recently improved the pruning rates by more than 10 points over other techniques such as GDP [32] and SOSP [41].
Nonetheless, SInGE is competitive with these methods and achieve a higher FLOP reduction while maintaining a higher accuracy.
We also evaluated the proposed method on the more compact (thus generally harder to prune)
MobileNet V2 architecture.
Results and comparison with existing approaches are shown in Table 5.
We consider three pruning goals of ≈ 30% , ≈ 40% and ≈ 50% parameters removed.
First, with near lossless pruning, we achieve results that are comparable to ManiDP-A [44] and Adapt-DCP [45] with a marginal improvement in accuracy.
Second, when targeting 40% parameters removed we improve by 0 .
89% the accuracy with 2.25% less parameters removed as compared to MDP [46].
Finally, in the higher pruning rates, we improve by 0 .
25% the accuracy with marginally more parameters pruned than Accs [47].
[Table] [Table]
Unstructured Pruning: While being harder to leverage
[Table]
unstructured pruning usually enables signiﬁcantly higher pruning rates.
Table 6 lists several state-of-the-art pruning methods evaluated on
ResNet 50.
We observe a common threshold in performance around 80% parameters and FLOPs removed among state-of-the-art techniques.
However, the proposed SInGE method manages to achieve very good accuracy of 73.77% while breaking the barrier of pruning performance at 90% parameters removed and almost 87% FLOPs removed.
These results in addition to the previous excellent results obtained on structured pruning conﬁrm the versatility of the proposed criterion and method for both structured and unstructured pruning.
Conclusion
In this paper, we pinpointed some limitations of some classical pruning criteria for assessing neuron importance prior to removing them.
In particular, we showed that magnitude-based approaches did not consider the sensitivity of the predictive function w.r.t.
this neuron weights, and that gradient-based approaches were limited to the locality of the measurements.
We drew inspiration on recent DNN attribution techniques to design a novel integrated gradients criterion, that consists in measuring the integral of the gradient variation on a path towards removing each individual neuron.
Furthermore, we proposed to entwine this criterion within the ﬁne-tuning steps.
We showed through extensive validation that the proposed method, dubbed SInGE , achieved superior accuracy v.s. pruning ratio as compared with existing approaches on a variety of benchmarks, including several datasets, architectures, and pruning scenarios.
Future work will involve introducing stochasticity in the model weights, similarly to [55], in order to smooth the decision function and ultimately the neuron relevance criterion.
Lasty, we will combine our approach with existing similarity-based pruning methods as well as with other DNN acceleration techniques, e.g. tensor decomposition or quantization techniques.
Acknowledgments
This work has been supported by the french National Association for Research and Technology (ANRT), the company Datakalab (CIFRE convention C20/1396) and by the French National Agency (ANR)
(FacIL, project ANR-17-CE33-0002).
This work was granted access to the HPC resources of
IDRIS under the allocation 2022-AD011013384 made by GENCI.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute
and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]