Neuron with Steady Response Leads to Better Generalization
Anonymous Author(s)
AffiliationAddress email
Abstract
Regularization can mitigate the generalization gap between training and inferenceby introducing inductive bias.
Existing works have already proposed various inductive biases from diverse perspectives.
However, none of them explores inductivebias from the perspective of class-dependent response distribution of individualneurons.
In this paper, we conduct a substantial analysis of the characteristics ofsuch distribution.
Based on the analysis results, we articulate the Neuron Steadiness
Hypothesis: the neuron with similar responses to instances of the same class leadsto better generalization.
Accordingly, we propose a new regularization methodcalled Neuron Steadiness Regularization (NSR) to reduce neuron intra-class response variance.
Based on Complexity Measure, we theoretically guarantee theeffectiveness of NSR for improving generalization.
We conduct extensive experiments on Multilayer Perceptron, Convolutional Neural Network, and Graph Neural
Network with popular benchmark datasets of diverse domains, which show thatour Neuron Steadiness Regularization consistently outperforms the vanilla versionof models with significant gain and low additional computational overhead.
Introduction
Deep Neural Network (DNN) achieves state-of-the-art results in a wide range of areas and has variousapplications across industries, including self driving cars [36], virtual assistants [37], intelligenthealthcare [31], personalized recommendation [33] etc.
DNN’s success usually relies on a plentyamount of training data.
However, DNN’s generalization is often hampered in many domains wheretraining data is insufficient because data annotation is labor-intensive and expensive.
Regularization is a popular method which helps to improve generalization via introducing inductivebias.
Regularization is one of the key elements of machine learning, particularly of deep learning[7].
Specifically, inductive bias represents assumptions about the model properties other thanthe consistency of outputs with targets.
There have been tremendous efforts in identifying suchdesired properties, which results in a series of widely used regularization methods.
For example, L
Regularization [35, 20] penalizes large norms of model weights, which puts constraints on “parameterscale".
L1 regularization improves “sparseness" by rewarding zero weight or neuron response.
Jacobian regularization [42, 12] minimizes the norm of the input-output Jacobian matrix to improve“smoothness" of the learned mapping function.
Orthogonal regularization [4, 1] enlarges “weightdiversity" to reduce the feature redundancy.
Batch Normalization [14] promotes “training dynamicsstability" by reducing the internal covariate shift.
Although the existing works have already proposed various inductive biases from diverse perspectives, including aforementioned “parameter scale", “sparseness", “smoothness", “weight diversity",“training dynamics stability", to the best of our knowledge, there is no work to explore inductivebias from the perspective regarding the characteristics of neuron response distribution on each class. From another point of view
[Figure] the existing works leverage the information related to weights (parameter scale regularization), weight correlations (orthogonal regularization), derivatives of mappingfunction (smoothness regularization), collective neurons responses (sparseness regularization, Batch Normalization) [Figure]
but none of them considers the intra-class response distribution of individual neurons
[Figure] [Figure]
two models on training set. In this paper, we study the characteristics of the intra-class response distribution of each individualneuron to identify the new regularization method.
In more detail, for each individual neuron, weanalyze the variance of its response to samples of the same class, which is called neuron intra-classresponse variance .
We find that such intra-class response variance has an obvious correlation withclassification correctness.
As shown in Figure 1, we find the correctly classified samples usuallyhave smaller intra-class response variance compared to the misclassified samples.
Besides, it can beobserved that the vanilla model with cross entropy as the optimization target usually could not controlintra-class response variance well, as shown in Figure 2, which leaves a potential improvement spacefor the regularization.
The details of experiments and observations are explained in Section 2.
Based on these observations, we articulate a Neuron Steadiness Hypothesis : neuron with similarresponses to instances of the same class, i.e., smaller neuron intra-class response variance can lead tobetter generalization.
Accordingly, we propose the regularization method called Neuron Steadiness
Regularization (NSR) to improve generalization by penalizing large neuron intra-class responsevariance.
Based on Complexity Measure theory in Section 3.3, we conduct a theoretical analysis on theeffectiveness of NSR for improving generalization.
In addition, our regularization method showssignificant improvements on various network architectures, including Graph Neural Network (GNN),
Convolution Neural Network (CNN), and Multilayer Perceptron (MLP).
To sum up, our contributions are as follows:
• We articulate a Neuron Steadiness Hypothesis and demonstrate its validity, which provides a new regularization perspective based on the neuron-level class-dependent responsedistribution.
• We propose a new regularization method called Neuron Steadiness Regularization to improves generalization ability.
The proposed method is computationally efficient and generalenough to be applied to various architectures and tasks.
The effectiveness of NSR isguaranteed via a theoretical analysis.• Extensive experiments are conducted on multiple types of datasets, like images, citationgraphs and product graphs, with various network architectures, including GNN, CNN andMLP.
Significant accuracy improvements evidently verify the effectiveness of the proposedNeuron Steadiness Regularization.
Observations
In this section, we verify the validity of our proposed Neuron Steadiness Hypothesis by experimentswith the following identified observations.
2.1 Correlation between neuron intra-class response variance and classification correctness
The neuron intra-class response variance is derived from neuron response distribution which could beobtained by recording neuron responses when we feed input samples to the model.
In this paper, forthe neuron with ReLU as its activation function, we do not take its zero-response into account forcalculation, because zero-response corresponds to the inactivated state where the neuron does notrespond at all.
In other words, for neurons with ReLU as the activation function, we only record itsnon-zero responses to represent its response distribution.
With the obtained response distribution of any given neuron, it is relatively straightforward to calculateour discussed statistics, i.e., intra-class response variance, which is the mean squared deviation ofthe neuron response from the mean of the intra-class response.
Then, for each neuron, we calculateintra-class response variance corresponding to the correctly classified samples and misclassifiedsamples, respectively.
Finally, we aggregate such two respective variances of all neurons in thepenultimate layer separately and present them in Figure 1.
Figure 1 shows that for different networks, the average intra-class response variance of correctlyclassified samples is smaller than that of misclassified ones on arbitrary class.
It indicates the strongcorrelation between classification correctness and neuron intra-class response variance.
2.2 Dynamics of neuron intra-class response variance during training procedure
We investigate the tendency of neuron intra-class response variance along with the training procedure.
For comparison, we perform the analysis on the vanilla model and the model with our proposedneuron steadiness regularization.
We calculate the intra-class response variance of each neuron onthe entire training set after each training epoch.
Then, the intra-class response variance of all neuronsare averaged and denoted as average variance in Figure 2 (c).
We also show the testing accuracy andthe training cross entropy loss in Figure 2 (a) and (b), respectively.
Other architectures demonstratesimilar tendencies and can be found in Appendix D.
From Figure 2, we could see that the cross entropy objective keeps being optimized during thetraining procedure which leads to increasing classification accuracy for both models.
However, forthe vanilla version, the average neuron intra-class response variance is growing larger because themodel training does not impose constraints on neuron intra-class response variance.
For the modeltrained with our proposed regularization, the neuron intra-class response variance is controlled anddecreases
after a few epochs.
More importantly, the learned model with well-controlled intra-classresponse variance has higher testing accuracy than the vanilla version although its correspondingcross entropy loss is even larger.
In addition, we could also see that regulating neuron intra-classresponse variance may also help the optimization procedure to achieve higher accuracy in earlierepochs.
To conclude all above observations, it is reasonable to design the regularization based on the neuronsteadiness hypothesis, i.e., reducing neuron intra-class response variance, for better generalization.
Method
In this section, we first describe the proposed Neuron Steadiness Regularization .
Then we furtherintroduce several techniques for computational efficiency.
3.1 Definition of Neuron Steadiness Regularization
Neuron Steadiness Regularization (NSR) for a specific neuron is defined as the summation ofintra-class response variances of different classes.
The NSR for the n -th neuron can be formulated as:
[Equation]
where X n,j is a random variable denoting the n -th neuron’s response for a sample belonging to thej -th class.
J is the number of classes.
α j = [Equation]
i z i is the prior probability of the j -th class where z jis the sample amount of j -th class.
Notice that α j is not a hyper-parameter, and it only presents theimportance of different classes.
With the NSR term defined for each individual neuron, the overallregularization is derived by applying NSR term on all neurons in the network as follows:
[Equation]
where L S represents the NSR term for the entire network, N is the number of neurons in the network.λ n is the hyper-parameter to control the regularization intensity.
In this paper, for practical simplicity,λ is set as the same value for all neurons.
Adding the overall regularization term to the main trainingtarget, i.e., the cross entropy loss, the final regularized loss function can be written as:
[Equation]
where L C represents cross entropy loss.
3.2 Practical Implementation
3.2.1 Mini-batch Training
In order to use mini-batch training, we adapt our NSR method to do forward and backward propagationbased on mini-batches of samples.
To be specific, we first transform Eq.
(1) as:
[Equation]
The first term has the same form as typical loss functions, i.e., an expectation of the function ofdata samples, which can be easily generalized to a mini-batch training setting via estimating theexpectation with a batch of samples.
Although the second term can be estimated in the same way, thesquare operation magnifies the estimation error, especially when the batch size is not large enough.
To alleviate the problem while not introduce much computing overhead, we propose a memory queuebased estimation method that allows us to leverage more history samples for estimation withoutadditional sampling and forward/backward computation.
For each class, we record the number ofsamples and the summation values of the neurons’ response within each batch.
To further reduce thestorage overhead, we only maintain these values of the latest M batches by two M -length queues.
More specifically, an element c m,j in the first memory queue is the count number of j -th classinstances in the m -th batch, represented as:
[Equation]
where Y m is the set of labels in the m -th batch, y i is the label of i -th sample, and δ is a characteristicfunction, i.e., δ ( condition )
= 1 if condition is satisfied, otherwise δ ( condition ) = 0 .
An elements ( n ) m,j in the second memory queue is the summation value of n -th neuron’s response for the samplesbelonging to the j -th class within the m -th batch, represented as:
[Equation]
) where X ( n ) m is the set of the n -th neuron’s response within the m -th batch and x ( n ) i is the n -th neuron’sresponse of the i -th sample.
When a new batch is fed, the estimation of expectation E [ X n,j ] can beupdated by the following steps:
[Equation]
) where ˆ E [ X n,j ] is the estimation of expectation E [ X n,j ] , C j = (cid:80) m c m,j , S ( n ) j =
[Equation] m s ( n ) m,j , andc ∗ ,j , s ( n ) ∗ ,j , as new elements appended to the queues, represent the count number and the summation forthe new batch.
Based on this dynamic update method, the additional memory overhead is negligible,and we give a space complexity analysis in Appendix E.
3.2.2 Layer Selection criterion for Applying NSR
We find that there is the correlation or redundancy among the steadiness constraints of differentlayers, meaning that applying NSR on different layers of network has an overlapping effect onneuron steadiness control.
In more details, from Figure 3, we can see that every layer’s variance ratiodecreases even only one specific layer is applied with NSR.
Such steadiness correlation or redundancyamong layers is not surprising because different layers are served as input/output of each other.
Due to the aforementioned redundancy, considering the trade-off between performance gain andcomputational overhead, we select only one layer to apply NSR.
As our NSR is used to reduce neuronintra-class response variance, naturally, the intuitive criterion is to apply NSR to the layer with thelargest aggregated neuron intra-class response variance.
Detailed experiment results in RQ4 of theexperiment section show such criterion works well.
In this paper, if not otherwise specified, we apply
NSR to only one particular layer determined by this layer selection criterion.
3.3 Theoretical Analysis
In this section, we theoretically analyze the difference in model generalization ability with or withoutour proposed regularization method NSR based on Complexity Measure [34].
Complexity Measureis one of the mainstream methods to measure the generalization ability of a deep learning model.
Alower complexity measure means a better generalization ability .
Formally, a Complexity Measureis a measure function M : {H , S} → R + where H is a class of models and S is a training set.
According to the definition, if a complexity measure of a given model tending to be 0, the modelshould have the best generalization ability.
Note that, several different complexity measures havebeen proposed.
We select Consistency of Representations [32] as the concrete Complexity Measure used in ourtheoretical analysis.
This measure is designed based on Davies-Bouldin Index [5], and is the
Winning Solution of the NeurIPS 2020 Competition on Predicting Generalization in Deep Learning.
Mathematically, for a given dataset and a model, we define the following statistics:
[Equation]
where i and j are two different classes, and n i is the number of samples belonging to the class i .
O ( τ ) i is the output representation of the τ -th sample belonging to class i for the final layer, and µ O i isthe cluster centroid of the representations of class i .
S i is a measure of scatter within representations
[Figure]
of class i , and M i,j
is a measure of separation between representations of classes i and j .
Then, thecomplexity measure based on the Davies-Bouldin Index is defined as:
[Equation]
Based on the definition of the complexity measure C , we have the following lemma:
Lemma 3.1.
For a multi-class classification problem, when (1) a deep learning model utilizing the
Cross Entropy loss with an NSR regularization term on any of its intermediate layer is optimized viagradient descent and (2) the capacity of the model is sufficiently large, the Consistency of Representations Complexity Measure on this model C will tend to be 0.
Under the same condition, when thedeep learning model is only optimized by the Cross Entropy loss without an NSR regularization term,there will be infinite local minima where the complexity measure C will be a positive number.
The proof is shown in Appendix A. Intuitively, this lemma shows that in an ideal condition (i.e.,the model is sufficiently large), if the definition of the Consistency of Representations Complexity
Measure is utilized, a deep learning model with NSR regularization term will be guaranteed toconverge to local minima with the best generalization ability.
In contrast, the model without NSRwill not have the same guarantee.
Experiment
In this section, we conduct experiments over a variety of datasets to examine the performance ofour neuron steadiness regularization on three extensively used neural network architectures.
Wedesign a series of experiments to answer following research questions and publish our code in link.
RQ1 : How does NSR perform on various datasets and different neural architectures?
RQ2 : Does
NSR outperform other classical regularization methods?
RQ3 : What is the effect of combining NSRwith other popular methods like Batch Normalization or Dropout?
RQ4 : What if we apply NSR tomultiple layers instead of the particular one layer?
4.1 Experiment Setup
Network Architectures and Datasets .
Three architectures utilized in experiments are Multilayer
Perceptron (MLP), Convolutional Neural Network (CNN) and Graph Neural Network (GNN).
Forvanilla models used as our baselines in different architectures, we adopt ResNet-18 [10], VGG-19[25] and ResNet-50 for CNN, GCN [16] and GraphSAGE [9] for GNN.
We run five MLP modelswhich are denoted as MLP-L, L indicates the number of layers including input layer.
We elaborateon the details of these vanilla models in Appendix C.2.
As for benchmark datasets used in ourexperiments, MLP and CNN are applied to image recognition task on MNIST [21], CIFAR-10 [18]and ImageNet [6] datasets, respectively.
GNN is applied to node classification on four real-worldgraph datasets: WikiCS [29], PubMed [47], Amazon-Photo and Amazon-Computers [40].
Noticethat ImageNet is a large benchmark dataset with 1000 classes.
Details are shown in Appendix C.4.
[Table] [Table]
Experiment Settings .
We follow the typical implementation settings to conduct ourexperiments.
For ResNet-18 and VGG19 on CIFAR-10, we follow the detailedsetting of [48, 8].
For ResNet-50 on ImageNet, we follow the official implementation provided by torchversion library 1 .
For
GraphSAGE and GCN, we follow the implementation setting of [27].
For MNISTdataset, we divide 60000 training images into the training set with 50000 samples and the validationset with the remaining 10000 samples to select hyper-parameter.
For each of four graph datasets, itis randomly split into training, validation, and testing sets with ratio 6:2:2.
Note that, SGD [39] isused to optimize MLP, and Adam [15] for other models except for ResNet-18 that is optimized by
Momentum [39] according to the implementation setting of [48].
We use the typical setting of batchsize as 100 for all experiments.
To ensure the model convergence, training epochs are set as 100 forboth MLP and GNN, and 200 for ResNet and 500 for VGG.
In all experiments except for RQ4, we apply NSR to only one particular layer with using the sameλ for each neuron.
For this only one hyper-parameter λ , like most of other regularization methods,we apply random search strategy to find its proper value ranging from 1e-2 to 10.
The error rateis the metric and each result is averaged over 5 runs with different random seeds.
The hardwareenvironments are detailed in Appendix B.
4.2 Experiment Results
RQ1: Performance of NSR : Here, we discuss the performance of NSR over different models andpresent the results on Tab. 1 - 3 “Gain” is the percentage of relative reduction in the error rate.
Tab.1 demonstrates that NSR can improve the performance of MLP with different layers: the relativeerror rate is reduced by 9.39% at least and 43.79% at most.
Besides, as the number of network layersincreases, the gain of NSR shows a roughly upward trend.
The four-layer MLP achieves the lowestclassification error rate among all vanilla version baselines, and the accuracy becomes worse as thenetworks grow deeper.
It indicates that deep MLP encounters a severe overfitting problem.
Thesuccess of our regularization in addressing such a problem reveals the importance of stabilizing theresponse of each individual neuron to instances from the same class.
For CNN models, Tab. 2 demonstrates that NSR can reduce the relative error rate for CIFAR-10 by11.97% on VGG-19 and 11.37% on Resnet-18, and reduce the relative error rate (top-5) for ImageNetby 10.74% on ResNet-50.
It is worth mentioning that VGG-19, ResNet-18 and ResNet-50 havealready adopted Batch Normalization, Dropout regularization and Weight Decay in vanilla models,the accuracy gain of our method reveals that NSR can have extra benefits to ulteriorly enhancegeneralization ability when combining with Batch Normalization, Dropout and Weight Decay.
Wewill show more evidence about this in RQ3.
The number of layers of two GNN models varies from 2 to 4.
It is a typical setting following empiricalexperiences as nodes will capture similar information from neighbors and result in over smoothnesswhen GNNs grow deeper.
Tab.
3 shows that GraphSAGE and GCN applied with NSR outperform thevanilla model with different layer depths on all datasets.
Specifically, GraphSAGE and GCN achievean average improvement of 8.6% and 5.8%, respectively, and 17.0% improvement at most.
RQ2: Comparison with Other Regularization : We first compare our NSR method with severalclassical regularization methods shown in Tab. 4. Notice that, both our NSR and the other regu-
[Table] [Table]
[Table] [Table]
[Table]
larization methods for comparison have only one hyper-parameter to tune, and we utilize the samehyper-parameter searching strategy to select the best hyper-parameter values for all of them.
Asshown in Tab. 4, our NSR performs best among these regularization methods on all three modelswith different architectures, i.e., MLP-4 (MLP), ResNet-18 (CNN), GraphSAGE (GNN).
The resultsindicate that our NSR has remarkable improvement and is general for various architectures.
RQ3: Combination with Other Regularization : We further investigate the effect of combiningour NSR with other popular methods like Batch Normalization and Dropout.
We conduct theexperiments on MLP-4 and the result are organized in Tab. 5.
From Tab. 5 we can find that both
Batch Normalization and Dropout can reduce error rate compared with vanilla baseline, and addingour NSR on the top of them can further promote the performance significantly.
It indicates that NSRcan provide complementary regularization benefits with Batch Normalization and Dropout.
RQ4: Layer selection for applying NSR : We have introduced how to select one particular layer toapply NSR in section 3.2.2.
To evaluate such layer selection criterion, we compare the performancestrained by applying NSR to the multiple layers and to the only one selected layer, respectively.
Taking
MLP-4 on MNIST and GraphSAGE-2 on Amazon-Computers as two examples, we apply NSR todifferent layer(s), and their corresponding error rates are listed in Tab.
6 ∼ Tab. 7, where model lmeans NSR is applied to the l th layer of the model.
Notice that, except the first layer of MLP-4, the input layer, for the second, third, and last layer of MLP-4, their neuron intra-class response varianceare 409, 510, and 1660, respectively.
For GraphSAGE-2, its neuron intra-class response variance are4.15 and 2.68, for the first and last layer, respectively.
The variance of MLP-4 and GraphSAGE-2 arequite different because of the data characteristic difference between MNIST and Amazon-Computers.
The results in Tab. 6 - 7 show that, no matter which layer(s) is applied with NSR, it could alwaysimprove the accuracy compared with the vanilla baseline for both MLP-4 and GraphSAGE-2.
Inaddition, according to our layer selection criterion, applying NSR to the layer with the biggestvariance, i.e., the last layer for MLP-4 and the first layer for GraphSAGE-2, could achieve the mostsignificant gain compared with applying NSR to other individual layers.
Also, it could achievesimilar accuracy compared with the best one obtained by applying NSR to multiple layers.
This isthe empirical evidence to demonstrate the rationality of our layer selection criterion.
Related Work
Regularization improves generalization ability by introducing the inductive bias based on priorknowledge.
According to the type of prior knowledge, existing works can be roughly categorizedinto Domain Specific Regularization and Model Generic Regularization.
Domain Specific Regularization shows success in various domains including Face Recognition[45, 23, 2, 24], Knowledge Graph Completion [49, 19, 30], Graph Neural Network [3, 13, 38, 46, 26].
In Face Recognition, [2] points out the existence of external factors, such as different situations ofenvironmental illumination, head poses, facial expressions, which brings the challenge that faceimages from the same person may have even larger differences than face images from differentpersons.
To address specific challenges in face recognition, [23] imposes the careful-designedregularization method which enforces the representations of face instances from the same person tobe similar.
In Graph Neural Network, the popular model usually suffers from the over-smoothingproblem due to the six degrees of separation [17].
[3] statistically analyzes the high correlationbetween smoothness and the mean average distance (MAD) among node representation.
Then aregularization called MADGap is proposed which punishes over smoothness by minimizing MAD.
Model Generic Regularization can be categorized into network-wise Regularization, layer-wise
Regularization, and neuron-wise Regularization, according to the granularity of the studied properties.
Network-wise Regularization encodes the desired property of the entire network, like thesparseness of the network, the smoothness of the mapping function between input and output.
{+L
 Regularization [35, 20]+} encourages small sum squared magnitude of model weights.
[12] introducesan efficient framework to minimize the norm of input-out Jacobian matrix for noise robustness.
Layer regularization [44, 35] becomes popular due to the success achieved by Batch Normalization[14].
Neuron-wise Regularization is the method with the most fine-grained granularity.
Dropout[11, 43] is the typical example belonging to this category.
It randomly removes neurons with a certainprobability to avoid strong co-adaption between neurons.
Actiavtion Regularization [28] adds the {+L
 Regularization on+} masked activations of neurons and Temporal Activation Regularization restrictsthe difference between RNN outputs at adjacent timesteps.
To minimize the variance of samplevariance and obtain a few distinct modes, Variance Constancy Loss [22] is applied on all neuronsbefore the activation as regularization.
Our proposed Neuron Steadiness Regularization belongs tothe neuron-wise regularization that leverages information of individual neuron response distribution.
Conclusion and Future Work
We explore the inductive bias from the new perspective of class-dependent response distributionof individual neurons.
Based on experimental observations, we articulate the Neuron Steadiness
Hypothesis and propose the Neuron Steadiness Regularization that penalizes large intra-class response variance of neurons.
Based on Complexity Measure, we provide theoretical analysis on itseffectiveness for improving generalization.
Additionally, we conduct extensive evaluations on diversedatasets with various network architectures to demonstrate its effectiveness.
Especially, we show itseffectiveness on the classification task with large model, i.e., ResNet-50, and large dataset with manyclasses, i.e., ImageNet with 1000 classes.
We also systematically consider the border impact, No riskis found.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] We provide
code in the anonymous project.
All datasets we utilized are open-source.
We share the
corresponding link in Appendix C
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See in Appendix C.
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] Yes, each results is calculated from five random seeds: [1,
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See in Appendix B
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] See in Appendix C.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]