Trading off Image Quality for Robustness is not Necessary with Deterministic Autoencoders
Anonymous Author(s)
AffiliationAddress email
Abstract
The susceptibility of Variational Autoencoders (VAEs) to adversarial attacks indicates the necessity to evaluate the robustness of the learned representations alongwith the generation performance.
The vulnerability of VAEs has been attributedto the limitations associated with their variational formulation.
Deterministic autoencoders could overcome the practical limitations associated with VAEs andoffer a promising alternative for image generation applications.
In this work, wepropose an adversarially robust deterministic autoencoder with superior performance in terms of both generation and robustness of the learned representations.
We introduce a regularization scheme to incorporate adversarially perturbed datapoints to the training pipeline without increasing the computational complexity orcompromising the generation fidelity by leveraging a loss based on the two-point
Kolmogorov–Smirnov test between representations.
We conduct extensive experimental studies on popular image benchmark datasets to quantify the robustnessof the proposed approach based on the adversarial attacks targeted at VAEs.
Ourempirical findings show that the proposed method achieves significant performancein both robustness and fidelity when compared to the robust VAE models.
Introduction
Variational autoencoders (VAEs) offer a powerful probabilistic framework to learn deep generativemodels.
They are widely employed in various domains such as computer vision, natural languageprocessing and representation learning [23, 16, 24, 25, 10].
A VAE is composed of two coupledbut independently parameterized models, an encoder or a recognition model that computes a latentrepresentation of the input space with stochastic latent variables and a decoder or a generativemodel to map the latent representations back to the original input space [18].
One of the majoradvantages of VAEs is that they provide semantically meaningful latent representations of highdimensional complex input distributions which can be further utilized for various downstreamtasks [17, 14, 27, 11].
However, VAEs are often limited due to theoretical and practical limitationssuch as over-regularization, prior-posterior mismatch resulting in trade-offs between generation andreconstruction fidelity [30, 9, 12].
Recently proposed deterministic versions of VAEs offer promisingalternatives to overcome these limitations [12, 26].
The semantically meaningful representations learned by VAEs can still be corrupted by so-calledadversarial attacks [29, 19, 20], where even small but specifically crafted changes to the input canlead to very different reconstructions.
This observation reveals a lack of generalization within suchmodels and is therefore a serious concern with respect to many practical applications.
While it is harder to attack VAEs when compared to classifier networks [13] it is essential to analyze therobustness of VAEs along with their generative performance, to validate whether the learned latentrepresentations are meaningful.
Hence, there has been increasing research interest in the deep learningfield towards training robust models, both for classifier models and for autoencoders, i.e. for robustrepresentation learning.
Borrowed from the training of robust classification models [22, 32], theconcept of adversarial training has proven to be able to smoothen the VAE encoder and improve therobustness of the learned representations [6].
Other attempts toward learning robust representationspaces introduce either complex network architectures or expensive regularization mechanisms toimprove the robustness of VAEs [31, 2].
Further, it has been pointed out in previous works thatthe robustness of VAEs can be improved by generating disentangled latent representations or byencouraging the smoothness or consistency of the encoding-decoding process [31, 5].
However,regularizing the VAE objective to enhance robustness often leads to poor generation ability comparedto its non-robust counterpart.
Hence, we seek to focus on improving the robustness of autoencoderswhile still maintaining the generation performance.
In this work, we introduce a simple and easy to train deterministic autoencoder which exhibitssuperior performance in terms of both generation and adversarial robustness.
We argue that the deterministic approach enhances the robustness of VAEs when the latent codes are properly regularized.
Consequently, we extend the training objective of the multi-modal deterministic autoencoders [26] toincorporate adversarially perturbed input data points in the latent space.
The proposed adversariallearning scheme is comparatively cheaper and easier to implement than existing robust VAE models.
We conduct extensive experimental analysis to evaluate the robustness of the trained model on popularbenchmarks such as MNIST, FASHIONMNIST, SVHN and CELEBA images.
Our empirical evaluations show that the proposed model consistently exhibits high adversarial robustness and significantlybetter generation performance compared to state-of-the-art robust VAE baseline models.
We alsoshow that by improving the robustness of the learned representations, a classifier trained on thelearned latent space of the model also exhibits better robustness.
Related Work
The ability to defend against adversarial attacks is closely related to the sensitivity of the learnedlatent representations to slight changes in the input data points.
In this section we first discuss seminalworks proposed to generate rich and meaningful latent codes.
This is followed by the review onadversaries for VAEs and strategies proposed to defend against such adversarial attacks.
Variational Autoencoders (VAEs)
In standard VAEs, the encoder and the decoder are jointlytrained to maximize the evidence lower bound (ELBO) of the log likelihood of the model.
Theresulting training objective is a combination of a reconstruction loss and the Kullback–Leibler (KL)divergence between the learned latent representations and a chosen prior - usually a uni-modal
Gaussian normal distribution.
Higgins et.al [16] improved the learned representation in VAEs byencouraging disentanglement in the latent space and introduces β -VAE for disentangled factorlearning.
β -VAE modifies VAEs by introducing a hyperparameter to balance the latent regularizationterm with the reconstruction performance.
Chen et.al [7] further decompose the ELBO term in VAEsto introduce total correlation (TC) regularization and propose β -TCVAE as a promising alternative toβ -VAEs.
Willets et.al.
[31] show that addition of TC term to the VAE objective also improves therobustness of VAEs.
Several methods were proposed to include complex and flexible priors to thetraining pipeline of VAEs to enhance the semantics of the learned latent representations [8, 34, 30].
Hierarchical VAEs [25, 3, 33, 21] extend the standard VAE framework by introducing a hierarchy oflatent variables and offer superior modelling capabilities.
Gosh et.al.
[12] question the variationalformulation of VAEs and introduce a simple and effective deterministic model without any priorassumptions followed by a post-hoc density estimation to approximate the learned posterior.
Amultimodal version to this approach, that utilizes the capacity of expressive multi-modal latentdistributions to yield high quality generation, was proposed in [26].
Adversarial attacks on VAEs Adversarial attacks targeted towards autoencoders were first discussedin [13].
Common attacks on VAEs follow procedures similar to attacks against classifiers, i.e.
they aimto maximize the network’s loss.
Usually, slight perturbations are added to the input images to make thereconstructions similar to a specific target image (targeted/supervised attack) or a completely differentimage (untargeted/unsupervised) [29, 19] such as to maximize the reconstruction loss.
In [31] Willetset.al.
show that applying the TC regularization introduced in TC-VAEs to hierarchical VAEs yieldsrobust representations.
Although the resulting model improves adversarial robustness, the trainingcomplexity is high when compared to a VAE.
Cemgil at.al [6] relate the robustness of VAEs to thesmoothness of the encoding process.
Similarly to Madry et.al.
[22], they introduce a regularizationscheme based on a selection mechanism in the latent space to generate additional data points tominimize the entropy regularized Wasserstein distance between latent representations.
Following thesame direction, Cemgil et.al.
[5] argue that the lack of consistency between the encoding-decodingprocess cause the susceptibility to adversarial attacks in VAES.
In contrast to the previous works,
Camuto et.al.
[4] provide a theoretical insight to the robustness of VAEs and introduce a novelcriterion for robustness in VAEs.
Barret et.al.
[2] propose to constrain the Lipschitz constants forboth encoder and decoder to ensure certifiable adversarial robustness of VAEs.
Although thesemethods comparatively improve the adversarial robustness of VAEs, they are often accompanied bycomplex network architectures and expensive training procedures.
In contrast, our approach adoptsan inexpensive adversarial training scheme to the latent space of deterministic autoencoders by anelegant extension to the regularization proposed in [26] to ensure both robustness and fidelity.
Towards robust deterministic autoencoders
Deterministic Autoencoders [12, 26] offer a promising alternative to VAEs for learning meaningfulrepresentations of complex input spaces with high fidelity.
Motivated by this fact, we aim to furtherexplore the robustness of the learned representations of the resulting model.
We are particularly interested in the multimodal prior setting in [26], since a flexible and expressive Gaussian mixture model(GMM) prior assumption facilitates encoding similar data points closer together while distancingdissimilar points far in the latent space - a behaviour that has also been found to be beneficial inlearning robust classifier models [11].
Consequently, we propose to adopt the regularization techniqueproposed in [26] to regularize the learned latent representations of our model towards a predefined
GMM prior.
For a model to be inherently robust, slight perturbations in the input space should notresult in substantial variations in the encoding space and the corresponding reconstructions.
Thiscould be also attributed to the smoothness of the learned encoder.
Hence, it is essential to investigatethe smoothness of the learned representations and the reconstructions of the model.
In the followingsubsections, we resume the regularization loss from [26] for the sake of completeness and illustratehow it can be extended to adversarial training samples.
3.1 Regularization of the learned representations – Encoder smoothness
Following the idea from [26], we aim to regularize the D -dimensional latent representations{ z 1 , ..., z N } of input datapoints { x 1 , ..., x N } learned by the encoder g of the model towards ak -modal GMM prior.
For k ≤ K , let µ k and Σ k be the mean and covariance matrix of the k -th modein the model.
Further, let w k > 0 be the weight of the k -th mode, (cid:80)
Kk
=
w k = 1 .
Inspired by thestatistical Kolmogorov-Smirnov (KS) test, the authors propose to regularize the empirical cumulativedistribution function (CDF) of the latent samples towards the CDF Φ( z ) of the prior distribution,Φ( z ) =
(cid:90)
[Equation]
where
z = ( z 1 , . . . , z D ) , t = ( t 1 , . .
. , t D ) .
While the original KS-test quantifies a metric between joint CDFs, the minimization of this KSdistance can be approximated by marginalization, i.e. by regularizing the empirical marginal CDFs
[Equation]
of the latent representations { z 1 , ..., z N } in dimensions d = 1 , . . .
, D towards the marginal CDFs
F GMM ,d ( z
[Equation]
=
[Equation] [Equation]
) of the GMM prior.
Here, [ · ] d indicates the d -th dimension of a vector, i.e. [ µ k ] d is the d -th entryof the mean vector µ k = ( µ k, 1 , . . . , µ k,D ) of mixture component k , i.e. a scalar.
Similarly, [ · ] d,dindicates the entry in row and column d of the matrix Σ k = diag( σ k, 1 , . . .
, σ k,D ) .
The respectiveloss is computed as
[Equation]
An additional loss compensates for the ambiguities on the target distribution introduced by themarginalization, i.e.
[Equation]
with Σ GMM =
(cid:80)
Kk
=
w k Σ
k
+ (cid:80)
Kk =1 w k ( µ k − ¯ µ ) ( µ k − ¯ µ ) T , ¯ µ = 1 k (cid:80)
Kk =1 µ k and ¯Σ being theempirical covariance matrix of the latent representations.
The total loss of the model in [26] is a combination of both regularization terms to enforce thelatent representations of the encoded data to match a predefined multi-modal prior distribution and areconstruction loss.
Motivated by [11], for equidistantly chosen modes µ i as in Figure 1, we expectsuch a model to already provide not only an improved generation fidelity as observed in [26] but alsoa more robust behavior.
We provide empirical evidence for this conjecture in an ablation study in
Section 4.
The GMM regularization implicitly clusters latent points such that similar points are closeto one another while dissimilar points are distant.
To further improve the robustness of our model we employ adversarial training - a widely populardefense strategy utilized to learn robust deep networks [22, 32].
That is, during training we utilize adversarial samples z adv n that are explicitly optimized to fall in an ϵ − ball around the original latentrepresentations z n for n ≤ N and to decode to damaged or semantically altered images.
In [1] itwas observed that such adversarial samples tend to explore the underrepresented regions in the latentspace.
In the following, we extend the losses in (4) and (5) to overcome this undesired behaviour.
This approach allows for a cheaper yet very effective adversarial training, while preserving thereconstruction and generation ability of the original model from [26].
3.2 Adversarial training data augmentation – Improving Robustness
To generate adversarial inputs, we adapt the fast gradient sign method [32] to the latent spaceof the model.
For a given ϵ > 0 and datapoint x n , the objective of the attack is to find thecorresponding adversary x
adv n that would introduce maximum distance in the encoding space.
Thatis, x adv n = x n + δ x n , where δ x n is the solution to the optimization problem
[Equation]
6) where g is the encoder of the model.
To prevent adversarial samples from exploring unexploredregions of the latent space, we assume that the joint distribution of latent encodings ( z n , z adv n ) (herez n = g ( x n ) and z adv n = g ( x n + δ x n )) of data points and their adversarial samples ( x n , x adv n ) tofollow the same multi-modal GMM prior (Figure 1).
One possible straight forward extension ofthe approach in [26] would be to consider adversarial examples as a specific data augmentation andregularize z 1 ,...,N and z adv1 ,...,N to the same GMM prior independently and ignoring cross-covariancebetween z 1 ,...,N and z adv1 ,...,N (here the off-diagonal elements in the covariance matrix of the GMMprior, that is the last matrix mentioned in equation (8), are zero) .
The corresponding losses in eqs.
(4)and (5) take the form
[Equation]
and
L aug CV ,k ( z 1 ,...,N , z
[Equation]
where ¯ Σ and ¯ Σ adv are the empirical covariance matrices of the latent representations z 1 ,...,N andtheir adversaries z adv1 ,...,N respectively, and ¯ Σ cross is the empirical cross-covariance between benignand adversarial samples.
While such a regularization preserves the overall distribution even underadversarial attacks, it can not control the distance of a specific adversarial sample to its benignz n .
In the worst case scenario, an adversarial example z adv n can be mapped to a different Gaussianmixture component than z n and therefore cause maximum damage in the reconstruction as shown in
Figure 1(left).
To evenly spread out the learned representations, we inject Gaussian noise to the latent vectors duringtraining.
Let x ϵ,n be the output of the decoder at z n + ϵ n , where ϵ n ∼ N (0 , I D ) .
The reconstructionloss equals the mean squared error between inputs x n and their noisy reconstructions x ϵ,n .
3.3 Coupling of z and z adv – A Two-Point KS-distance loss
To ensure that the adversarial examples remain in close proximity to the original mapping in thelearned latent space, we establish a strong coupling between the two distributions, z 1 ,...,N and z adv1 ,...,N .
Hence we propose to match the the empirical CDFs of z 1 ,...,N and z adv1 ,...,N and introduce a novelregularization based on the two-point KS-test [28].
By analogy to the one-point KS-test that testswhether a sample is drawn from a given, continuous distribution, the two-sample KS test determineswhether two samples with empirical CDF are drawn from the same distribution.
To this end, thetwo-sample KS test evaluates the supremum of the distance between the two CDFs.
Here, we proposeto minimize this distance computed from the marginalized two-point KS-test in order to align thedistributions of benign points and their adversaries.
The resulting loss is consistent with the previous regularization and allows to efficiently establish the desired coupling between the representations.
The first regularization loss of the adversarially extended model with pairwise coupling takes thefollowing form
[Equation]
where ¯ F d , ¯ F adv d are the empirical CDFs of z and z adv respectively.
The correlations between the latent representations and their adversarial samples need to be consideredseparately.
The degree or strength of the coupling is controlled by a coupling parameter | α |≤ 1 ,where α = 1 indicates the border condition where z
=
z adv .
The covariance loss of the extendedmodel becomes,
[Equation]
The total training objective of the model takes the following form,
[Equation]
The weights λ REC , λ KS and λ CV can be calculated by taking the statistics of samples from the GMMprior [26], see Appendix.
Experiments
We conduct an extensive experimental analysis to evaluate the robustness of the proposed model.
Weconsider the state of the art latent space attacks targeted at VAEs [13, 19, 2] and evaluate the robustnessbased on the quantitative metrics as described in Section 4.
Qualitative analysis are provided inthe Appendix.
Since the latent spaces of VAEs are often further utilized for various downstreamapplications, we also consider the impact of such adversarial attacks on a classifier trained in thelatent space of the model.
To evaluate the fidelity of the learned representations, we report the Fréchetinception distance (FID) [15] of the generated images.
Our model is compared with the followingbaseline models, Variational Autoencoder (VAE) [18], β -VAE [16], β -TCVAE [7], LipschitzVAE [2],
Smooth Encoders (SE) [6] and Autoencoding Variational Autoencoder (AVAE) [5].
The experimentalstudy is conducted on important image benchmark datasets such as MNIST, FASHIONMNIST, SVHNand CELEBA.
For simplicity we consider a fully connected network architecture for experiments on
MNIST and FASHIONMNIST images and a convolutional architecture for experiments on SVHNand CELEBA images.
Further details of the experiments are given in the Appendix 1 .
Adversarial attacks.
Adversarial attacks targeted at VAEs attempt to add small noise perturbationsto the input data points that fool the model to reconstruct the input image to a target adversarialimage or a completely different image.
Following recent literature [2, 31], we consider two types ofadversarial attacks in our experiments.
Latent space attacks or supervised attacks are considered the most effective mode of attacks on
VAEs.
Here, the attacker tries to add a noise perturbation δ to a data point x , such that the latentrepresentation z x + δ of the perturbed input x + δ is close to the latent representation z t of a chosentarget adversarial image, x t .
The attack involves solving the following optimization problem,
[Equation]
FASHIONMNIST
SVHN
CELEBA  5 0.
A tt a c k l o ss
VAEBetaTCVAELipschitzVAESEAVAEOurs
A tt a c k l o ss
VAEBetaTCVAELipschitzVAESEOurs  5 0.
A tt a c k l o ss
VAEBetaTCVAELipschitzVAESEOurs (a) Latent space attack.
A tt a c k l o s s
VAEBetaTCVAELipschitzVAESEAVAEOurs
A tt a c k l o s s
VAEBetaTCVAELipschitzVAESEOurs
A tt a c k l o s s
VAEBetaTCVAELipschitzVAESEOurs (b) Maximum damage attack.
Further, we consider maximum damage or output space attack.
In this setting, the adversary perturbsthe input data point to cause maximum damage in the reconstruction of the decoder f of the modeland optimizes the following objective,
[Equation]
In both scenarios, the noise perturbation is explicitly constrained by some constant λ > 0 to ensure aconsistent comparison with the baseline models.
Evaluation.
We consider three evaluation metrics to quantitatively estimate the robustness of thedifferent models for the above-mentioned adversarial attacks.
First, we evaluate the attack loss of theadversary.
That is, we report the achieved value of the optimization objectives in (12) and (13).
Theobserved attack losses for the two forms of adversarial attacks are shown in Figure 2 (MNIST resultsare given in the Appendix).
Higher attack losses correspond to less successful attacks and hence betterrobustness.
Due to the different regularization methods used in the baseline models, the inherentscale of the aggregated posterior changes and hence the value of the attack objective in eqn 12.
Hencefor latent space attacks, the values reported in Figure 2 might not be directly comparable.
Anotherstrong indicator for the robustness of the considered models is the similarity between the referenceimage x r and the reconstructions ˜ x a of the adversarially perturbed variants x a .
In the case of latentspace attacks, we consider the target image x r = x t as the reference images and compare with thereconstruction ˜ x a = f ( z x + δ ) of the perturbed latent representation.
For a maximum damage attack,we consider the original images x r = x as the reference images and compare them to ˜ x a .
Similarto [20], we use the perception-based similarity metric Multi-Scale Structural Similarity Index Measure(MSSSIM ∈ [0 , 1] ) to compare the images.
We also report the l 2 -distance as an alternative similaritymetric to MSSSIM in the Appendix.
Lower values of MSSSIM( x r , ˜ x a ) indicate less similaritybetween the reference (target) image and the adversarial reconstructions and correspond to lesssuccessful latent space attacks.
And higher value of MSSSIM( ˜ x r , ˜ x a ) corresponds to less successfulmaximum damage attacks as they correspond to high similarity between reference (original) and theadversarial reconstructions.
We report the observed values in Table 1.
Finally, it is also essential toevaluate how similar the adversarial images are to the original image.
Ideally, a successful attackimplies that both adversarial and original images look similar in appearance (MSSSIM ≈ 1 ).
Hencewe consider the MSSSIM between the original and adversarial images in Table 1.
We also report the
FID of the generated samples to compare the fidelity of the representations.
Results.
Overall we see that the proposed model outperforms all the considered baselines in terms ofboth robustness and offers superior generation performance.
Even for complex datasets like SVHN
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
and CELEBA, we observe the same trend with FID and robustness measures.
These results areespecially promising since we did not employ any extensive hyperparameter search for training.
Our results further confirm that both robust and high fidelity models are possible.
Since we employan FGSM based adversarial training, the training time required is cheaper when compared to theexpensive PGD-based training used in smooth encoders (SE).
The computation time for a singleiteration of SEs is two times more when compared to our method.
Ablation Study.
In this section, we aim to compare the robustness and fidelity of our model to theoriginal model definition in [26] and to the augmented model definded in eqs (7) (8).
The observedvalues are given in Table 2.
We see that the proposed model exhibits better robustness and maintainssimilar fidelity to the non robust counterpart GMM-DAE [26].
As discussed in Section 3, sincesimilar points are structured in same GMM component in a multimodal deterministic autoencoder,we observe improved robustness for GMM-DAE when compared to a standard VAE setting (Table 1).
Robustness to downstream applications.
Since the learned representations of VAEs are often usedfor various downstream tasks, it is also vital to verify how adversarial attacks affect the performanceof the same.
To showcase the effect of adversarial attacks to downstream classification tasks, wetrain a MLP classifier on the latent space of the model and observe the accuracy drop in the presenceof the latent space attacks for a constrained noise norm λ = 1 .
The observed values along with theclean accuracy are shown in Table 3.
The classifier trained on the latent space of the proposed modelachieves better accuracies when compared to the baseline models under latent space attack.
Discussion & Conclusion
Limitations.
Although cheaper when compared to the current adversarially trained robust VAEmodels, the FGSM based adversarial training scheme is still expensive when compared to the nonrobust counterpart.
The coupling parameter in the proposed method is an additional hyperparameterto tune and enforcing strong coupling i.e. α = 1 , might compromise the generation fidelity of theresulting model.
While we need the coupling to perform adversarial training, overly strong couplingwill necessarily lead to deteriorated reconstructions, similar as in VAEs.
It is worth noting that wedid not perform intensive hyperparameter optimization in this line of experiments.
Hyperparameteroptimzation might be needed in other datasets for this reason.
Further, datasets with large inter classvariance might not benefit from the underlying multimodal prior assumptions in the current approach.
It would be interesting to explore the impact of the adversarial attacks in other potential downstreamapplications such as optimization in the latent space of VAE models.
Conclusion.
Developing robust VAE models is crucial since the learned representations of VAEsare frequently used for various applications.
Motivated by the recent research towards deterministicalternatives to VAEs, we study the robustness of deterministic autoencoders.
We extend recentlydeveloped regularization schemes to efficiently couple the adversarial examples and the learned representations during training.
The proposed adversarial training scheme is comparatively cheap whencompared to the existing robust VAE models.
Our experiments show that with proper regularization,adversarially trained multimodal deterministic autoencoders offer significantly improved adversarialrobustness and high fidelity in the learned latent space.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section
(b) Did you include complete proofs of all theoretical results?
[Yes] See Section
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See Appendix
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Appendix
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes] See Appendix
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you citep the creators?
[Yes](b) Did you mention the license of the assets?
[Yes] See Appendix
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
See Appendix
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e ) Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
[N/A]5.
If you used crowdsourcing or conducted research with human subjects...
[Table] (a) Did you include the full text of instructions given to participants and screenshots, if [Table] applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]