Private Set Generation with Discriminative Information
Anonymous Author(s)
AffiliationAddress email
Abstract
Differentially private data generation techniques have become a promising solutionto the data privacy challenge –– it enables sharing of data while complying withrigorous privacy guarantees, which is essential for scientific progress in sensitivedomains.
Unfortunately, restricted by the inherent complexity of modeling highdimensional distributions, existing private generative models are struggling withthe utility of synthetic samples.
In contrast to existing works that aim at fittingthe complete data distribution, we directly optimize for a small set of samplesthat are representative of the distribution under the supervision of discriminativeinformation from downstream tasks, which is generally an easier task and moresuitable for private training.
Our work provides an alternative view for differentiallyprivate generation of high-dimensional data and introduces a simple yet effectivemethod that greatly improves the sample utility of state-of-the-art approaches.
Introduction
Data sharing is vital for the growth of machine learning applications in numerous domains.
However,in many application scenarios, data sharing is prohibited due to the private nature of data (e.g., individual data stored on mobile devices, medical treatments, and banking records) and the correspondingstringent regulations, which greatly hinders technological progress.
Differentially private (DP) datapublishing [7, 8, 10] provides a compelling solution to such challenge, where only a sanitized formof the data is publicly released.
Such sanitized synthetic data can be leveraged as if it were thereal data, analyzed with established toolchains, and can be shared openly to the public, facilitatingtechnological advance and reproducible research in sensitive domains.
Yet, generation of high-dimensional data with DP guarantees is highly challenging and traditional
DP algorithms designed for capturing low-dimensional statistical characteristics are not applicableto this task [27, 12, 4, 34]).
Instead, inspired by the great successes of deep generative models inlearning high-dimensional representations, recent works [5, 6, 39, 40, 2, 11] adopt deep generativeneural networks as the underlying generation backbone and incorporate the privacy constraints intothe training procedure, such that any privacy leakage upon disclosing the data generator is bounded.
However, these methods have common shortcomings: (i) deep generative models are known to be datademanding [15], which becomes even harder to train when considering the privacy constraints [6, 5];(ii)
they do not guarantee any optimal solution for the downstream task (e.g. classification).
Infact, existing models are still struggling to generate sanitized data that is useful for downstreamdata analysis tasks.
For example, when training a convolutional neural network (ConvNet) classifieron the private generated data, the highest test accuracy reported in literature is < 85% for MNISTdataset with ( ε, δ ) =
(10 , 10 − 5 ) [5], which lags far behind the discriminative baseline ( > 98% with( ε, δ ) =
(1 . 2 , 10 − 5 ) [31]) and makes private generative models less appealing for many practicalscenarios with data analysis as the end goal.
In this work, we learn to synthesize informative samples that are privacy-preserving and are optimizedto train neural networks for downstream tasks.
In contrast to existing approaches, we directly optimizea small set of samples instead of the deep generative models that is notoriously difficult to train in aprivate manner.
Moreover, we exploit discriminative information from downstream tasks to guidethe samples towards containing more useful information for downstream analysis.
Compared toexisting works, we improve the task utility by a large extent (up to 10 % downstream test accuracyimprovement over state-of-the-art approach), while still preserving the flexibility and generalityacross varying configurations for downstream analysis.
As an added benefit, our formulation naturallydistilled the knowledge of original data into a much small set, which largely save the memory andcomputational consumption for downstream analysis.
We summarize our main contributions as follows
• We present a new perspective of private high-dimensional data generation, with which weaim to bridge the utility and generality gap between the private generative and discriminative
models.
We believe this alternative view opens up new possibilities in different researchdirections ranging from private analysis to generation.•
We introduce a simple yet effective method for generating informative samples that are
optimized for training downstream neural networks, while maintaining the generality aswell as reducing the memory and computation consumption as added benefits.
• Experimental results demonstrate that, in comparison to existing works, our work improves
the sample utility by a large margin and offers superior practicability for real-world applica-
tion scenarios.
Related Work
Differentially Private Generative Models Training deep generative models in a private manner hasbecome the default choice for private high-dimensional data generation.
Existing methods typicallyadopt differentially private stochastic gradient descent (DP-SGD) [1, 30, 6, 5] or Private Aggregationof Teacher Ensembles (PATE)
[22, 23, 19, 35] to equip the deep generative models with rigorousprivacy guarantees.
Despite significant progress in mitigating training instabilities and improvinggeneration (visual) quality, existing works are still far from being optimal in terms of the sampleutility.
This is mainly because that existing works are attempting to solve a problem that is inherentlyhard and almost impossible to be solved accurately under current private training framework.
Incontrast, we directly optimize the samples (rather than the deep generative models that is much harderto train in a private setting) and exploit the knowledge from a general class of downstream tasks thatcan be employed on the samples to further guide the training.
Coreset Selection and Generation
Our work is largely motivated by recent success in distilling alarge dataset into a much smaller set of representative samples, i.e., the coreset.
For example, samplesfrom a dataset are selected to be representative based on their ability to mimic the gradient signal [21],hardness to fit [29], distance to the cluster centers [37, 26], etc.
Instead of selecting samples from thedataset, our work focus on synthesizing informative samples from scratch [36, 42, 41, 18] under DPconstraints, and optimize the sample utility for training downstream neural networks.
Moreover, ourset generation formulation is similar in spirit to works in the field of private queries release [27, 12, 4,13] which synthesize a set of pseudo-data (under DP guarantees) that is representative of the originaldata in answering linear queries.
However, as neural networks exhibit highly nonlinear properties,methods target at linear queries are generally not applicable to our case and are algorithmicallydistinct from approaches designed for neural nets.
Background
We consider the standard central model of DP in this paper.
We below present several definitions andtheorems that will be used in this work.
Definition 3.1.
(Differential Privacy (DP) [7])
A randomized mechanism M with range R is ( ε, δ ) -
DP, if
[Equation]
holds for any subset of outputs O ⊆ R and for any adjacent datasets D and D ′ , where D and D ′ differ 86 from each other with only one training example.
M corresponds to the set generation algorithm in 87 our case, ε is the upper bound of privacy loss, and δ is the probability of breaching DP constraints.
DPguarantees the difficulty of inferring the presence of an individual in the private dataset by observingthe generated set of samples M ( D ) .
Our approach is built on top of the Gaussian mechanism defined as follows.
Definition 3.2.
(Gaussian Mechanism [9]) Let f : X → R d be an arbitrary d -dimensional functionwith sensitivity being
[Equation]
over all adjacent datasets D and D ′ .
The Gaussian Mechanism M σ , parameterized by σ , adds noise 94 into the output, i.e.,
[Equation]
M σ is ( ε, δ ) -DP for σ ≥
[Equation]
Any privacy cost is bounded upon releasing the private set of generated data due to the closedness of
DP under post-processing.
Theorem 3.1.
(Post-processing [9])
If M satisfies ( ε, δ ) -DP, F ◦ M will satisfy ( ε, δ ) -DP for anyfunction F with ◦ denoting the composition operator.
Method
We consider a standard classfication task where we are given a private dataset D
= { ( x i , y i ) } Ni =1with x
i ∈ R
d the feature, y i ∈ { 1 , ..., L } the class label, N the number of samples, L the number of 103 label classes.
Our objective is to synthesize a set of samples
S = { ( x S i , y S i ) } Mi =1 such that (i) samples 104 in
S have the same form as data in D ; (ii) a neural network trained on S should maximally match 105 generalization performance of a deep neural network that is trained on
D ; (iii) the privacy leakage of
D when releasing S is upper bounded by a pre-defined privacy level ( ε, δ ) .
Let F ( · ; θ D ) and F ( · ; θ S ) be the deep neural networks parameterized by θ D and θ S that are trained 108 on
D and S respectively.
The objective can be formulated as:
[Equation]
) where ℓ denotes the loss function (e.g., cross-entropy for the classification task) and the expectationis taken over the real data distribution P D .
Equation 4 can be naturally achieved once θ S ≈ θ D .
In particular, when given the same initializationθ D 0 = θ S 0 , solving for θ S t ≈ θ D t at each training iteration t leads to θ S ≈ θ D as desired.
This can be 113 achieved by optimizing the synthetic set
S such that it yields a similar gradient as if the network istrained on the real dataset at each iteration t :
[Equation]
[Equation] where ∇ θ L ( S , θ t )) corresponds to the gradient of the classification loss on the synthetic set S ,∇ θ L ( D , θ t ) denotes the stochastic gradient on the real data, and L dis is a sum of cosine distances 117 between the gradients at each layer [42, 41]
(See supplementary material for more details).
sensitive (
[Equation]
private
[Figure]
inner loop outer loop
[Equation]
[Table]
[Figure]
To mimic the training procedure, S and the network
F ( · ; θ ) are updated jointly in a iterative manner,where in each outer iterations the S is trained to minimized the gradient matching loss L dis and in each 122 inner iterations the network parameters θ t are optimized towards minimizing the classification loss onthe synthetic set S .
Moreover, S is optimized over 125 multiple initialization of network parameters θ 0 toensure the generalization ability of S over different 127 random initialization when training a downstream model.
The objective can be summarize as follows [36, 42, 41]:
[Equation]
where P θ 0 stands for the distribution over the initialization of network parameters.
We incorporate DP constraints by sanitizing the stochastic gradient on real data ∇ θ L ( D , θ t ) at each 132 outer iteration, while leaving the inner iterations unchanged as their privacy is guaranteed by thepost-processing theorem 3.1.
The final objective can be formulated as follows:
[Equation]
where we use [Equation] g D θ t to denote the parameter gradient on D that is sanitized via Gaussian mechanism 3.2,and g S θ t to denote the parameter gradient on S .
The whole pipeline is summarized in Algorithm 1.
Weuse the subsampled Renyi-DP accountant [1, 20] to compute the overall privacy cost accumulated foriteratively updating S .
Note that the training procedure and the privacy computation are approximatelyas simple as training a classification network with DP-SGD, which in general has lower difficultythan training a DP deep generative models as done in existing works (witnessed by a significantperformance gap in terms of the classification accuracy).
Moreover, in contrast to previous works, oursynthetic set S provide utility guarantees for downstream tasks, while such guarantees generalizedover novel random initialization taken place in downstream analysis phase.
Input: Dataset D = { ( x i , y i ) } Ni =1 , learning rate for update network parameters τ θ and τ S , batch size B , DP noise scale σ , gradient clipping bound C , number of runs R , outer iterations T , inner iterations J , batches K , Output: Synthetic set S , Privacy cost ε
Initialize synthetic set S (features x S are from Gaussian noise; labels are balanced set depends on the pre-defined number of samples per class) ; for run in { 1 , ..., R } do Initialize model parameter θ∼ P [Equation] ; for outer_iter in { 1 , ..., T } do θ t +
=
θ t for batch_index in { 1 , ..., K } do Uniformly sample random batch { ( x i , y i ) } Bi =1 from D ; for each ( x i , y i ) do // Compute per-example gradients on real data
[Equation]
Clip gradients
[Equation]
end //
Add noise to average gradient with Gaussian mechanism
[Equation]
// Compute parameter gradients on synthetic data and update S
[Equation]
end endfor inner_iter in { 1 , ..., J } do // Update network parameter using S
[Equation]
[Equation]
end endreturn Synthetic set S , Privacy cost ε given a pre-defined δ
[Table]
Experiment
5.1 Classification
We first compare private set generation (PSG) with existing DP generative models on standardclassification benchmarks including MNIST [16] and FashionMNIST [38].
Setup.
We use by default a ConvNet with 3 blocks where each block contains one Conv layerwith 128 filters, followed by Instance Normalization [32], ReLU activation and AvgPooling modulesand a fully connected (FC) layer as the final output layer.
We initialize the network parametersusing Kaiming initialization [14] and the synthetic samples using standard Gaussian.
We reportthe averaged results over 3 runs of experiments for all the comparisons.
We list below the defaulthyperparameters used for the main experiments and refer to the supplementary material for moredetails: Clipping bound C = 0 .
1 , R = 1000 for ε = 10 (and 200 for ε = 1) , number of samples perclass (spc) ∈ { 10 , 20 } , K = 10 , T = 10 for spc=10 (and =20 for spc=20).
Comparison to state of the art.
We show in Table 1a the results of, to the best of our knowledge,all existing DP high-dimensional data generation methods (whose validity has been justified viapeer review at top-tier conferences) that report results on the benchmark datasets we consider.
These include DP-CGAN [30], G-PATE [19], DataLens [35], GS-WGAN [6], DP-Merf [11], DPSinkhorn [5].
For methods that are not open-sourced, we report the original results from the publishedpaper.
As shown in Table 1a, our formulation results in significant improvement in the sample utility(measured by test accuracy on real data) for training downstream classification models.
Specifically,the improvement is consistent and significant (around 5-10% increase over different configurations)for both the low privacy budget regime ( ε =1) (around 8-9% improvement over SOTA in this case)and a relatively high privacy regime ( ε =10) where all the investigated methods achieve convergence(around 10% and 5% increase in test accuracy for MNIST and FashionMNIST, respectively).
Notethat in contrast to most existing methods that show superiority only for a certain range of privacylevel, our improvement covers a wide range, if not all, of practical scenarios spanning across differentprivacy level.
We then focus on the open-sourced methods that are strictly comparable (e.g., G-PATE and DataLensprovide data-dependent ε , i.e., publishing ε value will introduce privacy cost, and are thus not directlycomparable) to ours and conduct a comprehensive investigation trough different angles.
Memory and computation cost.
We additionally show that our method is the only one that simultaneously show advantages in reducing the memory and computation consumption of downstreamanalysis.
As shown in Table 1b, training the classifier with full (6000 samples per class) size ofsamples in most cases yields an upper bound for the test accuracy, while training on randomly subsampled smaller sets will decrease the performance, unless the generated samples are not informativesuch that they can be harmful for the downstream tasks (e.g., for DP-CGAN).
In contrast, we directly
[Table]
[Figure]
optimize to compress the useful information into small set of samples and naturally save the memoryand computation consumption for downstream analysis tasks.
Generalization ability across different architectures.
One natural concern of our formulationcould be the generalization ability to unseen situations.
While we exploit discriminative informationto guide the training, we (in principle) inevitably trade the generality off against task-specific utility,leaving no performance guarantees for new models.
Interestingly, as shown in Table 2, we find thatour generated set still provides better utility than all baseline methods in most cases, even though themodels for evaluation have completely different architecture from the one we used for training.
Theonly case where our generated set do not work well is for training MLPs.
We conjecture that it is dueto the difference in the network properties that results in distinct gradient signals: for example, layersin MLPs are densely connected while being sparsely connected in ConvNets, and Convolutionallayers are translation equivalent while FC layers in MLPs are not.
Moreover, we argue that this maynot be a bug, but feature.
Note that the reference results on real data also indicate that the MLP isinferior to other architectures while models with ConvNet, VGG, or ResNet architecture performswell in most cases.
In this regard, results on our generated set generally align well with the result onreal data, which suggest the possibility of conducting model selection with our private generated set.
Convergence rate.
For most private (gradient-based) iterative methods, the privacy cost accumulatesin each training iteration, and thus faster convergence is highly preferable.
We show in Figure 2 thetraining curves where the y-axis denotes the utility and x-axis corresponds to the privacy.
We observethat our method generally has a much faster convergence rate than existing methods that need toaccumulate the privacy cost for each iteration.
In particular, our method already achieve a decentlevel of utility with ε ≤ 2 which is much lower than the privacy budget used in most previous works(normally ε = 10 ).
5.2 Application: Private Continual Learning
The utility guarantee of our formulation requires that the network architecture is known to the dataprovider/generator.
Fortunately, it is not a rare case in practice.
In particular, our method is naturallyapplicable to cases where (i) there are multiple parties involved in training a model and they agree on
[Figure]
one common training protocol (i.e., the network architecture is known to all participants); (ii) eachparty has its own data whose privacy need to be protected (i.e., the training need to be DP); (iii) dataon each party exhibit distinct property and is all informative for the final task (i.e., a synthetic set ofrepresentative samples that capture such properties would greatly aid the final task).
One example is continual learning [17, 28] where the training of the classification network is splitinto stages.
Here we consider a setting adjusted to the DP training: to protect the privacy of its data,each party is responsible for a different training stage where it performs DP training of the model onits data, and subsequently delivers the DP model to the party responsible for the next training stage.
Note that the raw data would not be transferred as otherwise the privacy would be leaked.
We conduct DP training on the SplitMNIST and SplitFashionMNIST datasets where the data ispartitioned into 5 parts (based on the class labels, which corresponds to the class-incremental [26]setup) and we assume each part is held by one party and can not be accessed by others for privacysake (See supplementary material for more details).
We show in Figure 3 (green curves) the baselineresults of DP training of model under the private class-incremental setting (i.e., each party finetunethe model is obtained from the previous stage on its own data using DP-SGD).
Apparently, this naivetraining scheme leads to catastrophic forgetting of information learned in the early stages.
Evenworse is that the common strategy to cope with this issue requires transferring a small set of realdata to other parties such that it can be replayed in the later training stage [26, 3, 24], which is notdirectly applicable to the private setting as transferring the data breaks privacy.
In contrast, privategeneration methods can be seamlessly applied to this case, where a set of DP synthetic samplesis transferred to enable the final model to learn the characteristics of each partition of data.
Inparticular, our formulation is better suitable for this setting than other generation methods as thenetwork architecture is known to all participants and samples can be tailored to the specific networkvia our formulation.
This is verified in Figure 3, where our synthetic samples are generally moreinformative for training the classifier when compared to DP-Merf – the overall best existing works interms of the downstream utility.
Moreover, as our formulation condenses the information into a smallset of samples by construction, we also enjoy the advantages when considering the computation,storage, and communication cost.
Discussion
In this section, we present several key factors that distinguish our approach from existing ones anddiscuss possible concerns regarding our private set generation formulation.
Trade-off between Visual Quality and Task Utility.
Our formulation is designed for optimizingthe utility of downstream analysis tasks instead of the visual appearance as done in previous works,thereby leaving no performance guarantee for the visual quality of the synthetic samples.
Moreover,the optimization of the private synthetic set is unconstrained and unregulated over the whole dataspace, with the gradient signal as the only guidance.
As the data to gradient mapping is generallynon-injective (i.e., different data can results in the same gradient), searching for the correct data given
[Figure]
[Figure]
the gradient is an indefinite problem, which inevitably leads to outcomes that are out of the datamanifold in practice.
This can be seen in the first row of Figure 4 where we plot our private syntheticsamples trained under the default setting.
Recall that one key difference between our formulation and existing works is that: we directlyoptimize for a set of samples instead of the deep generative models.
We then take a further step andinvestigate whether this difference is the key factor that determines the samples’ visual quality.
To dothis, we employ a DCGAN [25] model from [6] as the generator backbone (denoted as G ), let x S tobe outputs of G , and then optimize over the network parameter of G using the gradient matching lossas in Equation 5.
Mathematically, this transforms Equation 7 into:
[Equation]
with
[Equation]
where φ is the parameter of G , z i is random Gaussian noise (fixed during training).
Basically, thisformulation restricts the synthetic images to be within the output space of G , and the inductive biasintroduced by the convolutional structure serves as deep image prior [33] to regularize the visualappearance of the synthetic images.
We show the synthetic samples in the second row of Figure 4, and compare the utilities with ouroriginal formulation in Figure 5.
We observe that the prior from the deep generative model is indeedimportant for the visual quality.
However, interestingly, better visual quality does not mean betterutility.
Specifically, optimizing over the parameter of generator G exhibits a slower convergence thandirectly optimizing the samples, while the final performance is also inferior (See quantitative resultsin Table 3).
This gives several important indications that help inform future research in this field:
(i ) the goal of achieving better downstream utility may be incompatible with the goal of achievingbetter sample visual quality, while dedicated efforts towards different goals are necessary; (ii) deep generative models may not be the best option for the task of private data generation as they result insuboptimal utility (mainly due to its slow convergence), which questions the current default way ofthinking in this field.
Scalability & Transparency.
We discuss the possible issues when scaling to more complicateddatasets which: (i) contains a large number of label classes; (ii) are diverse and require a large numberof samples to capture the statistical characteristics of the data distribution.
For (i) , the complexity ofour (and all the other) approaches will definitely increase as the number of label classes increases.
When considering the number of variables that need to be optimized, the complexity increases linearlyfor our case, while for all methods (that optimize over the network parameters) the increase is noless than ours.
While the application of DP deep learning (of discriminative models) to datasets with>10 label classes is rare, we anticipate that dealing with a much larger number of label classes is tooambitious for DP generative modeling for now.
For (ii) , we conduct the experiment when varyingthe number of samples per class and present the results in 4, where we indeed observe the trainingdifficulty when the number of samples increases.
We conjecture that it is mainly because the gradientsignals for updating the synthetic samples get sparser when the number increases, which results ina lower convergence rate and thus worse results especially when the allowed privacy budget is low.
However, it is arguable whether this is a shortage as smaller amounts of samples allow more savingsin the storage and computation consumption while providing greater transparency of downstreamanalysis.
[Table]
MNIST
FashionMNIST
[Table] [Table]
Generality and Expressiveness.
Our formulation focus on the task of training downstream neuralnetworks, and thus have no guarantees for other (and more general) purpose.
In contrast, deepgenerative models are designed for capturing the complete data distribution and, once perfectlytrained, can be applied to more general cases.
While our formulation seems to be inferior in thisregard, we argue that this should not be a major shortcoming that outweighs the advantages: Firstof all, while deep generative models in principle have much greater expressiveness than a smallset of samples, such upper bound is hard, if not impossible, to be achieved in the privacy learningsetting.
Instead, compromising the upper bound for a more achievable target is worthy and showsgreat improvement over existing works as demonstrated in section 5.1.
Moreover, our formulationgeneralizes seamlessly to any gradient-based learning methods that a downstream analyst may adopt.
While such methods already cover the most part of the possible analysis algorithms that could beadopted for high-dimensional data, we believe that our approach does exhibit a good level of practicalapplicability.
Conclusion
We introduce a novel view of private high-dimensional data generation: instead of attempting to traindeep generative models in a DP manner, we directly optimize a set of samples under the supervisionof discriminative information for downstream utility.
We present a simple yet effective method thatallows synthesizing a small set of samples that are representative of the original data distribution andinformative for training downstream neural networks.
We demonstrate via extensive experimentsthat our formulation leads to great improvement over state-of-the-art approaches in terms of the taskutility, without losing the generality for performing analysis tasks in practice.
Moreover, our resultsquestion the current default way of thinking and provide insights for further pushing the frontier inthe field of private data generation. Our source code
Broader Impact
The success of many machine learning methods hinges upon the availability of (large) datasets, whichis problematic if the data is sensitive and contains private information.
Yet, novel applications e.g. inthe health domain have a tremendous potential to facilitate breakthroughs in diagnosis, treatment,and personalized medicine.
In contrast to direct privacy-preserving analytics, privacy-preserving generative models allow for asafe way to release data.
This difference has several important implications: (1) potentially higheradoption rate of privacy compliance, as data is private and algorithms don’t need to be changed (2)facilitating new scientific discovery that would otherwise be neglected due to data protection hurdles(3) facilitating public benchmarks/dataset in domains with sensitive data to foster comparison andreproducible research.
Therefore, our submission contributes to making the latest advances in generative modeling complywith data privacy – a commonly agreed on societal value.
Our method improves the state of the artin privacy-preserving generative models.
In particular, we show the success on high dimensionaldata, which will be key to bringing those methods to a broader range of applications.
To increase theimpact on the research community and support reproducible research, we will release the source codeand setup at the time of publication.
Consequently, we see the broader impact generally as positive.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .•
Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 6(c)
Did you discuss any potential negative societal impacts of your work?
[Yes](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] Code will bereleased upon publication.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Section
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] In supplementary
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]