Semantic uncertainty intervals for disentangled latent spaces
Anonymous Author(s)
AffiliationAddress email
Abstract
Meaningful uncertainty quantification in computer vision requires reasoning aboutsemantic information—say, the hair color of the person in a photo or the location ofa car on the street.
To this end, recent breakthroughs in generative modeling allowus to represent semantic information in disentangled latent spaces, but providinguncertainties on the semantic latent variables has remained challenging.
In thiswork, we provide principled uncertainty intervals that are guaranteed to contain thetrue semantic factors for any underlying generative model.
The method does thefollowing: (1) it uses quantile regression to output a heuristic uncertainty intervalfor each element in the latent space (2) calibrates these uncertainties such thatthey contain the true value of the latent for a new, unseen input.
The endpoints ofthese calibrated intervals can then be propagated through the generator to produceinterpretable uncertainty visualizations for each semantic factor.
This techniquereliably communicates semantically meaningful, principled, and instance-adaptiveuncertainty in inverse problems like image super-resolution and image completion.
Introduction
When making decisions with visual data, such as automated vehicle navigation with blurry images,uncertainty quantification is critical.
The relevant uncertainty pertains to a low-dimensional setof semantic properties, such as the locations of objects.
However, there is a wide class of imagevalued estimation problems—from super-resolution to inpainting—for which there does not currentlyexist a method of producing semantically meaningful uncertainties.
As an example, imagine doinguncertainty quantification for medical image reconstruction from, say, a fast but undersampled MRIscan. In such a setting, pixelwise intervals [16, 39, 5] are not very useful. We need uncertainty on theunderlying semantics—e.g., whether there is a tumor, and if so, of what size and shape.
We make progress on this problem by bringing techniques from quantile regression and distributionfree uncertainty quantification together with a disentangled latent space learned by a generativeadversarial network (GAN).
We call the coordinates of this latent space semantic factors , as each controls one meaningful aspect of the image, like age or hair color.
We do not require these semantic factors to be statistically independent.
Our method takes a corrupted image input and predicts eachsemantic factor along with an uncertainty interval that is guaranteed to contain the true semanticfactor.
When the model is unsure, the intervals are large, and vice-versa.
By propagating theseintervals through the GAN coordinate-wise, we can visualize uncertainty directly in image-spacewithout resorting to per-pixel intervals—see Figure 1.
The result of our procedure is a rich form ofuncertainty quantification directly on the estimates of semantic properties of the image
More concretely, we receive input images X and then predict an uncertainty interval for each of D semantic factors Z d , d = 1 , ..., D which are the elements of a disentangled latent space
The method
[Figure]
involves training an encoder (a neural network that takes images as input and produces outputs in thelatent space of the GAN) on ( X, Z ) pairs to give us three different outputs:
The point prediction, f ( X ) .
This is the encoder’s best guess at the semantic factors Z .
The estimated lower conditional quantile, q α 2 ( X ) .
The encoder believes that q α 2 ( X ) is a lower bound on the value of Z given X .3.
The estimated upper conditional quantile, q 1 − α 2 ( X ) .
The encoder believes that q 1 − α 2 ( X )is an upper bound on the value of Z given X .
Once the above encoder is trained, as described in Section 2.2, we use it to form an uncertaintyinterval for each semantic factor.
However, for the d th element of the latent code, the naive interval( q α 2 ( X ) d , q 1 − α 2 ( X ) d ) is not guaranteed to contain the ground truth value in finite samples.
Wepropose to perform a calibration procedure to fix this problem, yielding the sets
[Equation]
(1) where q cal is a calibrated version of q constructed using the tools in Section 2.3.
Once we have doneso, the intervals will contain α fraction of the true latent codes with high probability.
In other words,for any user-chosen levels α and δ , we can output intervals that with probability 1 − δ satisfy
[Equation]
for a new test point ( X, Z ) , regardless of the distribution of the data, the encoder used, and thenumber of data points used in the calibration procedure.
This guarantee, described more carefullyin Definition 2.1, says that the intervals cover 1 − α fraction of the semantic factors unless ourcalibration data is not representative of our test data (which only happens with a probability δ whichgoes to 0 as the number of calibration data points grows).
We visualize each of the d ∈ { 1 , ..., D }intervals in latent space by propagating the d th lower and upper endpoints through the generator withall other entries in the latent fixed to the point prediction (see Section 2.3 for a formal explanation).
1.1 Central Contribution
To our knowledge, this is the first algorithm for uncertainty intervals on a learned semantic representation with formal statistical guarantees.
By propagating these intervals through the generator, we areable to visualize uncertainty in an intuitive new way that directly encodes semantic meaning.
This isan important step towards interpretable uncertainties in general image-valued estimation problems.
However, the reader should note that our technique requires access to a disentangled latent space,such as that of a StyleGAN, and provides no guarantees about the degree of disentanglement.
Method
2.1 Notation and goal
Our data consist of pairs ( X, Z ) —the corrupted image X in X =
[0 , 1] H × W , and the latent code
Z ∈ Z , where Z = R D .
As mentioned in the introduction, we think of Z as a disentangled representation with d semantic factors — i.e. factors of variation corresponding to interpretable featuresin an image, such as hair color and expression.
For simplicity, assume each of the d dimensionscontrols a single semantic factor; in practice, we ignore those that do not.
In our sampling model, X is generated from Z by composing two functions.
The first function is afixed generator G : Z → Y , where
Y =
[0 , 1] H × W , which takes the latent vector Z and producesthe ground truth image Y ∈ Y (for ease of notation, we assume X and Y have the same shape).
The second function is a corruption model, F : Y → X , which degrades the ground truth image
Y to produce the corrupted image X , for example by randomly masking out part of the image.
Tosummarize our data-generating process, we have
[Equation]
Goal #1.
Our first goal is to train an encoder E to recover Z from X —in other words, to invertthe mapping F ◦ G —with a heuristic notion of uncertainty.
The encoder’s point prediction will bea function f : X → Z .
The uncertainty will be parameterized by two functions, q 1 − α 2 : X → Zand q α 2 : X → Z , denoting our estimates of the 1 − α 2 and α 2 conditional quantiles, respectively.
These conditional quantiles are potentially bad estimates; they do not natively possess the statisticalguarantee we desire.
Goal #2.
Having trained the encoder and the conditional quantile estimates, we will output uncertaintyintervals in the disentangled latent space.
Each dimension will get its own interval, which has theform in (2).
Ultimately, our uncertainty intervals will come with a statistical risk control guarantee,like in (2), that is distribution-free—i.e., valid irrespective of the model or data distribution. The riskfunction controlled is the false negative rate in the latent space . In other words, we bound the fractionof semantic factors not covered by their calibrated intervals above by α (say, α = 10% ).
Definition 2.1 (Risk-Controlling Prediction Set (RCPS)) .
A set-valued function T : X → 2 Z is an( α, δ ) risk-controlling prediction set if
[Equation]
where
[Equation]
The reader should note here that the function T depends on the calibration data.
The outer probabilityin (4) is over the randomness in this calibration procedure; the inner expectation is over the new testpoint, ( X, Y ) .
The reader should note that in our setting, because we can generate infinite data fromthe GAN, it is always possible to take δ arbitrarily close to 0 , effectively making T nonrandom.
Inthe two following subsections, we address each of our goals separately.
2.2 Goal #1: Training the encoder for quantile regression
Our job in this subsection is to learn the three functions f , q α 2 , and q 1 − α 2 .
We do so by training aneural network with three different loss functions, one for each of three linear heads on top of thesame feature extractor—see Figure 2 for the training protocol.
Loss function for point prediction.
We supervise the point prediction with two loss functions.
Thefirst is an L 1 loss directly in the latent space, and encourages f ( X ) to be close to Z .
[Equation]
The second is an identity loss on the generated image G ( f ( X )) , which encourages G ( f ( X )) tocontain objects that have the same “identity” — or semantic meaning — as those in Y .
We calculatethe identity loss using a pretrained embedding function, ID :
Y → R d ′ for some d ′ , which projects Yto an embedding space where images with different identities land far away from one another.
Finally,we calculate our loss using cosine similarity,
[Equation]
X
Y
E
G corrupt generator sample encoder image space , , latent space losses predict gradients
Finally, we combine these two loss functions to form the loss function for the prediction f ,
[Equation]
8) where c = 0 .
7 was chosen to balance the two losses.
Loss function for quantile regression.
Quantile regression [29, 13, 31, 26, 27, 28, 30] is a statisticalmethod for estimating the conditional quantiles of a distribution.
The key idea of quantile regressionis to supervise the regressor using a quantile loss ,
[Equation]
The minimizer of the quantile risk is the true β conditional quantile of Z | X .
We supervise ourconditional quantile estimates q α 2 and q 1 − α 2 with two separate instances of the quantile loss, L α 2 q and
L 1 − α 2 q respectively.
This concludes our explanation of the model training procedure, summarized in Algorithm 1.
Experimental details, such as the particular model architecture we use, are available in Section 3. Algorithm
Input: training dataset D , risk level α , number of epochs E , fixed generator G
Output: trained functions f, q α 2 , and q 1 − α 2 .
f, q α 2 , and q 1 − α 2 ← random model initialization for e = 1 to E do loss
2.3 Goal #2: Calibration
Having trained the model, we now calibrate it to achieve the statistical guarantee in Definition 2.1using a set of calibration data (cid:8)
( X i , Z i ) (cid:9) n i =1 generated from the model and the upper-confidencebound procedure from [8].
The output of the procedure will be the function T from (2); specifically,we will learn the calibrated conditional quantiles q cal α 2 and q cal1 − α 2 .
Our procedure will calibrate the conditional quantiles by rescaling their size multiplicatively.
We willultimately choose a multiplicative factor ˆ λ that gives us the desired guarantee.
Towards that end, weindex a family of uncertainty intervals scaled by a free parameter λ for each semantic factor, [Equation] λ
[Equation]
X
[Equation]
(cid:21) .
When λ grows, the interval T λ ( X )
d also grows, and thus, the loss function L ( T λ ( X ) , Y ) shrinks.
Therefore, by taking λ large enough, we can always ensure the loss is zero.
The challenge ahead is topick ˆ λ to be the smallest value such that T ˆ λ is an RCPS as in (4).
The algorithm for selecting ˆ λ involves forming an upper confidence bound (UCB) for the risk,then picking the smallest value of λ such that the upper confidence bound falls below α .
We give
Hoeffding’s UCB below, although we use the stronger Hoeffding-Bentkus bound from [8] in practice:
[Equation]
Note that in our setting, we can always generate enough samples to drive the last term to 0 for anyδ ; however, if we only had a finite sample from some population, this would not be the case.
Wecan then select ˆ λ by scanning from large to small values, ˆ λ = min (cid:110) λ : ˆ R + ( λ ′ ) ≤ α, ∀ α ′ ≥
α (cid:111) , orrunning binary search.
Proposition 2.2 ( T ˆ λ is an RCPS [8]) .
With ˆ λ selected as above, T ˆ λ satisfies Definition 2.1.
For the proof of this fact, along with a discussion the tighter confidence bounds used in our experiments and extensions to the underlying theory, see [8] and [3].
Having proven that T ˆ λ is an RCPS, we can simply set T ( X ) =
T ˆ λ
( X ) in (2); in other words, we setq cal α 2 ( X ) d
= f ( X ) d − ˆ λ (cid:0) f
( X ) d − q α 2 ( X ) d (cid:1) + and q cal1 − α 2 ( X ) d = f ( X ) d + ˆ λ (cid:0) q
[Equation]
− α
[Equation]
Visualizing uncertainty intervals in image space
We briefly describe our method for visualizing latent-space uncertainty intervals.
In order to see theeffect of a single semantic factor, we set it to either the lower or upper quantile and hold the otherfactors fixed to the point prediction.
More specifically, for a particular dimension d ∈ { 1 , ..., D } ,define the following vector:
[Equation]
We visualize the lower and upper quantiles in image space as G ( ˆ Z d α 2 ) and G ( ˆ Z d 1 − α 2 ) respectively.
Since each semantic factor corresponds to an attribute, visualizing the lower and upper quantilesper-factor gives interpretable meaning to the latent-space intervals.
For example, in Figure 1, theimages of the child smiling give a range of possible expressions the model believes are consistentwith the underlying image.
Experiments
3.1 Dataset descriptions
FFHQ We use the StyleGAN framework pretrained using the Flickr-Faces-HQ (FFHQ) dataset [24].
FFHQ is a publicly available dataset consisting of 70,000 high-quality images at 1024 × 1024resolution.
The data used to train the quantile encoder and for the experiments in this section issampled from the generator pretrained on FFHQ.
CelebA-HQ We use the CelebA-HQ dataset [22] to demonstrate the effectiveness of our approachon real world data.
The dataset contains 30,000 high-quality images of celebrity faces.
Since the realdataset is used only for evaluation, we use images from standard test split.
CLEVR.
In order to have a controlled setup where we can easily identify disentangled factors ofvariation, we generate synthetic images of objects based on the CLEVR dataset [21].
This datasetprovides a programmatic way of generating synthetic data by explicitly varying specific semanticfactors.
We create a synthetic dataset by varying { color, shape } and fixing the other factors such aslighting, material and camera jitter.
3.2 Experimental setup
Model architectures In all our experiments, we use the StyleGAN2 [23] framework for the generatorarchitecture G .
For the experiments involving faces, we use the pretrained model available from [Figure]
The corrupt image is provided as input to the encoder which outputs a pointwise prediction and quantile predictions for each style dimension
We plot the calibrated and uncalibrated as well as their visualizations in image-space.
[Best viewed in color, zoom in for detail.]
the official repository. For the CLEVR-2D experiments, we train a simpler variant of StyleGAN2generator from scratch.
For the quantile regression, we use a standard architecture: the encodernetwork consists of a ResNet-50 backbone [18] with the final layer branching into the point predictionand conditional quantiles.
The branching module is a standard combination of convolution andactivation blocks followed by a fully connected layer of the expected output dimension; we call thesethe heads of the model.
Specific details of the model architecture are provided in the supplementarymaterial.
Model training We start by pretraining the generative model or acquiring an off-the-shelf pretrainedgenerative model for the task at hand.
In generative models such as StyleGAN, the style space thatoffers fine grained control over image attributes, is very high dimensional.
From this high dimensionalspace, we extract only the disentangled dimensions following previous work on style space analysis[41].
In order to better focus the encoder’s capacity only on the disentangled dimensions, we maskout the irrelevant dimensions for applying the quantile loss.
However, the pointwise loss in (6) isapplied to the full style vector to ensure that the pointwise prediction is able to match the true latentsaccurately, while the quantile heads focus on learning variablity only in the disentangled dimensions.
During the encoder training 2.2, the generative model G is held frozen and only the parameters ofthe encoder E are updated.
The point prediction and conditional quantile heads are trained jointlywith the Ranger optimizer [41] and a flat learning rate of 0.001 for all our experiments.
For the image super-resolution training, we augment the input dataset by using different levels ofdownsampled inputs, i.e. , we take the raw input and apply a random downsampling factor from{ 1 , 4 , 8 , 16 , 32 } and resize it to the original dimensions.
For the image inpainting task, we vary thedifficulty by choosing a random threshold to create the mask – lower thresholds implies fewer pixelsare masked and vice-versa.
The mask is concatenated to the image resulting in a C + 1 -channelinput to the encoder, where C is the number of image channels.
The detailed description of the maskgeneration procedure can be found in the supplementary material.
Calibration and Evaluation For both the synthetic object experiments with CLEVR and faceexperiments with FFHQ, we train the quantile encoder on data points sampled from the latent spaceof the pretrained generative model.
This ensures that we have access to the true latents that resulted ineach image.
We generate 100k samples per model and generate a random 80-10-10 split for training,calibration and validation.
3.3 Findings
In the following experiments, we explore different properties of our intervals.
The problem typesinclude image super-resolution and image inpainting.
The risk level α and the user-specified errorthreshold δ are fixed to 0.1, unless specified otherwise.
[Figure]
Goal.
We qualitatively verify that the proposed approach outlined in Section 2 produces visuallymeaningful uncertainties.
Description.
We train a quantile encoder with Algorithm 1.
Then we generate n = 5000 images bysampling latents and propagating them through the encoder-generator combination.
We use these nimages as calibration data for the procedure in 2.3.
Finally, we randomly sample a new test point, passit through the calibrated encoder, and form the uncertainty intervals in image space as in Section 2.3.
Results.
The results are illustrated in Figure 3 on images sampled from the generator trained on
FFHQ (left) and CLEVR (right).
In case of the FFHQ image, the person is wearing glasses in thelower quantile image and not in the upper; hence, the model is not certain that the person is wearingglasses.
The model also expresses some uncertainty about the amount of gray versus brown hair.
Thisoutcome was predictable from the input image, where the fact that the person is wearing glasses isnot obvious, and there is some hair color ambiguity.
The results on the CLEVR dataset are analogous.
The lower and upper quantile images yield similar colors, which is predictable from the blurry input.
The model predicts that both a cylinder and sphere would be consistent with this blurry input.
Thecalibrated quantiles cover the ground truth color value, while the uncalibrated ones do not. 3.3.2 Experiments with real data
Goal.
We calibrate our encoder with real data sampled from the Celeb-A face dataset for the imageresolution task.
The objective is to test to what extent we can extract meaningful uncertainty intervalsfor real data without explicitly training with them.
Description The difficulty with calibrating using real data is that the targets i.e the true style vectorsthat generated the real images are not available to us.
To circumvent that difficulty, we back-projectthe real images into the latent space of the GAN using existing approaches to GAN inversion [23, 1].
Even though the mapping from the GAN latent space to the output is not one-one, the back-projectionoptimization procedure is able to visually match the appearance of the real images convincingly.
Thisprocedure provides us with the estimated "true" style vectors for each image in the real calibration set. We combine this with the generated data and perform the RCPS calibration procedure as described in [Figure] factor effect of calibration before after
[Figure]
Observe that in some cases, such as smile and glasses , there are clear visible changes due to the calibration approach.
In other cases such as hair , the changes are not visually explicit.
More such examples can be found in the supplementary material.
[Best viewed in color.
Zoom in for detail]
Section 2.3.
Note that, our encoder was only trained on generated data i.e. the same model as used in
Section 3.3.1, we use the real data only for the calibration procedure.
Qualitative results The predicted quantiles and the effect of the calibration procedure for a realtest image drawn from the Celeb-A dataset is visualized in Figure 5.
The pointwise prediction ofthe encoder does not match the identity of the true image due to the high level of input corruption.
However, using our approach, we are able to visualize the predicted uncertainty intervals in ameaningful manner.
The visual effect of our calibration procedure is demonstrated on the right.
Notice that some attributes such as smile and glasses have a visible change but others do not.
Thechange is due to the fact that the calibration procedure adjusts the intervals in such a way to satisfythe required coverage guarantee.
3.3.3 Exploratory results with purposeful corruptions
Goal.
We probe our procedure to see if it will have the expected qualitative behavior.
Description of experiment.
We sampled images from the held out set of the FFHQ pretrained GANand applied purposeful corruptions to check if the resulting quantile estimates had semantic meaning.
Both image resolution and image masking were used as corruption models.
We qualitatively analyzethe results by visualizing the predictions. We include a quantitative measurement of variabilityestimated using image based metrics in the supplementary material.
Qualitative results.
Figure 4 shows the results of this experiment for Image inpainting (left) andsuper-resolution (right).
In the inpainting case, when nothing is masked, the quantiles are roughlyidentical.
When the eyes are masked, the quantiles indicate the model does not know if the personwas wearing glasses.
When the mouth is masked, the model expresses uncertainty as to whether thewoman is showing her teeth in the smile.
Finally, when almost everything is masked, the quantileimages are very different, representing individuals with entirely different identities.
Similar behaviorcan be observed in the super-resolution case.
The results are shown for two separate inputs; in bothcases, the input in the top row is uncorrupted while in the bottom row, it is undersampled by 16x.
The model is able to predict almost perfectly in the absence of corruption.
The quantile predictionsare extremely close as well.
In the presence of corruption, both the pointwise prediction is off (as expected) and the quantile edges display much higher variability including in attributes such as hairshape, glasses, facial hair and perceived gender.
The results from both these experiments point to anexpected qualitative behavior of our proposed approach: the model exhibits more uncertainty withincreased information loss at the input.
Related Work
GANs for Inverse problems.
The remarkable image generation properties of recent approachessuch as BigGAN [11] and StyleGAN [23] has led to the increasing use of these models to solveinverse problems relating to image restoration such as image super-resolution and completion.
Allprior methods that use GANs for inverse problems such as the ones that use the pretrained generativemodel as an image prior [45, 9, 37, 35] or the ones that train an encoder model to project the inputinto the generator’s latent space [49, 41, 44, 40] focus on the accuracy of the point estimate but noton the uncertainty level of the input.
GANs for Interpretability.
Despite providing no guarantees of image likelihood, unlike others suchas Normalized Flow [25] or Score-based models [50], GANs have been used to develop interpretableapproaches to image generation.
The widespread use of GANs as opposed to other generative modelsin the interpretability is done due to the availability of an disentangled latent space [17, 48], which isa property we utilize in our work.
Quantile Regression.
Quantile regression was first proposed in [29].
Since then, many papers haveused the technique, applying it to machine learning [20, 36, 38], medical research [7], and more.
Most relevant to us is conformalized quantile regression [42], which gives quantile regression amarginal coverage guarantee using conformal prediction.
Our work instead uses risk-controllingprediction sets, a different distribution-free uncertainty quantification technique.
Conformal prediction and distribution-free uncertainty quantification.
At the core of ourproposed method is the distribution-free, marginal risk-control technique studied in [8] and [3].
These ideas have their roots in the distribution-free marginal guarantees of conformal prediction,proposed in [47].
Conformal prediction is a flexible method for forming prediction sets that satisfya marginal coverage guarantee, under no assumptions besides the exchangeability of the test pointwith the calibration data [47, 46, 33, 34, 32, 2].
Conformal prediction has been studied in computervision [19, 12, 6, 43, 4], natural language [14], drug discovery [15], criminal justice [10], and more.
We are not aware of work applying the notions of conformal prediction and quantile regression to thelatent space in generative models.
Conclusion
Experiments indicate that with an appropriately disentangled model, latent space uncertainty intervalsexpress a useful semantic notion of uncertainty previously unavailable in computer vision.
Limitationsof our method include that the calibration data must be reflective of the data distribution, and thatwe assume access to a disentangled latent space.
We see our work as part of a larger tapestry ofresults in generative models, and our technique will remain applicable as progress gets made ondisentanglement and backprojection.
Ethics
The ethics of generative modeling itself has been called into question given recent events, e.g. , thedevelopment of deep fakes.
Nonetheless, we believe the downstream consequences of this work willlikely be positive.
The techniques herein do not change the predictions of a generative model; theysimply provide a calibrated notion of its uncertainty in a relevant semantic space.
Thus, the standardcriticism of generative modeling—that it will enable widespread deep fakes—is not applicable.
Furthermore, we expect having a statistically valid and semantically rich notion of uncertainty willprovide a sobering reliability assessment of these models, perhaps mitigating the chance of harmfulfailures.
Finally, although we use face datasets due to their ubiquity in this literature, we haveattempted to ethically treat topics like gender and race where they arise. Checklist For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
(b) Did you describe the limitations of your work?
[Yes] We explicitly state the assumptions
in Section
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Explicitly
address this in Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] Explicitly address this in Section
If you are including
theoretical
results...
a)
Did you state the full set of assumptions of all theoretical results?
[Yes] The propositionis stated in Section 2.3.
We also state the assumptions in Section 5.
For more details,we redirect the reader to previous work where we derive the framework from.
( [8]and [3])
(b)
Did you include complete proofs of all theoretical results?
[No] The proposition isstated in Section 2.3.
For complete proofs, we redirect the reader to previous workwhere we derive the framework from.
( [8] and [3])3.
If you ran experiments...
(a)
Did you
include the code
data, and instructions
needed
to
reproduce
the
main experimental results (either in the supplemental material or as a URL)?
Yes
] Training detailsare provided in Section 3.
More complete details about the model architecture areprovided in the supplementary material.
The datasets we use are publicly available, weinclude references to that in Section 3.
The code will be released in the near future.
(b) Did you specify all the training details (e
g.
data splits
hyperparameters, how they
were chosen)?
[Yes
(c) Did you report error bars (e
g., with respect to the random seed after running exper
-iments
multiple times)?
[No] While
we expect there to be some randomness, thebehavior of the encoder is consistent across
different runs.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[No]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No] We cite the dataset creators who have
included their respective licenses along with the dataset.
We ensure that we follow the
terms of the license i.e. use the data for research use only.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No] Since we use a dataset that comes with a reseach license,
we
defer such concerns to the dataset creators.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No] Since we use a dataset that comes with a reseachlicense, we defer such concerns to the dataset creators.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]