CHIMLE: Conditional Hierarchical IMLE for Multimodal Conditional Image Synthesis
Abstract
A persistent challenge in conditional image synthesis has been to generate diverse output images from the same input image despite only one output image being observed per input image.
GAN-based methods are prone to mode collapse, which leads to low diversity.
To get around this, we leverage Implicit Maximum Likelihood Estimation (IMLE) which can overcome mode collapse fundamentally.
IMLE uses the same generator as GANs but trains it with a different, non-adversarial objective which ensures each observed image has a generated sample nearby.
Unfortunately, to generate high-ﬁdelity images, prior IMLE-based methods require a large number of samples, which is expensive.
In this paper, we propose a new method to get around this limitation, which we dub Conditional Hierarchical IMLE (CHIMLE), which can generate high-ﬁdelity images without requiring many samples.
We show CHIMLE signiﬁcantly outperforms the prior best IMLE, GAN and diffusion-based methods in terms of image ﬁdelity and mode coverage across four tasks, namely night-to-day, 16 ⇥ single image super-resolution, image colourization and image decompression.
Quantitatively, our method improves Fréchet Inception Distance (FID) by 36.9% on average compared to the prior best IMLE-based method, and by 27.5% on average compared to the best non-IMLE-based generalpurpose methods.
More results and code are available on the project website at https://niopeng.github.io/CHIMLE/ .
Introduction
Impressive advances in image synthesis have been made by generative models [9, 40, 83, 15, 78, 31, 38].
Generative models are probabilistic models; in the context of image synthesis, they aim to learn the probability distribution over natural images from examples of natural images.
Whereas unconditional generative models learn the marginal distribution over images, conditional generative models learn the conditional distribution over images given a conditioning input, such as a class label, textual description or image.
Conditional generative modelling is more challenging than unconditional generative modelling, due to the need to ensure consistency between the conditioning input and the generated output while still ensuring image ﬁdelity and diversity.
As the conditioning input becomes more speciﬁc, there are fewer images in the training set that are consistent with the conditioning input, resulting in weaker supervision.
So, in the order of decreasing supervision are class, text and image-conditional generative models – in class conditioning, many images correspond to each class; in text conditioning, multiple images can at times share the same textual description; in image conditioning (e.g., a grayscale image as the input and a colour image as the output), typically only one output image that corresponds to each input image is observed (e.g., only one way to colour a grayscale image is observed).
We 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
will focus on the most challenging of these settings, namely image conditioning.
Regardless of the supervision, the goal of conditional generative models is to learn the full distribution over all possible images consistent with the input.
Attaining this goal is challenging, especially when the supervision is weak, and so popular existing methods such as variational autoencoders (VAEs) [45, 70] and generative adversarial nets (GANs) [26, 27] focus on generating a point estimate of the conditional distribution, such as the mean (VAE) or an arbitrary mode (GAN).
Such methods are known as unimodal methods , whereas methods that aim to generate samples from the full distribution are known as multimodal methods , since they enable samples to be drawn from as many modes as there exist.
Extending popular unconditional generative models such as VAEs, GANs and diffusion models to the image-conditional multimodal setting has proven to be challenging.
VAEs suffer from posterior collapse [18, 57] where the encoder ignores the conditional input, leading the decoder to produce the conditional mean, which typically corresponds to a blurry image.
GANs suffer from mode collapse [37, 105], where the generator tends to ignore the latent noise, leading the generator to produce the same mode of the conditional distribution for the same input regardless of the latent noise.
As a result, other equally valid modes (e.g., other possible ways to colour a grayscale image) cannot be generated.
Recent and concurrent work [42, 6] extends diffusion models to this setting, but they can produce both too little and too much diversity by creating spurious modes and dropping modes.
We dub this problem mode forcing and include an analysis of its cause in the appendix.
A recent method [53] takes a different approach by extending an alternative generative modelling technique known as Implicit Maximum Likelihood Estimation (IMLE) [52] to the conditional synthesis setting.
At a high level, IMLE uses a generator like GANs, but instead of using an adversarial objective which makes each generated sample similar to some observed image , IMLE uses a different objective that ensures each observed image has some similar generated samples .
Hence, the generator can cover the whole data distribution without dropping modes.
We refer interested readers to [53, 52] for more details on the algorithm.
However, as shown in the top row of Figure 1, the output image from cIMLE [53] lacks ﬁne details.
For cIMLE to generate high quality samples, there must a sample out of m samples generated during training that is similar to the observed image.
This necessitates a large m ; unfortunately, generating a large number of samples is computationally expensive.
So we are faced with a tradeoff: do improvements in sample quality have to come at the expense of sample efﬁciency?
In this paper, we show that there is a way around this conundrum.
The idea is to generate samples in a clever way such that the best sample is about as similar to the observed image as if a large number of samples had been generated, without actually generating that many samples.
We propose three ideas: partitioning of the latent code, partial evaluation of latent code components and iterative construction of latent code as shown in the bottom row of Figure 1.
These three ideas give rise to a novel method known as Conditional Hierarchical IMLE (CHIMLE).
We demonstrate that CHIMLE signiﬁcantly outperforms the prior best IMLE-based method [53] in terms of both ﬁdelity and diversity across four challenging tasks, namely night-to-day, 16 ⇥ single image super-resolution, image colourization and image decompression.
Moreover, we show that CHIMLE achieves the state-of-the-art results in image ﬁdelity and mode coverage compared to leading general-purpose multimodal and task-speciﬁc methods, including both GAN-based and diffusion-based methods.
Method 2.1 Preliminaries: Conditional Implicit Maximum Likelihood Estimation (cIMLE)
In ordinary unimodal synthesis, the model is a function f ✓ parameterized by ✓ that maps the input to the generated output.
To support multimodal synthesis, one can add a latent random variable as an input, so now f ✓ takes in both the input x and a latent code z drawn from a standard Gaussian N (0 , I ) and produces an image b y as output.
To train such a network, we can use a conditional GAN (cGAN), which adds a discriminator network that tries to tell apart the observed image y and the generated output b y .
The generator is trained to make its output
b
y seem as real as possible to the discriminator.
Unfortunately, after training, f ✓ ( x , z ) often produces the same output for all values of z because of mode collapse.
Intuitively, this happens because making b y as real as possible would push it towards the observed image y , so the generator tries to make its output similar to the observed image y for all values of z .
As a result, naïvely applying cGANs to the problem of multimodal synthesis is difﬁcult.
Conditional IMLE (cIMLE) [53] proposes an alternative technique for training the generator network f ✓ .
Rather than trying to make all outputs generated from different values of z similar to the observed image y , it only tries to make some of them similar to the observed image y .
The generator is therefore only encouraged to map one value of z to the observed image y , allowing other values of z map to other reasonable outputs that are not observed.
This makes it possible to perform multimodal synthesis.
Also, unlike cGANs, cIMLE does not use a discriminator and therefore obviates adversarial training, making training more stable.
The cIMLE training objective is:
[Equation]
where d ( · , · ) is a distance metric, m is a hyperparameter, and x ( i ) and y ( i ) are the i th input and observed image in the dataset.
During each iteration of the training loop, a pool of m samples are generated from f ✓ for each input x ( i ) .
The cIMLE algorithm then selects the closest sample for each observed image
y
( i ) to optimizes ✓ .
2.2 Conditional Hierarchical IMLE (CHIMLE)
In order to generate high-quality images using cIMLE, we need some of the m generated samples to be similar to the observed image y during training.
Achieving this requires a large m , but generating samples is expensive.
This forces a tradeoff between the number of samples and the mode modelling accuracy, which is less than ideal.
The key observation is that the gradient of the loss depends solely on the closest sample for each observed image, as shown in Eqn. 1 – all other samples that are not selected do not contribute.
Therefore, if we could efﬁciently search for a latent code z that would produce a sample of a comparable degree of similarity to y as one that would be selected from a larger pool
z ⇤ , we could achieve high sample quality without actually generating that many samples.
To solve search problems, a time-tested paradigm in computer science is divide-and-conquer, which entails dividing the problem into simpler sub-problems, conquering each sub-problem and combining the solutions from the solved sub-problem.
In the context of searching for a latent code z that simulates the quality of z ⇤ , we propose dividing the latent code into components, evaluating each latent code component and combining the latent code components iteratively.
In the ensuing discussion, we will consider a common paradigm of image synthesis models [68, 37, 13, 39], where the model processes the image at L different scales.
We can feed different dimensions of the latent code as inputs to different layers operating at different resolutions, thereby controlling the variations of content at different resolutions.
Division of Latent Code To reduce the difﬁculty of the original problem, we can divide it into sub-problems.
Each of these sub-problems should be a simpler instance of the original problem.
We propose dividing the dimensions of the latent code z 2 R n into L different components S
= { z i } Li =1 , where z
i 2 R k i and
P
Li =
1 k i = n , which amounts to dividing the latent space into mutually orthogonal subspaces.
Each component z i corresponds to the dimensions that serve as input to layers operating at a particular resolution.
Partial Evaluation of Latent Code Components To determine how promising a latent code component is, we need a way to evaluate a partially constructed latent code, with only some components, indexed by S r ⇢ S , being realized with concrete values.
We propose using a partial output ˜ y p that depends only on the currently constructed components z S r , and evaluating ˜ y p against a part of the observed image y p that only contains details at resolutions modelled by z S r .
So instead of computing d (˜ y , y ) , we compute d (˜ y p , y p ) .
By doing so, we can evaluate the quality of z S r more accurately, since ˜ y p does not depend on the unconstructed components
z S \ S r .
To realize this, we add output heads at the L different resolutions that the model operates on to produce the partial outputs ˜
y p and downsample the observed image y to the corresponding resolutions to produce the partial observed images y p .
The values of the partially realized components that result in the greatest similarity to y p is the ﬁnal selected component, i.e., z
⇤ p = arg min d (˜ y p , y p ) .
gorithm for a Single Data Point
Require: One input image at L increasing resolutions { x l } Ll =1 , the set of partial observed images { y l } Ll =1 at corresponding resolutions and the gener- ator f ✓ ( · , · ) that produces L outputs at correspond-
ing resolutions for l = 1 to L do
Randomly generate m i.i.d.
latent codes for the l th component z (1) l , . .
, z ( m )
l ˜ y
( j ) l l th output from f ✓ (( x 1 , . . .
, x l ) , ( z ⇤ 1 , . . .
, z ⇤ l  1 , z ( j ) l )) 8 j{ 1 , . . .
, m } z ⇤ l arg min j d ( y l ,
˜ y ( j ) l ) 8 j 2 { 1 , . .
. , m } end for return { z ⇤ l } Ll =
Iterative Combination of Latent Code Components To maximize efﬁciency, the solution to one sub-problem should reveal some information on which other sub-problems to solve.
So, we need to decide on which sub-problems to solve ﬁrst and how to use their solutions to decide on which sub-problem to solve next.
We leverage the fact that solving for the appearance at lower resolutions (e.g., ﬁnding out that the coarse structure of an apple is red) can inform on the probable appearances
Putting everything together we obtain the Conditional Hierarchical IMLE (CHIMLE) algorithm, which is detailed in Alg. 1 for the special case of one input image.
We validate the effectiveness of
CHIMLE by comparing the number of generated samples CHIMLE needs to obtain the same level of similarity to the observed image as cIMLE.
As shown in Figure 2, CHIMLE consistently requires fewer samples than cIMLE, demonstrating a signiﬁcant improvement in sample efﬁciency.
2.3 Model Architecture
As shown in Figure 3, our architecture contains a sequence of modules, each of which handles an input image of a particular resolution and outputs an image whose resolution is doubled along each side.
We downsample the input image repeatedly by half to obtain input images at different resolutions, which are fed into different modules.
As mentioned in Sect.
2.2, we divide the dimensions of the latent code into components and feed each to a different module.
Note that this architecture generalizes to varying levels of output resolution, since we can simply add more modules for high-resolution outputs.
We add intermediate output heads to each module and add supervision to the output of each module.
We choose LPIPS [101] as the distance metric used in the IMLE objective function.
For the design of each module, we follow the paradigm of prior work on modelling global and local image features [59, 10, 84, 58, 74] and design a module that comprises of two branches, a local processing branch and a global processing branch.
The local processing branch takes in the conditional image input, a latent code with the matching resolution as the image and the output from the previous module if applicable.
The global processing branch takes in a ﬁxed-sized latent code input and produces scaling factors and offsets for the different channels of the intermediate features in the local processing branch.
For more implementation details, please refer to the appendix.
Due to the hierarchical nature of the architecture and its role as the implicit model in IMLE (also known as the “generator” in GAN parlance), we dub this architecture the Tower Implicit Model (TIM).
Experiments
Baselines To validate our main contribution of this paper, namely improving the output ﬁdelity of IMLE-based methods, we compare to the leading IMLE-based method, cIMLE
As a
[Figure]
cIMLE
[Figure]
secondary comparison, we demonstrate potential impact in a broader context and compare our method to the leading multimodal general-purpose conditional image synthesis methods according to a recent survey [65], including four GAN-based methods, such as BicyleGAN [105], MSGAN [60],
DivCo [56] and MoNCE [95], and two diffusion-based methods, such as DDRM [42] and NDM [6].
As a tertiary comparison, we also compare our method to popular task-speciﬁc methods based on leaderboard rankings in challenges and recent survey papers [96, 3].
Tasks We apply our method to four different conditional image synthesis tasks, namely night-today, 16 ⇥ single image super-resolution, image colourization and image decompression.
We pick night-to-day because it is a classic task in the multimodal general-purpose conditional image synthesis literature [37, 105].
We pick the remaining tasks because they are more challenging than typical tasks considered in the literature as evidenced by the availability of task-speciﬁc methods tailored to each.
Training Details We use a four-module TIM model (described in Sect. 2.3) for all tasks.
The input to each module is downsampled from the full resolution input to resolution the module operates on.
We trained all models for 150 epochs with a mini-batch size of 1 using the Adam optimizer [44] on an NVIDIA V100 GPU.
Details for datasets are included in the appendix.
Evaluation Metrics We evaluate the visual ﬁdelity of generated output images using the Fréchet
Inception Distance (FID) [30] and Kernel Inception Distance (KID) [7].
We also measure the diversity of the generated output images since it is a crucial objective of multimodal conditional image synthesis.
One metric to measure diversity is the LPIPS diversity score (not to be confused with the
LPIPS distance metric) [105], which is the average LPIPS distance between different output samples for the same input.
Unfortunately it has limitations, because it may favour a set of samples where some are of poor ﬁdelity as shown in Figure 5.
To get around this, we use faithfulness-weighted variance (FwV)
[53] instead, which is the average LPIPS distance d LPIPS between the output samples and the mean, weighted by the consistency with the original image measured by a Gaussian kernel 1 :
[Equation]
where ˜ y ( i,j ) is the i th generated sample from the j th input.
¯ y ( j ) is the mean of all generated samples from the j th input.
y ( j ) is the j th observed image.
The kernel bandwidth parameter  trades off the importance of consistency vs. diversity.
We measure how well the modes in the model distribution and the true distribution match along two axes, mode accuracy (precision) and mode coverage (recall).
We measure the former with the
F 1 / 8 score component of the PRD metric [71], which is heavily weighted towards precision, and the latter with the F 8 score component of the PRD metric [71], which is heavily weighted towards recall, and the inference-via-optimization (IvO) metric [61], which measures the ﬁnal mean squared error (MSE) between the the observed image and the best output image that can be generated (i.e., where the latent code is optimized w.r.t.
MSE while keeping model weights ﬁxed).
In the conditional image synthesis setting, IvO is a more sensitive metric than the F 8 score in PRD, since it would detect cases where the output samples for a given input are not diverse but the output samples for different inputs are diverse in aggregate.
In other words, IvO measures mode coverage in the conditional distribution, whereas the F 8 score measures mode coverage in the marginal distribution.
In light of the large number of noise samples in NDM (2 for each of the 1000 diffusion steps) which all need to be optimized in calculating IvO, we omit it due to the lack of sufﬁcient computational resources.
3.1 Quantitative Results
We compare the perceptual quality and output diversity in Table 1.
As shown, CHIMLE outperforms best prior IMLE-based method, cIMLE, general-purpose conditional image synthesis methods and task-speciﬁc methods in terms of FID and KID across all tasks.
CHIMLE also outperforms all multimodal baselines in terms of FwV across all tasks.
These results show that CHIMLE can produce more realistic and diverse images than the baselines, thereby setting a new state-of-the-art.
We compare the mode accuracy and coverage in Table 2.
As shown, CHIMLE outperforms or achieves comparable performance to cIMLE and other multimodal general-purpose baselines in [Table] terms of PRD scores.
In addition, CHIMLE signiﬁcantly outperforms cIMLE and other multimodal general-purpose baselines in terms of IvO. These results show that CHIMLE achieves better mode accuracy and coverage of the true distribution compared to the baselines.
3.2 Qualitative Results
We show the qualitative comparison of our method (CHIMLE) and selected baselines on various tasks in Figure 4.
As shown, CHIMLE generates high-quality results, like the high contrasts in the building and the bushes in night-to-day ( 1 st row), the sharp appearance of the wing pattern of the butterﬂy wing and the ﬂowers in the background ( 2 nd row), the vibrant yet detailed colouring of the parrot ( 3 rd row) and the high-frequency details in the sculpture ( 4 th row).
For all methods, we show videos of different samples generated from the same input on the project website and in the supplementary materials.
Due to mode collapse, GAN-based baselines incorporate regularizers that improve the diversity of samples at the expense of their ﬁdelity, resulting in more varied but less realistic samples as shown in the videos.
Diffusion-based baselines suffer from mode forcing (see appendix D for more details) and generate spurious samples that are either excessively diverse or with little diversity.
Prior best IMLE-based method (cIMLE) avoids spurious samples but falls short in image ﬁdelity.
In contrast, CHIMLE generates samples that are both diverse and of high ﬁdelity.
Figure 6 shows the qualitative comparison of the best reconstruction of the observed image found by different methods using IvO [61], which tries to approximately invert each model by optimizing over the latent code input to ﬁnd a generated image closest to the observed image.
As shown, the reconstruction of the CHIMLE model is the closest to the observed image and achieves almost pixel-level accuracy.
In addition, we found that for our model, IvO can consistently converge to a high-quality sample, whereas it would diverge quite often for the baselines, suggesting that inverting our model is much easier than the baselines.
This demonstrates that CHIMLE successfully covers the mode corresponding to the observed image.
3.3 Ablation Study
We incrementally remove (1) iterative combination (IC), (2) partial evaluation (PE), (3) latent code division (CD).
As shown in Figure 7, each is critical to achieving the best results.
Notably, we see an average increase in terms of FID by 7 . 7% after removing IC, by another 7 . 8% after removing PE and by another 12 . 8% after removing CD, thereby validating the importance of each of our contributions.
[Table]
[Table]
[Figure]
covered by CHIMLE.
Related Work
General-Purpose Conditional Image Synthesis Conditional image synthesis generate images from a conditioning input.
The input can come in various forms, such as a class label [62, 64], a textual description [69], or an image [37].
The latter is sometimes known as image-to-image translation, which is the setting we consider.
General image-conditional methods can be paired or unpaired.
In the paired setting, each image in the source domain corresponds to an image in the target domain [37, 93, 86, 75].
In the unpaired setting, images in the source domain may not correspond to images in the target domain [104, 92, 81, 55, 16, 8, 67].
Whereas these previous methods are all unimodal, other methods aim for the more challenging task of multimodal synthesis in the paired setting [105, 24, 22] and in the unpaired setting [34, 50, 51, 17].
Since many of the aforementioned works use GANs as their generators which suffer from mode collapse, some works aim to address this issue by introducing mode seeking terms [60] or adding contrastive losses [56, 95, 33].
Another
Full Model
No IC
No IC, PE
No IC, PE, CD
SR
DC
Full Model 16 .73 .
No IC 17 .81 .
No IC, PE 18 .87 .
No IC, PE, CD 21 .101 .
comparison as we gradually remove iterative latent code division (CD) on two tasks, 16 ⇥ super-resolution (SR) (ﬁrst row in the
and image decompression (DC) (second row in the
As shown in the , the generated image contains less ﬁne details (as in SR, where the strawberry seeds become less pronounced) or colour saturation (as in DC) as we remove each component, which is also reﬂected by the FID shown in the table.
line of work uses diffusion models [42, 6].
These methods can both generate spurious modes and drop modes (due to the problem of mode forcing , which we analyze in appendix D), and need to devote part of the model capacity to reconstruct the information encoded in the conditioning input from pure noise and retain it over the course of diffusion.
In this paper, we focus on the paired image-conditional multimodal setting, and sidestep mode collapse by building on a recent generative modelling technique, conditional IMLE [53].
Task-Speciﬁc Approaches There is a large body of work on super-resolution, most of which consider upscaling factors of 2  4 ⇥ .
See [91, 63, 89] for comprehensive surveys.
Many methods regress to the high-resolution image directly and differ widely in the architecture [23, 43, 47, 80, 28, 102, 19, 54].
Various conditional GANs have also been developed for the problem [49, 72, 88, 94, 66, 90, 87, 76].
For image colourization, a recent survey [3] provides a detailed overview of different methods.
Some methods [14, 12, 100, 11] use deep networks to directly predict the colour and combine it with the input to produce the output.
Other methods [36, 48, 21, 103, 79] take advantage of features from different levels or branches in the network for colour prediction.
There is relatively little work on image decompression to our knowledge.
[4] targets DCT-based compression methods and explicitly models the quantization errors of DCT coefﬁcients.
More work was done on learned image compression [2, 1], which changes the encoding of the compressed image itself.
Another related area is image denoising, see [82] for a survey of deep learning methods.
Many of the methods uses a ResNet [29] backbone or other variants of deep CNNs [97, 98, 99, 85].
Discussion and Conclusion
Limitation Our method can only generate output images that are similar to some training images.
Take colourization as an example – if there were no green apple in the training data, then our model would not be able to generate green apples.
Societal Impact
The TIM architecture adopts a modular design, its capacity could be expanded by adding more modules.
While it may produce more reﬁned results, training such a huge model may result in increased greenhouse gas emissions if the electricity used is generated from fossil fuels.
Conclusion In this paper, we developed an improved method for the challenging problem of multimodal conditional image synthesis based on IMLE.
We addressed the undesirable tradeoff between sample efﬁciency and sample quality of the prior best IMLE-based method by introducing a novel hierarchical algorithm.
We demonstrated on a wide range of tasks that the proposed method achieves the state-of-the-art results in image ﬁdelity without compromising diversity or mode coverage.
Acknowledgements This research was supported by NSERC, WestGrid and the Digital Research
Alliance of Canada.
We thank Tristan Engst, Yanshu Zhang, Saurabh Mishra, Mehran Aghabozorgi and Shuman Peng for helpful comments and suggestions, and Kuan-Chieh Wang, Devin Guillory,
Tianhao Zhang and Jason Lawrence for feedback on an early draft of this paper.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 5 for details.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Section 5 for details.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] No theoretical results.
(b) Did you include complete proofs of all theoretical results?
[N/A] No theoretical results.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See the appendix and the project website.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] Each experiment is computationally expensive, but we during the course of developing the method, we found it to be quite robust to different seeds.
We believe that this is because our method optimizes a non-adversarial objective, which makes the training stable.
We demonstrate the training stability exhibited by our method in Sect.
C of the appendix.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 3 for details.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] Existing assets are cited.
(b) Did you mention the license of the assets?
[N/A] Assets used in this research carry no license/access restrictions.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
No new assets.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A]
No new assets.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A]
No personally identiﬁable information in the data.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] No crowdsourcing.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] No crowdsourcing.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A] No crowdsourcing.