CHIMLE: Conditional Hierarchical IMLE for Multimodal Conditional Image Synthesis
Abstract
A persistent challenge in conditional image synthesis has been generating diverse output images from the same input image due to the problem of mode collapse.
Implicit Maximum Likelihood Estimation (IMLE) is a recently proposed alternative that aims to address this issue.
IMLE uses the same generator as GANs but adopts a different objective function which ensures each observed image has a generated sample nearby.
To generate high-ﬁdelity images, prior IMLE-based methods require a large number of samples.
Doing so is expensive, and so this limits image ﬁdelity in practice.
In this paper, we propose a new method to get around this limitation, which we dub Conditional Hierarchical IMLE (CHIMLE), which can generate high-ﬁdelity images without requiring many samples.
We show on fours tasks, namely night-to-day, 16 ⇥ single image super-resolution, image colourization and image decompression, that CHIMLE signiﬁcantly improves generated image ﬁdelity, as demonstrated by a reduction in Fréchet Inception Distance (FID) by 36.9% on average compared to the prior best IMLE-based method, and by 27.5% on average over the best non-IMLE-based general-purpose methods. We also show that CHIMLE outperforms GAN-based and diffusion-based methods in image ﬁdelity while better preserving modes in the data. More results and code are available on the project website at https://niopeng.github.io/CHIMLE/ .
Introduction
Impressive advances in image synthesis have been made by generative models.
Generative models are probabilistic models; in the context of image synthesis, they aim to learn the probability distribution over natural images from examples of natural images.
Whereas unconditional generative models learn the marginal distribution over images, conditional generative models learn the conditional distribution over images given a conditioning input, such as a class label, textual description or image.
Conditional generative modelling is more challenging than unconditional generative modelling, due to the need to ensure consistency between the conditioning input and the generated output while still ensuring image ﬁdelity and diversity. As the conditioning input becomes more speciﬁc, there are fewer images in the training set that are consistent with the conditioning input, resulting in weaker supervision.
So, in the order of decreasing supervision are class, text and image-conditional generative models – in class conditioning, many images correspond to each class; in text conditioning, multiple images can at times share the same textual description; in image conditioning (e.g., a grayscale image as the input and a colour image as the output), typically only one output image that corresponds to each input image is observed (e.g., only one way to colour a grayscale image is observed). We will focus on the most challenging of these settings, namely image conditioning. Regardless of the supervision, the goal of conditional generative models is to learn the full distribution over all possible images consistent with the input.
Attaining this goal is challenging, especially when the supervision is weak, and so popular existing methods such as variational autoencoders (VAEs) [39, 63] and generative adversarial nets (GANs) [24, 25] focus on generating a point estimate of the conditional distribution, such as the mean (VAE) or an arbitrary mode (GAN). Such methods are known as unimodal methods , whereas methods that aim to generate samples from the full distribution (which enables samples to be drawn from as many modes as there exist) are known as multimodal methods .
Extending VAEs and GANs to the multimodal setting has proven to be challenging.
VAEs suffer from posterior collapse [16, 51] where the encoder ignores the conditional input, leading the decoder to produce the conditional mean.
GANs suffer from mode collapse [33, 95], where the generator tends to ignore the latent noise, leading the generator to produce the same mode of the conditional distribution, regardless of the noise.
Both are less than ideal in the context of image synthesis, since the conditional mean is usually blurry, whereas other equally valid modes (e.g., other possible ways to colour a grayscale image) cannot be generated.
A recent method [47] takes a different approach by extending an alternative generative modelling technique known as Implicit Maximum Likelihood Estimation (IMLE) [46] to the conditional synthesis setting.
At a high level, IMLE uses a generator like GANs, but instead of using an adversarial objective which makes each generated sample similar to some observed image , IMLE uses a different objective that ensures each observed image has some similar generated samples .
Hence, the generator can cover the whole data distribution without dropping modes.
We refer interested readers to [47, 46] for more details on the algorithm.
However, as shown in the top row of Figure 1, the output image from cIMLE [47] lacks ﬁne details.
For cIMLE to generate high quality samples, there must a sample out of m samples generated during training that is similar to the observed image.
This necessitates a large m ; unfortunately, generating a large number of samples is computationally expensive.
So we are faced with a tradeoff: do improvements in sample quality have to come at the expense of sample efﬁciency?
In this paper, we show that there is a way around this conundrum.
The idea is to generate samples in a clever way such that the best sample is about as similar to the observed image as if a large number of samples had been generated, without actually generating that many samples.
We propose three ideas: partitioning of the latent code, partial evaluation of latent code components and iterative construction of latent code as shown in the bottom row of Figure 1.
These three ideas give rise to a novel method known as Conditional Hierarchical IMLE (CHIMLE).
We demonstrate that CHIMLE signiﬁcantly outperforms the prior best IMLE-based method [47] in terms of both ﬁdelity and diversity on a variety of tasks.
Moreover, we show that CHIMLE achieves the state-of-the-art results in image ﬁdelity and mode coverage compared to leading general-purpose multimodal and task-speciﬁc methods, including both GAN-based and diffusion-based methods.
Method 2.1 Preliminaries: Conditional Implicit Maximum Likelihood Estimation (cIMLE)
In ordinary unimodal synthesis, the model is a function f ✓ parameterized by ✓ that maps the input to the generated output.
To support multimodal synthesis, one can add a latent random variable as an input, so now f ✓ takes in both the input x and a latent code z drawn from a standard Gaussian N (0 , I ) and produces an image b y as output.
To train such a network, we can use a conditional GAN (cGAN), which adds a discriminator network that tries to tell apart the observed image y and the generated output b y .
The generator is trained to make its output b y seem as real as possible to the discriminator.
Unfortunately, after training, f ✓ ( x , z ) often produces the same output for all values of z because of mode collapse.
Intuitively, this happens because making b y as real as possible would push it towards the observed image y , so the generator tries to make its output similar to the observed image y for all values of z .
As a result, naïvely applying cGANs to the problem of multimodal synthesis is difﬁcult.
Conditional IMLE (cIMLE) [47] proposes an alternative technique for training the generator network f ✓ .
Rather than trying to make all outputs generated from different values of z similar to the observed image y , it only tries to make some of them similar to the observed image y .
The generator is therefore only encouraged to map one value of z to the observed image y , allowing other values of z map to other reasonable outputs that are not observed.
This makes it possible to perform multimodal synthesis.
Also, unlike cGANs, cIMLE does not use a discriminator and therefore obviates adversarial training, making training more stable.
The cIMLE training objective is:
[Equation]
where d ( · , · ) is a distance metric, m is a hyperparameter, and x ( i ) and y ( i ) are the i th input and observed image in the dataset.
During each iteration of the training loop, a pool of m samples are generated from f ✓ for each input x ( i ) .
The cIMLE algorithm then selects the closest sample for each observed image
y ( i ) to optimizes ✓ .
2.2 Conditional Hierarchical IMLE (CHIMLE)
In order to generate high-quality images using cIMLE, we need some of the m generated samples to be similar to the observed image y during training.
Achieving this requires a large m , but generating samples is expensive.
This forces a tradeoff between the number of samples and the mode modelling accuracy, which is less than ideal.
The key observation is that the gradient of the loss depends solely on the closest sample for each observed image, as shown in Eqn. 1 – all other samples that are not selected do not contribute.
Therefore, if we could efﬁciently search for a latent code z that would produce a sample of a comparable degree of similarity to y as one that would be selected from a larger pool z ⇤ , we could achieve high sample quality without actually generating that many samples.
To solve search problems, a time-tested paradigm in computer science is divide-and-conquer, which entails dividing the problem into simpler sub-problems, conquering each sub-problem and combining the solutions from the solved sub-problem.
In the context of searching for a latent code z that simulates the quality of z ⇤ , we propose dividing the latent code into components, evaluating each latent code component and combining the latent code components iteratively.
In the ensuing discussion, we will consider a common paradigm of image synthesis models [62, 33, 12, 34], where the model processes the image at L different scales.
We can feed different dimensions of the latent code as inputs to different layers operating at different resolutions, thereby controlling the variations of content at different resolutions.
Division of Latent Code To reduce the difﬁculty of the original problem, we can divide it into sub-problems.
Each of these sub-problems should be a simpler instance of the original problem.
We propose dividing the dimensions of the latent code z 2 R n into L different components S
= { z i } Li =1 , where z
i 2 R k i and P Li =
1 k i = n , which amounts to dividing the latent space into mutually orthogonal subspaces.
Each component z i corresponds to the dimensions that serve as input to layers operating at a particular resolution.
Partial Evaluation of Latent Code Components To determine how promising a latent code component is, we need a way to evaluate a partially constructed latent code, with only some components, indexed by S r ⇢ S , being realized with concrete values.
We propose using a partial output ˜ y p that depends only on the currently constructed components z S r , and evaluating ˜ y p against a part of the observed image y p that only contains details at resolutions modelled by z S r .
So instead of computing d (˜ y , y ) , we compute d (˜ y p , y p ) .
By doing so, we can evaluate the quality of z S r more accurately, since ˜ y p does not depend on the unconstructed components z S \ S r .
To realize this, we add output heads at the L different resolutions that the model operates on to produce the partial outputs ˜
y p and downsample the observed image y to the corresponding resolutions to produce the partial observed images y p .
The values of the partially realized components that result in the greatest similarity to y p is the ﬁnal selected component, i.e., z ⇤ p = arg min d (˜ y p , y p ) .
Require: One input image at L increasing resolutions { x l } Ll =1 , the set of partial observed images { y l } Ll =1 at corresponding resolutions and the gener- ator f ✓ ( · , · ) that produces L outputs at correspond-
ing resolutions for l = 1 to L do
Randomly generate m i.i.d.
latent codes for the l th component z (1) l , . .
, z ( m )
l ˜ y
( j ) l l th output from f ✓ (( x 1 , . . .
, x l ) , ( z ⇤ 1 , . . .
, z ⇤ l  1 , z ( j ) l )) 8 j{ 1 , . . .
, m } z ⇤ l arg min j d ( y l ,
˜ y ( j ) l ) 8 j 2 { 1 , . .
. , m } end for return { z ⇤ l } Ll =
Iterative Combination of Latent Code Components To maximize efﬁciency, the solution to one sub-problem should reveal some information on which other sub-problems to solve.
So, we need to decide on which sub-problems to solve ﬁrst and how to use their solutions to decide on which sub-problem to solve next.
We leverage the fact that solving for the appearance at lower resolutions (e.g., ﬁnding out that the coarse structure of an apple is red) can inform on the probable appearances at higher resolutions (e.g., the ﬁne details of the apple are likely red as well).
We propose starting the construction of the latent code from the component operating at the lowest resolution to the highest, i.e. from z 1 to z L , where z i ’s are assumed to be ordered by resolution in increasing order.
At a given resolution l 2 { 1 . . .
L } , we set the components at all lower resolutions to the values selected in previous steps z ⇤ 1 , . . .
, z ⇤ l  1 to construct z S l = ( z ⇤ 1 , . . .
, z ⇤ l  1 , z l ) , and evaluate the quality of z S l using the partial evaluation method proposed above.
If we put everything together, we obtain the Conditional Hierarchical IMLE (CHIMLE) algorithm, which is detailed in Alg. 1 for the special case of one input image.
We empirically validate the effectiveness of CHIMLE by comparing the number of generated samples CHIMLE needs to obtain the same level of similarity to the observed image as cIMLE.
As shown in Figure 2, CHIMLE consistently requires fewer samples than cIMLE, demonstrating a signiﬁcant improvement in sample efﬁciency.
2.3 Model Architecture
As shown in Figure 3, our architecture contains a sequence of modules, each of which handles an input image of a particular resolution and outputs an image whose resolution is increased by a factor of 2.
We downsample the input image repeatedly by a factor of 2 to obtain a set of input images at different resolutions and feed them into different modules.
As introduced in Sect.
2.2, we divide the dimensions of the latent code into components and feed them to each of the modules.
Note that this architecture generalizes to varying levels of output resolution, since we can simply add more modules for high-resolution outputs.
We add intermediate output heads to each module and add supervision to the output of each module.
We choose the LPIPS [91] distance as the distance metric used in the
IMLE objective function.
For the design of each module, we follow prior works on modelling global and local image features [53, 9, 74, 52, 67] and design a module that comprises of two branches, a local processing branch and a global processing branch.
The local processing branch takes in the conditional image input, a latent code with the matching resolution as the image and the output from the previous module if applicable.
The global processing branch takes in a ﬁxed-sized latent code input and produces scaling factors and offsets for the different channels of the intermediate features in the local processing branch.
For more implementation details, please refer to the appendix.
Due to the hierarchical nature of the architecture and its role as the implicit model in IMLE (also known as the “generator” in GAN parlance), we dub this architecture the Tower Implicit Model (TIM).
Experiments
Baselines To validate our main contribution of this paper, namely improving the output ﬁdelity of IMLE-based methods , we compare to the leading IMLE-based method, cIMLE [47].
As a secondary comparison, we demonstrate potential impact in a broader context and compare our method to the leading multimodal general-purpose conditional image synthesis methods according to a recent survey [59], including four GAN-based methods, such as BicyleGAN [95], MSGAN [54], DivCo [50],
[Figure]
NDM [6]
BicycleGAN [95] cIMLE [47]
CHIMLE (Ours)
[Figure]
MoNCE [85]
RFB-ESRGAN [69] cIMLE [47]
CHIMLE (Ours)
Input
MoNCE [85]
InstColorization [70] cIMLE [47]
CHIMLE (Ours)
Input
BicycleGAN [95] DnCNN [87] cIMLE
CHIMLE (Ours) MoNCE [85], and two diffusion-based methods, such as DDRM [36] and NDM [6]. As a tertiary comparison, we also compare our method to popular task-speciﬁc methods based on leaderboard rankings in challenges and recent survey papers [86, 3].
Tasks We apply our method to four different conditional image synthesis tasks, namely night-today, 16 ⇥ single image super-resolution, image colourization and image decompression.
We pick night-to-day because it is a classic task in the multimodal general-purpose conditional image synthesis literature [33, 95].
We pick the remaining tasks because they are more challenging as evidenced by the availability of task-speciﬁc methods tailored to each task.
Training Details We use a four-module model introduced in Sect.
2.3 for all tasks.
The input to each module is downsampled from the full resolution input to resolution the module operates on.
We trained all models for 150 epochs with a mini-batch size of 1 using the Adam optimizer [38] on an NVIDIA V100 GPU.
Details for datasets are included in the appendix.
Evaluation Metrics We evaluate perceptual quality of predictions using the Fréchet Inception
Distance (FID) [28] and Kernel Inception Distance (KID) [7].
We also measure the diversity of the predictions since it is a crucial criterion of multimodal conditional image synthesis.
One metric to measure diversity is the LPIPS diversity score (not to be confused with the LPIPS distance metric) [95], which is the average LPIPS distance between different output samples for the same input.
We observe that the LPIPS diversity score may favour a set of samples where some are of poor visual ﬁdelity as shown in Figure 5.
To get around this limitation, we use faithfulness-weighted variance (FwV) [47] instead, which is the average LPIPS distance d LPIPS between the output samples and the mean, weighted by the consistency with the original image measured by a Gaussian kernel:
M =
X i X j w i d LPIPS (˜ y
( i,j ) , ¯ y ( j ) ) , where w i = exp ✓  d LPIPS (˜ y
( i,j ) , y ( j ) ) 2 ◆ (2) where ˜ y ( i,j ) is the i th generated sample from the j th input.
¯ y ( j ) is the mean of all generated samples from the j th input.
y ( j ) is the j th observed image.
The kernel bandwidth parameter  trades off the importance of consistency vs. diversity.
We measure how well the modes match the model distribution and the true distribution along two axes, mode accuracy (precision) and mode coverage (recall).
We measure the former with the F 1 /score component of the PRD metric [64], which is heavily weighted towards precision, and the latter with the F 8 score component of the PRD metric [64], which is heavily weighted towards recall, and the inference-via-optimization (IvO) metric [55], which measures the ﬁnal mean squared error (MSE) between the generated image and the observed image after optimizing over the latent code input while keeping the model weights ﬁxed. In our conditional image synthesis setting, IvO is more suitable than the F 8 score in PRD, as IvO measures mode coverage in the conditional distribution, whereas the F 8 score measures mode coverage in the marginal distribution. Due to a large number of noise samples in NDM (2 for each of the 1000 diffusion steps), we do not have sufﬁcient computation power to calculate IvO for it and therefore omit it.
3.1 Quantitative Results
We compare the perceptual quality and output diversity in Table 1.
As shown, CHIMLE outperforms best prior IMLE-based method, cIMLE, general-purpose conditional image synthesis methods and task-speciﬁc methods in terms of FID and KID across all tasks.
CHIMLE also outperforms all multimodal baselines in terms of FwV across all tasks.
These results show that CHIMLE can produce more realistic and diverse images than the baselines, thereby setting a new state-of-the-art.
We compare the mode accuracy and coverage in Table 2.
As shown, CHIMLE outperforms or achieves comparable performance to cIMLE and other multimodal general-purpose baselines in terms of PRD scores.
In addition, CHIMLE signiﬁcantly outperforms cIMLE and other multimodal general-purpose baselines in terms of IvO. These results show that CHIMLE achieves better mode accuracy and coverage of the true distribution compared to the baselines. [Table]
3.2 Qualitative Results
We show the qualitative comparison of our method (CHIMLE) and selected baselines on various tasks in Figure 4.
As shown, CHIMLE generates high-quality results, like the high contrasts in the building and the bushes in night-to-day ( 1 st row), the sharp appearance of the wing pattern of the butterﬂy wing and the ﬂowers in the background ( 2 nd row), the vibrant yet detailed colouring of the parrot ( 3 rd row) and the high-frequency details in the sculpture ( 4 th row).
Figure 6 shows the qualitative comparison of the best reconstruction of the observed image found by different methods using IvO [55], which tries to approximately invert each model by optimizing over the latent code input to ﬁnd a generated image closest to the observed image.
As shown, the reconstruction of the CHIMLE model is the closest to the observed image and achieves almost pixel-level precision.
In addition, we found that for our model, IvO can consistently converge to a high-quality sample, whereas it would diverge quite often for the baselines, suggesting that inverting our model is much easier than the baselines.
This demonstrates that CHIMLE successfully covers the mode corresponding to the observed image.
More comparisons to the baselines are included in the appendix and on the project website.
3.3 Ablation Study
We incrementally remove (1) iterative combination (IC), (2) partial evaluation (PE), (3) latent code division (CD).
As shown in Figure 7, each component is critical to achieving the best results.
Notably, we see an average increase in terms of FID by 7 . 7% after removing IC, by another 7 . 8% after removing PE and by another 12 . 8% after removing CD, thereby validating the importance of each of our contributions.
Related Work
General-Purpose Conditional Image Synthesis Conditional image synthesis can be divided into class conditioning [56, 58] where the output is conditioned on class labels, and image conditioning which takes an image as conditional input.
The latter is often viewed as an image-to-image translation problem, which is most closely related to the setting we consider.
General image-to-image translation methods can be paired or unpaired.
In the paired setting, each image in the source domain corresponds to an image in the target domain [33, 83, 76, 68].
In the unpaired setting, images in the source domain may not correspond to images in the target domain [94, 82, 72, 49, 14, 8, 61].
Whereas these previous methods are all unimodal, other methods aim for the more challenging task of multimodal [Table] [Figure] synthesis in the paired setting [95, 22, 20] and in the unpaired setting [30, 44, 45, 15].
Since many of the aforementioned works use GANs as their generators which suffer from mode collapse, some works aim to address this issue by introducing mode seeking terms [54] or adding contrastive losses [50, 85, 29].
Another line of work uses diffusion models [36, 6], where the model generates the output by iteratively denoising an image with pure noise.
Despite achieving promising results, these methods still need to devote part of the model capacity to reconstruct the information encoded in the conditioning input from pure noise, which limits their expressiveness given the same compute budget compared to one-shot generative models such as GAN, VAE and IMLE, which can directly map the conditional input to output.
In this paper, we focus on the paired multimodal setting with one-shot generative models, and sidestep mode collapse by building on a recent generative modelling technique, conditional IMLE [47].
Full Model
No IC
No IC, PE
No IC, PE, CD
SR
DC
Full Model 16 .73 .
No IC 17 .81 .
No IC, PE 18 .87 .
No IC, PE, CD 21 .101 .
comparison as we gradually remove (1) iterative combination (IC), (2) partial evaluation (PE), (3) latent code division (CD) on two tasks, 16 ⇥ super-resolution (SR) (ﬁrst row in the ﬁgure) and image decompression (DC) (second row in the ﬁgure).
As shown in the ﬁgure, the generated image contains less ﬁne details (as in SR, where the strawberry seeds become less pronounced) or colour saturation (as in DC) as we remove each component, which is also reﬂected by the FID shown in the table.
Task-Speciﬁc Approaches There is a large body of work on super-resolution, most of which consider upscaling factors of 2  4 ⇥ .
See [81, 57, 79] for comprehensive surveys.
Many methods regress to the high-resolution image directly and differ widely in the architecture [21, 37, 41, 71, 26, 92, 17, 48].
Various conditional GANs have also been developed for the problem [43, 65, 78, 84, 60, 80, 77, 69].
For image colourization, a recent survey [3] provides a detailed overview of different methods.
Some methods [13, 11, 90, 10] use deep networks to directly predict the colour and combine it with the input to produce the output.
Other methods [32, 42, 19, 93, 70] take advantage of features from different levels or branches in the network for colour prediction.
There is relatively little work on image decompression to our knowledge.
[4] targets DCT-based compression methods and explicitly models the quantization errors of DCT coefﬁcients.
More work was done on learned image compression [2, 1], which changes the encoding of the compressed image itself.
Another related area is image denoising, see [73] for a survey of deep learning methods.
Many of the methods uses a ResNet [27] backbone or other variants of deep CNNs [87, 88, 89, 75].
Discussion and Conclusion
Limitation Our method can only generate output images that are similar to some training images.
Take colourization as an example, if there is no green apple in the training data, then our model cannot produce green apples.
Societal Impact CHIMLE could be applied to a broad range of conditional image synthesis tasks.
While most of the applications are harmless, it could be potentially used for tasks like demosaicing which may constitute a privacy or copyright violation.
In addition, since CHIMLE adopts a modular design and could include more modules to produce more reﬁned results, training such huge model may increase greenhouse gas emissions.
Conclusion In this paper, we developed an improved method for the challenging problem of multimodal conditional image synthesis based on the IMLE framework.
We addressed the undesirable tradeoff between sample efﬁciency and sample quality of the prior best IMLE-based method by introducing a novel hierarchical algorithm.
We demonstrated on a wide range of tasks that the proposed method achieves the state-of-the-art results in image quality without compromising diversity or mode coverage.
Acknowledgement We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC). This research was enabled in part by support provided by WestGrid (westgrid.ca) and the Digital Research Alliance of Canada (alliancecan.ca). We thank Tristan Engst,
Yanshu Zhang, Saurabh Mishra, Mehran Aghabozorgi and Shuman Peng for their comments and suggestions. Checklist For all authors.. (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work? [Yes] See Section 5 for details. (c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 5 for details.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] 2. If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results? [N/A] No theoretical results.
(b) Did you include complete proofs of all theoretical results? [N/A] No theoretical results. If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] See supplementary material for details.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 3 for details.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators? [Yes] Existing assets are cited.
(b) Did you mention the license of the assets?
[N/A] Assets used in this research carry no license/access restrictions. (c) Did you include any new assets either in the supplemental material or as a URL? [N/A] No new assets.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating? [N/A] No new assets. (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] No personally identiﬁable information in the data.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] No crowdsourcing.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] No crowdsourcing.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A] No crowdsourcing.