Masked Generative Adversarial Networks are Robust Generation Learners
Abstract
This paper shows that masked generative adversarial networks (MaskedGAN) are robust image generation learners with limited training data.
The idea of
MaskedGAN is simple: it randomly masks out certain image information for effective GAN training with limited data.
We develop two masking strategies that work along orthogonal dimensions of training images, including a shifted spatial masking that masks the images in spatial dimensions with random shifts, and a balanced spectral masking that masks certain image spectral bands with selfadaptive probabilities.
The two masking strategies complement each other which together encourage more challenging holistic learning from limited training data, ultimately suppressing trivial solutions and failures in GAN training.
Albeit simple, extensive experiments show that MaskedGAN achieves superior performance consistently across different network architectures ( e.g. , CNNs including BigGAN and StyleGAN-v2 and Transformers including TransGAN and GANformer) and datasets ( e.g. , CIFAR-10, CIFAR-100, ImageNet, 100-shot, AFHQ, FFHQ and
Cityscapes).
Introduction
“What I cannot create, I do not understand.” —Richard Feynman.
Generative Adversarial Network (GAN) [18] is such an unsupervised framework learning to understand and create: the discriminator aims to understand “what are real and fake images” by binary discrimination while the generator strives to produce realistic images according to discriminator’s feedback.
In recent years, GANs have achieved great success under the presence of large amounts of training data.
When only limited training data is available, they instead experience obviously increased generation failures [52, 28].
By reversing the dictum, we get “What I do not understand, I can not create” [40], which might be the perfect metaphor of such generation failures.
Specifically, with limited training data, the discriminator tends to achieve discrimination via certain less meaningful shortcuts by memorizing training data or merely focusing on easy-to-discriminate image locations and spectra without holistic understanding of images, ultimately leading to trivial solutions that fail to “create” realistic images [52, 28] as shown in Fig. [Figure]
Trivial solutions are ubiquitous in unsupervised learning [3, 6, 7, 10], and so in Masked Autoencoders (MAE) [45, 39, 8, 17, 2, 20, 48] as one prevalent unsupervised representation method.
MAE works by masking and reconstructing the image, where image masking is the core design that suppresses trivial solutions and enables learning high-level semantic features [20, 5].
For instance, [20] presents a high-ratio masking strategy for MAE: 1) it can impede shortcuts in image reconstruction, e.g. , local spatial interpolation that recovers a masked patch from adjacent patches without high-level understanding [20, 5]; 2) it encourages global spatial interpolation that requires modelling inter-patch relationship globally, ultimately learning nontrivial and meaningful representations [20, 5].
In this work, we explore the idea of image masking training for GAN, aiming to develop robust image generation learners with limited training data.
By viewing image masking as one general means of suppressing training shortcuts, we hypothesize that image masking in GAN training should be conducted over both spatial and spectral dimensions for preventing discriminators from merely focusing on easy-to-discriminate image locations and spectra.
In another word, applying random spatial and spectral masking creates more challenging tasks for discriminator training which forces discriminators to learn more holistic instead of easy-to-discriminate information only.
At the other end, as applying masking to discriminators slows down their learning, it is crucial to decelerate the generator learning synchronously for an overall balanced and stable adversarial learning.
Driven by this motivation, we design MaskedGAN, a robust image generation network that can learn effectively with limited training data.
The idea of MaskedGAN is simple: it randomly masks certain image information to suppress trivial solutions in GAN training.
We develop two masking strategies that work along orthogonal dimensions of images.
The first is shifted spatial masking that masks the image in spatial dimensions with a random shift.
Intuitively, random masking along spatial dimensions enforces discriminator to learn over all image locations instead of merely focusing on easy-to-discriminate locations.
The second is balanced spectral masking that decomposes an image into multiple bands in spectral space and masks a portion of spectral bands with self-adaptive probabilities.
Random masking along spectral dimension encourages discriminator to learn hard-todiscriminate bands ( e.g. , high-frequency bands capturing shapes and structures) instead of largely on easy bands ( e.g. , low-frequency bands capturing color and brightness).
The two masking strategies complement each other which together suppress trivial solutions and training failures by encouraging more challenging holistic discriminator learning from limited training data.
MaskedGAN also applies the two masking strategies to the generator training.
Differently, on generator side, the two image masking operations work on outputs by sparsifying the training signals backpropagated from the discriminator.
This design slows down the generator learning and stabilizes the overall adversarial learning by keeping similar learning paces for discriminator and generator.
It ensures that MaskedGAN can converge to a Local Nash Equilibrium under certain conditions [21].
We summarize the contributions of this work in three aspects.
First , this paper investigates image masking training for GAN, aiming for building robust image generation learners with limited training data.
Second , we propose MaskedGAN and design spatial and spectral masking for training robust and effective GAN with limited data.
The two masking strategies complement and encourage more challenging holistic understanding of images which help suppress trivial solutions and stabilize the overall adversarial training.
Third , extensive experiments show that a simple implementation of MaskedGAN achieves superior generation performance consistently across different network architectures and datasets.
Related Work
This work is related to two main fields of research, namely, generative adversarial networks for image generation and masked autoencoders for unsupervised representation learning.
Generative Adversarial Network (GAN) [18] is an unsupervised data synthesis framework that aims to generate a model distribution that mimics a given target distribution.
It consists of a generator that generates model distribution and a discriminator that distinguishes the model distribution from the target.
In recent years, a series of GANs have been proposed to improve the performance and training stability from different perspectives, such as designing more stable training objectives [1, 33, 19, 34, 42], building better network architectures [41, 35, 36, 50, 30, 4, 25, 23], involving various training strategies [15, 27, 32], etc.
Despite the great success obtained, they require large amounts of training data for training effective generation models.
When only limited training data is available, they often suffer from clearly increased generation failures with severely degraded performance [52, 28, 24, 9].
In this work, we focus on training effective GANs with limited training data.
Training GAN with limited data: Recently, the task of training effective GANs with limited data has attracted increasing attention for alleviating the requirement of large amounts of training data.
Most existing efforts address this challenging task from two perspectives.
The first is data augmentation [52, 28, 24] that works via massive hand-crafted data augmentation policies, e.g. , [52] employs different types of differentiable augmentation to stabilize GAN training.
The second is model regularization [43, 13, 9] which stabilizes GAN training by regularizing the discriminator training via label flipping [43], network co-training [13], lottery ticket hypothesis [9], etc.
We tackle this challenging task from a perspective of image masking training.
Specifically, we design MaskedGAN that exploits two orthogonal image masking strategies for encouraging more challenging holistic understanding of images.
Extensive evaluations show that MaskedGAN stabilizes the adversarial training process and suppresses trivial solutions and training failures effectively.
Masked Autoencoders (MAE) has greatly advanced the research of unsupervised representation learning recently.
Despite different designs, MAE is essentially a denoising autoencoder (DAE) [44] that learns semantic representations by corrupting and reconstructing the input signals.
Most existing work can be viewed as a generalized DAE under different types of corruptions, such as masking image pixels, patches, and regions [45, 39, 8, 17, 2, 20], or masking image color channels [51].
For example, Context Encoder [39] masks random image regions of different shapes and learns to reconstruct them.
Color Encoder [39] masks image color channels and learns to reconstruct them from grayscale images.
Recently, [17, 2, 20] mask random image patches for MAE training.
In particular, it is demonstrated in [20] that various image masking strategies can help suppress trivial solutions in MAE training while large mask ratios help learn better representations.
Different from most existing work that explores image masking in MAE training for learning better unsupervised representations, we explore the idea of image masking for GAN training and aim to develop robust image generation learners with limited training data.
We design two orthogonal image masking strategies, namely, shifted spatial masking and balanced spectral masking that help learn robust image generation models effectively, more details to be described in the ensuing subsections.
Method 3.1 Task Definition
This work focuses on the task of training GAN with limited training data.
GAN is an unsupervised framework that aims to generate a model distribution that mimics a given target distribution: the generator G learns to generate images G ( z ) given the latent code z while the discriminator D learns to distinguish the generated images G ( z ) from the real images x .
Conventionally, GAN is optimized alternatively with a discriminator loss L d and a generator loss L g that can be defined as:
[Equation]
where p data denotes the distribution of the provided real images and p z is the prior distribution ( e.g. ,
Gaussian distribution).
[Figure]
GAN training failures with limited data .
Prior studies [52, 28] show that GAN often experiences generation failures with severely degraded generation performance when only limited training data is available.
Specifically, with limited training data, the discriminator tends to discriminate via meaningless shortcuts by merely focusing on easy-to-discriminate image locations and spectra instead of holistic understanding of images.
This can be viewed clearly in Fig.
1 of the appendix where the Gini Coefficient [16] of discriminator’s spatial attentions increases quickly along the training iteration (when only limited training data is available).
Note that the Gini coefficient [16] is negatively correlated with equality, i.e., the discriminator will pay more unevenly distributed attention to each spatial location while the Gini coefficient increases from ‘0’ to ‘1’.
For image generation with GAN, the large Gini coefficient thus means that the discriminator starts to focus on certain spatial locations (easy to discriminate) while ignoring other spatial locations (hard to discriminate), ultimately leading to an over-confident discriminator and training collapse.
3.2 Masked Generative Adversarial Network
We tackle the challenging task of training GAN with limited data from a perspective of image masking training [20].
Specifically, we design two orthogonal image masking strategies, namely, shifted spatial masking and balanced spectral masking as illustrated in Fig.
The two masking strategies complement each other and encourage more challenging holistic understanding of images which help stabilize the adversarial training process and suppress trivial solutions and training failures.
Shifted spatial masking masks random image patches in spatial dimensions, aiming to force discriminator to learn over all image locations instead of merely focusing on easy-to-discriminate locations.
It first generates a random patch-based mask and then introduces a random shift to the generated mask and finally applies the shifted mask to the training image.
The shift operation allows to conduct spatial masking continuously along spatial dimensions: a masked patch could appear on any possible locations instead of on fixed grid locations [20, 2].
This feature is critical to image generation that requires high-fidelity along spatial dimensions.
Given an image x ∈ R H × W , we first generate a random patch-based spatial mask m spatial ∈ { 0 , 1 } H × W and apply a random spatial shift d ∈ [0 , D ] on m spatial to acquire the shifted spatial mask m ′ spatial .
The shifted spatial masking function can be defined by: : M spatial ( x ) = x × m ′ spatial .
In the patch-based mask generation, we set the grid size as N × N , i.e. , the patch size is HN × WN .
We set the random shift range D as half of patch size, which allows the masked patch to appear on any possible locations.
For images with multiple color channels, each channel are masked in the same manner.
Relations to the spatial masking in MAE .
MAE aims to encode an image patch into a feature vector, which is a downsampling process and does not require high precision along spatial dimensions ( i.e. , pixels within a patch are considered as the same).
GAN, on the contrary, aims for decoding a latent code into a high-resolution image, which is an upsampling process and requires high fidelity along spatial dimensions ( i.e. , pixels within a patch should not be completely the same).
We therefore introduce an additional “mask shift” to allow the masked patches to appear at any image locations (instead of fixed grid locations only as in the spatial masking in MAE [20, 2]), which helps generate images with high-fidelity along spatial dimensions.
Balanced spectral masking masks random spectral bands of an image in the decomposed spectral space, aiming to encourage the discriminator to learn from all spectral bands instead of focusing on easy bands ( e.g. , low-frequency bands capturing color and brightness) only.
Specifically, balanced spectral masking decomposes an image into multiple bands in spectral space and masks a portion of spectral bands with self-adaptive probabilities, i.e., a spectral band is masked with a higher (or lower) probability if it contains more (or less) contents.
The balanced masking is crucial to the spectral masking as image contents are usually unevenly distributed along the spectral dimension.
Specifically, the contents of natural images are distributed very densely around low-frequency bands but very sparsely in high-frequency bands.
Therefore, uniform masking along the spectral dimension is imbalanced and does not work well for image generation.
Given an image x ∈ R H × W , we first employ Fourier transformation [22] to decompose it into multi-band representation x mul ∈ R H × W × C , where C denotes the number of spectral bands.
The image contents in each band I x ∈ R C can then be computed by summing all contents in each band, i.e. , I x = (cid:80) H (cid:80) W x
( H,W,C ) mul , and followed by an normalization operation: I ′ x
= I ( C ) x / (cid:80) C
I ( C ) x .
With the calculated I ′ x , The balanced spectral mask m spectral ∈ { 0 , 1 } C can be generated by masking c -th band with probability I ′ ( c ) x .
Finally, the balanced spectral masking function can be defined by:
M spectral
[Equation]
Loss functions.
We apply the two image masking strategies to both discriminator and generator training, where the overall training losses can be formulated as follows:
[Equation]
where M ( · ) =
M spectral ( M spatial ( · )) is the combination of shifted spatial masking and balanced spectral masking .
Note the two masking strategies work in different manners in the discriminator and generator training.
Specifically, by applying image masking on input images in discriminator training, we could create a challenging discriminator task that requires holistic learning beyond certain easy-to-discriminate information.
Differently, in generator training, the image masking operation works on output side by sparsifying the training signals back-propagated from the discriminator.
One major consideration is to slow down the generator training so that it has similar learning pace as the discriminator which further stabilizes the overall adversarial learning between the discriminator and the generator.
3.3 Theoretical Insights
The proposed Masked Generative Adversarial Network (MaskedGAN) is inherently connected with the theory of stochastic approximation.
Proposition 1 The MaskedGAN can be modeled as an instance of the Two Time-Scale Update Rule.
Proposition 2 The MaskedGAN converges to a Local Nash Equilibrium under certain conditions.
The proofs of Propositions 1 and 2 are provided in the Appendix.
Experiments
We evaluate MaskedGAN over different network architectures and public datasets.
Sections 4. presents experiments with BigGAN over datasets CIFAR-10 [31], CIFAR-100 [31] and ImageNet [14].
Section 4.2 reports experimental results with StyleGAN-v2 over datasets 100-shot [52], AFHQ [11] and FFHQ [29].
Section 4.3 presents experiments with two transformer-based GANs (TransGAN and
GANformer) over datasets CIFAR-10, CIFAR-100 and Cityscapes [12].
In addition, we present ablation studies in Section 4.1 and discuss different features of the proposed MaskedGAN in Section [-4.
-]4.
The implementation and dataset details are provided in appendix. [Table] [Table] [Table]
CIFAR-10 and CIFAR-100.
Table 1 reports class-conditional image generation results with baseline
BigGAN [4] on CIFAR-10 and CIFAR-100.
All models are trained with 100%, 20% or 10% training data ( i.e. , 50K, 10K or 5K images), and evaluated over the validation set (10K images).
It shows that
MaskedGAN ( i.e. , Masked-BigGAN) achieves superior performance over CIFAR-10 and CIFARas compared with state-of-the-art methods.
Results are averaged with three runs.
Ablation studies over CIFAR-10.
We perform ablation studies with the widely adopted BigGAN [4] over 10% CIFAR-10 data as shown in Table 2.
As the core of the proposed MaskedGAN, we examine how our designed shifted spatial masking and balanced spectral masking contribute to the overall performance of data-limited image generation.
As shown in Table 2, the baseline (BigGAN) in the first row does not perform well with limited training data.
Including the basic Spatial Masking in
GAN training improves the baseline clearly as shown in the 2nd row, while further involving Random
Shift on top of Spatial Masking brings clear improvement in FID as shown in the 3rd row.
On the other hand, the basic Spectral Masking (masks spectral bands with the same probability) does not work well as shown in the 4th row as image contents are usually unevenly distributed along the spectral dimension.
However, Balanced spectral masking which includes the proposed Self-adaptive [Table] Obama [Table]
Panda Scale/shift [Table] [Table]
Yes
[Table]
[Table]
[Table]
MineGAN [46]
Yes
[Table]
[Table]
[Table]
TransferGAN [47]
Yes 48.
TransferGAN + DA [52]
Yes 39.
FreezeD [37]
Yes 41.
StyleGAN-v2 [30]
[Table]
No 80.
[Table]
ADA [28]
No 45.
LeCam-
[Table]
No 38.
GenCo [13]
No 36.
AdvAug [9]
No 52.
DA [52]
No 46.
APA [24]
No 43.
InsGen [49]
No 45.
[Table]
No 33.78 ± 0.
20.06 ± 0.
8.93 ± 0.
calculate
[Table]
Probabilities as shown in the 5th row, improves image generation clearly in FID (19.30 above the baseline), showing that the balanced masking strategy is crucial for effective spectral masking.
In addition, the two image masking strategies work in orthogonal dimensions ( i [Table] e. , spatial and spectral dimensions), which complement each other in data-limited image generation.
We can observe clearly that combining these two masking strategies ( MaskedGAN in the last row) performs the best.
ImageNet.
Table 3 reports class-conditional generation results with BigGAN [4] over ImageNet at 64 × 64 resolution.
The models are trained with different amounts of training data including ~10% (100K images), ~5% (50K images) and ~2.5% (25K images)
[Table]
and evaluated over the whole training set (~1.2 million images).
Experiments show that MaskedGAN works effectively with different data setups over the very diverse and large-scale ImageNet.
Results are averaged with three runs.
4.2 Unconditional image generation with StyleGAN-v2 on 100-shot, AFHQ and FFHQ 100-shot.
Table 4 reports unconditional image generation results with StyleGAN-v2 [4] on 100-shot dataset.
All models are trained and evaluated over 100 images of Obama, Grumpy Cat and Panda, respectively.
Experimental results show that the proposed MaskedGAN works effectively with only 100 images, i.e. , it brings significant gains over StyleGAN-v2 (baseline) and outperforms state-of-theart methods by clear margins.
Besides, MaskedGAN surpasses all transfer learning methods but does not require pre-training over large-scale datasets.
Results are averaged with three runs.
[Table]
[Table]
AFHQ and FFHQ.
Table 5 presents unconditional image generation results with StyleGAN-v2 [4] on AFHQ and FFHQ.
For AFHQ, the models are trained with different amounts of training data including ~20% (1K images), ~10% (500 images) and ~5% (250 images) [Table] and evaluated over the whole training set ( 5K images).
For FFHQ, the models are trained with ~10% (7K images), ~5% (3.5K images) and ~2.5% (1.75K images) data, and evaluated over the whole training set (70K images).
It can be observed that MaskedGAN works effectively with limited training data over animal and human face generation tasks.
Results are averaged with three runs.
4.3 Unconditional image generation with TransGAN and GANformer
We evaluate MaskedGAN with recent transformer-based architectures including TransGAN [25] and
GANformer [23].
Specifically, TransGAN is the first work that uses purely transformers for image generation.
GANformer employs a bipartite structure for computing soft attention, which circumvents the computation constraints in standard transformers and leads to improved image generation.
CIFAR-10 and CIFAR-100 with TransGAN.
Tables 6 reports unconditional image generation with
TransGAN on CIFAR-10 and CIFAR-100.
The models are trained with 100%, 20% or 10% training data ( i.e. , 50K, 10K or 5K images), and evaluated over the whole training set (50K images).
We can observe that our image masking strategies work effectively with transformer-based GAN as well.
Results are acquired with three runs.
Cityscapes with GANformer.
Tables 7 reports unconditional image generation with GANformer on Cityscapes.
The models are trained with ~5%, ~2.5% or ~1.25% training data ( i.e. , 1.25K, 0.63K or 0.31K images ), and evaluated over the whole training set ( i.e. , ~25K images ).
Experiments show that our image masking strategies bring significant gains over the baseline GANformer consistently over scene generation task.
Results are acquired with three runs.
4.4 Discussion
Generalization across different GAN architectures: We examine the generalization of the
proposed
MaskedGAN by evaluating it with four representative GAN architectures, including two CNN-based ( i.e. , BigGAN and StyleGAN-v2) and two Transformer-based ( i.e. , TransGAN and GANformer).
Experimental results in Tables 1- 7 show that our MaskedGAN brings clear generation improvements consistently over different GAN architectures.
Generalization across different generation tasks: We study the generalization of the proposed
MaskedGAN from the perspective of image generation tasks.
Specifically, we perform extensive evaluations over object generation tasks with CIFAR-10, CIFAR-100, and ImageNet, face generation tasks with 100-shot AFHQ, and FFHQ, and scene generation tasks with Cityscapes. Experimental [Figure] [Figure]
[Figure] results in Tables 1- 7 show that the proposed MaskedGAN achieves superior generation consistently over different generation tasks.
Generalization over different numbers of training samples: We study the generalization of the proposed MaskedGAN from the perspective of the number of training samples.
Specifically, we benchmark MaskedGAN on CIFAR-10 and CIFAR-100 with 100%, 20% and 10% data, ImageNet with 10%, 5% and 2.5% data, AFHQ with 20%, 10% and 5% data , FFHQ and Cityscapes with 5%, 2.5% and 1.25% data and 100-shot dataset with only 100 images.
Experiments in Tables 1- 7 show that MaskedGAN achieves superior performance consistently with different amounts of training data.
Convergence comparison across different network architectures and datasets: We examine the convergence of the proposed MaskedGAN by benchmarking it over various network architectures ( e.g. ,
CNNs-based including BigGAN and StyleGAN-v2 and Transformers-based including TransGAN and
GANformer) and datasets ( e.g. , CIFAR-10, ImageNet, 100-shot and Cityscapes).
Fig.
3 provides the line charts of FID versus training iteration .
It shows clearly that most state-of-the-art GANs [30, 4] experience generation failures (or called training collapses) with severely degraded generation performance when only limited training data is available.
In addition, prior studies on data-limited generation ( e.g. , data augmentation method [52] and model regularization method [43]) could improve the performance in some degree but still suffer from generation failures and training collapses.
As a comparison, the proposed MaskedGAN converges well consistently across different amounts of training data, network architectures and datasets.
The great convergence of MaskedGAN is largely attributed to two factors: its image masking designs suppress trivial solutions and training failures directly; it keeps similar learning paces for discriminator and generator which ensures that network converge to a Local Nash Equilibrium under certain conditions [21]. [Figure]
We provide qualitative illustrations in Fig.
4 and 5, which show that MaskedGAN outperforms the baseline and state-of-the-art models clearly.
Specifically, with limited training data, introducing the proposed image masking strategies into GAN training helps generate more realistic and high-fidelity images, especially in terms of image shapes and structures.
Due to the space limit, we provide additional discussions and qualitative illustrations (including comparisons with the state-of-the-arts) in appendix.
Conclusion
This paper presents MaskedGAN, a robust image generation network that can learn effectively with limited training data.
MaskedGAN introduces two image masking strategies that work along orthogonal dimensions of training images, namely, shifted spatial masking and balanced spectral masking .
The two masking strategies complement each other which together encourage more challenging holistic learning from limited training data, ultimately suppressing trivial solutions and failures in GAN training.
Extensive experiments show that MaskedGAN achieves superior performance consistently across different network architectures ( e.g. , CNNs including BigGAN and
StyleGAN-v2 and Transformers including TransGAN and GANformer) and datasets ( e.g. , CIFAR-10,
CIFAR-100, ImageNet, 100-shot, AFHQ, FFHQ and Cityscapes).
Moving forward, we will explore data masking training in other generation tasks such multi-modality generation.
Acknowledgement
This study is supported under the RIE2020 Industry Alignment Fund – Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from Singapore
Telecommunications Limited (Singtel), through Singtel Cognitive and Artificial Intelligence Lab for
Enterprises (SCALE@NTU). This research was also carried out on the High Performance Computing resources at Inception Institute of Artificial Intelligence, Abu Dhabi.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu- tions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[Yes] Following please find them.
This work could mitigate ethical concerns of privacy violation for using massive images in training GAN.
Current GANs require using large amounts of training data for training effective generation models which could raise concerns in data privacy when collecting large amounts of training data.
This work proposes MaskedGAN for data-limited image generation, which aims to learn effective GANs with limited data (e g., 100 images).
Thus, it could relieve the reliance on large amounts of training data and so the risk of privacy violation.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the
[Table]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re us- ing/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]