Original: I S T HERE M ODE C OLLAPSE ? A C ASE S TUDY ON F ACE G ENERATION AND I TS B LACK - BOX C ALIBRATION
Revised: I S T HERE M ODE C OLLAPSE ? A C ASE S TUDY ON F ACE G ENERATION AND I TS B LACK - BOX C ALIBRATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Generative adversarial networks (GANs) nowadays are capable of producing images of incredible realism.
Revised: Generative adversarial networks (GANs) nowadays are capable of producing images of incredible realism.

Original: One concern raised is whether the state-of-the-art
Revised: One concern raised is whether the state-of-the-art

Original: GAN’s learned distribution still suffers from mode collapse.
Revised: GAN’s learned distribution still suffers from mode collapse.

Original: Existing evaluation metrics for image synthesis focus on low-level perceptual quality.
Revised: Existing evaluation metrics for image synthesis focus on low-level perceptual quality.

Original: Diversity tests of samples from GANs are usually conducted qualitatively on a small scale.
Revised: Diversity tests of samples from GANs are usually conducted qualitatively on a small scale.

Original: In this work, we devise a set of statistical tools, that are broadly applicable to quantitatively measuring the mode collapse of GANs.
Revised: In this work, we devise a set of statistical tools, that are broadly applicable to quantitatively measuring the mode collapse of GANs.

Original: Strikingly, we consistently observe strong mode collapse on several state-of-the-art GANs using our toolset.
Revised: Strikingly, we consistently observe strong mode collapse on several state-of-the-art GANs using our toolset.

Original: We analyze possible causes, and for the ﬁrst time present two simple yet effective “black-box” methods to calibrate the GAN learned distribution, without accessing either model parameters or the original training data.
Revised: We analyze possible causes, and for the ﬁrst time present two simple yet effective “black-box” methods to calibrate the GAN learned distribution, without accessing either model parameters or the original training data.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Generative adversarial networks (GANs) (Goodfellow et al., 2014) have demonstrated unprecedented power for various image generation tasks.
Revised: Generative adversarial networks (GANs) (Goodfellow et al., 2014) have demonstrated unprecedented power for various image generation tasks.

Original: However, GANs have also been suffering from generation bias and/or loss of diversity.
Revised: However, GANs have also been suffering from generation bias and/or loss of diversity.

Original: The underlying reasons could be compound, ranging from the data imbalance to the training difﬁculty of GANs, and more:
Revised: The underlying reasons could be compound, ranging from the data imbalance to the training difﬁculty of GANs, and more:

Original: • First of all, the training data for GANs, especially for the typical unconditional/unsupervised generation tasks (Karras et al., 2017; 2018), might possess various subject or attribute imbalances.
Revised: • First of all, the training data for GANs, especially for the typical unconditional/unsupervised generation tasks (Karras et al., 2017; 2018), might possess various subject or attribute imbalances.

Original: As a result, GANs trained with them might be further biased towards the denser areas, similarly to the classiﬁer bias towards the majority class in imbalanced classiﬁcation.
Revised: As a result, GANs trained with them might be further biased towards the denser areas, similarly to the classiﬁer bias towards the majority class in imbalanced classiﬁcation.

Original: • More intrinsically, even when the training dataset “looks" balanced, training GANs is notoriously more unstable (sometimes even uncontrollable) than training classiﬁers, potentially constituting another source of mode collapse.
Revised: • More intrinsically, even when the training dataset “looks" balanced, training GANs is notoriously more unstable (sometimes even uncontrollable) than training classiﬁers, potentially constituting another source of mode collapse.

Original: One most common hurdle of GANs is the loss of diversity due to mode collapse (Goodfellow,
Revised: One most common hurdle of GANs is the loss of diversity due to mode collapse (Goodfellow,

Original: 2016), wherein the generator concentrates too large a probability mass on a few modes of the true distribution.
Revised: 2016), wherein the generator concentrates too large a probability mass on a few modes of the true distribution.

Original: Another widely reported issue, known as co-variate shift (Santurkar et al., 2017), could be viewed as a nuanced version of mode collapse.
Revised: Another widely reported issue, known as co-variate shift (Santurkar et al., 2017), could be viewed as a nuanced version of mode collapse.

Original: This paper seeks to explore: do the state-of-the-art GANs still suffer from mode collapse?
Revised: This paper seeks to explore: do the state-of-the-art GANs still suffer from mode collapse?

Original: Can we have a toolkit to detect that?
Revised: Can we have a toolkit to detect that?

Original: And if the mode collapse happens, is there any “easy and quick" remedy for calibrating the GAN’s learned distribution to alleviate the mode collapse?
Revised: And if the mode collapse happens, is there any “easy and quick" remedy for calibrating the GAN’s learned distribution to alleviate the mode collapse?

Original: Evaluation of Mode Collapse There are several popular metrics for GAN evaluation, e.g. Inception
Revised: Evaluation of Mode Collapse There are several popular metrics for GAN evaluation, e.g. Inception

Original: Score (IS) (Salimans et al., 2016), Fréchet Inception Distance (FID) (Heusel et al., 2017), MODE (Che et al., 2016) and birthday paradox based diversity test (Arora & Zhang, 2017).
Revised: Score (IS) (Salimans et al., 2016), Fréchet Inception Distance (FID) (Heusel et al., 2017), MODE (Che et al., 2016) and birthday paradox based diversity test (Arora & Zhang, 2017).

Original: IS, FID and
Revised: IS, FID and

Original: MODE score takes both visual ﬁdelity and diversity into account.
Revised: MODE score takes both visual ﬁdelity and diversity into account.

Original: Birthday paradox based diversity test gives a rough estimation of support size under the assumption of uniform sampling.
Revised: Birthday paradox based diversity test gives a rough estimation of support size under the assumption of uniform sampling.

Original: Recently, a classiﬁcation-based metric (Santurkar et al., 2017) was proposed for a quantitative assessment of the mode distribution learned by GANs.
Revised: Recently, a classiﬁcation-based metric (Santurkar et al., 2017) was proposed for a quantitative assessment of the mode distribution learned by GANs.

Original: However, their approach hinge on a classiﬁer trained on the original (balanced) GAN training set, with class labels known, available and well-deﬁned ( e.g. , object classes in CIFAR-10, or face gender in CelebA), making it non-straightforward to extend to data subjects where classes are hard to be deﬁned, and/or are not enumerable (e.g, open set problems).
Revised: However, their approach hinge on a classiﬁer trained on the original (balanced) GAN training set, with class labels known, available and well-deﬁned ( e.g. , object classes in CIFAR-10, or face gender in CelebA), making it non-straightforward to extend to data subjects where classes are hard to be deﬁned, and/or are not enumerable (e.g, open set problems).

Original: To tackle this problem, we propose a hypothesis test method by analyzing the clustering pattern of samples.
Revised: To tackle this problem, we propose a hypothesis test method by analyzing the clustering pattern of samples.

Original: We exploit a statistical tool from spatial analysis, called Ripley’s K function, to quantitatively measure the mode collapse.
Revised: We exploit a statistical tool from spatial analysis, called Ripley’s K function, to quantitatively measure the mode collapse.

Original: We demonstrate the application of our tool set in analyzing the bias in unconditional face image generation : a popular benchmark task nowadays for GANs, yet remaining rather unclear how to measure its mode collapse using existing tools since every generated identity is expected to be new.
Revised: We demonstrate the application of our tool set in analyzing the bias in unconditional face image generation : a popular benchmark task nowadays for GANs, yet remaining rather unclear how to measure its mode collapse using existing tools since every generated identity is expected to be new.

Original: The study of face identity generation bias has profound practical values for understanding facial privacy (Filipovych et al., 2011) and fairness (Holstein et al., 2018).
Revised: The study of face identity generation bias has profound practical values for understanding facial privacy (Filipovych et al., 2011) and fairness (Holstein et al., 2018).

Original: Using our tools, we ﬁnd the mode collapse still a prevailing problem in state-of-the-art face generation GANs (Karras et al., 2018; 2017), and further analyze several possible causes.
Revised: Using our tools, we ﬁnd the mode collapse still a prevailing problem in state-of-the-art face generation GANs (Karras et al., 2018; 2017), and further analyze several possible causes.

Original: Calibration Approaches on GAN Many approaches have been proposed to alleviate mode collapse problem, ranging from better optimization objectives (Arjovsky et al., 2017; Mao et al., 2017), to specialized builing blocks (Durugkar et al., 2016; Ghosh et al., 2018; Liu & Tuzel, 2016).
Revised: Calibration Approaches on GAN Many approaches have been proposed to alleviate mode collapse problem, ranging from better optimization objectives (Arjovsky et al., 2017; Mao et al., 2017), to specialized builing blocks (Durugkar et al., 2016; Ghosh et al., 2018; Liu & Tuzel, 2016).

Original: However, they require either tedious (re-)training, or at least the access to training data, as well as to model parameters: we refer to the existing methods as white-box approaches.
Revised: However, they require either tedious (re-)training, or at least the access to training data, as well as to model parameters: we refer to the existing methods as white-box approaches.

Original: In contrast, we are interested in an almost unexplored aspect: assuming some generation bias is known, how can be calibrate the GAN, without accessing either the training data or the current model parameters?
Revised: In contrast, we are interested in an almost unexplored aspect: assuming some generation bias is known, how can be calibrate the GAN, without accessing either the training data or the current model parameters?

Original: Such black-box calibration is desirable due to many practical demands: the training data might be protected or no longer available; the GAN model might be provided as a black box and cannot be altered ( e.g. , as APIs); or we simply want to adjust the generated distribution of any GAN with minimized re-training efforts.
Revised: Such black-box calibration is desirable due to many practical demands: the training data might be protected or no longer available; the GAN model might be provided as a black box and cannot be altered ( e.g. , as APIs); or we simply want to adjust the generated distribution of any GAN with minimized re-training efforts.

Original: For the ﬁrst time, we explore two “black-box” approaches to calibrate the GAN learned distribution, i.e., latent space reshaping via Gaussian mixture models, and importance sampling.
Revised: For the ﬁrst time, we explore two “black-box” approaches to calibrate the GAN learned distribution, i.e. , latent space reshaping via Gaussian mixture models, and importance sampling.

Original: They are observed to alleviate the mode collapse without re-touching training data, nor even needing any access to model parameters.
Revised: They are observed to alleviate the mode collapse without re-touching training data, nor even needing any access to model parameters.

Original:  R ELATED W ORKS 2. E VALUATION M ETRICS OF M ODE C OLLAPSE IN GAN S
Revised:  R ELATED W ORKS 2. E VALUATION M ETRICS OF M ODE C OLLAPSE IN GAN S

Original: GAN models are often observed to suffer from the mode collapse problem (Salimans et al., 2016); (Sutskever et al., 2015), where only small modes subsets of distribution are characterized by the generator.
Revised: GAN models are often observed to suffer from the mode collapse problem (Salimans et al., 2016); (Sutskever et al., 2015), where only small modes subsets of distribution are characterized by the generator.

Original: The problem is especially prevalent for high-dimensional data, e.g. face image generation, where the training samples are low-density w.r.t.
Revised: The problem is especially prevalent for high-dimensional data, e.g. face image generation, where the training samples are low-density w.r.t.

Original: the high-dimensional feature space.
Revised: the high-dimensional feature space.

Original: Salimans et al.
Revised: Salimans et al.

Original: (2016) presented the popular metric of Inception Score (IS) to measure the individual sample quality.
Revised: (2016) presented the popular metric of Inception Score (IS) to measure the individual sample quality.

Original: IS does not directly reﬂect the population-level generation quality, e.g. , the overﬁtting and loss of diversity.
Revised: IS does not directly reﬂect the population-level generation quality, e.g. , the overﬁtting and loss of diversity.

Original: It also requires pre-trained perceptual models on ImageNet or other speciﬁc datasets (Barratt & Sharma, 2018).
Revised: It also requires pre-trained perceptual models on ImageNet or other speciﬁc datasets (Barratt & Sharma, 2018).

Original: Heusel et al.
Revised: Heusel et al.

Original: (2017) propose the Fréchet Inception Distance (FID), which models the distribution of image features as multivariate Gaussian distribution and computes the distance between the distribution of real images and the distribution of fakes images.
Revised: (2017) propose the Fréchet Inception Distance (FID), which models the distribution of image features as multivariate Gaussian distribution and computes the distance between the distribution of real images and the distribution of fakes images.

Original: Unlike IS, FID can detect intra-class mode dropping.
Revised: Unlike IS, FID can detect intra-class mode dropping.

Original: However, the multivariate Gaussian distribution assumption hardly holds very well on real images, and low FID score cannot rule out the possibility of the generator’s simply copying the training data.
Revised: However, the multivariate Gaussian distribution assumption hardly holds very well on real images, and low FID score cannot rule out the possibility of the generator’s simply copying the training data.

Original: Besides the two most popular metrics, (Che et al., 2016) develop an assessment for both visual quality and variety of samples, known as the
Revised: Besides the two most popular metrics, (Che et al., 2016) develop an assessment for both visual quality and variety of samples, known as the

Original: MODE score and later shown to be similar to IS (Zhou et al., 2017).
Revised: MODE score and later shown to be similar to IS (Zhou et al., 2017).

Original: (Arora et al., 2018) and (Arora & Zhang, 2017) proposed a test based upon the birthday paradox for estimating the support size of the generated distribution.
Revised: (Arora et al., 2018) and (Arora & Zhang, 2017) proposed a test based upon the birthday paradox for estimating the support size of the generated distribution.

Original: Although the test can detect severe cases of mode collapse, it falls short in measuring how well a generator captures the true data distribution.
Revised: Although the test can detect severe cases of mode collapse, it falls short in measuring how well a generator captures the true data distribution.

Original: It also heavily relies on human annotation, making it challenging to scale up to larger-scale evaluation.
Revised: It also heavily relies on human annotation, making it challenging to scale up to larger-scale evaluation.

Original: (Santurkar et al., 2017) took a classiﬁcation-based perspective and view loss of diversity as a form of covariate shift.
Revised: (Santurkar et al., 2017) took a classiﬁcation-based perspective and view loss of diversity as a form of covariate shift.

Original: As we discussed above, their approach cannot be straightforwardly extended to data subjects without pre-known and closed-set class deﬁnition, in addition to the need of training an extra classiﬁer on the original labeled training set.
Revised: As we discussed above, their approach cannot be straightforwardly extended to data subjects without pre-known and closed-set class deﬁnition, in addition to the need of training an extra classiﬁer on the original labeled training set.

Original: 2. M ODEL C ALIBRATION A PPROACHES OF GAN S
Revised: 2. M ODEL C ALIBRATION A PPROACHES OF GAN S

Original: There are many efforts to address the mode collapse problem in GANs.
Revised: There are many efforts to address the mode collapse problem in GANs.

Original: Some focus on discriminators by introducing different divergence metrics (Metz et al., 2016) and optimization losses (Arjovsky et al., 2017; Mao et al., 2017).
Revised: Some focus on discriminators by introducing different divergence metrics (Metz et al., 2016) and optimization losses (Arjovsky et al., 2017; Mao et al., 2017).

Original: The minibatch discrimination scheme allows the discriminator to discriminate between whole mini-batches of samples instead of between individual samples.
Revised: The minibatch discrimination scheme allows the discriminator to discriminate between whole mini-batches of samples instead of between individual samples.

Original: (Durugkar et al., 2016) adopted multiple discriminators to alleviate mode collapse.
Revised: (Durugkar et al., 2016) adopted multiple discriminators to alleviate mode collapse.

Original: ModeGAN (Che et al., 2016) and VEEGAN (Srivastava et al., 2017) enforce the bijection mapping between the input noise vectors and generated images with additional encoder networks.
Revised: ModeGAN (Che et al., 2016) and VEEGAN (Srivastava et al., 2017) enforce the bijection mapping between the input noise vectors and generated images with additional encoder networks.

Original: Multiple generators (Ghosh et al., 2018) and weight-sharing generators (Liu & Tuzel, 2016) are developed to capture more modes of the distribution.
Revised: Multiple generators (Ghosh et al., 2018) and weight-sharing generators (Liu & Tuzel, 2016) are developed to capture more modes of the distribution.

Original: However, these approaches are designed to easily calibrating trained GANs.
Revised: However, these approaches are designed to easily calibrating trained GANs.

Original: A handful of existing works attempt to combine GANs with sampling methods to improve generation quality.
Revised: A handful of existing works attempt to combine GANs with sampling methods to improve generation quality.

Original: (Turner et al., 2018) introduced the Metropolis-Hastings generative adversarial network (MH-GAN).
Revised: (Turner et al., 2018) introduced the Metropolis-Hastings generative adversarial network (MH-GAN).

Original: The MH-GAN uses the learned discriminator from GAN training to build a wrapper for the generator for improved sampling, at the generation inference stage.
Revised: The MH-GAN uses the learned discriminator from GAN training to build a wrapper for the generator for improved sampling, at the generation inference stage.

Original: With a perfect discriminator, the wrapped generator can sample from the true distribution exactly even with a deﬁcient generator.
Revised: With a perfect discriminator, the wrapped generator can sample from the true distribution exactly even with a deﬁcient generator.

Original: (Azadi et al., 2018) proposed discriminator rejection sampling (DRS) for GANs, which performs rejection sampling on the outputs of the generator by using the probabilities given by the discriminator, to approximately correct errors in the generator’s distribution.
Revised: (Azadi et al., 2018) proposed discriminator rejection sampling (DRS) for GANs, which performs rejection sampling on the outputs of the generator by using the probabilities given by the discriminator, to approximately correct errors in the generator’s distribution.

Original: Yet still, these approaches are white-box calibration since both require access to trained discriminators (which might be even less available/accessible than the generator after a GAN is trained).
Revised: Yet still, these approaches are white-box calibration since both require access to trained discriminators (which might be even less available/accessible than the generator after a GAN is trained).

Original:  M ETHOD
Revised:  M ETHOD

Original: We instead study the bias of the most representative features of the generated face, the face identity distribution, that almost all face attribute can be derived based on the representations.
Revised: We intend to study the bias of the most representative features of the generated face, the face identity distribution, since almost all face attribute can be derived based on this representations.

Original: To detect face identity collapse, we are aiming to detect high-density regions in features space caused by any possible attribute non-diversiﬁed.
Revised: To detect face identity collapse, we are aiming to detect high-density regions in features space caused by any possible attribute non-diversiﬁed.

Original: Or, if being slightly imprecise in terms,(Santurkar et al., 2017) examined the marginalized distribution through some discrete categorical attributes’ lens, while ours looks at the joint distribution of all possible attributes in the continuous feature space holistically.
Revised: Or, if being slightly imprecise in terms,(Santurkar et al., 2017) examined the marginalized distribution through some discrete categorical attributes’ lens, while ours looks at the joint distribution of all possible attributes in the continuous feature space holistically.

Original: (cid:46)
Revised: (cid:46)

Original: Given a pre-trained generator G , an identity descriptor f id , a random distribution N (0 , Σ) , a neighbor distance threshold d 0 and a face embedding space distance range [ d b , d e ] d s ( d s : step size) (cid:46) S ← { I S 1 , · · · , I S m } // Randomly sampled m face images for each I S i ∈ S
Revised: Given a pre-trained generator G , an identity descriptor f id , a random distribution N (0 , Σ) , a neighbor distance threshold d 0 and a face embedding space distance range [ d b , d e ] d s ( d s : step size) (cid:46) S ← { I S 1 , · · · , I S m } // Randomly sampled m face images for each I S i ∈ S

Original: do //
Revised: do //

Original: Count neighbors within d 0 distance for each sampled I S i (cid:46) N I S
Revised: Count neighbors within d 0 distance for each sampled I S i (cid:46) N I S

Original: i ← N
Revised: i ← N

Original: ( I S i , S\ I S i , d 0 )
Revised: ( I S i , S\ I S i , d 0 )

Original: (cid:46) R obs ← { ˜ I S 1 , · · · , ˜ I S p } //
Revised: (cid:46) R obs ← { ˜ I S 1 , · · · , ˜ I S p } //

Original: Find the region for observation by selecting the top p face images in S with largest N I S
Revised: Find the region for observation by selecting the top p face images in S with largest N I S

Original: i (cid:46) R ref ← { ˆ I S 1 , · · · , ˆ I S q } // Find the region for reference by randomly selecting q face images from S (cid:46) T ← { I T 1 , · · · , I T M } // Randomly sampled M face images ( M (cid:29) m ) for each d in [ d b , d e ] d s do for each ˜
Revised: i (cid:46) R ref ← { ˆ I S 1 , · · · , ˆ I S q } // Find the region for reference by randomly selecting q face images from S (cid:46) T ← { I T 1 , · · · , I T M } // Randomly sampled M face images ( M (cid:29) m ) for each d in [ d b , d e ] d s do for each ˜

Original: I S i ∈
Revised: I S i ∈

Original: R obs do //
Revised: R obs do //

Original: Count neighbors within d distance for each ˜ I S i in R obs (cid:46) N d
Revised: Count neighbors within d distance for each ˜ I S i in R obs (cid:46) N d

Original: ˜
Revised: ˜

Original: I S
Revised: I S

Original: i ← N (˜ I S
Revised: i ← N (˜ I S

Original: i , T , d ) for each ˆ I S i ∈
Revised: i , T , d ) for each ˆ I S i ∈

Original: R ref do //
Revised: R ref do //

Original: Count neighbors within d distance for each ˆ I S i in R ref (cid:46) N
Revised: Count neighbors within d distance for each ˆ I S i in R ref (cid:46) N

Original: d ˆ I S i ← N
Revised: d ˆ I S i ← N

Original: (ˆ I S
Revised: (ˆ I S

Original: i , T , d ) (cid:46) Compute the pointwise conﬁdence regions of [ N d ˆ I Si | 1 − α 2 , N d
Revised: i , T , d ) (cid:46) Compute the pointwise conﬁdence regions of [ N d ˆ I Si | 1 − α 2 , N d

Original: ˆ I Si | α 2 ] for each d ∈ [ d b , d e ] d s , at conﬁdence level of α (default 0.05).
Revised: ˆ I Si | α 2 ] for each d ∈ [ d b , d e ] d s , at conﬁdence level of α (default 0.05).

Original: The intervals between the upper and lower conﬁdence bounds for all samples in R ref deﬁne the conﬁdence band (Eubank & Speckman, 1993).
Revised: The intervals between the upper and lower conﬁdence bounds for all samples in R ref deﬁne the conﬁdence band (Eubank & Speckman, 1993).

Original: (cid:46) Reject the hypothesis that the clustering pattern of R obs is the same as that of R ref , if the curve of N d ˜ I Si falls outside of the conﬁdence band.
Revised: (cid:46) Reject the hypothesis that the clustering pattern of R obs is the same as that of R ref , if the curve of N d ˜ I Si falls outside of the conﬁdence band.

Original: Given an unconditional face generator G and an identity descriptor f id , we sample images I =
Revised: Given an unconditional face generator G and an identity descriptor f id , we sample images I =

Original: G ( z ) using a random distribution z ∼ N (0 , Σ) .
Revised: G ( z ) using a random distribution z ∼ N (0 , Σ) .

Original: The unit vector f id ( I ) describes the identity feature in the face embedding space.
Revised: The unit vector f id ( I ) describes the identity feature in the face embedding space.

Original: The normalized cosine distance between image I 

Revised: The normalized cosine distance between image I

Original: and I 1 is deﬁned as:
Revised: and I 1 is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: For a given anchor face image I 0 , a distance threshold d 0 and a collection of randomly sampled face images S , the neighboring function N ( I 0 , S , d 0 ) is deﬁned to compute the number of neighbors within d 0 distance of I 0 , among all images in S :
Revised: For a given anchor face image I 0 , a distance threshold d 0 and a collection of randomly sampled face images S , the neighboring function N ( I 0 , S , d 0 ) is deﬁned to compute the number of neighbors within d 0 distance of I 0 , among all images in S :

Original: [Equation]
Revised: [Equation]

Original: We refer to the tool of Ripley’s K function (Dixon, 2014), a spatial analysis method used to describe point patterns over a given area of interest.
Revised: We refer to the tool of Ripley’s K function (Dixon, 2014), a spatial analysis method used to describe point patterns over a given area of interest.

Original: Ripley’s K function can be used to determine if the points of interest appears to be dispersed, clustered, or randomly distributed throughout the area.
Revised: Ripley’s K function can be used to determine if the points of interest appears to be dispersed, clustered, or randomly distributed throughout the area.

Original: Our deﬁned neighboring function N ( I 0 , S , d 0 ) serves as a surrogate of the Ripley’s K function K ( d ) .
Revised: Our deﬁned neighboring function N ( I 0 , S , d 0 ) serves as a surrogate of the Ripley’s K function K ( d ) .

Original: Hypothesis Testing Given an observed high-identity-density region R obs and a reference region
Revised: Hypothesis Testing Given an observed high-identity-density region R obs and a reference region

Original: R ref , we want to test the hypothesis that the clustering pattern of R obs is the same as R ref .
Revised: R ref , we want to test the hypothesis that the clustering pattern of R obs is the same as R ref .

Original: We use
Revised: We use

Original: N to get the clustering pattern for the anchor images in R obs and R ref respectively.
Revised: N to get the clustering pattern for the anchor images in R obs and R ref respectively.

Original: We can reject the hypothesis if the clustering pattern of R obs is signiﬁcantly different from R ref .
Revised: We can reject the hypothesis if the clustering pattern of R obs is signiﬁcantly different from R ref .

Original: The detailed algorithm is outlined in Algorithm 1.
Revised: The detailed algorithm is outlined in Algorithm 1.

Original:  E MPIRICAL S TUDY AND A NALYSIS
Revised:  E MPIRICAL S TUDY AND A NALYSIS

Original: We choose two state-of-the-art GANs: PGGAN (Karras et al., 2017) and StyleGAN (Karras et al., 2018), as our model subjects of study.
Revised: We choose two state-of-the-art GANs: PGGAN (Karras et al., 2017) and StyleGAN (Karras et al., 2018), as our model subjects of study.

Original: Both are known to be able to produce high-resolution, realistic and diverse images.
Revised: Both are known to be able to produce high-resolution, realistic and diverse images.

Original: We ﬁnd that the observations below drawn from the two models also generalize to a few other GAN models.
Revised: We ﬁnd that the observations below drawn from the two models also generalize to a few other GAN models.

Original: We choose the CelebAHQ benchmark (Karras et al., 2017) and FFHQ benchmark (Karras et al., 2018) as our data subject of study.
Revised: We choose the CelebAHQ benchmark (Karras et al., 2017) and FFHQ benchmark (Karras et al., 2018) as our data subject of study.

Original: Both benchmarks are composed of diverse and realistic face images.
Revised: Both benchmarks are composed of diverse and realistic face images.

Original: All images are 1024 × 1024 resolution unless otherwise speciﬁed.
Revised: All images are 1024 × 1024 resolution unless otherwise speciﬁed.

Original: We use ensemble model of InsightFace (Deng et al., 2019b;
Revised: We use ensemble model of InsightFace (Deng et al., 2019b;

Original: Guo et al., 2018;
Revised: Guo et al., 2018;

Original: Deng et al., 2018; 2019a), FaceNet (Schroff et al., 2015) and CosFace (Wang et al., 2018) as f id to serve as the face identity descriptor.
Revised: Deng et al., 2018; 2019a), FaceNet (Schroff et al., 2015) and CosFace (Wang et al., 2018) as f id to serve as the face identity descriptor.

Original: We emphasize that the due diligence of “sanity check” has been performed on those classiﬁers, e.g. , their face recognition results are manually inspected one-by-one and conﬁrmed to be highly reliable on the generated images.
Revised: We emphasize that the due diligence of “sanity check” has been performed on those classiﬁers, e.g. , their face recognition results are manually inspected one-by-one and conﬁrmed to be highly reliable on the generated images.

Original: q ( |R ref | ) is set to be 1000 .
Revised: q ( |R ref | ) is set to be 1000 .

Original: We empirically set d b , d e and d s are set to be 0 . 1 , 0 .
Revised: We empirically set d b , d e and d s are set to be 0 . 1 , 0 .

Original: 5 and 0 .
Revised: 5 and 0 .

Original: 01 respectively.
Revised: 01 respectively.

Original: 4. O BSERVATION OF THE M ODE C OLLAPSE
Revised: 4. O BSERVATION OF THE M ODE C OLLAPSE

Original: Mode Collapse Analysis For both StyleGAN and PGGAN, despite of the observed diversity and high quality of their generated images, we empirically ﬁnd some high-density regions in both learned distributions.
Revised: Mode Collapse Analysis For both StyleGAN and PGGAN, despite of the observed diversity and high quality of their generated images, we empirically ﬁnd some high-density regions in both learned distributions.

Original: Figure 1 shows that the clustering pattern of R obs is signiﬁcantly different from that of
Revised: Figure 1 shows that the clustering pattern of R obs is signiﬁcantly different from that of

Original: R ref , showing that even the learned distributions of two currently best models have strong dense regions towards some speciﬁc identities.
Revised: R ref , showing that even the learned distributions of two currently best models have strong dense regions towards some speciﬁc identities.

Original: For simplicity, our study target is the worst-case dense mode, i.e. the identity with the largest number of neighbors within a given distance threshold.
Revised: For simplicity, our study target is the worst-case dense mode, i.e. the identity with the largest number of neighbors within a given distance threshold.

Original: Consistency of the Dense Mode
Revised: Consistency of the Dense Mode

Original: The dense region R obs is obtained by selecting the top p images in S with the largest number of neighbors.
Revised: The dense region R obs is obtained by selecting the top p images in S with the largest number of neighbors.

Original: In order to test the consistency of the worst-case dense mode I m against sampling, we visualize the I m w.r.t.
Revised: In order to test the consistency of the worst-case dense mode I m against sampling, we visualize the I m w.r.t.

Original: different size of S in Figure 2.
Revised: different size of S in Figure 2.

Original: We consistently observe roughly the same identity as the sampling size increases.
Revised: We consistently observe roughly the same identity as the sampling size increases.

Original: I m can be reliably obtained even when |S| = 1 k .
Revised: I m can be reliably obtained even when |S| = 1 k .

Original: The consistency of I m demonstrate that the support size of I m is unnegligible.
Revised: The consistency of I m demonstrate that the support size of I m is unnegligible.

Original: 4. E MPIRICAL S TUDY OF THE C AUSE OF M ODE C OLLAPSE
Revised: 4. E MPIRICAL S TUDY OF THE C AUSE OF M ODE C OLLAPSE

Original: We hypothesize multiple factors that may potentially lead to the observed dense mode of face identity.
Revised: We hypothesize multiple factors that may potentially lead to the observed dense mode of face identity.

Original: We perform additional experiments, aiming to validate one by one: unfortunately, none of them was observed to reduce the observed mode collapse.
Revised: We perform additional experiments, aiming to validate one by one: unfortunately, none of them was observed to reduce the observed mode collapse.

Original: That implies the existence of some more intrinsic reason for the mode collapse in GAN, which we leave for future exploration.
Revised: That implies the existence of some more intrinsic reason for the mode collapse in GAN, which we leave for future exploration.

Original: Imbalance of Training Data?
Revised: Imbalance of Training Data?

Original: CelebAHQ is a highly imbalanced dataset: among its 30 , 000 highresolution face images of 6 , 217 different celebrities, the largest identity class has 28 images and the smallest one has only 1.
Revised: CelebAHQ is a highly imbalanced dataset: among its 30 , 000 highresolution face images of 6 , 217 different celebrities, the largest identity class has 28 images and the smallest one has only 1.

Original: Would a balanced dataset alleviate the mode collapse?
Revised: Would a balanced dataset alleviate the mode collapse?

Original: [Figure]
Revised: [Figure]

Original: The blue region is a conﬁdence band formed by the pointwise intervals between the upper and lower conﬁdence bounds for all identities in R ref .
Revised: The blue region is a conﬁdence band formed by the pointwise intervals between the upper and lower conﬁdence bounds for all identities in R ref .

Original: The red curve is the neighboring function curve for identity in R obs , the worst-case dense mode.
Revised: The red curve is the neighboring function curve for identity in R obs , the worst-case dense mode.

Original: We empirically set m ( |S| ) to be 100 , 000 and M ( |T | ) to be 10 , 000 , 000 .
Revised: We empirically set m ( |S| ) to be 100 , 000 and M ( |T | ) to be 10 , 000 , 000 .

Original: To study the worst-case dense mode, p ( |R obs | ) is set to be 1 .
Revised: To study the worst-case dense mode, p ( |R obs | ) is set to be 1 .

Original: [Figure]
Revised: [Figure]

Original: We turn to the Flickr-Faces-HQ Dataset (FFHQ), a high-quality human face dataset created in (Karras et al., 2018), consisting of 70 , 000 high-resolution face images, without repeated identities (we manually examined the dataset to ensure so.
Revised: We turn to the Flickr-Faces-HQ Dataset (FFHQ), a high-quality human face dataset created in (Karras et al., 2018), consisting of 70 , 000 high-resolution face images, without repeated identities (we manually examined the dataset to ensure so.

Original: It is thus “balanced” in terms of identity, in the sense that each identity class has one sample.
Revised: It is thus “balanced” in terms of identity, in the sense that each identity class has one sample.

Original: We train StyleGAN on FFHQ: somehow surprisingly, the mode collapse persists and seems no less than StyleGAN on CelebAHQ, as shown in Figure 1c and 1d.
Revised: We train StyleGAN on FFHQ: somehow surprisingly, the mode collapse persists and seems no less than StyleGAN on CelebAHQ, as shown in Figure 1c and 1d.

Original: [Figure]
Revised: [Figure]

Original: | ) to be 1 , 000 , 000 .
Revised: | ) to be 1 , 000 , 000 .

Original: To study the worst-case dense mode, p ( |R obs | ) and is set to be 1 .
Revised: To study the worst-case dense mode, p ( |R obs | ) and is set to be 1 .

Original: Randomness during Initialization/Optimization?
Revised: Randomness during Initialization/Optimization?

Original: We repeat training StyleGAN on CelebAHQ ( 128 × 128 ) for 10 times.
Revised: We repeat training StyleGAN on CelebAHQ ( 128 × 128 ) for 10 times.

Original: The experimental results are shown in Figure 3a, with the shaded areas denoting the variances.
Revised: The experimental results are shown in Figure 3a, with the shaded areas denoting the variances.

Original: Despite the variance for the neighboring function curves plotted for repeated experiments, a large gap between the curves of R obs and R ref can be consistently observed.
Revised: Despite the variance for the neighboring function curves plotted for repeated experiments, a large gap between the curves of R obs and R ref can be consistently observed.

Original: Unﬁtting/Overﬁtting in Training?
Revised: Unﬁtting/Overﬁtting in Training?

Original: We train StyleGAN on CelebAHQ ( 128 × 128 ) again, and store model checkpoints at iteration 7707 (FID = 7.67, same hereinafter), 8307 (7.02), 8908 (6.89), 9508 (6.63), 10108 (6.41), and 12000 (6.32).
Revised: We train StyleGAN on CelebAHQ ( 128 × 128 ) again, and store model checkpoints at iteration 7707 (FID = 7.67, same hereinafter), 8307 (7.02), 8908 (6.89), 9508 (6.63), 10108 (6.41), and 12000 (6.32).

Original: We plot their corresponding neighboring function curves in Figure 3b.
Revised: We plot their corresponding neighboring function curves in Figure 3b.

Original: Similarly, despite the variances, the identity mode collapse persists due to the consistent large gap between R obs and R ref curves.
Revised: Similarly, despite the variances, the identity mode collapse persists due to the consistent large gap between R obs and R ref curves.

Original: Model Architecture Differences?
Revised: Model Architecture Differences?

Original: Both StyleGAN and PGGAN progressively grow their architectures that can generate images of different resolutions: 128 , 256 , 512 and 1024 .
Revised: Both StyleGAN and PGGAN progressively grow their architectures that can generate images of different resolutions: 128 , 256 , 512 and 1024 .

Original: Utilizing this property, we train StyleGAN and PGGAN on CelebAHQ-128, CelebAHQ-256, CelebAHQ-512 and
Revised: Utilizing this property, we train StyleGAN and PGGAN on CelebAHQ-128, CelebAHQ-256, CelebAHQ-512 and

Original: CelebAHQ-1024 respectively, and plot the neighboring function curves correspondingly.
Revised: CelebAHQ-1024 respectively, and plot the neighboring function curves correspondingly.

Original: According to Figures 3c and 3d, varying the architectures does not eliminate the mode collapse either.
Revised: According to Figures 3c and 3d, varying the architectures does not eliminate the mode collapse either.

Original:  B LACK - BOX C ALIBRATION A PPROACHES
Revised:  B LACK - BOX C ALIBRATION A PPROACHES

Original: Given a pre-trained generator G and target dense mode for alleviation, the goals of calibration are three-fold: (1) the density of the mode is maximally alleviated; (2) the diversity and quality of the generated images (measured by FID) are minimally sacriﬁced; and (3) the calibration is black-box, which does not require access to training data or model parameters.
Revised: Given a pre-trained generator G and target dense mode for alleviation, the goals of calibration are three-fold: (1) the density of the mode is maximally alleviated; (2) the diversity and quality of the generated images (measured by FID) are minimally sacriﬁced; and (3) the calibration is black-box, which does not require access to training data or model parameters.

Original: We propose two calibration approaches: reshaping latent space via Gaussian mixture models and importance sampling.
Revised: We propose two calibration approaches: reshaping latent space via Gaussian mixture models and importance sampling.

Original: They operate on the latent codes, and require no modiﬁcation of the trained model, nor even any access to the model parameters or training data, making them “black-box".
Revised: They operate on the latent codes, and require no modiﬁcation of the trained model, nor even any access to the model parameters or training data, making them “black-box".

Original: Both approaches are evaluated with StyleGAN trained on CelebAHQ-128.
Revised: Both approaches are evaluated with StyleGAN trained on CelebAHQ-128.

Original: For simplicity, we only target to eliminating the worst-case dense mode I m , i.e. the identity with the largest number of neighbors within a speciﬁed distance threshold.
Revised: For simplicity, we only target to eliminating the worst-case dense mode I m , i.e. the identity with the largest number of neighbors within a speciﬁed distance threshold.

Original: 5. R ESHAPING L ATENT S PACE VIA G AUSSIAN M IXTURE M ODELS
Revised: 5. R ESHAPING L ATENT S PACE VIA G AUSSIAN M IXTURE M ODELS

Original: Since we consistently observe close neighbors to I m , when interpolating near I m , we hypothesize that the latent codes of a dense mode I m lay on a smooth manifold.
Revised: Since we consistently observe close neighbors to I m , when interpolating near I m , we hypothesize that the latent codes of a dense mode I m lay on a smooth manifold.

Original: Based on this assumption, we attempt to re-shape that into a Gaussian mixture.
Revised: Based on this assumption, we attempt to re-shape that into a Gaussian mixture.

Original: 5.1. M ETHOD D ESCRIPTION
Revised: 5.1. M ETHOD D ESCRIPTION

Original: The original latent space distribution φ ( z ; θ 0 ) can be approximated with a mixture of Gaussian distributions
Revised: The original latent space distribution φ ( z ; θ 0 ) can be approximated with a mixture of Gaussian distributions

Original: K (cid:88) i =
Revised: K (cid:88) i =

Original: w i φ ( z ; θ i ) .
Revised: w i φ ( z ; θ i ) .

Original: We randomly sample N latent code and use K -means to explore θ i =
Revised: We randomly sample N latent code and use K -means to explore θ i =

Original: ( µ i , σ i ) .
Revised: ( µ i , σ i ) .

Original: We denote p ( I m ) as the probability of sampling the worst-case dense mode I m .
Revised: We denote p ( I m ) as the probability of sampling the worst-case dense mode I m .

Original: p ( I m ) =
Revised: p ( I m ) =

Original: (cid:90) p ( I m | z ) φ ( z ; θ 0 )
Revised: (cid:90) p ( I m | z ) φ ( z ; θ 0 )

Original: dz =
Revised: dz =

Original: K (cid:88) i =
Revised: K (cid:88) i =

Original: w i (cid:90) p ( I m | z ) φ i ( z ; θ i ) dz .
Revised: w i (cid:90) p ( I m | z ) φ i ( z ; θ i ) dz .

Original: If p ( I m | θ i ) is large, we reduce w i to make the overall p ( I m ) small.
Revised: If p ( I m | θ i ) is large, we reduce w i to make the overall p ( I m ) small.

Original: p ( I m | θ i ) is estimated by the number of neighbors within ddistance to I m in cluster C i , i.e. N ( I m , C i , d 0 ) .
Revised: p ( I m | θ i ) is estimated by the number of neighbors within ddistance to I m in cluster C i , i.e. N ( I m , C i , d 0 ) .

Original: 5.1. E XPERIMENTS
Revised: 5.1. E XPERIMENTS

Original: [Figure]
Revised: [Figure]

Original: Starting from
Revised: Starting from

Original: StyleGAN model pretrained on CelebAHQ-128, we aim at alleviating the collapse on the worst-case dense mode I m in R obs with the largest number of neighbors.
Revised: StyleGAN model pretrained on CelebAHQ-128, we aim at alleviating the collapse on the worst-case dense mode I m in R obs with the largest number of neighbors.

Original: We reshape the latent space of M via Gaussian mixture models to get the new model M (cid:48) .
Revised: We reshape the latent space of M via Gaussian mixture models to get the new model M (cid:48) .

Original: We get the new worst-case dense mode I (cid:48) m in the new region R (cid:48) obs with the largest number of neighbors.
Revised: We get the new worst-case dense mode I (cid:48) m in the new region R (cid:48) obs with the largest number of neighbors.

Original: We next randomly sample 10 6 images from the original Gaussian distribution and new GMM distribution, to form T and T (cid:48) respectively.
Revised: We next randomly sample 10 6 images from the original Gaussian distribution and new GMM distribution, to form T and T (cid:48) respectively.

Original: We then plot the neighboring function curves for I m in T and T (cid:48) , and I (cid:48) m in T and T (cid:48) respectively.
Revised: We then plot the neighboring function curves for I m in T and T (cid:48) , and I (cid:48) m in T and T (cid:48) respectively.

Original: We expect the reshaping latent space via Gaussian mixture models to alleviate the worst-case dense mode with the minimal sacriﬁce of generated image quality and diversity.
Revised: We expect the reshaping latent space via Gaussian mixture models to alleviate the worst-case dense mode with the minimal sacriﬁce of generated image quality and diversity.

Original: As shown in Figure 4, the latent space reshaping could suppress the clustering of I m (indicated by a large gap between the two red curves) without intensifying the clustering of I (cid:48) m (indicated by a little gap between the two green curves), resulting in a reduction of mode collapse on I m .
Revised: As shown in Figure 4, the latent space reshaping could suppress the clustering of I m (indicated by a large gap between the two red curves) without intensifying the clustering of I (cid:48) m (indicated by a little gap between the two green curves), resulting in a reduction of mode collapse on I m .

Original: Such an alleviation is achieved with an unnoticeable degradation of generation quality, with FID increasing from 5.93 ( M ) to 5.95 ( M (cid:48) ).
Revised: Such an alleviation is achieved with an unnoticeable degradation of generation quality, with FID increasing from 5.93 ( M ) to 5.95 ( M (cid:48) ).

Original: The large overlapping between conﬁdence bands N M R ref and N M (cid:48)
Revised: The large overlapping between conﬁdence bands N M R ref and N M (cid:48)

Original: R ref shows that the diversity of generation is not sacriﬁced either.
Revised: R ref shows that the diversity of generation is not sacriﬁced either.

Original: 5. I MPORTANCE S AMPLING
Revised: 5. I MPORTANCE S AMPLING

Original: Under the same hypothesis of smooth manifold in section 5.1, the high-density region corresponding to the worst-case dense mode I m can be approximated with a convex hull.
Revised: Under the same hypothesis of smooth manifold in section 5.1, the high-density region corresponding to the worst-case dense mode I m can be approximated with a convex hull.

Original: 5.2. M ETHOD D ESCRIPTION
Revised: 5.2. M ETHOD D ESCRIPTION

Original: Importance sampling is a variance reduction strategy in the Monte Carlo method.
Revised: Importance sampling is a variance reduction strategy in the Monte Carlo method.

Original: Let the estimated neighboring function densities for the dense and sparse regions be p 1 and p 2 respectively.
Revised: Let the estimated neighboring function densities for the dense and sparse regions be p 1 and p 2 respectively.

Original: We accept the samples from G falling in the high-density region with a probability of p 2 /p 1 , so that the calibrated densities can match.
Revised: We accept the samples from G falling in the high-density region with a probability of p 2 /p 1 , so that the calibrated densities can match.

Original: We approximate the high-density region with a convex hull formed by the collection of latent codes
Revised: We approximate the high-density region with a convex hull formed by the collection of latent codes

Original: Z I m corresponding to the identities similar to I m :
Revised: Z I m corresponding to the identities similar to I m :

Original: [Equation]
Revised: [Equation]

Original: 5.2. E XPERIMENT
Revised: 5.2. E XPERIMENT

Original: The experiment setting is mostly similar to the reshaping latent space via the Gaussian mixture models case.
Revised: The experiment setting is mostly similar to the reshaping latent space via the Gaussian mixture models case.

Original: We integrate importance sampling to the latent code generation stage.
Revised: We integrate importance sampling to the latent code generation stage.

Original: Given the dense mode I m , we can ﬁnd the collection of latent codes Z I m via sampling:
Revised: Given the dense mode I m , we can ﬁnd the collection of latent codes Z I m via sampling:

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: , before/after importance sampling.
Revised: , before/after importance sampling.

Original: Z I m is obtained from the top 10 2 latent codes whose corresponding images have the smallest distances (1) to I m , among the 10 6 random samples.
Revised: Z I m is obtained from the top 10 2 latent codes whose corresponding images have the smallest distances (1) to I m , among the 10 6 random samples.

Original: We randomly sample 10 6 images from M and M (cid:48) to form T and
Revised: We randomly sample 10 6 images from M and M (cid:48) to form T and

Original: T (cid:48) respectively.
Revised: T (cid:48) respectively.

Original: We plot the neighboring function curves for I M in T and T (cid:48) , and
Revised: We plot the neighboring function curves for I M in T and T (cid:48) , and

Original: I M (cid:48) in T and T (cid:48) respectively.
Revised: I M (cid:48) in T and T (cid:48) respectively.

Original: As shown in Figure 5, the mode collapse is again alleviated (indicated by a gap between the two red curves), without intensifying the clustering of I (cid:48) m (indicated by a little gap between the two green curves), while FID only marginally increases from 5.93 ( M ) to 5.94 ( M (cid:48) ).
Revised: As shown in Figure 5, the mode collapse is again alleviated (indicated by a gap between the two red curves), without intensifying the clustering of I (cid:48) m (indicated by a little gap between the two green curves), while FID only marginally increases from 5.93 ( M ) to 5.94 ( M (cid:48) ).

Original: The conﬁdence band N M R ref is overlapped with N M (cid:48)
Revised: The conﬁdence band N M R ref is overlapped with N M (cid:48)

Original: R ref , showing no loss of the diversity.
Revised: R ref , showing no loss of the diversity.

Original: Additionally, in the appendix, we show a white-box counterpart to the importance sampling approach, where the latent codes Z I m are obtained via explicit optimization (accessing and altering model parameters).
Revised: Additionally, in the appendix, we show a white-box counterpart to the importance sampling approach, where the latent codes Z I m are obtained via explicit optimization (accessing and altering model parameters).

Original: The white-box approach does not seem to notably outperform than our above black-box way, implying the relative effectiveness of the latter.
Revised: The white-box approach does not seem to notably outperform than our above black-box way, implying the relative effectiveness of the latter.

Original:  D ISCUSSIONS AND F UTURE W ORK
Revised:  D ISCUSSIONS AND F UTURE W ORK

Original: This paper is intended as a pilot study on exploring the mode collapse issue of GANs.
Revised: This paper is intended as a pilot study on exploring the mode collapse issue of GANs.

Original: Using face generation as a study subject, we quantify the general mode collapse via statistical tools, discuss and verify possible causes, as well as propose two black-box calibration approaches for the ﬁrst time to alleviate the mode collapse.
Revised: Using face generation as a study subject, we quantify the general mode collapse via statistical tools, discuss and verify possible causes, as well as propose two black-box calibration approaches for the ﬁrst time to alleviate the mode collapse.

Original: Despite the preliminary success, the current study remains to be limited in many ways.
Revised: Despite the preliminary success, the current study remains to be limited in many ways.

Original: First, there are inevitably prediction errors for the identity descriptors from generated images, even we have performed our best efforts to ﬁnd the three most accurate descriptor predictions.
Revised: First, there are inevitably prediction errors for the identity descriptors from generated images, even we have performed our best efforts to ﬁnd the three most accurate descriptor predictions.

Original: Moreover, the fundamental causes of GAN mode collapse demand deeper understandings.
Revised: Moreover, the fundamental causes of GAN mode collapse demand deeper understandings.

Original: Besides, the two calibration approaches only handle one worst-case dense mode, leaving much improvement room open for future work.
Revised: Besides, the two calibration approaches only handle one worst-case dense mode, leaving much improvement room open for future work.