Original: C URRICULUM L EARNING FOR D EEP G ENERATIVE M ODELS 
Revised: C URRICULUM L EARNING FOR D EEP G ENERATIVE M ODELS WITH C

Original: WITH C LUSTERING
Revised: LUSTERING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Training generative models like Generative Adversarial Network (GAN) is challenging for noisy data.
Revised: Training generative models like Generative Adversarial Networks (GANs) is challenging for noisy data.

Original: A novel curriculum learning algorithm pertaining to clustering is proposed to address this issue in this paper.
Revised: A novel curriculum learning algorithm pertaining to clustering is proposed to address this issue in this paper.

Original: The curriculum construction is based on the centrality of underlying clusters in data points.
Revised: The curriculum construction is based on the centrality of underlying clusters in data points.

Original: The data points of high centrality takes  priority of being fed into generative models during training.
Revised: The data points of high centrality takes the priority of being fed into generative models during training.

Original: To make our algorithm scalable to large-scale data, the active set is devised, in the sense that every round of training proceeds only on an active subset containing a small fraction of already trained data and the incremental data of lower centrality.
Revised: To make our algorithm scalable to large-scale data, the active set is devised, in the sense that every round of training proceeds only on an active subset containing a small fraction of already trained data and the incremental data of lower centrality.

Original: Moreover, the geometric analysis is presented to interpret the necessity of cluster curriculum for generative models.
Revised: Moreover, the geometric analysis is presented to interpret the necessity of cluster curriculum for generative models.

Original: The experiments on  cat and human-face data validate that our algorithm is able to learn the optimal generative models (e.g.
Revised: The experiments on the cat and human-face data validate that our algorithm is able to learn the optimal generative models (e.g.

Original: ProGAN)
Revised: ProGAN)

Original: with respect to speciﬁed quality metrics for noisy data.
Revised: with respect to speciﬁed quality metrics for noisy data.

Original: An interesting ﬁnding is that the optimal cluster curriculum is closely related to the critical point of the geometric percolation process formulated in the paper.
Revised: An interesting ﬁnding is that the optimal cluster curriculum is closely related to the critical point of the geometric percolation process formulated in the paper.

Revised: Besides, the centrality-accuracy curve in our algorithm provides an effective way of visualizing the model state with curriculum training.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep generative models pique researchers’ interest in the past decade.
Revised: Deep generative models have piqued researchers’ interest in the past decade.

Original: The fruitful progress has been achieved on this topic, such as auto-encoder (Hinton & Salakhutdinov, 2006) and variational auto-encoder (VAE) (Kingma & Welling, 2013; Rezende et al., 2014), generative adversarial network (GAN) (Goodfellow et al., 2014; Radford et al., 2016;
Revised: The fruitful progress has been achieved on this topic, such as auto-encoder (Hinton & Salakhutdinov, 2006) and variational auto-encoder (VAE) (Kingma & Welling, 2013; Rezende et al., 2014), generative adversarial network (GAN) (Goodfellow et al., 2014; Radford et al., 2016;

Original: Arjovsky et al., 2017), normalizing ﬂow (Rezende & Mohamed, 2015; Dinh et al., 2015; 2017; Kingma & Dhariwal, 2018), and autoregressive models (van den Oord et al., 2016b;a; 2017).
Revised: Arjovsky et al., 2017), normalizing ﬂow (Rezende & Mohamed, 2015; Dinh et al., 2015; 2017; Kingma & Dhariwal, 2018), and autoregressive models (van den Oord et al., 2016b;a; 2017).

Original: However, it is non-trial to train a deep generative model that can converge to a proper minimum of  associated optimization.
Revised: However, it is non-trivial to train a deep generative model that can converge to a proper minimum of the associated optimization.

Original: For example,
Revised: For example,

Original: GAN suffers non-stability, model collapse, and generative distortion during training.
Revised: GAN suffers non-stability, mode collapse, and generative distortion during training.

Original: Many insightful algorithms have been proposed to circumvent those issues, including feature engineering (Salimans et al., 2016), various discrimination metrics (Mao et al., 2016; Arjovsky et al., 2017; Berthelot et al., 2017), distinctive gradient penalties (Gulrajani et al., 2017; Mescheder et al., 2018), spectral normalization to discriminator (Miyato et al., 2018), and orthogonal regularization to generator (Brock et al., 2019).
Revised: Many insightful algorithms have been proposed to circumvent those issues, including feature engineering (Salimans et al., 2016), various discrimination metrics (Mao et al., 2016; Arjovsky et al., 2017; Berthelot et al., 2017), distinctive gradient penalties (Gulrajani et al., 2017; Mescheder et al., 2018), spectral normalization to discriminator (Miyato et al., 2018), and orthogonal regularization to generator (Brock et al., 2019).

Original: What is particularly of interest is that the breakthrough for GAN has been made with a simple technique of progressively growing neural networks of generator and discriminator from low-resolution images to high-resolution counterparts (Karras et al., 2018a).
Revised: What is particularly of interest is that the breakthrough for GANs has been made with a simple technique of progressively growing neural networks of generators and discriminators from low-resolution images to high-resolution counterparts (Karras et al., 2018a).

Original: This kind of progressive growing also helps push the state of the art to a new level by enabling StyleGAN to produce photo-realistic and detail-sharp results (Karras et al., 2018b), shedding new light on wide applications of GANs in solving real problems.
Revised: This kind of progressive growing also helps push the state of the arts to a new level by enabling StyleGAN to produce photo-realistic and detail-sharp results (Karras et al., 2018b), shedding new light on wide applications of GANs in solving real problems.

Original: This idea of progressive learning is actually a general manner of cognition process (Elman, 1993; Oudeyer et al., 2007), which has been formally named curriculum learning in machine learning (Bengio et al., 2009).
Revised: This idea of progressive learning is actually a general manner of cognition process (Elman, 1993; Oudeyer et al., 2007), which has been formally named curriculum learning in machine learning (Bengio et al., 2009).

Original: The central topic of this paper is to explore a new curriculum for training deep generative models.
Revised: The central topic of this paper is to explore a new curriculum for training deep generative models.

Original: To facilitate robust training of deep generative models with noisy data, we propose curriculum learning with clustering.
Revised: To facilitate robust training of deep generative models with noisy data, we propose curriculum learning with clustering.

Original: The key contributions are listed as follows:
Revised: The key contributions are listed as follows:

Original: • We ﬁrst summarize four representative curricula for generative models, i.e. architecture (generation capacity), semantics (data content), dimension (data space), and cluster (data distribution).
Revised: • We ﬁrst summarize four representative curricula for generative models, i e. architecture (generation capacity), semantics (data content), dimension (data space), and cluster (data structure).

Original: Among these curricula, cluster curriculum is newly proposed in this paper
Revised: Among these curricula, cluster curriculum is newly proposed in this paper.

Original: • Cluster curriculum is to treat data according to  centrality of each data point, which is pictorially illustrated and explained in detail.
Revised: • Cluster curriculum is to treat data according to the centrality of each data point, which is pictorially illustrated and explained in detail.

Original: To foster large-scale learning, we devise the active set algorithm that only needs an active data subset of small ﬁxed size for training.
Revised: To foster large-scale learning, we devise the active set algorithm that only needs an active data subset of small ﬁxed size for training.

Original: • The geometric principle is formulated to analyze hardness of noisy data and advantage of cluster curriculum.
Revised: • The geometric principle is formulated to analyze hardness of noisy data and advantage of cluster curriculum.

Original: The geometry pertains to counting a small sphere packed in an ellipsoid, on which is based the percolation theory we use.
Revised: The geometry pertains to counting a small sphere packed in an ellipsoid, on which is based the percolation theory we use.

Original: The research on curriculum learning is diverse.
Revised: The research on curriculum learning is diverse.

Original: Our work focuses on curricula that are closely related to data attributes, beyond which is not the scope we concern in this paper.
Revised: Our work focuses on curricula that are closely related to data attributes, beyond which is not the scope we concern in this paper.

Original:  C URRICULUM LEARNING
Revised:  C URRICULUM LEARNING

Original: Curriculum learning has been a basic learning approach to promoting performance of algorithms in machine learning.
Revised: Curriculum learning has been a basic learning approach to promoting performance of algorithms in machine learning.

Original: We quote the original words from the seminal paper (Bengio et al., 2009) as its deﬁnition:
Revised: We quote the original words from the seminal paper (Bengio et al., 2009) as its deﬁnition:

Original: Curriculum learning.
Revised: Curriculum learning.

Original: “The basic idea is to start small, learn easier aspects of the task or easier sub-tasks, and then gradually increase the difﬁculty level” according to pre-deﬁned or self-learned curricula.
Revised: “The basic idea is to start small, learn easier aspects of the task or easier sub-tasks, and then gradually increase the difﬁculty level” according to pre-deﬁned or self-learned curricula.

Original: From cognitive perspective, curriculum learning is common for human and animal learning when they interact with environments (Elman, 1993), which is the reason why it is natural as a learning rule for machine intelligence.
Revised: From cognitive perspective, curriculum learning is common for human and animal learning when they interact with environments (Elman, 1993), which is the reason why it is natural as a learning rule for machine intelligence.

Original: The learning process of cognitive development is gradual and progressive (Oudeyer et al., 2007).
Revised: The learning process of cognitive development is gradual and progressive (Oudeyer et al., 2007).

Original: In practice, the design of curricula is task-dependent and data-dependent.
Revised: In practice, the design of curricula is task-dependent and data-dependent.

Original: Here we summarize the representative curricula that are developed for generative models.
Revised: Here we summarize the representative curricula that are developed for generative models.

Original: Architecture curriculum .
Revised: Architecture curriculum .

Original: The deep neural architecture itself can be viewed as a curriculum from the viewpoint of learning concepts (Hinton & Salakhutdinov, 2006; Bengio et al., 2006) or disentangling representations (Lee et al., 2011).
Revised: The deep neural architecture itself can be viewed as a curriculum from the viewpoint of learning concepts (Hinton & Salakhutdinov, 2006; Bengio et al., 2006) or disentangling representations (Lee et al., 2011).

Original: For example, the different layers decompose distinctive features of objects for recognition (Lee et al., 2011; Zeiler & Fergus, 2014;
Revised: For example, the different layers decompose distinctive features of objects for recognition (Lee et al., 2011; Zeiler & Fergus, 2014;

Original: Zhou et al., 2016) and generation (Bau et al., 2018).
Revised: Zhou et al., 2016) and generation (Bau et al., 2018).

Original: Besides, Progressive growing of neural architectures is successfully exploited in
Revised: Besides, Progressive growing of neural architectures is successfully exploited in

Original: GANs (Karras et al., 2018a; Heljakka et al., 2018; Korkinof et al., 2018; Karras et al., 2018b).
Revised: GANs (Karras et al., 2018a; Heljakka et al., 2018; Korkinof et al., 2018; Karras et al., 2018b).

Original: Semantics curriculum .
Revised: Semantics curriculum .

Original: The most intuitive content for each datum is the semantic information that the datum conveys.
Revised: The most intuitive content for each datum is the semantic information that the datum conveys.

Original: The hardness of semantics determines the difﬁculty of learning knowledge from data.
Revised: The hardness of semantics determines the difﬁculty of learning knowledge from data.

Original: Therefore, the semantics can be a common curriculum.
Revised: Therefore, the semantics can be a common curriculum.

Original: For instance, the environment for a game in deep reinforcement learning (Justesen et al., 2018) and the number sense of learning cognitive concepts with neural networks (Zou & McClelland, 2013) can be such curricula.
Revised: For instance, the environment for a game in deep reinforcement learning (Justesen et al., 2018) and the number sense of learning cognitive concepts with neural networks (Zou & McClelland, 2013) can be such curricula.

Original: Dimension curriculum .
Revised: Dimension curriculum .

Original: The high dimension usually poses the difﬁculty of machine learning due to the curse of dimensionality (Donoho, 2000), in the sense that the amount of data points for learning grows exponentially with dimension of variables (Vershynin, 2018).
Revised: The high dimension usually poses the difﬁculty of machine learning due to the curse of dimensionality (Donoho, 2000), in the sense that the amount of data points for learning grows exponentially with dimension of variables (Vershynin, 2018).

Original: Therefore, the algorithms are expected to be beneﬁcial from growing dimensions.
Revised: Therefore, the algorithms are expected to be beneﬁcial from growing dimensions.

Original: The effectiveness of dimension curriculum is evident from recent progress on deep generative models, such as ProGANs (Karras et al., 2018a;b) by gradually enlarging image resolution and language generation from short sequences to long sequences of more complexity (Rajeswar et al., 2017; Press et al., 2017).
Revised: The effectiveness of dimension curriculum is evident from recent progress on deep generative models, such as ProGANs (Karras et al., 2018a;b) by gradually enlarging image resolution and language generation from short sequences to long sequences of more complexity (Rajeswar et al., 2017; Press et al., 2017).

Original:  C LUSTER CURRICULUM
Revised:  C LUSTER CURRICULUM

Original: For ﬁtting distributions, dense data points are generally easier to handle than sparse data or outliers.
Revised: For ﬁtting distributions, dense data points are generally easier to handle than sparse data or outliers.

Original: To train generative models robustly, therefore, it is plausible to raise cluster curriculum, meaning that generative algorithms ﬁrst learn from data points close to cluster centers and then with more data progressively approaching cluster boundaries.
Revised: To train generative models robustly, therefore, it is plausible to raise cluster curriculum, meaning that generative algorithms ﬁrst learn from data points close to cluster centers and then with more data progressively approaching cluster boundaries.

Original: Thus the stream of feeding data points to models for curriculum learning is the process of clustering data points according to cluster centrality that will be explained in section 3.2.
Revised: Thus the stream of feeding data points to models for curriculum learning is the process of clustering data points according to cluster centrality that will be explained in section 3.2.

Original: The toy example in Figure 1 illustrates how to form cluster curriculum.
Revised: The toy example in Figure 1 illustrates how to form cluster curriculum.

Original: 3. W HY CLUSTERS MATTER
Revised: 3. W HY CLUSTERS MATTER

Original: The importance of clusters for data points is actually obvious from geometric point of view.
Revised: The importance of clusters for data points is actually obvious from geometric point of view.

Original: The data sparsity in high-dimensional spaces causes the difﬁculty of ﬁtting the underlying distribution of
Revised: The data sparsity in high-dimensional spaces causes the difﬁculty of ﬁtting the underlying distribution of

Original: [Figure]
Revised: [Figure]

Original: The value n is the number of data points taken with centrality order data points (Vershynin, 2018).
Revised: The value n is the number of data points taken with centrality order data points (Vershynin, 2018).

Original: So generative algorithms may be beneﬁcial when proceeding from the local spaces where data points are relatively dense.
Revised: So generative algorithms may be beneﬁcial when proceeding from the local spaces where data points are relatively dense.

Original: Such data points form clusters that are generally informative subsets with respect to  entire dataset.
Revised: Such data points form clusters that are generally informative subsets with respect to the entire dataset.

Original: In addition, clusters contain common regular patterns of data points, where generative models are easier to converge.
Revised: In addition, clusters contain common regular patterns of data points, where generative models are easier to converge.

Original: What is most important is that noisy data points deteriorate performance of algorithms .
Revised: What is most important is that noisy data points deteriorate performance of algorithms .

Original: For classiﬁcation, the effectiveness of curriculum learning is theoretically proven to circumvent the negative inﬂuence of noisy data (Gong et al., 2016).
Revised: For classiﬁcation, the effectiveness of curriculum learning is theoretically proven to circumvent the negative inﬂuence of noisy data (Gong et al., 2016).

Original: We will analyze this aspect for generative models with geometric facts.
Revised: We will analyze this aspect for generative models with geometric facts.

Original: 3. G ENERATIVE MODELS WITH CLUSTER CURRICULUM
Revised: 3. G ENERATIVE MODELS WITH CLUSTER CURRICULUM

Original: With cluster curriculum, we are allowed to gradually learn generative models from dense clusters to cluster boundaries and ﬁnally to all data points.
Revised: With cluster curriculum, we are allowed to gradually learn generative models from dense clusters to cluster boundaries and ﬁnally to all data points.

Original: In this way, generative algorithms are capable of avoiding the direct harm of noise or outliers.
Revised: In this way, generative algorithms are capable of avoiding the direct harm of noise or outliers.

Original: To this end, we ﬁrst need a measure called centrality that is the terminology in graph-based clustering.
Revised: To this end, we ﬁrst need a measure called centrality that is the terminology in graph-based clustering.

Original: It quantiﬁes the compactness of a cluster in data points or a community in complex networks (Newman, 2010).
Revised: It quantiﬁes the compactness of a cluster in data points or a community in complex networks (Newman, 2010).

Original: A large centrality implies that the associated data point is close to one of cluster centers.
Revised: A large centrality implies that the associated data point is close to one of cluster centers.

Original: For easy reference, we provide the algorithm of the centrality we use in appendix.
Revised: For easy reference, we provide the algorithm of the centrality we use in Appendix.

Original: For experiments in this paper, all the cluster curricula are constructed by the centrality of stationary probability distribution, i.e. the eigenvector corresponding to the largest eigenvalue of the transition probability matrix drawn from the data.
Revised: For experiments in this paper, all the cluster curricula are constructed by the centrality of stationary probability distribution, i.e. the eigenvector corresponding to the largest eigenvalue of the transition probability matrix drawn from the data.

Original: To be speciﬁc, let c ∈ R m denote the centrality vector of m data points.
Revised: To be speciﬁc, let c ∈ R m denote the centrality vector of m data points.

Original: Namely, the i -th entry c i of c is the centrality of data point x i .
Revised: Namely, the i -th entry c i of c is the centrality of data point x i .

Original: Sorting c in descending order and adjusting the order of original data points accordingly give data points arranged by cluster centrality.
Revised: Sorting c in descending order and adjusting the order of original data points accordingly give data points arranged by cluster centrality.

Original: Let −→X = {−→X 0 , −→X 1 , . .
Revised: Let −→X = {−→X 0 , −→X 1 , . .

Original: . , −→X l } signify the set of centrality-sorted data points, where −→X 0 is the base set that guarantees a proper convergence of generative models, and the rest of −→X is evenly divided into l subsets according to centrality order.
Revised: . , −→X l } signify the set of centrality-sorted data points, where −→X 0 is the base set that contains sufﬁcient data to attain convergent generative models, and the rest of −→X is evenly divided into l subsets according to centrality order.

Original: In general, the number of data points in −→X 0 is moderate compared to m and determined according to X .
Revised: In general, the number of data points in −→X 0 is much less than m and determined according to X .

Original: Such division of −→X 0 serves to efﬁciency of training, because we do not need to train models from a very small dataset.
Revised: Such division of −→X 0 serves to efﬁciency of training, because we do not need to train models from a very small dataset.

Original: The cluster curriculum learning is carried out by incrementally feeding subsets in −→X into generative algorithms.
Revised: The cluster curriculum learning is carried out by incrementally feeding subsets in −→X into generative algorithms.

Original: In other words, algorithms are successively trained on −→X 0 ← −→X 0 ∪ −→X i after −→X 0 , meaning that the curriculum for each round of training is accumulated with −→X i .
Revised: In other words, algorithms are successively trained on −→X 0 ← −→X 0 ∪ −→X i after −→X 0 , meaning that the curriculum for each round of training is accumulated with −→X i .

Original: In order to determine the optimal curriculum −→X 0 ∪ −→X 1 ∪ · · · ∪ −→X i , we need the aid of quality metric of generative models, such as Fréchet inception distance (FID) or sliced Wasserstein distance (SWD) (Borji, 2018).
Revised: In order to determine the optimal curriculum −→X 0 ∪ −→X 1 ∪ · · · ∪ −→X i , we need the aid of quality metric of generative models, such as Fréchet inception distance (FID) or sliced Wasserstein distance (SWD) (Borji, 2018).

Original: For generative models trained with each curriculum, we calculate the associated score s i via the speciﬁed quality metric.
Revised: For generative models trained with each curriculum, we calculate the associated score s i via the speciﬁed quality metric.

Original: The optimal curriculum for effective training can be identiﬁed by the minimal value for all s i , where i = 1 , . . .
Revised: The optimal curriculum for effective training can be identiﬁed by the minimal value for all s i , where i = 1 , . . .

Original: , l + 1 .
Revised: , l + 1 .

Original: The interesting phenomenon of this score curve will be illustrated in the experiment.
Revised: The interesting phenomenon of this score curve will be illustrated in the experiment.

Original: The minimum of score s is apparently metric-dependent.
Revised: The minimum of score s is apparently metric-dependent.

Original: One can refer to (Borji, 2018) for the review of evaluation metrics.
Revised: One can refer to (Borji, 2018) for the review of evaluation metrics.

Original: In practice, we can opt one of reliable metrics to use or multiple metrics for decision-making of the optimal model.
Revised: In practice, we can opt one of reliable metrics to use or multiple metrics for decision-making of the optimal model.

Original: There are two ways of using  incremental subset −→X i during training.
Revised: There are two ways of using the incremental subset −→X i during training.

Original: One is that the parameters of models are re-randomized when  new data are used, the procedure of which is given in Algorithm 1 in appendix.
Revised: One is that the parameters of models are re-randomized when the new data are used, the procedure of which is given in Algorithm 1 in Appendix.

Original: The other is that the parameters are ﬁne-tuned based on pre-training of  previous model, which will be presented with a fast learning algorithm in the following section.
Revised: The other is that the parameters are ﬁne-tuned based on pre-training of the previous model, which will be presented with a fast learning algorithm in the following section.

Original: 3. A CTIVE SET FOR SCALABLE TRAINING
Revised: 3. A CTIVE SET FOR SCALABLE TRAINING

Original: To obtain the precise minimum of s , the cardinality of −→X i needs to be set much smaller than m , meaning that l will be large even for a dataset of moderate scale.
Revised: To obtain the precise minimum of s , the cardinality of −→X i needs to be set much smaller than m , meaning that l will be large even for a dataset of moderate scale.

Original: The training of many loops will lead to time-consuming.
Revised: The training of many loops will be time-consuming.

Original: Here we propose the active set to address the issue, in the sense that for each loop of cluster curriculum,  generative models are always trained with a subset of  small ﬁxed size instead of −→X 0 ← −→X 0 ∪ −→X i whose size becomes incrementally large. 
Revised: Here we propose the active set to address the issue, in the sense that for each loop of cluster curriculum, the generative models are always trained with a subset of a small ﬁxed size instead of −→X 0 ← −→X 0 ∪ −→X i whose size becomes incrementally large. [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: curriculum. Here |−→X 0 | = |−→X 1 | = |−→X 2 | = 100 .

Original: [Equation]
Revised: The cardinality |A| of the active set A is 200 . When −→X 2 is taken for training, we need to randomly sample another 100 (i.e.

Original: [Figure]
Revised: |A| − |−→X 2 | ) data points from the history data −→X 0 ∪ −→X 1 to form −→A 0 . Then the complete active set is composed by A = −→A 0 ∪ −→X 2 . We can see that data points in −→X 0 ∪ −→Xbecome less dense after sampling.

Original: To form the active set A , the subset −→A 0 of data points are randomly sampled from −→X 0 to combine with −→X i for the next loop, where |−→A 0 | = |A| − |−→X i | .
Revised: To form the active set A , the subset −→A 0 of data points are randomly sampled from −→X 0 to combine with −→X i for the next loop, where |−→A 0 | = |A| − |−→X i | .

Original: For easy understanding, we illustrate the active set with toy example in Figure 2.
Revised: For easy understanding, we illustrate the active set with toy example in Figure 2.

Original: In this scenario, progressive pre-training must be applied, meaning that the update of model parameters for the current training is based on parameters of previous loop.
Revised: In this scenario, progressive pre-training must be applied, meaning that the update of model parameters for the current training is based on parameters of previous loop.

Original: The procedure of cluster curriculum with active set is detailed in Algorithm 2 in appendix.
Revised: The procedure of cluster curriculum with active set is detailed in Algorithm 2 in Appendix.

Original: The active set allows us to train generative models with a small dataset that is actively adapted, thereby signiﬁcantly reducing the training time for large-scale data.
Revised: The active set allows us to train generative models with a small dataset that is actively adapted, thereby signiﬁcantly reducing the training time for large-scale data.

Original:  G EOMETRIC VIEW OF CLUSTER CURRICULUM
Revised:  G EOMETRIC VIEW OF CLUSTER CURRICULUM

Original: Cluster curriculum bears the interesting relation to high-dimensional geometry, which can provide geometric understanding of our algorithm.
Revised: Cluster curriculum bears the interesting relation to high-dimensional geometry, which can provide geometric understanding of our algorithm.

Original: Without loss of generality, we work on a cluster obeying  normal distribution.
Revised: Without loss of generality, we work on a cluster obeying the normal distribution.

Original: The characteristic of the cluster can be extended into other clusters of the same distribution.
Revised: The characteristic of the cluster can be extended into other clusters of the same distribution.

Original: For easy analysis, let us begin with a toy example.
Revised: For easy analysis, let us begin with a toy example.

Original: As Figure 3(a) shows, the conﬁdence ellipse E 2 ﬁtted from the subset of centrality-ranked data points is nearly conformal to E 1 of all data points, which allow us to put the relation of these two ellipses by virtue of the conﬁdence-level equation.
Revised: As Figure 3(a) shows, the conﬁdence ellipse E 2 ﬁtted from the subset of centrality-ranked data points is nearly conformal to E 1 of all data points, which allows us to put the relation of these two ellipses by virtue of the conﬁdence-level equation.

Original: Let N ( 0 , Σ ) signify the center and covariance matrix of the cluster C of interest, where
Revised: Let N ( 0 , Σ ) signify the center and covariance matrix of the cluster C of interest, where

Original: C = { x i | x i ∈ R d , i = 1 , . . . , n } .
Revised: C = { x i | x i ∈ R d , i = 1 , . . . , n } .

Original: To make it formal, we can write the equation by
Revised: To make it formal, we can write the equation by

Original: [Equation]
Revised: [Equation]

Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original:  where χ 2 d,α can be the chi-squared distribution or Mahalanobis distance square, d is the degree of freedom, and (1 − α ) ∈
Revised: [Figure] [Equation] [Equation] where χ 2 d,α can be the chi-squared distribution or Mahalanobis distance square, d is the degree of freedom, and (1 − α ) ∈

Original: [0 , 1] is the conﬁdence level.
Revised: [0 , 1] is the conﬁdence level.

Original: For conciseness, we write χ 2 d,α as χ 2 α in the following context.
Revised: For conciseness, we write χ 2 d,α as χ 2 α in the following context.

Original: Then the ellipses E α 1 and E α 2 correspond to χ 2 α 1 and χ 2 α 2 , respectively, where α 1 ≤ α 2 .
Revised: Then the ellipses E α 1 and E α 2 correspond to χ 2 α 1 and χ 2 α 2 , respectively, where α 1 ≤ α 2 .

Original: To analyze the hardness of training generative model, a fundamental aspect is to examine the number n ( E ) of given data points falling in a geometric entity E 1 and the number N ( E ) of lattice points in it .
Revised: To analyze the hardness of training generative models, a fundamental aspect is to examine the number n ( E ) of given data points falling in a geometric entity E 1 and the number N ( E ) of lattice points in it .

Original: The less n ( E ) is compared to N ( E ) , the harder the problem will be.
Revised: The less n ( E ) is compared to N ( E ) , the harder the problem will be.

Original: However, the enumeration of lattice points is computationally prohibitive for high dimensions.
Revised: However, the enumeration of lattice points is computationally prohibitive for high dimensions.

Original: Inspired by the information theory of encoding data of normal distributions (Roman, 1996), we count the number of small spheres S ε of radius ε packed in the ellipsoid E instead.
Revised: Inspired by the information theory of encoding data of normal distributions (Roman, 1996; Ma et al., 2007), we count the number of small spheres S ε of radius ε packed in the ellipsoid E instead.

Original: Thus we can use this number to replace the role of N ( E ) as long as the radius of the sphere S ε is set properly.
Revised: Thus we can use this number to replace the role of N ( E ) as long as the radius of the sphere S ε is set properly.

Original: With a little abuse of  notation, we still use N ( E ) to denote the packing number in the following context.
Revised: With a little abuse of cluster curriculum, it [Figure] notation, we still use N ( E ) to denote the packing number in the following context.

Original: Theorem 1 gives the exact form of N ( E ) .
Revised: Theorem 1 gives the exact form of N ( E ) .

Original: [Figure]

Original: Theorem 1.
Revised: Theorem 1.

Original: For a set C = { x i | x i ∈ R d } of n data points drawn from normal distribution N ( 0 , Σ ) , the ellipsoid E α of conﬁdence 1 − α is deﬁned as x (cid:62) Σ − 1 x ≤ χ 2 α , where Σ has no zero eigenvalues and α ∈ [0 , 1] .
Revised: For a set C = { x i | x i ∈ R d } of n data points drawn from normal distribution N ( 0 , Σ ) , the ellipsoid E α of conﬁdence 1 − α is deﬁned as x (cid:62) Σ − 1 x ≤ χ 2 α , where Σ has no zero eigenvalues and α ∈ [0 , 1] .

Original: Let N ( E α ) be the number of spheres of radius ε packed in the ellipsoid E α .
Revised: Let N ( E α ) be the number of spheres of radius ε packed in the ellipsoid E α .

Original: Then we can establish
Revised: Then we can establish

Original: [Equation]
Revised: [Equation]

Original: We  see that N ( E α ) admits a tidy form with Mahalanobis distance χ α , dimension d , and sphere radius ε as variables.
Revised: We can see that N ( E α ) admits a tidy form with Mahalanobis distance χ α , dimension d , and sphere radius ε as variables.

Original: The proof is provided in appendix.
Revised: The proof is provided in Appendix.

Original: The geometric region of interest for cluster curriculum is the annulus A formed by removing the ellipsoid 2 E α 2 from the ellipsoid E α 1 , as Figure 3(b) displays.
Revised: The geometric region of interest for cluster curriculum is the annulus A formed by removing the ellipsoid 2 E α 2 from the ellipsoid E α 1 , as Figure 3(b) displays.

Original: We investigate the varying law between n ( A ) and N ( A ) in the annulus A when the inner ellipse E α 2 grows with cluster curriculum.
Revised: We investigate the varying law between n ( A ) and N ( A ) in the annulus A when the inner ellipse E α 2 grows with cluster curriculum.

Original: For this purpose, we need the following two corollaries that immediately follows from Theorem 1.
Revised: For this purpose, we need the following two corollaries that immediately follows from Theorem 1.

Original: Corollary 1.
Revised: Corollary 1.

Original: Let N ( A ) be the number of spheres of radius ε packed in the annulus A that is formed by removing the ellipsoid E α 1 from the ellipsoid E α 1 , where α 1 ≤ α 2 .
Revised: Let N ( A ) be the number of spheres of radius ε packed in the annulus A that is formed by removing the ellipsoid E α 1 from the ellipsoid E α 1 , where α 1 ≤ α 2 .

Original: Then the following identity holds
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: Corollary 2.
Revised: Corollary 2.

Original: [Equation]
Revised: [Equation]

Original: It is obvious that N ( A ) goes inﬁnite when d → ∞ under the conditions that χ α 1 > χ
Revised: It is obvious that N ( A ) goes inﬁnite when d → ∞ under the conditions that χ α 1 > χ

Original: α 2 and ε is bounded.
Revised: α 2 and ε is bounded.

Original: Besides, when E α 2 (cluster) grows, N ( A ) reduces with exponent d ifthe ellipsoid E α 1 is ﬁxed.
Revised: Besides, when E α 2 (cluster) grows, N ( A ) reduces with exponent d if E α 1 is ﬁxed.

Original: In light of Corollary 1, we can now demonstrate the functional law between n ( A ) and N ( A ) .
Revised: In light of Corollary 1, we can now demonstrate the functional law between n ( A ) and N ( A ) .

Original: First, we determine χ α 1 as follows
Revised: First, we determine χ α 1 as follows

Original: [Equation]
Revised: [Equation]

Original: which means that E α 1 is the ellipsoid of minimal Mahalanobis distance to the center that contains all the data points in the cluster.
Revised: which means that E α 1 is the ellipsoid of minimal Mahalanobis distance to the center that contains all the data points in the cluster.

Original: In addition, we need to estimate a suitable sphere radius ε , such that n ( E α 1 ) and N ( E α 1 ) have comparable scales in order to make n ( A ) and N ( A ) comparable in scale.
Revised: In addition, we need to estimate a suitable sphere radius ε , such that n ( E α 1 ) and N ( E α 1 ) have comparable scales in order to make n ( A ) and N ( A ) comparable in scale.

Original: To achieve this, we deﬁne an oracle ellipse E where n ( E ) =
Revised: To achieve this, we deﬁne an oracle ellipse E where n ( E ) =

Original: N ( E ) .
Revised: N ( E ) .

Original: For simplicity, we let E α 1 be the oracle ellipse.
Revised: For simplicity, we let E α 1 be the oracle ellipse.

Original: Thus we can determine ε with Corollary 3.
Revised: Thus we can determine ε with Corollary 3.

Original: Corollary 3.
Revised: Corollary 3.

Original: If we let E α 1 be the oracle ellipse such that n ( E α 1 ) =
Revised: If we let E α 1 be the oracle ellipse such that n ( E α 1 ) =

Original: N ( E α 1 ) , then the free parameter ε can be computed with ε = χ
Revised: N ( E α 1 ) , then the free parameter ε can be computed with ε = χ

Original: α 1 (cid:0)(cid:112) det( Σ ) /n
Revised: α 1 (cid:0)(cid:112) det( Σ ) /n

Original: ( E α 1 ) (cid:1) 1 d .
Revised: ( E α 1 ) (cid:1) 1 d .

Original: To make the demonstration amenable to handle, data points we use for simulation are assumed to obey the isotropic normal distribution, meaning that data points are generated with nearly equal variance along each dimension.
Revised: To make the demonstration amenable to handle, data points we use for simulation are assumed to obey the isotropic normal distribution, meaning that data points are generated with nearly equal variance along each dimension.

Original: Figure 4 shows that n ( A ) gradually exhibits the critical phenomena of percolation processes 3 when the dimension d goes large, implying that the data points in the annulus A are signiﬁcantly reduced when E α 2 grows a little bigger near the critical point.
Revised: Figure 4 shows that n ( A ) gradually exhibits the critical phenomena of percolation processes 3 when the dimension d goes large, implying that the data points in the annulus A are signiﬁcantly reduced when E α 2 grows a little bigger near the critical point.

Original: In contrast, the number N ( A ) of lattice points is still large and varies negligibly until E α 2 approaches the boundary.
Revised: In contrast, the number N ( A ) of lattice points is still large and varies negligibly until E α 2 approaches the boundary.

Original: This discrepancy indicates clearly that ﬁtting data points in the annulus is pretty hard and guaranteeing the precision is nearly impossible when crossing the critical point of n ( A ) even for a moderate dimension (
Revised: This discrepancy indicates clearly that ﬁtting data points in the annulus is pretty hard and guaranteeing the precision is nearly impossible when crossing the critical point of n ( A ) even for a moderate dimension (

Original: e.g. d = 500 ).
Revised: e.g. d = 500 ).

Original: Therefore, the plausibility of cluster curriculum can be drawn naturally from this geometric fact.
Revised: Therefore, the plausibility of cluster curriculum can be drawn naturally from this geometric fact.

Original:  E XPERIMENT
Revised:  E XPERIMENT

Original: The generative model that we use for experiments are Progressive growing of GAN (ProGAN) (Karras et al., 2018a).
Revised: The generative model that we use for experiments are Progressive growing of GAN (ProGAN) (Karras et al., 2018a).

Original: This algorithm is chosen because ProGAN is the state-of-the-arts algorithm of GANs with ofﬁcial open sources available.
Revised: This algorithm is chosen because ProGAN is the state-of-the-arts algorithm of GANs with ofﬁcial open sources available.

Original: According to convention, we opt the Fréchet inception distance (FID) (Borji, 2018) for ProGAN as the quality metric.
Revised: According to convention, we opt the Fréchet inception distance (FID) (Borji, 2018) for ProGAN as the quality metric.

Original: 5. D ATASET AND EXPERIMENTAL SETTING
Revised: 5. D ATASET AND EXPERIMENTAL SETTING

Original: We randomly sample  200,000 cat images from the LSUN dataset (Yu et al., 2015).
Revised: We randomly sample the 200,000 cat images from the LSUN dataset (Yu et al., 2015).

Original: These cat images are captured in the wild.
Revised: These cat images are captured in the wild.

Original: So their styles vary signiﬁcantly.
Revised: So their styles vary signiﬁcantly.

Original: Figure 5 shows the cat examples of high and low centralities.
Revised: Figure 5 shows the cat examples of high and low centralities.

Original: We can see that  noisy cat images differ much from the clean ones.
Revised: We can see that the noisy cat images differ much from the clean ones.

Original: There actually contain the images of very few informative cat features, which are  outliers we refer to.
Revised: There actually contain the images of very few informative cat features, which are the outliers we refer to.

Original: The curriculum parameters are set as |−→X 0 | = 20 , 000 and |−→X i | = 10 , 000 , which means that the algorithms are trained with 20,000 images ﬁrst and after the initial training, another 10,000 images according to centrality order are merged into the current training data for further re-training.
Revised: The curriculum parameters are set as |−→X 0 | = 20 , 000 and |−→X i | = 10 , 000 , which means that the algorithms are trained with 20,000 images ﬁrst and after the initial training, another 10,000 images according to centrality order are merged into the current training data for further re-training.

Original: For active set, its size is ﬁxed to be 30 , 000 .
Revised: For active set, its size is ﬁxed to be 30 , 000 .

Original: The CelebA dataset is a large-scale face attribute dataset (Liu et al., 2015).
Revised: The CelebA dataset is a large-scale face attribute dataset (Liu et al., 2015).

Original: We use the cropped and well-aligned faces with a bit of image backgrounds preserved for generation task.
Revised: We use the cropped and well-aligned faces with a bit of image backgrounds preserved for generation task.

Original: For clustercurriculum learning, we randomly sample 70,000 faces as the training set.
Revised: For clustercurriculum learning, we randomly sample 70,000 faces as the training set.

Original: The face examples of different centralities are shown in Figure 5.
Revised: The face examples of different centralities are shown in Figure 5.

Original: The curriculum parameters are set as |−→X 0 | = 10 ,and |−→X i | = 5 , 000 .
Revised: The curriculum parameters are set as |−→X 0 | = 10 ,and |−→X i | = 5 , 000 .

Original: We bypass the experiment of  active set on faces because it is used for  large-scale data.
Revised: We bypass the experiment of the active set on faces because it is used for the large-scale data.

Original: Each image in two databases is resized to be 64 × 64 .
Revised: Each image in two databases is resized to be 64 × 64 .

Original: To form cluster curricula, we exploit
Revised: To form cluster curricula, we exploit

Original: ResNet34 (He et al., 2016)
Revised: ResNet34 (He et al., 2016)

Original: pre-trained on ImageNet (Russakovsky et al., 2015) to extract 512dimensional features for each face and cat images.
Revised: pre-trained on ImageNet (Russakovsky et al., 2015) to extract 512dimensional features for each face and cat images.

Original: The directed graphs are built with these feature vectors.
Revised: The directed graphs are built with these feature vectors.

Original: We determine the parameter σ of edge weights by enforcing the geometric mean of weights to be 0.8.
Revised: We determine the parameter σ of edge weights by enforcing the geometric mean of weights to be 0.8.

Revised: The robustness of varying the value was validated in (Zhao & Tang, 2008) for clustering.

Original: The number of nearest neighbors is set to be K = 4 log m .
Revised: The number of nearest neighbors is set to be K = 4 log m .

Original: The centrality is the stationary probability distribution.
Revised: The centrality is the stationary probability distribution.

Original: All codes are written with TensorFlow.
Revised: All codes are written with TensorFlow.

Original: [Figure]
Revised: [Figure]

Original: 5. E XPERIMENTAL RESULT
Revised: 5. E XPERIMENTAL RESULT

Original: From Figure 6a we can see that the FID curves are all nearly V-shaped, indicating that  global minima exist amid the training process.
Revised: From Figure 6a, we can see that the FID curves are all nearly V-shaped, indicating that the global minima exist amid the training process.

Original: This is  clear evidence that  noisy data and outliers deteriorate the quality of generative models during training.
Revised: This is the clear evidence that the noisy data and outliers deteriorate the quality of generative models during training.

Original: From the optimal curricula found by two algorithms (i e. curricula at 110,000 and 100,000) we can see that the curriculum of the active set differs from that of normal training only by one-step data increment, implying that the active set is reliable for fast cluster-curriculum learning.
Revised: From the optimal curricula found by two algorithms (i.e. curricula at 110,000 and 100,000), we can see that the curriculum of the active set differs from that of normal training only by one-step data increment, implying that the active set is reliable for fast cluster-curriculum learning.

Original: The performance of the active set measured by FID is much worse than that of normal training, especially when more noisy data are fed into generative models.
Revised: The performance of the active set measured by FID is much worse than that of normal training, especially when more noisy data are fed into generative models.

Original: However, this does not change the whole V-shape of the accuracy curve.
Revised: However, this does not change the whole V-shape of the accuracy curve.

Original: Namely, it is applicable as long as the active set admits the metric minimum corresponding to the appropriate curriculum.
Revised: Namely, it is applicable as long as the active set admits the metric minimum corresponding to the appropriate curriculum.

Original: The V-shape of the centrality-FID curve on the cat data is due to that the noisy data of low centrality contains little effective information to characterize the cats, as already displayed in Figure 5.
Revised: The V-shape of the centrality-FID curve on the cat data is due to that the noisy data of low centrality contains little effective information to characterize the cats, as already displayed in Figure 5.

Original: However, it is different for the CelebA face dataset where the face images of low centrality also convey  part of face features.
Revised: However, it is different for the CelebA face dataset where the face images of low centrality also convey the part of face features.

Original: As evident by Figure 6b, ProGAN keeps being optimized by the majority of  data until the curriculum of size 55 , 000 .
Revised: As evident by Figure 6b, ProGAN keeps being optimized by the majority of the data until the curriculum of size 55 , 000 .

Original: To highlight the meaning of this nearly negligible minimum, we also conduct the exactly same experiment on the FFHQ face dataset containing 70 , 000 face images of high-quality (Karras et al., 2018b).
Revised: To highlight the meaning of this nearly negligible minimum, we also conduct the exactly same experiment on the FFHQ face dataset containing 70 , 000 face images of high-quality (Karras et al., 2018b).

Original: For FFHQ data, the noisy face data can be ignored.
Revised: For FFHQ data, the noisy face data can be ignored.

Original: The gray curve of normal training in Figure 6b indicates that the FID of ProGAN is monotonically decreased for all curricula.
Revised: The gray curve of normal training in Figure 6b indicates that the FID of ProGAN is monotonically decreased for all curricula.

Original: This gentle difference of the FID curves at the ends between CelebA and FFHQ clearly demonstrate the difﬁculty of noisy data to generative algorithms.
Revised: This gentle difference of the FID curves at the ends between CelebA and FFHQ clearly demonstrates the difﬁculty of noisy data to generative algorithms.

Original: 5. G EOMETRIC INVESTIGATION
Revised: 5. G EOMETRIC INVESTIGATION

Original: To understand cluster curriculum deeply, we employ the geometric method formulated in section 4 to analyze the cat and face data.
Revised: To understand cluster curriculum deeply, we employ the geometric method formulated in section 4 to analyze the cat and face data.

Original: The percolation processes are both conducted with 512-dimensional features from ResNet34.
Revised: The percolation processes are both conducted with 512-dimensional features from ResNet34.

Original: Figure 7 displays the curve of n ( A ) that is the variable of interest in this scenario.
Revised: Figure 7 displays the curve of n ( A ) that is the variable of interest in this scenario.

Original: As expected, the critical point in the percolation process occurs for both cases, as shown by blue curves.
Revised: As expected, the critical point in the percolation process occurs for both cases, as shown by blue curves.

Original: An obvious fact is that the optimal curricula (red strips) both fall into the (feasible) domains of percolation processes after  critical points, as indicated by gray color.
Revised: An obvious fact is that the optimal curricula (red strips) both fall into the (feasible) domains of percolation processes after the critical points, as indicated by gray color.

Original: This is a desirable property because data become rather sparse in the annuli when crossing the critical points.
Revised: This is a desirable property because data become rather sparse in the annuli when crossing the critical points.

Original: Then noisy data play the non-negligible role in tuning  parameters of generative models.
Revised: Then noisy data play the non-negligible role on tuning the parameters of generative models.

Original: Therefore, a fast learning strategy can be derived from  percolation process.
Revised: Therefore, a fast learning strategy can be derived from the percolation process.

Original: The training may begin from the curriculum speciﬁed by the critical point, thus signiﬁcantly accelerating cluster-curriculum learning.
Revised: Training may begin from the curriculum speciﬁed by the critical point, thus signiﬁcantly accelerating cluster-curriculum learning.

Original: 1    critical point 0.
Revised: 1    critical point 0.

Original: critical point
Revised: critical point

Original: [Figure]
Revised: [Figure]

Original: The  strips  intervals of optimal curricula derived by generative models.
Revised: The pink strips are intervals of optimal curricula derived by generative models.

Original: For example, the valueof the pink interval in (a) is obtained by 9 = 20 − 11 , where 11 is one of the minima (i.e.110,000) in
Revised: For example, the value 9 of the pink interval in (a) is obtained by 9 = 20 − 11 , where 11 is one of the minima (i.e.

Original:  Figure 6a.
Revised: 110,000) in Figure 6a.

Original: The others are derived in the same way.
Revised: The others are derived in the same way.

Original: The subtraction transforms the data number in the cluster to be the one in the annulus.
Revised: The subtraction transforms the data number in the cluster to be the one in the annulus.

Original: The critical points are determined by searching the maxima of the absolute discrete difference of the associated curves.
Revised: The critical points are determined by searching the maxima of the absolute discrete difference of the associated curves.

Original: The scales of y -axes are normalized by 10,000.
Revised: The scales of y -axes are normalized by 10,000.

Original: Another intriguing phenomenon is that the more noisy the data, the closer the optimal interval (red strip) is to the critical point.
Revised: Another intriguing phenomenon is that the more noisy the data, the closer the optimal interval (red strip) is to the critical point.

Original: We can see that the optimal interval of the cat data is much closer to the critical point than that of the face data.
Revised: We can see that the optimal interval of the cat data is much closer to the critical point than that of the face data.

Original: What surprises us here is that the optimal interval of cluster curricula associated with the cat data nearly coincides with the critical point of the percolation process in the annulus!
Revised: What surprises us here is that the optimal interval of cluster curricula associated with the cat data nearly coincides with the critical point of the percolation process in the annulus!

Original: This means that the optimal curriculum may be found at the intervals close to the critical point of of n ( A ) percolation for heavily noisy data, thus affording great convenience to learning an appropriate generative model for such datasets.
Revised: This means that the optimal curriculum may be found at the intervals close to the critical point of of n ( A ) percolation for heavily noisy data, thus affording great convenience to learning an appropriate generative model for such datasets.

Original:  A NALYSIS AND CONCLUSION
Revised:  A NALYSIS AND CONCLUSION

Original: The cluster curriculum is proposed for robust training of generative models.
Revised: Cluster curriculum is proposed for robust training of generative models.

Original: The active set of cluster curriculum is devised to facilitate scalable learning.
Revised: The active set of cluster curriculum is devised to facilitate scalable learning.

Original: The geometric principle behind cluster curriculum is analyzed in detail as well.
Revised: The geometric principle behind cluster curriculum is analyzed in detail as well.

Original: The experimental results on  LSUN cat dataset and CelebA face dataset demonstrate that  generative models trained with cluster curriculum is capable of learning the optimal parameters with respect to the speciﬁed quality metric such as Fréchet inception distance and sliced Wasserstein distance.
Revised: The experimental results on the LSUN cat dataset and CelebA face dataset demonstrate that the generative models trained with cluster curriculum is capable of learning the optimal parameters with respect to the speciﬁed quality metric such as Fréchet inception distance and sliced Wasserstein distance.

Original: Geometric analysis indicates that the optimal curricula obtained from generative models are closely related to  critical points of the associated percolation processes established in this paper.
Revised: Geometric analysis indicates that the optimal curricula obtained from generative models are closely related to the critical points of the associated percolation processes established in this paper.

Original: This intriguing geometric phenomenon is worth being explored deeply in terms of the theoretical connection between generative models and high-dimensional geometry.
Revised: This intriguing geometric phenomenon is worth being explored deeply in terms of the theoretical connection between generative models and high-dimensional geometry.

Original: It is worth emphasizing that the meaning of model optimality refers to the global minimum of the centrality-FID curve.
Revised: It is worth emphasizing that the meaning of model optimality refers to the global minimum of the centrality-FID curve.

Original: As we already noted, the optimality is metric-dependent.
Revised: As we already noted, the optimality is metric-dependent.

Original: We are able to obtain the optimal model with cluster curriculum, which does not mean that the algorithm only serves to this purpose.
Revised: We are able to obtain the optimal model with cluster curriculum, which does not mean that the algorithm only serves to this purpose.

Original: We know that more informative data can help learn a more powerful model covering  large data diversity.
Revised: We know that more informative data can help learn a more powerful model covering the large data diversity.

Original: Here a trade-off arises, i.e. the robustness against noise and the capacity of ﬁtting more data.
Revised: Here a trade-off arises, i.e. the robustness against noise and the capacity of ﬁtting more data.

Original: The centrality-FID curve provides a visual tool to monitor the state of model training, thus aiding us in understanding the learning process and selecting suitable models according to noisy degree of given data.
Revised: The centrality-FID curve provides a visual tool to monitor the state of model training, thus aiding us in understanding the learning process and selecting suitable models according to noisy degree of given data.

Original: For instance, we can pick the trained model close to the optimal curriculum for heavily noisy data or the one near the end of the centrality-FID curve for datasets of little noise.
Revised: For instance, we can pick the trained model close to the optimal curriculum for heavily noisy data or the one near the end of the centrality-FID curve for datasets of little noise.

Original: In fact, this may be the most common way of using cluster curriculum.
Revised: In fact, this may be the most common way of using cluster curriculum.

Original: In this paper, we do not investigate the cluster-curriculum learning for the multi-class case, e.g. the
Revised: In this paper, we do not investigate the cluster-curriculum learning for the multi-class case, e.g. the

Original: ImageNet dataset with BigGAN (Brock et al., 2019).
Revised: ImageNet dataset with BigGAN (Brock et al., 2019).

Original: The cluster-curriculum learning of multiple classes is more complex than that we have already analyzed on  face and cat data.
Revised: The cluster-curriculum learning of multiple classes is more complex than that we have already analyzed on the face and cat data.

Original: We leave this study for future work.
Revised: We leave this study for future work.

Original: A A PPENDIX A. C ENTRALITY MEASURE
Revised: A A PPENDIX A. C ENTRALITY MEASURE

Original: The centrality or clustering coefﬁcient pertaining to a cluster in data points or a community in a complex network is a well-studied traditional topic in machine learning and complex systems.
Revised: The centrality or clustering coefﬁcient pertaining to a cluster in data points or a community in a complex network is a well-studied traditional topic in machine learning and complex systems.

Original: Here we introduce the graph-theoretic centrality for the utilization of cluster curriculum.
Revised: Here we introduce the graph-theoretic centrality for the utilization of cluster curriculum.

Original: Firstly, we construct a directed graph (digraph) with K nearest neighbors.
Revised: Firstly, we construct a directed graph (digraph) with K nearest neighbors by the method in (Zhao & Tang, 2008).

Original: The weighted adjacency matrix W of the digraph can be formed in this way: W ij = exp( − d 2 ij /σ 2 ) if x j is one of the nearest neighbors of x i and 0 otherwise, where d ij is the distance between x i and x j and σ is a free parameter.
Revised: The weighted adjacency matrix W of the digraph can be formed in this way: W ij = exp( − d 2 ij /σ 2 ) if x j is one of the nearest neighbors of x i and 0 otherwise, where d ij is the distance between x i and x j and σ is a free parameter.

Original: The density of data points can be quantiﬁed with  stationary probability distribution of a Markov chain.
Revised: The density of data points can be quantiﬁed with the stationary probability distribution of a Markov chain.

Original: For a digraph built from data, the transition probability matrix can be derived by row normalization, say, P ij =
Revised: For a digraph built from data, the transition probability matrix can be derived by row normalization, say, P ij =

Original: W ij / (cid:80)
Revised: W ij / (cid:80)

Original: j W ij .
Revised: j W ij .

Original: Then the stationary probability u can be obtained by solving an eigenvalue problem
Revised: Then the stationary probability u can be obtained by solving an eigenvalue problem

Original: [Equation]
Revised: [Equation]

Original: where (cid:62) denotes the matrix transpose.
Revised: where (cid:62) denotes the matrix transpose.

Original: It is straightforward to know that u is the eigen-vector of P (cid:62) corresponding to the largest eigenvalue (i.e. 1 ).
Revised: It is straightforward to know that u is the eigen-vector of P (cid:62) corresponding to the largest eigenvalue (i.e. 1 ).

Original: u is also deﬁned as a kind of PageRank in many scenarios.
Revised: u is also deﬁned as a kind of PageRank in many scenarios.

Original: For density-based cluster curriculum, the centrality c coincides with the stationary probability u .
Revised: For density-based cluster curriculum, the centrality c coincides with the stationary probability u .

Original: Figure 1 in the main context shows the plausibility of using the stationary probability distribution to quantify  data density.
Revised: Figure 1 in the main context shows the plausibility of using the stationary probability distribution to quantify the data density.

Original: A. T HEOREM 1 AND P ROOF
Revised: A. T HEOREM 1 AND P ROOF

Original: Theorem 1.
Revised: Theorem 1.

Original: For a set C = { x i | x i ∈ R d } of n data points drawn from normal distribution N ( 0 , Σ ) , the ellipsoid E α of conﬁdence 1 − α is deﬁned as x (cid:62) Σ − 1 x ≤ χ 2 α , where Σ has no zero eigenvalues and α ∈ [0 , 1] .
Revised: For a set C = { x i | x i ∈ R d } of n data points drawn from normal distribution N ( 0 , Σ ) , the ellipsoid E α of conﬁdence 1 − α is deﬁned as x (cid:62) Σ − 1 x ≤ χ 2 α , where Σ has no zero eigenvalues and α ∈ [0 , 1] .

Original: Let N ( E α ) be the number of spheres of radius ε packed in the ellipsoid E α .
Revised: Let N ( E α ) be the number of spheres of radius ε packed in the ellipsoid E α .

Original: Then we can establish
Revised: Then we can establish

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: As explained in the main context, the ellipse equation with respect to the conﬁdence α can be expressed by the following equation
Revised: As explained in the main context, the ellipse equation with respect to the conﬁdence α can be expressed by the following equation

Original: [Equation]
Revised: [Equation]

Original: Suppose that λ 1 , . .
Revised: Suppose that λ 1 , . .

Original: , λ d are the eigenvalues of Σ .
Revised: , λ d are the eigenvalues of Σ .

Original: Then equation (7) can be written as
Revised: Then equation (7) can be written as

Original: [Equation]
Revised: [Equation]

Original: Further, eliminating χ 2 α on the right side gives
Revised: Further, eliminating χ 2 α on the right side gives

Original: [Equation]
Revised: [Equation]

Original: Then we derive the length of semi-axis with respect to χ α , i.e.
Revised: Then we derive the length of semi-axis with respect to χ α , i.e.

Original: [Equation]
Revised: [Equation]

Original: For a d -dimensional ellipsoid E , the volume of E is
Revised: For a d -dimensional ellipsoid E , the volume of E is

Original: [Equation]
Revised: [Equation]

Original:  where r i the leng of semi-axis of E and Γ( · ) is the Gamma function.
Revised: (11) where r i the leng of semi-axis of E and Γ( · ) is the Gamma function.

Original: Substituting (10) into the above equation, we obtain the ﬁnal formula of volume
Revised: Substituting (10) into the above equation, we obtain the ﬁnal formula of volume

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation] Using the volume formula in (11), it is straightforward to get the volume of  packing sphere S ε 
Revised: Using the volume formula in (11), it is straightforward to get the volume of the packing sphere S ε [Equation]

Original: [Equation]
Revised: [Equation]

Original: By the deﬁnition of N ( E α ) , we can write
Revised: By the deﬁnition of N ( E α ) , we can write

Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We conclude the proof of the theorem.
Revised: We conclude the proof of the theorem.

Original: A. P ROCEDURES OF A LGORITHM 1 AND A LGORITHM
Revised: A. P ROCEDURES OF A LGORITHM 1 AND A LGORITHM

Original: Require: 1: X , dataset containing data points 2: GenerativeModel
Revised: Require: 1: X , dataset containing data points 2: GenerativeModel

Original: , generative models 3: QualityScore for generative results 4: l , number of subsets
Revised: , generative models 3: QualityScore for generative results 4: l , number of subsets

Original: Ensure: 5: //
Revised: Ensure: 5: //

Original: Solve centralities 6: c =
Revised: Solve centralities 6: c =

Original: Centrality ( X ) (cid:46) section A.
Revised: Centrality ( X ) (cid:46) section A.

Original: 7: // Cluster curriculum 8: Get sorted data −→X
Revised: 7: // Cluster curriculum 8: Get sorted data −→X

Original: according to descending order of c 9: Divide −→X to be {−→X 0 , −→X 1 , . . .
Revised: according to descending order of c 9: Divide −→X to be {−→X 0 , −→X 1 , . . .

Original: , −→X l } 10: // Train generative models 11: for −→X i ∈ {−→X 0 , −→X 1 , . . .
Revised: , −→X l } 10: // Train generative models 11: for −→X i ∈ {−→X 0 , −→X 1 , . . .

Original: , −→X l } do 12:
Revised: , −→X l } do 12:

Original: Initialize model parameters θ randomly 13: θ i =
Revised: Initialize model parameters θ randomly 13: θ i =

Original: GenerativeModel ( −→ X 0 , θ ) (cid:46) e.g. GAN 14: −→X 0 ← {−→X 0 ∪ −→X i } 15: end for 16: //
Revised: GenerativeModel ( −→ X 0 , θ ) (cid:46) e.g. GAN 14: −→X 0 ← {−→X 0 ∪ −→X i } 15: end for 16: //

Original: Search the optimal set 17: for θ i ∈ { θ 1 . . .
Revised: Search the optimal set 17: for θ i ∈ { θ 1 . . .

Original: , θ l +1 } do 18:
Revised: , θ l +1 } do 18:

Original: Generate data X with model of parameter θ i 19: s i =
Revised: Generate data X with model of parameter θ i 19: s i =

Original: QualityScore ( X ) (cid:46) e.g. FID 20: end for 21: i ∗ =
Revised: QualityScore ( X ) (cid:46) e.g. FID 20: end for 21: i ∗ =

Original: arg min s i , i = 1 , . . .
Revised: arg min s i , i = 1 , . . .

Original: , l +22:
Revised: , l +22:

Original: Return the optimal model parameter θ i ∗
Revised: Return the optimal model parameter θ i ∗

Original: Require: 1: X , GenerativeModel (), QualityScore () as in Algorithm2: |A| , cardinality of active set
Revised: Require: 1: X , GenerativeModel (), QualityScore () as in Algorithm2: |A| , cardinality of active set

Original: A
Revised: A

Original: Ensure: 3: //
Revised: Ensure: 3: //

Original: Solve centralities 4: c =
Revised: Solve centralities 4: c =

Original: Centrality ( X ) (cid:46) section A.
Revised: Centrality ( X ) (cid:46) section A.

Original: Cluster curriculum 6: Get sorted data −→X according to descending order of c 7: Divide −→X to be {−→X 0 , −→X 1 , . .
Revised: Cluster curriculum 6: Get sorted data −→X according to descending order of c 7: Divide −→X to be {−→X 0 , −→X 1 , . .

Original: . , −→X l } 8: // Train generative models 9: Initialize model parameters θ randomly 10: θ 

Revised: . , −→X l } 8: // Train generative models 9: Initialize model parameters θ randomly 10: θ

Original: =
Revised: =

Original: GenerativeModel ( −→X 0 , θ ) (cid:46) e.g. GAN 11: for −→X i ∈ {−→X 1 , . . .
Revised: GenerativeModel ( −→X 0 , θ ) (cid:46) e.g. GAN 11: for −→X i ∈ {−→X 1 , . . .

Original: , −→X l } do 12:
Revised: , −→X l } do 12:

Original: Derive −→A 0 by randomly sampling −→X13: −→A 

Revised: Derive −→A 0 by randomly sampling −→X13: −→A

Original: ← {−→A 0 ∪ −→X i } 14: //
Revised: ← {−→A 0 ∪ −→X i } 14: //

Original: Use pre-training model 15: θ i =
Revised: Use pre-training model 15: θ i =

Original: GenerativeModel ( −→A 0 , θ i − 1 ) 16: −→X 0 ← {−→X 0 ∪ −→X i } 17: end for 18: //
Revised: GenerativeModel ( −→A 0 , θ i − 1 ) 16: −→X 0 ← {−→X 0 ∪ −→X i } 17: end for 18: //

Original: Search the optimal set 19: for θ i ∈ { θ 1 . . .
Revised: Search the optimal set 19: for θ i ∈ { θ 1 . . .

Original: , θ l +1 } do 20:
Revised: , θ l +1 } do 20:

Original: Generate data X with model of parameter θ i by sampling a prior (cid:46) e.g. Gaussian 21: s i =
Revised: Generate data X with model of parameter θ i by sampling a prior (cid:46) e.g. Gaussian 21: s i =

Original: QualityScore ( X ) (cid:46) e.g. FID 22: end for 23: i ∗ =
Revised: QualityScore ( X ) (cid:46) e.g. FID 22: end for 23: i ∗ =

Original: arg min s i , i = 1 , . . .
Revised: arg min s i , i = 1 , . . .

Original: , l +24:
Revised: , l +24:

Original: Return the optimal model parameter θ i ∗
Revised: Return the optimal model parameter θ i ∗