Original: S TEIN B RIDGING : E NABLING M UTUAL R EINFORCEMENT BETWEEN E XPLICIT AND I MPLICIT G ENERATIVE M ODELS
Revised: S TEIN B RIDGING : E NABLING M UTUAL R EINFORCEMENT BETWEEN E XPLICIT AND I MPLICIT G ENERATIVE M ODELS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep generative models are generally categorized into explicit models and implicit models.
Revised: Deep generative models are generally categorized into explicit models and implicit models.

Original: The former deﬁnes an explicit density form, whose normalizing constant is often unknown; while the latter, including generative adversarial networks (GANs), generates samples without explicitly deﬁnes a density function.
Revised: The former deﬁnes an explicit density form, whose normalizing constant is often unknown; while the latter, including generative adversarial networks (GANs), generates samples without explicitly deﬁning a density function.

Original: In spite of substantial recent advances demonstrating the power of the two classes of generative models in many applications, both of them, when used alone, suffer from respective limitations and drawbacks.
Revised: In spite of substantial recent advances demonstrating the power of the two classes of generative models in many applications, both of them, when used alone, suffer from respective limitations and drawbacks.

Original: To mitigate these issues, we propose Stein Bridging , a novel joint training framework that connects an explicit  density estimator and an implicit sample generator with Stein discrepancy.
Revised: To mitigate these issues, we propose Stein Bridging , a novel joint training framework that connects an explicit (un-normalized) density estimator and an implicit sample generator with Stein discrepancy.

Original: We show that the Stein Bridge induces new regularization schemes for both explicit and implicit models.
Revised: We show that the Stein Bridge induces new regularization schemes for both explicit and implicit models.

Original: Convergence analysis and extensive experiments demonstrate that the Stein Bridging i) improves the stability and sample quality of the GAN training, and ii) facilitates the density estimator to seek more modes in data and alleviate the mode-collapse issue.
Revised: Convergence analysis and extensive experiments demonstrate that the Stein Bridging i) improves the stability and sample quality of the GAN training, and ii) facilitates the density estimator to seek more modes in data and alleviate the mode-collapse issue.

Original: Additionally, we discuss several applications of Stein Bridging and useful tricks in practical implementation used in our experiments.
Revised: Additionally, we discuss several applications of Stein Bridging and useful tricks in practical implementation used in our experiments.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep generative model, as a powerful unsupervised framework for learning the distribution of highdimensional multi-modal data, has been extensively studied in recent literature.
Revised: Deep generative model, as a powerful unsupervised framework for learning the distribution of highdimensional multi-modal data, has been extensively studied in recent literature.

Original: Typically, there are two types of generative models (Goodfellow et al., 2014).
Revised: Typically, there are two types of generative models (Goodfellow et al., 2014).

Original: Explicit models deﬁne an explicit (unnormalized) density function, while implicit models learn to sample from the distribution without explicitly deﬁne a density function.
Revised: Explicit models deﬁne an explicit (unnormalized) density function, while implicit models learn to sample from the distribution without explicitly deﬁning a density function.

Original: Explicit models have wide applications in undirected graphical models (LeCun et al., 2006;
Revised: Explicit models have wide applications in undirected graphical models (LeCun et al., 2006;

Original: Salakhutdinov & Hinton, 2009; Hinton et al., 2006; Ngiam et al., 2011), random graph theory (Robins et al., 2007), energy-based reinforcement learning (Haarnoja et al., 2017), etc.
Revised: Salakhutdinov & Hinton, 2009; Hinton et al., 2006; Ngiam et al., 2011), random graph theory (Robins et al., 2007), energy-based reinforcement learning (Haarnoja et al., 2017), etc.

Original: However, the unknown normalizing constant makes the model hard to train and sample from, and the explicit models might not be able to capture the complex structure of true samples while maintaining tractability.
Revised: However, the unknown normalizing constant makes the model hard to train and sample from, and the explicit models might not be able to capture the complex structure of true samples while maintaining tractability.

Original: In contrast, implicit models are more ﬂexible in training and easy to sample from, and in pariticular, generative adverarial networks (GANs) have shown great power in learning representations of images, natural languages, graphs, etc.
Revised: In contrast, implicit models are more ﬂexible in training and easy to sample from, and in pariticular, generative adverarial networks (GANs) have shown great power in learning representations of images, natural languages, graphs, etc.

Original: (Goodfellow et al., 2014; Radford et al., 2016;
Revised: (Goodfellow et al., 2014; Radford et al., 2016;

Original: Arjovsky et al., 2017; Brock et al., 2019).
Revised: Arjovsky et al., 2017; Brock et al., 2019).

Original: Nevertheless, due to the minimax game between generator and discriminator/critic in GANs, the training process often suffers from instability, and produces undesirable samples often associated with missing modes in data or generating extra modes out of data.
Revised: Nevertheless, due to the minimax game between generator and discriminator/critic in GANs, the training process often suffers from instability, and produces undesirable samples often associated with missing modes in data or generating extra modes out of data.

Revised: Therefore, it motivates us to consider jointly learning of two models that can presumably compensate and reinforce each other in the training process.

Original: More discussion on related work is in Appendix A.
Revised: (More discussions about related works are in Appendix A.)

Original: There are  situations where we  need both an explicit  density and a ﬂexible implicit sampler.
Revised: Furthermore, there are many of situations where we do need both an explicit (unnormalized) density and a ﬂexible implicit sampler.

Original: For sample evaluation, it is not enough to merely distinguish samples from real to faked one, and one may also expect to provide ﬁne-grained evaluation on generated samples, where the energy values given by the explicit models can be a good metric (Dai et al., 2017).
Revised: For sample evaluation, it is not enough to merely distinguish samples between real and faked one, and one may also expect to provide ﬁne-grained evaluation on generated samples, where the energy values given by the explicit models can be a good metric (Dai et al., 2017).

Original: Another situation is outlier detection.
Revised: Another situation is outlier detection.

Original: Implicit models often leverage all true samples (possibly mixed with corrupted samples) as true examples for training.
Revised: Implicit models often leverage all true samples (possibly mixed with corrupted samples) as true examples for training.

Original: To make up for the issue, explicit models could help to detect out-of-distribution samples via the estimated densities (Zhai et al., 2016).
Revised: To make up for the issue, explicit models could help to detect out-of-distribution samples via the estimated densities (Zhai et al., 2016).

Original: Also, when given insufﬁcient observed samples, explicit models may fail to capture an accurate distribution, in which case implicit model may help with data augmentation and facilitate training for density estimation.
Revised: Also, when given insufﬁcient observed samples, explicit models may fail to capture an accurate distribution, in which case implicit model may help with data augmentation and facilitate training for density estimation.

Original: These situations motivate us to combine both of the worldsin an effective way so as to make the two models compensate and reinforce each other.
Revised: These situations inspire us to combine both of the worlds so as to take advantage of two models in an effective way.

Original: In this work, we aim at jointly learning explicit and implicit generative models.
Revised: In this work, we aim at jointly learning explicit and implicit generative models.

Original: In our framework, an implicit energy model is used to estimate the unnormalized densities of true samples via minimizing a Stein discrepancy; in the meantime, an explicit generator model is exploited to minimize the Wasserstein metric (or Jensen-Shannon divergence) between distributions of true and generated samples.
Revised: In our framework, an explicit energy model is used to estimate the unnormalized densities of true samples via minimizing a Stein discrepancy; in the meantime, an implicit generator model is exploited to minimize the Wasserstein metric (or Jensen-Shannon divergence) between distributions of true and generated samples.

Original: On top of these, another Stein discrepancy, acting as a bridge between implicit generated samples and explicit estimated densities, is introduced and pushes the two models to achieve a consensus.
Revised: On top of these, another Stein discrepancy, acting as a bridge between implicit generated samples and explicit estimated densities, is introduced and pushes the two models to achieve a consensus.

Original: We show that the Stein bridge allows the two generative models to reinforce each other by imposing new regularizations on both models, which help the generator to output high-quality samples and facilitate the energy model to avoid mode-collapse.
Revised: We show that the Stein bridge allows the two generative models to reinforce each other by imposing new regularizations on both models, which help the generator to output high-quality samples and facilitate the energy model to avoid mode-collapse.

Original: Moreover, we show that the joint training helps to stabilize GAN training via a convergence analysis.
Revised: Moreover, we show that the joint training helps to stabilize GAN training via a convergence analysis.

Original: Extensive experiments on various tasks verify our theoretical ﬁndings as well as demonstrate the superiority of proposed methods compared with existing deep energy models and GAN-based models.
Revised: Extensive experiments on various tasks verify our theoretical ﬁndings as well as demonstrate the superiority of proposed methods compared with existing deep energy models and GAN-based models.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: In this section, we brieﬂy provide some technical background used in our model.
Revised: In this section, we brieﬂy provide some technical background used in our model.

Original: Energy Model.
Revised: Energy Model.

Original: The energy model assigns each data x  with a scalar energy value E φ ( x ) , where E φ ( · ) is called the energy function and is parameterized by φ .
Revised: The energy model assigns each data x ∈ R d with a scalar energy value E φ ( x ) , where E φ ( · ) is called the energy function and is parameterized by φ .

Original: The model is expected to assign low energy to true samples according to a Gibbs distribution p φ ( x ) =
Revised: The model is expected to assign low energy to true samples according to a Gibbs distribution p φ ( x ) =

Original: exp {− E φ ( x ) } /Z
Revised: exp {− E φ ( x ) } /Z

Original: φ , where Z φ is a normalizing constant dependent on φ .
Revised: φ , where Z φ is a normalizing constant dependent on φ .

Original: The normalizing term Z φ is often hard to compute, making the training intractable, and various methods are proposed to detour such term (see Appendix A).
Revised: The normalizing term Z φ is often hard to compute, making the training intractable, and various methods are proposed to detour such term (see Appendix A).

Original: Stein Discrepancy.
Revised: Stein Discrepancy.

Original: Stein discrepancy (Gorham & Mackey, 2015; Liu et al., 2016;
Revised: Stein discrepancy (Gorham & Mackey, 2015; Liu et al., 2016;

Original: Chwialkowski et al., 2016; Oates et al., 2017) is a measure of closeness between two probability distributions that does not require the knowledge for the normalizing constant of one of the compared distributions.
Revised: Chwialkowski et al., 2016; Oates et al., 2017) is a measure of closeness between two probability distributions that does not require the knowledge for the normalizing constant of one of the compared distributions.

Original: Let P and Q be two probability distributions on X ⊂ R d , and assume Q has a  density q .
Revised: Let P and Q be two probability distributions on X ⊂ R d , and assume Q has a (unnormalized) density q .

Original: The Stein discrepancy S ( P , Q ) is deﬁned as
Revised: The Stein discrepancy S ( P , Q ) is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where F is often chosen to be a Stein class (see, e.g., Deﬁnition 2.1 in Liu et al.
Revised: where F is often chosen to be a Stein class (see, e.g., Deﬁnition 2.1 in Liu et al.

Revised: , f : R d →

Revised: R d (cid:48) is a vector-valued function called Stein critic and φ is an operation that transforms a d × d (cid:48) matrix into a scalar value.

Revised: One common choice 1 of φ is trace operation when d (cid:48)

Original: (2016)), or a unit ball in some reproducing kernel Hilbert space (RKHS) with a positive deﬁnite kernel k .
Revised: = d . If F is a unit ball in some reproducing kernel Hilbert space (RKHS) with a positive deﬁnite kernel k , it induces

Original: The latter is referred to as Kernel Stein Discrepancy (KSD).
Revised: Kernel Stein Discrepancy (KSD).

Original: We refer to Appendix B for more details.
Revised: We provide more details in Appendix B.

Original: Wasserstein Metric.
Revised: Wasserstein Metric.

Original: Wasserstein metric is suitable for measuring distances between two distributions with non-overlapping supports (Arjovsky et al., 2017).
Revised: Wasserstein metric is suitable for measuring distances between two distributions with non-overlapping supports (Arjovsky et al., 2017).

Original: The Wasserstein-1 metric between distributions P and Q is deﬁned as W ( P , Q ) :
Revised: The Wasserstein-1 metric between distributions P and Q is deﬁned as W ( P , Q ) :

Original: = min γ E ( x , y ) ∼ γ [ (cid:107) x − y (cid:107) ] , where the minimization is over all joint distributions with marginals P and Q .
Revised: = min γ E ( x , y ) ∼ γ [ (cid:107) x − y (cid:107) ] , where the minimization is over all joint distributions with marginals P and Q .

Original: By Kantorovich-Rubinstein duality, it has a dual representation
Revised: By Kantorovich-Rubinstein duality, it has a dual representation

Original: [Equation]
Revised: [Equation]

Original: where the maximization is over all 1-Lipschitz continuous functions.
Revised: where the maximization is over all 1-Lipschitz continuous functions.

Original: Sobolev space and Sobolev dual norm.
Revised: Sobolev space and Sobolev dual norm.

Original: Use L 2 to denote the canonical Hilbert space on R d equipped with an inner product (cid:104) u, v (cid:105)
Revised: Use L 2 to denote the canonical Hilbert space on R d equipped with an inner product (cid:104) u, v (cid:105)

Original: L 2 :
Revised: L 2 :

Original: = (cid:82) R d uvd x .
Revised: = (cid:82) R d uvd x .

Original: The Sobolev space H 1 is deﬁned as the closure of C ∞ 0 , the set of smooth functions on R d with compact support, with respect to the norm (cid:107) u (cid:107) H 1 :=
Revised: The Sobolev space H 1 is deﬁned as the closure of C ∞ 0 , the set of smooth functions on R d with compact support, with respect to the norm (cid:107) u (cid:107) H 1 :=

Original: (cid:0) (cid:82) R d ( u 2 + (cid:107)∇ u (cid:107) 22 ) d x (cid:1) 1 / 2 .
Revised: (cid:0) (cid:82) R d ( u 2 + (cid:107)∇ u (cid:107) 22 ) d x (cid:1) 1 / 2 .

Original: For v ∈ L 2 , its Sobolev dual norm (cid:107) v (cid:107) H − 1 is deﬁned by (Evans & Society, 2010)
Revised: For v ∈ L 2 , its Sobolev dual norm (cid:107) v (cid:107) H − 1 is deﬁned by (Evans & Society, 2010)

Original: [Equation]
Revised: [Equation]

Revised: The constraint (cid:82) R d u ( x ) d x = 0 is necessary in ensuring the ﬁniteness of the supremum.

Original: Therefore, (cid:107)·(cid:107) H − 1 can be viewed as a measure of smoothness, which measures the similarity (in terms of largest L 2 -norm) between v and a subset of smooth functions in H 1 .
Revised: (cid:107)·(cid:107) H − 1 can be viewed as a measure of smoothness, which measures the similarity (in terms of largest L 2 -norm) between v and a subset of smooth functions in H 1 .

Original:  P ROPOSED M ODEL
Revised:  P ROPOSED M ODEL

Original: In this section, we formulate our model, Stein Bridging , and highlight its regularization effects.
Revised: In this section, we formulate our model, Stein Bridging , and highlight its regularization effects.

Original: 3. M ODEL F ORMULATION
Revised: 3. M ODEL F ORMULATION

Original: We denote by P real the underlying real distribution from which the data { x } are sampled.
Revised: We denote by P real the underlying real distribution from which the data { x } are sampled.

Original: We simultaneously learn two generative models – one explicit and one implicit – that represent estimates of P real .
Revised: We simultaneously learn two generative models – one explicit and one implicit – that represent estimates of P real .

Original: The explicit generative model has an explicit probability density P E proportional to exp( − E ( x )) , where E is referred to as an energy function.
Revised: The explicit generative model has an explicit probability density P E proportional to exp( − E ( x )) , where E is referred to as an energy function.

Original: The implicit generative model transforms an easy-to-sample random noise z with distribution P 0 via a generator G to a generated sample (cid:101) x
Revised: The implicit generative model transforms an easy-to-sample random noise z with distribution P 0 via a generator G to a generated sample (cid:101) x

Original: = G ( z ) with distribution P G .
Revised: = G ( z ) with distribution P G .

Original: We use the Stein discrepancy as a measure of closeness between the explicit  density P E and the real distribution P real , and use the Wasserstein metric as a measure of closeness between the implicit distribution P G and P real .
Revised: We use the Stein discrepancy as a measure of closeness between the explicit unnormalized density P E and the real distribution P real , and use the Wasserstein metric as a measure of closeness between the implicit distribution P G and P real .

Original: To jointly learn the two generative models P G and P E , arguably the most straightforward approach is to minimize the sum of the Stein discrepancy and the Wasserstein metric:
Revised: To jointly learn the two generative models P G and P E , arguably the most straightforward approach is to minimize the sum of the Stein discrepancy and the Wasserstein metric:

Original: [Equation]
Revised: [Equation]

Original: where λ ≥ 0 is a weight coefﬁcient.
Revised: where λ ≥ 0 is a weight coefﬁcient.

Original: However, this approach appears no different than learning the two generative models separately.
Revised: However, this approach appears no different than learning the two generative models separately.

Original: To better train the model, we incorporate the objective another term S ( P G , P E ) – called the Stein bridge – that measures the closeness between the explicit  density P E and the implicit distribution P G :
Revised: To better train the model, we incorporate the objective another term S ( P G , P E ) – called the Stein bridge – that measures the closeness between the explicit unnormalized density P E and the implicit distribution P G :

Original: [Equation]
Revised: [Equation]

Original: ) where λ 1 , λ 2 ≥ 0 are weight coefﬁcients.
Revised: where λ 1 , λ 2 ≥ 0 are weight coefﬁcients.

Original: Although the Stein bridge might seem redundant mathematically, we show that it helps regularize the models in Section 3.2.
Revised: Although the Stein bridge might seem redundant mathematically, we show that it helps regularize the models in Section 3.2.

Original: The Wasserstein term in (3) is implemented using its equivalent dual representation (2).
Revised: The Wasserstein term in (3) is implemented using its equivalent dual representation (2).

Original: The two
Revised: The two

Original: Stein terms in (3) can be implemented using (1) with either a Stein critic parameterized by a neural network, or the Kernel Stein Discrepancy.
Revised: Stein terms in (3) can be implemented using (1) with either a Stein critic parameterized by a neural network, or the Kernel Stein Discrepancy.

Original: To reduce the computational cost, the two Stein critics share their parameters, namely, kernels or neural networks.
Revised: To reduce the computational cost, the two Stein critics share their parameters, namely, kernels or neural networks.

Original: A scheme of our framework is presented in Fig.
Revised: A scheme of our framework is presented in Fig.

Original: We also discuss some related works that attempt to combine both of the worlds (such as energy-based GAN, contrastive learning and cooperative learning) in Appendix A.3, and highlight the difference between our method and theirs in terms of the objective in Table 1.
Revised: We also discuss some related works that attempt to combine both of the worlds (such as energy-based GAN, contrastive learning and cooperative learning) in Appendix A.3, and highlight the difference between our method and theirs in terms of the objective in Table 1.

Original: Remark .
Revised: Remark .

Original: In general, we can also choose other statistical distances in (3) to measure closeness between probability distributions.
Revised: In general, we can also choose other statistical distances in (3) to measure closeness between probability distributions.

Original: For example, the Wasserstein metric W ( P real , P G ) can be replaced by other common choices for implicit generative models, such as Jensen-Shannon divergence used in the original GAN paper (Goodfellow et al., 2014).
Revised: For example, the Wasserstein metric W ( P real , P G ) can be replaced by other common choices for implicit generative models, such as Jensen-Shannon divergence used in the original GAN paper (Goodfellow et al., 2014).

Original: If the normalizing constant of P E is known or easy to calculate, one can replace the Stein discrepancy by the Kullback-Leibler divergence, which is equivalent to the maximum likelihood estimation.
Revised: If the normalizing constant of P E is known or easy to calculate, one can replace the Stein discrepancy by the Kullback-Leibler divergence, which is equivalent to the maximum likelihood estimation.

Original: We present details for model speciﬁcations in various forms and training algorithm in Appendix E.2.[Figure] els, where D 1 :=
Revised: We present details for model speciﬁcations in various forms and training algorithm in Appendix E.2.

Original: D 1 ( P real , P G ) ,

Original: D 2 :

Original: = D 2 ( P real , P E ) and D 3 :=

Original: D 3 ( P G , P E ) denote general statistical distances between two distributions.

Original: 3. R EGULARIZATION E FFECTS BY V IRTUE OF T HE S TEIN B RIDGE
Revised: 3. R EGULARIZATION E FFECTS BY V IRTUE OF T HE S TEIN B RIDGE

Original: The intuitive motivation of the Stein bridge term in (3) is to push the two models to achieve a consensus.
Revised: The intuitive motivation of the Stein bridge term in (3) is to push the two models to achieve a consensus.

Original: In this subsection, we theoretically show that the Stein bridge allows the two models to reinforce each other by imposing regularizations on the critics.
Revised: In this subsection, we theoretically show that the Stein bridge allows the two models to reinforce each other by imposing regularizations on the critics.

Original: 3.2. K ERNEL S OBOLEV DUAL NORM REGULARIZATION ON THE W ASSERSTEIN CRITIC
Revised: 3.2. K ERNEL S OBOLEV DUAL NORM REGULARIZATION ON THE W ASSERSTEIN CRITIC

Original: Weﬁrst show the regularization effect of the Stein bridge on the Wasserstein critic.
Revised: We show the regularization effect of the Stein bridge on the Wasserstein critic.

Original: Fixing the energy function E , consider the max-min problem over the Wasserstein critic D and the generator G :
Revised: Deﬁne the kernel Sobolev dual norm as

Original: [Equation]
Revised: [Equation]

Original: We deﬁne the kernel Sobolev dual norm as
Revised: [Table] Model Objective GAN

Original: [Equation]
Revised: D 1 Energy Model D 2 Energy-based GAN (Zhao et al. D

Revised: Contrastive Learning (Kim & Bengio (2017))

Revised: D

Original: , which can be viewed as a kernel generalization of the Sobolev dual norm deﬁned in Section 2, which reduces to the Sobolev dual norm when k ( x , x (cid:48) )
Revised: Cooperative Learning (Xie et al. D 2 + D Stein Bridging (ours) D 1 + D 2 + D which can be viewed as a kernel generalization of the Sobolev dual norm deﬁned in Section 2, which reduces to the Sobolev dual norm when k ( x , x (cid:48) )

Original: =
Revised: =

Original: I ( x = x (cid:48) ) and P being the Lebesgue measure.
Revised: I ( x = x (cid:48) ) and P being the Lebesgue measure.

Original: Assuming that { P G } G exhausts all probability distributions, we have the following result.
Revised: We have the following result.

Original: Theorem 1.
Revised: Theorem 1.

Revised: Assume that { P G } G exhausts all continuous probability distributions and S is chosen as kernel Stein discrepancy.

Original: Formally, problem (4) is equivalent to
Revised: Then problem (3) is equivalent to

Original: [Equation]
Revised: [Equation]

Original: Note that if D is an optimal Wasserstein critic in (2)

Original: , so does D − r , r ∈ R .

Original: This is consistent to the penalty term inf t ∈ R (cid:107) D − t (cid:107) H − 1 ( P E ; k ) , since the penalties are identical for D and D − r .

Original: According to Section 2, the regularization term would penalize the non-smoothness of the Wasserstein critic D , which is in the same spirit of gradient-based penalty (e.g., Gulrajani et al.
Revised: According to Section 2, the regularization term would penalize the non-smoothness of the Wasserstein critic D , which is in the same spirit of gradient-based penalty (e.g., Gulrajani et al.

Original: ; Roth et al.
Revised: Roth et al.

Original: (2017)), but with a new way to encouraging smoothness.
Revised: , but with a new way to encouraging smoothness.

Original: Another way to interpret the Sobolev dual norm penalty is by observing that if k ( x , x (cid:48) )
Revised: Another way to interpret the Sobolev dual norm penalty is by observing that if k ( x , x (cid:48) )

Original: =
Revised: =

Original: I ( x = x (cid:48) ) , and E P E
Revised: I ( x = x (cid:48) ) and and E P E

Original: [ D− t ] = [-
-] (Villani, 2008), then
Revised: [ D ] = [-
-]0 (Villani, 2008), then

Original: [Equation]
Revised: [Equation]

Original: , where W 2 denotes the 2-Wasserstein metric.
Revised: where W 2 denotes the 2-Wasserstein metric.

Original: Therefore, the regularization ensures that D would not change suddenly on the high-density region of P E , and the explicit model reinforces the learning of the Wasserstein critic.
Revised: Therefore, the regularization ensures that D would not change suddenly on the high-density region of P E , and the explicit model reinforces the learning of the Wasserstein critic.

Original: 3.2. L IPSCHITZ R EGULARIZATION ON THE S TEIN CRITIC
Revised: 3.2. L IPSCHITZ R EGULARIZATION ON THE S TEIN CRITIC

Original: We next investigate how the Stein bridge helps to regularize the Stein critic.
Revised: We next investigate how the Stein bridge helps to regularize the Stein critic.

Original: Recall that the two
Revised: Recall that the two

Original: Stein terms in (3) share the same Stein critic.
Revised: Stein terms in (3) share the same Stein critic.

Original: Fixing the energy function E , consider the max-min problem over the Stein critic f and the generator G : [Equation] Assuming that { P G } G exhausts all probability distributions, we have the following result.
Revised: We have the following result.

Original: Theorem 2.
Revised: Theorem 2.

Revised: Assume { P G } G exhausts all continuous probability distributions, and the Stein class deﬁning the Stein discrepancy is compact (in some linear topological space).

Original: Problem (5) is equivalent to
Revised: Then problem (3) is equivalent to

Original: [Equation]
Revised: [Equation]

Original: where Lip( A P E f  ) denotes the Lipschitz constant of the function A P E f .
Revised: where M λ 2 A P E f ( · ) denotes the (generalized) Moreau-Yosida regularization of the function A P E f with parameter λ 2 , i.e., M λ 2 A P E f

Original:  Theorem 2 shows that the Stein bridge, together with the Wasserstein metric W ( P real , P G ) , plays as a smoothness regularization on the Stein critic f via  the constraint Lip(tr( f (cid:62) ∇ log P E+ ∇ f)) ≤ 1 /λ 2 . The regularization will penalize large variation of values given by Stein operators on adjacent instances and further encourage the energy model to seek more modes in data instead of focusing on some dominated modes, thus helping to alleviate the mode-collapse issue.
Revised: ( x ) = min y ∈X {A P E f ( y ) + 1 λ 2 || x − y ||} . Theorem 2 shows that the Stein bridge, together with the Wasserstein metric W ( P real , P G ) , plays as a smoothness regularization on the Stein critic f via Moreau-Yosida regularization, which smoothens the Stein operator A P E f and further encourages the energy model to seek more modes in data instead of focusing on some dominated modes, thus helps alleviate the mode-collapse issue.

Original: To the best of our knowledge, this suggests a novel regularization scheme for Stein-based GAN.
Revised: To the best of our knowledge, this suggests a novel regularization scheme for Stein-based GAN.

Original:  C ONVERGENCE A NALYSIS
Revised:  C ONVERGENCE A NALYSIS

Original: In Section 3.2, we justify Stein Bridging by showing the regularization effects.
Revised: In Section 3.2, we justify Stein Bridging by showing the regularization effects.

Original: In this section, we further show that it could help to stabilize GAN training with local convergence guarantee.
Revised: In this section, we further show that it could help to stabilize GAN training with local convergence guarantee.

Original: To this end, we ﬁrst compare the behaviors of WGAN, likelihood- and entropy-regularized WGAN, and our Stein Bridging under SGD via an easy to comprehend toy example.
Revised: To this end, we ﬁrst compare the behaviors of WGAN, likelihood- and entropy-regularized WGAN, and our Stein Bridging under SGD via an easy to comprehend toy example.

Original: Then we give a formal result that interprets why the introduction of  density estimator could stablize GAN training and help for convergence.
Revised: Then we give a formal result that interprets why the introduction of (unnormalized) density estimator could stablize GAN training and help for convergence.

Original: 4. A NALYSIS OF A L INEAR S YSTEM
Revised: 4. A NALYSIS OF A L INEAR S YSTEM

Original: The training for minimax game in GAN is difﬁcult.
Revised: The training for minimax game in GAN is difﬁcult.

Original: When using traditional gradient methods, the training would suffer from some oscillatory behaviors (Goodfellow (2017); Liang & Stokes (2019)).
Revised: When using traditional gradient methods, the training would suffer from some oscillatory behaviors (Goodfellow (2017); Liang & Stokes (2019)).

Original: In order to better understand the optimization behaviors, we ﬁrst study a one-dimension linear system that provides some insights on this problem.
Revised: In order to better understand the optimization behaviors, we ﬁrst study a one-dimension linear system that provides some insights on this problem.

Original: Note that such toy example (or a similar one) is also utilized by Gidel et al.
Revised: Note that such toy example (or a similar one) is also utilized by Gidel et al.

Original: ; Nagarajan & Kolter (2017) to shed lights on the instability of WGAN training 1 . Consider a linear critic D ψ ( x ) =
Revised: ; Nagarajan & Kolter (2017) to shed lights on the instability of WGAN training 2 . Consider a linear critic D ψ ( x ) =

Original: ψx and generator G θ ( z ) = θx .
Revised: ψx and generator G θ ( z ) = θx .

Original: Then the Wasserstein
Revised: Then the Wasserstein

Original: GAN objective can be written as a constrained bilinear problem: min θ max | ψ |≤ 1 ψ E [ x ] − ψθ E
Revised: GAN objective can be written as a constrained bilinear problem: min θ max | ψ |≤ 1 ψ E [ x ] − ψθ E

Original: [ z ] , which could be further simpliﬁed as an unconstrained version (the behaviors could be generalized to multi-dimensional cases (Gidel et al.
Revised: [ z ] , which could be further simpliﬁed as an unconstrained version (the behaviors could be generalized to multi-dimensional cases (Gidel et al.

Original: [Equation]
Revised: [Equation]

Original: Unfortunately, such simple objective cannot guarantee convergence by traditional gradient methods like SGD with alternate updating 2 : θ k +

Revised: Unfortunately, such simple objective cannot guarantee convergence by traditional gradient methods like SGD with alternate updating 3 : θ k +

Original: =
Revised: =

Original: θ k − ηψ k , , ψ k +1 =
Revised: θ k + ηψ k , , ψ k +1 =

Original: ψ k + ηθ k +1  .
Revised: ψ k + η (1 − θ k +1 ) .

Original: Such optimization would suffer from an oscillatory behavior, i.e., the updated parameters go around the optimum point ( [ θ ∗ , ψ ∗ ] =
Revised: Such optimization would suffer from an oscillatory behavior, i.e., the updated parameters go around the optimum point ( [ ψ ∗ , θ ∗ ] =

Original: [1 , 0] ) forming a circle without converging to the centrality, which is shown in Fig. 2(a).
Revised: [0 , 1] ) forming a circle without converging to the centrality, which is shown in Fig. 2(a).

Original: A recent study in Liang & Stokes (2019) theoretically show that such oscillation is due to the interaction term in (6).
Revised: A recent study in Liang & Stokes (2019) theoretically show that such oscillation is due to the interaction term in (4).

Original: One solution to the instability of GAN training is to add (likelihood) regularization, which has been widely studied by recent literatures (Warde-Farley & Bengio (2017); Li & Turner (2018)).
Revised: One solution to the instability of GAN training is to add (likelihood) regularization, which has been widely studied by recent literatures (Warde-Farley & Bengio (2017); Li & Turner (2018)).

Original: With regularization term, the objective changes into min θ max | ψ |≤ 1 ψ E [ x ] − ψθ E
Revised: With regularization term, the objective changes into min θ max | ψ |≤ 1 ψ E [ x ] − ψθ E

Original: [ z ] − λ E [log µ ( θz )] , where µ ( · ) denotes the likelihood function and λ is a hyperparameter.
Revised: [ z ] − λ E [log µ ( θz )] , where µ ( · ) denotes the likelihood function and λ is a hyperparameter.

Original: A recent study (Tao et al.
Revised: A recent study (Tao et al.

Original: (2019)) proves that when λ < 0 (likelihood-regularization), the extra term is equivalent to maximizing sample evidence, helping to stabilize GAN training; when λ > 0 (entropy-regularization), the extra term maximizes sample entropy, which encourages diversity of generator.
Revised: (2019)) proves that when λ < 0 (likelihood-regularization), the extra term is equivalent to maximizing sample evidence, helping to stabilize GAN training; when λ > 0 (entropy-regularization), the extra term maximizes sample entropy, which encourages diversity of generator.

Original: Here we consider a
Revised: Here we consider a

Original: Gaussian likelihood function for generated sample x (cid:48) , µ ( x (cid:48) )
Revised: Gaussian likelihood function for generated sample x (cid:48) , µ ( x (cid:48) )

Original: =
Revised: =

Original: exp( − 12 ( x (cid:48) − b ) 2 ) which is up to a constant, and then the objective becomes (see Appendix D.1 for details):
Revised: exp( − 12 ( x (cid:48) − b ) 2 ) which is up to a constant, and then the objective becomes (see Appendix D.1 for details):

Original: [Equation]
Revised: [Equation]

Original: [Figure] [Table] The above system would converge with λ < 0 and diverge with λ > 0 in gradient-based optimization, shown in Fig. 2(a).
Revised: The above system would converge with λ < 0 and diverge with λ > 0 in gradient-based optimization, shown in Fig. 2(a).

Original: Another issue of likelihood-regularization is that the extra term changes the optimum point and makes the model converge to a biased distribution, as proved by Tao et al.
Revised: Another issue of likelihood-regularization is that the extra term changes the optimum point and makes the model converge to a biased distribution, as proved by Tao et al.

Original: In this case, one can verify that the optimum point becomes [ ψ ∗ , θ ∗ ] =
Revised: In this case, one can verify that the optimum point becomes [ ψ ∗ , θ ∗ ] =

Original: [ − λ, 1] , resulting a bias.
Revised: [ − λ, 1] , resulting a bias.

Original: To avoid this issue, Tao et al.
Revised: To avoid this issue, Tao et al.

Original: (2019) proposes to temporally decrease | λ | through training.
Revised: (2019) proposes to temporally decrease | λ | through training.

Original: However, such method would also be stuck in oscillation when | λ | gets close to zero as is shown in Fig. 2(a).
Revised: However, such method would also be stuck in oscillation when | λ | gets close to zero as is shown in Fig. 2(a).

Original: Finally, let us consider our proposed model.
Revised: Finally, let us consider our proposed model.

Original: We also simplify the density estimator as a basic energy model p φ ( x ) = exp(  12 x 2 + φx ) whose score function is ∇ x log p φ ( x ) =  x + φ .
Revised: We also simplify the density estimator as a basic energy model p φ ( x ) = exp( − 12 x 2 − φx ) whose score function is ∇ x log p φ ( x ) = − x − φ .

Original: Then if we specify the two Stein discrepancies in (3) as KSD, we have the objective,
Revised: Then if we specify the two Stein discrepancies in (3) as KSD, we have the objective (see Appendix D.1 for details),

Original: [Equation]
Revised: [Equation]

Original: Interestingly, one can verify that for ∀ λ 1 , λ 2 , the optimum point remains the same [ ψ ∗ , θ ∗ , φ ∗ ] =
Revised: Interestingly, one can verify that for ∀ λ 1 , λ 2 , the optimum point remains the same [ ψ ∗ , θ ∗ , φ ∗ ] =

Original: [0 , 1 , − 1] .
Revised: [0 , 1 , − 1] .

Original: Then we show that the optimization can guarantee convergence to [ ψ ∗ , θ ∗ , φ ∗ ] .
Revised: Then we show that the optimization can guarantee convergence to [ ψ ∗ , θ ∗ , φ ∗ ] .

Original: Proposition 1.
Revised: Proposition 1.

Original: Using alternate SGD for (8) geometrically decreases the square norm N t = | ψ t | 2 + | θ − [Equation] | 2 + | φ + 1 | 2 , for any 0 < η < 1 with λ 1 = λ 2 = 1 ,
Revised: Using alternate SGD for (6) geometrically decreases the square norm N t = | ψ t | 2 + | θ − 1 | 2 + | φ + 1 | 2 , for any 0 < η < 1 with λ 1 = λ 2 = 1 ,

Original: [Equation]
Revised: [Equation]

Original: In Fig. 2(a), we can see that Stein Bridging achieves a good convergence to the right optimum.
Revised: In Fig. 2(a), we can see that Stein Bridging achieves a good convergence to the right optimum.

Original: Compared with (6), the objective (8) adds a new bilinear term φ · θ , which acts like a connection between the two generator and estimator, and two other quadratic terms, which help to push the values to decrease through training.
Revised: Compared with (4), the objective (6) adds a new bilinear term φ · θ , which acts like a connection between the two generator and estimator, and two other quadratic terms, which help to push the values to decrease through training.

Original: The added terms and the original terms in (8) cooperate to guarantee convergence to a unique optimum.
Revised: The added terms and the original terms in (6) cooperate to guarantee convergence to a unique optimum.

Original: (More discussions in Appendix D.1).
Revised: (More discussions in Appendix D.1).

Original: We further generalize the analysis to multi-dimensional bilinear system F ( ψ , θ )
Revised: We further generalize the analysis to multi-dimensional bilinear system F ( ψ , θ )

Original: = θ (cid:62)
Revised: = θ (cid:62)

Original: A ψ − b (cid:62) θ − c (cid:62) ψ which is extensively used by researches for analysis of GAN stability (Goodfellow (2017);
Revised: A ψ − b (cid:62) θ − c (cid:62) ψ which is extensively used by researches for analysis of GAN stability (Goodfellow (2017);

Original: Gemp & Mahadevan (2018); Liang & Stokes (2019); Gidel et al.
Revised: Gemp & Mahadevan (2018); Liang & Stokes (2019); Gidel et al.

Original: For any bilinear system, with the added term H ( φ , θ ) = 12 ( θ + φ ) (cid:62) B ( θ + φ )
Revised: For any bilinear system, with the added term H ( φ , θ ) = 12 ( θ + φ ) (cid:62) B ( θ + φ )

Original: where B =
Revised: where B =

Original: ( AA (cid:62) )
Revised: ( AA (cid:62) )

Original: 12 to the objective, we can prove that i) the optimum point remains the same as the original system (Proposition 2) and ii) using alternate SGD algorithm for the new objective can guarantee convergence (Theorem 4).
Revised: 12 to the objective, we can prove that i) the optimum point remains the same as the original system (Proposition 2) and ii) using alternate SGD algorithm for the new objective can guarantee convergence (Theorem 4).

Original: The results are given in Appendix D.3.
Revised: The results are given in Appendix D.3.

Original: 4. L OCAL C ONVERGENCE FOR A G ENERAL M ODEL
Revised: 4. L OCAL C ONVERGENCE FOR A G ENERAL M ODEL

Original: To study the convergence for Stein Bridging, we proceed to consider a general optimization objective
Revised: To study the convergence for Stein Bridging, we proceed to consider a general optimization objective

Original: [Equation]
Revised: [Equation]

Original: where L ( θ , ψ , φ ) =
Revised: where L ( θ , ψ , φ ) =

Original: F ( θ , ψ )+ H ( θ , φ ) , and ω f
Revised: F ( θ , ψ )+ H ( θ , φ ) , and ω f

Original: = [ θ , ψ ] and ω h =
Revised: = [ θ , ψ ] and ω h =

Original: [ θ , φ ] ( θ is a shared parameter set).
Revised: [ θ , φ ] ( θ is a shared parameter set).

Original: Use ω ∗ =
Revised: Use ω ∗ =

Original: [ θ ∗ , ψ ∗ , φ ∗ ] to denote the optimum point of L and ω ∗ f
Revised: [ θ ∗ , ψ ∗ , φ ∗ ] to denote the optimum point of L and ω ∗ f

Original: = [ θ ∗ , ψ ∗ ] , ω ∗ h
Revised: = [ θ ∗ , ψ ∗ ] , ω ∗ h

Original: =
Revised: =

Original: [ θ ∗ , φ ∗ ] represent the optimum points of F and H respectively.
Revised: [ θ ∗ , φ ∗ ] represent the optimum points of F and H respectively.

Original: Deﬁne Ω f
Revised: Deﬁne Ω f

Original: = Ω θ × Ω ψ and Ω h
Revised: = Ω θ × Ω ψ and Ω h

Original: = Ω θ × Ω φ , where Ω θ , Ω ψ , Ω φ denote constraint sets for θ , ψ , φ respectively.
Revised: = Ω θ × Ω φ , where Ω θ , Ω ψ , Ω φ denote constraint sets for θ , ψ , φ respectively.

Original: Function H is µ -strongly convex, and F is µ -strongly convex for θ and µ -strongly concave for ψ (see Appendix E.4 for deﬁnition of strongly convex condition).
Revised: Function H is µ -strongly convex, and F is µ -strongly convex for θ and µ -strongly concave for ψ (see Appendix E.4 for deﬁnition of strongly convex condition).

Original: Here we deﬁne h ( ω h ) =
Revised: Here we deﬁne h ( ω h ) =

Original: ∇ θ H + ∇ φ H , f ( ω f ) = ∇ θ F − ∇ ψ F , and then we have the following theorem.
Revised: ∇ θ H + ∇ φ H , f ( ω f ) = ∇ θ F − ∇ ψ F , and then we have the following theorem.

Original: Theorem 3.
Revised: Theorem 3.

Original: If F is µ -strongly convex-concave and H is µ -strongly convex, we can leverage the alternate SGD algorithm, i.e.
Revised: If F is µ -strongly convex-concave and H is µ -strongly convex, we can leverage the alternate SGD algorithm, i.e.

Original: [Equation]
Revised: [Equation]

Original: (11) where ω t +1 / 2 h=
Revised: where ω t +1 / 2 h

Original:  [ θ t , ψ t ] , ω 
Revised: = [ θ t , ψ t ] , ω t +1 h

Original: [Equation]
Revised: = [ θ t +1 / 2 , ψ t +1 ] , ω t +1 / 2 f = [ θ t +1 / 2 , φ t ] , ω t +1 f

Original: = [Equation] and P Ω ( ω ) =
Revised: = [ θ t +1 , φ t +1 ] , and P Ω ( ω ) =

Original: arg min ω (cid:48)
Revised: arg min ω (cid:48)

Original: ∈ Ω (cid:107) ω
Revised: ∈ Ω (cid:107) ω

Original: − ω (cid:48) (cid:107) 22 denotes the projection mapping to Ω .
Revised: − ω (cid:48) (cid:107) 22 denotes the projection mapping to Ω .

Original: Then we can achieve the convergence by using 12 µ < η < 1 µ .
Revised: Then we can achieve the convergence by using 12 µ < η < 1 µ .

Original: Theorem 3 shows that Stein Bridging could converge to at least a local optimum.
Revised: Theorem 3 shows that Stein Bridging could converge to at least a local optimum.

Original: Due to the unknown and intricate landscape of deep neural networks, the global optimization and convergence analysis for GAN has remained as an unexplored problem.
Revised: Due to the unknown and intricate landscape of deep neural networks, the global optimization and convergence analysis for GAN has remained as an unexplored problem.

Original: Despite the fact that strong convexity assumption cannot be guaranteed with deep neural networks, the optimization could converge to a stable point once there exists a local region that satisﬁes the strongly convex conditions.
Revised: Despite the fact that strong convexity assumption cannot be guaranteed with deep neural networks, the optimization could converge to a stable point once there exists a local region that satisﬁes the strongly convex conditions.

Original: In the experiments, we will empirically compare the training stability of each method on various datasets to validate our theoretical discussions.
Revised: In the experiments, we will empirically compare the training stability of each method on various datasets to validate our theoretical discussions.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we conduct experiments to verify the effectiveness of proposed method from multifaceted views.
Revised: In this section, we conduct experiments to verify the effectiveness of proposed method from multifaceted views.

Original: First, we select three tasks with different evaluation metrics in Section 5.1, 5.2 and 5.3.
Revised: First, we select three tasks with different evaluation metrics in Section 5.1, 5.2 and 5.3.

Original: Then we further discuss some applications of joint training as well as some useful tricks in
Revised: Then we further discuss some applications of joint training as well as some useful tricks in

Original: Section 5.4, 5.5 and 5.6.
Revised: Section 5.4, 5.5 and 5.6.

Original: The codes will be released later.
Revised: The codes will be released later.

Original: We consider two synthetic datasets with mixtures of Gaussian distributions: Two-Circle and TwoSpiral.
Revised: We consider two synthetic datasets with mixtures of Gaussian distributions: Two-Circle and TwoSpiral.

Original: The ﬁrst one is composed of 24 Gaussian mixtures that lie in two circles.
Revised: The ﬁrst one is composed of 24 Gaussian mixtures that lie in two circles.

Original: Such dataset is extended from the 8-Gaussian-mixture scenario which is widely used in previous GAN papers and is more difﬁcult, so that we can use it to test the quality of generated samples and mode coverage of learned density.
Revised: Such dataset is extended from the 8-Gaussian-mixture scenario which is widely used in previous GAN papers and is more difﬁcult, so that we can use it to test the quality of generated samples and mode coverage of learned energy.

Original: The second synthetic dataset consists of 100 Gaussian mixtures whose centers are densely arranged on two centrally symmetrical spiral-shaped curves.
Revised: The second synthetic dataset consists of 100 Gaussian mixtures whose centers are densely arranged on two centrally symmetrical spiral-shaped curves.

Original: This dataset can be used to examine the power of generative model on complicated data distributions.
Revised: This dataset can be used to examine the power of generative model on complicated data distributions.

Original: The ground-truth distributions and samples are shown in Fig.
Revised: The ground-truth distributions and samples are shown in Fig.

Original: 3 (a) and Fig.
Revised: 3 (a) and Fig.

Original: 4 (a).
Revised: 4 (a).

Original: Furthermore, we also apply the methods to MNIST and CIFAR datasets which require the model to deal with high-dimensional data.
Revised: Furthermore, we also apply the methods to MNIST and CIFAR datasets which require the model to deal with high-dimensional data.

Original: In each dataset, we use observed samples as input of the model and leverage them to train the generators and the estimators.
Revised: In each dataset, we use observed samples as input of the model and leverage them to train the generators and the estimators.

Original: The details for each dataset are reported in Appendix E.1.
Revised: The details for each dataset are reported in Appendix E.1.

Original: In our experiments, we also replace the Wasserstein metric in (3) by JS divergence.
Revised: In our experiments, we also replace the Wasserstein metric in (3) by JS divergence.

Original: To well distinguish different speciﬁcations, we term the model Joint-W if using Wasserstein metric and Joint-JS if using JS divergence in this section.
Revised: To well distinguish different speciﬁcations, we term the model Joint-W if using Wasserstein metric and Joint-JS if using JS divergence in this section.

Original: We consider several competitors.
Revised: We consider several competitors.

Original: First, for implicit generative models, we consider valina GAN, WGAN-GP (Gulrajani et al.
Revised: First, for implicit generative models, we consider valina GAN, WGAN-GP (Gulrajani et al.

Original: (2017)), likelihood-regularized
Revised: (2017)), likelihood-regularized

Original: GAN/WGAN-GP (short as GAN+LR/WGAN+LR), entropy-regularized GAN/WGAN-GP (short as GAN+ER/WGAN+ER) and a recently proposed variational annealing regularization (Tao et al.
Revised: GAN/WGAN-GP (short as GAN+LR/WGAN+LR), entropy-regularized GAN/WGAN-GP (short as GAN+ER/WGAN+ER) and a recently proposed variational annealing regularization (Tao et al.

Original: for GAN (short as GAN+VA/WGAN+VA) to compare the quality of generated samples.
Revised: for GAN (short as GAN+VA/WGAN+VA) to compare the quality of generated samples.

Original: We employ the denoising auto-encoder to estimate the gradient for regularization penalty, which is proposed by Alain & Bengio (2014) and utilized by Tao et al.
Revised: We employ the denoising auto-encoder to estimate the gradient for regularization penalty, which is proposed by Alain & Bengio (2014) and utilized by Tao et al.

Original: Second, for explicitdensity models, we consider Deep Energy Model (DEM) which is optimized based on Stein discrepancy, and energy-based GAN (EGAN) (Dai et al.
Revised: Second, for explicit models, we consider Deep Energy Model (DEM) which is optimized based on Stein discrepancy, and energybased GAN (EGAN) (Dai et al.

Original: Besides, we also compare with Deep Directed Generative (DGM) Model (Kim & Bengio (2017)) which adopts contrastive divergence to unite sample generator and density estimator.
Revised: Besides, we also compare with Deep Directed Generative (DGM) Model (Kim & Bengio (2017)) which adopts contrastive divergence to unite sample generator and energy estimator.

Original: See Appendix A for brief introduction of these methods and Appendix E.3 for implementation details for each method.
Revised: See Appendix A for brief introduction of these methods and Appendix E.3 for implementation details for each method.

Original: 5. S AMPLE Q UALITY OF I MPLICIT M ODEL
Revised: 5. S AMPLE Q UALITY OF I MPLICIT M ODEL

Original: Calibrating explicit  density model with implicit generator is expected to improve the quality of generated samples.
Revised: Calibrating explicit (unnormalized) density model with implicit generator is expected to improve the quality of generated samples.

Original: In Fig.
Revised: In Fig.

Original: 3 and Fig.
Revised: 3 and Fig.

Original: 4 we show the results of different generators inTwo-Circle and
Revised: 4 we show the results of different generators in

Original:  Two-Spiral datasets.
Revised: Two-Circle and Two-Spiral datasets.

Original: As we can see, in Two-Circle, there are a large number of generated samples
Revised: As we can see, in Two-Circle, there are a large number of gen-

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original:  given by GAN, WGAN-GP and DGM (the worst one in this case) locating between two Gaussian components, and the boundary for each component is not distinguishable.
Revised: erated samples given by GAN, WGAN-GP and DGM (the worst one in this case) locating between two Gaussian components, and the boundary for each component is not distinguishable.

Original: Since the ground-truth densities of regions between two components are very low, such generated samples possess lowquality, which depicts that these models capture the combinations of two dominated features (i
Revised: Since the ground-truth densities of regions between two components are very low, such generated samples possess low-quality, which depicts that these models capture the combinations of two dominated features (i

Original: [Figure]
Revised: [Figure]

Original: e., modes) in the data but such combination does not make sense in practice.
Revised: e., modes) in the data but such combination does not make sense in practice.

Original: By contrast, Joint-JS and Joint-W could alleviate such issue, reduce the low-quality samples and produce more distinguishable boundaries for components.
Revised: By contrast, Joint-JS and Joint-W could alleviate such issue, reduce the low-quality samples and produce more distinguishable boundaries for components.

Original: In Two-Spiral, similarly, the generated samples given by GAN and WGAN-GP form a circle instead of two spirals while the samples of DGM ‘link’ two spirals.
Revised: In Two-Spiral, similarly, the generated samples given by GAN and WGAN-GP form a circle instead of two spirals while the samples of DGM ‘link’ two spirals.

Original: Joint-JS manages to focus more on true high densities compared to GAN and Joint-W provides the best results.
Revised: Joint-JS manages to focus more on true high densities compared to GAN and Joint-W provides the best results.

Original: To quantitatively measure the sample quality, we adopt two metrics: Maximum Mean Discrepancy (MMD) and High-quality Sample Rate (HSR).
Revised: To quantitatively measure the sample quality, we adopt two metrics: Maximum Mean Discrepancy (MMD) and High-quality Sample Rate (HSR).

Original: The detailed deﬁnitions are given in Appendix [-E.
-] and we report the results in Table 5.
Revised: The detailed deﬁnitions are given in Appendix [-E.
-]E.4 and we report the results in Table 5.

Original: We visualize the generated digits/images on MNIST/CIFAR-10 datasets in Fig.
Revised: We visualize the generated digits/images on MNIST/CIFAR-10 datasets in Fig.

Original: 9 and Fig.
Revised: 9 and Fig.

Original: 10 and use Inception Score and conditional entropy of predicted classes (CEPC) to measure the sample quality (See Appendix E.4 for details).
Revised: 10 and use Inception Score and conditional entropy of predicted classes (CEPC) to measure the sample quality (See Appendix E.4 for details).

Original: As shown in Table 2, Joint-W (resp.
Revised: As shown in Table 2, Joint-W (resp.

Original: Joint-JS) is superior than WGAN-GP (resp.
Revised: Joint-JS) is superior than WGAN-GP (resp.

Original: DCGAN), regularized WGAN (resp. DCGAN) and DGM.
Revised: DCGAN), regularized WGAN (resp. DCGAN) and DGM.

Original: The CEPC characterizes how well the picture can be distinguished by a pre-trained classiﬁer, i.e., the quality of picture, so the results depict that proposed method could give higher-quality generated pictures.
Revised: The CEPC characterizes how well the picture can be distinguished by a pre-trained classiﬁer, i.e., the quality of picture, so the results depict that proposed method could give higher-quality generated pictures.

Original: 5. D ENSITY E STIMATION OF E XPLICIT M ODEL
Revised: 5. D ENSITY E STIMATION OF E XPLICIT M ODEL

Original: Another advantage of joint learning is that the generator could help the density estimator to capture more accurate distribution.
Revised: Another advantage of joint learning is that the generator could help the density estimator to capture more accurate distribution.

Original: As shown in Fig 3, both Joint-JS and Joint-W manage to capture all
Revised: As shown in Fig 3, both Joint-JS and Joint-W manage to capture all

Original: Gaussian components while other methods miss some of modes.
Revised: Gaussian components while other methods miss some of modes.

Original: In Fig 4, Joint-JS and Joint-W exactly ﬁt the ground-truth distribution.
Revised: In Fig 4, Joint-JS and Joint-W exactly ﬁt the ground-truth distribution.

Original: By contrast, DEM misses one spiral while EGAN degrades to a uniform-like distribution.
Revised: By contrast, DEM misses one spiral while EGAN degrades to a uniform-like distribution.

Original: DGM manages to ﬁt two spirals but allocate high densities to regions that have low densities in the groung-truth distribution.
Revised: DGM manages to ﬁt two spirals but allocate high densities to regions that have low densities in the groung-truth distribution.

Original: To quantitatively measure the performance, we introduce three evaluation metrics: KL & JS divergence between the ground-truth and estimated densities and Area Under the Curve (AUC) for false-positive rate v.s. true-positive rate where we select points with true high (resp. low) densities as positive (resp.
Revised: To quantitatively measure the performance, we introduce three evaluation metrics: KL & JS divergence between the ground-truth and estimated densities and Area Under the Curve (AUC) for false-positive rate v.s. true-positive rate where we select points with true high (resp. low) densities as positive (resp.

Original: negative) examples.
Revised: negative) examples.

Original: The detailed information and results are given in Appendix E.4 and Table 5 respectively.
Revised: The detailed information and results are given in Appendix E.4 and Table 5 respectively.

Original: The values show that
Revised: The values show that

Original: Joint-W and Joint-JS could provide more accurate  density estimation than other competitors.
Revised: Joint-W and Joint-JS could provide more accurate (unnormalized) density estimation than other competitors.

Original: We also rank the generated digits (and true digits) on MNIST w.r.t the densities given by the energy model in Fig.
Revised: We also rank the generated digits (and true digits) on MNIST w.r.t the densities given by the energy model in Fig.

Original: 11, Fig.
Revised: 11, Fig.

Original: 12 and Fig. 13.
Revised: 12 and Fig. 13.

Original: As depicted in the ﬁgures, the digits with high densities (orlow densities) given by Joint-JS possess enough diversity (the thickness, the inclination angles as
Revised: As depicted in the ﬁgures, the digits with high densities (or

Original: [Figure]
Revised: [Figure]

Original:  well as the shapes of digits diverses).By constrast [Figure]
Revised: low densities) given by Joint-JS possess enough diversity (the thickness, the inclination angles as well as the shapes of digits diverses).

Original:  all the digits with high densities given by DGM tend to be thin and digits with low densities are very thick.
Revised: By constrast, all the digits with high densities given by DGM tend to be thin and digits with low densities are very thick.

Original: Also, as for EGAN, digits with high (or low) densities appear to have the same inclination angle (for high densities, ‘1’ keeps straight and ‘9’ ’leans’ to the left while for low densities, just the opposite).
Revised: Also, as for EGAN, digits with high (or low) densities appear to have the same inclination angle (for high densities, ‘1’ keeps straight and ‘9’ ’leans’ to the left while for low densities, just the opposite).

Original: Such phenomenon indicates that DGM and EGAN tend to allocate high (or low) densities to data with certain modes and would miss some modes that possibly possess high densities in ground-truth distributions.
Revised: Such phenomenon indicates that DGM and EGAN tend to allocate high (or low) densities to data with certain modes and would miss some modes that possibly possess high densities in ground-truth distributions.

Original: Fortunately, our method overcomes the issue and manages to capture complicated distributions.
Revised: Fortunately, our method overcomes the issue and manages to capture complicated distributions.

Original: 5. E NHANCING THE S TABILITY OF GAN
Revised: 5. E NHANCING THE S TABILITY OF GAN

Original: Our discussions and analysis show that joint training helps to stabilize GAN training.
Revised: Our discussions and analysis show that joint training helps to stabilize GAN training.

Original: In Fig.we present the learning curves of Joint-W (resp.
Revised: In Fig.we present the learning curves of Joint-W (resp.

Original: Joint-JS)
Revised: Joint-JS)

Original: compared with WGAN (resp.
Revised: compared with WGAN (resp.

Original: GAN or
Revised: GAN or

Original: DCGAN) and its regularization-based variants on different datasets.
Revised: DCGAN) and its regularization-based variants on different datasets.

Original: One can clearly see from the curves that joint training could reduce the variance of metric values especially during the second half of training.
Revised: One can clearly see from the curves that joint training could reduce the variance of metric values especially during the second half of training.

Original: Furthermore, we visualize the generated pictures given by the same noise z in adjacent epoches in Fig.
Revised: Furthermore, we visualize the generated pictures given by the same noise z in adjacent epoches in Fig.

Original: The results show that Joint-W outputs more stable generation in adjacent epoches while the generated samples given by WGAN-GP and WGAN+VA exhibit an obvious variation.
Revised: The results show that Joint-W outputs more stable generation in adjacent epoches while the generated samples given by WGAN-GP and WGAN+VA exhibit an obvious variation.

Original: Especially, some digits generated by WGAN-GP and WGAN+VA change from one class to another.
Revised: Especially, some digits generated by WGAN-GP and WGAN+VA change from one class to another.

Original: Such phenomenon is quite similar to the oscillatory behavior with non-convergence in optimization that we discuss in Section 4.1.
Revised: Such phenomenon is quite similar to the oscillatory behavior with non-convergence in optimization that we discuss in Section 4.1.

Original: Another issue discussed in Section 4.1 is the bias of model distribution for regularized GAN methods.
Revised: Another issue discussed in Section 4.1 is the bias of model distribution for regularized GAN methods.

Original: To quantify this evaluation, we calculate l 1 and l 2 distances between the means ofgenerated digits (resp. images) and 50000 true digits (resp. images) in MNIST (reps. CIFAR-10).
Revised: To quantify this evaluation, we calculate l 1 and l 2 distances between the means ofgenerated digits (resp. images) and 50000 true digits (resp. images) in MNIST (reps. CIFAR-10).

Original: The results are shown in Table 3.
Revised: The results are shown in Table 3.

Original: The smaller distances given by Joint-W indicate that it converges to a better local optimum with smaller bias from the original data distribution.
Revised: The smaller distances given by Joint-W indicate that it converges to a better local optimum with smaller bias from the original data distribution.

Original: Also, in Table 6 (resp.
Revised: Also, in Table 6 (resp.

Original: Table 7), we report the distances for digits (resp. images) in each class on MNIST (resp.
Revised: Table 7), we report the distances for digits (resp. images) in each class on MNIST (resp.

Original: CIFAR).
Revised: CIFAR).

Original: 5. D ETECTING O UT - OF -D ISTRIBUTION S AMPLES
Revised: 5. D ETECTING O UT - OF -D ISTRIBUTION S AMPLES

Original: The explicit model estimates densities for each sample and one of its applications is to detect outliers in the input data.
Revised: The explicit model estimates densities for each sample and one of its applications is to detect outliers in the input data.

Original: Here, we adopt CIFAR-10 to measure the ability of our estimator to distinguish the in-distribution samples and (true/false) out-of-distribution samples.
Revised: Here, we adopt CIFAR-10 to measure the ability of our estimator to distinguish the in-distribution samples and (true/false) out-of-distribution samples.

Original: We consider four situations and in each case, we consider the test images of CIFAR-10 as positive set (expected to allocate high densities) and construct a negative set (expected to allocate low densities).
Revised: We consider four situations and in each case, we consider the test images of CIFAR-10 as positive set (expected to allocate high densities) and construct a negative set (expected to allocate low densities).

Original: We let the model output densities for images in two sets, rank them according to the densities and plot the ROC curve for false-positive rate v.s. true-positive rate in Fig.
Revised: We let the model output densities for images in two sets, rank them according to the densities and plot the ROC curve for false-positive rate v.s. true-positive rate in Fig.

Original: In the ﬁrst case, we ﬂip each image in the positive set as negative set.
Revised: In the ﬁrst case, we ﬂip each image in the positive set as negative set.

Original: Note that such ﬂipped images are not out-of-distribution samples, so the model is expected to allocate high densities to them, i.e., the ROC curve should be close to a straight line from (0 , 0) to (1 , 1) .
Revised: Note that such ﬂipped images are not out-of-distribution samples, so the model is expected to allocate high densities to them, i.e., the ROC curve should be close to a straight line from (0 , 0) to (1 , 1) .

Original: The results show that Joint-W, EGAN and DEM give the exact results while DGM assigns all ﬂipped images with lower densities, which means that it fails to capture the semantics  in images.
Revised: The results show that Joint-W, EGAN and DEM give the exact results while DGM assigns all ﬂipped images with lower densities, which means that it fails to capture the semantics [Figure] [Figure] in images.

Original: In the following three cases, we i) generate random noise, ii) average two images with [Figure]
Revised: In the following three cases, we i) generate random noise, ii) average two images with different CIFAR classes

Original: [Figure]
Revised: [Figure]

Original: different CIFAR classes, and iii) adopt Lsun Bedroom dataset as the negative set, respectively.
Revised: and iii) adopt Lsun Bedroom dataset as the negative set, respectively.

Original: In these situations, the model is expected to distinguish the images in two sets.
Revised: In these situations, the model is expected to distinguish the images in two sets.

Original: The results in Fig.show that DGM provides the best results while the performance of Joint-W is quite close to DGM and much better than DEM and EGAN.
Revised: The results in Fig.show that DGM provides the best results while the performance of Joint-W is quite close to DGM and much better than DEM and EGAN.

Original: 5. A DDRESSING D ATA I NSUFFICIENCY AND N OISY D ATA
Revised: 5. A DDRESSING D ATA I NSUFFICIENCY AND N OISY D ATA

Original: We proceed to test the model performance in some extreme situations where the observed samples are mixed with noises or the observed samples are quite insufﬁcient.
Revised: We proceed to test the model performance in some extreme situations where the observed samples are mixed with noises or the observed samples are quite insufﬁcient.

Original: The results are presented in
Revised: The results are presented in

Original: Fig.
Revised: Fig.

Original: 7(a) where we add different ratios of random noise to the true samples in Two-Circle dataset and Fig.
Revised: 7(a) where we add different ratios of random noise to the true samples in Two-Circle dataset and Fig.

Original: 7(b) where we only sample insufﬁcient data for training in Two-Spiral dataset.
Revised: 7(b) where we only sample insufﬁcient data for training in Two-Spiral dataset.

Original: The details are in Appendix E.1.
Revised: The details are in Appendix E.1.

Original: The noise in data impacts the performance of WGAN and Joint-W, but comparatively, the performance decline for Joint-W is less insigniﬁcant than WGAN, which indicates better robustness of joint training w.r.t noised data.
Revised: The noise in data impacts the performance of WGAN and Joint-W, but comparatively, the performance decline for Joint-W is less insigniﬁcant than WGAN, which indicates better robustness of joint training w.r.t noised data.

Original: In Fig. 7(b), when the sample size decreases from 2000 to 100, the AUC value of DEM declines dramatically, showing its dependency on sufﬁcient training samples.
Revised: In Fig. 7(b), when the sample size decreases from 2000 to 100, the AUC value of DEM declines dramatically, showing its dependency on sufﬁcient training samples.

Original: By contrast, the AUC of Joint-W exhibits a small decline when the sample size is more than 500 and suffers from an obvious decline when it is less than 300.
Revised: By contrast, the AUC of Joint-W exhibits a small decline when the sample size is more than 500 and suffers from an obvious decline when it is less than 300.

Original: Such phenomenon demonstrates lower sensitivity of joint training to observed sample size.
Revised: Such phenomenon demonstrates lower sensitivity of joint training to observed sample size.

Original: 5. W HEN TO S TART J OINT L EARNING
Revised: 5. W HEN TO S TART J OINT L EARNING

Original: In our experiment, we also observe an interesting phenomenon: the performance achieved at convergence would be better if we start joint training after some iterations with independent training for the generator and the estimator.
Revised: In our experiment, we also observe an interesting phenomenon: the performance achieved at convergence would be better if we start joint training after some iterations with independent training for the generator and the estimator.

Original: In other words, at the beginning, we could set λ 2 = 0 (or some very small values) in (3) and after some iterations set it as a normal level.
Revised: In other words, at the beginning, we could set λ 2 = 0 (or some very small values) in (3) and after some iterations set it as a normal level.

Original: We report the inception scores on MNIST with different numbers of iterations for independent training in Fig.
Revised: We report the inception scores on MNIST with different numbers of iterations for independent training in Fig.

Original: 7(c) where we can see that the score ﬁrstly goes up and then goes down when we increase iterations for independent training.
Revised: 7(c) where we can see that the score ﬁrstly goes up and then goes down when we increase iterations for independent training.

Original: Such phenomenon is quite similar to the ‘warm up’ trick used for training deep networks where one can use small learning rates at iterations in the begining and amplify its value for further training.
Revised: Such phenomenon is quite similar to the ‘warm up’ trick used for training deep networks where one can use small learning rates at iterations in the begining and amplify its value for further training.

Original: One intuitive reason behind this phenomenon is that at the beginning, both the generator and estimator are weak and if we minimize the discrepancy between them at this point, they would possibly constrain each other and get limited in some bad local optima.
Revised: One intuitive reason behind this phenomenon is that at the beginning, both the generator and estimator are weak and if we minimize the discrepancy between them at this point, they would possibly constrain each other and get limited in some bad local optima.

Original: When they become strong enough after some training iterations, uniting them through joint training would help them compensate and reinforce each other as our discussions.
Revised: When they become strong enough after some training iterations, uniting them through joint training would help them compensate and reinforce each other as our discussions.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper, we aim at uniting the training for implicit generative model (represented by GAN) and explicit generative model (represented by a deep energy-based model).
Revised: In this paper, we aim at uniting the training for implicit generative model (represented by GAN) and explicit generative model (represented by a deep energy-based model).

Original: Besides two loss terms for
Revised: Besides two loss terms for

Original: GAN and energy-based model, we introduce the third loss characterized via Stein discrepancy between the generator in GAN and the energy-based model.
Revised: GAN and energy-based model, we introduce the third loss characterized via Stein discrepancy between the generator in GAN and the energy-based model.

Original: Theoretically, we show that joint training could i) help to stablize GAN training and facilitate its convergence, and ii) enforcing dual regularization effects on both models and help to escape from local optima in optimization.
Revised: Theoretically, we show that joint training could i) help to stablize GAN training and facilitate its convergence, and ii) enforcing dual regularization effects on both models and help to escape from local optima in optimization.

Original: We also conduct extensive experiments with different tasks and application senarios to verify our theoretical ﬁndings as well as demonstrate the superiority of our method compared with various GAN models and deep energy-based models.
Revised: We also conduct extensive experiments with different tasks and application senarios to verify our theoretical ﬁndings as well as demonstrate the superiority of our method compared with various GAN models and deep energy-based models.

Original: A L ITERATURE R EVIEWS
Revised: A L ITERATURE R EVIEWS

Original: We discuss some of related literatures and shed lights on the relationship between our work with others.
Revised: We discuss some of related literatures and shed lights on the relationship between our work with others.

Original: A. E XPLICIT G ENERATIVE M ODELS
Revised: A. E XPLICIT G ENERATIVE M ODELS

Original: Explicit generative models are interested in ﬁtting each instance with a scaler  density expected to explicitly capture the distribution behind data.
Revised: Explicit generative models are interested in ﬁtting each instance with a scaler (unnormalized) density expected to explicitly capture the distribution behind data.

Original: Such densities are often up to a constant and called as energy functions which are common in undirected graphical models (LeCun et al.
Revised: Such densities are often up to a constant and called as energy functions which are common in undirected graphical models (LeCun et al.

Original: Hence, explicit generative models are also termed as energy-based models.
Revised: Hence, explicit generative models are also termed as energy-based models.

Original: An early version of energybased models is the FRAME (Filters, Random ﬁeld, And Maximum Entropy) model (Zhu et al.
Revised: An early version of energy-based models is the FRAME (Filters, Random ﬁeld, And Maximum Entropy) model (Zhu et al.

Original: ; Wu et al.
Revised: ; Wu et al.

Original: Later on, some works leverage deep neural networks to model the energy function (Ngiam et al.
Revised: Later on, some works leverage deep neural networks to model the energy function (Ngiam et al.

Original: Xie et al.
Revised: Xie et al.

Original: (2016b))
Revised: (2016b))

Original: and pave the way for researches on deep energy model (DEM)
Revised: and pave the way for researches on deep energy model (DEM)

Original: (e.g., Liu & Wang (2017); Kim & Bengio (2017); Zhai et al.
Revised: (e.g., Liu & Wang (2017); Kim & Bengio (2017); Zhai et al.

Original: Haarnoja et al.
Revised: Haarnoja et al.

Original: (2017); Du & Mordatch (2019); Nijkamp et al.
Revised: Du & Mordatch (2019); Nijkamp et al.

Original: Apart from DEM, there are also some other forms of deep explicit models based on restricted Boltzmann machines like deep belief networks (Hinton et al.
Revised: Apart from DEM, there are also some other forms of deep explicit models based on restricted Boltzmann machines like deep belief networks (Hinton et al.

Original: and deep Boltzmann machines (Salakhutdinov & Hinton (2009)).
Revised: and deep Boltzmann machines (Salakhutdinov & Hinton (2009)).

Original: The normalized constant under the energy function requires an intractable integral over all possible instances, which makes the model hard to learn via Maximum Likelihood Estimation (MLE).
Revised: The normalized constant under the energy function requires an intractable integral over all possible instances, which makes the model hard to learn via Maximum Likelihood Estimation (MLE).

Original: To solve this issue, some works propose to approximate the constant by MCMC methods (Geman &
Revised: To solve this issue, some works propose to approximate the constant by MCMC methods (Geman &

Original: Geman (1984); Neal (2011)).
Revised: Geman (1984); Neal (2011)).

Original: However, MCMC requires an inner-loop samples in each training, which induces high computational costs.
Revised: However, MCMC requires an inner-loop samples in each training, which induces high computational costs.

Original: Another solution is to optimize an alternate surrogate loss function.
Revised: Another solution is to optimize an alternate surrogate loss function.

Original: For example, contrastive divergence (CD) (Liu & Wang (2017)) is proposed to measure how much KL divergence can be improved by running a small numbers of Markov chain steps towards the intractable likelihood, while score matching (SM) (Hyv ¨ arinen (2005)) detours the constant by minimizing the distance for gradients of log-likelihoods.
Revised: For example, contrastive divergence (CD) (Liu & Wang (2017)) is proposed to measure how much KL divergence can be improved by running a small numbers of Markov chain steps towards the intractable likelihood, while score matching (SM) (Hyv ¨ arinen (2005)) detours the constant by minimizing the distance for gradients of log-likelihoods.

Original: Moreover, the intractable normalized constant makes it hard to sample from.
Revised: Moreover, the intractable normalized constant makes it hard to sample from.

Original: To obtain an accurate samples from unnormalized densities, many studies propose to approximate the generation by diffusion-based processes, like generative ﬂow (Nguyen et al.
Revised: To obtain an accurate samples from unnormalized densities, many studies propose to approximate the generation by diffusion-based processes, like generative ﬂow (Nguyen et al.

Original: and variational gradient descent (Liu & Wang (2016)).
Revised: and variational gradient descent (Liu & Wang (2016)).

Original: Also, a recent work (Hu et al.
Revised: Also, a recent work (Hu et al.

Original: leverages Stein discrepancy to design a neural sampler from unnormalized densities.
Revised: leverages Stein discrepancy to design a neural sampler from unnormalized densities.

Original: The fundamental disadvantage of explicit model is that the energy-based learning is difﬁcult to accurately capture the distribution of true samples due to the low manifold of real-world instances (Liu & Wang (2017)).
Revised: The fundamental disadvantage of explicit model is that the energy-based learning is difﬁcult to accurately capture the distribution of true samples due to the low manifold of real-world instances (Liu & Wang (2017)).

Original: A. I MPLICIT G ENERATIVE M ODELS
Revised: A. I MPLICIT G ENERATIVE M ODELS

Original: Implicit generative models focus on a generation mapping from random noises to generated samples.
Revised: Implicit generative models focus on a generation mapping from random noises to generated samples.

Original: Such mapping function is often called as generator and possesses better ﬂexibility compared with explicit models.
Revised: Such mapping function is often called as generator and possesses better ﬂexibility compared with explicit models.

Original: Two typical implicit models are Variational Auto-Encoder (VAE) (Kingma & Welling (2014)) and Generative Adversarial Networks (GAN) (Goodfellow et al.
Revised: Two typical implicit models are Variational Auto-Encoder (VAE) (Kingma & Welling (2014)) and Generative Adversarial Networks (GAN) (Goodfellow et al.

Original: VAE introduces a latent variable and attempts to maximize the variational lower bound for likelihood of joint distribution of latent variable and observable variable, while GAN targets an adversarial game between the generator and a discriminator (or critic in WGAN) that aims at discriminating the generated and true samples.
Revised: VAE introduces a latent variable and attempts to maximize the variational lower bound for likelihood of joint distribution of latent variable and observable variable, while GAN targets an adversarial game between the generator and a discriminator (or critic in WGAN) that aims at discriminating the generated and true samples.

Original: In this paper, we focus on GAN and its variants (e.g., WGAN (Arjovsky et al.
Revised: In this paper, we focus on GAN and its variants (e.g., WGAN (Arjovsky et al.

Original: WGAN-GP (Gulrajani et al.
Revised: WGAN-GP (Gulrajani et al.

Original: , DCGAN (Radford et al.
Revised: , DCGAN (Radford et al.

Original: (2016)), etc.)
Revised: (2016)), etc.)

Original: as the implicit generative model
Revised: as the implicit generative model

Original: and we leave the discussions on VAE as future work.
Revised: and we leave the discussions on VAE as future work.

Original: Two important issues concerning GAN and its variants are instability of training and local optima.
Revised: Two important issues concerning GAN and its variants are instability of training and local optima.

Original: The typical local optima for GAN can be divided into two categories: mode-collapse (the model fails to capture all the modes in data) and mode-redundance (the model generates modes that do not exist in data).
Revised: The typical local optima for GAN can be divided into two categories: mode-collapse (the model fails to capture all the modes in data) and mode-redundance (the model generates modes that do not exist in data).

Original: Recently there are many attempts to solve these issues from various perspectives.
Revised: Recently there are many attempts to solve these issues from various perspectives.

Original: One perspective is from regularization.
Revised: One perspective is from regularization.

Original: Two typical regularization methods are likelihood-based and entropy-based regularization with the prominent examples Warde-Farley & Bengio (2017) and Li &
Revised: Two typical regularization methods are likelihood-based and entropy-based regularization with the prominent examples Warde-Farley & Bengio (2017) and Li &

Original: Turner (2018) that respectively leverage denoising feature matching and implicit gradient approximation to enforce the regularization constraints.
Revised: Turner (2018) that respectively leverage denoising feature matching and implicit gradient approximation to enforce the regularization constraints.

Original: The likelihood and entropy regularizations could respectively help the generator to focus on data distribution and encourage more diverse samples, and a recent work (Tao et al.
Revised: The likelihood and entropy regularizations could respectively help the generator to focus on data distribution and encourage more diverse samples, and a recent work (Tao et al.

Original: (2019)) uses Langevin dynamics to indicate that i) the entropy and likelihood regularizations are equivalent and share an opposite relationship in mathematics, and ii) both regularizations would make the model converge to a surrogate point with a bias from original data distribution.
Revised: (2019)) uses Langevin dynamics to indicate that i) the entropy and likelihood regularizations are equivalent and share an opposite relationship in mathematics, and ii) both regularizations would make the model converge to a surrogate point with a bias from original data distribution.

Original: Then Tao et al.
Revised: Then Tao et al.

Original: (2019) proposes a variational annealing strategy to empirically unite two regularizations and tackle the biased distributions.
Revised: (2019) proposes a variational annealing strategy to empirically unite two regularizations and tackle the biased distributions.

Original: To deal with the instability issue, there are also some recent literatures from optimization perspectives and proposes different algorithms to address the non-convergence of minimax game optimization (for instance, Gemp & Mahadevan (2018); Liang & Stokes (2019); Gidel et al.
Revised: To deal with the instability issue, there are also some recent literatures from optimization perspectives and proposes different algorithms to address the non-convergence of minimax game optimization (for instance, Gemp & Mahadevan (2018); Liang & Stokes (2019); Gidel et al.

Original: Moreover, the disadvantage of implicit models is the lack of explicit densities over instances, which disables the black-box generator to characterize the distributions behind data.
Revised: Moreover, the disadvantage of implicit models is the lack of explicit densities over instances, which disables the black-box generator to characterize the distributions behind data.

Original: A. A TTEMPTS TO C OMBINE B OTH OF THE W ORLDS
Revised: A. A TTEMPTS TO C OMBINE B OTH OF THE W ORLDS

Original: Recently, there are several studies that attempt to combine explicit and implicit generative models from different ways.
Revised: Recently, there are several studies that attempt to combine explicit and implicit generative models from different ways.

Original: For instance, Zhao et al.
Revised: For instance, Zhao et al.

Original: (2017) proposes energy-based GAN that leverages energy model as discriminator to distinguish the generated and true samples.
Revised: (2017) proposes energy-based GAN that leverages energy model as discriminator to distinguish the generated and true samples.

Original: The similar idea is also used by Kim & Bengio (2017) and Dai et al.
Revised: The similar idea is also used by Kim & Bengio (2017) and Dai et al.

Original: (2017) which let the discriminator estimate a scaler energy value for each sample.
Revised: (2017) which let the discriminator estimate a scaler energy value for each sample.

Original: Such discriminator is optimized to give high energy to generated samples and low energy to true samples while the generator aims at generating samples with low energy.
Revised: Such discriminator is optimized to give high energy to generated samples and low energy to true samples while the generator aims at generating samples with low energy.

Original: The fundamental difference is that Zhao et al.
Revised: The fundamental difference is that Zhao et al.

Original: (2017) and Dai et al.
Revised: (2017) and Dai et al.

Original: (2017) both aim at minimizing the discrepancy between distributions of generated and true samples while the motivation of Kim & Bengio (2017) is to minimize the KL divergence between estimated densities and true samples.
Revised: (2017) both aim at minimizing the discrepancy between distributions of generated and true samples while the motivation of Kim & Bengio (2017) is to minimize the KL divergence between estimated densities and true samples.

Original: Kim & Bengio (2017) adopts contrastive divergence (CD) to link MLE for energy model over true data with the adversarial training of energy-based GAN.
Revised: Kim & Bengio (2017) adopts contrastive divergence (CD) to link MLE for energy model over true data with the adversarial training of energy-based GAN.

Original: However, both CD-based method and energy-based GAN have limited power for both generator and discriminator.
Revised: However, both CD-based method and energy-based GAN have limited power for both generator and discriminator.

Original: Firstly, if the generated samples resemble true samples, then the gradients for discriminator given by true and generated samples are just the opposite and will counteract each other, and the training will stop before the discriminitor captures accurate data distribution.
Revised: Firstly, if the generated samples resemble true samples, then the gradients for discriminator given by true and generated samples are just the opposite and will counteract each other, and the training will stop before the discriminitor captures accurate data distribution.

Original: Second, since the objective boils down to minimizing the KL divergence (for Kim & Bengio (2017)) or Wasserstein distance (for Dai et al.
Revised: Second, since the objective boils down to minimizing the KL divergence (for Kim & Bengio (2017)) or Wasserstein distance (for Dai et al.

Original: (2017)) between model and true distributions, the issues concerning GAN (or WGAN) like training instability and mode-collapse would also bother these methods.
Revised: (2017)) between model and true distributions, the issues concerning GAN (or WGAN) like training instability and mode-collapse would also bother these methods.

Original: Another way for combination is by cooperative training.
Revised: Another way for combination is by cooperative training.

Original: Xie et al.
Revised: Xie et al.

Original: (2016a)
Revised: (2016a)

Original: (and its improved version
Revised: (and its improved version

Original: Xie et al.
Revised: Xie et al.

Original: (2018)) leverages the samples of generator as the MCMC initialization for energy-based model.
Revised: (2018)) leverages the samples of generator as the MCMC initialization for energy-based model.

Original: The synthesized samples produced from ﬁnite-step MCMC are closer to the energy model and the generator is optimized to make the ﬁnite-step MCMC revise its initial samples.
Revised: The synthesized samples produced from ﬁnite-step MCMC are closer to the energy model and the generator is optimized to make the ﬁnite-step MCMC revise its initial samples.

Original: Also, a recent work Du et al.
Revised: Also, a recent work Du et al.

Original: (2018) proposes to regard the explicit model as a teacher net who guides the training of implicit generator as a student net to produce samples that could overcome the mode-collapse issue.
Revised: (2018) proposes to regard the explicit model as a teacher net who guides the training of implicit generator as a student net to produce samples that could overcome the mode-collapse issue.

Original: The main drawback of cooperative training is that they indirectly optimize the discrepancy between the generator and data distribution via the energy model as a ‘mediator’, which leads to a fact that once the energy model gets stuck in a local optimum (e.g., mode-collapse or mode-redundance) the training for the generator would be affected.
Revised: The main drawback of cooperative training is that they indirectly optimize the discrepancy between the generator and data distribution via the energy model as a ‘mediator’, which leads to a fact that once the energy model gets stuck in a local optimum (e.g., mode-collapse or mode-redundance) the training for the generator would be affected.

Original: In other words, the training for two models would constrain rather than exactly compensate each other.
Revised: In other words, the training for two models would constrain rather than exactly compensate each other.

Original: In Table 1, we do a high-level comparison among the above-mentioned generative models w.r.t the objectives.
Revised: In Table 1, we do a high-level comparison among the above-mentioned generative models w.r.t the objectives.

Original: Different from other methods, our model considers three discrepancies simultaneously as a triangle to jointly train the generator and the estimator, enabling them to compensate and reinforce each other.
Revised: Different from other methods, our model considers three discrepancies simultaneously as a triangle to jointly train the generator and the estimator, enabling them to compensate and reinforce each other.

Original: B B ACKGROUND FOR S TEIN D ISCREPANCY
Revised: B B ACKGROUND FOR S TEIN D ISCREPANCY

Original: Assume q ( x ) to be a continuously differentiable density supported on X ⊂ R d and f ( x ) a smooth vector function.
Revised: Assume q ( x ) to be a continuously differentiable density supported on X ⊂ R d and f : R d → R d (cid:48) a smooth vector function.

Original: Deﬁne A q [ f ( x )]
Revised: Deﬁne A q [ f ( x )]

Original: = ∇ x log q ( x ) f ( x ) (cid:62) + ∇ x f ( x ) as a Stein operator.
Revised: = ∇ x log q ( x ) f ( x ) (cid:62) + ∇ x f ( x ) as a Stein operator.

Original: If f is a Stein class (satisfying some mild boundary conditions) then we have the following Stein identity property:
Revised: If f is a Stein class (satisfying some mild boundary conditions) then we have the following Stein identity property:

Original: [Equation]
Revised: [Equation]

Original: Such property induces the Stein discrepancy between distributions P : p ( x ) and Q : q ( x ) , x ∈ X : 
Revised: Such property induces the Stein discrepancy between distributions P : p ( x ) and Q : q ( x ) , x ∈ X : [Equation]

Original: [Equation]
Revised: [Equation]

Original: where f is what we call Stein critic that exploits over function space F and if F is large enough then
Revised: where f is what we call Stein critic that exploits over function space F and if F is large enough then

Original: S ( Q , P ) = 

Revised: S ( Q , P ) =

Original: if and only if Q = P .
Revised: if and only if Q = P .

Original: Note that in (1), we do not need the normalized constant for p ( x ) which enables Stein discrepancy to deal with unnormalized density.
Revised: Note that in (1), we do not need the normalized constant for p ( x ) which enables Stein discrepancy to deal with unnormalized density.

Original: If F is a unit ball in a Reproducing Kernel Hilbert Space (RKHS) with a positive deﬁnite kernel function k ( · , · ) , then the supremum in (1) would have a close form (see Liu et al.
Revised: If F is a unit ball in a Reproducing Kernel Hilbert Space (RKHS) with a positive deﬁnite kernel function k ( · , · ) , then the supremum in (1) would have a close form (see Liu et al.

Original: Chwialkowski et al.
Revised: Chwialkowski et al.

Original: ; Oates et al.
Revised: ; Oates et al.

Original: (2017) for more details):
Revised: (2017) for more details):

Original: [Equation]
Revised: [Equation]

Original: (13) where u p ( x , x (cid:48) ) = log p ( x ) (cid:62) k ( x , x (cid:48) ) log p ( x (cid:48) ) + log p ( x ) (cid:62) ∇ x k ( x, x (cid:48) ) 
Revised: ) where

Original: + ∇ x k ( x , x (cid:48) ) (cid:62) log p ( x (cid:48) ) + tr ( ∇ x , x (cid:48) k ( x , x (cid:48) )) .
Revised: [Equation]

Original: The (13) gives the Kernel Stein Discrepancy (KSD).
Revised: The (11) gives the Kernel Stein Discrepancy (KSD).

Original: C P ROOFS OF R ESULTS IN S ECTION 3. C. P ROOF OF T HEOREM
Revised: C P ROOFS OF R ESULTS IN S ECTION 3. C. P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: Fixing the Wasserstein critic D , we are going to solve for
Revised: Using Kantorovich’s duality [Equation] we rewrite the problem as

Original: [Equation]
Revised: [Equation]

Original: By deﬁnition, if there exists a P E -measure zero set with positive supp( P ) -measure, then S ( P , P E ) = ∞ .
Revised: where the minimization with respect to E is over all energy functions [Equation] the minimization with respect to P is over all probability distributions with continuous density

Original: Hence, to solve (14), it sufﬁces to consider distributions whose support belongs to supp( P E ) .
Revised: [Equation]

Original: Since C ∞ is dense in L 2 , we can restrict to those P ’s that are absolutely continuous with respect to P E
Revised: and the maximization is over all 1-Lipschitz continuous functions. From the deﬁnition of kernel Stein discrepancy

Original: [Equation]
Revised: [Equation]

Revised: S ( P , P E ) is inﬁnite if P is not absolutely continuous with respect to P E .

Revised: Hence, to minimize the objective, it sufﬁces to consider those P ’s that are absolutely continuous with respect to P E .

Original: Set h ( x ) :=
Revised: We set h ( x ) :

Original:  P ( x ) / P E ( x ) − 1 ,  the problem above becomes [Equation]
Revised: = d P /d P E ( x ) − 1 , where d P /d P E is the Radon-Nikodym derivative.

Original: For theminimization problem above, invoking Lagrangian duality gives
Revised: It follows that the problem becomes

Original: [Equation]
Revised: [Equation]

Original: Applying the approximation log(1 + a ) = a + O ( a 2 ) to the minimization problem above yields
Revised: where the minimization with respect to h is over all L 1 ( P E ) functions with P E -expectation zero.

Original: [Equation] Consider a further approximation [Equation]
Revised: Fixing E , we claim that we can swap min h and max D .

Original: By deﬁnition of (cid:107) D− t (cid:107) H − 1 ( P E ; k ) {+= 
 for some element x 0+} ,  the inner inﬁmum equals − 14 λ 2 (cid:107) D − t (cid:107) H − 1 ( P E ; k ) , and hence (15) equals
Revised: Indeed, without loss of generality, we can restrict D to be such that D ( x 0 ) {+= 
 for some element x 0+} , as a constant shift does not change the value of E P E [(1 + h ) D ] − E P real

Original: [Equation]
Revised: [ D ] .

Original: Formally, the gap between (14) and (15) is O (1 /λ 22 ) , which completes the proof. We note that  a sufﬁcient condition to make the above formal derivation hold is that X is compact  and  the kernel is bounded, although this can be greatly weakened.
Revised: The set of Lipschitz functions that vanish at x 0 is a Banach space, and the subset of 1-Lipschtiz functions is compact Weaver (1999). Moreover, L 1 ( P E ) is also a Banach space. The above veriﬁes the condition of Sion’s minimax theorem, and thus the claim is proved.

Revised: Swapping min h and max D , we consider

Revised: [Equation]

Revised: where the second equality follows from the chain rule of the derivative.

Revised: For the inner minimum

Revised: [Equation]

Revised: we have that

Revised: [Equation]

Revised: where the ﬁrst equality follows the introduction of the auxiliary variable r , the second equality follows a change of variable from h to rh ; and the second equality the third equality follows from the deﬁnition of the kernel Sobolev dual norm.

Revised: Plugging in the previous equation yields the ideal result.

Original: C. P ROOF FOR T HEOREM
Revised: C. P ROOF FOR T HEOREM

Original: Proof.
Revised: Proof.

Original: Essentially the result is a consequence of distributionally robust optimization with Wasserstein metric (Gao & Kleywegt, 2016; Gao et al., 2017).
Revised: Essentially, the result is a consequence of distributionally robust optimization with Wasserstein metric (Gao & Kleywegt, 2016; Gao et al., 2017).

Original: Here we provide a simpliﬁed version for completeness.
Revised: Here we provide a simpliﬁed version for completeness.

Original: Consider
Revised: Using the deﬁnition of Stein discrepancy, we rewrite the problem as [Equation] Using the compactness of the Stein class and similar argument to the proof of Theorem 1, we can swap the minimization over P and the maximization over f . Fixing f , consider

Original: [Equation]
Revised: [Equation]

Original: By assumption, using the deﬁnition of Wasserstein metric, we write the problem above as 
Revised: Using the deﬁnition of Wasserstein metric [Equation] where the minimization is over all joint distributions of ( x , y ) with x -marginal P real and y -marginal P , we write the problem above as [Equation]

Revised: where γ has marginals P real and P .

Revised: Since P is unconstrained, the above problem is further equivalent to

Original: [Equation]
Revised: [Equation]

Original: where the minimization is over all joint distributions of ( x , y ) with x -marginal being P real .
Revised: where the minimization is over all joint distributions of ( x , y ) with x -marginal being P real .

Original: Using the law of total expectation, the problem above is equivalent to
Revised: Using the law of total expectation, the problem above is equivalent to

Original: [Equation]
Revised: [Equation]

Original: where the minimization  is over γ x , all conditional distributions of y given x. If Lip( λ 2 A P E f ) < 1 , then the minimal value equals −∞ , otherwise the minimal value equals λ 2 E y ∼ P real [ A P E f ( y )]
Revised: where the minimization in the ﬁrst equation is over γ x , all conditional distributions of y given x , and the exchanging of min and E follows from the interchangebability principle (Shapiro et al., 2009); the second equality holds because the inﬁmum can be restricted to the set of point masses. We ﬁnally have the original problem is equivalent to

Original: [Equation]
Revised: [Equation]

Original: Hence the proof is completed.
Revised: Hence the proof is completed using the deﬁnition of Moreau-Yosida regularization.

Original: D P ROOFS AND M ORE D ISCUSSIONS IN S ECTIOND. D ETAILS FOR O NE -D IMENSIONAL C ASE
Revised: D P ROOFS AND M ORE D ISCUSSIONS IN S ECTIOND. D ETAILS FOR O NE -D IMENSIONAL C ASE

Original: For the analysis of 1-dim regularized WGAN in section 3.1.1, we assume a Gaussian likelihood function forgenerated sample x(cid:48) , µ ( x(cid:48) ) =
Revised: For the analysis of 1-dim regularized WGAN in section 3.1.1, we assume a Gaussian likelihood function for sample x , µ ( x ) =

Original: exp( − 12 ( x(cid:48) − b ) 2 ) which is up to a constant.
Revised: exp( − 12 ( x − b ) 2 ) which is up to a constant.

Original: Its parameter can be estimated by b =
Revised: Its parameter can be estimated by b =

Original: E [ x ] .
Revised: E [ x ] .

Original: Then since x (cid:48)
Revised: Then for generated sample x (cid:48)

Original: = θz , we have E (log µ ( θz ))
Revised: = θz , we have E (log µ ( θz ))

Original: = − 12 E
Revised: = − 1 2 E

Original: [ z 2 ] θ 2 +
Revised: [ z 2 ] θ 2 +

Original: E [ z ] E [ x ] θ − 12 E [ x ] 2 .
Revised: E [ z ] E [ x ] θ − 12 E [ x ] 2 .

Original: Like the case in WGAN, we consider E [ x ] =
Revised: Like the case in WGAN, we consider E [ x ] =

Original: E
Revised: E

Original: [ z ] = 1 .
Revised: [ z ] = 1 .

Original: Assume Var[ z ] =
Revised: Assume Var[ z ] =

Original: and we have E [ z 2 ] = 1 + E [ z ] .
Revised: and we have E [ z 2 ] = 1 + E [ z ] .

Original: Hence, for the analysis on likelihood- (and entropy-) regularized
Revised: Hence, for the analysis on likelihood- (and entropy-) regularized

Original: WGAN, we can study the following system:
Revised: WGAN, we can study the following system:

Original: [Equation]
Revised: [Equation]

Original: When λ = 1 , the above objective degrades to (6); when λ < 0 (likelihood-regularization), the the gradient of regularization term pushes θ to shrink, which helps for convergence; when λ >(entropy-regularization), the added term forms an ampliﬁying strength on θ and leads to divergence.
Revised: When λ = 1 , the above objective degrades to (4); when λ < 0 (likelihood-regularization), the the gradient of regularization term pushes θ to shrink, which helps for convergence; when λ >(entropy-regularization), the added term forms an ampliﬁying strength on θ and leads to divergence.

Original: Interestingly, the added terms λ 1 2 (1 + φ  )  2 + λ 2 2 ( θ + φ )  2 in (8) and the original terms ψ − ψ · θ in WGAN play both necessary roles to guarantee the convergence to the unique optimum points [ ψ ∗ , θ ∗ , φ ∗ ] =
Revised: We proceed to consider 1-dim case of Stein Bridging with energy model p φ ( x ) = exp( − 12 x 2 − φx ) . If using KSD with kernel k ( x 1 , x 2 ) = I ( x 1 = x 2 ) , then S ( P real , P E ) =

Revised: E x 1 ,

Revised: [Equation]

Revised: [Equation]

Revised: ( x

Revised: [Equation]

Revised: µ ( x

Revised: [Equation]

Revised: ( x 1 , x

Revised: [Equation]

Revised: ∇

Revised: [Equation]

Revised: x

Revised: [Equation]

Revised: [Equation]

Revised: E x

Revised: [( ∇ x log p

Revised: [Equation]

Revised: ( x ) − ∇ x log µ ( x )) 2 ] = ( φ + E [ x ]) 2 .

Revised: Similarly, one can obtain S ( P G , P E ) =

Revised: ( φ + θ E

Revised: [ z ]) 2 .

Revised: Therefore we arrive at the objective in (6)

Revised: [Equation]

Revised: Interestingly, the added terms λ 1 2 (1 + φ ) 2 + λ 2 2 ( θ + φ ) 2 in (6) and the original terms ψ − ψ · θ in WGAN play both necessary roles to guarantee the convergence to the unique optimum points [ ψ ∗ , θ ∗ , φ ∗ ] =

Original: [0 , 1 , − 1] .
Revised: [0 , 1 , − 1] .

Original: If we remove the critic and optimize θ and φ with the remaining loss terms, we would ﬁnd that the training would converge but not necessarily to [ ψ ∗ , θ ∗ ] =
Revised: If we remove the critic and optimize θ and φ with the remaining loss terms, we would ﬁnd that the training would converge but not necessarily to [ ψ ∗ , θ ∗ ] =

Original: [0 , 1] (since the optimum points are not unique in this case).
Revised: [0 , 1] (since the optimum points are not unique in this case).

Original: On the other hand, if we remove the estimator, the system degrades to (6) and would not converge to the unique optimum point [ ψ ∗ , θ ∗ ] =
Revised: On the other hand, if we remove the estimator, the system degrades to (4) and would not converge to the unique optimum point [ ψ ∗ , θ ∗ ] =

Original: If we consider both of the world and optimize three terms together, the training would converge to a unique global optimum [ ψ ∗ , θ ∗ , φ ∗ ] =
Revised: If we consider both of the world and optimize three terms together, the training would converge to a unique global optimum [ ψ ∗ , θ ∗ , φ ∗ ] =

Original: [0 , 1 , − 1] .
Revised: [0 , 1 , − 1] .

Original: D. P ROOF FOR P ROPOSITION
Revised: D. P ROOF FOR P ROPOSITION

Original: Proof.
Revised: Proof.

Original: Instead of directly studying the optimization for (8), we ﬁrst prove the following problem will converge to the unique optimum,
Revised: Instead of directly studying the optimization for (6), we ﬁrst prove the following problem will converge to the unique optimum,

Original: [Equation]
Revised: [Equation]

Original: Applying alternate SGD we have the following iterations:
Revised: Applying alternate SGD we have the following iterations:

Original: [Equation]
Revised: [Equation]

Original: Then we obtain the relationship between adjacent iterations:
Revised: Then we obtain the relationship between adjacent iterations:

Original: [Equation]
Revised: [Equation]

Original: We further calculate the eigenvalues for matrix M and have the following equations (assume the eigenvalue as λ ): 
Revised: We further calculate the eigenvalues for matrix M and have the following equations (assume the eigenvalue as λ ): [Equation] ( λ [Equation]

Revised: +

Original: [Equation]
Revised: [Equation]

Original:  One can verify that the solutions to the above equation satisfy | λ | < (cid:112) (1 − η + η 2 )(1 + η − η 2 ) .
Revised: = [Equation] One can verify that the solutions to the above equation satisfy | λ | < (cid:112) (1 − η + η 2 )(1 + η − η 2 ) .

Original: Then we have the following relationship
Revised: Then we have the following relationship

Original: [Equation]
Revised: [Equation]

Original: where λ m denotes the eigenvalue with the maximum absolute value of matrix M .
Revised: where λ m denotes the eigenvalue with the maximum absolute value of matrix M .

Original: Hence, we have
Revised: Hence, we have

Original: [Equation]

Original: +

Original: [Equation]

Original: + φ 2 t +

Original: [Equation]
Revised: [Equation]

Original: t

Original: [Equation]

Original: 1 ≤

Original: [Equation]

Original: 1 − η + η

Original: [Equation]

Original: + η − η [Equation] [ ψ 2 t + φ 2 t + θ 2 t ] . We proceed to replace ψ , φ and θ in (16) by ψ (cid:48) , φ (cid:48) and θ (cid:48) respectively and conduct a change of variable: let θ (cid:48)
Revised: We proceed to replace ψ , φ and θ in (13) by ψ (cid:48) , φ (cid:48) and θ (cid:48) respectively and conduct a change of variable: let θ (cid:48)

Original: = 1 − θ and φ (cid:48)
Revised: = 1 − θ and φ (cid:48)

Original: = − 1 − φ .
Revised: = − 1 − φ .

Original: Then we get the conclusion in the proposition.
Revised: Then we get the conclusion in the proposition.

Original: D. G ENERALIZATION TO B ILINEAR S YSTEMS
Revised: D. G ENERALIZATION TO B ILINEAR S YSTEMS

Original: Our analysis in the one-dimension case inspires us that we can add afﬁliated variable to modify the objective and stabilize the training for general bilinear system.
Revised: Our analysis in the one-dimension case inspires us that we can add afﬁliated variable to modify the objective and stabilize the training for general bilinear system.

Original: The bilinear system is of wide interest for researchers focusing on stability of GAN training (Goodfellow (2017); Liang & Stokes (2019);
Revised: The bilinear system is of wide interest for researchers focusing on stability of GAN training (Goodfellow (2017); Liang & Stokes (2019);

Original: Gidel et al.
Revised: Gidel et al.

Original: ; Gemp & Mahadevan (2018)).
Revised: ; Gemp & Mahadevan (2018)).

Original: The general bilinear function can be written as
Revised: The general bilinear function can be written as

Original: [Equation]
Revised: [Equation]

Original:  where ψ , θ are both r -dimensional vectors and the objective is min θ max ψ F ( ψ , θ ) which can be seen as a basic form of various GAN objectives.
Revised: [Equation] where ψ , θ are both r -dimensional vectors and the objective is min θ max ψ F ( ψ , θ ) which can be seen as a basic form of various GAN objectives.

Original: Unfortunately, if we directly use simultaneous (resp.
Revised: Unfortunately, if we directly use simultaneous (resp.

Original: alternate) SGD to optimize such objectives, one can obtain divergence (resp.
Revised: alternate) SGD to optimize such objectives, one can obtain divergence (resp.

Original: ﬂuctuation).
Revised: ﬂuctuation).

Original: To solve the issue, some recent papers propose several optimization algorithms, like extrapolation from the past (Gidel et al.
Revised: To solve the issue, some recent papers propose several optimization algorithms, like extrapolation from the past (Gidel et al.

Original: crossing the curl (Gemp & Mahadevan (2018)) and consensus optimization (Liang & Stokes (2019)).
Revised: crossing the curl (Gemp & Mahadevan (2018)) and consensus optimization (Liang & Stokes (2019)).

Original: Also, Liang & Stokes (2019) shows that it is the interaction term which generates non-zero values for ∇ θψ F and ∇ ψθ F that leads to such instability of training.
Revised: Also, Liang & Stokes (2019) shows that it is the interaction term which generates non-zero values for ∇ θψ F and ∇ ψθ F that leads to such instability of training.

Original: Different from previous works that focused on algorithmic perspective, we propose to add new afﬁliated variables which modify the objective function and allow the SGD algorithm to achieve convergence without changing the optimum points.
Revised: Different from previous works that focused on algorithmic perspective, we propose to add new afﬁliated variables which modify the objective function and allow the SGD algorithm to achieve convergence without changing the optimum points.

Original: Based on the minimax objective of (17) we add afﬁliated r -dimensional variable φ (corresponding to the estimator in our model) the original system and tackle the following problem:
Revised: Based on the minimax objective of (14) we add afﬁliated r -dimensional variable φ (corresponding to the estimator in our model) the original system and tackle the following problem:

Original: [Equation]
Revised: [Equation]

Original: (18) where H ( φ , θ ) =
Revised: (15) where H ( φ , θ ) =

Original: 12 ( θ + φ )
Revised: 12 ( θ + φ )

Original: (cid:62) B ( θ + φ ) , B = ( AA (cid:62) ) 12 and α is a non-negative constant.
Revised: (cid:62) B ( θ + φ ) , B = ( AA (cid:62) ) 12 and α is a non-negative constant.

Original: Theoretically, the new problem keeps the optimum points of (17) unchanged.
Revised: Theoretically, the new problem keeps the optimum points of (14) unchanged.

Original: Let L ( ψ , φ , θ ) =
Revised: Let L ( ψ , φ , θ ) =

Original: F ( ψ , θ ) + αG ( φ , θ )
Revised: F ( ψ , θ ) + αG ( φ , θ )

Original: Proposition 2.
Revised: Proposition 2.

Original: Assume the optimum point of min θ max ψ F ( ψ , θ ) are [ ψ ∗ , θ ∗ ] , then the optimum points of (18) would be [ ψ ∗ , θ ∗ , φ ∗ ] where φ ∗ = − θ ∗ .
Revised: Assume the optimum point of min θ max ψ F ( ψ , θ ) are [ ψ ∗ , θ ∗ ] , then the optimum points of (15) would be [ ψ ∗ , θ ∗ , φ ∗ ] where φ ∗ = − θ ∗ .

Original: Proof.
Revised: Proof.

Original: The condition tells us that ∇ θ F ( ψ ∗ , θ ) = 0 and ∇ ψ F ( ψ , θ ∗ ) = 0 .
Revised: The condition tells us that ∇ θ F ( ψ ∗ , θ ) = 0 and ∇ ψ F ( ψ , θ ∗ ) = 0 .

Original: Then we derive the gradients for L ( ψ, φ, θ ) ,
Revised: Then we derive the gradients for L ( ψ, φ, θ ) ,

Original: [Equation]
Revised: [Equation]

Original: Combining (20) and (21) we get φ ∗ = − θ ∗ .
Revised: Combining (17) and (18) we get φ ∗ = − θ ∗ .

Original: Hence, the optimum point of (18) is [ ψ ∗ , θ ∗ , φ ∗ ] where φ ∗ = − θ ∗ .
Revised: Hence, the optimum point of (15) is [ ψ ∗ , θ ∗ , φ ∗ ] where φ ∗ = − θ ∗ .

Original: The advantage of the new problem is that it can be solved by SGD algorithm and guarantees convergence theoretically.
Revised: The advantage of the new problem is that it can be solved by SGD algorithm and guarantees convergence theoretically.

Original: We formulate the results in the following theorem.
Revised: We formulate the results in the following theorem.

Original: Theorem 4.
Revised: Theorem 4.

Original: For problem min θ max ψ min φ L ( ψ , φ , θ ) using alternate SGD algorithm, i.e.,
Revised: For problem min θ max ψ min φ L ( ψ , φ , θ ) using alternate SGD algorithm, i.e.,

Original: [Equation]
Revised: [Equation]

Original:  we can achieve convergence to [ ψ ∗ , θ ∗ , φ ∗ ] where φ ∗ = − θ ∗ with at least linear rate of (1 − η 1 + η 22 )
Revised: [Equation] we can achieve convergence to [ ψ ∗ , θ ∗ , φ ∗ ] where φ ∗ = − θ ∗ with at least linear rate of (1 − η 1 + η 22 )

Original: (1 + η 2 − η 21 ) where η 1 = ησ min , η 2 = ησ max and σ min (resp.
Revised: (1 + η 2 − η 21 ) where η 1 = ησ min , η 2 = ησ max and σ min (resp.

Original: σ max ) denotes the maximum (resp. minimum) singular value of matrix A .
Revised: σ max ) denotes the maximum (resp. minimum) singular value of matrix A .

Original: To prove Theorem 3, we can prove a more general argument.
Revised: To prove Theorem 3, we can prove a more general argument.

Original: Lemma 1.
Revised: Lemma 1.

Original: If we consider any ﬁrst-order optimization method on (18), i.e.,
Revised: If we consider any ﬁrst-order optimization method on (15), i.e.,

Original: [Equation]
Revised: [Equation]

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: where U and V are the singular vectors decomposed by matrix A using SVD decomposition, i.e.,
Revised: where U and V are the singular vectors decomposed by matrix A using SVD decomposition, i.e.,

Original: A = UDV (cid:62) and the triple ([ (cid:101) ψ t ] i , [ (cid:101) φ t ] i , [ (cid:101)
Revised: A = UDV (cid:62) and the triple ([ (cid:101) ψ t ] i , [ (cid:101) φ t ] i , [ (cid:101)

Original: [Equation]
Revised: [Equation]

Original: 1 ≤ i ≤ r follows the update rule with step size σ i η as the same optimization method on a unidimensional problem
Revised: 1 ≤ i ≤ r follows the update rule with step size σ i η as the same optimization method on a unidimensional problem

Original: [Equation]
Revised: [Equation]

Original: with step size η , where σ i denotes the i -th singular value on the diagonal of D .
Revised: with step size η , where σ i denotes the i -th singular value on the diagonal of D .

Original: Proof.
Revised: Proof.

Original: The proof is extended from the proof of Lemma 3 in Gidel et al.
Revised: The proof is extended from the proof of Lemma 3 in Gidel et al.

Original: The general class of ﬁrst-order optimization methods derive the following updations:
Revised: The general class of ﬁrst-order optimization methods derive the following updations:

Original: [Equation]
Revised: [Equation]

Original: where ρ st , δ st , µ st ∈ R depend on speciﬁc optimization method (for example, in SGD, ρ tt = δ tt = µ tt remain as a non-zero constant for ∀ t and other coefﬁcients are zero).
Revised: where ρ st , δ st , µ st ∈ R depend on speciﬁc optimization method (for example, in SGD, ρ tt = δ tt = µ tt remain as a non-zero constant for ∀ t and other coefﬁcients are zero).

Original: Using SVD A
Revised: Using SVD A

Original: = UDV (cid:62) and the fact θ ∗ = − φ ∗ , B
Revised: = UDV (cid:62) and the fact θ ∗ = − φ ∗ , B

Original: = ( UDD (cid:62) U (cid:62) )
Revised: = ( UDD (cid:62) U (cid:62) )

Original: =
Revised: =

Original: D , we have
Revised: D , we have

Original: [Equation]
Revised: [Equation]

Original: and equivalently,
Revised: and equivalently,

Original: [Equation]
Revised: [Equation]

Original: Note that D is a rectangular matrix with non-zero elements on a diagonal block of size r .
Revised: Note that D is a rectangular matrix with non-zero elements on a diagonal block of size r .

Original: Hence, the above r -dimensional problem can be reduced to r unidimensional problems:
Revised: Hence, the above r -dimensional problem can be reduced to r unidimensional problems:

Original: [Equation]
Revised: [Equation]

Original: The above iterations can be conducted independently in each dimension where the optimization in i -th dimension follows the same updating rule with step size σ i η as problem in (23).
Revised: The above iterations can be conducted independently in each dimension where the optimization in i -th dimension follows the same updating rule with step size σ i η as problem in (20).

Original: Furthermore, since problem (23) can achieve convergence with a linear rate of (1 − η + η 2 )(1+ η − η 2 ) using alternate SGD (the proof is similar to that of ((16))), the multi-dimensional problem in (18) can achieve convergence by SGD with at least a rate of (1 − η 1 + η 22 )
Revised: Furthermore, since problem (20) can achieve convergence with a linear rate of (1 − η + η 2 )(1+ η − η 2 ) using alternate SGD (the proof is similar to that of ((13))), the multi-dimensional problem in (15) can achieve convergence by SGD with at least a rate of (1 − η 1 + η 22 )

Original: (1+ η 2 − η 21 ) where η 1 = ησ max , η 2 = ησ min and σ max (resp.
Revised: (1+ η 2 − η 21 ) where η 1 = ησ max , η 2 = ησ min and σ max (resp.

Original: σ min ) denotes the maximum (resp. minimum) singular value of matrix
Revised: σ min ) denotes the maximum (resp. minimum) singular value of matrix

Original: A .
Revised: A .

Original: We conclude the proof for Theorem 4.
Revised: We conclude the proof for Theorem 4.

Original: Theorem 4 suggests that the added term H ( φ , θ ) with afﬁliated variables φ could help the SGD algorithm achieve convergence to the the same optimum points as directly optimizing F ( ψ , θ ) .
Revised: Theorem 4 suggests that the added term H ( φ , θ ) with afﬁliated variables φ could help the SGD algorithm achieve convergence to the the same optimum points as directly optimizing F ( ψ , θ ) .

Original: Our method is related to consensus optimization algorithm
Revised: Our method is related to consensus optimization algorithm

Original: (Liang & Stokes (2019)) which adds a regularization term (cid:107)∇ θ F ( ψ , θ ) (cid:107) + (cid:107)∇ ψ F ( ψ , θ ) (cid:107) to (17) resulting extra quadratic terms for θ and ψ .
Revised: (Liang & Stokes (2019)) which adds a regularization term (cid:107)∇ θ F ( ψ , θ ) (cid:107) + (cid:107)∇ ψ F ( ψ , θ ) (cid:107) to (14) resulting extra quadratic terms for θ and ψ .

Original: The disadvantage of such method is the requirement of Hessian matrix of F ( ψ , θ ) which is computational expensive for high-dimensional data.
Revised: The disadvantage of such method is the requirement of Hessian matrix of F ( ψ , θ ) which is computational expensive for high-dimensional data.

Original: By contrast, our solution only requires the ﬁrst-order derivatives.
Revised: By contrast, our solution only requires the ﬁrst-order derivatives.

Original: D. S TRONGLY C ONVEXITY
Revised: D. S TRONGLY C ONVEXITY

Original: In section 3.1.2, we assume H ( θ , φ ) as a µ -strongly convex function which indicates that it satisﬁes the conditions:
Revised: In section 3.1.2, we assume H ( θ , φ ) as a µ -strongly convex function which indicates that it satisﬁes the conditions:

Original: [Equation]
Revised: [Equation]

Original: Bedises, F ( θ , ψ ) is µ -strongly convex for θ and µ -strongly concave for ψ so it satisﬁes:
Revised: Bedises, F ( θ , ψ ) is µ -strongly convex for θ and µ -strongly concave for ψ so it satisﬁes:

Original: [Equation]
Revised: [Equation]

Original: In section 3.1.
Revised: In section 3.1.

Original: [Equation]
Revised: [Equation]

Original: we also deﬁne h ( ω h ) =
Revised: we also deﬁne h ( ω h ) =

Original: ∇ θ H + ∇ φ H and f ( ω f ) = ∇ θ F − ∇ ψ F , so the above condition can be written in a more compact form,
Revised: ∇ θ H + ∇ φ H and f ( ω f ) = ∇ θ F − ∇ ψ F , so the above condition can be written in a more compact form,

Original: [Equation]
Revised: [Equation]

Original: D. P ROOF FOR T HEOREM
Revised: D. P ROOF FOR T HEOREM

Original: The proof relies on two lemmas,
Revised: The proof relies on two lemmas,

Original: Lemma 2.
Revised: Lemma 2.

Original: For any ω ∈ Ω and ω + = P Ω ( ω + u ) , then we have
Revised: For any ω ∈ Ω and ω + = P Ω ( ω + u ) , then we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Since ω + is a projection of ω + u on a convex set Ω , we have
Revised: Since ω + is a projection of ω + u on a convex set Ω , we have

Original: [Equation]
Revised: [Equation]

Original: Rearranging the above inequality one can easily get the lemma.
Revised: Rearranging the above inequality one can easily get the lemma.

Original: Lemma 3.
Revised: Lemma 3.

Original: If function Φ( ω ) is µ -strongly convex, we have
Revised: If function Φ( ω ) is µ -strongly convex, we have

Original: [Equation]
Revised: [Equation]

Original: Similarly, if Φ( ω ) is µ -strongly concave, we have µ ( ω − ω ∗ ) ≤ ∇ − Φ( ω ) (cid:62) ( ω − ω ∗ ) .
Revised: Similarly, if Φ( ω ) is µ -strongly concave, we have µ ( ω − ω ∗ ) ≤ ∇ − Φ( ω ) (cid:62) ( ω − ω ∗ ) .

Original: Proof.
Revised: Proof.

Original: By optimality of ω ∗ , we have
Revised: By optimality of ω ∗ , we have

Original: [Equation]
Revised: [Equation]

Original: Since Φ is µ -convex, we can further derive
Revised: Since Φ is µ -convex, we can further derive

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: (Proof for Theorem 3)
Revised: (Proof for Theorem 3)

Original: We apply Lemma 2 to (11) with ( ω , u , ω + ) =
Revised: We apply Lemma 2 to (9) with ( ω , u , ω + ) =

Original: ( ω t +1 / 2 f , − ηf ( ω t +1 / 2 f ) , ω t +1 f ) and we have
Revised: ( ω t +1 / 2 f , − ηf ( ω t +1 / 2 f ) , ω t +1 f ) and we have

Original: [Equation]
Revised: [Equation]

Original: ω
Revised: ω

Original: [Equation]
Revised: [Equation]

Original: +1 f
Revised: +1 f

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: According to Lemma 3, we have
Revised: According to Lemma 3, we have

Original: [Equation]
Revised: [Equation]

Original: Plug (27) into (26) and we get
Revised: Plug (24) into (23) and we get

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The above inequality is equivalent to
Revised: The above inequality is equivalent to

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Similarly, one can obtain
Revised: Similarly, one can obtain

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Combining (29) and (31) we have
Revised: Combining (26) and (28) we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Hence
Revised: Hence

Original: [Equation]
Revised: [Equation]

Original: if 12 µ < η < 1 µ we have 0 < 2 − 2 ηµ < 1 and (cid:107) ω t − ω ∗ (cid:107) 22 ≤ (2 − 2 ηµ ) t (cid:107) ω 0 − ω ∗ (cid:107)
Revised: if 12 µ < η < 1 µ we have 0 < 2 − 2 ηµ < 1 and (cid:107) ω t − ω ∗ (cid:107) 22 ≤ (2 − 2 ηµ ) t (cid:107) ω 0 − ω ∗ (cid:107)

Original: E D ETAILS FOR E XPERIMENT S ETUP E. S YNTHETIC D ATASETS
Revised: E D ETAILS FOR E XPERIMENT S ETUP E. S YNTHETIC D ATASETS

Original: We provide the details for two synthetic datasets.
Revised: We provide the details for two synthetic datasets.

Original: The Two-Circle dataset consists of 24 Gaussian mixtures where 8 of them are located in an inner circle with radius r 1 = 4 and 16 of them lie in an outer circle with radius r 2 = 8 .
Revised: The Two-Circle dataset consists of 24 Gaussian mixtures where 8 of them are located in an inner circle with radius r 1 = 4 and 16 of them lie in an outer circle with radius r 2 = 8 .

Original: For each Gaussian component, the covariance matrix is (cid:18) 0 .
Revised: For each Gaussian component, the covariance matrix is (cid:18) 0 .

Original:  0 .(cid:19)
Revised:  0 .(cid:19)

Original: =
Revised: =

Original: σ 1 I and the mean value is [ r 1 cos t, r 1 sin t ] , where t = 2 π · k 8 , k = 1 , · · · , 8 , for the inner circle, and [ r 2 cos t, r 2 sin t ] , where t = 2 π · k 16 , k = 1 , · · · , 16 for the outer circle.
Revised: σ 1 I and the mean value is [ r 1 cos t, r 1 sin t ] , where t = 2 π · k 8 , k = 1 , · · · , 8 , for the inner circle, and [ r 2 cos t, r 2 sin t ] , where t = 2 π · k 16 , k = 1 , · · · , 16 for the outer circle.

Original: We sample N 1 = 2000 points as true observed samples for model training.
Revised: We sample N 1 = 2000 points as true observed samples for model training.

Original: In section 5.5, we consider noised data scenario.
Revised: In section 5.5, we consider noised data scenario.

Original: In this case, we randomly add n noise points sampled from Gaussian distribution N ( 0 , σ 0 I ) where σ 0 = 2 to the original true samples.
Revised: In this case, we randomly add n noise points sampled from Gaussian distribution N ( 0 , σ 0 I ) where σ 0 = 2 to the original true samples.

Original: Here we set n = [40 , 100 , 160 , 300 , 400 , 600 , 800 , 1000] .
Revised: Here we set n = [40 , 100 , 160 , 300 , 400 , 600 , 800 , 1000] .

Original: The Two-Spiral dataset contains 100 Gaussian mixtures whose centers locate on two spiral-shaped curves.
Revised: The Two-Spiral dataset contains 100 Gaussian mixtures whose centers locate on two spiral-shaped curves.

Original: For each Gaussian component, the covariance matrix is (cid:18) 0 .
Revised: For each Gaussian component, the covariance matrix is (cid:18) 0 .

Original:  0 .(cid:19)
Revised:  0 .(cid:19)

Original: =
Revised: =

Original: σ 2 I and the mean value is [ − c 1 cos c 1 , c 1 sin c 1 ] , where c 1 = 2 π 3 + linspace (0 , 0 .
Revised: σ 2 I and the mean value is [ − c 1 cos c 1 , c 1 sin c 1 ] , where c 1 = 2 π 3 + linspace (0 , 0 .

Original: 5 , 50) · 2 π , for one spiral, and [ c 2 cos c 2 , − c 2 sin c 2 ] , where c 2 = 2 π 3 + linspace (0 , 0 . 5 , 50) · 2 π for another spiral.
Revised: 5 , 50) · 2 π , for one spiral, and [ c 2 cos c 2 , − c 2 sin c 2 ] , where c 2 = 2 π 3 + linspace (0 , 0 . 5 , 50) · 2 π for another spiral.

Original: We sample
Revised: We sample

Original: N 2 = 5000 points as true observed samples.
Revised: N 2 = 5000 points as true observed samples.

Original: In section 5.5, we consider insufﬁcient data scenario.
Revised: In section 5.5, we consider insufﬁcient data scenario.

Original: In this case
Revised: In this case

Original: [Equation]
Revised: [Equation]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] 
Revised: [Table] [Equation] [Table]

Original:  [Table]
Revised: [Equation] [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation] 
Revised: [Equation] [Table] [Equation] [Table]

Original:  [Table]
Revised: [Equation] [Table] [Equation] [Table]

Original: [Equation] 
Revised: [Equation] [Table] [Equation] [Table]

Original:  [Table]
Revised: [Equation] [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: We specify D 1 as Wasserstein distance or JS divergence in our paper and for D 2 and D 3 we consider the general Stein discrepancy or kernel Stein discrepancy.
Revised: We specify D 1 as Wasserstein distance or JS divergence in our paper and for D 2 and D 3 we consider the general Stein discrepancy or kernel Stein discrepancy.

Original: Here we use W , J S to denote Wasserstein distance and JS divergence respectively, and S , S k to represent general Stein discrepancy and kernel Stein discrepancy respectively.
Revised: Here we use W , J S to denote Wasserstein distance and JS divergence respectively, and S , S k to represent general Stein discrepancy and kernel Stein discrepancy respectively.

Original: We omit the gradient penalty term for Wasserstein distance here but use it in experiments.
Revised: We omit the gradient penalty term for Wasserstein distance here but use it in experiments.

Original: E. M ODEL S PECIFICATIONS AND T RAINING A LGORITHM
Revised: E. M ODEL S PECIFICATIONS AND T RAINING A LGORITHM

Original: In different tasks, we consider different model speciﬁcations in order to meet the demand of capacify as well as test the effectiveness under various settings.
Revised: In different tasks, we consider different model speciﬁcations in order to meet the demand of capacify as well as test the effectiveness under various settings.

Original: Our proposed framework (3) adopts Wasserstein distance for the ﬁrst term and two Stein discrepancies for the second and the third terms.
Revised: Our proposed framework (3) adopts Wasserstein distance for the ﬁrst term and two Stein discrepancies for the second and the third terms.

Original: We can write (3) as a more general form
Revised: We can write (3) as a more general form

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where D 1 , D 2 , D 3 denote three general discrepancy measures for distributions.
Revised: where D 1 , D 2 , D 3 denote three general discrepancy measures for distributions.

Original: As stated in our remark, D 1 can be speciﬁed as arbitrary discrepancy measures for implicit generative models.
Revised: As stated in our remark, D 1 can be speciﬁed as arbitrary discrepancy measures for implicit generative models.

Original: Here we also use JS divergence, the objective for valina GAN.
Revised: Here we also use JS divergence, the objective for valina GAN.

Original: To well distinguish them, we call the model using Wasserstein distance (resp.
Revised: To well distinguish them, we call the model using Wasserstein distance (resp.

Original: JS divergence) as Joint-W (resp.
Revised: JS divergence) as Joint-W (resp.

Original: Joint-JS) in our experiments.
Revised: Joint-JS) in our experiments.

Original: On the other hand, the two Stein discrepancies in (3) can be speciﬁed by KSD (as deﬁned by S k in (13)) or general Stein discrepancy with an extra critic (as deﬁned by S in (1)).
Revised: On the other hand, the two Stein discrepancies in (3) can be speciﬁed by KSD (as deﬁned by S k in (11)) or general Stein discrepancy with an extra critic (as deﬁned by S in (1)).

Original: Hence, the two speciﬁcations for D 1 and the two for D 2 ( D 3 ) compose four different combinations in total, and we organize the objectives in each case in Table 4.
Revised: Hence, the two speciﬁcations for D 1 and the two for D 2 ( D 3 ) compose four different combinations in total, and we organize the objectives in each case in Table 4.

Original: In our experiments, we use KSD with RBF kernels for D 2 and D 3 in Joint-W and Joint-JS on two synthetic datasets.
Revised: In our experiments, we use KSD with RBF kernels for D 2 and D 3 in Joint-W and Joint-JS on two synthetic datasets.

Original: For MNIST with conditional training (given the digit class as model input), we also use KSD with RBF kernels.
Revised: For MNIST with conditional training (given the digit class as model input), we also use KSD with RBF kernels.

Original: For MNIST and CIFAR with unconditional training (the class is not given as known information), we ﬁnd that KSD cannot provide desirable results so we adopt general Stein discrepancy for higher model capacity.
Revised: For MNIST and CIFAR with unconditional training (the class is not given as known information), we ﬁnd that KSD cannot provide desirable results so we adopt general Stein discrepancy for higher model capacity.

Original: The objectives in Table 4 appear to be comutationally expensive.
Revised: The objectives in Table 4 appear to be comutationally expensive.

Original: In the worst case (using general
Revised: In the worst case (using general

Original: Stein discrepancy), there are two minimax operations where one is from GAN or WGAN and one is from Stein discrepancy estimation.
Revised: Stein discrepancy), there are two minimax operations where one is from GAN or WGAN and one is from Stein discrepancy estimation.

Original: To guarantee training efﬁciency, we alternatively update the generator, estimator, Wasserstein critic and Stein critic over the parameters θ , φ , ψ and π respectively.
Revised: To guarantee training efﬁciency, we alternatively update the generator, estimator, Wasserstein critic and Stein critic over the parameters θ , φ , ψ and π respectively.

Original: Speciﬁcally, in one iteration, we optimize the generator over θ and the estimator over φ with one step respectively, and then optimize the Wasserstein critic over ψ with n d steps and the Stein critic over π with n c steps.
Revised: Speciﬁcally, in one iteration, we optimize the generator over θ and the estimator over φ with one step respectively, and then optimize the Wasserstein critic over ψ with n d steps and the Stein critic over π with n c steps.

Original: Such training approach guarantees the same time complexity order of proposed method as that of GAN or WGAN, and the training time for our model can be bounded within constant times the time for training GAN model.
Revised: Such training approach guarantees the same time complexity order of proposed method as that of GAN or WGAN, and the training time for our model can be bounded within constant times the time for training GAN model.

Original: In our experiment, we set n d = n c = 5 and empirically ﬁnd that our model Stein Bridging would be two times slower than WGAN on average.
Revised: In our experiment, we set n d = n c = 5 and empirically ﬁnd that our model Stein Bridging would be two times slower than WGAN on average.

Original: We present the training algorithm for Stein Bridging in Algorithm 1.
Revised: We present the training algorithm for Stein Bridging in Algorithm 1.

Original: E. I MPLEMENTATION D ETAILS
Revised: E. I MPLEMENTATION D ETAILS

Original: We give the information of network architectures and hyper-parameter settings for our model as well as each competitor in our experiments.
Revised: We give the information of network architectures and hyper-parameter settings for our model as well as each competitor in our experiments.

Original: : observed training samples real .
Revised: : observed training samples real .

Original: 2 REQUIRE: θ 0 , φ 0 , ψ 0 , π 0 , initial parameters for generator, estimator, Wasserstein critic and Stein critic models respectively.
Revised: 2 REQUIRE: θ 0 , φ 0 , ψ 0 , π 0 , initial parameters for generator, estimator, Wasserstein critic and Stein critic models respectively.

Original: α E = 0 . 0002 , β E 1 = 0 .
Revised: α E = 0 . 0002 , β E 1 = 0 .

Original: 9 , β E 2 = 0 .
Revised: 9 , β E 2 = 0 .

Original: 999 , Adam hyper-parameters for explicit models.
Revised: 999 , Adam hyper-parameters for explicit models.

Original: α I = 0 .
Revised: α I = 0 .

Original: 0002 , β I 1 = 0 .
Revised: 0002 , β I 1 = 0 .

Original: 5 , β I 2 = 0 .
Revised: 5 , β I 2 = 0 .

Original: 999 , Adam hyper-parameters for implicit models.
Revised: 999 , Adam hyper-parameters for implicit models.

Original: λ 1 = 1 , λ 2 , weights for D 2 and D 3 (we suggest increasing λ 2 from 0 to 1 through training).
Revised: λ 1 = 1 , λ 2 , weights for D 2 and D 3 (we suggest increasing λ 2 from 0 to 1 through training).

Original: n d = 5 , n c = 5 number of iterations for Wasserstein critic and Stein critic, respectively, before one iteration for generator and estimator.
Revised: n d = 5 , n c = 5 number of iterations for Wasserstein critic and Stein critic, respectively, before one iteration for generator and estimator.

Original: B = 100 , batch size.
Revised: B = 100 , batch size.

Original: [Table]
Revised: [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: The energy function is often parametrized as a sum of multiple experts (Hinton (1999)) and each expert can have various function forms depending on the distributions.
Revised: The energy function is often parametrized as a sum of multiple experts (Hinton (1999)) and each expert can have various function forms depending on the distributions.

Original: If using sigmoid distribution, the energy function becomes (see section 2.1 in Kim & Bengio (2017) for details)
Revised: If using sigmoid distribution, the energy function becomes (see section 2.1 in Kim & Bengio (2017) for details)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where n ( x ) maps input x to a feature vector and could be speciﬁed as a deep neural network, which corresponds to deep energy model (Ngiam et al.
Revised: where n ( x ) maps input x to a feature vector and could be speciﬁed as a deep neural network, which corresponds to deep energy model (Ngiam et al.

Original: For synthetic datasets, we set the noise dimension as 4.
Revised: For synthetic datasets, we set the noise dimension as 4.

Original: All the generators are speciﬁed as a threelayer fully-connected (FC) neural network with neuron size 4 − 128 − 128 − 2 , and all the Wasserstein critics (or the discriminators in JS-divergence-based GAN) are also a three-layer FC network with neuron size 2 − 128 − 128 − 1 .
Revised: All the generators are speciﬁed as a threelayer fully-connected (FC) neural network with neuron size 4 − 128 − 128 − 2 , and all the Wasserstein critics (or the discriminators in JS-divergence-based GAN) are also a three-layer FC network with neuron size 2 − 128 − 128 − 1 .

Original: For the estimators, we set the expert number as 4 and the feature function n ( x ) is a FC network with neuron size 2 − 128 − 128 − 4 .
Revised: For the estimators, we set the expert number as 4 and the feature function n ( x ) is a FC network with neuron size 2 − 128 − 128 − 4 .

Original: Then in the last layer we sum the outputs from each expert as the energy value E ( x ) .
Revised: Then in the last layer we sum the outputs from each expert as the energy value E ( x ) .

Original: The activation units are searched within [ LeakyReLU, tanh, sigmoid, softplus ] .
Revised: The activation units are searched within [ LeakyReLU, tanh, sigmoid, softplus ] .

Original: The learning rate [1 e − 6 , 1 e − 5 , 1 e − 4 , 1 e − 3 , 1 e − 2] and the batch size [50 , 100 , 150 , 200] .
Revised: The learning rate [1 e − 6 , 1 e − 5 , 1 e − 4 , 1 e − 3 , 1 e − 2] and the batch size [50 , 100 , 150 , 200] .

Original: The gradient penalty weight for WGAN is searched in [0 , 0 . 1 , 1 , 10 , 100] .
Revised: The gradient penalty weight for WGAN is searched in [0 , 0 . 1 , 1 , 10 , 100] .

Original: For MNIST dataset, we set the noise dimension as 100.
Revised: For MNIST dataset, we set the noise dimension as 100.

Original: All the critics/discriminators are implemented as a four-layer network where the ﬁrst two layers adopt convolution operations with ﬁlter size 5 and stride [2 , 2] and the last two layers are FC layers.
Revised: All the critics/discriminators are implemented as a four-layer network where the ﬁrst two layers adopt convolution operations with ﬁlter size 5 and stride [2 , 2] and the last two layers are FC layers.

Original: The size for each layer is 1 − 64 − 128 − 256 − 1 .
Revised: The size for each layer is 1 − 64 − 128 − 256 − 1 .

Original: All the generators are implemented as a four-layer networks where the ﬁrst two layers are FC and the last two adopt deconvolution operations with ﬁlter size 5 and stride [2 , 2] .
Revised: All the generators are implemented as a four-layer networks where the ﬁrst two layers are FC and the last two adopt deconvolution operations with ﬁlter size 5 and stride [2 , 2] .

Original: The size for each layer is 100 − 256 − 128 − 64 − 1 .
Revised: The size for each layer is 100 − 256 − 128 − 64 − 1 .

Original: For the estimators, we consider the expert number as 128 and the feature function is the same as the Wasserstein critic except that the size of last layer is 128 .
Revised: For the estimators, we consider the expert number as 128 and the feature function is the same as the Wasserstein critic except that the size of last layer is 128 .

Original: Then we sum the outputs from each expert as the energy
Revised: Then we sum the outputs from each expert as the energy

Original: [Table]
Revised: [Table]

Original: value.
Revised: value.

Original: The activation units are searched within [ ReLU, LeakyReLU, tanh ] .
Revised: The activation units are searched within [ ReLU, LeakyReLU, tanh ] .

Original: The learning rate [2 e − 5 , 2 e − 4 , 2 e − 3 , 2 e − 2] and the batch size [32 , 64 , 100 , 128] .
Revised: The learning rate [2 e − 5 , 2 e − 4 , 2 e − 3 , 2 e − 2] and the batch size [32 , 64 , 100 , 128] .

Original: The gradient penalty weight for
Revised: The gradient penalty weight for

Original: WGAN is searched in [1 , 10 , 100 , 1000] .
Revised: WGAN is searched in [1 , 10 , 100 , 1000] .

Original: For CIFAR dataset, we adopt the same architecture as DCGAN for critics and generators.
Revised: For CIFAR dataset, we adopt the same architecture as DCGAN for critics and generators.

Original: As for the estimator, the architecture of feature function is the same as the critics except the last year where we set the expert number as 128 and sum each output as the output energy value.
Revised: As for the estimator, the architecture of feature function is the same as the critics except the last year where we set the expert number as 128 and sum each output as the output energy value.

Original: The architectures for
Revised: The architectures for

Original: Stein critic are the same as Wasserstein critic for both MNIST and CIFAR datasets.
Revised: Stein critic are the same as Wasserstein critic for both MNIST and CIFAR datasets.

Revised: In other words, we consider d (cid:48)

Revised: = 1 in (1) and further simply φ as an average of each dimension of E x ∼ P [ A Q f ( x )] .

Revised: Empirically we found this setting can provide efﬁcient computation and decent performance.

Original: E. E VALUATION M ETRICS
Revised: E. E VALUATION M ETRICS

Original: We adopt some quantitative metrics to evaluate the performance of each method on different tasks.
Revised: We adopt some quantitative metrics to evaluate the performance of each method on different tasks.

Original: In section 4.1, we use two metrics to test the sample quality: Maximum Mean Discrepancy (MMD) and High-quality Sample Rate (HSR).
Revised: In section 4.1, we use two metrics to test the sample quality: Maximum Mean Discrepancy (MMD) and High-quality Sample Rate (HSR).

Original: MMD measures the discrepancy between two distributions X and Y , MMD ( X, Y ) =
Revised: MMD measures the discrepancy between two distributions X and Y , MMD ( X, Y ) =

Original: (cid:107) 1 n (cid:80) ni
Revised: (cid:107) 1 n (cid:80) ni

Original: =1 Φ( x i ) − 1 m (cid:80) mj
Revised: =1 Φ( x i ) − 1 m (cid:80) mj

Original: =1 Φ( y i ) (cid:107) where x i and y j denote samples from
Revised: =1 Φ( y i ) (cid:107) where x i and y j denote samples from

Original: X and Y respectively and Φ maps each sample to a RKHS.
Revised: X and Y respectively and Φ maps each sample to a RKHS.

Original: Here we use RBF kernel and calculate
Revised: Here we use RBF kernel and calculate

Original: MMD between generated samples and true samples.
Revised: MMD between generated samples and true samples.

Original: HSR statistics the rate of high-quality samples over all generated samples.
Revised: HSR statistics the rate of high-quality samples over all generated samples.

Original: For Two-Cirlce dataset, we deﬁne the generated points whose distance from the nearest Gaussian component is less than σ 1 as high-quality samples.
Revised: For Two-Cirlce dataset, we deﬁne the generated points whose distance from the nearest Gaussian component is less than σ 1 as high-quality samples.

Original: We generatepoints in total and statistic HSR.
Revised: We generatepoints in total and statistic HSR.

Original: For Two-Spiral dataset, we set the distance threshold as 5 σ 2 and generate 5000 points to calculate HSR.
Revised: For Two-Spiral dataset, we set the distance threshold as 5 σ 2 and generate 5000 points to calculate HSR.

Original: As for Inception Score and CEPC.
Revised: As for Inception Score and CEPC.

Original: For MNIST, we pre-train a classiﬁer for 10 digits which can provide the test accuracy up to 99% for calculation of scores.
Revised: For MNIST, we pre-train a classiﬁer for 10 digits which can provide the test accuracy up to 99% for calculation of scores.

Original: The conditional entropy of predicted classes (CEPC) for given samples is deﬁned as H ( y | x ) ≈ 1 n (cid:80) ni =1 (cid:80) 10 k =1 p
Revised: The conditional entropy of predicted classes (CEPC) for given samples is deﬁned as H ( y | x [Equation]

Original: ( y k | x i ) log p ( y k | x i ) where x is a generated instance and y denotes the predicted class given x from a pre-trained classiﬁer.
Revised: where x is a generated instance and y denotes the predicted class given x from a pre-trained classiﬁer.

Original: CEPC measures how well a given sample can be classﬁed into a right class, i.e. the quality of such sample.
Revised: CEPC measures how well a given sample can be classﬁed into a right class, i.e. the quality of such sample.

Original: For CIFAR, we use the Inception V3 Network in Tensorﬂow as pre-trained classiﬁer.
Revised: For CIFAR, we use the Inception V3 Network in Tensorﬂow as pre-trained classiﬁer.

Original: In section 4.2, we use three metrics to characterize the performance for density estimation: KL divergence, JS divergence and AUC.
Revised: In section 4.2, we use three metrics to characterize the performance for density estimation: KL divergence, JS divergence and AUC.

Original: We divide the map into a 300 meshgrid, calculate the  density values of each point given by the estimators and compute the KL and JS divergences between estimated density and ground-truth density.
Revised: We divide the map into a 300 meshgrid, calculate the unnormalized density values of each point given by the estimators and compute the KL and JS divergences between estimated density and ground-truth density.

Original: Besides, we select the centers of each Gaussian components as positive examples (expected to have high densities) and randomly sample 10 points within a circle around each center as negative examples (expected to have relatively low densities) and rank them according to the densities given by the model.
Revised: Besides, we select the centers of each Gaussian components as positive examples (expected to have high densities) and randomly sample 10 points within a circle around each center as negative examples (expected to have relatively low densities) and rank them according to the densities given by the model.

Original: Then we obtain the area under the curve (AUC) for false-positive rate v.s. true-positive rate.
Revised: Then we obtain the area under the curve (AUC) for false-positive rate v.s. true-positive rate.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure][Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]