Original: GO G RADIENT FOR E XPECTATION -B ASED O BJECTIVES
Revised: GO G RADIENT FOR E XPECTATION -B ASED O BJECTIVES

Original: A BSTRACT
Revised: A BSTRACT

Original: Within many machine learning algorithms, a fundamental problem concerns efﬁcient calculation of an unbiased gradient wrt parameters γ for expectation-based objectives E q γ ( y )
Revised: Within many machine learning algorithms, a fundamental problem concerns efﬁcient calculation of an unbiased gradient wrt parameters γ for expectation-based objectives E q γ ( y )

Original: [ f ( y )] .
Revised: [ f ( y )] .

Original: Most existing methods either ( i ) suffer from high variance, seeking help from (often) complicated variance-reduction techniques; or ( ii ) they only apply to reparameterizable continuous random variables and employ a reparameterization trick.
Revised: Most existing methods either ( i ) suffer from high variance, seeking help from (often) complicated variance-reduction techniques; or ( ii ) they only apply to reparameterizable continuous random variables and employ a reparameterization trick.

Original: To address these limitations, we propose a General and
Revised: To address these limitations, we propose a General and

Original: One-sample (GO) gradient that ( i ) applies to many distributions associated with non-reparameterizable continuous or discrete random variables, and ( ii ) has the same low-variance as the reparameterization trick.
Revised: One-sample (GO) gradient that ( i ) applies to many distributions associated with non-reparameterizable continuous or discrete random variables, and ( ii ) has the same low-variance as the reparameterization trick.

Original: We ﬁnd that the GO gradient often works well in practice based on only one Monte Carlo sample (although one can of course use more samples if desired).
Revised: We ﬁnd that the GO gradient often works well in practice based on only one Monte Carlo sample (although one can of course use more samples if desired).

Original: Alongside the GO gradient, we develop a means of propagating the chain rule through distributions, yielding statistical back-propagation, coupling neural networks to common random variables.
Revised: Alongside the GO gradient, we develop a means of propagating the chain rule through distributions, yielding statistical back-propagation, coupling neural networks to common random variables.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Neural networks, typically trained using back-propagation for parameter optimization, have recently demonstrated signiﬁcant success across a wide range of applications.
Revised: Neural networks, typically trained using back-propagation for parameter optimization, have recently demonstrated signiﬁcant success across a wide range of applications.

Original: There has been interest in coupling neural networks with random variables, so as to embrace greater descriptive capacity.
Revised: There has been interest in coupling neural networks with random variables, so as to embrace greater descriptive capacity.

Original: Recent examples of this include black-box variational inference (BBVI) (Kingma & Welling, 2014; Rezende et al., 2014;
Revised: Recent examples of this include black-box variational inference (BBVI) (Kingma & Welling, 2014; Rezende et al., 2014;

Original: Ranganath et al., 2014; Hernández-Lobato et al., 2016; Ranganath et al., 2016b; Li & Turner, 2016; Ranganath et al., 2016a;
Revised: Ranganath et al., 2014; Hernández-Lobato et al., 2016; Ranganath et al., 2016b; Li & Turner, 2016; Ranganath et al., 2016a;

Original: Zhang et al., 2018) and generative adversarial networks (GANs) (Goodfellow et al., 2014;
Revised: Zhang et al., 2018) and generative adversarial networks (GANs) (Goodfellow et al., 2014;

Original: Radford et al., 2015; Zhao et al., 2016;
Revised: Radford et al., 2015; Zhao et al., 2016;

Original: Arjovsky et al., 2017; Li et al., 2017; Gan et al., 2017; Li et al., 2018).
Revised: Arjovsky et al., 2017; Li et al., 2017; Gan et al., 2017; Li et al., 2018).

Original: Unfortunately, efﬁciently backpropagating gradients through general distributions (random variables) remains a bottleneck.
Revised: Unfortunately, efﬁciently backpropagating gradients through general distributions (random variables) remains a bottleneck.

Original: Most current methodology focuses on distributions with continuous random variables, for which the reparameterization trick may be readily applied (Kingma & Welling, 2014; Grathwohl et al., 2017).
Revised: Most current methodology focuses on distributions with continuous random variables, for which the reparameterization trick may be readily applied (Kingma & Welling, 2014; Grathwohl et al., 2017).

Original: As an example, the aforementioned bottleneck greatly constrains the applicability of BBVI, by limiting variational approximations to reparameterizable distributions.
Revised: As an example, the aforementioned bottleneck greatly constrains the applicability of BBVI, by limiting variational approximations to reparameterizable distributions.

Original: This limitation excludes discrete random variables and many types of continuous ones.
Revised: This limitation excludes discrete random variables and many types of continuous ones.

Original: From the perspective of GAN, the need to employ reparameterization has constrained most applications to continuous observations.
Revised: From the perspective of GAN, the need to employ reparameterization has constrained most applications to continuous observations.

Original: There are many forms of data that are more-naturally discrete.
Revised: There are many forms of data that are more-naturally discrete.

Original: The fundamental problem associated with the aforementioned challenges is the need to efﬁciently calculate an unbiased low-variance gradient wrt parameters γ for an expectation objective of the form E q γ ( y ) [ f ( y )] 1 .
Revised: The fundamental problem associated with the aforementioned challenges is the need to efﬁciently calculate an unbiased low-variance gradient wrt parameters γ for an expectation objective of the form E q γ ( y ) [ f ( y )] 1 .

Original: We are interested in general distributions q γ ( y ) , for which the components of y may be either continuous or discrete.
Revised: We are interested in general distributions q γ ( y ) , for which the components of y may be either continuous or discrete.

Original: Typically the components of y have a hierarchical structure, and a subset of the components of y play a role in evaluating f ( y ) .
Revised: Typically the components of y have a hierarchical structure, and a subset of the components of y play a role in evaluating f ( y ) .

Original: Unfortunately, classical methods for estimating gradients of E q γ ( y )
Revised: Unfortunately, classical methods for estimating gradients of E q γ ( y )

Original: [ f ( y )]
Revised: [ f ( y )]

Original: wrt γ have limitations.
Revised: wrt γ have limitations.

Original: The REINFORCE gradient (Williams, 1992), although generally applicable ( e.g. , for continuous and discrete random variables), exhibits high variance with Monte Carlo (MC) estimation of the expectation, forcing one to apply additional variance-reduction techniques.
Revised: The REINFORCE gradient (Williams, 1992), although generally applicable ( e.g. , for continuous and discrete random variables), exhibits high variance with Monte Carlo (MC) estimation of the expectation, forcing one to apply additional variance-reduction techniques.

Original: The reparameterization trick (Rep) (Salimans et al., 2013; Kingma & Welling, 2014; Rezende et al., 2014) works well, with as few as only one MC sample, but it is limited to continuous reparameterizable y .
Revised: The reparameterization trick (Rep) (Salimans et al., 2013; Kingma & Welling, 2014; Rezende et al., 2014) works well, with as few as only one MC sample, but it is limited to continuous reparameterizable y .

Original: Many efforts have been devoted to improving these two formulations, as detailed in Section 6.
Revised: Many efforts have been devoted to improving these two formulations, as detailed in Section 6.

Original: However, none of these methods is characterized by generalization (applicable to general distributions) and efﬁciency (working well with as few as one MC sample).
Revised: However, none of these methods is characterized by generalization (applicable to general distributions) and efﬁciency (working well with as few as one MC sample).

Original: The key contributions of this work are based on the recognition that REINFORCE and Rep are seeking to solve the same objective, but in practice Rep yields lower-variance estimations, albeit for a narrower class of distributions.
Revised: The key contributions of this work are based on the recognition that REINFORCE and Rep are seeking to solve the same objective, but in practice Rep yields lower-variance estimations, albeit for a narrower class of distributions.

Original: Recent work (Ranganath et al., 2016b) has made a connection between
Revised: Recent work (Ranganath et al., 2016b) has made a connection between

Original: REINFORCE and Rep, recognizing that the former estimates a term the latter evaluates analytically.
Revised: REINFORCE and Rep, recognizing that the former estimates a term the latter evaluates analytically.

Original: The high variance by which REINFORCE approximates this term manifests high variance in the gradient estimation.
Revised: The high variance by which REINFORCE approximates this term manifests high variance in the gradient estimation.

Original: Extending these ideas, we make the following main contributions.
Revised: Extending these ideas, we make the following main contributions.

Original: ( i )
Revised: ( i )

Original: We propose a new General and One-sample (GO) gradient in Section 3, that principally generalizes Rep to many non-reparameterizable distributions and justiﬁes two recent methods (Figurnov et al., 2018;
Revised: We propose a new General and One-sample (GO) gradient in Section 3, that principally generalizes Rep to many non-reparameterizable distributions and justiﬁes two recent methods (Figurnov et al., 2018;

Original: Jankowiak & Obermeyer, 2018); the “One sample” motivating the name GO is meant to highlight the low variance of the proposed method, although of course one may use more than one sample if desired.
Revised: Jankowiak & Obermeyer, 2018); the “One sample” motivating the name GO is meant to highlight the low variance of the proposed method, although of course one may use more than one sample if desired.

Original: ( ii ) We ﬁnd that the core of the GO gradient is something we term a variable-nabla , which can be interpreted as the gradient of a random variable wrt a parameter.
Revised: ( ii ) We ﬁnd that the core of the GO gradient is something we term a variable-nabla , which can be interpreted as the gradient of a random variable wrt a parameter.

Original: ( iii ) Utilizing variablenablas to propagate the chain rule through distributions, we broaden the applicability of the GO gradient in Sections 4-5 and present statistical back-propagation, a statistical generalization of classic back-propagation (Rumelhart & Hinton, 1986).
Revised: ( iii ) Utilizing variablenablas to propagate the chain rule through distributions, we broaden the applicability of the GO gradient in Sections 4-5 and present statistical back-propagation, a statistical generalization of classic back-propagation (Rumelhart & Hinton, 1986).

Original: Through this generalization, we may couple neural networks to general random variables, and compute needed gradients with low variance.
Revised: Through this generalization, we may couple neural networks to general random variables, and compute needed gradients with low variance.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: To motivate this paper, we begin by brieﬂy elucidating common machine learning problems for which there is a need to efﬁciently estimate gradients of γ for functions of the form E q γ ( y )
Revised: To motivate this paper, we begin by brieﬂy elucidating common machine learning problems for which there is a need to efﬁciently estimate gradients of γ for functions of the form E q γ ( y )

Original: [ f ( y )] .
Revised: [ f ( y )] .

Original: Assume access to data samples { x i } i =1 ,N , drawn i.i.d.
Revised: Assume access to data samples { x i } i =1 ,N , drawn i.i.d.

Original: from the true (and unknown) underlying distribution q ( x ) .
Revised: from the true (and unknown) underlying distribution q ( x ) .

Original: We seek to learn a model p θ ( x ) to approximate q ( x ) .
Revised: We seek to learn a model p θ ( x ) to approximate q ( x ) .

Original: A classic approach to such learning is to maximize the expected log likelihood ˆ θ = argmax θ E q ( x ) [log p θ ( x )] , perhaps with an added regularization term on θ .
Revised: A classic approach to such learning is to maximize the expected log likelihood ˆ θ = argmax θ E q ( x ) [log p θ ( x )] , perhaps with an added regularization term on θ .

Original: Expectation E q ( x ) ( · ) is approximated via the available data samples, as
Revised: Expectation E q ( x ) ( · ) is approximated via the available data samples, as

Original: [Equation]
Revised: [Equation]

Original: It is often convenient to employ a model with latent variables z , i.e. , p
Revised: It is often convenient to employ a model with latent variables z , i.e. , p

Original: [Equation]
Revised: [Equation]

Original: p θ ( x | z ) p ( z ) d z , with prior p ( z ) on z .
Revised: p θ ( x | z ) p ( z ) d z , with prior p ( z ) on z .

Original: The integral wrt z is typically intractable, motivating introduction of the approximate posterior q φ ( z | x ) , with parameters φ .
Revised: The integral wrt z is typically intractable, motivating introduction of the approximate posterior q φ ( z | x ) , with parameters φ .

Original: The well-known evidence lower bound (ELBO) (Jordan et al., 1999; Bishop, 2006) is deﬁned as
Revised: The well-known evidence lower bound (ELBO) (Jordan et al., 1999; Bishop, 2006) is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where p θ ( z | x ) is the true posterior, and KL ( ·(cid:107)· ) represents the Kullback-Leibler divergence.
Revised: where p θ ( z | x ) is the true posterior, and KL ( ·(cid:107)· ) represents the Kullback-Leibler divergence.

Original: Variational learning seeks (ˆ θ , ˆ φ ) =
Revised: Variational learning seeks (ˆ θ , ˆ φ ) =

Original: argmax θ , φ (cid:80) Ni =1 ELBO ( θ , φ ; x i ) .
Revised: argmax θ , φ (cid:80) Ni =1 ELBO ( θ , φ ; x i ) .

Original: While computation of the ELBO has been considered for many years, a problem introduced recently concerns adversarial learning of p θ ( x ) , or, more precisely, learning a model that allows one to efﬁciently and accurately draw samples x ∼ p θ ( x ) that are similar to x ∼ q ( x ) .
Revised: While computation of the ELBO has been considered for many years, a problem introduced recently concerns adversarial learning of p θ ( x ) , or, more precisely, learning a model that allows one to efﬁciently and accurately draw samples x ∼ p θ ( x ) that are similar to x ∼ q ( x ) .

Original: With generative adversarial networks (GANs) (Goodfellow et al., 2014), one seeks to solve
Revised: With generative adversarial networks (GANs) (Goodfellow et al., 2014), one seeks to solve

Original: [Equation]
Revised: [Equation]

Original: where D β ( x ) is a discriminator with parameters β , quantifying the probability x was drawn from q ( x ) , with 1 − D β ( x ) representing the probability that it was drawn from p θ ( x ) .
Revised: where D β ( x ) is a discriminator with parameters β , quantifying the probability x was drawn from q ( x ) , with 1 − D β ( x ) representing the probability that it was drawn from p θ ( x ) .

Original: There have been many recent extensions of GAN (Radford et al., 2015; Zhao et al., 2016;
Revised: There have been many recent extensions of GAN (Radford et al., 2015; Zhao et al., 2016;

Original: Arjovsky et al., 2017; Li et al., 2017; Gan et al., 2017; Li et al., 2018), but the basic setup in (3) holds for most.
Revised: Arjovsky et al., 2017; Li et al., 2017; Gan et al., 2017; Li et al., 2018), but the basic setup in (3) holds for most.

Original: To optimize (1) and (3), the most challenging gradients that must be computed are of the form ∇ γ E q γ ( y )
Revised: To optimize (1) and (3), the most challenging gradients that must be computed are of the form ∇ γ E q γ ( y )

Original: [ f ( y )] ; for (1)
Revised: [ f ( y )] ; for (1)

Original: y =
Revised: y =

Original: z and γ = φ , while for (3) y
Revised: z and γ = φ , while for (3) y

Original: = x and γ = θ .
Revised: = x and γ = θ .

Original: The need to evaluate expressions like ∇ γ E q γ ( y )
Revised: The need to evaluate expressions like ∇ γ E q γ ( y )

Original: [ f ( y )] arises in many other machine learning problems, and consequently it has generated much prior attention.
Revised: [ f ( y )] arises in many other machine learning problems, and consequently it has generated much prior attention.

Original: Evaluation of the gradient of the expectation is simpliﬁed if the gradient can be moved inside the expectation.
Revised: Evaluation of the gradient of the expectation is simpliﬁed if the gradient can be moved inside the expectation.

Original: REINFORCE (Williams, 1992) is based on the identity
Revised: REINFORCE (Williams, 1992) is based on the identity

Original: [Equation]
Revised: [Equation]

Original: While simple in concept, this estimate is known to have high variance when the expectation E q γ ( y ) ( · ) is approximated (as needed in practice) by a ﬁnite number of samples.
Revised: While simple in concept, this estimate is known to have high variance when the expectation E q γ ( y ) ( · ) is approximated (as needed in practice) by a ﬁnite number of samples.

Original: An approach (Salimans et al., 2013; Kingma & Welling, 2014; Rezende et al., 2014) that has attracted recent attention is termed the reparameterization trick, applicable when q γ ( y ) can be reparametrized as y =
Revised: An approach (Salimans et al., 2013; Kingma & Welling, 2014; Rezende et al., 2014) that has attracted recent attention is termed the reparameterization trick, applicable when q γ ( y ) can be reparametrized as y =

Original: τ γ ( (cid:15) ) , with (cid:15) ∼ q ( (cid:15) ) , where τ γ ( (cid:15) ) is a differentiable transformation and q ( (cid:15) ) is a simple distribution that may be readily sampled.
Revised: τ γ ( (cid:15) ) , with (cid:15) ∼ q ( (cid:15) ) , where τ γ ( (cid:15) ) is a differentiable transformation and q ( (cid:15) ) is a simple distribution that may be readily sampled.

Original: To keep consistency with the literature, we call a distribution reparameterizable if and only if those conditions are satisﬁed 2 .
Revised: To keep consistency with the literature, we call a distribution reparameterizable if and only if those conditions are satisﬁed 2 .

Original: In this case we have
Revised: In this case we have

Original: [Equation]
Revised: [Equation]

Original: This gradient, termed Rep, is typically characterized by relatively low variance, when approximating E q ( (cid:15) ) ( · ) with a small number of samples (cid:15) ∼ q ( (cid:15) ) .
Revised: This gradient, termed Rep, is typically characterized by relatively low variance, when approximating E q ( (cid:15) ) ( · ) with a small number of samples (cid:15) ∼ q ( (cid:15) ) .

Original: This approach has been widely employed for computation of the ELBO and within GAN, but it limits one to models that satisfy the assumptions of
Revised: This approach has been widely employed for computation of the ELBO and within GAN, but it limits one to models that satisfy the assumptions of

Original: Rep.
Revised: Rep.

Original:  GO G RADIENT
Revised:  GO G RADIENT

Original: The reparameterization trick (Rep) is limited to reparameterizable random variables y with continuous components.
Revised: The reparameterization trick (Rep) is limited to reparameterizable random variables y with continuous components.

Original: There are situations for which Rep is not readily applicable, e.g., where the components of y may be discrete or nonnegative Gamma distributed.
Revised: There are situations for which Rep is not readily applicable, e.g., where the components of y may be discrete or nonnegative Gamma distributed.

Original: We seek to gain insights from the relationship between REINFORCE and Rep, and generalize the types of random variables y for which the latter approach may be effected.
Revised: We seek to gain insights from the relationship between REINFORCE and Rep, and generalize the types of random variables y for which the latter approach may be effected.

Original: We term our proposed approach a General and One-sample (GO) gradient.
Revised: We term our proposed approach a General and One-sample (GO) gradient.

Original: In practice, we ﬁnd that this approach works well with as few as one sample for evaluating the expectation, and it is applicable to more general settings than Rep.
Revised: In practice, we ﬁnd that this approach works well with as few as one sample for evaluating the expectation, and it is applicable to more general settings than Rep.

Original: Recall that Rep was ﬁrst applied within the context of variational learning (Kingma & Welling, 2014), as in (1).
Revised: Recall that Rep was ﬁrst applied within the context of variational learning (Kingma & Welling, 2014), as in (1).

Original: Speciﬁcally, it was assumed q γ ( y ) =
Revised: Speciﬁcally, it was assumed q γ ( y ) =

Original: (cid:81) v q γ ( y v ) , omitting explicit dependence on data x , for notational convenience; y v is component v of y .
Revised: (cid:81) v q γ ( y v ) , omitting explicit dependence on data x , for notational convenience; y v is component v of y .

Original: In Kingma & Welling (2014) q γ ( y v ) corresponded to a Gaussian distribution q γ ( y v ) =
Revised: In Kingma & Welling (2014) q γ ( y v ) corresponded to a Gaussian distribution q γ ( y v ) =

Original: N ( y v ; µ v ( γ ) , σ 2 v ( γ )) , with mean µ v ( γ ) and variance σ 2 v ( γ ) .
Revised: N ( y v ; µ v ( γ ) , σ 2 v ( γ )) , with mean µ v ( γ ) and variance σ 2 v ( γ ) .

Original: In the following we generalize q γ ( y v ) such that it need not be Gaussian.
Revised: In the following we generalize q γ ( y v ) such that it need not be Gaussian.

Original: Applying integration by parts (Ranganath et al., 2016b)
Revised: Applying integration by parts (Ranganath et al., 2016b)

Original: [Equation]
Revised: [Equation]

Original: where y − v denotes y with y v excluded, and Q γ ( y v ) is the cumulative distribution function (CDF) of q γ ( y v ) .
Revised: where y − v denotes y with y v excluded, and Q γ ( y v ) is the cumulative distribution function (CDF) of q γ ( y v ) .

Original: The “0” term is readily proven to be zero for any Q γ ( y v ) , with the assumption that f ( y ) doesn’t tend to inﬁnity faster than ∇ γ Q γ ( y v ) tending to zero when y v → ±∞ .
Revised: The “0” term is readily proven to be zero for any Q γ ( y v ) , with the assumption that f ( y ) doesn’t tend to inﬁnity faster than ∇ γ Q γ ( y v ) tending to zero when y v → ±∞ .

Original: The “Key” term exactly recovers the one-dimensional Rep when reparameterization y v = τ γ ( (cid:15) v ) , (cid:15) v ∼ q ( (cid:15) v ) exists (Ranganath et al., 2016b).
Revised: The “Key” term exactly recovers the one-dimensional Rep when reparameterization y v = τ γ ( (cid:15) v ) , (cid:15) v ∼ q ( (cid:15) v ) exists (Ranganath et al., 2016b).

Original: Further, applying ∇ γ q γ ( y v ) =
Revised: Further, applying ∇ γ q γ ( y v ) =

Original: q γ ( y v ) ∇ γ log q γ ( y v ) in (6) yields REINFORCE.
Revised: q γ ( y v ) ∇ γ log q γ ( y v ) in (6) yields REINFORCE.

Original: Consequently, it appears that Rep yields low variance by analytically setting to zero the unnecessary but high-variance-injecting “0” term, while in contrast REINFORCE implicitly seeks to numerically implement both terms in (7).
Revised: Consequently, it appears that Rep yields low variance by analytically setting to zero the unnecessary but high-variance-injecting “0” term, while in contrast REINFORCE implicitly seeks to numerically implement both terms in (7).

Original: We generalize q γ ( y ) for discrete y v , here assuming y v ∈ { 0 , 1 , . . .
Revised: We generalize q γ ( y ) for discrete y v , here assuming y v ∈ { 0 , 1 , . . .

Original: , ∞} .
Revised: , ∞} .

Original: It is shown in Appendix
Revised: It is shown in Appendix

Original: A.2 that this framework is also applicable to discrete y v with a ﬁnite alphabet.
Revised: A.2 that this framework is also applicable to discrete y v with a ﬁnite alphabet.

Original: It may be shown (see
Revised: It may be shown (see

Original: Appendix A.2)
Revised: Appendix A.2)

Original: that
Revised: that

Original: [Equation]
Revised: [Equation]

Original:  where Q γ ( y v )=
Revised: 1 (cid:15)<P with (cid:15) ∼ U (0 , 1) . But y is called non-reparameterizable because the transformation y = is not differentiable where Q γ ( y v )

Original:  (cid:80) y v n =0 q γ ( n ) , and Q γ ( ∞ )
Revised: = (cid:80) y v n =0 q γ ( n ) , and Q γ ( ∞ )

Original: = 1 for all γ .
Revised: = 1 for all γ .

Original: Theorem 1 ( GO Gradient ) .
Revised: Theorem 1 ( GO Gradient ) .

Original: For expectation objectives E q γ ( y ) [ f ( y )] , where q γ ( y ) satisﬁes ( i ) q γ ( y ) =
Revised: For expectation objectives E q γ ( y ) [ f ( y )] , where q γ ( y ) satisﬁes ( i ) q γ ( y ) =

Original: (cid:81) v q γ ( y v ) ; ( ii ) the corresponding CDF Q γ ( y v ) is differentiable wrt parameters γ ; and ( iii ) one can calculate ∇ γ Q γ ( y v ) , the General and One-sample (GO) gradient is deﬁned as
Revised: (cid:81) v q γ ( y v ) ; ( ii ) the corresponding CDF Q γ ( y v ) is differentiable wrt parameters γ ; and ( iii ) one can calculate ∇ γ Q γ ( y v ) , the General and One-sample (GO) gradient is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where G q γ ( y ) γ speciﬁes G q ( y ) κ
Revised: where G q γ ( y ) γ speciﬁes G q ( y ) κ

Original: [Equation]
Revised: [Equation]

Original: · ·  , g 
Revised: · · · , g q

Original: [Equation]
Revised: [Equation]

Original: ) κ ,
Revised: ) κ ,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: with variable-nabla g q ( y v ) κ
Revised: with variable-nabla g q ( y v ) κ

Original: [Equation]
Revised: [Equation]

Original: Q
Revised: Q

Original: [Equation]
Revised: [Equation]

Original: D
Revised: D

Original: [Equation]
Revised: [Equation]

Original: f
Revised: f

Original: [Equation]
Revised: [Equation]

Original: y )]
Revised: y )]

Original: = (cid:2) · · · , D y v [ f ( y )] , · · · (cid:3) T , and
Revised: = (cid:2) · · · , D y v [ f ( y )] , · · · (cid:3) T , and

Original: [Equation]
Revised: [Equation]

Original: Continousy
Revised: Continous

Original: [Equation]
Revised: [Equation]

Original: Discrete y
Revised: Discrete y

Original: [Equation]
Revised: [Equation]

Original: All proofs are provided in Appendix A, where we also list g q γ ( y v ) γ for a wide selection of possible q γ ( y ) , for both continuous and discrete y .
Revised: All proofs are provided in Appendix A, where we also list g q γ ( y v ) γ for a wide selection of possible q γ ( y ) , for both continuous and discrete y .

Original: Note for the special case with continuous y , GO reduces to Implicit Rep gradients (Figurnov et al., 2018) and pathwise derivatives (Jankowiak & Obermeyer, 2018); in other words, GO provides a principled explanation for their low variance, namely their foundation (implicit differentiation) originates from integration by parts.
Revised: Note for the special case with continuous y , GO reduces to Implicit Rep gradients (Figurnov et al., 2018) and pathwise derivatives (Jankowiak & Obermeyer, 2018); in other words, GO provides a principled explanation for their low variance, namely their foundation (implicit differentiation) originates from integration by parts.

Original: For high-dimensional discrete y , calculating D y [ f ( y )] is computationally expensive.
Revised: For high-dimensional discrete y , calculating D y [ f ( y )] is computationally expensive.

Original: Fortunately, for f ( y ) often used in practice special properties hold that can be exploited for efﬁcient parallel computing.
Revised: Fortunately, for f ( y ) often used in practice special properties hold that can be exploited for efﬁcient parallel computing.

Original: Also for discrete y v with ﬁnite support, it is possible that one could analytically evaluate a part of expectations in (9) for lower variance, mimicking the local idea in Titsias & Lázaro-Gredilla (2015);
Revised: Also for discrete y v with ﬁnite support, it is possible that one could analytically evaluate a part of expectations in (9) for lower variance, mimicking the local idea in Titsias & Lázaro-Gredilla (2015);

Original: Titsias (2015).
Revised: Titsias (2015).

Original: Appendix I shows an example illustrating how to handle these two issues in practice.
Revised: Appendix I shows an example illustrating how to handle these two issues in practice.

Original:  D EEP GO G RADIENT
Revised:  D EEP GO G RADIENT

Original: The GO gradient in Theorem 1 can only handle single-layer mean-ﬁeld q γ ( y ) , characterized by an independence assumption on the components of y .
Revised: The GO gradient in Theorem 1 can only handle single-layer mean-ﬁeld q γ ( y ) , characterized by an independence assumption on the components of y .

Original: One may enlarge the descriptive capability of q γ ( y ) by modeling it as a marginal distribution of a deep model (Ranganath et al., 2016b; Bishop, 2006).
Revised: One may enlarge the descriptive capability of q γ ( y ) by modeling it as a marginal distribution of a deep model (Ranganath et al., 2016b; Bishop, 2006).

Original: Hereafter, we focus on this situation, and begin with a 2-layer model for simple demonstration.
Revised: Hereafter, we focus on this situation, and begin with a 2-layer model for simple demonstration.

Original: Speciﬁcally, consider
Revised: Speciﬁcally, consider

Original: [Equation]
Revised: [Equation]

Original: where γ = { γ y , γ λ } , y is the leaf variable, and the internal variable λ is assumed to be continuous.
Revised: where γ = { γ y , γ λ } , y is the leaf variable, and the internal variable λ is assumed to be continuous.

Original: Components of y are assumed to be conditionally independent given λ , but upon marginalizing out λ this independence is removed.
Revised: Components of y are assumed to be conditionally independent given λ , but upon marginalizing out λ this independence is removed.

Original: The objective becomes E q γ ( y )
Revised: The objective becomes E q γ ( y )

Original: [ f ( y )]
Revised: [ f ( y )]

Original: =
Revised: =

Original: E q γy ( y | λ ) q γλ ( λ ) [ f ( y )] , and via Theorem
Revised: E q γy ( y | λ ) q γλ ( λ ) [ f ( y )] , and via Theorem

Original: [Equation]
Revised: [Equation]

Original: Lemma 1.
Revised: Lemma 1.

Original: Equation (10) exactly recovers the Rep gradient in (5) , if γ = γ y and q γ ( y ) has reparameterization y =
Revised: Equation (10) exactly recovers the Rep gradient in (5) , if γ = γ y and q γ ( y ) has reparameterization y =

Original: τ γ ( λ ) , λ ∼ q ( λ ) for differentiable τ γ ( λ ) and easily sampled q ( λ ) .
Revised: τ γ ( λ ) , λ ∼ q ( λ ) for differentiable τ γ ( λ ) and easily sampled q ( λ ) .

Original: Lemma 1 shows that Rep is a special case of our deep GO gradient in the following Theorem 2.
Revised: Lemma 1 shows that Rep is a special case of our deep GO gradient in the following Theorem 2.

Original: Note neither Implicit Rep gradients nor pathwise derivatives can recover Rep in general, because a neural-network-parameterized y =
Revised: Note neither Implicit Rep gradients nor pathwise derivatives can recover Rep in general, because a neural-network-parameterized y =

Original: τ γ ( λ ) may lead to non-trivial CDF Q γ ( y ) .
Revised: τ γ ( λ ) may lead to non-trivial CDF Q γ ( y ) .

Original: For the gradient wrt γ λ , we ﬁrst apply Theorem 1, yielding
Revised: For the gradient wrt γ λ , we ﬁrst apply Theorem 1, yielding

Original: [Equation]
Revised: [Equation]

Original: For continuous internal variable λ one can apply Theorem 1 again, from which
Revised: For continuous internal variable λ one can apply Theorem 1 again, from which

Original: [Equation]
Revised: [Equation]

Original: Now extending the same procedure to deeper models with L layers, we generalize the GO gradient in Theorem 2.
Revised: Now extending the same procedure to deeper models with L layers, we generalize the GO gradient in Theorem 2.

Original: Random variable y
Revised: Random variable y

Original: ( L ) is assumed to be the leaf variable of interest, and may be continuous or discrete; latent/internal random variables { y (1) , . . .
Revised: ( L ) is assumed to be the leaf variable of interest, and may be continuous or discrete; latent/internal random variables { y (1) , . . .

Original: , y ( L − 1) } are assumed continuous (these generalize λ from above).
Revised: , y ( L − 1) } are assumed continuous (these generalize λ from above).

Original: Theorem 2 ( Deep GO Gradient ) .
Revised: Theorem 2 ( Deep GO Gradient ) .

Original: For expectation objectives E q γ ( y ( L ) )
Revised: For expectation objectives E q γ ( y ( L ) )

Original: [ f ( y ( L ) )] with q γ ( y ( L ) ) being the marginal distribution of
Revised: [ f ( y ( L ) )] with q γ ( y ( L ) ) being the marginal distribution of

Original: [Equation]
Revised: [Equation]

Original: where γ = { γ (1) , · · · , γ ( L ) } , all internal variables y (1) , · · · , y ( L − 1) are continuous, the leaf variable
Revised: where γ = { γ (1) , · · · , γ ( L ) } , all internal variables y (1) , · · · , y ( L − 1) are continuous, the leaf variable

Original: y
Revised: y

Original: ( L ) is either continuous or discrete,
Revised: ( L ) is either continuous or discrete,

Original: q γ ( l ) ( y ( l ) | y
Revised: q γ ( l ) ( y ( l ) | y

Original: ( l − 1) )
Revised: ( l − 1) )

Original: = (cid:81) v q γ ( l ) ( y ( l ) v | y ( l − 1) ) , and one has access to variable-nablas g q γ ( l ) ( y ( l ) v | y ( l − 1) )
Revised: = (cid:81) v q γ ( l ) ( y ( l ) v | y ( l − 1) ) , and one has access to variable-nablas g q γ ( l ) ( y ( l ) v | y ( l − 1) )

Original: γ ( l ) and g q γ ( l ) ( y ( l ) v | y ( l − 1) )
Revised: γ ( l ) and g q γ ( l ) ( y ( l ) v | y ( l − 1) )

Original: y ( l − 1) , as deﬁned in Theorem 1, the
Revised: y ( l − 1) , as deﬁned in Theorem 1, the

Original: General and One-sample (GO) gradient is deﬁned as
Revised: General and One-sample (GO) gradient is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: , (12) where BP [ y ( L ) ]
Revised: , (12) where BP [ y ( L ) ]

Original: =
Revised: =

Original: D y
Revised: D y

Original: ( L )
Revised: ( L )

Original: [ f ( y ( L ) )] and BP [ y ( l ) ]
Revised: [ f ( y ( L ) )] and BP [ y ( l ) ]

Original: =
Revised: =

Original: G q γ ( l +1)
Revised: G q γ ( l +1)

Original: ( y ( l +1)
Revised: ( y ( l +1)

Original: | y ( l ) )
Revised: | y ( l ) )

Original: y ( l )
Revised: y ( l )

Original: BP [ y ( l +1) ] for l < L .
Revised: BP [ y ( l +1) ] for l < L .

Original: Corollary 1.
Revised: Corollary 1.

Original: The deep GO gradient in Theorem 2 exactly recovers back-propagation (Rumelhart & Hinton, 1986) when each element distribution q γ ( l ) ( y ( l ) v | y ( l − 1) ) is speciﬁed as the Dirac delta function located at the activated value after activation function.
Revised: The deep GO gradient in Theorem 2 exactly recovers back-propagation (Rumelhart & Hinton, 1986) when each element distribution q γ ( l ) ( y ( l ) v | y ( l − 1) ) is speciﬁed as the Dirac delta function located at the activated value after activation function.

Original: Figure 1 is presented for better understanding.
Revised: Figure 1 is presented for better understanding.

Original: With variable-nablas, one can readily verify that the deep GO gradient in Theorem 2 in expectation obeys the chain rule.
Revised: With variable-nablas, one can readily verify that the deep GO gradient in Theorem 2 in expectation obeys the chain rule.

Original: Assume all continuous variables y , and abuse notations for better understanding.
Revised: Assume all continuous variables y , and abuse notations for better understanding.

Original: [Equation]
Revised: [Equation]

Original:  S TATISTICAL B ACK -P ROPAGATION AND H IERARCHICAL V ARIATIONAL I NFERENCE
Revised:  S TATISTICAL B ACK -P ROPAGATION AND H IERARCHICAL V ARIATIONAL I NFERENCE

Original: Recall the motivating discussion in Section 2, in which we considered generative model p θ ( x , z ) and inference model q φ ( z | x ) , the former used to model synthesis of the observed data x and the latter used for inference of z given observed x .
Revised: Recall the motivating discussion in Section 2, in which we considered generative model p θ ( x , z ) and inference model q φ ( z | x ) , the former used to model synthesis of the observed data x and the latter used for inference of z given observed x .

Original: In recent deep architectures, a hierarchical representation for cumulative latent variables z =
Revised: In recent deep architectures, a hierarchical representation for cumulative latent variables z =

Original: ( z (1) , . . .
Revised: ( z (1) , . . .

Original: , z ( L ) ) has been considered (Rezende et al., 2014;
Revised: , z ( L ) ) has been considered (Rezende et al., 2014;

Original: Ranganath et al., 2015; Zhou et al., 2015; 2016; Ranganath et al., 2016b; Cong et al., 2017; Zhang et al., 2018).
Revised: Ranganath et al., 2015; Zhou et al., 2015; 2016; Ranganath et al., 2016b; Cong et al., 2017; Zhang et al., 2018).

Original: As an example, there are models with p θ ( x , z )
Revised: As an example, there are models with p θ ( x , z )

Original: =
Revised: =

Original: p θ ( x | z (1) ) (cid:2) (cid:81)
Revised: p θ ( x | z (1) ) (cid:2) (cid:81)

Original: L − 1 l =1 p θ ( z ( l ) | z ( l +1) )
Revised: L − 1 l =1 p θ ( z ( l ) | z ( l +1) )

Original: (cid:3) p θ ( z ( L ) ) .
Revised: (cid:3) p θ ( z ( L ) ) .

Original: When performing inference for such models, it is intuitive to consider ﬁrst-order Markov chain structure for q φ ( z | x ) =
Revised: When performing inference for such models, it is intuitive to consider ﬁrst-order Markov chain structure for q φ ( z | x ) =

Original: q φ ( z (1) | x ) (cid:81)
Revised: q φ ( z (1) | x ) (cid:81)

Original: L l =2 q φ ( z ( l )
Revised: L l =2 q φ ( z ( l )

Original: | z ( l − 1) ) .
Revised: | z ( l − 1) ) .

Original: The discussion in this section is most relevant for variational inference, for computation of ∇ φ E q φ ( z (1) ,..., z ( L ) | x ) [ f ( z (1) , . .
Revised: The discussion in this section is most relevant for variational inference, for computation of ∇ φ E q φ ( z (1) ,..., z ( L ) | x ) [ f ( z (1) , . .

Original: , z ( L ) )] , and consequently we specialize to that notation in the subsequent discussion (we consider representations in terms of z , rather than the more general y notation employed in Section 4).
Revised: , z ( L ) )] , and consequently we specialize to that notation in the subsequent discussion (we consider representations in terms of z , rather than the more general y notation employed in Section 4).

Original: Before proceeding, we seek to make clear the distinction between this section and Section 4.
Revised: Before proceeding, we seek to make clear the distinction between this section and Section 4.

Original: In the latter, only the leaf variable z ( L ) appears in ∇ γ E q γ ( z ( L ) )
Revised: In the latter, only the leaf variable z ( L ) appears in ∇ γ E q γ ( z ( L ) )

Original: [ f ( z ( L ) )] ; see (12), with y ( L ) →
Revised: [ f ( z ( L ) )] ; see (12), with y ( L ) →

Original: z ( L ) .
Revised: z ( L ) .

Original: That is because in Section 4 the underlying model is a marginal distribution of z ( L ) , i.e. , q γ ( z ( L ) ) , which is relevant to the generators of GANs; see (3), with x → z ( L ) and p θ ( x ) → q γ ( z ( L ) ) .
Revised: That is because in Section 4 the underlying model is a marginal distribution of z ( L ) , i.e. , q γ ( z ( L ) ) , which is relevant to the generators of GANs; see (3), with x → z ( L ) and p θ ( x ) → q γ ( z ( L ) ) .

Original: Random variables z (1) , . . .
Revised: Random variables z (1) , . . .

Original: , z ( L − 1) were marginalized out of q γ ( z (1) , . . .
Revised: , z ( L − 1) were marginalized out of q γ ( z (1) , . . .

Original: , z ( L − 1) , z ( L ) ) to represent q γ ( z ( L ) ) .
Revised: , z ( L − 1) , z ( L ) ) to represent q γ ( z ( L ) ) .

Original: As discussed in Section 4, z (1) , . .
Revised: As discussed in Section 4, z (1) , . .

Original: , z ( L − 1) were added there to enhance the modeling ﬂexibility of q γ ( z ( L ) ) .
Revised: , z ( L − 1) were added there to enhance the modeling ﬂexibility of q γ ( z ( L ) ) .

Original: In this section, the deep set of random variables z =
Revised: In this section, the deep set of random variables z =

Original: ( z (1) , . . .
Revised: ( z (1) , . . .

Original: , z ( L ) ) are inherent components of the underlying generative model for x , i.e. , p θ ( x , z )
Revised: , z ( L ) ) are inherent components of the underlying generative model for x , i.e. , p θ ( x , z )

Original: =
Revised: =

Original: p θ ( x , z (1) , . . . ,
Revised: p θ ( x , z (1) , . . . ,

Original: z ( L ) ) .
Revised: z ( L ) ) .

Original: Hence, all components of z manifested via inference model q φ ( z | x ) =
Revised: Hence, all components of z manifested via inference model q φ ( z | x ) =

Original: q φ ( z (1) , . . .
Revised: q φ ( z (1) , . . .

Original: , z ( L ) | x ) play a role in f ( z ) .
Revised: , z ( L ) | x ) play a role in f ( z ) .

Original: Besides, no speciﬁc structure is imposed on p θ ( x , z ) and q φ ( z | x ) in this section, moving beyond the aforementioned ﬁrst-order Markov structure.
Revised: Besides, no speciﬁc structure is imposed on p θ ( x , z ) and q φ ( z | x ) in this section, moving beyond the aforementioned ﬁrst-order Markov structure.

Original: For a practical application, one may employ domain knowledge to design suitable graphical models for p θ ( x , z ) and q φ ( z | x ) , and then use the following Theorem 3 for training.
Revised: For a practical application, one may employ domain knowledge to design suitable graphical models for p θ ( x , z ) and q φ ( z | x ) , and then use the following Theorem 3 for training.

Original: Theorem 3 ( Statistical Back-Propagation ) .
Revised: Theorem 3 ( Statistical Back-Propagation ) .

Original: For expectation objectives
Revised: For expectation objectives

Original: [Equation]
Revised: [Equation]

Original: where { z ( i ) } Ii =1 denotes I continuous internal variables with at least one child variable, { z ( j ) }
Revised: where { z ( i ) } Ii =1 denotes I continuous internal variables with at least one child variable, { z ( j ) }

Original: Lj =
Revised: Lj =

Original: I + represents L − I continuous or discrete leaf variables with no children except f ( · ) , and q φ ( · ) is constructed as a
Revised: I + represents L − I continuous or discrete leaf variables with no children except f ( · ) , and q φ ( · ) is constructed as a

Original: [Equation]
Revised: [Equation]

Original: with each element distribution q φ ( z v | pa ( z v )) having accessible variable-nablas as deﬁned in Theorem 1, pa ( z v ) denotes the parent variables of z v , the General and One-sample (GO) gradient for φ k ∈ φ is deﬁned as
Revised: with each element distribution q φ ( z v | pa ( z v )) having accessible variable-nablas as deﬁned in Theorem 1, pa ( z v ) denotes the parent variables of z v , the General and One-sample (GO) gradient for φ k ∈ φ is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where ch ( φ k ) denotes the children variables of φ k , and with z v ∈ ch ( φ k ) ,
Revised: where ch ( φ k ) denotes the children variables of φ k , and with z v ∈ ch ( φ k ) ,

Original: G
Revised: G

Original: [Equation]
Revised: [Equation]

Original: = [ · · · , g q φ ( z
Revised: = [ · · · , g q φ ( z

Original: [Equation]
Revised: [Equation]

Original: | pa ( z v
Revised: | pa ( z v

Original: [Equation]
Revised: [Equation]

Original: φ
Revised: φ

Original: [Equation]
Revised: [Equation]

Original: BP [ ch ( φ k )] =
Revised: BP [ ch ( φ k )] =

Original: [ · · · , BP [ z v ] , · · · ] T , and BP [ z v ] is iteratively calculated as BP [ z v ] =
Revised: [ · · · , BP [ z v ] , · · · ] T , and BP [ z v ] is iteratively calculated as BP [ z v ] =

Original: G q φ [ ch ( z v )]
Revised: G q φ [ ch ( z v )]

Original: z v
Revised: z v

Original: BP [ ch ( z v )] , until leaf variables where
Revised: BP [ ch ( z v )] , until leaf variables where

Original: BP [ z v ] =
Revised: BP [ z v ] =

Original: D z v [ f ( { z ( i ) } Li =1 )] .
Revised: D z v [ f ( { z ( i ) } Li =1 )] .

Original: Statistical back-propagation in Theorem 3 is relevant to hierarchical variational inference (HVI) (Ranganath et al., 2016b; Hoffman & Blei, 2015; Mnih & Gregor, 2014)
Revised: Statistical back-propagation in Theorem 3 is relevant to hierarchical variational inference (HVI) (Ranganath et al., 2016b; Hoffman & Blei, 2015; Mnih & Gregor, 2014)

Original: (see Appendix G), greatly generalizing GO gradients to the inference of directed acyclic probabilistic graphical models.
Revised: (see Appendix G), greatly generalizing GO gradients to the inference of directed acyclic probabilistic graphical models.

Original: In
Revised: In

Original: HVI variational distributions are speciﬁed as hierarchical graphical models constructed by neural networks.
Revised: HVI variational distributions are speciﬁed as hierarchical graphical models constructed by neural networks.

Original: Using statistical back-propagation, one may rely on GO gradients to perform HVI with low variance, while greatly broadening modeling ﬂexibility.
Revised: Using statistical back-propagation, one may rely on GO gradients to perform HVI with low variance, while greatly broadening modeling ﬂexibility.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: There are many methods directed toward low-variance gradients for expectation-based objectives.
Revised: There are many methods directed toward low-variance gradients for expectation-based objectives.

Original: Attracted by the generalization of REINFORCE, many works try to improve its performance via efﬁcient variance-reduction techniques, like control variants (Mnih & Gregor, 2014; Titsias & LázaroGredilla, 2015;
Revised: Attracted by the generalization of REINFORCE, many works try to improve its performance via efﬁcient variance-reduction techniques, like control variants (Mnih & Gregor, 2014; Titsias & LázaroGredilla, 2015;

Original: Gu et al., 2015; Mnih & Rezende, 2016; Tucker et al., 2017; Grathwohl et al., 2017) or via data augmentation and permutation techniques (Yin & Zhou, 2018).
Revised: Gu et al., 2015; Mnih & Rezende, 2016; Tucker et al., 2017; Grathwohl et al., 2017) or via data augmentation and permutation techniques (Yin & Zhou, 2018).

Original: Most of this research focuses on discrete random variables, likely because Rep (if it exists) works well for continuous random variables but it may not exist for discrete random variables.
Revised: Most of this research focuses on discrete random variables, likely because Rep (if it exists) works well for continuous random variables but it may not exist for discrete random variables.

Original: Other efforts are devoted to continuously relaxing discrete variables, to combine both REINFORCE and Rep for variance reduction (Jang et al., 2016; Maddison et al., 2016; Tucker et al., 2017; Grathwohl et al., 2017).
Revised: Other efforts are devoted to continuously relaxing discrete variables, to combine both REINFORCE and Rep for variance reduction (Jang et al., 2016; Maddison et al., 2016; Tucker et al., 2017; Grathwohl et al., 2017).

Original: Inspired by the low variance of Rep, there are methods that try to generalize its scope.
Revised: Inspired by the low variance of Rep, there are methods that try to generalize its scope.

Original: The Generalized Rep (GRep) gradient (Ruiz et al., 2016) employs an approximate reparameterization whose transformed distribution weakly depends on the parameters of interest.
Revised: The Generalized Rep (GRep) gradient (Ruiz et al., 2016) employs an approximate reparameterization whose transformed distribution weakly depends on the parameters of interest.

Original: Rejection sampling variational inference (RSVI)
Revised: Rejection sampling variational inference (RSVI)

Original: (Naesseth et al., 2016) exploits highly-tuned transformations in mature rejection sampling simulation to better approximate Rep for non-reparameterizable distributions.
Revised: (Naesseth et al., 2016) exploits highly-tuned transformations in mature rejection sampling simulation to better approximate Rep for non-reparameterizable distributions.

Original: Compared to the aforementioned methods, the proposed GO gradient, containing Rep as a special case for continuous random variables, applies to both continuous and discrete random variables with the same low-variance as the Rep gradient.
Revised: Compared to the aforementioned methods, the proposed GO gradient, containing Rep as a special case for continuous random variables, applies to both continuous and discrete random variables with the same low-variance as the Rep gradient.

Original: Implicit Rep gradients (Figurnov et al., 2018) and pathwise derivatives (Jankowiak & Obermeyer, 2018) are recent low-variance methods that exploit the gradient of the expected function; they are special cases of GO in the single-layer continuous settings.
Revised: Implicit Rep gradients (Figurnov et al., 2018) and pathwise derivatives (Jankowiak & Obermeyer, 2018) are recent low-variance methods that exploit the gradient of the expected function; they are special cases of GO in the single-layer continuous settings.

Original: The idea of gradient backpropagation through random variables has been exploited before.
Revised: The idea of gradient backpropagation through random variables has been exploited before.

Original: RELAX (Grathwohl et al., 2017), employing neural-network-parametrized control variants to assist
Revised: RELAX (Grathwohl et al., 2017), employing neural-network-parametrized control variants to assist

Original: REINFORCE for that goal, has a variance potentially as low as the Rep gradient.
Revised: REINFORCE for that goal, has a variance potentially as low as the Rep gradient.

Original: SCG (Schulman et al., 2015) utilizes the generalizability of REINFORCE to construct widely-applicable stochastic computation graphs.
Revised: SCG (Schulman et al., 2015) utilizes the generalizability of REINFORCE to construct widely-applicable stochastic computation graphs.

Original: However, REINFORCE is known to have high variance, especially for highdimensional problems, where the proposed methods are preferable when applicable (Schulman et al., 2015).
Revised: However, REINFORCE is known to have high variance, especially for highdimensional problems, where the proposed methods are preferable when applicable (Schulman et al., 2015).

Original: Stochastic back-propagation (Rezende et al., 2014; Fan et al., 2015), focusing mainly on reparameterizable Gaussian random variables and deep latent Gaussian models, exploits the product rule for an integral to derive gradient backpropagation through several continuous random variables.
Revised: Stochastic back-propagation (Rezende et al., 2014; Fan et al., 2015), focusing mainly on reparameterizable Gaussian random variables and deep latent Gaussian models, exploits the product rule for an integral to derive gradient backpropagation through several continuous random variables.

Original: By comparison, the proposed statistical back-propagation based on the GO gradient is applicable to most distributions for continuous random variables.
Revised: By comparison, the proposed statistical back-propagation based on the GO gradient is applicable to most distributions for continuous random variables.

Original: Further, it also ﬂexibly generalizes to hierarchical probabilistic graphical models with continuous internal variables and continuous/discrete leaf ones.
Revised: Further, it also ﬂexibly generalizes to hierarchical probabilistic graphical models with continuous internal variables and continuous/discrete leaf ones.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We examine the proposed GO gradients and statistical back-propagation with four experiments: ( i ) simple one-dimensional (gamma and negative binomial) examples are presented to verify the
Revised: We examine the proposed GO gradients and statistical back-propagation with four experiments: ( i ) simple one-dimensional (gamma and negative binomial) examples are presented to verify the

Original: GO gradient in Theorem 1, corresponding to nonnegative and discrete random variables; ( ii ) the discrete variational autoencoder experiment from Tucker et al.
Revised: GO gradient in Theorem 1, corresponding to nonnegative and discrete random variables; ( ii ) the discrete variational autoencoder experiment from Tucker et al.

Original: (2017) and Grathwohl et al.
Revised: (2017) and Grathwohl et al.

Original: (2017) is reproduced to compare GO with the state-of-the-art variance-reduction methods; ( iii ) a multinomial
Revised: (2017) is reproduced to compare GO with the state-of-the-art variance-reduction methods; ( iii ) a multinomial

Original: GAN, generating discrete observations, is constructed to demonstrate the deep GO gradient in
Revised: GAN, generating discrete observations, is constructed to demonstrate the deep GO gradient in

Original: Theorem 2; ( iv ) hierarchical variational inference (HVI) for two deep non-conjugate Bayesian models is developed to verify statistical back-propagation in Theorem 3.
Revised: Theorem 2; ( iv ) hierarchical variational inference (HVI) for two deep non-conjugate Bayesian models is developed to verify statistical back-propagation in Theorem 3.

Original: Note the experiments of Figurnov et al.
Revised: Note the experiments of Figurnov et al.

Original: and Jankowiak & Obermeyer (2018) additionally support our GO in the single-layer continuous settings.
Revised: and Jankowiak & Obermeyer (2018) additionally support our GO in the single-layer continuous settings.

Original: Many mature machine learning frameworks, like TensorFlow (Abadi et al.)
Revised: Many mature machine learning frameworks, like TensorFlow (Abadi et al.)

Original: and PyTorch (Paszke et al., 2017), are optimized for implementation of methods like back-propagation.
Revised: and PyTorch (Paszke et al., 2017), are optimized for implementation of methods like back-propagation.

Original: Fortunately, all gradient calculations in the proposed theorems obey the chain rule in expectation, enabling convenient incorporation of the proposed approaches into existing frameworks.
Revised: Fortunately, all gradient calculations in the proposed theorems obey the chain rule in expectation, enabling convenient incorporation of the proposed approaches into existing frameworks.

Original: Experiments presented below were implemented in TensorFlow or PyTorch with a Titan Xp GPU.
Revised: Experiments presented below were implemented in TensorFlow or PyTorch with a Titan Xp GPU.

Original: Code for all experiments can be found at github.com/YulaiCong/GOgradient .
Revised: Code for all experiments can be found at github.com/YulaiCong/GOgradient .

Original: Notation Gam( α, β ) denotes the gamma distribution with shape α and rate β , NB( r, P ) the negative binomial distribution with number of failures r and success probability P , Bern( P ) the Bernoulli distribution with probability P , Mult( n, P ) the multinomial distribution with number of trials n and event probabilities P , Pois( λ ) the Poisson distribution with rate λ , and Dir( α ) the Dirichlet distribution with concentration parameters α .
Revised: Notation Gam( α, β ) denotes the gamma distribution with shape α and rate β , NB( r, P ) the negative binomial distribution with number of failures r and success probability P , Bern( P ) the Bernoulli distribution with probability P , Mult( n, P ) the multinomial distribution with number of trials n and event probabilities P , Pois( λ ) the Poisson distribution with rate λ , and Dir( α ) the Dirichlet distribution with concentration parameters α .

Original: 7. G AMMA AND NB O NE - DIMENSIONAL S IMPLE E XAMPLES
Revised: 7. G AMMA AND NB O NE - DIMENSIONAL S IMPLE E XAMPLES

Original: We ﬁrst consider illustrative one-dimensional “toy” problems, to examine the GO gradient for both continuous and discrete random variables.
Revised: We ﬁrst consider illustrative one-dimensional “toy” problems, to examine the GO gradient for both continuous and discrete random variables.

Original: The optimization objective is expressed as
Revised: The optimization objective is expressed as

Original: [Equation]
Revised: [Equation]

Original: where for continuous z we assume p ( z | x ) =
Revised: where for continuous z we assume p ( z | x ) =

Original: Gam( z ; α 0 , β 0 ) for given set ( α 0 , β 0 ) , with q φ ( z ) =
Revised: Gam( z ; α 0 , β 0 ) for given set ( α 0 , β 0 ) , with q φ ( z ) =

Original: Gam( z ; α, β ) and φ = { α, β } ; for discrete z we assume p ( z | x ) =
Revised: Gam( z ; α, β ) and φ = { α, β } ; for discrete z we assume p ( z | x ) =

Original: NB( z ; r 0 , p 0 ) for given set ( r 0 , p 0 ) , with q φ ( z ) =
Revised: NB( z ; r 0 , p 0 ) for given set ( r 0 , p 0 ) , with q φ ( z ) =

Original: NB( z ; r, p ) and φ = { r, p } .
Revised: NB( z ; r, p ) and φ = { r, p } .

Original: Stochastic gradient ascent with onesample-estimated gradients is used to optimize the objective, which is equivalent to minimizing
Revised: Stochastic gradient ascent with onesample-estimated gradients is used to optimize the objective, which is equivalent to minimizing

Original: KL ( q φ ( z ) (cid:107) p ( z | x )) .
Revised: KL ( q φ ( z ) (cid:107) p ( z | x )) .

Original: Figure 2 shows the results (see Appendix H for additional details).
Revised: Figure 2 shows the results (see Appendix H for additional details).

Original: For the nonnegative continuous z associated with the gamma distribution, we compare our GO gradient with GRep (Ruiz et al., 2016),
Revised: For the nonnegative continuous z associated with the gamma distribution, we compare our GO gradient with GRep (Ruiz et al., 2016),

Original: [Figure]
Revised: [Figure]

Original: RSVI (Naesseth et al., 2016), and their modiﬁed version using the “sticking” idea (Roeder et al., 2017), denoted as GRep-Stick and RSVI-Stick, respectively.
Revised: RSVI (Naesseth et al., 2016), and their modiﬁed version using the “sticking” idea (Roeder et al., 2017), denoted as GRep-Stick and RSVI-Stick, respectively.

Original: For RSVI and RSVI-Stick, the shape augmentation parameter is set as 5 by default.
Revised: For RSVI and RSVI-Stick, the shape augmentation parameter is set as 5 by default.

Original: The only difference between GRep and GRep-Stick (also RSVI and RSVI-Stick) is the latter does not analytically express the entropy E q φ ( z ) [ − log q φ ( z )] .
Revised: The only difference between GRep and GRep-Stick (also RSVI and RSVI-Stick) is the latter does not analytically express the entropy E q φ ( z ) [ − log q φ ( z )] .

Original: Figure 2(a) clearly shows the utility of employing sticking to reduce variance; without it, GRep and
Revised: Figure 2(a) clearly shows the utility of employing sticking to reduce variance; without it, GRep and

Original: RSVI exhibit high variance, that destabilizes the optimization for small gamma shape parameters, as shown in Figures 2(b) and 2(c).
Revised: RSVI exhibit high variance, that destabilizes the optimization for small gamma shape parameters, as shown in Figures 2(b) and 2(c).

Original: We adopt the sticking approach hereafter for all the compared methods.
Revised: We adopt the sticking approach hereafter for all the compared methods.

Original: Among methods with sticking, GO exhibits the lowest variance in general, as shown in
Revised: Among methods with sticking, GO exhibits the lowest variance in general, as shown in

Original: Figures 2(a) and 2(b).
Revised: Figures 2(a) and 2(b).

Original: GO empirically provides more stable learning curves, as shown in Figure 2(c).
Revised: GO empirically provides more stable learning curves, as shown in Figure 2(c).

Original: For the discrete case corresponding to the NB distribution, GO is compared to REINFORCE (Williams, 1992).
Revised: For the discrete case corresponding to the NB distribution, GO is compared to REINFORCE (Williams, 1992).

Original: To address the concern about comparison with the same number of evaluations of the expected function 3 , another curve of REINFORCE using 2 samples is also added, termed
Revised: To address the concern about comparison with the same number of evaluations of the expected function 3 , another curve of REINFORCE using 2 samples is also added, termed

Original: REINFORCE2.
Revised: REINFORCE2.

Original: It is apparent from Figure 2(d) that, thanks to analytically removing the “0” terms in (8), the GO gradient has much lower variance, even in this simple one-dimensional case.
Revised: It is apparent from Figure 2(d) that, thanks to analytically removing the “0” terms in (8), the GO gradient has much lower variance, even in this simple one-dimensional case.

Original: 7. D ISCRETE V ARIATIONAL A UTOENCODER
Revised: 7. D ISCRETE V ARIATIONAL A UTOENCODER

Original: To demonstrate the low variance of the proposed GO gradient, we consider the discrete variational autoencoder (VAE) experiment from REBAR (Tucker et al., 2017) and RELAX (Grathwohl et al., 2017), to make a direct comparison with state-of-the-art variance-reduction methods.
Revised: To demonstrate the low variance of the proposed GO gradient, we consider the discrete variational autoencoder (VAE) experiment from REBAR (Tucker et al., 2017) and RELAX (Grathwohl et al., 2017), to make a direct comparison with state-of-the-art variance-reduction methods.

Original: Since the statistical back-propagation in Theorem 3 cannot handle discrete internal variables, we focus on the single-latent-layer settings (1 layer of 200 Bernoulli random variables), i.e.,
Revised: Since the statistical back-propagation in Theorem 3 cannot handle discrete internal variables, we focus on the single-latent-layer settings (1 layer of 200 Bernoulli random variables), i.e.,

Original: [Equation]
Revised: [Equation]

Original: where P z is the parameters of the prior p θ ( z ) ,
Revised: where P z is the parameters of the prior p θ ( z ) ,

Original: NN P x | z ( z ) means using a neural network to project the latent binary code z to the parameters P x | z of the likelihood p θ ( x | z ) , and NN P z | x ( x ) is similarly deﬁned for q φ ( z | x ) .
Revised: NN P x | z ( z ) means using a neural network to project the latent binary code z to the parameters P x | z of the likelihood p θ ( x | z ) , and NN P z | x ( x ) is similarly deﬁned for q φ ( z | x ) .

Original: The objective is given in (1).
Revised: The objective is given in (1).

Original: See Appendix I for more details
Revised: See Appendix I for more details

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Figure 3 (also Figure 9 of Appendix I) shows the training curves versus iteration and running time for the compared methods.
Revised: Figure 3 (also Figure 9 of Appendix I) shows the training curves versus iteration and running time for the compared methods.

Original: Even without any variance-reduction techniques, GO provides better performance, faster convergence rate, and better running efﬁciency (about ten times faster in achieving the best training ELBO of RERAR/RELAX in this experiment).
Revised: Even without any variance-reduction techniques, GO provides better performance, faster convergence rate, and better running efﬁciency (about ten times faster in achieving the best training ELBO of RERAR/RELAX in this experiment).

Original: We believe GO’s better performance originates from: ( i ) its inherent low-variance nature; ( ii ) GO has less parameters compared to REBAR and RELAX (no control variant is adopted for GO); ( iii ) efﬁcient batch processing methods (see Appendix I) are adopted to beneﬁt from parallel computing.
Revised: We believe GO’s better performance originates from: ( i ) its inherent low-variance nature; ( ii ) GO has less parameters compared to REBAR and RELAX (no control variant is adopted for GO); ( iii ) efﬁcient batch processing methods (see Appendix I) are adopted to beneﬁt from parallel computing.

Original: Table 1 presents the best training/validation ELBOs under various experimental settings for the compared methods.
Revised: Table 1 presents the best training/validation ELBOs under various experimental settings for the compared methods.

Original: GO provides the best performance in all situations.
Revised: GO provides the best performance in all situations.

Original: Additional experimental results are given in
Revised: Additional experimental results are given in

Original: Appendix I.
Revised: Appendix I.

Original: Many variance-reduction techniques can be used to further reduce the variance of GO, especially when complicated models are of interest.
Revised: Many variance-reduction techniques can be used to further reduce the variance of GO, especially when complicated models are of interest.

Original: Compared to RELAX, GO cannot be directly applied when f ( y ) is not computable or where the interested model has discrete internal variables (like multilayer
Revised: Compared to RELAX, GO cannot be directly applied when f ( y ) is not computable or where the interested model has discrete internal variables (like multilayer

Original: Sigmoid belief networks (Neal, 1992)).
Revised: Sigmoid belief networks (Neal, 1992)).

Original: For the latter issue, we present in Appendix B.4 a procedure to assist GO (or statistical back-propagation in Theorem 3) in handling discrete internal variables.
Revised: For the latter issue, we present in Appendix B.4 a procedure to assist GO (or statistical back-propagation in Theorem 3) in handling discrete internal variables.

Original: 7. M ULTINOMIAL GAN
Revised: 7. M ULTINOMIAL GAN

Original: To demonstrate the deep GO gradient in Theorem 2, we adopt multinomial leaf variables x and construct a new multinomial GAN (denoted as MNGAN-GO) for generating discrete observations with a ﬁnite alphabet.
Revised: To demonstrate the deep GO gradient in Theorem 2, we adopt multinomial leaf variables x and construct a new multinomial GAN (denoted as MNGAN-GO) for generating discrete observations with a ﬁnite alphabet.

Original: The corresponding generator p θ ( x ) is expressed as
Revised: The corresponding generator p θ ( x ) is expressed as

Original: [Equation]
Revised: [Equation]

Original: For brevity, we integrate the generator’s parameters θ into the NN notation, and do not explicitly express them.
Revised: For brevity, we integrate the generator’s parameters θ into the NN notation, and do not explicitly express them.

Original: Details for this example are provided in Appendix J.
Revised: Details for this example are provided in Appendix J.

Original: We compare MNGAN-GO with the recently proposed boundary-seeking GAN (BGAN) (Hjelm et al., 2018) on 1-bit (1-state, Bernoulli leaf variables x ), 1-bit (2-state), 2-bit (4-state), 3-bit (8-state) and
Revised: We compare MNGAN-GO with the recently proposed boundary-seeking GAN (BGAN) (Hjelm et al., 2018) on 1-bit (1-state, Bernoulli leaf variables x ), 1-bit (2-state), 2-bit (4-state), 3-bit (8-state) and

Original: Table 2 presents inception scores (Salimans et al., 2016) of both methods.
Revised: Table 2 presents inception scores (Salimans et al., 2016) of both methods.

Original: MNGAN-GO performs better in general.
Revised: MNGAN-GO performs better in general.

Original: Further, with GO’s assistance, MNGAN-GO shows more potential to beneﬁt from richer information coming from more quantized states.
Revised: Further, with GO’s assistance, MNGAN-GO shows more potential to beneﬁt from richer information coming from more quantized states.

Original: For demonstration, Figure 4 shows the generated samples from the 4-bit experiment, where better image quality and higher diversity are observed for the samples from MNGAN-GO.
Revised: For demonstration, Figure 4 shows the generated samples from the 4-bit experiment, where better image quality and higher diversity are observed for the samples from MNGAN-GO.

Original: 7. HVI FOR D EEP E XPONENTIAL F AMILIES AND D EEP L ATENT D IRICHLET A LLOCATION
Revised: 7. HVI FOR D EEP E XPONENTIAL F AMILIES AND D EEP L ATENT D IRICHLET A LLOCATION

Original: To demonstrate statistical back-propagation in Theorem 3, we design variational inference nets for two nonconjugate hierarchical Bayesian models,
Revised: To demonstrate statistical back-propagation in Theorem 3, we design variational inference nets for two nonconjugate hierarchical Bayesian models,

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: e., deep exponential families (DEF) (Ranganath et al., 2015) and deep latent Dirichlet allocation (DLDA)
Revised: e., deep exponential families (DEF) (Ranganath et al., 2015) and deep latent Dirichlet allocation (DLDA)

Original: (Zhou et al., 2015; 2016;
Revised: (Zhou et al., 2015; 2016;

Original: Cong et al., 2017).
Revised: Cong et al., 2017).

Original: [Equation]
Revised: [Equation]

Original: For demonstration
Revised: For demonstration

Original: [Equation]
Revised: [Equation]

Original: we design the inference nets q φ ( z | x ) following the ﬁrst-order Markov chain construction in Section 5, namely
Revised: we design the inference nets q φ ( z | x ) following the ﬁrst-order Markov chain construction in Section 5, namely

Original: [Equation]
Revised: [Equation]

Original: Further details are provided in Appendix K. One might also wish to design inference nets that have structure beyond the above ﬁrst-order Markov chain construction, as in Zhang et al.
Revised: Further details are provided in Appendix K. One might also wish to design inference nets that have structure beyond the above ﬁrst-order Markov chain construction, as in Zhang et al.

Original: ; we do not consider that here, but Theorem 3 is applicable to that case.
Revised: ; we do not consider that here, but Theorem 3 is applicable to that case.

Original: 2 Iteration
Revised: 2 Iteration

Original: E L
Revised: E L

Original: [Figure]
Revised: [Figure]

Original: O
Revised: O

Original: GO
Revised: GO

Original: HVI for a 2-layer DEF is ﬁrst performed, with the ELBO curves shown in Figure 5.
Revised: HVI for a 2-layer DEF is ﬁrst performed, with the ELBO curves shown in Figure 5.

Original: GO enables faster and more stable convergence.
Revised: GO enables faster and more stable convergence.

Original: Figure 6 presents the HVI results for a 3-layer DLDA, for which stable ELBOs are again observed.
Revised: Figure 6 presents the HVI results for a 3-layer DLDA, for which stable ELBOs are again observed.

Original: More importantly, with the GO gradient, one can utilize pure gradient-based methods to efﬁciently train such complicated nonconjugate models for meaningful dictionaries (see Appendix K for more implementary details).
Revised: More importantly, with the GO gradient, one can utilize pure gradient-based methods to efﬁciently train such complicated nonconjugate models for meaningful dictionaries (see Appendix K for more implementary details).

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: For expectation-based objectives, we propose a General and One-sample (GO) gradient that applies to continuous and discrete random variables.
Revised: For expectation-based objectives, we propose a General and One-sample (GO) gradient that applies to continuous and discrete random variables.

Original: We further generalize the GO gradient to cases for which the underlying model is deep and has a marginal distribution corresponding to the latent variables of interest, and to cases for which the latent variables are hierarchical.
Revised: We further generalize the GO gradient to cases for which the underlying model is deep and has a marginal distribution corresponding to the latent variables of interest, and to cases for which the latent variables are hierarchical.

Original: The GO-gradient setup is demonstrated to yield the same low-variance estimation as the reparameterization trick, which is only applicable to reparameterizable continuous random variables.
Revised: The GO-gradient setup is demonstrated to yield the same low-variance estimation as the reparameterization trick, which is only applicable to reparameterizable continuous random variables.

Original: Alongside the GO gradient, we constitute a means of propagating the chain rule through distributions.
Revised: Alongside the GO gradient, we constitute a means of propagating the chain rule through distributions.

Original: Accordingly, we present statistical back-propagation, to ﬂexibly integrate deep neural networks with general classes of random variables.
Revised: Accordingly, we present statistical back-propagation, to ﬂexibly integrate deep neural networks with general classes of random variables.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: We thank the anonymous reviewers for their useful comments.
Revised: We thank the anonymous reviewers for their useful comments.

Revised: The research was supported by part by DARPA, DOE, NIH, NSF and ONR.

Original: The Titan Xp GPU used was donated by the NVIDIA Corporation.
Revised: The Titan Xp GPU used was donated by the NVIDIA Corporation.

Original: We also wish to thank Chenyang Tao, Liqun Chen, and Chunyuan Li for helpful discussions.
Revised: We also wish to thank Chenyang Tao, Liqun Chen, and Chunyuan Li for helpful discussions.

Original: A P ROOF OF T HEOREM
Revised: A P ROOF OF T HEOREM

Original: We ﬁrst prove (7) in the main manuscript, followed by its discrete counterpart, i.e., (8) in the main manuscript.
Revised: We ﬁrst prove (7) in the main manuscript, followed by its discrete counterpart, i.e., (8) in the main manuscript.

Original: Then, it is easy to verify Theorem 1.
Revised: Then, it is easy to verify Theorem 1.

Original: A. P ROOF OF E QUATION (7) IN THE M AIN M ANUSCRIPT
Revised: A. P ROOF OF E QUATION (7) IN THE M AIN M ANUSCRIPT

Original: Similar proof in one-dimension is also given in the supplemental materials of Ranganath et al.
Revised: Similar proof in one-dimension is also given in the supplemental materials of Ranganath et al.

Original: (2016b).
Revised: (2016b).

Original: We want to calculate
Revised: We want to calculate

Original: [Equation]
Revised: [Equation]

Original: where y − v denotes y with y v excluded.
Revised: where y − v denotes y with y v excluded.

Original: Without loss of generality
Revised: Without loss of generality

Original: [Equation]
Revised: [Equation]

Original: we assume y v ∈ ( −∞ , ∞ ) .
Revised: we assume y v ∈ ( −∞ , ∞ ) .

Original: Let v (cid:48)
Revised: Let v (cid:48)

Original: ( y v ) =
Revised: ( y v ) =

Original: ∇ γ q γ ( y v )
Revised: ∇ γ q γ ( y v )

Original: [Equation]
Revised: [Equation]

Original: and we have
Revised: and we have

Original: [Equation]
Revised: [Equation]

Original: where Q γ ( y v ) is the cumulative distribution function (CDF) of q γ ( y v ) .
Revised: where Q γ ( y v ) is the cumulative distribution function (CDF) of q γ ( y v ) .

Original: Further deﬁne u ( y v ) =
Revised: Further deﬁne u ( y v ) =

Original: f ( y v , y − v )
Revised: f ( y v , y − v )

Original: [Equation]
Revised: [Equation]

Original: we then apply integration by parts (or partial integration) to get
Revised: we then apply integration by parts (or partial integration) to get

Original: [Equation]
Revised: [Equation]

Original: With Q γ ( ∞ ) = 1 and Q γ ( −∞ ) = 0 , it’s straightforward to verify that the ﬁrst term is always zero for any Q γ ( y v ) , thus named the “0” term.
Revised: With Q γ ( ∞ ) = 1 and Q γ ( −∞ ) = 0 , it’s straightforward to verify that the ﬁrst term is always zero for any Q γ ( y v ) , thus named the “0” term.

Original: A. P ROOF OF E QUATION (8) IN THE M AIN M ANUSCRIPT
Revised: A. P ROOF OF E QUATION (8) IN THE M AIN M ANUSCRIPT

Original: For discrete variables y , we have
Revised: For discrete variables y , we have

Original: [Equation]
Revised: [Equation]

Original: where y v ∈ { 0 , 1 , · · · , N } and N is the size of the alphabet.
Revised: where y v ∈ { 0 , 1 , · · · , N } and N is the size of the alphabet.

Original: To handle the summation of products of two sequences and develop discrete counterpart of (7), we ﬁrst introduce Abel transformation.
Revised: To handle the summation of products of two sequences and develop discrete counterpart of (7), we ﬁrst introduce Abel transformation.

Original: Abel transformation.
Revised: Abel transformation.

Original: Given two sequences { a n } and { b n } , with n ∈ { 0 , · · · , N } , we deﬁne
Revised: Given two sequences { a n } and { b n } , with n ∈ { 0 , · · · , N } , we deﬁne

Original: B 0 = b 0 and B n = (cid:80) nk =

Revised: B 0 = b 0 and B n = (cid:80) nk =

Original: b n for n ≥ 1 .
Revised: b n for n ≥ 1 .

Original: Accordingly, we have
Revised: Accordingly, we have

Original: [Equation]
Revised: [Equation]

Original: Substituting n = y v , a n = f ( y ) , b n = ∇ γ q γ ( y v ) , and B n = ∇ γ Q γ ( y v ) into the above equation, we have
Revised: Substituting n = y v , a n = f ( y ) , b n = ∇ γ q γ ( y v ) , and B n = ∇ γ Q γ ( y v ) into the above equation, we have

Original: [Equation]
Revised: [Equation]

Original: Note the ﬁrst term equals zero for both ﬁnite alphabet, i
Revised: Note the ﬁrst term equals zero for both ﬁnite alphabet, i

Original: [Table]
Revised: [Table]

Original: e., N < ∞ , and inﬁnite alphabet, i
Revised: e., N < ∞ , and inﬁnite alphabet, i

Original: [Table]
Revised: [Table]

Original: e.,
Revised: e.,

Original: N = ∞ .
Revised: N = ∞ .

Original: When N = ∞ , we get (8) in the main manuscript.
Revised: When N = ∞ , we get (8) in the main manuscript.

Original: With the above proofs for Eqs.
Revised: With the above proofs for Eqs.

Original: (7) and (8)
Revised: (7) and (8)

Original: [Table]
Revised: [Table]

Original: one could straightforwardly verify Theorem
Revised: one could straightforwardly verify Theorem

Original: [Table] [Table]
Revised: [Table] [Table]

Original: +

Original: [Equation]
Revised: [Equation]

Original: =  
Revised: =  

Original: [Equation]
Revised: [Equation]

Original:  
Revised:  

Original: [Equation]
Revised: [Equation]

Original: = 
Revised: = 

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: g
Revised: 

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: Exponential ( λ ) g
Revised: Exponential ( λ ) g

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: = − yλ
Revised: = − yλ

Original: Student’s t ( v )
Revised: Student’s t ( v )

Original: [Equation]
Revised: [Equation]

Original:  
Revised: v

Original: [Equation]
Revised: [Equation]

Original: F
Revised:  

Original: [Equation]
Revised: [Equation]

Original: + y
Revised: + y

Original: [Equation]
Revised: [Equation]

Original: F
Revised: F

Original: [Equation]
Revised: [Equation]

Original:  
Revised:  

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: v
Revised: v

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: − y 2 v
Revised: − y 2 v

Original: [Equation]
Revised: [Equation]

Original: − y 2 v   
Revised: − y 2 v   

Original: 
Revised: 

Original: Weibull ( [Table] [Table] ) g 
Revised: Weibull ( λ,k ) g q γ

Original: [Table]

Original: ( y ) λ
Revised: ( y ) λ

Original: =
Revised: =

Original: yλ g q γ ( y ) k
Revised: yλ g q γ ( y ) k

Original: = yk log
Revised: = yk log

Original: [Table]

Original: [Table]
Revised: [Table]

Original: B W HY D ISCRETE Internal V ARIABLES A RE C HALLENGING ?
Revised: B W HY D ISCRETE Internal V ARIABLES A RE C HALLENGING ?

Original: For simpler demonstration, we ﬁrst use a 2-layer model to show why discrete internal variables are challenging.
Revised: For simpler demonstration, we ﬁrst use a 2-layer model to show why discrete internal variables are challenging.

Original: Then, we present an importance-sampling proposal that might be useful under speciﬁc situations.
Revised: Then, we present an importance-sampling proposal that might be useful under speciﬁc situations.

Original: Finally, we present a strategy to learn discrete internal variables with the statistical back-propagation in Theorem 3 of the main manuscript.
Revised: Finally, we present a strategy to learn discrete internal variables with the statistical back-propagation in Theorem 3 of the main manuscript.

Original: Assume q γ ( y ) being the marginal distribution of the following 2-layer model
Revised: Assume q γ ( y ) being the marginal distribution of the following 2-layer model

Original: [Equation]
Revised: [Equation]

Original: where γ = { γ y , γ λ } , q γ ( y , λ ) =
Revised: where γ = { γ y , γ λ } , q γ ( y , λ ) =

Original: q γ y ( y | λ ) q γ λ
Revised: q γ y ( y | λ ) q γ λ

Original: [Equation]
Revised: [Equation]

Original: λ ) =
Revised: λ ) =

Original: (cid:81) v q γ y ( y v | λ ) · (cid:81) k
Revised: (cid:81) v q γ y ( y v | λ ) · (cid:81) k

Original: q γ λ ( λ k ) , and both the leaf variable y and the internal variable λ could be either continuous or discrete.
Revised: q γ λ ( λ k ) , and both the leaf variable y and the internal variable λ could be either continuous or discrete.

Original: Accordingly, the objective becomes
Revised: Accordingly, the objective becomes

Original: [Equation]
Revised: [Equation]

Original: [ f ( y )] =
Revised: [ f ( y )] =

Original: E q γy ( y | λ ) q γλ ( λ ) [ f ( y )] .
Revised: E q γy ( y | λ ) q γλ ( λ ) [ f ( y )] .

Original: For gradient wrt γ y , using Theorem 1, it is straight to show
Revised: For gradient wrt γ y , using Theorem 1, it is straight to show

Original: [Equation]
Revised: [Equation]

Original: For gradient wrt γ λ , we ﬁrst have
Revised: For gradient wrt γ λ , we ﬁrst have

Original: [Equation]
Revised: [Equation]

Original: With ˆ f ( λ ) =
Revised: With ˆ f ( λ ) =

Original: E q γy ( y | λ )
Revised: E q γy ( y | λ )

Original: [ f ( y )]
Revised: [ f ( y )]

Original: [Equation]
Revised: [Equation]

Original: we then apply Theorem 1 and get
Revised: we then apply Theorem 1 and get

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where D λ [ ˆ f ( λ )] =
Revised: where D λ [ ˆ f ( λ )] =

Original: [ · · · , D λ k
Revised: [ · · · , D λ k

Original: [ ˆ f ( λ )] , · · · ] T , and
Revised: [ ˆ f ( λ )] , · · · ] T , and

Original: [Equation]
Revised: [Equation]

Original: Continous
Revised: Continous

Original: [Equation]
Revised: [Equation]

Original: Discrete λ k
Revised: Discrete λ k

Original: Next
Revised: Next

Original: [Equation]
Revised: [Equation]

Original: we separately discuss the situations where λ k is continuous or discrete
Revised: we separately discuss the situations where λ k is continuous or discrete

Original: [Equation]
Revised: [Equation]

Original: B. F OR CONTINUOUS λ k
Revised: B. F OR CONTINUOUS λ k

Original: One can directly apply Theorem 1 again, namely
Revised: One can directly apply Theorem 1 again, namely

Original: [Equation]
Revised: [Equation]

Original: Substituting (17) into (16), we have
Revised: Substituting (17) into (16), we have

Original: [Equation]
Revised: [Equation]

Original: B. F OR DISCRETE λ k
Revised: B. F OR DISCRETE λ k

Original: In this case
Revised: In this case

Original: [Equation]
Revised: [Equation]

Original: we need to calculate D λ k [ ˆ f ( λ )]
Revised: we need to calculate D λ k [ ˆ f ( λ )]

Original: = ˆ f ( λ − k , λ k + 1) − ˆ f ( λ ) .
Revised: = ˆ f ( λ − k , λ k + 1) − ˆ f ( λ ) .

Original: The keys are again partial integration and Abel transformation.
Revised: The keys are again partial integration and Abel transformation.

Original: For simplicity, we ﬁrst assume one-dimensional y , and separately discuss y being continuous and discrete.
Revised: For simplicity, we ﬁrst assume one-dimensional y , and separately discuss y being continuous and discrete.

Original: For continuous y
Revised: For continuous y

Original: [Equation]
Revised: [Equation]

Original: we apply partial integration to ˆ f ( λ ) and get
Revised: we apply partial integration to ˆ f ( λ ) and get

Original: [Equation]
Revised: [Equation]

Original: Accordingly
Revised: Accordingly

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Removing the “0” term
Revised: Removing the “0” term

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: For discrete y
Revised: For discrete y

Original: [Equation]
Revised: [Equation]

Original: by similarly exploiting Abel transformation
Revised: by similarly exploiting Abel transformation

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Accordingly, we get
Revised: Accordingly, we get

Original: [Equation]
Revised: [Equation]

Original: Unifying (19) for continuous y and (20) for discrete y , we have
Revised: Unifying (19) for continuous y and (20) for discrete y , we have

Original: [Equation]
Revised: [Equation]

Original: where we deﬁne
Revised: where we deﬁne

Original: [Equation]
Revised: [Equation]

Original: Multi-dimensional y .
Revised: Multi-dimensional y .

Original: Based on the above one-dimensional foundation
Revised: Based on the above one-dimensional foundation

Original: [Equation]
Revised: [Equation]

Original: we next move on to multidimensional situations.
Revised: we next move on to multidimensional situations.

Original: With deﬁnitions y : i (cid:44) { y 1 , · · · , y i } and y i : (cid:44) { y i , · · · , y V } , where V is the dimensionality of y
Revised: With deﬁnitions y : i (cid:44) { y 1 , · · · , y i } and y i : (cid:44) { y i , · · · , y V } , where V is the dimensionality of y

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Apply (21) and we have
Revised: Apply (21) and we have

Original: [Equation]
Revised: [Equation]

Original: Similarly
Revised: Similarly

Original: [Equation]
Revised: [Equation]

Original: we add extra terms to the above equation to enable applying (21) again as
Revised: we add extra terms to the above equation to enable applying (21) again as

Original: [Equation]
Revised: [Equation]

Original: Accordingly, we apply (21) to the ﬁrst two terms and have
Revised: Accordingly, we apply (21) to the ﬁrst two terms and have

Original: [Equation]
Revised: [Equation]

Original: So forth
Revised: So forth

Original: [Equation]
Revised: [Equation]

Original: we summarize the pattern into the following equation as
Revised: we summarize the pattern into the following equation as

Original: [Equation]
Revised: [Equation]

Original: where A λ k ( y , V ) is iteratively calculated as
Revised: where A λ k ( y , V ) is iteratively calculated as

Original: [Equation]
Revised: [Equation]

Original: Despite elegant structures within (23)
Revised: Despite elegant structures within (23)

Original: [Equation]
Revised: [Equation]

Original: to calculate it
Revised: to calculate it

Original: [Equation]
Revised: [Equation]

Original: one must iterate over all dimensions of y
Revised: one must iterate over all dimensions of y

Original: [Equation]
Revised: [Equation]

Original: which is computational expensive in practice.
Revised: which is computational expensive in practice.

Original: More importantly, it is straightforward to show that deeper models will have similar but much more complicated expressions
Revised: More importantly, it is straightforward to show that deeper models will have similar but much more complicated expressions

Original: [Equation]
Revised: [Equation]

Original: B. A N I MPORTANCE - SAMPLING P ROPOSAL TO H ANDLE D ISCRETE Internal V ARIABLES
Revised: B. A N I MPORTANCE - SAMPLING P ROPOSAL TO H ANDLE D ISCRETE Internal V ARIABLES

Original: Next, we present an intuitive proposal that might be useful under speciﬁc situations
Revised: Next, we present an intuitive proposal that might be useful under speciﬁc situations

Original: [Equation]
Revised: [Equation]

Original: The key idea is to use different extra items to enable “easy-to-use” expression for D λ k
Revised: The key idea is to use different extra items to enable “easy-to-use” expression for D λ k

Original: [ ˆ f ( λ )] , namely
Revised: [ ˆ f ( λ )] , namely

Original: [Equation]
Revised: [Equation]

Original: Apply (21) to the adjacent two terms for V times
Revised: Apply (21) to the adjacent two terms for V times

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where we can apply the idea of importance sampling and modify the above equation to
Revised: where we can apply the idea of importance sampling and modify the above equation to

Original: [Equation]
Revised: [Equation]

Original: Note that importance sampling may not always work well in practice (Bishop, 2006).
Revised: Note that importance sampling may not always work well in practice (Bishop, 2006).

Original: We further deﬁne the generalized variable-nabla as
Revised: We further deﬁne the generalized variable-nabla as

Original: [Equation]
Revised: [Equation]

Original: With the generalized variable-nabla, we unify (24) for discrete λ k and (17) for continuous λ k and get
Revised: With the generalized variable-nabla, we unify (24) for discrete λ k and (17) for continuous λ k and get

Original: [Equation]
Revised: [Equation]

Original: which apparently obeys the chain rule.
Revised: which apparently obeys the chain rule.

Original: Accordingly, we have the gradient for γ λ in (16) as
Revised: Accordingly, we have the gradient for γ λ in (16) as

Original: [Equation]
Revised: [Equation]

Original: One can straightforwardly verify that, with the generalized variable-nabla deﬁned in (25), the chain rule applies to ∇ γ E q γ ( y ( L ) )
Revised: One can straightforwardly verify that, with the generalized variable-nabla deﬁned in (25), the chain rule applies to ∇ γ E q γ ( y ( L ) )

Original: [ f ( y ( L ) )] , where one can freely specify both leaf and internal variables to be either continuous or discrete.
Revised: [ f ( y ( L ) )] , where one can freely specify both leaf and internal variables to be either continuous or discrete.

Original: The only problem is that, for discrete internal variables, the importance sampling trick used in (24) may not always work as expected.
Revised: The only problem is that, for discrete internal variables, the importance sampling trick used in (24) may not always work as expected.

Original: B. S TRATEGY TO L EARN D ISCRETE I NTERNAL V ARIABLES WITH S TATISTICAL B ACK -P ROPAGATION
Revised: B. S TRATEGY TO L EARN D ISCRETE I NTERNAL V ARIABLES WITH S TATISTICAL B ACK -P ROPAGATION

Original: Practically, if one has to deal with a q γ ( y (1) , · · · , y ( L ) ) with discrete internal variables y ( l ) , l < L , we suggest the strategy in Figure 7, with which one should expect a close performance but enjoy much easier implementation with statistical back-propagation in Theorem 3 of the main manuscript.
Revised: Practically, if one has to deal with a q γ ( y (1) , · · · , y ( L ) ) with discrete internal variables y ( l ) , l < L , we suggest the strategy in Figure 7, with which one should expect a close performance but enjoy much easier implementation with statistical back-propagation in Theorem 3 of the main manuscript.

Original: In fact, one can always add additional continuous internal variables to the graphical models to remedy the performance loss or even boost the performance.
Revised: In fact, one can always add additional continuous internal variables to the graphical models to remedy the performance loss or even boost the performance.

Original: [Figure]
Revised: [Figure]

Original: C P ROOF OF L EMMA
Revised: C P ROOF OF L EMMA

Original: First, a marginal distribution q γ ( y ) with reparameterization y =
Revised: First, a marginal distribution q γ ( y ) with reparameterization y =

Original: τ γ ( (cid:15) ) , (cid:15) ∼ q (
Revised: τ γ ( (cid:15) ) , (cid:15) ∼ q (

Original: [Equation]
Revised: [Equation]

Original: ) can be expressed as a joint distribution, namely
Revised: ) can be expressed as a joint distribution, namely

Original: [Equation]
Revised: [Equation]

Original: where q γ ( y | (cid:15) )
Revised: where q γ ( y | (cid:15) )

Original: =
Revised: =

Original: δ ( y − τ γ ( (cid:15) )) , δ ( · ) is the Dirac delta function, and τ γ ( (cid:15) ) could be ﬂexibly speciﬁed as a injective, surjective, or bijective function.
Revised: δ ( y − τ γ ( (cid:15) )) , δ ( · ) is the Dirac delta function, and τ γ ( (cid:15) ) could be ﬂexibly speciﬁed as a injective, surjective, or bijective function.

Original: Next, we align notations and rewrite (10) as
Revised: Next, we align notations and rewrite (10) as

Original: [Equation]
Revised: [Equation]

Original: ) (cid:2) G
Revised: ) (cid:2) G

Revised: [Equation]

Revised: (cid:15)

Original: [Equation]
Revised: [Equation]

Original: γ
Revised: γ

Original: [Equation]
Revised: [Equation]

Original: ] (cid:3)(cid:105) , (26) where G q γ ( y | (cid:15) ) γ = (cid:2) · · · , g q γ ( y v | (cid:15) ) γ , · · · (cid:3) and g q γ ( y v | (cid:15) ) γ (cid:44) − 1 q γ ( y v | (cid:15) ) ∇ γ Q γ ( y v | (cid:15) ) .
Revised: ] (cid:3)(cid:105) , (26) where G q γ ( y | (cid:15) ) γ = (cid:2) · · · , g q γ ( y v | (cid:15) ) γ , · · · (cid:3) and g q γ ( y v | (cid:15) ) γ (cid:44) − 1 q γ ( y v | (cid:15) ) ∇ γ Q γ ( y v | (cid:15) ) .

Original: With q γ ( y v | (cid:15) )
Revised: With q γ ( y v | (cid:15) )

Original: =
Revised: =

Original: δ ( y v − [ τ γ ( (cid:15) )] v ) , we have
Revised: δ ( y v − [ τ γ ( (cid:15) )] v ) , we have

Original: [Equation]
Revised: [Equation]

Original: Accordingly, we have
Revised: Accordingly, we have

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Substituting the above equations into (26), we get
Revised: Substituting the above equations into (26), we get

Original: [Equation]
Revised: [Equation]

Original: which is the multi-dimensional Rep gradient in (5) of the main manuscript.
Revised: which is the multi-dimensional Rep gradient in (5) of the main manuscript.

Original: D P ROOF OF T HEOREM
Revised: D P ROOF OF T HEOREM

Original: Firstly, with the internal variable λ being continuous, (10) and (11) in the main manuscript are proved by (15) and (18) in Section B, respectively.
Revised: Firstly, with the internal variable λ being continuous, (10) and (11) in the main manuscript are proved by (15) and (18) in Section B, respectively.

Original: Then, by iteratively generalizing the similar derivations to deep models and utilizing the fact that the GO gradients with variable-nablas in expectation obey the chain rule for models with continuous internal variables, Theorem 2 could be readily veriﬁed.
Revised: Then, by iteratively generalizing the similar derivations to deep models and utilizing the fact that the GO gradients with variable-nablas in expectation obey the chain rule for models with continuous internal variables, Theorem 2 could be readily veriﬁed.

Original: E P ROOFS FOR C OROLLARY
Revised: E P ROOFS FOR C OROLLARY

Original: When all q γ ( i ) (cid:0) y ( i ) | y ( i − 1) (cid:1) s are speciﬁed as Dirac delta functions, namely
Revised: When all q γ ( i ) (cid:0) y ( i ) | y ( i − 1) (cid:1) s are speciﬁed as Dirac delta functions, namely

Original: [Equation]
Revised: [Equation]

Original: where σ ( γ ( i ) , y ( i − 1) ) denotes the activated values after activation functions, the objective becomes
Revised: where σ ( γ ( i ) , y ( i − 1) ) denotes the activated values after activation functions, the objective becomes

Original: [Equation]
Revised: [Equation]

Original: where γ = { γ (1) , · · · , γ ( N ) }
Revised: where γ = { γ (1) , · · · , γ ( N ) }

Original: Back-Propagation.
Revised: Back-Propagation.

Original: For the objective in (28), the Back-Propagation is expressed as
Revised: For the objective in (28), the Back-Propagation is expressed as

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Deep GO Gradient.
Revised: Deep GO Gradient.

Original: We consider the continuous special case, where D y ( L ) (cid:2)
Revised: We consider the continuous special case, where D y ( L ) (cid:2)

Original: f ( y ( L ) ) (cid:3) = ∇ y
Revised: f ( y ( L ) ) (cid:3) = ∇ y

Original: ( L ) f
Revised: ( L ) f

Original: ( y ( L ) ) .
Revised: ( y ( L ) ) .

Original: With q γ ( i +1) (cid:0) y
Revised: With q γ ( i +1) (cid:0) y

Original: ( i +1)
Revised: ( i +1)

Original: | y
Revised: | y

Original: ( i ) (cid:1) s being Dirac delta functions, namely,
Revised: ( i ) (cid:1) s being Dirac delta functions, namely,

Original: [Equation]
Revised: [Equation]

Revised: y

Revised: [Equation]

Revised: i +

Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Taking derivative wrt y
Revised: Taking derivative wrt y

Original: ( i ) v , we got
Revised: ( i ) v , we got

Original: [Equation]
Revised: [Equation]

Original: Accordingly, we have
Revised: Accordingly, we have

Original: [Equation]
Revised: [Equation]

Original: By substituting the above equation into (12) in Theorem 2, and then comparing it with(29), one can easily verify Corollary 1.
Revised: By substituting the above equation into (12) in Theorem 2, and then comparing it with(29), one can easily verify Corollary 1.

Original: F P ROOF FOR T HEOREM
Revised: F P ROOF FOR T HEOREM

Original: Based on the proofs for Theorem 1 and Theorem 2, it is clear that, if one constrains all internal variables to be continuous, the GO gradients in expectation obey the chain rule.
Revised: Based on the proofs for Theorem 1 and Theorem 2, it is clear that, if one constrains all internal variables to be continuous, the GO gradients in expectation obey the chain rule.

Original: Therefore, one can straightforwardly utilizing the chain rule to verify Theorem 3.
Revised: Therefore, one can straightforwardly utilizing the chain rule to verify Theorem 3.

Original: Actually, Theorem 3 may be seen as the chain rule generalized with random variables, among which the internal ones are only allowed to be continuous.
Revised: Actually, Theorem 3 may be seen as the chain rule generalized with random variables, among which the internal ones are only allowed to be continuous.

Original: G D ERIVATIONS FOR H IERARCHICAL V ARIATIONAL I NFERENCE
Revised: G D ERIVATIONS FOR H IERARCHICAL V ARIATIONAL I NFERENCE

Original: In Hierarchical Variational Inference, the objective is to maximize the evidence lower Bound (ELBO)
Revised: In Hierarchical Variational Inference, the objective is to maximize the evidence lower Bound (ELBO)

Original: [Equation]
Revised: [Equation]

Original: For the common case with z = { z (1) , · · · , z ( L ) } , it is obvious that Theorem 3 of the main manuscript can be applied when optimizing φ .
Revised: For the common case with z = { z (1) , · · · , z ( L ) } , it is obvious that Theorem 3 of the main manuscript can be applied when optimizing φ .

Original: Practically, there are situations where one might further put a latent variable λ in reference q φ ( z | x ) , namely q φ ( z | x ) = (cid:82)
Revised: Practically, there are situations where one might further put a latent variable λ in reference q φ ( z | x ) , namely q φ ( z | x ) = (cid:82)

Original: q φ z ( z | λ ) q φ λ ( λ ) d λ with φ = { φ z , φ λ } .
Revised: q φ z ( z | λ ) q φ λ ( λ ) d λ with φ = { φ z , φ λ } .

Original: Following Ranganath et al.
Revised: Following Ranganath et al.

Original: (2016b), we brieﬂy discuss this situation here.
Revised: (2016b), we brieﬂy discuss this situation here.

Original: We ﬁrst show that there is another unnecessary variance-injecting “0” term.
Revised: We ﬁrst show that there is another unnecessary variance-injecting “0” term.

Original: [Equation]
Revised: [Equation]

Original: where the second “0” term is straightly veriﬁed as
Revised: where the second “0” term is straightly veriﬁed as

Original: [Equation]
Revised: [Equation]

Original: Eliminating the “0” term from (31), one still has another problem, that is, log q φ ( z | x ) is usually non-trivial when q φ ( z | x ) is marginal.
Revised: Eliminating the “0” term from (31), one still has another problem, that is, log q φ ( z | x ) is usually non-trivial when q φ ( z | x ) is marginal.

Original: For this problem, we follow Ranganath et al.
Revised: For this problem, we follow Ranganath et al.

Original: (2016b) to use another lower bound ELBO2 of the ELBO in (30).
Revised: (2016b) to use another lower bound ELBO2 of the ELBO in (30).

Original: [Equation]
Revised: [Equation]

Original: where r ω ( λ | z , x ) , evaluable, is an additional variational distribution to approximate the variational posterior q φ ( λ | z , x ) .
Revised: where r ω ( λ | z , x ) , evaluable, is an additional variational distribution to approximate the variational posterior q φ ( λ | z , x ) .

Original: Accordingly, we get the ELBO2 for Hierarchical Variational Inference as
Revised: Accordingly, we get the ELBO2 for Hierarchical Variational Inference as

Original: [Equation]
Revised: [Equation]

Original: Note similar to (31), the unnecessary “0” term related to log q φ ( z , λ | x ) should also be removed.
Revised: Note similar to (31), the unnecessary “0” term related to log q φ ( z , λ | x ) should also be removed.

Original: Accordingly, we have
Revised: Accordingly, we have

Original: [Equation]
Revised: [Equation]

Original: Obviously, Theorem 3 is readily applicable to provide GO gradients.
Revised: Obviously, Theorem 3 is readily applicable to provide GO gradients.

Original: H D ETAILS OF G AMMA AND NB O NE - DIMENSIONAL S IMPLE E XAMPLES
Revised: H D ETAILS OF G AMMA AND NB O NE - DIMENSIONAL S IMPLE E XAMPLES

Original: We ﬁrst consider illustrative one-dimensional “toy” problems, to examine the GO gradient in Theorem 1 for both continuous and discrete random variables.
Revised: We ﬁrst consider illustrative one-dimensional “toy” problems, to examine the GO gradient in Theorem 1 for both continuous and discrete random variables.

Original: The optimization objective is expressed as
Revised: The optimization objective is expressed as

Original: [Equation]
Revised: [Equation]

Original: where for continuous z we assume p ( z | x ) =
Revised: where for continuous z we assume p ( z | x ) =

Original: Gam( z ; α 0 , β 0 ) for set ( α 0 , β 0 ) , with q φ ( z ) =
Revised: Gam( z ; α 0 , β 0 ) for set ( α 0 , β 0 ) , with q φ ( z ) =

Original: Gam( z ; α, β ) and φ = { α, β } ; for discrete
Revised: Gam( z ; α, β ) and φ = { α, β } ; for discrete

Original: z we assume p ( z | x ) =
Revised: z we assume p ( z | x ) =

Original: NB( z ; r 0 , p 0 ) for set ( r 0 , p 0 ) , with q φ ( z ) =
Revised: NB( z ; r 0 , p 0 ) for set ( r 0 , p 0 ) , with q φ ( z ) =

Original: NB( z ; r, p ) and φ = { r, p } .
Revised: NB( z ; r, p ) and φ = { r, p } .

Original: Stochastic gradient ascent with one-sample-estimated gradients is used to optimize the objective, which is equivalent to minimizing KL ( q φ ( z ) (cid:107) p ( z | x )) .
Revised: Stochastic gradient ascent with one-sample-estimated gradients is used to optimize the objective, which is equivalent to minimizing KL ( q φ ( z ) (cid:107) p ( z | x )) .

Original: Figure 8 shows the experimental results.
Revised: Figure 8 shows the experimental results.

Original: For the nonnegative continuous z associated with the gamma distribution, we compare our GO gradient with GRep (Ruiz et al., 2016), RSVI (Naesseth et al., 2016), and their modiﬁed version using the “sticking” idea (Roeder et al., 2017), denoted as GRep-Stick and
Revised: For the nonnegative continuous z associated with the gamma distribution, we compare our GO gradient with GRep (Ruiz et al., 2016), RSVI (Naesseth et al., 2016), and their modiﬁed version using the “sticking” idea (Roeder et al., 2017), denoted as GRep-Stick and

Original: RSVI-Stick respectively.
Revised: RSVI-Stick respectively.

Original: For RSVI and RSVI-Stick, the shape augmentation parameter is set as 5 by default.
Revised: For RSVI and RSVI-Stick, the shape augmentation parameter is set as 5 by default.

Original: The only difference between GRep and GRep-Stick (also RSVI and RSVI-Stick) is the latter does NOT analytically express the entropy E q φ ( z ) [ − log q φ ( z )] .
Revised: The only difference between GRep and GRep-Stick (also RSVI and RSVI-Stick) is the latter does NOT analytically express the entropy E q φ ( z ) [ − log q φ ( z )] .

Original: One should apply sticking because ( i ) Figures 8(a)-8(c) clearly show its utility in reducing variance; and ( ii ) without it, GRep and RSVI exhibit high variance that unstabilizes the optimization for small gamma shape parameters, as shown in Figures 8(d)-8(f).
Revised: One should apply sticking because ( i ) Figures 8(a)-8(c) clearly show its utility in reducing variance; and ( ii ) without it, GRep and RSVI exhibit high variance that unstabilizes the optimization for small gamma shape parameters, as shown in Figures 8(d)-8(f).

Original: We adopt the sticking approach hereafter for all the compared methods.
Revised: We adopt the sticking approach hereafter for all the compared methods.

Original: Since the gamma rate parameter β is reparameterizable, its gradient calculation is the same for all sticking methods, including GO, GRep-Stick, and RSVI-Stick.
Revised: Since the gamma rate parameter β is reparameterizable, its gradient calculation is the same for all sticking methods, including GO, GRep-Stick, and RSVI-Stick.

Original: Therefore, similar variances are observed in
Revised: Therefore, similar variances are observed in

Original: Figures 8(b) and 8(e).
Revised: Figures 8(b) and 8(e).

Original: Among methods with sticking, GO exhibits the lowest variance in general, as shown in Figures 8(a) and 8(d).
Revised: Among methods with sticking, GO exhibits the lowest variance in general, as shown in Figures 8(a) and 8(d).

Original: Note it is high variance that causes optimization issues.
Revised: Note it is high variance that causes optimization issues.

Original: As a result, GO empirically provides more stable learning curves, as shown in Figures 8(c) and 8(f).
Revised: As a result, GO empirically provides more stable learning curves, as shown in Figures 8(c) and 8(f).

Original: For the discrete case corresponding to the NB distribution, GO is compared to REINFORCE (Williams, 1992).
Revised: For the discrete case corresponding to the NB distribution, GO is compared to REINFORCE (Williams, 1992).

Original: To estimate gradient, REINFORCE uses 1 sample of z and 1 evaluation of the expected function; whereas GO uses 1 sample and 2 evaluations.
Revised: To estimate gradient, REINFORCE uses 1 sample of z and 1 evaluation of the expected function; whereas GO uses 1 sample and 2 evaluations.

Original: To address the concern about comparison with the same number of evaluations of the expected function, another curve of REINFORCE using 2 samples (thus 2 evaluations of the expected function) is also added, termed REINFORCE2.
Revised: To address the concern about comparison with the same number of evaluations of the expected function, another curve of REINFORCE using 2 samples (thus 2 evaluations of the expected function) is also added, termed REINFORCE2.

Original: It is apparent from Figures 8(g)-8(l) that, thanks to analytically removing the “0” terms, the GO gradient has much lower variance and thus faster convergence, even in this simple one-dimensional case.
Revised: It is apparent from Figures 8(g)-8(l) that, thanks to analytically removing the “0” terms, the GO gradient has much lower variance and thus faster convergence, even in this simple one-dimensional case.

Original: I D ETAILS OF THE D ISCRETE VAE E XPERIMENT
Revised: I D ETAILS OF THE D ISCRETE VAE E XPERIMENT

Original: Complementing the discrete VAE experiment of the main manuscript, we present below its experimental settings, implementary details, and additional results.
Revised: Complementing the discrete VAE experiment of the main manuscript, we present below its experimental settings, implementary details, and additional results.

Original: Since the presented statistical back-propagation in Theorem 3 of the main manuscript cannot handle discrete internal variables, we focus on the single-latent-layer settings (1 layer of 200 Bernoulli random variables) for fairness, i.e.,
Revised: Since the presented statistical back-propagation in Theorem 3 of the main manuscript cannot handle discrete internal variables, we focus on the single-latent-layer settings (1 layer of 200 Bernoulli random variables) for fairness, i.e.,

Original: [Equation]
Revised: [Equation]

Original: Referring to the experimental settings in Grathwohl et al.
Revised: Referring to the experimental settings in Grathwohl et al.

Original: (2017), we consider
Revised: (2017), we consider

Original: [Figure]
Revised: [Figure]

Original: • 1-layer linear model:
Revised: • 1-layer linear model:

Original: [Equation]
Revised: [Equation]

Original: where σ ( · ) is the sigmoid function.
Revised: where σ ( · ) is the sigmoid function.

Original: • Nonlinear model:
Revised: • Nonlinear model:

Original: [Equation]
Revised: [Equation]

Original: where tanh( · ) is the hyperbolic-tangent function.
Revised: where tanh( · ) is the hyperbolic-tangent function.

Original: The used datasets and other experimental settings, including the hyperparameter search strategy, are the same as those in Grathwohl et al.
Revised: The used datasets and other experimental settings, including the hyperparameter search strategy, are the same as those in Grathwohl et al.

Original: For such single-latent-layer settings, it is obvious that Theorem 3 (also Theorem 1) can be straightforwardly applied.
Revised: For such single-latent-layer settings, it is obvious that Theorem 3 (also Theorem 1) can be straightforwardly applied.

Original: However, since Bernoulli distribution has ﬁnite support, as mentioned in the main manuscript, we should analytically express some expectations for lower variance, as detailed below.
Revised: However, since Bernoulli distribution has ﬁnite support, as mentioned in the main manuscript, we should analytically express some expectations for lower variance, as detailed below.

Original: Notations of (8) and (9) of the main manuscript are used for clarity and also for generalization.
Revised: Notations of (8) and (9) of the main manuscript are used for clarity and also for generalization.

Original: In fact, we should take a step back and start from (8) of the main manuscript, which is equivalent to analytically express an expectation in (9), namely
Revised: In fact, we should take a step back and start from (8) of the main manuscript, which is equivalent to analytically express an expectation in (9), namely

Original: [Equation]
Revised: [Equation]

Original:  where Q γ ( y v ) = (cid:26) 1 − P v ( γ ) y v = y v = 1 with P v ( γ ) being the Bernoulli probability of Bernoulli random variable y v .
Revised: 33) where Q γ ( y v ) = (cid:26) 1 − P v ( γ ) y v = y v = 1 with P v ( γ ) being the Bernoulli probability of Bernoulli random variable y v .

Original: Accordingly, we have
Revised: Accordingly, we have

Original: [Equation]
Revised: [Equation]

Original: For better understanding only, with abused notations ∇ γ P = [ · · · , ∇ γ P v ( γ ) , · · · ] T , ∇ P y
Revised: For better understanding only, with abused notations ∇ γ P = [ · · · , ∇ γ P v ( γ ) , · · · ] T , ∇ P y

Original: =
Revised: =

Original: I , and ∇ y f ( y )
Revised: I , and ∇ y f ( y )

Original: =
Revised: =

Original: [ · · · , { f ( y − v , y v = 1) − f
Revised: [ · · · , { f ( y − v , y v = 1) − f

Original: ( y − v , y v = 0) } , · · · ] T , one should observe a chain rule within the above equation.
Revised: ( y − v , y v = 0) } , · · · ] T , one should observe a chain rule within the above equation.

Original: To assist better understanding of how to practically cooperate the presented GO gradients with deep learning frameworks like TensorFlow or PyTorch, we take (34) as an example, and present for it the following simple algorithm.
Revised: To assist better understanding of how to practically cooperate the presented GO gradients with deep learning frameworks like TensorFlow or PyTorch, we take (34) as an example, and present for it the following simple algorithm.

Original: Algorithm 1 An algorithm for (34) as an example to demonstrate how to practically cooperate GO gradients with deep learning frameworks like TensorFlow or PyTorch.
Revised: Algorithm 1 An algorithm for (34) as an example to demonstrate how to practically cooperate GO gradients with deep learning frameworks like TensorFlow or PyTorch.

Original: One sample is assumed for clarity.
Revised: One sample is assumed for clarity.

Original: Practically, an easy-to-use trick for changing gradients of any function h ( x ) is to deﬁne ˆ h ( x ) = x T StopGradient [ g ] + StopGradient [ h ( x ) − x T g ] with g the desired gradients.
Revised: Practically, an easy-to-use trick for changing gradients of any function h ( x ) is to deﬁne ˆ h ( x ) = x T StopGradient [ g ] + StopGradient [ h ( x ) − x T g ] with g the desired gradients.

Original: Forward-
Revised: # Forward-Propagation

Original: Propagation

Original: γ → P ( γ ) :
Revised: γ → P ( γ ) :

Original: Calculate Bernoulli probabilities P ( γ ) 2.
Revised: Calculate Bernoulli probabilities P ( γ ) 2.

Original: P ( γ ) →
Revised: P ( γ ) →

Original: y : Sample y ∼ Bern( P ( γ ))
Revised: y : Sample y ∼ Bern( P ( γ ))

Original: (cid:46) Change gradient: ∇ P ( γ )
Revised: (cid:46) Change gradient: ∇ P ( γ )

Original: y
Revised: y

Original: = I 3.
Revised: = I 3.

Original: y
Revised: y

Original: → f ( y ) :
Revised: → f ( y ) :

Original: Calculate the loss f ( y ) (cid:46) Change gradient: [ ∇ y f ( y )] v = f
Revised: Calculate the loss f ( y ) (cid:46) Change gradient: [ ∇ y f ( y )] v = f

Original: ( y − v , y v = 1) − f
Revised: ( y − v , y v = 1) − f

Original: ( y − v , y v = 0)
Revised: ( y − v , y v = 0)

Original: Back
Revised: Back

Original: -Propagation
Revised: -Propagation

Original: Rely on the mature auto-differential software for back-propagating gradients
Revised: Rely on the mature auto-differential software for back-propagating gradients

Original: For efﬁcient implementation of D y f ( y ) , one should exploit the prior knowledge of function f ( y ) .
Revised: For efﬁcient implementation of D y f ( y ) , one should exploit the prior knowledge of function f ( y ) .

Original: For example, f ( y ) s are often neural-network-parameterized.
Revised: For example, f ( y ) s are often neural-network-parameterized.

Original: Under that settings, one should be able to exploit tensor operation to design efﬁcient implementation of D y f ( y ) .
Revised: Under that settings, one should be able to exploit tensor operation to design efﬁcient implementation of D y f ( y ) .

Original: Again we take (34) as an example, and assume f ( y ) has the special structure
Revised: Again we take (34) as an example, and assume f ( y ) has the special structure

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where σ ( · ) is an element-wise nonlinear activation function, and r ( · ) is a function that takes in a vector and outputs a scalar.
Revised: where σ ( · ) is an element-wise nonlinear activation function, and r ( · ) is a function that takes in a vector and outputs a scalar.

Original: One can easily modify the above f ( y ) for the considered discrete VAE experiment.
Revised: One can easily modify the above f ( y ) for the considered discrete VAE experiment.

Original: Since y v s are now Bernoulli random variables with support { 0 , 1 } , we have that
Revised: Since y v s are now Bernoulli random variables with support { 0 , 1 } , we have that

Original: [Equation]
Revised: [Equation]

Original: where a v =
Revised: where a v =

Original: (cid:26)y v =−y v = 1 is the v -th element of vector
Revised: (cid:26)y v =−y v = 1 is the v -th element of vector

Original: a
Revised: a

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure][Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: Then to efﬁciently calculate the f ( y − v , y v + a v ) s, we use the following batch processing procedure to beneﬁt from parallel computing.
Revised: Then to efﬁciently calculate the f ( y − v , y v + a v ) s, we use the following batch processing procedure to beneﬁt from parallel computing.

Original: • Step 1: Deﬁne Ξ y h as the matrix whose element [Ξ y h ] vj represents the “new” h ∗ j when input { y − v , y v + a v } in (35).
Revised: • Step 1: Deﬁne Ξ y h as the matrix whose element [Ξ y h ] vj represents the “new” h ∗ j when input { y − v , y v + a v } in (35).

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: where W : j is the j -th column of the matrix W .
Revised: where W : j is the j -th column of the matrix W .

Original: Note the v th row of Ξ y h , i.e., [Ξ y h ] v : , happens to be the “new” h ∗ when input { y − v , y v + a v } .
Revised: Note the v th row of Ξ y h , i.e., [Ξ y h ] v : , happens to be the “new” h ∗ when input { y − v , y v + a v } .

Original: • Step 2: Similarly, we deﬁne Ξ y f as the vector whose element [Ξ y f ] v = f
Revised: • Step 2: Similarly, we deﬁne Ξ y f as the vector whose element [Ξ y f ] v = f

Original: ( y − v , y v + a v ) .
Revised: ( y − v , y v + a v ) .

Original: Utilizing Ξ y h obtained in Step 1, we have
Revised: Utilizing Ξ y h obtained in Step 1, we have

Original: [Equation]
Revised: [Equation]

Original: where r ( · ) is applied to each row of the matrix ([Ξ y h ] ϑ + c T ) .
Revised: where r ( · ) is applied to each row of the matrix ([Ξ y h ] ϑ + c T ) .

Original: Note the above batch processing procedure can be easily extended to deeper neural networks.
Revised: Note the above batch processing procedure can be easily extended to deeper neural networks.

Original: Accordingly, we have
Revised: Accordingly, we have

Original: [Equation]
Revised: [Equation]

Original: where (cid:12) represents the matrix element-wise product.
Revised: where (cid:12) represents the matrix element-wise product.

Original: Now we can rely on Algorithm 1 to solve the problem whose objective has its gradient expressed as (34), for example the inference of the single-latent-layer discrete VAE in (32).
Revised: Now we can rely on Algorithm 1 to solve the problem whose objective has its gradient expressed as (34), for example the inference of the single-latent-layer discrete VAE in (32).

Original: All training curves versus iteration/running-time are given in Figure 9, where it is apparent that
Revised: All training curves versus iteration/running-time are given in Figure 9, where it is apparent that

Original: GO provides better performance, a faster convergence rate, and a better running efﬁciency in all situations.
Revised: GO provides better performance, a faster convergence rate, and a better running efﬁciency in all situations.

Original: The average running time per 100 iterations for the compared methods are given in Table 4, where GO is 2 − 4 times faster in ﬁnishing the same number of training iterations.
Revised: The average running time per 100 iterations for the compared methods are given in Table 4, where GO is 2 − 4 times faster in ﬁnishing the same number of training iterations.

Original: We also quantify the running efﬁciency of GO by considering its running time to achieve the best training ELBO (
Revised: We also quantify the running efﬁciency of GO by considering its running time to achieve the best training ELBO (

Original: within 1 , 000 , 000 training iterations) of RERAR/RELAX, referring to the black lines shown in the second-column subﬁgures of Figure 9.
Revised: within 1 , 000 , 000 training iterations) of RERAR/RELAX, referring to the black lines shown in the second-column subﬁgures of Figure 9.

Original: It is clear that GO is approximately 5 − 10 times more efﬁcient than REBAR/RELAX in the considered experiments.
Revised: It is clear that GO is approximately 5 − 10 times more efﬁcient than REBAR/RELAX in the considered experiments.

Original: As shown in the second and fourth rows of Figures 9, for the experiments with nonlinear models all methods suffer from over-ﬁtting, which originates from the redundant complexity of the adopted neural networks and appeals for model regularizations.
Revised: As shown in the second and fourth rows of Figures 9, for the experiments with nonlinear models all methods suffer from over-ﬁtting, which originates from the redundant complexity of the adopted neural networks and appeals for model regularizations.

Original: We detailedly clarify these experimental results as follows
Revised: We detailedly clarify these experimental results as follows

Original: • All the compared methods are given the same and only objective, namely to maximize the training ELBO on the same training dataset with the same model; GO clearly shows its power in achieving a better objective.
Revised: • All the compared methods are given the same and only objective, namely to maximize the training ELBO on the same training dataset with the same model; GO clearly shows its power in achieving a better objective.

Original: • The “level” of over-ﬁtting is ultimately determined by the used dataset, model, and objective; it is independent of the adopted optimization method.
Revised: • The “level” of over-ﬁtting is ultimately determined by the used dataset, model, and objective; it is independent of the adopted optimization method.

Original: Different optimization methods just reveal different optimizing trajectories, which show different sequences of training objectives and over-ﬁtting levels (validation objectives).
Revised: Different optimization methods just reveal different optimizing trajectories, which show different sequences of training objectives and over-ﬁtting levels (validation objectives).

Original: • Since all methods are given the same dataset, model, and objective, they have the same over-ﬁtting level.
Revised: • Since all methods are given the same dataset, model, and objective, they have the same over-ﬁtting level.

Original: Because GO has a lower variance, and thus more powerful optimization capacity, it gets to the similar situations much faster than REBAR/RELAX.
Revised: Because GO has a lower variance, and thus more powerful optimization capacity, it gets to the similar situations much faster than REBAR/RELAX.

Original: Note this does not mean GO suffers more from over-ﬁtting.
Revised: Note this does not mean GO suffers more from over-ﬁtting.

Original: In fact, GO provides better validation ELBOs in all situations, as shown in
Revised: In fact, GO provides better validation ELBOs in all situations, as shown in

Original: Figure
Revised: Figure

Original: and also Table
Revised: and also Table

Original: of the main manuscript.
Revised: of the main manuscript.

Original: In practice, GO can beneﬁt from the early-stopping trick to get a better generalization ability.
Revised: In practice, GO can beneﬁt from the early-stopping trick to get a better generalization ability.

Original: J D ETAILS OF THE M ULTINOMIAL GAN
Revised: J D ETAILS OF THE M ULTINOMIAL GAN

Original: Complementing the multinomial GAN experiment in the main manuscript, we present more details as follows.
Revised: Complementing the multinomial GAN experiment in the main manuscript, we present more details as follows.

Original: For a quantitative assessment of the computational complexity, our PyTorch code takes about 30 minutes to get the most challenging 4-bit task in Figs.
Revised: For a quantitative assessment of the computational complexity, our PyTorch code takes about 30 minutes to get the most challenging 4-bit task in Figs.

Original: 4 and 11, with a Titan Xp GPU.
Revised: 4 and 11, with a Titan Xp GPU.

Original: Firstly, recall that the generator p θ ( x ) of the developed multinomial GAN (denoted as MNGAN-GO) is expressed as
Revised: Firstly, recall that the generator p θ ( x ) of the developed multinomial GAN (denoted as MNGAN-GO) is expressed as

Original: [Equation]
Revised: [Equation]

Original: where NN P ( (cid:15) ) denotes use of a neural network to project (cid:15) to distribution parameters P .
Revised: where NN P ( (cid:15) ) denotes use of a neural network to project (cid:15) to distribution parameters P .

Original: For brevity, we integration the generator’s parameters θ into the NN notation and do not explicitly express them.
Revised: For brevity, we integration the generator’s parameters θ into the NN notation and do not explicitly express them.

Original: Multinomial leaf variables x is used to describe discrete observations with a ﬁnite alphabet.
Revised: Multinomial leaf variables x is used to describe discrete observations with a ﬁnite alphabet.

Original: To train
Revised: To train

Original: MNGAN-GO, the vanilla GAN loss (Goodfellow et al., 2014) is used.
Revised: MNGAN-GO, the vanilla GAN loss (Goodfellow et al., 2014) is used.

Original: A deconvolutional neural network as in Radford et al.
Revised: A deconvolutional neural network as in Radford et al.

Original: (2015) is used to map (cid:15) to P in the generator.
Revised: (2015) is used to map (cid:15) to P in the generator.

Original: The discriminator is constructed as a multilayer perceptron.
Revised: The discriminator is constructed as a multilayer perceptron.

Original: Detailed model architectures are given in Table 5.
Revised: Detailed model architectures are given in Table 5.

Original: Figure 10 illustrates the pipeline of MNGAN-GO.
Revised: Figure 10 illustrates the pipeline of MNGAN-GO.

Original: Note MNGAN-GO has a smaller number of parameters, compared to BGAN (Hjelm et al., 2018).
Revised: Note MNGAN-GO has a smaller number of parameters, compared to BGAN (Hjelm et al., 2018).

Original: For clarity, we brieﬂy discuss the employed data preprocessing.
Revised: For clarity, we brieﬂy discuss the employed data preprocessing.

Original: Taking MNIST for an example, the original data are 8 -bit grayscale images, with pixel intensities ranging from 0 to 255 .
Revised: Taking MNIST for an example, the original data are 8 -bit grayscale images, with pixel intensities ranging from 0 to 255 .

Original: For the n -bit experiment, we obtain the real data, such as the 2 -bit one in Figure 10, by rescaling and quantizing the pixel intensities to the range [0 , 2 n − 1] , having 2 n different states (values).
Revised: For the n -bit experiment, we obtain the real data, such as the 2 -bit one in Figure 10, by rescaling and quantizing the pixel intensities to the range [0 , 2 n − 1] , having 2 n different states (values).

Original: For the 1 -bit special case, the multinomial distribution reduces to the Bernoulli distribution.
Revised: For the 1 -bit special case, the multinomial distribution reduces to the Bernoulli distribution.

Original: Of course, one could intuitively employ the redundant multinomial distribution, which is denoted as 1-bit (2-state) in Table 2.
Revised: Of course, one could intuitively employ the redundant multinomial distribution, which is denoted as 1-bit (2-state) in Table 2.

Original: An alternative and popular approach is to adopt the Bernoulli distribution to remove the redundancy by only modeling its probability parameters; we denote this case as 1-bit (1-state, Bernoulli) in Table 2.
Revised: An alternative and popular approach is to adopt the Bernoulli distribution to remove the redundancy by only modeling its probability parameters; we denote this case as 1-bit (1-state, Bernoulli) in Table 2.

Original: Figure 11 shows the generated samples from the compared models on different quantized MNIST.
Revised: Figure 11 shows the generated samples from the compared models on different quantized MNIST.

Original: It is obvious that MNGAN-GO provides images with better quality and wider diversity in general.
Revised: It is obvious that MNGAN-GO provides images with better quality and wider diversity in general.

Original: K D ETAILS OF HVI FOR DEF AND DLDA
Revised: K D ETAILS OF HVI FOR DEF AND DLDA

Original: Complementing the HVI experiments in the main manuscript, we present more details as follows.
Revised: Complementing the HVI experiments in the main manuscript, we present more details as follows.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: For a quantitative assessment of the computational complexity
Revised: For a quantitative assessment of the computational complexity

Original: [Table]
Revised: [Table]

Original: the TensorFlow code used takes about 0 .
Revised: the TensorFlow code used takes about 0 .

Original: 08 seconds per iteration (including > 40 , 000 Meijer-G function calculations, with an approximate algorithm coded also with TensorFlow)
Revised: 08 seconds per iteration (including > 40 , 000 Meijer-G function calculations, with an approximate algorithm coded also with TensorFlow)

Original: [Table]
Revised: [Table]

Original: Deep exponential families (DEF) (Ranganath et al., 2015) is expressed as
Revised: Deep exponential families (DEF) (Ranganath et al., 2015) is expressed as

Original: [Equation]
Revised: [Equation]

Original: where α z = 0 . 1 , α 0 = 0 .
Revised: where α z = 0 . 1 , α 0 = 0 .

Original: 3 , and β 0 = 0 .
Revised: 3 , and β 0 = 0 .

Original: 1 following Ruiz et al.
Revised: 1 following Ruiz et al.

Original: ; Naesseth et al.
Revised: ; Naesseth et al.

Original: Since the variables of interest, W ( l ) and z ( l ) , are gamma-distributed, we design the variational approximations as
Revised: Since the variables of interest, W ( l ) and z ( l ) , are gamma-distributed, we design the variational approximations as

Original: [Equation]
Revised: [Equation]

Original: where NN ( l ) ( · ) s are set to the same shapes as the corresponding z ( l ) , α ( l ) W and β ( l ) W also have the shape of W ( l ) .
Revised: where NN ( l ) ( · ) s are set to the same shapes as the corresponding z ( l ) , α ( l ) W and β ( l ) W also have the shape of W ( l ) .

Original: We employ a simple two-layer DEF for demonstration, with z (1) and z (2) havingand 64 components, respectively.
Revised: We employ a simple two-layer DEF for demonstration, with z (1) and z (2) havingand 64 components, respectively.

Original: The mini-batch size is set to 200.
Revised: The mini-batch size is set to 200.

Original: One-sample gradient estimates are used to train the model for the compared methods.
Revised: One-sample gradient estimates are used to train the model for the compared methods.

Original: For RSVI (Naesseth et al., 2016), the shape augmentation parameter B is set to 5.
Revised: For RSVI (Naesseth et al., 2016), the shape augmentation parameter B is set to 5.

Original: All experimental settings, except for different ways to calculate gradients, are the same for the compared methods.
Revised: All experimental settings, except for different ways to calculate gradients, are the same for the compared methods.

Original: Deep latent Dirichlet allocation (DLDA)
Revised: Deep latent Dirichlet allocation (DLDA)

Original: (Zhou et al., 2015; 2016;
Revised: (Zhou et al., 2015; 2016;

Original: Cong et al., 2017) is expressed as
Revised: Cong et al., 2017) is expressed as

Original: [Equation]
Revised: [Equation]

Original: Pois
Revised: Pois

Original: [Equation]
Revised: [Equation]

Original: (37) where hyperparameters are chosen following Cong et al.
Revised: (37) where hyperparameters are chosen following Cong et al.

Original: Compared to DEF, DLDA is more complicated: 1) model is constructed via the highly nonlinear Gamma shape parameters; 2) dictionaries are described by more challenging Dirichlet distributions; 3) more interested random variables
Revised: Compared to DEF, DLDA is more complicated: 1) model is constructed via the highly nonlinear Gamma shape parameters; 2) dictionaries are described by more challenging Dirichlet distributions; 3) more interested random variables

Original: [Equation]
Revised: [Equation]

Original: The variational approximations for DLDA are designed as
Revised: The variational approximations for DLDA are designed as

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: where motivated by the original upward-downward Gibbs sampler developed in Zhou et al.
Revised: where motivated by the original upward-downward Gibbs sampler developed in Zhou et al.

Original: , we specify NN ( l ) β z ( · ) as a scaler to mimic the Gibbs conditional posteriors.
Revised: , we specify NN ( l ) β z ( · ) as a scaler to mimic the Gibbs conditional posteriors.

Original: The other NN ( · ) s and also η ( l ) , α r , β r have the same shapes of the corresponding random variables.
Revised: The other NN ( · ) s and also η ( l ) , α r , β r have the same shapes of the corresponding random variables.

Original: A three-layer DLDA, having 128, 64, 32 components for latent z (1) , z (2) , z (3) respectively, is trained on MNIST with mini-batches of size 200.
Revised: A three-layer DLDA, having 128, 64, 32 components for latent z (1) , z (2) , z (3) respectively, is trained on MNIST with mini-batches of size 200.

Original: With the learned variational inference nets, one could efﬁciently project the observed x to its latent variables z , c during testing.
Revised: With the learned variational inference nets, one could efﬁciently project the observed x to its latent variables z , c during testing.

Original: For applications requiring realtime processing, this is a clear advantage.
Revised: For applications requiring realtime processing, this is a clear advantage.

Original: For demonstration, Figure 12 shows test data samples x and their reconstruction
Revised: For demonstration, Figure 12 shows test data samples x and their reconstruction

Original: [Equation]
Revised: [Equation]

Original: where one Monte Carlo sample is used to calculate ˆ z (1) .
Revised: where one Monte Carlo sample is used to calculate ˆ z (1) .

Original: [Figure]
Revised: [Figure]

Original: Note for the challenging DLDA task in (37), we ﬁnd it tricky to naively apply pure-gradient-based learning methods.
Revised: Note for the challenging DLDA task in (37), we ﬁnd it tricky to naively apply pure-gradient-based learning methods.

Original: The main reason is: the latent code z ( l ) s and their gamma shape parameters Φ ( l +1)
Revised: The main reason is: the latent code z ( l ) s and their gamma shape parameters Φ ( l +1)

Original: z ( l +1) s are usually extremely sparse, meaning most elements are almost zero; a gamma distribution z ∼ Gam( α, β ) with almost-zero α has an increasingly steep slope when z approaches zero, namely the gradient wrt z shall have an enormous magnitude that unstablize the learning procedure.
Revised: z ( l +1) s are usually extremely sparse, meaning most elements are almost zero; a gamma distribution z ∼ Gam( α, β ) with almost-zero α has an increasingly steep slope when z approaches zero, namely the gradient wrt z shall have an enormous magnitude that unstablize the learning procedure.

Original: Even though it might not be sufﬁcient to just use the ﬁrst-order gradient information, empirically the following tricks help us get the presented reasonable results.
Revised: Even though it might not be sufﬁcient to just use the ﬁrst-order gradient information, empirically the following tricks help us get the presented reasonable results.

Original: • Let z ( l ) ≥ T z , where T z = 1 e − 5 is used in the experiments; • Let c ( l ) ≥ T c , where T c = 1 e − 5 ; • Let Φ ( l +1)
Revised: • Let z ( l ) ≥ T z , where T z = 1 e − 5 is used in the experiments; • Let c ( l ) ≥ T c , where T c = 1 e − 5 ; • Let Φ ( l +1)

Original: z ( l +1) ≥ T α with T α = 0 . 2 ; • Use a factor to compromise the likelihood and prior for each z ( l ) .
Revised: z ( l +1) ≥ T α with T α = 0 . 2 ; • Use a factor to compromise the likelihood and prior for each z ( l ) .

Original: For more details, please refer to our released code.
Revised: For more details, please refer to our released code.

Original: We are working on exploiting higher-order information (such as Hessian) to help remedy this issue.
Revised: We are working on exploiting higher-order information (such as Hessian) to help remedy this issue.