Original: L OWER B OUNDING R ATE -D ISTORTION F ROM S AMPLES
Revised: L OWER B OUNDING R ATE -D ISTORTION F ROM S AMPLES

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The rate-distortion function R ( D ) tells us the minimal number of bits on average to compress a random object within a given distortion tolerance.
Revised: The rate-distortion function R ( D ) tells us the minimal number of bits on average to compress a random object within a given distortion tolerance.

Original: A lower bound on R ( D ) therefore represents a fundamental limit on the best possible rate-distortion performance of any lossy compression algorithm, and can help us assess the potential room for improvement.
Revised: A lower bound on R ( D ) therefore represents a fundamental limit on the best possible ratedistortion performance of any lossy compression algorithm, and can help us assess the potential room for improvement.

Original: We make a ﬁrst attempt at an algorithm for computing such a lower bound, applicable to any data source that we have samples of.
Revised: We make a ﬁrst attempt at an algorithm for computing such a lower bound, applicable to general memoryless data sources that we have samples of.

Original: Based on a dual characterization of R ( D ) in terms of constrained maximization, our method approximates the exact constraint function by an asymptotically unbiasedsample-based estimator, allowing for stochastic optimization.
Revised: Based on a dual characterization of R ( D ) in terms of constrained maximization, our method approximates the exact constraint function by an asymptotically unbiased estimator, allowing for stochastic optimization.

Original: On a 2D Gaussian source, we obtain a lower bound within 1 bit of the true R ( D ) on average.
Revised: On a 2D Gaussian source, we obtain a lower bound within 1 bit of the analytical R ( D ) .

Original:  I NTRODUCTION AND B ACKGROUND
Revised:  I NTRODUCTION AND B ACKGROUND

Original: Let X ∈ X be a random variable with distribution P X 1 that represents a memoryless data source, let
Revised: Let X ∈ X be a random variable with distribution P X 1 that represents a memoryless data source, let

Original: Y be the set of lossy representations, and let ρ : X × Y → [0 , ∞ ) be a distortion cost function.
Revised: Y be the set of lossy representations, and let ρ : X × Y → [0 , ∞ ) be a distortion cost function.

Original: The (information) rate-distortion function R ( D ) is deﬁned as the solution to the optimization problem,
Revised: The (information) rate-distortion function R ( D ) is deﬁned as the solution to the optimization problem,

Original: [Equation]
Revised: [Equation]

Original: (1) where we consider all stochastic transforms P Y | X whose expected distortion is within the given threshold D , and minimize the mutual information between the source X and its reproduced representation Y .
Revised: (1) where we consider all stochastic transforms P Y | X whose expected distortion is within the given threshold D , and minimize the mutual information between the source X and its reproduced representation Y .

Original: Rate-distortion theory (Shannon, 1959; Cover, 1999) gives operational meaning to the above deﬁnition as the asymptotic fundamental limit of lossy data compression.
Revised: Rate-distortion theory (Shannon, 1959; Cover, 1999) gives operational meaning to the above deﬁnition as the asymptotic fundamental limit of lossy data compression.

Original: In essence, R ( D ) gives the minimal average number of bits (or nats, depending the base of log ) needed to convey i.i.d.
Revised: In essence, R ( D ) delimits the minimal average number of bits (or nats, depending the base of log ) needed to convey i.i.d.

Original: samples of X within average distortion D , for any block code that is allowed arbitrarily long blocks.
Revised: samples of X within average distortion D , for any block code that is allowed arbitrarily long blocks or high complexity, regardless of its implementation technology.

Original: Therefore, the rate-distortion function, or a lower bound on the rate-distortion function, may help analyze how close lossy compression algorithms are to their theoretical performance limits, and is the motivation behind this work.
Revised: Therefore, the ratedistortion function, or a lower bound on it, can help analyze how close lossy compression algorithms are to their theoretical performance limits.

Original: If the operational rate and distortion of a compression algorithm lies far above the R ( D ) curve (or its lower bound), then we may expect room for further improvement; otherwise, the algorithm’s rate-distortion performance is already close to the theoretical optimum, so we may better focus our attention on improving other aspects of the algorithm.
Revised: If the operational rate and distortion of a compression algorithm lies far above the R ( D ) curve (or its lower bound), then we may expect room for further improvement; otherwise, the algorithm’s rate-distortion performance is already close to theoretically optimal, so we may better focus our attention on improving other aspects of the algorithm.

Original: Although R ( D ) has no analytical form in general, it can be computed in principle by solving the convex optimization problem Eq.
Revised: Although R ( D ) has no analytical form in general, it can be computed in principle by solving the convex optimization problem Eq.

Original: The mutual information I ( X ; Y ) , however, is typically challenging to compute, so a tractable upper bound I ( X ; Y ) ≤ I ( X [Equation] Y )
Revised: The mutual information I ( X ; Y ) , however, is typically challenging to compute, so a tractable upper bound I ( X ; Y ) ≤ I ( X ; Y )

Original: + KL ( P Y (cid:107) ˜
Revised: + KL ( P Y (cid:107) ˜

Original: P 
Revised: P Y )

Original: [Equation] ) =E X [Equation]
Revised: =

Original:  ( P Y [Equation] X (cid:107) ˜ P Y )] is often used instead in the optimization problem.
Revised: E X [ KL ( P Y | X (cid:107) ˜ P Y )] is often used instead in the optimization problem.

Original: Solving the resulting minimization problem then leads to upper bounds on R ( D ) , and is the basis of the Blahut–Arimoto algorithm (Arimoto, 1972).
Revised: Solving the resulting minimization problem then leads to upper bounds on R ( D ) , and is the basis of the Blahut–Arimoto algorithm (Arimoto, 1972).

Original: The same mutual information upper bound has also made numerous appearances in machine learning (Alemi et al., 2017; Poole et al., 2019), notably as the aggregate KL-divergence regularizer in
Revised: The same mutual information upper bound has also made numerous appearances in machine learning (Alemi et al., 2017; Poole et al., 2019), notably as the aggregate KL-divergence regularizer in

Original: Variational-Autoencoders (VAEs) (Kingma & Welling, 2013).
Revised: Variational-Autoencoders (VAEs) (Kingma & Welling, 2013).

Original: Traditionally, the set of lossy presentations Y is often a subset of the data space X , in the form of quantization points, and distortion ρ is ﬁxed.
Revised: Traditionally, the set of lossy presentations Y is often a subset of the data space X , in the form of quantization points, and distortion ρ is ﬁxed.

Original: In VAEs and their recent application to learned data compression (Ball´e et al., 2017), Y has theinterpretation of a latent space , ˜ P Y is the prior , and ρ ( x, y )
Revised: In VAEs and their recent application to learned data compression (Ball´e et al., 2017), Y has the of a latent space , ˜ P Y is the prior , and ρ ( x, y )

Original: =
Revised: =

Original: ρ 0 ( x, f θ ( y )) corresponds to the negative log likelihood log p θ ( x | y ) parameterized by a decoder f
Revised: ρ 0 ( x, f θ ( y )) corresponds to the negative log likelihood log p θ ( x | y ) parameterized by a decoder f

Original: θ : Y → X (e.g., p θ ( x | y ) is a Gaussian density with mean f θ ( y ) , ρ 0 is a squared error), so the distortion ρ is learned.
Revised: θ : Y → X (e.g., p θ ( x | y ) is a Gaussian density with mean f θ ( y ) , ρ 0 is a squared error), so the distortion ρ is learned.

Original: Thenegative ELBO (Kingma & Welling, 2013) then corresponds to the Lagrangian E X [ KL ( P Y |
Revised: The

Original: [Equation] (cid:107) ˜ P Y )]
Revised: Lagrangian E X [ KL ( P Y | X (cid:107) ˜ P Y )]

Original: + λ E
Revised: + λ E

Original: [ ρ ( X, Y )] , which provides the basis for learning.
Revised: [ ρ ( X, Y )] then corresponds to the negative ELBO (Kingma & Welling, 2013) for learning and inference. Using the analytical density of P Y | X that optimizes the

Original:  Huang et al.
Revised: Lagrangian (assumed to exist), Huang et al.

Original: (2020) apply Annealed Importance Sampling to optimizing the Lagrangian, giving optimistic upper bounds on R ( D ) for trained VAEs and other generative models.
Revised: (2020) sample from this optimal P Y | X to compute optimistic upper bounds on R ( D ) of trained VAEs and other generative models, hinting at their possible compression performance.

Original: Like (Huang et al., 2020), our lower bound for R ( D ) is agnostic to the choice of representations Y or distortion ρ , and can  bedirectly extended to compute R ( D ) lower bounds of trained generative models; however, unlike the upper bound of (Huang et al., 2020) which tells us what kind of rate-distortion performance is theoretically possible, ourR ( D ) lower bound establishes what kind of rate-distortion performance isso good that it is theoretically impossible to obtain.
Revised: Our method is agnostic to the choice of representations Y or distortion ρ , and can also be extended to compute R ( D ) lower bounds of trained decoder-based models. Unlike (Huang et al., 2020), however, we do not assume (or require) a prior distribution ˜ P Y , and our lower bound establishes what kind of lossy compression performance is theoretically impossible to obtain.

Original: Lower bounds of R ( D ) , by contrast, have received considerably less attention, potentially because of difﬁculty in obtaining them.
Revised: Lower bounds of R ( D ) , by contrast, have received considerably less attention, potentially because of difﬁculty in obtaining them.

Original: A variational lower bound of R ( D ) already appeared in Shannon’s landmark paper (Shannon, 1948), which was later extended (Berger, 1971) and proved rigorously for X belonging to a general abstract probability space (Csiszar, 1974); the versionappearning in (Kostina, 2016) is  the basis for this work.
Revised: A variational lower bound of R ( D ) already appeared in Shannon’s landmark paper (Shannon, 1948), which was later extended (Berger, 1971) and proved rigorously for X in a general abstract probability space by Csisz´ar (1974); we adopt the version in (Kostina, 2016). The special case of a discrete and known source P X is considered by Chiang & Boyd (2004), who solve the corresponding (ﬁnite-dimensional) dual problem of R ( D ) by geometric programming.

Original:  M ETHOD 2. D UAL FORMULATION OF R ( D )
Revised:  M ETHOD 2. D UAL CHARACTERIZATION OF R ( D )

Original: We employ an alternative characterization of R ( D ) in terms of a constrained maximization problem:
Revised: We use the below general characterization of R ( D ) in terms of a constrained maximization problem:

Original: Theorem 2.1.
Revised: Theorem 2.1.

Original: (Kostina, 2016)
Revised: (Kostina, 2016)

Original: Under basic assumptions that avoids degeneracy (see Appendix Section.
Revised: Under basic regularity conditions (e.g., satisﬁed by a bounded ρ ; see

Original:  A.1), for each distortion tolerance D > D min : 
Revised: Appendix Sec. A.1), for any distortion tolerance D > D min : = inf { D

Original: = inf { D : R ( D ) ≤ ∞} , we have that
Revised: [Equation] R ( D [Equation] ≤ ∞} , it holds that

Original: [Equation]
Revised: [Equation]

Original: where the maximization is over g ( x ) ≥ 0 and λ ≥ 0 satisfying the 
Revised: where the maximization is over g ( x ) ≥ 0 and λ ≥ 0 satisfying the constraint

Original: [Equation]
Revised: [Equation]

Original: In other words, every pair of feasible ( λ, g ) satisfying Eq.
Revised: In other words, every pair of feasible ( λ, g ) satisfying Eq.

Original: 3 yields a lower bound of R ( D ) , i.e., E [ − log g ( X )] − λD ≤ R ( D ) , and we obtain R ( D ) via the tightest such lower bound.
Revised: 3 yields a lower bound of R ( D ) , i.e., E [ − log g ( X )] − λD ≤ R ( D ) , and we obtain R ( D ) via the tightest such lower bound.

Original: This characterization of R ( D ) can be derived from Lagrange duality (Chiang & Boyd, 2004) in the case of a ﬁnite alphabet X ; the extension to a general X requires a more delicate argument (Csiszar, 1974).
Revised: This characterization of R ( D ) can be derived from Lagrange duality (Chiang & Boyd, 2004) in the case of a ﬁnite alphabet X ; the extension to a general X requires a more involved argument (Csisz´ar, 1974).

Original: 2. L OWER B OUNDING R ( D ) VIA NUMERICAL OPTIMIZATION
Revised: 2. L OWER B OUNDING R ( D ) VIA NUMERICAL OPTIMIZATION

Original: Unfortunately, P X is rarely known to us; e.g., when P X is induced by natural images, characterizing the support of P X (which is a low dimensional manifold of Euclidean space) alone is non-trivial.
Revised: We rarely know P X explicitly; e.g., when P X is induced by natural images, characterizing the support of P X (a low dimensional manifold of Euclidean space) alone is non-trivial.

Original: However, if we can compute the expectations in Eqs. 2 and 3, then we can still obtain a lower bound of R ( D ) by numerically solving a constrained optimization problem.
Revised: However, if we can compute the expectations in Eqs. 2 and 3, then we can still obtain a lower bound of R ( D ) by numerically solving a constrained optimization problem.

Original: The main idea is to replace maximizing over a function g ≥ 0 by a subset of functions g θ ≥ 0 parameterized by a vector θ .
Revised: The main idea is to replace maximizing over functions g ≥ 0 by a subset of functions g θ ≥ 0 parameterized by a vector θ .

Original: A feasible ( g θ , λ ) would still give us a (possibly loose) lower bound on R ( D ) .
Revised: Any feasible ( g θ , λ ) would then give us a (possibly loose) lower bound on R ( D ) .

Original: We can use a ﬂexible family of functions for g θ , such as neural networks, with a non-negativefunction at the output layer to ensure g θ ≥ 0 (or, we can parameterize log g instead for numerical stability).
Revised: We can use a ﬂexible family of functions for g θ , such as neural networks with a non-negative output activation to ensure g θ ≥ 0 (or, we can parameterize log g instead).

Original: Noting that the constraint Eq.
Revised: Noting that the constraint Eq.

Original: 3 is equivalent to the constraint function c ( g, λ ) :
Revised: 3 is equivalent to the constraint function c ( g, λ ) :

Original: = sup y E [exp( − λρ ( X, y )) /g
Revised: = sup y E [exp( − λρ ( X, y )) /g

Original: ( X )] − 1 being non-positive, we can then consider solving the following constrained optimization problem to obtain a R ( D ) lower bound, R L ( D ) : 
Revised: ( X )] − 1 being non-positive, we can then consider solving the following constrained maximization problem to obtain a R ( D ) lower bound, R L ( D ) : [Equation] where f ( θ, λ ) : = E [ − log g θ ( X

Original: [Equation]
Revised: [Equation]

Original: subject to
Revised: subject to

Original: [Equation]
Revised: [Equation]

Original: Assuming the supremum in Eq.
Revised: Assuming the supremum in Eq.

Original: 5 can be computed exactly, then a variety of algorithms can be used to ﬁnd a local optimum of this problem.
Revised: 5 can be computed exactly, then a variety of algorithms can be used to ﬁnd a local optimum of this problem.

Original: We consider a simple formulation based on exact penalty method (Nocedal & Wright, 2006): we solve a sequence of unconstrained problems
Revised: We consider a simple formulation based on exact penalty method (Nocedal & Wright, 2006): we solve a sequence of unconstrained problems

Original: [Equation]
Revised: [Equation]

Original:  where max { 0 , c ( θ, λ ) } penalizes the objective by the amount of constraint violation, and the sequence of penalty coefﬁcients γ t is gradually increased.
Revised: 6) where max { 0 , c ( θ, λ ) } penalizes the objective by the amount of constraint violation, and the sequence of penalty coefﬁcients γ t is gradually increased.

Original: Under fairly general conditions, a feasible solution { ( θ ∗ t , λ ∗ t ) } to Eq.
Revised: Under fairly general conditions, a feasible solution { ( θ ∗ t , λ ∗ t ) } to Eq.

Original: 6 is also a local optimum of the original constrained optimization problem in Eqs.
Revised: 6 is also a local optimum of the original constrained optimization problem in Eqs.

Original: 4, 5 (Theorem 17.4 of Nocedal & Wright (2006)).
Revised: 4, 5 (Theorem 17.4 of Nocedal & Wright (2006)).

Original: Then we can compute R L = f
Revised: Then we can estimate R L = f

Original: ( θ ∗ t , λ ∗ t ) .
Revised: ( θ ∗ t , λ ∗ t ) .

Original: 2. A N A PPROXIMATE A LGORITHM USING STOCHASTIC ESTIMATORS
Revised: 2. A N A PPROXIMATE A LGORITHM USING STOCHASTIC ESTIMATORS

Original: In most practical problems, the expectations in the optimization problem can only be approximated by Monte-Carlo averaging over samples of X .
Revised: In most practical problems, the expectations in the optimization problem can only be approximated by Monte-Carlo averaging over samples of X .

Original: Let’s focus on the constraint function c for now, and denote ψ ( x, y ) :
Revised: Let’s focus on the constraint function c for now, and denote ψ ( x, y ) :

Original: = exp( − λρ ( x,y ))
Revised: = exp( − λρ ( x,y ))

Original: g ( x ) − 1 , suppressing its dependence on ( g, λ ) for brevity.
Revised: g ( x ) − 1 , suppressing its dependence on ( g, λ ) for brevity.

Original: Note, however, that c
Revised: Note, however, that c

Original: = sup y E
Revised: = sup y E

Original: [ ψ ( X, y )] does not have the form of an expectation due to the maximization over y .
Revised: [ ψ ( X, y )] does not have the form of an expectation due to the maximization over y .

Original: Still, we may try to estimate c by maximizing the sampled approximation of E [ ψ ( X, y )] , i.e., compute the k -sample estimator C k :=
Revised: Still, we may try to estimate c by maximizing the sampled approximation of E [ ψ ( X, y )] , i.e., compute the k -sample estimator C k :=

Original: sup y 1 k
Revised: sup y 1 k

Original: (cid:80)
Revised: (cid:80)

Original: ki =1 ψ ( X i , y ) , with X i ∼ P X .
Revised: ki =1 ψ ( X i , y ) , with X i ∼ P X .

Original: What’s the relation between C k and c ?
Revised: What’s the relation between C k and c ?

Original: Given a candidate ( g, λ ) , can we verify its feasibility using C k , instead of the intractable c ?
Revised: Given a candidate ( g, λ ) , can we verify its feasibility using C k , instead of the intractable c ?

Original: We answer these questions in the following theorem (proved in Appendix Sec.
Revised: We answer these questions in the following theorem (proved in Appendix Sec.

Original: A.2):
Revised: A.2):

Original: Theorem 2.2.
Revised: Theorem 2.2.

Original: Let X 1 , X 2 , ... be a sequence of i.i.d.
Revised: Let X 1 , X 2 , ... be a sequence of i.i.d.

Original: random variables with the shared data distribution X k ∼ P X .
Revised: random variables with the shared data distribution X k ∼ P X .

Original: Deﬁne the sequence of random variables
Revised: Deﬁne the sequence of random variables

Original: C k := sup y 1 k
Revised: C k : = sup y 1 k

Original: (cid:80)
Revised: (cid:80)

Original: i ψ ( X i , y ) .
Revised: i ψ ( X i , y ) .

Original: Then 
Revised: Then 1.

Original: [Equation]
Revised: E [ C k ] =

Original:  k (cid:80) i ψ ( X i , y )] ≥ sup y E [ ψ ( X, y )]
Revised: E X 1 ,...,X k [sup y 1 k (cid:80) i ψ ( X i , y )] ≥ sup y E [ ψ ( X, y )]

Original: =: c ; 2.
Revised: =: c ; 2.

Original: E [ C 1 ] ≥ E [ C 2 ] ≥ ... ≥ E [ C k ] ≥ E
Revised: E [ C 1 ] ≥ E [ C 2 ] ≥ ... ≥ E [ C k ] ≥ E

Original: [ C k +1 ] ≥ ... sup y E [ ψ ( X, y )]
Revised: [ C k +1 ] ≥ ... sup y E [ ψ ( X, y )]

Original: = c ; 3.
Revised: = c ; 3.

Original: If ψ ( x, y ) is bounded and continuous in y , and if Y is compact, then C k converges to c almost surely (as well as in probability,
Revised: If ψ ( x, y ) is bounded and continuous in y , and if Y is compact, then C k converges to c almost surely (as well as in probability,

Original: i
Revised: i

Original: e., lim k →∞ P ( | C k − c | > (cid:15) )
Revised: e., lim k →∞ P ( | C k − c | > (cid:15) )

Original: = 0 , ∀ (cid:15) > 0 ), and lim k →∞ E
Revised: = 0 , ∀ (cid:15) > 0 ), and lim k →∞ E

Original: [ C k ] = c .
Revised: [ C k ] = c .

Original: Theorem 2.2 tells us that C k is on average an over-estimator of the value of the constraint function c ; and like the Importance-Weighted ELBO (Burda et al., 2015), the bias of the estimator decreases monotonically as k → ∞ , and that under continuity assumptions, C k is asymptotically unbiased and converges to c .
Revised: Theorem 2.2 tells us that C k is on average an over-estimator of the value of the constraint function c ; and like the Importance-Weighted ELBO (Burda et al., 2015), the bias of the estimator decreases monotonically as k → ∞ , and that under continuity assumptions, C k is asymptotically unbiased and converges to c .

Original: This means that we can form an empirical estimate of the constraint function c by computing ˆ C k :=
Revised: This means that we can form an empirical estimate of the constraint function c by computing ˆ C k :=

Original: sup y 1 k
Revised: sup y 1 k

Original: (cid:80)
Revised: (cid:80)

Original: i ψ ( x i , y ) from a large enough number of samples { x 1 , x 2 , ..., x k } ; if the empirical estimate of the constraint is satisﬁed,i.e., ˆ C k ≤ 0 , then we can claim that the true constraint c is satisﬁed with near certainty.
Revised: i ψ ( x i , y ) from a large batch of samples { x 1 , x 2 , ..., x k } (or, to be more precise, we average many such ˆ C k computed from multiple batches to estimate E [ C k ] ); if the empirical estimate of c is satisﬁed, then we can be conﬁdent so is the true constraint c satisﬁed.

Original: In light of this result, we propose to replace c by its sample-based estimator ˆ C k , and also use a
Revised: In light of this result, we propose to replace c by its sample-based estimator ˆ C k , and also use a

Original: Monte-Carlo estimate of the objective ˆ f := 1 m (cid:80) mj =1 − log g ( x j ) − λD .
Revised: Monte-Carlo estimate of the objective ˆ f := 1 m (cid:80) m j =1 − log g ( x j ) − λD .

Original: Our proposed stochastic optimization algorithm then solves the following sequence of penalty problems,
Revised: Our proposed stochastic optimization algorithm then solves the following sequence of penalty problems,

Original: [Equation]
Revised: [Equation]

Original: where given a penalty coefﬁcient γ t , we perform stochastic gradient ascent on the unconstrained objective ˆ (cid:96) ( θ, λ ) till convergence; we then increase the penalty e.g., γ t +1 :=
Revised: where given a penalty coefﬁcient γ t , we perform stochastic gradient ascent on the unconstrained objective ˆ (cid:96) ( θ, λ ) till convergence; we then increase the penalty e.g., γ t +1 :=

Original: βγ t with β > 1 , and solve the maximization problem w.r.t.
Revised: βγ t with β > 1 , and solve the maximization problem w.r.t.

Original: ( θ, λ ) again, until we can verify feasibility of resulting ( θ, λ ) (i.e., ˆ C k ≤ 0 ) with high conﬁdence.
Revised: ( θ, λ ) again, until we can verify feasibility of resulting ( θ, λ ) (e.g., ˆ C k ≤ 0 ) with high conﬁdence.

Original: We specify Algorithm 1 in pseudocode in Appendix Sec.
Revised: We specify Algorithm 1 in pseudocode in Appendix Sec.

Original: A.3.
Revised: A.3.

Original: 2. I NNER- LOOP OPTIMIZATION WITH RESPECT TO y
Revised: 2. I NNER OPTIMIZATION WITH RESPECT TO y

Original: So far we have assumed that we can exactly compute the supremum ˆ C k
Revised: So far we have assumed that we can exactly compute the supremum ˆ C k

Original: = sup y 1 k
Revised: = sup y 1 k

Original: (cid:80)
Revised: (cid:80)

Original: i ψ ( x i , y ) .
Revised: i ψ ( x i , y ) .

Original: In practice, 1 k (cid:80) i ψ ( x i , y ) is rarely concave in y , and we can only ﬁnd a local optimum, e.g., with a gradient-based method.
Revised: In practice, 1 k (cid:80) i ψ ( x i , y ) is rarely concave in y , and we can only ﬁnd a local optimum, e.g., with a gradient-based method.

Original: This is less problematic during training, as long as ( g, λ ) receive the appropriate training signal to eventually arrive at a feasible solution.
Revised: This is less problematic during training, as long as ( g, λ ) receive the appropriate training signal to eventually arrive at a feasible solution.

Original: However, once the training terminates with some candidate solution ( g ∗ , λ ∗ ) , we need to compute ˆ C k exactly (ideally with as large k as possible), in order to verify that the solution ( g ∗ , λ ∗ ) is feasible (with  high probability) and that ˆ f ( g ∗ , λ ∗ ) gives a valid estimate of a R ( D ) lower bound.
Revised: However, once the training terminates with some candidate solution ( g ∗ , λ ∗ ) , we need to compute ˆ C k exactly (ideally with as large k as possible), in order to verify that the solution ( g ∗ , λ ∗ ) is feasible (with [Figure] high probability) and that ˆ f ( g ∗ , λ ∗ ) gives a valid estimate of a R ( D ) lower bound.

Original: This requirement makes it computationally expensive (if not impossible) to verify a solution, and indeed poses an open challenge for our method; techniques from global optimization might offer useful tools.
Revised: This requirement makes it computationally expensive (if not infeasible) to verify a solution and indeed poses an open challenge for our method; techniques from global optimization might offer useful tools.

Original: We note that in the simple case X
Revised: We note that in the simple case X

Original: =
Revised: =

Original: Y = R n and ρ is the (mean) squared error, maximizing[Figure] φ ( y ) :
Revised: Y = R n and ρ is the (mean) squared error, maximizing φ ( y ) :

Original: = 1 k (cid:80) i ψ ( x i , y )
Revised: = 1 k (cid:80) i ψ ( x i , y )

Original: + 1 =
Revised: + 1 =

Original: α (cid:80) k
Revised: α (cid:80) k

Original: α k exp( − λ (cid:107) x
Revised: α k exp( − λ (cid:107) x

Original: i − y (cid:107) 2 ) is equivalent to ﬁnding the mode of a k -component Gaussian mixture density.
Revised: i − y (cid:107) 2 ) is equivalent to ﬁnding the mode of a k -component Gaussian mixture density.

Original: Global optimization algorithms for the latter problem exist with (partial) optimality guarantees (Carreira-Perpinan, 2000); e.g., we can run k gradient ascent procedures separately from each of the component modes x 1 , ..., x k , and take the largest result.
Revised: Global optimization algorithms for the latter problem exist with (partial) optimality guarantees (Carreira-Perpinan, 2000); e.g., we can run k gradient ascent procedures separately from each of the component modes x 1 , ..., x k , and take the largest result.

Original:  R ESULTS
Revised:  R ESULTS

Original: We apply our algorithm to a 2D standard Gaussian source, using mean-squared error as distortion ρ .
Revised: We apply our algorithm to a 2D standard Gaussian source, using mean-squared error as distortion ρ .

Original: We parameterize log g by a two-layer fully connected neural network.
Revised: We parameterize log g by a two-layer fully connected neural network.

Original: During training, we use k = 100 for the constraint estimator ˆ C k , and run a simple gradient ascent procedure in the inner optimization w.r.t.
Revised: During training, we use k = 100 for the constraint estimator ˆ C k , and run a simple gradient ascent procedure in the inner optimization w.r.t.

Original: y for computational efﬁciency.
Revised: y for computational efﬁciency.

Original: We start with a penalty parameter γ 0 = 1 , and multiply itby 2 every 500 training steps (which seemed sufﬁcient for convergence).
Revised: We start with a penalty parameter γ 0 = 1 , and double it every 1000 training steps (which seemed sufﬁcient for convergence).

Original: We stop training after 2000 steps, at which point our simple estimate of constraint ˆ C k is rarely violated across training steps; see Fig.
Revised: We stop training after 4000 steps, at which point our simple estimate of constraint ˆ C k is frequently satisﬁed across training steps; see Fig.

Original: 1 (Left) for example training curves.
Revised: 1 (Left) for example training curves.

Original: To certify the resulting solution ( g ∗ , λ ∗ ) as valid, we then compute ˆ C k with the global optimization algorithm described in Section 2.4, using a large number of samples ( k = 1000 ); in all our experiments, ˆ C 1000 ( g ∗ , λ ∗ ) < 0 , so we conclude the solutions are feasible, and ˆ f ( g ∗ , λ ∗ ) provide valid lower bounds of R ( D ) with very high probability.
Revised: To certify the resulting solution ( g ∗ , λ ∗ ) as valid, we then compute ˆ C k with the global optimization algorithm described in Section 2.4, using a large number of samples ( k = 10000 ); in all our experiments, ˆ C 10000 ( g ∗ , λ ∗ ) < 0 , so we conclude the solutions are feasible, and ˆ f ( g ∗ , λ ∗ ) gives a valid lower bound of R ( D ) with very high probability.

Original: We repeat the aboveprocedure with D in { 0.01, 0.02, 0.1, 0.2, 0.5, 0.8 } , and plot the corresponding ˆ 
Revised: We repeat the above with D ranging from 0.02 to 0.8, and plot the corresponding ˆ R L ( D )

Original: R L ( D ) =
Revised: =

Original: ˆ f ( g ∗ , λ ∗ ) to trace out an R ( D ) lower bound.
Revised: ˆ f ( g ∗ , λ ∗ ) to trace out an R ( D ) lower bound.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 1 (Middle), the resulting lower bound shows reasonable agreement with the true R ( D ) , with an average gap of 0.88 bit; the gap decreases with increasing distortion tolerance, from 1.63 bit at D = 0.01, to 0.15 bit at D = 0.8.
Revised: 1 (Middle), our lower bound shows good agreement with the true R ( D ) , with an average gap of 0.74 bit.

Original: We attribute this behavior to the inaccuracy of our simple gradient ascent procedure for computing ˆ C k during training: with a smaller D , there is less pressure to drive down λ , which makes the component functions φ x i more sharply peaked, and the overall inner objective φ ( y ) more multimodal and harder to optimize.

Original: Fig.
Revised: Fig.

Original: 1 (Right) visualizes the optimization problem for D = 0 .
Revised: 1 (Right) visualizes the optimization problem for D = 0 .

Original: 5 , where we take a vertical slice of R 2 and show the learned g θ ∗ ( x ) , the theoretically derived optimal g ( x ) (a scaled Gaussian density), and the ˆ C k optimization objective φ ( y ) and its component functions φ x i ( y ) centered on  20 random samples, plugging in the solution ( g θ ∗ , λ ∗ ) .
Revised: 6 , where we take a vertical slice of R 2 and show the learned g θ ∗ ( x ) , the theoretically derived optimal g ( x ) (a scaled Gaussian density), and the ˆ C k optimization objective φ ( y ) and its component functions φ x i ( y ) centered on k = 20 random samples, plugging in the learned ( g θ ∗ , λ ∗ ) .

Original: In this case, we see  that ˆ 
Revised: Note that the component functions have the form of a Gaussian kernel, whose “sharpness” is controlled by λ . When λ is large or k is small, the resulting ˆ C k = max y φ ( y ) −more severely overestimates the true constraint c . With a smaller λ , or by increasing k , the landscape of φ ( y ) is more smoothed out, so the overestimating effect is reduced. Indeed, in this ﬁgure we see max y φ ( y ) > 1 . 25 ; by increasing the sample size k from 20 to 10000, max y φ ( y ) would decrease to around 0.93 instead (so that ˆ C k ≈ − 0 .

Original:  C k = max y φ ( y ) − 1 > 0 ,however this does not contradict the feasibility of ( g θ ∗ , λ ∗ ) ; in fact, ˆ C k decreases to become negative when we use more (e.g., k ≥ 50 )samples, as Theorem 2.2 predicts.
Revised: 07 < 0 ), as predicted by Theorem 2.2. In the low distortion (small D ) regime, λ is driven to large values, so that ˆ C k exhibits higher bias, and constraint violations max { 0 , ˆ C k } are often penalized overly harshly (as ˆ C k tends to overestimate c ); this likely explains why our lower bound has a larger gap to the true R ( D ) as we decrease D .

Original: We  provide more experimental details, and a visualization of the contours of g θ being optimized during training, in Appendix Section.
Revised: We expect the bound can be improved by more accurately estimating the gradient of the penalty/constraint function (we elaborate this point in Appendix Sec. A.3), and/or training with a larger k especially in the lower distortion regime. We provide more experimental details and visualizations in Appendix Sec.

Original: A.4.
Revised: A.4.

Original: As shown in Appendix Figure. 2 the learned g θ eventually converges to a uni-modal function with mode at the origin, roughly resembling the theoretically optimal g (which is equal to the source density scaled by a factor of 2 πD ).
Revised: This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. HR001120C0021. Any opinions, ﬁndings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reﬂect the views of the Defense Advanced Research Projects Agency (DARPA).

Original: Again, we see the agreement is worse for smaller D (e.g., D = 0 . 01 ), likely due to the inaccuracy of estimating ˆ C k by a single gradient ascent run.
Revised: Yibo Yang acknowledges funding from the Hasso Plattner Foundation.

Original: This could potentially be addressed byﬁne-tuning the solution with a larger number of samples k (which has the effect of smoothing out the inner objective φ ( y ) ), and/or using a global optimization procedure for computing ˆ C k instead.
Revised: Furthermore, this work was supported by the National Science Foundation under Grants 1928718, 2003237 and 2007719, as well as Intel and Qualcomm.

Original:  A. F ULL V ERSION OF T HEOREM 2. 
Revised: A A PPENDIX A. F ULL V ERSION OF T HEOREM 2.

Original: Theorem A.1.
Revised: Theorem A.1.

Original: (Kostina, 2016) Suppose that the following basic assumptions are satisﬁed.
Revised: (Kostina, 2016) Suppose that the following basic assumptions are satisﬁed.

Original: R ( D ) is ﬁnite for some D , i.e., D min :
Revised: R ( D ) is ﬁnite for some D , i.e., D min :

Original: = inf { D : R ( D ) ≤ ∞} < ∞ ; 2.
Revised: = inf { D : R ( D ) ≤ ∞} < ∞ ; 2.

Original: The distortion metric ρ is such that there exists a ﬁnite set E ⊂ Y such that
Revised: The distortion metric ρ is such that there exists a ﬁnite set E ⊂ Y such that

Original: [Equation]
Revised: [Equation]

Original: Then, for each D > D min ,
Revised: Then, for each D > D min ,

Original: it holds that
Revised: it holds that

Original: [Equation]
Revised: [Equation]

Original: ) where the maximization is over g ( x ) ≥ 0 and λ ≥ 0 satisfying the constraint
Revised: where the maximization is over g ( x ) ≥ 0 and λ ≥ 0 satisfying the constraint

Original: [Equation]
Revised: [Equation]

Original: Note: the  maximization over g ( x ) ≥ 0 can be restricted to only 1 ≥ g ( x ) ≥ 0 .
Revised: Note: the basic assumption 2 is trivially satisﬁed when the distortion ρ is bounded from above; the maximization over g ( x ) ≥ 0 can be restricted to only 1 ≥ g ( x ) ≥ 0 .

Original: Unless stated otherwise, we use log base e in this work, so the R ( D ) above is in terms of nats.
Revised: Unless stated otherwise, we use log base e in this work, so the R ( D ) above is in terms of nats.

Original: A. P ROOF OF T HEOREM 2. 
Revised: A. P ROOF OF T HEOREM 2.

Original: Theorem A.2.
Revised: Theorem A.2.

Original: (We restate the theorem for completeness).
Revised: (We restate the theorem for completeness).

Original: Let X 1 , X 2 , ... be a sequence of i.i.d.
Revised: Let X 1 , X 2 , ... be a sequence of i.i.d.

Original: random variables with the shared data distribution X k ∼ P X .
Revised: random variables with the shared data distribution X k ∼ P X .

Original: Deﬁne the sequence of random variables
Revised: Deﬁne the sequence of random variables

Original: C k := sup y 1 k
Revised: C k := sup y 1 k

Original: (cid:80)
Revised: (cid:80)

Original: i ψ ( X i , y ) .
Revised: i ψ ( X i , y ) .

Original: Then1.
Revised: Then

Original: E [ C k ] = E [ sup
Revised: [Equation] [Equation]

Original: y 1 k (cid:80) i ψ ( X i , y )] ≥ sup y E [ ψ ( X, y )]
Revised: 2 ] ≥ ...

Original: =: c ; E [ C 1 ] ≥ E [ C 2 ] ≥ ... ≥ E
Revised: ≥ E

Original: [ C k ] ≥ E [ C k +1 ] ≥ ... sup y E
Revised: [ C k ] ≥ E [ C k +1 ] ≥ ... sup y E

Original: [ ψ ( X, y )]
Revised: [ ψ ( X, y )]

Original: = c ; 3.
Revised: = c ; 3.

Original: If ψ ( x, y ) is bounded and is continuous in y , and if Y is compact , then C k converges to c = sup y E
Revised: If ψ ( x, y ) is bounded and is continuous in y , and if Y is compact , then C k converges to c = sup y E

Original: [ ψ ( X, y )] almost surely (as well as in probability), and we have lim k →∞ E [ C k ] = c .
Revised: [ ψ ( X, y )] almost surely (as well as in probability), and we have lim k →∞ E [ C k ] = c .

Original: Proof.
Revised: Proof.

Original: We prove each in turn:
Revised: We prove each in turn:

Original: E [ C k ] =
Revised: E [ C k ] =

Original: E [sup y 1 k (cid:80) i ψ ( X i , y )] ≥ sup y E
Revised: E [sup y 1 k (cid:80) i ψ ( X i , y )] ≥ sup y E

Original: [ 1 k (cid:80) i ψ ( X i , y )]
Revised: [ 1 k (cid:80) i ψ ( X i , y )]

Original: = sup y E
Revised: = sup y E

Original: [ ψ ( X, y )]
Revised: [ ψ ( X, y )]

Original: =
Revised: =

Original: c 2.
Revised: c 2.

Original: First, note that E [ C 1 ] ≥ E [ C k ] since
Revised: First, note that E [ C 1 ] ≥ E [ C k ] since

Original: [Equation]
Revised: [Equation]

Revised: X

Revised: [Equation]

Revised: [Equation]

Original: We therefore have
Revised: We therefore have

Original: [Equation]
Revised: [Equation]

Original: The proof for this resembles that of Theorem 1 in (Burda et al., 2015).
Revised: The proof for this resembles that of Theorem 1 in (Burda et al., 2015).

Revised: We use standard arguments from probability theory and real analysis.

Original: Fix y ∈ Y , and consider the random variable M k
Revised: Fix y ∈ Y , and consider the random variable M k

Original: = 1 k (cid:80) ki
Revised: = 1 k (cid:80) ki

Original: =1 ψ ( X i , y ) .
Revised: =1 ψ ( X i , y ) .

Original: If ψ is bounded, then it follows from the Strong Law of Large Numbers that M k converges to E [ M 1 ] =
Revised: If ψ is bounded, then it follows from the Strong Law of Large Numbers that M k converges to E [ M 1 ] =

Original: E [ ψ ( X, y )] almost surely; in other words, for every ω belonging to some set with measure 1,
Revised: E [ ψ ( X, y )] almost surely; in other words, for every ω outside a set of measure zero,

Original: [Equation]
Revised: [Equation]

Original: Then, for every such ω
Revised: Then, for every such ω

Original: [Equation]
Revised: [Equation]

Original: where we used the fact that the sequence of continuous functions s k ( y ) :
Revised: where we used the fact that the sequence of continuous functions s k ( y ) :

Original: = 1 k [Equation] ki =1 ψ ( X i ( ω )) , y ) converges pointwise to s ( y ) :
Revised: = 1 k (cid:80) ki =1 ψ ( X i ( ω )) , y ) converges pointwise to s ( y ) :

Original: = E [ ψ ( X ( ω ) , y )] on a compact set Y , so s k converges to s also uniformly, so we are allowed to exchange limit and supremum, i.e.,
Revised: = E [ ψ ( X ( ω ) , y )] on a compact set Y , so s k converges to s also uniformly, so we are allowed to exchange limit and supremum, i.e.,

Original: lim k →∞ sup y s k
Revised: lim k →∞ sup y s k

Original: ( y ) =
Revised: [Equation] y ) =

Original: sup
Revised: sup

Original: y lim k →∞ s k ( y )
Revised: y lim k →∞ s k ( y )

Original: =
Revised: =

Original: sup y s ( y ) .
Revised: sup y s ( y ) .

Original: But the above equation precisely means that C k converges to E [ ψ ( X, y )] almost surely.
Revised: But the above equation precisely means that C k converges to E [ ψ ( X, y )] almost surely.

Original: Therefore C k also converges to E [ ψ ( X, y )] in probability, and lim k →∞ E [ C k ]
Revised: Therefore C k also converges to E [ ψ ( X, y )] in probability, and lim k →∞ E [ C k ]

Original: = c .
Revised: = c .

Original: A. P ROPOSED S TOCHASTIC O PTIMIZATION A LGORITHM
Revised: A. P ROPOSED S TOCHASTIC O PTIMIZATION A LGORITHM

Original: We give a pseudocode implementation ofthe penalty method, using our proposed estimators for the optimization objectives, in Algorithm 1.
Revised: We give a pseudocode implementation of our proposed stochastic optimization procedure in Algorithm 1.

Original: The subroutine estimate constraint used during gradient ascent on the unconstrained penalty objective ˆ (cid:96) ( θ, λ ) is intended to be a computationally cheaper alternative to the exact (but expensive) estimate constraint global opt procedure, which we reserve only for verifying the feasibility of a candidate solution.
Revised: We follow a simple exact penalty method (Nocedal & Wright, 2006), but other constrained optimization methods like augmented Lagrangian can also be used instead. Recall the exact form of the unconstrained penalty problem that we would like to solve: [Equation] where

Revised: [Equation]

Original: We give an example implementation of estimate constraint that simply performs a single gradient ascentrun on φ ( y ) to estimateˆ C k , which proved sufﬁcient for training in our experiment; however, since gradient ascent only gives us a local maximum ofφ ( y ) , the unconstrained optimization of ˆ (cid:96) ( θ, λ ) with this naive implementation of ˆ C k may run into convergence issues.
Revised: and [Equation] To apply stochastic gradient ascent on this problem, we need an estimate of the gradient with respect to θ and λ . Focusing just on θ and dropping dependence on λ to simplify notation, we note that an unbiased gradient estimate of f is straightforward to obtain by a sample mean, as

Revised: [Equation]

Original: We appeal to Danskin’s theorem in order to compute the gradient  ∇  ( θ,λ ) ˆ C k = ∇ ( θ,λ )  sup y φ (  y ) by ∇ ( θ,λ ) φ (  y∗ ) , when doing gradient ascent on ˆ (cid:96) ( θ, λ ) .
Revised: Estimating the gradient of the penalty function, ∇ θ max { 0 , c ( θ ) } , requires more work. Let’s switch to the lighter notation c ( θ ) = sup y E [ ψ θ ( X, y )] , where ψ θ ( x, y ) : = exp( − λρ ( x,y

Revised: [Equation]

Revised: ) g ( x ) − 1 .

Revised: Suppose y ∗ achieves the supremum in the deﬁnition of c , i.e., c ( θ )

Revised: = sup y E

Revised: [ ψ θ ( X, y )]

Revised: =

Revised: E [ ψ θ ( X, y ∗ )] .

Revised: Focusing on the case where the constraint is violated, i.e., c ( θ ) > 0 and max { 0 , c ( θ ) } =

Revised: c ( θ ) (for otherwise we can simply set the (sub)gradient ∇ θ max { 0 , c ( θ ) } to zero), we have ∇ θ max { 0 , c ( θ ) } =

Revised: ∇ θ c ( θ ) = ∇ θ sup y E [ ψ θ ( X, y )]

Revised: = ∇ θ E [ ψ θ ( X, y ∗ )] , where in the last equality we removed the dependence of the constraint gradient on y by appealing to an envelope theorem (Milgrom & Segal, 2002).

Revised: Then we can form a simple unbiased gradient estimator by a sample mean, ∇ θ E [ ψ θ ( X, y ∗ )]

Revised: ≈ 1 k (cid:80) ki =1 ∇ θ ψ θ ( x i , y ∗ ) .

Revised: The remaining crucial problem, then, is that of computing y ∗ =

Revised: arg max y E [ ψ ( X, y )] in the ﬁrst place.

Revised: Here, we implement the approximation y ∗ ≈ arg max y 1 k (cid:80) ki

Revised: =1 ψ θ ( x i , y ) in the subroutine estimate constraint , where we also compute the constraint estimator ˆ C k =

Original: Requires: Model g θ (e.g., a neural network), initial parameters ( g θ 0 , λ 0 ) , initial penalty parameter γ 

Revised: Requires: Model g θ (e.g., a neural network), initial parameters ( g θ 0 , λ 0 ) , initial penalty parameter γ

Original: = 1 , batch sizes k, m , gradient ascent step size η , step counter t =while True do // Find a local maximum of ˆ (cid:96) ( θ, λ ) given the current γ t while ( θ t , λ t ) not converged do
Revised: = 1 , batch sizes k, m , gradient ascent step size η , step counter t =while True do // Find a local maximum of ˆ (cid:96) ( θ, λ ) given the current γ t while ( θ t , λ t ) not converged do

Original: Draw two batches of data samples { x 1 , ..., x k } and { x 1 , ..., x m } y ∗ , ˆ C k = estimate constraint ( θ t , λ t , { x 1 , ..., x k } )
Revised: Draw two batches of data samples { x 1 , ..., x k } and { x 1 , ..., x m } y ∗ , ˆ C k = estimate constraint ( θ t , λ t , { x 1 , ..., x k } )

Original: Update parameters by ( θ t +1 , λ t +1 )
Revised: Update parameters by ( θ t +1 , λ t +1 )

Original: = ( θ t , λ t ) + η ∇ ( θ,λ ) 
Revised: = ( θ t , λ t ) + η ∇ ( θ,λ ) (cid:16) 1 m (cid:80) mj

Original: [Table]
Revised: =1 − log g θ t ( x j ) − γ t max { 0 , 1 k (cid:80)

Original: −
Revised: ki

Original: [Table] } (cid:17)
Revised: =1 exp {− λ t ρ ( x i ,y ∗ ) } g θt ( x i ) − 1 } (cid:17)

Original: Update training step counter t :=
Revised: Update training step counter t :=

Original: t +end  y ∗ , ˆ C k = estimate constraint global opt ( θ t , λ t , { x 1 , ..., x k } ) if ˆ C k ≤ 0 then //
Revised: t +end Sample a large batch { x 1 , ..., x k } (as large as we can afford for global optimization below) y ∗ , ˆ C k = estimate constraint global opt ( θ t , λ t , { x 1 , ..., x k } ) if ˆ C k ≤ 0 then //

Original: True constraint c is satisfied with high probability
Revised: True constraint c is satisfied with high probability

Original: Return solution ( θ t , λ t ) , and lower bound estimate ˆ R L = (cid:80) m j =1 − log g θ t ( x j ) else
Revised: Return solution ( θ t , λ t ) , and lower bound estimate ˆ R L = (cid:80) mj =1 − log g θ t ( x j ) else

Original: Increase penalty parameter, γ t := βγ t , with e.g., β = 2 orend endSubroutine estimate constraint ( θ, λ, { x 1 , ..., x k } ) :
Revised: Increase penalty parameter, γ t := βγ t , with e.g., β = 2 orend endSubroutine estimate constraint ( θ, λ, { x 1 , ..., x k } ) :

Original: Initialize y , for example, using the mean of data samples, y = (cid:80) ki

Original: =1 x i

Original: Update y by gradient ascent on φ ( y ) :
Revised: Run gradient ascent on φ ( y ) :

Original: = (cid:80) ki
Revised: = (cid:80) ki

Original: =1 exp {− λρ ( x i , y ) } /g θ ( x i ) , until convergence ˆ C k = φ ( y ∗ )
Revised: =1 exp {− λρ ( x i , y ) } /g θ ( x i ) , until converging to y ∗ Compute ˆ C k = φ ( y ∗ )

Original: Return (converged y ∗ , ˆ C k )
Revised: Return ( y ∗ , ˆ C k )

Original: Subroutine estimate constraint global opt ( θ, λ, { x 1 , ..., x k } ) : Find global maximizer y ∗ 
Revised: Subroutine estimate constraint global opt ( θ, λ, { x 1 , ..., x k } ) : Find global maximizer y ∗ = arg max y φ ( y ) Compute ˆ C k = φ ( y ∗ )

Original: = arg max y  (cid:80) ki =1 exp {− λρ ( x i , y ) } /g θ ( x i )
Revised: Return ( y ∗ , ˆ C k ) max y 1 k (cid:80) ki = 1 ψ θ ( x i , y ) =

Original: ˆ C k= (cid:80) 
Revised: 1 k (cid:80) ki =

Original: ki =1 exp {− λρ ( x i , y ∗ )  } /g θ  ( x i ) −
Revised: 1 ψ θ ( x i , y ∗ ) along the way, so that ∇ θ max { 0 , c ( θ ) } ≈ ∇ θ max { 0 , ˆ C k ( θ ) } .

Original: Return ( y ∗  , ˆ C k )
Revised: This is done by a simple gradient ascent procedure, intended as a computationally cheaper alternative to the exact (but expensive) estimate constraint global opt procedure, which we reserve only for verifying the feasibility of a candidate solution. Alternatively, we could approximately ﬁnd y ∗ by running stochastic gradient ascent on E [ ψ ( X, y )] with respect to y , but we ﬁnd this can yield worse local maxima than our deterministic optimization procedure.

Original: A. M ORE EXPERIMENTAL  RESULTS
Revised: A. M ORE EXPERIMENTAL DETAILS AND RESULTS

Original: We used a simple 2 layer fully connected network with a scalar output for log g , with 8 hidden units in each layer.
Revised: We used a simple 2 layer fully connected network with a scalar output for log g , with 8 hidden units in each layer.

Original: We parameterized λ as λ = exp ˜
Revised: We parameterized λ as λ = exp ˜

Original: λ using an unconstrained variable ˜ λ .
Revised: λ using an unconstrained variable ˜ λ .

Original: We used  Adam optimizer for all gradient based optimization.
Revised: We used the Adam optimizer for all gradient based optimization.

Original: During training, we set  k
Revised: During training, we set batch size k

Original: = m = 100 ,using the same batch of data samples for ˆ f andˆ C k for simplicity. We implement estimate constraint by a single gradient ascent run, initializing y to the mean of k data samples.
Revised: = m = 100 , and implement estimate constraint by a single gradient ascent run, initializing y to the data sample in the batch that achieves the highest value of φ ( · ) (we ﬁnd that randomly initializing y to a data sample to also work reasonably well).

Original: We used m = 10000 samples when computing ˆ f for  the ﬁnal R ( D ) lower bound; in Figure 1 (Middle), we plotted the mean of 10 different estimates of ˆ f (each with  10000 samples); the (mean, std) of the 10 estimates are (2.9730887, 0.011416249), (2.3030891, 0.0050365757), (1.4062412, 0.004873369), (0.92046356, 0.0046269516), (0.34891954, 0.007819665), (0.07717345, 0.0040917634), for D = 0 .
Revised: We used the ( g ∗ , λ ∗ ) trained after 3500 steps for estimating the ﬁnal R ( D ) lower bound. We ﬁrst veriﬁed their feasibility by running the global optimization procedure of Carreira-Perpinan (2000) with k = 10000 , as assured by the negative values of the resulting ˆ C 10000 , which were − 0 .

Original: Note these results are in terms of nats (unlike in Figure 1 (Middle) which we converted to bits).
Revised: 199 , − 0 .

Original: The values of ˆ C 1000 for verifying feasibility of the solutions were -0.235, -0.532, -0.388, -0.335, -0.245, -0.145, and -0.04, for
Revised: 106 , − 0 . 254 , − 0 . 202 , − 0 . 115 , − 0 .

Original:  D = 0 . 
Revised: 068 , − 0 . 052 , for D = 0 . Then we compute ˆ f m from m = samples, repeat for different batches, and plot the mean ˆ f as the lower bound ˆ R L ( D ) in

Original: 5 , 0 .  8 , computed with the global optimization algorithm of (CarreiraPerpinan, 2000).
Revised: Figure (Middle), corresponding to the ( D, R ) points (0 . 02 , 4 . 062) , (0 . 05 , 3 . 343) , (0 . 1 , 2 . 424) , (0 . 2 , 1 . 600) , (0 . 4 , 0 . 853) , (0 . 6 , 0 . 432) , (0 . 8 , 0 . 125) , in terms of (MSE, bits). The standard deviations of ˆ f from the 10 different batches are between 0 . 005 and 0 . 007 , so the error bars do not appear noticeable in the ﬁgure.

Original: In our setup, the optimal g ( x ) can be found analytically as g ( x ) =
Revised: In our problem setting, the optimal g ( x ) can be found analytically as g ( x ) =

Original: f X ( x )(2 πD ) n/ 2 , where n =and f X is the density of the source X (2D standard Gaussian).
Revised: f X ( x )(2 πD ) n/ 2 , where n = 2 and f X is the density of the source X (2D standard Gaussian).

Original: The Shannon lower bound coincides with the true R ( D ) (Cover, 1999), giving R ( D )
Revised: The Shannon lower bound coincides with the true R ( D ) (Cover, 1999), giving R ( D )

Original: =
Revised: =

Original: h ( X ) − h
Revised: h ( X ) − h

Original: ( D ) , where h ( X ) , h ( D ) are the differential entropy of X , and an isotropic n -dimensional Gaussian distribution, respectively.
Revised: ( D ) , where h ( X ) , h ( D ) are the differential entropy of X , and an isotropic n -dimensional Gaussian distribution with diagonal covariances equal to D , respectively.

Original: In the next ﬁgure, we visualize the evolution of g θ during training, when we run the algorithm with a few example settings of D .
Revised: In the next ﬁgure, we visualize the evolution of g θ during training for a few experiments.