Original: L OWER B OUNDING R ATE -D ISTORTION F ROM S AMPLES
Revised: L OWER B OUNDING R ATE -D ISTORTION F ROM S AMPLES

Original: A BSTRACT
Revised: A BSTRACT

Original: The rate-distortion function R ( D ) tells us the minimal number of bits on average to compress a random object within a given distortion tolerance.
Revised: The rate-distortion function R ( D ) tells us the minimal number of bits on average to compress a random object within a given distortion tolerance.

Original: A lower bound on R ( D ) therefore represents a fundamental limit on the best possible ratedistortion performance of any lossy compression algorithm, and can help us assess the potential room for improvement.
Revised: A lower bound on R ( D ) therefore represents a fundamental limit on the best possible ratedistortion performance of any lossy compression algorithm, and can help us assess the potential room for improvement.

Original: We make a ﬁrst attempt at an algorithm for computing such a lower bound, applicable to general memoryless data sources that we have samples of.
Revised: We make a ﬁrst attempt at an algorithm for computing such a lower bound, applicable to general memoryless data sources that we have samples of.

Original: Based on a dual characterization of R ( D ) in terms of constrained maximization, our method approximates the exact constraint function by an asymptotically unbiased estimator, allowing for stochastic optimization.
Revised: Based on a dual characterization of R ( D ) in terms of constrained maximization, our method approximates the exact constraint function by an asymptotically unbiased estimator, allowing for stochastic optimization.

Original: On a 2D Gaussian source, we obtain a lower bound within 1 bit of the analytical R ( D ) .
Revised: On a 2D Gaussian source, we obtain a lower bound within 1 bit of the analytical R ( D ) .

Original:  I NTRODUCTION AND B ACKGROUND
Revised:  I NTRODUCTION AND B ACKGROUND

Original: Let X ∈ X be a random variable with distribution P X 1 that represents a memoryless data source, let
Revised: Let X ∈ X be a random variable with distribution P X 1 that represents a memoryless data source, let

Original: Y be the set of lossy representations, and let ρ : X × Y → [0 , ∞ ) be a distortion cost function.
Revised: Y be the set of lossy representations, and let ρ : X × Y → [0 , ∞ ) be a distortion cost function.

Original: The (information) rate-distortion function R ( D ) is deﬁned as the solution to the optimization problem,
Revised: The (information) rate-distortion function R ( D ) is deﬁned as the solution to the optimization problem,

Original: [Equation]
Revised: [Equation]

Original: (1) where we consider all stochastic transforms P Y | X whose expected distortion is within the given threshold D , and minimize the mutual information between the source X and its reproduced representation Y .
Revised: (1) where we consider all stochastic transforms P Y | X whose expected distortion is within the given threshold D , and minimize the mutual information between the source X and its reproduced representation Y .

Original: Rate-distortion theory (Shannon, 1959; Cover, 1999) gives operational meaning to the above deﬁnition as the asymptotic fundamental limit of lossy data compression.
Revised: Rate-distortion theory (Shannon, 1959; Cover, 1999) gives operational meaning to the above deﬁnition as the asymptotic fundamental limit of lossy data compression.

Original: In essence,
Revised: In essence,

Original: R ( D ) delimits the minimal average number of bits (or nats, depending the base of log ) needed to convey i.i.d.
Revised: R ( D ) delimits the minimal average number of bits (or nats, depending the base of log ) needed to convey i.i.d.

Original: samples of X within average distortion D , for any block code that is allowed arbitrarily long blocks or high complexity, regardless of its implementation technology.
Revised: samples of X within average distortion D , for any block code that is allowed arbitrarily long blocks or high complexity, regardless of its implementation technology.

Original: Therefore, the ratedistortion function, or a lower bound on it, can help analyze how close lossy compression algorithms are to their theoretical performance limits.
Revised: Therefore, the ratedistortion function, or a lower bound on it, can help analyze how close lossy compression algorithms are to their theoretical performance limits.

Original: If the operational rate and distortion of a compression algorithm lies far above the R ( D ) curve (or its lower bound), then we may expect room for further improvement; otherwise, the algorithm’s rate-distortion performance is already close to theoretically optimal, so we may better focus our attention on improving other aspects of the algorithm.
Revised: If the operational rate and distortion of a compression algorithm lies far above the R ( D ) curve (or its lower bound), then we may expect room for further improvement; otherwise, the algorithm’s rate-distortion performance is already close to theoretically optimal, so we may better focus our attention on improving other aspects of the algorithm.

Original: Although R ( D ) has no analytical form in general, it can be computed in principle by solving the convex optimization problem Eq.
Revised: Although R ( D ) has no analytical form in general, it can be computed in principle by solving the convex optimization problem Eq.

Original: The mutual information I ( X ; Y ) , however, is typically challenging to compute, so a tractable upper bound I ( X ; Y ) ≤ I ( X ; Y )
Revised: The mutual information I ( X ; Y ) , however, is typically challenging to compute, so a tractable upper bound I ( X ; Y ) ≤ I ( X ; Y )

Original: + KL ( P Y (cid:107) ˜
Revised: + KL ( P Y (cid:107) ˜

Original: P Y )
Revised: P Y )

Original: =
Revised: =

Original: E X [ KL ( P Y | X (cid:107) ˜ P Y )] is often used instead in the optimization problem.
Revised: E X [ KL ( P Y | X (cid:107) ˜ P Y )] is often used instead in the optimization problem.

Original: Solving the resulting minimization problem then leads to upper bounds on R ( D ) , and is the basis of the Blahut–Arimoto algorithm (Arimoto, 1972).
Revised: Solving the resulting minimization problem then leads to upper bounds on R ( D ) , and is the basis of the Blahut–Arimoto algorithm (Arimoto, 1972).

Original: The same mutual information upper bound has also made numerous appearances in machine learning (Alemi et al., 2017; Poole et al., 2019), notably as the aggregate KL-divergence regularizer in
Revised: The same mutual information upper bound has also made numerous appearances in machine learning (Alemi et al., 2017; Poole et al., 2019), notably as the aggregate KL-divergence regularizer in

Original: Variational-Autoencoders (VAEs) (Kingma & Welling, 2013).
Revised: Variational-Autoencoders (VAEs) (Kingma & Welling, 2013).

Original: Traditionally, the set of lossy presentations Y is often a subset of the data space X , in the form of quantization points, and distortion ρ is ﬁxed.
Revised: Traditionally, the set of lossy presentations Y is often a subset of the data space X , in the form of quantization points, and distortion ρ is ﬁxed.

Original: In VAEs and their recent application to learned data compression (Ball´e et al., 2017), Y has the of a latent space , ˜ P Y is the prior , and ρ ( x, y )
Revised: In VAEs and their recent application to learned data compression (Ball´e et al., 2017), Y has the of a latent space , ˜ P Y is the prior , and ρ ( x, y )

Original: =
Revised: =

Original: ρ 0 ( x, f θ ( y )) corresponds to the negative log likelihood log p θ ( x | y ) parameterized by a decoder f
Revised: ρ 0 ( x, f θ ( y )) corresponds to the negative log likelihood log p θ ( x | y ) parameterized by a decoder f

Original: θ : Y → X (e.g., p θ ( x | y ) is a Gaussian density with mean f θ ( y ) , ρ 0 is a squared error), so the distortion ρ is learned.
Revised: θ : Y → X (e.g., p θ ( x | y ) is a Gaussian density with mean f θ ( y ) , ρ 0 is a squared error), so the distortion ρ is learned.

Original: The
Revised: The

Original: Lagrangian E X [ KL ( P Y | X (cid:107) ˜ P Y )]
Revised: Lagrangian E X [ KL ( P Y | X (cid:107) ˜ P Y )]

Original: + λ E
Revised: + λ E

Original: [ ρ ( X, Y )] then corresponds to the negative ELBO (Kingma & Welling, 2013) for learning and inference.
Revised: [ ρ ( X, Y )] then corresponds to the negative ELBO (Kingma & Welling, 2013) for learning and inference.

Original: Using the analytical density of P Y | X that optimizes the
Revised: Using the analytical density of P Y | X that optimizes the

Original: Lagrangian (assumed to exist), Huang et al.
Revised: Lagrangian (assumed to exist), Huang et al.

Original: (2020) sample from this optimal P Y | X to compute optimistic upper bounds on R ( D ) of trained VAEs and other generative models, hinting at their possible compression performance.
Revised: (2020) sample from this optimal P Y | X to compute optimistic upper bounds on R ( D ) of trained VAEs and other generative models, hinting at their possible compression performance.

Original: Our method is agnostic to the choice of representations Y or distortion ρ , and can also be extended to compute R ( D ) lower bounds of trained decoder-based models.
Revised: Our method is agnostic to the choice of representations Y or distortion ρ , and can also be extended to compute R ( D ) lower bounds of trained decoder-based models.

Original: Unlike (Huang et al., 2020), however, we do not assume (or require) a prior distribution ˜ P Y , and our lower bound establishes what kind of lossy compression performance is theoretically impossible to obtain.
Revised: Unlike (Huang et al., 2020), however, we do not assume (or require) a prior distribution ˜ P Y , and our lower bound establishes what kind of lossy compression performance is theoretically impossible to obtain.

Original: Lower bounds of R ( D ) , by contrast, have received considerably less attention, potentially because of difﬁculty in obtaining them.
Revised: Lower bounds of R ( D ) , by contrast, have received considerably less attention, potentially because of difﬁculty in obtaining them.

Original: A variational lower bound of R ( D ) already appeared in Shannon’s landmark paper (Shannon, 1948), which was later extended (Berger, 1971) and proved rigorously for
Revised: A variational lower bound of R ( D ) already appeared in Shannon’s landmark paper (Shannon, 1948), which was later extended (Berger, 1971) and proved rigorously for

Original: X in a general abstract probability space by Csisz´ar (1974); we adopt the version in (Kostina, 2016).
Revised: X in a general abstract probability space by Csisz´ar (1974); we adopt the version in (Kostina, 2016).

Original: The special case of a discrete and known source P X is considered by Chiang & Boyd (2004), who solve the corresponding (ﬁnite-dimensional) dual problem of R ( D ) by geometric programming.
Revised: The special case of a discrete and known source P X is considered by Chiang & Boyd (2004), who solve the corresponding (ﬁnite-dimensional) dual problem of R ( D ) by geometric programming.

Original:  M ETHOD 2. D UAL CHARACTERIZATION OF R ( D )
Revised:  M ETHOD 2. D UAL CHARACTERIZATION OF R ( D )

Original: We use the below general characterization of R ( D ) in terms of a constrained maximization problem:
Revised: We use the below general characterization of R ( D ) in terms of a constrained maximization problem:

Original: Theorem 2.1.
Revised: Theorem 2.1.

Original: (Kostina, 2016)
Revised: (Kostina, 2016)

Original: Under basic regularity conditions (e.g., satisﬁed by a bounded ρ ; see
Revised: Under basic regularity conditions (e.g., satisﬁed by a bounded ρ ; see

Original: Appendix Sec. A.1), for any distortion tolerance D > D min :
Revised: Appendix Sec. A.1), for any distortion tolerance D > D min :

Original: = inf { D
Revised: = inf { D

Original: [Equation]
Revised: [Equation]

Original: R ( D
Revised: R ( D

Original: [Equation]
Revised: [Equation]

Original: ≤ ∞} , it holds that
Revised: ≤ ∞} , it holds that

Original: [Equation]
Revised: [Equation]

Original: where the maximization is over g ( x ) ≥ 0 and λ ≥ 0 satisfying the constraint
Revised: where the maximization is over g ( x ) ≥ 0 and λ ≥ 0 satisfying the constraint

Original: [Equation]
Revised: [Equation]

Original: In other words, every pair of feasible ( λ, g ) satisfying Eq.
Revised: In other words, every pair of feasible ( λ, g ) satisfying Eq.

Original: 3 yields a lower bound of R ( D ) , i.e., E [ − log g ( X )] − λD ≤ R ( D ) , and we obtain R ( D ) via the tightest such lower bound.
Revised: 3 yields a lower bound of R ( D ) , i.e., E [ − log g ( X )] − λD ≤ R ( D ) , and we obtain R ( D ) via the tightest such lower bound.

Original: This characterization of R ( D ) can be derived from Lagrange duality (Chiang & Boyd, 2004) in the case of a ﬁnite alphabet X ; the extension to a general X requires a more involved argument (Csisz´ar, 1974).
Revised: This characterization of R ( D ) can be derived from Lagrange duality (Chiang & Boyd, 2004) in the case of a ﬁnite alphabet X ; the extension to a general X requires a more involved argument (Csisz´ar, 1974).

Original: 2. L OWER B OUNDING R ( D ) VIA NUMERICAL OPTIMIZATION
Revised: 2. L OWER B OUNDING R ( D ) VIA NUMERICAL OPTIMIZATION

Original: We rarely know P X explicitly; e.g., when P X is induced by natural images, characterizing the support of P X (a low dimensional manifold of Euclidean space) alone is non-trivial.
Revised: We rarely know P X explicitly; e.g., when P X is induced by natural images, characterizing the support of P X (a low dimensional manifold of Euclidean space) alone is non-trivial.

Original: However, if we can compute the expectations in Eqs. 2 and 3, then we can still obtain a lower bound of R ( D ) by numerically solving a constrained optimization problem.
Revised: However, if we can compute the expectations in Eqs. 2 and 3, then we can still obtain a lower bound of R ( D ) by numerically solving a constrained optimization problem.

Original: The main idea is to replace maximizing over functions g ≥ 0 by a subset of functions g θ ≥ 0 parameterized by a vector θ .
Revised: The main idea is to replace maximizing over functions g ≥ 0 by a subset of functions g θ ≥ 0 parameterized by a vector θ .

Original: Any feasible ( g θ , λ ) would then give us a (possibly loose) lower bound on R ( D ) .
Revised: Any feasible ( g θ , λ ) would then give us a (possibly loose) lower bound on R ( D ) .

Original: We can use a ﬂexible family of functions for g θ , such as neural networks with a non-negative output activation to ensure g θ ≥ 0 (or, we can parameterize log g instead).
Revised: We can use a ﬂexible family of functions for g θ , such as neural networks with a non-negative output activation to ensure g θ ≥ 0 (or, we can parameterize log g instead).

Original: Noting that the constraint Eq.
Revised: Noting that the constraint Eq.

Original: 3 is equivalent to the constraint function c ( g, λ ) :
Revised: 3 is equivalent to the constraint function c ( g, λ ) :

Original: = sup y E [exp( − λρ ( X, y )) /g
Revised: = sup y E [exp( − λρ ( X, y )) /g

Original: ( X )] − 1 being non-positive, we can then consider solving the following constrained maximization problem to obtain a R ( D ) lower bound, R L ( D ) :
Revised: ( X )] − 1 being non-positive, we can then consider solving the following constrained maximization problem to obtain a R ( D ) lower bound, R L ( D ) :

Original: [Equation]
Revised: [Equation]

Original: where f ( θ, λ) :
Revised: where f ( θ, λ

Revised: [Equation]

Original: = E [ − log g θ ( X
Revised: := E [ − log g θ [Equation] X

Original: [Equation]
Revised: [Equation]

Original: subject to
Revised: subject to

Original: [Equation]
Revised: [Equation]

Original: Assuming the supremum in Eq.
Revised: Assuming the supremum in Eq.

Original: 5 can be computed exactly, then a variety of algorithms can be used to ﬁnd a local optimum of this problem.
Revised: 11 can be computed exactly, then a variety of algorithms can be used to ﬁnd a local optimum of this problem.

Original: We consider a simple formulation based on exact penalty method (Nocedal & Wright, 2006): we solve a sequence of unconstrained problems
Revised: We consider a simple formulation based on exact penalty method (Nocedal & Wright, 2006): we solve a sequence of unconstrained problems

Original: [Equation]
Revised: [Equation]

Original: 6) where  max { 0 , c ( θ, λ ) } penalizes the objective by the amount of constraint violation, and the sequence of penalty coefﬁcients γ t is gradually increased.
Revised: 6) where [ c ( θ, λ )] + = max { 0 , c ( θ, λ ) } measures the amount of constraint violation, and the sequence of penalty coefﬁcients γ t is gradually increased.

Original: Under fairly general conditions, a feasible solution { ( θ ∗ t , λ ∗ t ) } to Eq.
Revised: Under fairly general conditions, a feasible solution { ( θ ∗ t , λ ∗ t ) } to Eq.

Original: 6 is also a local optimum of the original constrained optimization problem in Eqs.
Revised: 6 is also a local optimum of the original constrained optimization problem in Eqs.

Original: 4, 5 (Theorem 17.4 of Nocedal & Wright (2006)).
Revised: 10, 11 (Theorem 17.4 of Nocedal & Wright (2006)).

Original: Then we can estimate R L = f
Revised: Then we can estimate R L = f

Original: ( θ ∗ t , λ ∗ t ) .
Revised: ( θ ∗ t , λ ∗ t ) .

Original: 2. A N A PPROXIMATE A LGORITHM USING STOCHASTIC ESTIMATORS
Revised: 2. A N A PPROXIMATE A LGORITHM USING STOCHASTIC ESTIMATORS

Original: In most practical problems, the expectations in the optimization problem can only be approximated by Monte-Carlo averaging over samples of X .
Revised: In most practical problems, the expectations in the optimization problem can only be approximated by Monte-Carlo averaging over samples of X .

Original: Let’s focus on the constraint function c for now, and denote ψ ( x, y ) :
Revised: Let’s focus on the constraint function c for now, and denote ψ ( x, y ) :

Original: = exp( − λρ ( x,y ))
Revised: = exp( − λρ ( x,y ))

Original: g ( x ) − 1 , suppressing its dependence on ( g, λ ) for brevity.
Revised: g ( x ) − 1 , suppressing its dependence on ( g, λ ) for brevity.

Original: Note, however, that c
Revised: Note, however, that c

Original: = sup y E
Revised: = sup y E

Original: [ ψ ( X, y )] does not have the form of an expectation due to the maximization over y .
Revised: [ ψ ( X, y )] does not have the form of an expectation due to the maximization over y .

Original: Still, we may try to estimate c by maximizing the sampled approximation of E [ ψ ( X, y )] , i.e., compute the k -sample estimator C k :=
Revised: Still, we may try to estimate c by maximizing the sampled approximation of E [ ψ ( X, y )] , i.e., compute the k -sample estimator C k :=

Original: sup y 1 k
Revised: sup y 1 k

Original: (cid:80)
Revised: (cid:80)

Original: ki =1 ψ ( X i , y ) , with X i ∼ P X .
Revised: ki =1 ψ ( X i , y ) , with X i ∼ P X .

Original: What’s the relation between C k and c ?
Revised: What’s the relation between C k and c ?

Original: Given a candidate ( g, λ ) , can we verify its feasibility using C k , instead of the intractable c ?
Revised: Given a candidate ( g, λ ) , can we verify its feasibility using C k , instead of the intractable c ?

Original: We answer these questions in the following theorem (proved in Appendix Sec.
Revised: We answer these questions in the following theorem (proved in Appendix Sec.

Original: A.2):
Revised: A.2):

Original: Theorem 2.2.
Revised: Theorem 2.2.

Original: Let X 1 , X 2 , ... be a sequence of i.i.d.
Revised: Let X 1 , X 2 , ... be a sequence of i.i.d.

Original: random variables with the shared data distribution X k ∼ P X .
Revised: random variables with the shared data distribution X k ∼ P X .

Original: Deﬁne the sequence of random variables
Revised: Deﬁne the sequence of random variables

Original: C k : = sup y 1 k
Revised: C k := sup y 1 k

Original: (cid:80)
Revised: (cid:80)

Original: i ψ ( X i , y ) .
Revised: i ψ ( X i , y ) .

Original: Then1.
Revised: Then

Original: E [ C k ] =
Revised: E [ C k ] =

Original: E X 1 ,...,X k [sup y 1 k (cid:80)
Revised: E X 1 ,...,X k [sup y 1 k (cid:80)

Original: i ψ ( X i , y )] ≥ sup y E
Revised: i ψ ( X i , y )] ≥ sup y E

Original: [ ψ ( X, y )]
Revised: [ ψ ( X, y )]

Original: =: c ; 2.
Revised: =: c ; 2.

Original: E [ C 1 ] ≥ E [ C 2 ] ≥ ... ≥ E [ C k ] ≥ E
Revised: E [ C 1 ] ≥ E [ C 2 ] ≥ ... ≥ E [ C k ] ≥ E

Original: [ C k +1 ] ≥ ... sup y E [ ψ ( X, y )]
Revised: [ C k +1 ] ≥ ... sup y E [ ψ ( X, y )]

Original: = c ; 3.
Revised: = c ; 3.

Original: If ψ ( x, y ) is bounded and continuous in y , and if Y is compact, then C k converges to c almost surely (as well as in probability,
Revised: If ψ ( x, y ) is bounded and continuous in y , and if Y is compact, then C k converges to c almost surely (as well as in probability,

Original: i
Revised: i

Original: e., lim k →∞ P ( | C k − c | > (cid:15) )
Revised: e., lim k →∞ P ( | C k − c | > (cid:15) )

Original: = 0 , ∀ (cid:15) > 0 ), and lim k →∞ E 
Revised: = 0 , ∀ (cid:15) > 0 ) and lim k →∞ E [ C k ]

Original: [ C k ] = c .
Revised: = c .

Original: Theorem 2.2 tells us that C k is on average an over-estimator of the value of the constraint function c ; and like the Importance-Weighted ELBO (Burda et al., 2015), the bias of the estimator decreases monotonically as k → ∞ , and that under continuity assumptions, C k is asymptotically unbiased and converges to c .
Revised: Theorem 2.2 tells us that C k is on average an over-estimator of the value of the constraint function c ; and like the Importance-Weighted ELBO (Burda et al., 2015), the bias of the estimator decreases monotonically as k → ∞ , and that under continuity assumptions, C k is asymptotically unbiased and converges to c .

Original: This means that we can form an empirical estimate of the constraint function c by computing ˆ C k :=
Revised: This means that we can form an empirical estimate of the constraint function c by computing ˆ C k :=

Original: sup y 1 k
Revised: sup y 1 k

Original: (cid:80)
Revised: (cid:80)

Original: i ψ ( x i , y ) from a large batch of samples { x 1 , x 2 , ..., x k } (or, to be more precise, we average many such ˆ C k computed from multiple batches to estimate E [ C k ] ); if the empirical estimate of c is satisﬁed, then we can be conﬁdent so is the true constraint c satisﬁed.
Revised: i ψ ( x i , y ) from a large batch of samples { x 1 , x 2 , ..., x k } (or, to be more precise, we average many such ˆ C k computed from multiple batches to estimate E [ C k ] ); if the empirical estimate of c is satisﬁed, then we can be conﬁdent so is the true constraint c satisﬁed.

Original: In light of this result, we propose to replace  c  by its sample-based estimator ˆ C k , and also use a
Revised: In light of this, we can replace the original constraint c ≤ 0 by the more conservative constraint

Original: Monte-Carlo estimate of the objective ˆ f := 1 m (cid:80) m j =1 − log g ( x j ) − λD . Our proposed stochastic optimization algorithmthen solves the following sequence of penalty problems,
Revised: E [ C k ] ≤ 0 . Using the unbiased estimator ˆ C k for E [ C k ] , and similarly the sample-mean ˆ f :=m (cid:80) m j =1 − log g ( x j ) − λD for f , our algorithm solves the following sequence of penalty problems,

Original: [Equation]
Revised: [Equation]

Original: where given a penalty coefﬁcient γ t , we perform stochastic gradient ascent on the unconstrained objective ˆ (cid:96) ( θ, λ ) till convergence; we then increase the penalty e.g., γ t +1 :=
Revised: where given a penalty coefﬁcient γ t , we perform stochastic gradient ascent on the unconstrained objective ˆ (cid:96) ( θ, λ ) till convergence; we then increase the penalty e.g., γ t +1 :=

Original: βγ t with β > 1 , and solve the maximization problem w.r.t.
Revised: βγ t with β > 1 , and solve the maximization problem w.r.t.

Original: ( θ, λ ) again, until we can verify feasibility of resulting ( θ, λ ) (e.g., ˆ C k ≤ 0 ) with high conﬁdence.
Revised: ( θ, λ ) again, until we can verify feasibility of resulting ( θ, λ ) (e.g., ˆ C k ≤ 0 ) with high conﬁdence.

Original: We specify Algorithm 1 in pseudocode in Appendix Sec.
Revised: We specify Algorithm 1 in full detail in Appendix Sec.

Original: A.3.
Revised: A.3.

Original: 2. I NNER OPTIMIZATION WITH RESPECT TO y
Revised: 2. I NNER OPTIMIZATION WITH RESPECT TO y

Original: So far we have assumed that we can exactly compute the supremum ˆ C k
Revised: So far we have assumed that we can exactly compute the supremum ˆ C k

Original: = sup y 1 k
Revised: = sup y 1 k

Original: (cid:80)
Revised: (cid:80)

Original: i ψ ( x i , y ) .
Revised: i ψ ( x i , y ) .

Original: In practice, 1 k (cid:80) i ψ ( x i , y ) is rarely concave in y , and we can only ﬁnd a local optimum, e.g., with a gradient-based method.
Revised: In practice, 1 k (cid:80) i ψ ( x i , y ) is rarely concave in y , and we can only ﬁnd a local optimum, e.g., with a gradient-based method.

Original: This is less problematic during training, as long as ( g, λ ) receive the appropriate training signal to eventually arrive at a feasible solution.
Revised: This is less problematic during training, as long as ( g, λ ) receive the appropriate training signal to eventually arrive at a feasible solution.

Original: However, once the training terminates with some candidate solution ( g ∗ , λ ∗ ) , we need to compute ˆ C k exactly (ideally with as large k as possible), in order to verify that the solution ( g ∗ , λ ∗ ) is feasible (with
Revised: However, once the training terminates with some candidate solution ( g ∗ , λ ∗ ) , we need to compute ˆ C k exactly (ideally with as large k as possible), in order to verify that the solution ( g ∗ , λ ∗ ) is feasible (with

Original: [Figure]
Revised: [Figure]

Original: high probability) and that ˆ f ( g ∗ , λ ∗ ) gives a valid estimate of a R ( D ) lower bound.
Revised: high probability) and that ˆ f ( g ∗ , λ ∗ ) gives a valid estimate of a R ( D ) lower bound.

Original: This requirement makes it computationally expensive (if not infeasible) to verify a solution and indeed poses an open challenge for our method; techniques from global optimization might offer useful tools.
Revised: This requirement makes it computationally expensive (if not infeasible) to verify a solution and indeed poses an open challenge for our method; techniques from global optimization might offer useful tools.

Original: We note that in the simple case X
Revised: We note that in the simple case X

Original: =
Revised: =

Original: Y = R n and ρ is the (mean) squared error, maximizing φ ( y ) :
Revised: Y = R n and ρ is the (mean) squared error, maximizing φ ( y ) :

Original: = 1 k (cid:80) i ψ ( x i , y )
Revised: = 1 k (cid:80) i ψ ( x i , y )

Original: + 1 =
Revised: + 1 =

Original: α (cid:80) k
Revised: α (cid:80) i

Original: α k exp( − λ (cid:107) x
Revised: α i exp( − λ (cid:107) x

Original: i − y (cid:107) 2 ) is equivalent to ﬁnding the mode of a k -component Gaussian mixture density.
Revised: i − y (cid:107) 2 ) is equivalent to ﬁnding the mode of a k -component Gaussian mixture density.

Original: Global optimization algorithms for the latter problem exist with (partial) optimality guarantees (Carreira-Perpinan, 2000); e.g., we can run k gradient ascent procedures separately from each of the component modes x 1 , ..., x k , and take the largest result.
Revised: Global optimization algorithms for the latter problem exist with (partial) optimality guarantees (Carreira-Perpinan, 2000); e.g., we can run k gradient ascent procedures separately from each of the component modes x 1 , ..., x k , and take the largest result.

Original:  R ESULTS
Revised:  R ESULTS

Original: We apply our algorithm to a 2D standard Gaussian source, using mean-squared error as distortion ρ .
Revised: We apply our algorithm to a 2D standard Gaussian source, using mean-squared error as distortion ρ .

Original: We parameterize log g by a two-layer fully connected neural network.
Revised: We parameterize log g by a two-layer fully connected neural network.

Original: During training, we use k = 100 for the constraint estimator ˆ C k , and run a simple gradient ascent procedure in the inner optimization w.r.t.
Revised: During training, we use k = 100 for the constraint estimator ˆ C k , and run a simple gradient ascent procedure in the inner optimization w.r.t.

Original: y for computational efﬁciency.
Revised: y for computational efﬁciency.

Original: We start with a penalty parameter γ 0 = 1 , and double it every 1000 training steps (which seemed sufﬁcient for convergence).
Revised: We start with a penalty parameter γ 0 = 1 , and double it every 1000 training steps (which seemed sufﬁcient for convergence).

Original: We stop training after 4000 steps, at which point our simple estimate of constraint ˆ C k is frequently satisﬁed across training steps; see Fig.
Revised: We stop training after 4000 steps, at which point our simple estimate of constraint ˆ C k is frequently satisﬁed across training steps; see Fig.

Original: 1 (Left) for example training curves.
Revised: 1 (Left) for example training curves.

Original: To certify the resulting solution ( g ∗ , λ ∗ ) as valid, we then compute ˆ C k with the global optimization algorithm described in Section 2.4, using a large number of samples ( k = 10000 ); in all our experiments, ˆ C 10000 ( g ∗ , λ ∗ ) < 0 , so we conclude the solutions are feasible, and ˆ f ( g ∗ , λ ∗ ) gives a valid lower bound of R ( D ) with very high probability.
Revised: To certify the resulting solution ( g ∗ , λ ∗ ) as valid, we then compute ˆ C k with the global optimization algorithm described in Section 2.4, using a large number of samples ( k = 10000 ); in all our experiments, ˆ C 10000 ( g ∗ , λ ∗ ) < 0 , so we conclude the solutions are feasible, and ˆ f ( g ∗ , λ ∗ ) gives a valid lower bound of R ( D ) with very high probability.

Original: We repeat the above with D ranging from 0.02 to 0.8, and plot the corresponding ˆ R L ( D )
Revised: We repeat the above with D ranging from 0.02 to 0.8, and plot the corresponding ˆ R L ( D )

Original: =
Revised: =

Original: ˆ f ( g ∗ , λ ∗ ) to trace out an R ( D ) lower bound.
Revised: ˆ f ( g ∗ , λ ∗ ) to trace out an R ( D ) lower bound.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 1 (Middle), our lower bound shows good agreement with the true R ( D ) , with an average gap of 0.74 bit.
Revised: 1 (Middle), our lower bound shows good agreement with the true R ( D ) , with an average gap of 0.74 bit.

Original: Fig.
Revised: Fig.

Original: 1 (Right) visualizes the optimization problem for D = 0 .
Revised: 1 (Right) visualizes the optimization problem for D = 0 .

Original: 6 , where we take a vertical slice of R 2 and show the learned g θ ∗ ( x ) , the theoretically derived optimal g ( x ) (a scaled Gaussian density), and the ˆ C k optimization objective φ ( y ) and its component functions φ x i ( y ) centered on k = 20 random samples, plugging in the learned ( g θ ∗ , λ ∗ ) .
Revised: 6 , where we take a vertical slice of R 2 and show the learned g θ ∗ ( x ) , the theoretically derived optimal g ( x ) (a scaled Gaussian density), and the ˆ C k optimization objective φ ( y ) and its component functions φ x i ( y ) centered on k = 20 random samples, plugging in the learned ( g θ ∗ , λ ∗ ) .

Original: Note that the component functions have the form of a Gaussian kernel, whose “sharpness” is controlled by λ .
Revised: Note that the component functions have the form of a Gaussian kernel, whose steepness is controlled by λ .

Original: When λ is large or k is small, the resulting ˆ C k = max y φ ( y ) −more severely overestimates the true constraint c .
Revised: When λ is large or k is small, the resulting ˆ C k = max y φ ( y ) − 1 more severely overestimates the true constraint c .

Original: With a smaller λ , or by increasing k , the landscape of φ ( y ) is more smoothed out, so the overestimating effect is reduced.
Revised: With a smaller λ , or by increasing k , the landscape of φ ( y ) is more smoothed out, so the overestimating effect is reduced.

Original: Indeed, in this ﬁgure we see max y φ ( y ) > 1 . 25 ; by increasing the sample size k from 20 to 10000, max y φ ( y ) would decrease to around 0.93 instead (so that ˆ C k ≈ − 0 .
Revised: Indeed, in this ﬁgure we see max y φ ( y ) > 1 . 25 ; by increasing the sample size k from 20 to 10000, max y φ ( y ) would decrease to around 0.93 instead (so that ˆ C k ≈ − 0 .

Original: 07 < 0 ), as predicted by Theorem 2.2.
Revised: 07 < 0 ), as predicted by Theorem 2.2.

Original: In the low distortion (small D ) regime, λ is driven to large values, so that ˆ C k exhibits higher bias, and constraint violations max { 0 , ˆ C k } are often penalized overly harshly (as ˆ C k tends to overestimate c ); this likely explains why our lower bound has a larger gap to the true R ( D ) as we decrease D .
Revised: In the low distortion (small D ) regime, λ is driven to large values, so that ˆ C k exhibits higher bias, and constraint violations max { 0 , ˆ C k } are often penalized overly harshly (as ˆ C k tends to overestimate c ); this likely explains why our lower bound has a larger gap to the true R ( D ) as we decrease D .

Original: We expect the bound can be improved by more accurately estimating the gradient of the penalty/constraint function (we elaborate this point in Appendix Sec. A.3), and/or training with a larger k especially in the lower distortion regime.
Revised: We expect the bound can be improved by more accurately estimating the gradient of the penalty/constraint function (we elaborate this point in Appendix Sec. A.3), and/or training with a larger k especially in the lower distortion regime.

Original: We provide more experimental details and visualizations in Appendix Sec.
Revised: We provide more experimental details and visualizations in Appendix Sec.

Original: A.4.
Revised: A.4.

Original:  A CKNOWLEDGEMENTS
Revised:  A CKNOWLEDGEMENTS

Original: This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. HR001120C0021.
Revised: This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. HR001120C0021.

Original: Any opinions, ﬁndings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reﬂect the views of the Defense Advanced Research Projects Agency (DARPA).
Revised: Any opinions, ﬁndings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reﬂect the views of the Defense Advanced Research Projects Agency (DARPA).

Original: Yibo Yang acknowledges funding from the Hasso Plattner Foundation.
Revised: Yibo Yang acknowledges funding from the Hasso Plattner Foundation.

Original: Furthermore, this work was supported by the National
Revised: Furthermore, this work was supported by the National

Original: Science Foundation under Grants 1928718, 2003237 and 2007719, as well as Intel and Qualcomm.
Revised: Science Foundation under Grants 1928718, 2003237 and 2007719, as well as Intel and Qualcomm.

Original: A A PPENDIX A. F ULL V ERSION OF T HEOREM 2. 
Revised: A A PPENDIX A. F ULL V ERSION OF T HEOREM 2.

Original: Theorem A.1.
Revised: Theorem A.1.

Original: (Kostina, 2016) Suppose that the following basic assumptions are satisﬁed.
Revised: (Kostina, 2016) Suppose that the following basic assumptions are satisﬁed.

Original: R ( D ) is ﬁnite for some D , i.e., D min :
Revised: R ( D ) is ﬁnite for some D , i.e., D min :

Original: = inf { D : R ( D ) ≤ ∞} < ∞ ; 2.
Revised: = inf { D : R ( D ) ≤ ∞} < ∞ ; 2.

Original: The distortion metric ρ is such that there exists a ﬁnite set E ⊂ Y such that
Revised: The distortion metric ρ is such that there exists a ﬁnite set E ⊂ Y such that

Original: [Equation]
Revised: [Equation]

Original: Then, for each D > D min ,
Revised: Then, for each D > D min ,

Original: it
Revised: it

Original: holds that
Revised: holds that

Original: [Equation]
Revised: [Equation]

Original: where the maximization is over g ( x ) ≥ 0 and λ ≥ 0 satisfying the constraint
Revised: where the maximization is over g ( x ) ≥ 0 and λ ≥ 0 satisfying the constraint

Original: [Equation]
Revised: [Equation]

Original: Note: the basic assumption 2 is trivially satisﬁed when the distortion ρ is bounded from above; the maximization over g ( x ) ≥ 0 can be restricted to only 1 ≥ g ( x ) ≥ 0 .
Revised: Note: the basic assumption 2 is trivially satisﬁed when the distortion ρ is bounded from above; the maximization over g ( x ) ≥ 0 can be restricted to only 1 ≥ g ( x ) ≥ 0 .

Original: Unless stated otherwise, we use log base e in this work, so the R ( D ) above is in terms of nats.
Revised: Unless stated otherwise, we use log base e in this work, so the R ( D ) above is in terms of nats.

Original: A. P ROOF OF T HEOREM 2. 
Revised: A. P ROOF OF T HEOREM 2.

Original: Theorem A.2.
Revised: Theorem A.2.

Original: (We restate the theorem for completeness).
Revised: (We restate the theorem for completeness).

Original: Let X 1 , X 2 , ... be a sequence of i.i.d.
Revised: Let X 1 , X 2 , ... be a sequence of i.i.d.

Original: random variables with the shared data distribution X k ∼ P X .
Revised: random variables with the shared data distribution X k ∼ P X .

Original: Deﬁne the sequence of random variables
Revised: Deﬁne the sequence of random variables

Original: C k := sup y 1 k
Revised: C k := sup y 1 k

Original: (cid:80)
Revised: (cid:80)

Original: i ψ ( X i , y ) .
Revised: i ψ ( X i , y ) .

Original: Then
Revised: Then

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 2 ] ≥ ...
Revised: 2 ] ≥ ...

Original: ≥ E
Revised: ≥ E

Original: [ C k ] ≥ E [ C k +1 ] ≥ ... sup y E
Revised: [ C k ] ≥ E [ C k +1 ] ≥ ... sup y E

Original: [ ψ ( X, y )]
Revised: [ ψ ( X, y )]

Original: = c ; 3.
Revised: = c ; 3.

Original: If ψ ( x, y ) is bounded and is continuous in y , and if Y is compact , then C k converges to c = sup y E
Revised: If ψ ( x, y ) is bounded and is continuous in y , and if Y is compact , then C k converges to c = sup y E

Original: [ ψ ( X, y )] almost surely (as well as in probability), and we have lim k →∞ E [ C k ] = c .
Revised: [ ψ ( X, y )] almost surely (as well as in probability), and we have lim k →∞ E [ C k ] = c .

Original: Proof.
Revised: Proof.

Original: We prove each in turn:
Revised: We prove each in turn:

Original: E [ C k ] =
Revised: E [ C k ] =

Original: E [sup y 1 k (cid:80) i ψ ( X i , y )] ≥ sup y E
Revised: E [sup y 1 k (cid:80) i ψ ( X i , y )] ≥ sup y E

Original: [ 1 k (cid:80) i ψ ( X i , y )]
Revised: [ 1 k (cid:80) i ψ ( X i , y )]

Original: = sup y E
Revised: = sup y E

Original: [ ψ ( X, y )]
Revised: [ ψ ( X, y )]

Original: =
Revised: =

Original: c 2.
Revised: c 2.

Original: First, note that E [ C 1 ] ≥ E [ C k ] since
Revised: First, note that E [ C 1 ] ≥ E [ C k ] since

Original: [Equation]
Revised: [Equation]

Original: X
Revised: X

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We therefore have
Revised: We therefore have

Original: [Equation]
Revised: [Equation]

Original: The proof for this resembles that of Theorem 1 in (Burda et al., 2015).
Revised: The proof for this resembles that of Theorem 1 in (Burda et al., 2015).

Original: We use standard arguments from probability theory and real analysis.
Revised: We use standard arguments from probability theory and real analysis.

Original: Fix y ∈ Y , and consider the random variable M k
Revised: Fix y ∈ Y , and consider the random variable M k

Original: = 1 k (cid:80) ki
Revised: = 1 k (cid:80) ki

Original: =1 ψ ( X i , y ) .
Revised: =1 ψ ( X i , y ) .

Original: If ψ is bounded, then it follows from the Strong Law of
Revised: If ψ is bounded, then it follows from the Strong Law of

Original: Large Numbers that M k converges to E [ M 1 ] =
Revised: Large Numbers that M k converges to E [ M 1 ] =

Original: E [ ψ ( X, y )] almost surely; in other words, for every ω outside a set of measure zero, 
Revised: E [ ψ ( X, y )] almost surely; in other words, for every ω outside a set of measure zero, [Equation]

Revised: k →∞

Original: [Equation]
Revised: [Equation]

Original: Then, for every such ω
Revised: Then, for every such ω

Original: [Equation]
Revised: [Equation]

Original: where we used the fact that the sequence of continuous functions s k ( y ) :
Revised: where we used the fact that the sequence of continuous functions s k ( y ) :

Original: = 1 k (cid:80) ki =1 ψ ( X i ( ω )) , y ) converges pointwise to s ( y ) :
Revised: = 1 k (cid:80) ki =1 ψ ( X i ( ω )) , y ) converges pointwise to s ( y ) :

Original: = E [ ψ ( X ( ω ) , y )] on a compact set Y , so s k converges to s also uniformly, so we are allowed to exchange limit and supremum, i.e.,
Revised: = E [ ψ ( X ( ω ) , y )] on a compact set Y , so s k converges to s also uniformly, so we are allowed to exchange limit and supremum, i.e.,

Original: lim k →∞ sup y s k
Revised: lim k →∞ sup y s k

Original: [Equation]

Original:  y ) =
Revised: ( y ) =

Original: sup
Revised: sup

Original: y lim k →∞ s k ( y )
Revised: y lim k →∞ s k ( y )

Original: =
Revised: =

Original: sup y s ( y ) .
Revised: sup y s ( y ) .

Original: But the above equation precisely means that C k converges to E [ ψ ( X, y )] almost surely.
Revised: But the above equation precisely means that C k converges to E [ ψ ( X, y )] almost surely.

Original: Therefore C k also converges to E [ ψ ( X, y )] in probability, and lim k →∞ E [ C k ]
Revised: Therefore C k also converges to E [ ψ ( X, y )] in probability, and lim k →∞ E [ C k ]

Original: = c .
Revised: = c .

Original: A. P ROPOSED S TOCHASTIC O PTIMIZATION A LGORITHM
Revised: Corollary A.2.1. Under the conditions of Theorem 2.1, if any pair of non-negative ( g, λ ) satisﬁes E [ C k ] ≤ 0 , where C k := sup y 1 k (cid:80)

Original: We give a pseudocode implementation of our proposed stochastic optimization procedure in Algorithm 1.
Revised: i ψ ( X i , y ) = sup y 1 k (cid:80) i ψ ( X i , y ) and ψ ( x, y ) :

Original: We follow a simple exact penalty method (Nocedal & Wright, 2006), but other constrained optimization methods like augmented Lagrangian can also be used instead.
Revised: = exp( − λρ ( x,y )) g ( x ) − 1 as before, then for every D > D min , it holds that E [ − log g θ ( X )] − λD ≤ R ( D ) . Proof. The feasibility of such ( g, λ ) trivially follows from the fact that c ≤ E [ C k ] ≤ 0 by Theorem 2.2; the R ( D ) lower bound E [ − log g θ ( X )] − λD holds for every D > D min by the fact that R ( D ) is a convex function.

Original: Recall the exact form of the unconstrained penalty problem that we would like to solve:
Revised: Our algorithm aims to solve the following constrained optimization problem:

Original: [Equation] where 
Revised: [Equation] where f ( θ, λ ) : = E [ − log g θ ( X )] − λD

Original: [Equation]
Revised: [Equation]

Original: and
Revised: 10) subject to

Original: [Equation]
Revised: [Equation]

Original: To apply stochastic gradient ascent on this problem, we need an estimate of the gradient with respect to θ and λ . Focusing just on θ and dropping dependence on λ to simplify notation, we note that an unbiased gradient estimate of f is straightforward to obtain by a sample mean, as
Revised: which, by Theorem 2.2 and Corollary A.2.1, would guarantee a feasible solution to the original problem and yield a valid lower bound on R ( D ) . We apply the penalty method and solve the following sequence of unconstrained problems:

Original: [Equation]
Revised: [Equation]

Original: Estimating the gradient of the penalty function, ∇ θ max { 0 , c ( θ ) } , requires more work.
Revised: where we have essentially replaced c ( θ, λ ) in Eq. 6 by its overestimator E [ C k ( θ, λ )] .

Original: Let’s switch to the lighter notation c ( θ ) = sup y E [ ψ θ ( X, y )] , where ψ θ ( x, y ) : = exp( − λρ ( x,y
Revised: We estimate E [ C k ( θ, λ )]] + by the plug-in estimator:

Original: [Equation]
Revised: [Equation]

Original: ) g ( x ) − 1 . Suppose y ∗ achieves the supremum in the deﬁnition of c , i.e., c ( θ ) = sup y E [ ψ θ ( X, y )]
Revised: where we draw s mini-batches of data samples, each mini-batch containing k samples.

Original: =

Original: E [ ψ θ ( X, y∗ )] . Focusing on the case where the constraint is violated, i.e., c ( θ ) > 0 and max { 0 ,c ( θ ) } = c ( θ ) (for otherwise wecan simply set the (sub)gradient ∇ θ max { 0 , c ( θ ) } to zero), we have ∇ θ max { 0 , c ( θ ) } = ∇ θ c ( θ ) = ∇ θ sup y E [ ψ θ ( X, y )]
Revised: Since computing each ˆ C k requires solving a global optimization problem with respect to y , we simply use s = 1 for computational efﬁciency, i.e.,

Original: = ∇ θ E [ ψ θ ( X, y ∗ )] , where in the last equality we removed the dependence of the constraint gradient on y by appealing to an envelope theorem (Milgrom & Segal, 2002). Then we can form a simple unbiased gradient estimator by a sample mean, ∇ θ E [ ψ θ ( X, y ∗ )] ≈ 1 k (cid:80) ki =1 ∇ θ ψ θ ( x i , y ∗ ) . The remaining crucial problem, then, is that of computing y ∗ =
Revised: [Equation]

Original: arg max y E [ ψ ( X, y )] in the ﬁrst place.

Original: Here, we implement the approximation y ∗ ≈ arg max y 1 k (cid:80) ki

Original: =1 ψ θ ( x i , y ) in the subroutine estimate constraint , where we also compute the constraint estimator ˆ C k =

Original: Requires: Model g θ (e.g., a neural network), initial parameters ( g θ 0 , λ 0 ) , initial penalty parameter γ 

Revised: Requires: Model g θ (e.g., a neural network), initial parameters ( g θ 0 , λ 0 ) , initial penalty parameter γ

Original: = 1 , batch sizes k, m , gradient ascent step size η , step counter t =while True do // Find a local maximum of ˆ (cid:96) ( θ, λ ) given the current γ t while ( θ t , λ t ) not converged do
Revised: = 1 , batch sizes k, m , gradient ascent step size η , step counter t =while True do // Find a local maximum of ˆ (cid:96) ( θ, λ ) given the current γ t while ( θ t , λ t ) not converged do

Original: Draw two batches of data samples { x 1 , ..., x k } and { x 1 , ..., x m } y ∗ , ˆ C k = estimate constraint ( θ t , λ t , { x 1 , ..., x k } )
Revised: Draw two batches of data samples { x 1 , ..., x k } and { x 1 , ..., x m } y ∗ , ˆ C k = estimate constraint ( θ t , λ t , { x 1 , ..., x k } )

Original: Update parameters by ( θ t +1 , λ t +1 )
Revised: Update parameters by ( θ t +1 , λ t +1 )

Original: = ( θ t , λ t ) + η ∇ ( θ,λ ) (cid:16) 1 m (cid:80) mj
Revised: = ( θ t , λ t ) + η ∇ ( θ,λ ) (cid:16) 1 m (cid:80) mj

Original: =1 − log g θ t ( x j ) − γ t max { 0 , 1 k (cid:80)
Revised: =1 − log g θ t ( x j ) − γ t max { 0 , 1 k (cid:80)

Original: ki
Revised: ki

Original: =1 exp {− λ t ρ ( x i ,y ∗ ) } g θt ( x i ) − 1 } (cid:17)
Revised: =1 exp {− λ t ρ ( x i ,y ∗ ) } g θt ( x i ) − 1 } (cid:17)

Original: Update training step counter t :=
Revised: Update training step counter t :=

Original: t +endSample a large batch { x 1 , ..., x k } (as large as we can afford for global optimization below) y ∗ , ˆ C k = estimate constraint global opt ( θ t , λ t , { x 1 , ..., x k } ) if ˆ C k ≤ 0 then //
Revised: t +end y ∗ , ˆ C k = estimate constraint global opt ( θ t , λ t , { x 1 , ..., x k } ) if ˆ C k ≤ 0 then //

Original: True constraint c is  satisfiedwith high probability
Revised: True constraint c is likely satisfied

Original: Return solution ( θ t , λ t ) , and lower bound estimate ˆ R L = (cid:80) mj =1 − log g θ t ( x j ) else
Revised: Return solution ( θ t , λ t ) , and lower bound estimate ˆ R L = (cid:80) m j =1 − log g θ t ( x j ) else

Original: Increase penalty parameter, γ t := βγ t , with e.g., β = 2 orend endSubroutine estimate constraint ( θ, λ, { x 1 , ..., x k } ) :
Revised: Increase penalty parameter, γ t := βγ t , with e.g., β = 2 orend endSubroutine estimate constraint ( θ, λ, { x 1 , ..., x k } ) :

Original: Run gradient ascent on φ ( y ) :
Revised: Run gradient ascent on φ ( y ) :

Original: = (cid:80) ki
Revised: = (cid:80) ki

Original: =1 exp {− λρ ( x i , y ) } /g θ ( x i ) , until converging to y ∗
Revised: =1 exp {− λρ ( x i , y ) } /g θ ( x i ) , until converging to y ∗

Original: Compute ˆ C k = φ ( y ∗ ) 
Revised: Compute ˆ C k = φ ( y ∗ ) −

Original: Return ( y ∗ , ˆ C k )
Revised: Return ( y ∗ , ˆ C k )

Original: Subroutine estimate constraint global opt ( θ, λ, { x 1 , ..., x k } ) : Find global maximizer y ∗ = arg max y φ ( y )
Revised: Subroutine estimate constraint global opt ( θ, λ, { x 1 , ..., x k } ) : Find global maximizer y ∗ = arg max y φ ( y )

Original: Compute ˆ C k = φ ( y ∗ ) 
Revised: Compute ˆ C k = φ ( y ∗ ) −

Original: Return ( y ∗ , ˆ C k )max y 1 k
Revised: Return ( y ∗ , ˆ C k )

Original: (cid:80) ki =
Revised: The sampled loss function of the unconstrained penalty (sub)problem is then [Equation] To apply stochastic gradient ascent, we derive the gradient for each of the two terms with respect to θ (the gradient with respect to λ is similar). The gradient of ˆ f is simply a sample average:

Original: 1 ψ θ ( x i , y ) =
Revised: [Equation]

Original: 1 k (cid:80) ki =
Revised: The gradient with respect to the penalty term requires more work. First we need to compute ˆ C k by solving a global optimization problem, ﬁnding a y ∗ such that [Equation] and then we have

Original: 1 ψ θ ( x i , y ∗ ) along the way, so that ∇ θ max { 0 , c ( θ ) } ≈ ∇ θ max { 0 , ˆ C k ( θ ) } .
Revised: [Equation] where in the case ˆ C k > 0 , we plug in y ∗ and remove the dependence of the gradient on y by appealing to an envelope theorem (Milgrom & Segal, 2002). We give a pseudocode implementation of our proposed stochastic optimization procedure in Algorithm 1. We follow a simple exact penalty method (Nocedal & Wright, 2006), but other constrained optimization methods like augmented Lagrangian can also be used instead.

Original: This isdone by a simple gradient ascent procedure, intended as a computationally cheaper alternative to the exact (but expensive) estimate constraint global opt procedure, which we reserveonly for verifying the feasibility of a candidate solution.
Revised: The subroutine estimate constraint in the inner loop is intended as a computationally cheaper alternative to the exact (but expensive) estimate constraint global opt procedure, which we reserve for verifying the feasibility of a candidate solution.

Original: Alternatively, we could approximately ﬁnd y ∗ by running stochastic gradient ascent  on E [ ψ (X, y )] with respect to  y , but we ﬁnd this can yield worse local maxima than our deterministic optimization procedure.
Revised: We give an example implementation of estimate constraint that simply performs a single gradient ascent run on φ ( y ) to estimate ˆ C k (and its gradient), which proved sufﬁcient for training in our experiments; however, since gradient ascent only gives us a local maximum of φ ( y ) in general, this naive implementation of the algorithm may run into convergence issues.

Original: A. M ORE EXPERIMENTAL DETAILS AND RESULTS
Revised: A. M ORE EXPERIMENTAL DETAILS AND RESULTS

Original: We used a simple 2 layer fully connected network with a scalar output for log g , with 8 hidden units in each layer.
Revised: We used a simple 2 layer fully connected network with a scalar output for log g , with 8 hidden units in each layer.

Original: We parameterized λ as λ = exp ˜
Revised: We parameterized λ as λ = exp ˜

Original: λ using an unconstrained variable ˜ λ .
Revised: λ using an unconstrained variable ˜ λ .

Original: We used the
Revised: We used the

Original: Adam optimizer for all gradient based optimization.
Revised: Adam optimizer for all gradient based optimization.

Original: During training, we set batch size k
Revised: During training, we set batch size k

Original: = m = 100 , and implement estimate constraint by a single gradient ascent run, initializing y to the data sample in the batch that achieves the highest value of φ ( · ) (we ﬁnd thatrandomly initializing y to a data sample to also work reasonably well).
Revised: = m = 100 , and implement estimate constraint by a single gradient ascent run, initializing y to the data sample in the batch that achieves the highest value of φ ( · ) (we ﬁnd that initializing y to the batch mean 1 k (cid:80) ki =1 x i also works well).

Original: We used the ( g ∗ , λ ∗ ) trained after 3500 steps for estimating the ﬁnal R ( D ) lower bound.
Revised: We used the ( g ∗ , λ ∗ ) trained after 3500 steps for estimating the ﬁnal R ( D ) lower bound.

Original: We ﬁrst veriﬁed their feasibility by running the global optimization procedure of Carreira-Perpinan (2000) with k = 10000 , as assured by the negative values of the resulting ˆ C 10000 , which were − 0 .
Revised: We ﬁrst veriﬁed their feasibility by running the global optimization procedure of Carreira-Perpinan (2000) with k = 10000 , as assured by the negative values of the resulting ˆ C 10000 , which were − 0 .

Original: 199 , − 0 .
Revised: 199 , − 0 .

Original: 106 , − 0 .
Revised: 106 , − 0 .

Original: 254 , − 0 .
Revised: 254 , − 0 .

Original: 202 , − 0 .
Revised: 202 , − 0 .

Original: 115 , − 0 .
Revised: 115 , − 0 .

Original: 068 , − 0 .
Revised: 068 , − 0 .

Original: 052 , for D = 0 .
Revised: 052 , for D = 0 .

Original: Then we compute ˆ f m from m =  samples, repeat for  different batches, and plot the mean ˆ f  as the lower bound ˆ R L ( D ) in
Revised: Then we compute ˆ f m from m = 10000 samples, repeat for 10 different batches, and plot the mean ˆ f m as the lower bound ˆ R L ( D ) in

Original: Figure (Middle), corresponding to the ( D, R ) points (0 . 02 , 4 . 062) , (0 . 05 , 3 . 343) , (0 . 1 , 2 . 424) , (0 . 2 , 1 . 600) , (0 . 4 , 0 . 853) , (0 . 6 , 0 . 432) , (0 . 8 , 0 . 125) , in terms of (MSE, bits).
Revised: Figure (Middle), corresponding to the ( D, R ) points (0 . 02 , 4 . 062) , (0 . 05 , 3 . 343) , (0 . 1 , 2 . 424) , (0 . 2 , 1 . 600) , (0 . 4 , 0 . 853) , (0 . 6 , 0 . 432) , (0 . 8 , 0 . 125) , in terms of (MSE, bits).

Original: The standard deviations of ˆ f from the 10 different batches are between 0 .
Revised: The standard deviations of ˆ f from the 10 different batches are between 0 .

Original: 005 and 0 .
Revised: 005 and 0 .

Original: 007 , so the error bars do not appear noticeable in the ﬁgure.
Revised: 007 , so the error bars do not appear noticeable in the ﬁgure.

Original: In our problem setting, the optimal g ( x ) can be found analytically as g ( x ) =
Revised: In our problem setting, the optimal g ( x ) can be found analytically as g ( x ) =

Original: f X ( x )(2 πD ) n/ 2 , where n = 2 and f X is the density of the source X (2D standard Gaussian).
Revised: f X ( x )(2 πD ) n/ 2 , where n = 2 and f X is the density of the source X (2D standard Gaussian).

Original: The Shannon lower bound coincides with the true R ( D ) (Cover, 1999), giving R ( D )
Revised: The Shannon lower bound coincides with the true R ( D ) (Cover, 1999), giving R ( D )

Original: =
Revised: =

Original: h ( X ) − h
Revised: h ( X ) − h

Original: ( D ) , where h ( X ) , h ( D ) are the differential entropy of X , and an isotropic n -dimensional Gaussian distribution with diagonal covariances equal to D , respectively.
Revised: ( D ) , where h ( X ) , h ( D ) are the differential entropy of X , and an isotropic n -dimensional Gaussian distribution with diagonal covariances equal to D , respectively.

Original: In the next ﬁgure, we visualize the evolution of g θ during training for a few experiments.
Revised: In the next ﬁgure, we visualize the evolution of g θ during training for a few experiments.