Original: An Information-Theoretic Framework for Deep Learning
Revised: An Information-Theoretic Framework for Deep Learning

Original: Abstract
Revised: Abstract

Original: Each year, deep learning demonstrates new and improved empirical results with deeper and wider neural networks.
Revised: Each year, deep learning demonstrates new and improved empirical results with deeper and wider neural networks.

Original: Meanwhile, with existing theoretical frameworks, it is difficult to analyze networks deeper than two layers without resorting to counting parameters or encountering sample complexity bounds that are exponential in depth.
Revised: Meanwhile, with existing theoretical frameworks, it is difficult to analyze networks deeper than two layers without resorting to counting parameters or encountering sample complexity bounds that are exponential in depth.

Original: Perhaps it may be fruitful to try to analyze modern machine learning under a different lens.
Revised: Perhaps it may be fruitful to try to analyze modern machine learning under a different lens.

Original: In this paper, we propose a novel information-theoretic framework with its own notions of regret and sample complexity for analyzing the data requirements of machine learning.
Revised: In this paper, we propose a novel information-theoretic framework with its own notions of regret and sample complexity for analyzing the data requirements of machine learning.

Original: We use this framework to study the sample complexity of learning from data generated by deep ReLU neural networks and deep networks that are infinitely wide but have a bounded sum of weights.
Revised: We use this framework to study the sample complexity of learning from data generated by deep ReLU neural networks and deep networks that are infinitely wide but have a bounded sum of weights.

Original: We establish that the sample complexity of learning under these data generating processes is at most linear and quadratic, respectively, in network depth.
Revised: We establish that the sample complexity of learning under these data generating processes is at most linear and quadratic, respectively, in network depth.

Original:  Introduction
Revised:  Introduction

Original: In modern machine learning, the apparent capabilities of empirical methods have rapidly outpaced what is soundly understood theoretically.
Revised: In modern machine learning, the apparent capabilities of empirical methods have rapidly outpaced what is soundly understood theoretically.

Original: Modern neural network architectures have scaled immensely in both parameter count and depth.
Revised: Modern neural network architectures have scaled immensely in both parameter count and depth.

Original: GPT3, for example, encodes about 175 billion parameters in 96 decoder blocks, each with many layers within.
Revised: GPT3, for example, encodes about 175 billion parameters in 96 decoder blocks, each with many layers within.

Original: Yet, contrary to traditional intuition, these deep neural networks with gargantuan parameter counts are able to generalize well and produce useful models with tractable amounts of data.
Revised: Yet, contrary to traditional intuition, these deep neural networks with gargantuan parameter counts are able to generalize well and produce useful models with tractable amounts of data.

Original: This gap between what has been shown theoretically versus empirically makes it quite enticing to develop a coherent framework that could potentially explain this phenomenon.
Revised: This gap between what has been shown theoretically versus empirically makes it quite enticing to develop a coherent framework that could potentially explain this phenomenon.

Original: Existing theory breaks down when trying to explain learning under models that are simultaneously very deep (many layers) and wide (many hidden units per layer).
Revised: Existing theory breaks down when trying to explain learning under models that are simultaneously very deep (many layers) and wide (many hidden units per layer).

Original: Classical results such as those of
Revised: Classical results such as those of

Original: Haussler [1992] and Bartlett et al.
Revised: Haussler [1992] and Bartlett et al.

Original: [1998] can potentially handle the deep but narrow case.
Revised: [1998] can potentially handle the deep but narrow case.

Original: These results bound the sample complexity of a learning a neural network function in terms of the number of parameters and the depth.
Revised: These results bound the sample complexity of a learning a neural network function in terms of the number of parameters and the depth.

Original: More recently, Harvey et al.
Revised: More recently, Harvey et al.

Original: [2017] established a general result that suggests that for neural networks with piecewise-linear activation units, the sample complexity grows linearly in the product of parameter count and depth.
Revised: [2017] established a general result that suggests that for neural networks with piecewise-linear activation units, the sample complexity grows linearly in the product of parameter count and depth.

Original: However, when we consider neural networks with arbitrary width, these bounds become vacuous.
Revised: However, when we consider neural networks with arbitrary width, these bounds become vacuous.

Original: As an alternative to parameter count methods, researchers have produced sample complexity bounds that depend on the product of norms of realized weight matrices.
Revised: As an alternative to parameter count methods, researchers have produced sample complexity bounds that depend on the product of norms of realized weight matrices.

Original: Bartlett et al.
Revised: Bartlett et al.

Original: [2017] and Neyshabur et al.
Revised: [2017] and Neyshabur et al.

Original: [2018], for example, establish sample complexity bounds that scale with the product of spectral norms.
Revised: [2018], for example, establish sample complexity bounds that scale with the product of spectral norms.

Original: Neyshabur et al.
Revised: Neyshabur et al.

Original: [2015] and Golowich et al.
Revised: [2015] and Golowich et al.

Original: [2018] establish similar bounds that instead scale in the product of Frobenius norms.
Revised: [2018] establish similar bounds that instead scale in the product of Frobenius norms.

Original: While this line of work provides sample complexity bounds that are width -independent, they pay for it via an exponential dependence on depth , which is also inconsistent with empirical results.
Revised: While this line of work provides sample complexity bounds that are width -independent, they pay for it via an exponential dependence on depth , which is also inconsistent with empirical results.

Original: A large line of work has tried to ameliorate this exponential depth dependence via so-called datadependent quantities Dziugaite and Roy [2017], Arora et al.
Revised: A large line of work has tried to ameliorate this exponential depth dependence via so-called datadependent quantities Dziugaite and Roy [2017], Arora et al.

Original: [2018], Nagarajan and Kolter [2018],
Revised: [2018], Nagarajan and Kolter [2018],

Original: Wei and Ma [2019].
Revised: Wei and Ma [2019].

Original: Among these, the most relevant to our work is Wei and Ma [2019], which bounds sample complexity as a function of depth and statistics of trained neural network weights.
Revised: Among these, the most relevant to our work is Wei and Ma [2019], which bounds sample complexity as a function of depth and statistics of trained neural network weights.

Original: While difficult to interpret due to dependence on complicated data-dependent statistics, their bound suggests a nonic dependence on depth.
Revised: While difficult to interpret due to dependence on complicated data-dependent statistics, their bound suggests a nonic dependence on depth.

Original: Arora et al.
Revised: Arora et al.

Original: [2018] also utilize concepts of compression in their analysis, which we generalize and expand upon.
Revised: [2018] also utilize concepts of compression in their analysis, which we generalize and expand upon.

Original: While they establish a sample complexity bound that suggests quadratic dependence on depth, further dependence may be hidden in datadependent terms.
Revised: While they establish a sample complexity bound that suggests quadratic dependence on depth, further dependence may be hidden in datadependent terms.

Original: We suspect that the looseness of these results in comparison to empirical findings are due to a worstcase analysis framework.
Revised: We suspect that the looseness of these results in comparison to empirical findings are due to a worstcase analysis framework.

Original: In this paper, we study an average-case notion of regret and sample complexity that is motivated by information theory.
Revised: In this paper, we study an average-case notion of regret and sample complexity that is motivated by information theory.

Original: Our information-theoretic framework generalizes that developed by Haussler et al.
Revised: Our information-theoretic framework generalizes that developed by Haussler et al.

Original: [1994], which provided a basis for understanding the relationship between prediction error and information.
Revised: [1994], which provided a basis for understanding the relationship between prediction error and information.

Original: In a similar vein, Russo and Zou [2019] introduced tools that establish general relationships between mutual information and error.
Revised: In a similar vein, Russo and Zou [2019] introduced tools that establish general relationships between mutual information and error.

Original: Using these results, Xu and Raginsky [2017] established upper bounds on the generalization error of learning algorithms with countably infinite hypothesis spaces.
Revised: Using these results, Xu and Raginsky [2017] established upper bounds on the generalization error of learning algorithms with countably infinite hypothesis spaces.

Original: We extend these results in several directions to enable analysis of data generating processes related to deep learning.
Revised: We extend these results in several directions to enable analysis of data generating processes related to deep learning.

Original: For example, the results of Haussler et al.
Revised: For example, the results of Haussler et al.

Original: [1994] do not address noisy observations, and all three aforementioned papers do not accommodate continuous parameter spaces, let alone nonparametric data generating processes.
Revised: [1994] do not address noisy observations, and all three aforementioned papers do not accommodate continuous parameter spaces, let alone nonparametric data generating processes.

Original: A distinction of our work is that it builds on rate-distortion theory to address these limitations.
Revised: A distinction of our work is that it builds on rate-distortion theory to address these limitations.

Original: While
Revised: While

Original: Nokleby et al.
Revised: Nokleby et al.

Original: [2016] also use rate-distortion theory to study bayes risk, these results are again limited to parametric classification and only offer lower bounds.
Revised: [2016] also use rate-distortion theory to study bayes risk, these results are again limited to parametric classification and only offer lower bounds.

Original: The rate-distortion function that we study is equivalent to one defined the information bottleneck Tishby et al.
Revised: The rate-distortion function that we study is equivalent to one defined the information bottleneck Tishby et al.

Original: However, instead of using it as a basis for optimization methods as do Shwartz-Ziv and Tishby [2017], we develop tools to study sample complexity and arrive at concrete and novel results.
Revised: However, instead of using it as a basis for optimization methods as do Shwartz-Ziv and Tishby [2017], we develop tools to study sample complexity and arrive at concrete and novel results.

Original: In this paper, we consider contexts in which an agent learns from an iid sequence of data pairs.
Revised: In this paper, we consider contexts in which an agent learns from an iid sequence of data pairs.

Original: We establish tight upper and lower bounds for the average regret and sample complexity that depend on the rate-distortion function.
Revised: We establish tight upper and lower bounds for the average regret and sample complexity that depend on the rate-distortion function.

Original: With these information-theoretic tools, we analyze two deep neural network data generating processes and quantify the number of samples required to arrive at a useful model.
Revised: With these information-theoretic tools, we analyze two deep neural network data generating processes and quantify the number of samples required to arrive at a useful model.

Original: We establish novel sample complexity bounds for ReLU neural network data generating processes that are roughly linear in the parameter count (as opposed to linear in the product of parameter count and depth as in Harvey et al.
Revised: We establish novel sample complexity bounds for ReLU neural network data generating processes that are roughly linear in the parameter count (as opposed to linear in the product of parameter count and depth as in Harvey et al.

Original: For a multilayer process with arbitrary width but bounded sum of weights, we establish sample complexity bounds with only a quadratic depth dependence as opposed to exponential or high-order polynomial dependence as in
Revised: For a multilayer process with arbitrary width but bounded sum of weights, we establish sample complexity bounds with only a quadratic depth dependence as opposed to exponential or high-order polynomial dependence as in

Original: Bartlett et al.
Revised: Bartlett et al.

Original: [2017] and Wei and Ma [2019] respectively.
Revised: [2017] and Wei and Ma [2019] respectively.

Original: The main contribution of this paper is an elegant and intuitive information-theoretic framework for analyzing machine learning problems .
Revised: The main contribution of this paper is an elegant and intuitive information-theoretic framework for analyzing machine learning problems .

Original: We demonstrate its promise by deriving the two aforementioned theoretical results pertaining to deep neural networks which were not possible to show with existing analysis methods.
Revised: We demonstrate its promise by deriving the two aforementioned theoretical results pertaining to deep neural networks which were not possible to show with existing analysis methods.

Original:  Prediction and Error
Revised:  Prediction and Error

Original: We begin by introducing the structure of data generating processes we consider in our framework and defining our notions of error and sample complexity.
Revised: We begin by introducing the structure of data generating processes we consider in our framework and defining our notions of error and sample complexity.

Original: 2.1 Data Generating Process
Revised: 2.1 Data Generating Process

Original: We consider a stochastic process that generates a sequence (( X t , Y t +1 ) :
Revised: We consider a stochastic process that generates a sequence (( X t , Y t +1 ) :

Original: t = 0 , . .
Revised: t = 0 , . .

Original: , T − 1) of data pairs.
Revised: , T − 1) of data pairs.

Original: We refer to each X t as an input and each Y t +1 as an output .
Revised: We refer to each X t as an input and each Y t +1 as an output .

Original: We define these and all other random variables we will consider with respect to a probability space (Ω , F , P ) .
Revised: We define these and all other random variables we will consider with respect to a probability space (Ω , F , P ) .

Original: Elements of the sequence ( X t : t = 0 , . .
Revised: Elements of the sequence ( X t : t = 0 , . .

Original: , T − 1) are independent and identically distributed.
Revised: , T − 1) are independent and identically distributed.

Original: Denote the history of data generated through time t by H t =
Revised: Denote the history of data generated through time t by H t =

Original: ( X 0 , Y 1 , . .
Revised: ( X 0 , Y 1 , . .

Original: , X t − 1 , Y t , X t ) .
Revised: , X t − 1 , Y t , X t ) .

Original: We assume that each ( X t , Y t +1 ) pair is exchangeable.
Revised: We assume that each ( X t , Y t +1 ) pair is exchangeable.

Original: As a result of de Finetti’s theorem, there exists a latent variable E for which the pairs ( X t , Y t +1 ) are iid conditioned on E .
Revised: As a result of de Finetti’s theorem, there exists a latent variable E for which the pairs ( X t , Y t +1 ) are iid conditioned on E .

Original: Here, E (the environment) is a random function that specifies a conditional output distribution P ( Y t +1 ∈ ·|E , X t = x ) =
Revised: Here, E (the environment) is a random function that specifies a conditional output distribution P ( Y t +1 ∈ ·|E , X t = x ) =

Original: E ( ·| x ) for each input x .
Revised: E ( ·| x ) for each input x .

Original: Initial uncertainty about E is expressed by the prior distribution P ( E ∈ · ) .
Revised: Initial uncertainty about E is expressed by the prior distribution P ( E ∈ · ) .

Original: 2.2 Prediction
Revised: 2.2 Prediction

Original: We consider an agent that predict the next response Y t
Revised: We consider an agent that predict the next response Y t

Original: +1 given the history H t .
Revised: +1 given the history H t .

Original: Rather than a point estimate, the agent provides as its prediction a probability distribution P t over possible responses.
Revised: Rather than a point estimate, the agent provides as its prediction a probability distribution P t over possible responses.

Original: We characterize the agent in terms of a function π for which P t =
Revised: We characterize the agent in terms of a function π for which P t =

Original: π ( H t ) .
Revised: π ( H t ) .

Original: We now introduce some notation for referring to particular predictions.
Revised: We now introduce some notation for referring to particular predictions.

Original: We will generally use P t as a dummy variable – that is a generic prediction whose definition depends on context.
Revised: We will generally use P t as a dummy variable – that is a generic prediction whose definition depends on context.

Original: We denote the prediction conditioned on the environment, which could only be produced by a prescient agent, by
Revised: We denote the prediction conditioned on the environment, which could only be produced by a prescient agent, by

Original: [Equation]
Revised: [Equation]

Original: We will refer to this as the target distribution as it represents what an agent aims to learn.
Revised: We will refer to this as the target distribution as it represents what an agent aims to learn.

Original: Finally, we use ˆ P t to denote the posterior-predictive conditioned on H t , which will turn out to be optimal for the objective we will define next.
Revised: Finally, we use ˆ P t to denote the posterior-predictive conditioned on H t , which will turn out to be optimal for the objective we will define next.

Original: [Equation]
Revised: [Equation]

Original: .3 Error
Revised: .3 Error

Original: We assess the error of a prediction P t in terms of the KL-divergence relative to P ∗ t :
Revised: We assess the error of a prediction P t in terms of the KL-divergence relative to P ∗ t :

Original: [Equation]
Revised: [Equation]

Original: This quantifies mismatch between the prediction P t and target distribution P ∗ t .
Revised: This quantifies mismatch between the prediction P t and target distribution P ∗ t .

Original: This notion of error is closely related to more common error notions from machine learning.
Revised: This notion of error is closely related to more common error notions from machine learning.

Original: When Y takes values in a countable set, this notion of error is equivalent to expected cross-entropy loss.
Revised: When Y takes values in a countable set, this notion of error is equivalent to expected cross-entropy loss.

Original: Meanwhile when Y is continuous, this notion of error is upper and lower bounded by linear functions of mean-squared error under reasonable circumstances (refer to Appendix A).
Revised: Meanwhile when Y is continuous, this notion of error is upper and lower bounded by linear functions of mean-squared error under reasonable circumstances (refer to Appendix A).

Original:  Regret and Sample Complexity
Revised:  Regret and Sample Complexity

Original: We assess an agent’s performance over duration T in terms of the expected cumulative error
Revised: We assess an agent’s performance over duration T in terms of the expected cumulative error

Original: [Equation]
Revised: [Equation]

Original: The focus of this paper is on understanding how well an optimal agent can perform, given particular data generating processes.
Revised: The focus of this paper is on understanding how well an optimal agent can perform, given particular data generating processes.

Original: We will use regret to refer to the optimal performance defined below.
Revised: We will use regret to refer to the optimal performance defined below.

Original: Definition 1.
Revised: Definition 1.

Original: (optimal regret) For all T ∈
Revised: (optimal regret) For all T ∈

Original: Z + , environments E and data generating processes (( X 0 , Y 1 ) , ( X 1 , Y 2 ) , . . . )
Revised: Z + , environments E and data generating processes (( X 0 , Y 1 ) , ( X 1 , Y 2 ) , . . . )

Original: , the optimal regret is
Revised: , the optimal regret is

Original: [Equation]
Revised: [Equation]

Original: We will also consider sample complexity, which we take to be the duration required to attain expected average error within some fraction ϵ ≥ 0 of an optimal uninformed prediction:
Revised: We will also consider sample complexity, which we take to be the duration required to attain expected average error within some fraction ϵ ≥ 0 of an optimal uninformed prediction:

Original: Definition 2.
Revised: Definition 2.

Original: (sample complexity)
Revised: (sample complexity)

Original: For all ϵ ≥ 0 , environments E and data generating processes (( X 0 , Y 1 ) , ( X 1 , Y 2 ) , . . . )
Revised: For all ϵ ≥ 0 , environments E and data generating processes (( X 0 , Y 1 ) , ( X 1 , Y 2 ) , . . . )

Original: , the sample complexity is
Revised: , the sample complexity is

Original: T ϵ := min (cid:26)
Revised: T ϵ := min (cid:26)

Original: T : R ( T )
Revised: T : R ( T )

Original: T ≤ ϵ (cid:27) .
Revised: T ≤ ϵ (cid:27) .

Original: Note that this definition of sample complexity measures the number of samples necessary to achieve average cumulative error below ϵ .
Revised: Note that this definition of sample complexity measures the number of samples necessary to achieve average cumulative error below ϵ .

Original: Traditionally, generalization error quantifies the number of samples necessary to achieve out-of-sample error (such as I ( Y T +1 ; E| H T ) ) below ϵ .
Revised: Traditionally, generalization error quantifies the number of samples necessary to achieve out-of-sample error (such as I ( Y T +1 ; E| H T ) ) below ϵ .

Original: Due to a simple monotonicity result, our notion of sample complexity upper bounds the one for out-of-sample error i.e I ( Y T +1 ; E| H T ) ≤ R ( T ) T .
Revised: Due to a simple monotonicity result, our notion of sample complexity upper bounds the one for out-of-sample error i.e I ( Y T +1 ; E| H T ) ≤ R ( T ) T .

Original: 3.1 Optimal Predictions
Revised: 3.1 Optimal Predictions

Original: In this paper, we focus on how well an optimal agent performs, rather than on how to design practical agents that economize on memory and computation.
Revised: In this paper, we focus on how well an optimal agent performs, rather than on how to design practical agents that economize on memory and computation.

Original: Recall that an agent is characterized by a function π , which generates predictions P t
Revised: Recall that an agent is characterized by a function π , which generates predictions P t

Original: =
Revised: =

Original: π
Revised: π

Original: ( H t , Z t ) , where Z t represents algorithmic randomness.
Revised: ( H t , Z t ) , where Z t represents algorithmic randomness.

Original: The following result establishes that the conditional distribution ˆ P t =
Revised: The following result establishes that the conditional distribution ˆ P t =

Original: P ( Y t +1 ∈ ·| H t ) offers an optimal prediction.
Revised: P ( Y t +1 ∈ ·| H t ) offers an optimal prediction.

Original: Theorem 3.
Revised: Theorem 3.

Original: (optimal prediction)
Revised: (optimal prediction)

Original: For all t ≥ 0 ,
Revised: For all t ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: where P t = π ( H t , Z t ) .
Revised: where P t = π ( H t , Z t ) .

Original: A proof may be found in Appendix B. A corollary that characterizes the performance shortfall of a suboptimal agent may also be found there.
Revised: A proof may be found in Appendix B. A corollary that characterizes the performance shortfall of a suboptimal agent may also be found there.

Original: This corrolary opens future avenues regarding learning under misspecification or compute constraints.
Revised: This corrolary opens future avenues regarding learning under misspecification or compute constraints.

Original: In the remainder of the paper we will study an agent that generates optimal predictions P t =
Revised: In the remainder of the paper we will study an agent that generates optimal predictions P t =

Original: P ( Y t +1 ∈ ·| H t ) , as illustrated in Figure 1.
Revised: P ( Y t +1 ∈ ·| H t ) , as illustrated in Figure 1.

Original: We consider an agent that, given a history H t , generates an optimal prediction ˆ P t .
Revised: We consider an agent that, given a history H t , generates an optimal prediction ˆ P t .

Original:  Information
Revised:  Information

Original: We define concepts for quantifying uncertainty and the information gained from observations.
Revised: We define concepts for quantifying uncertainty and the information gained from observations.

Original: The entropy H ( E ) of the environment quantifies the agent’s initial degree of uncertainty in terms of the information required to identify E .
Revised: The entropy H ( E ) of the environment quantifies the agent’s initial degree of uncertainty in terms of the information required to identify E .

Original: We will measure information in nats , each of which is equivalent to 1 / ln 2 bits.
Revised: We will measure information in nats , each of which is equivalent to 1 / ln 2 bits.

Original: For example, if E occupies a countable range Θ then H ( E ) =
Revised: For example, if E occupies a countable range Θ then H ( E ) =

Original: − (cid:80) θ ∈ Θ P ( E = θ ) ln P ( E = θ ) .
Revised: − (cid:80) θ ∈ Θ P ( E = θ ) ln P ( E = θ ) .

Original: Uncertainty at time t can be expressed in terms of the conditional entropy H ( E| H t ) , which is the number of remaining nats after observing H t .
Revised: Uncertainty at time t can be expressed in terms of the conditional entropy H ( E| H t ) , which is the number of remaining nats after observing H t .

Original: The mutual information I ( E ; H t ) =
Revised: The mutual information I ( E ; H t ) =

Original: H ( E ) − H
Revised: H ( E ) − H

Original: ( E| H t ) quantifies the information about E gained from H t .
Revised: ( E| H t ) quantifies the information about E gained from H t .

Original: Proofs for all lemmas in this section may be found in Appendix B
Revised: Proofs for all lemmas in this section may be found in Appendix B

Original: 4.1 Learning from Errors
Revised: 4.1 Learning from Errors

Original: Each data pair ( X t , Y t +1 ) provides I ( E ; ( X t , Y t +1 )
Revised: Each data pair ( X t , Y t +1 ) provides I ( E ; ( X t , Y t +1 )

Original: | H t − 1 , Y t ) nats of new information about the environment.
Revised: | H t − 1 , Y t ) nats of new information about the environment.

Original: By the chain rule of mutual information, this is the sum
Revised: By the chain rule of mutual information, this is the sum

Original: I ( E ; ( X t , Y t +1 ) | H t − 1 , Y t ) =
Revised: I ( E ; ( X t , Y t +1 ) | H t − 1 , Y t ) =

Original: I ( E ; X t | H t − 1 , Y t )
Revised: I ( E ; X t | H t − 1 , Y t )

Original: + I ( E ; Y t +1 | H t ) of the information gained from X t and Y t +1 .
Revised: + I ( E ; Y t +1 | H t ) of the information gained from X t and Y t +1 .

Original: The former term I ( E ; X t | H t − 1 , Y t ) is equal tobecause X t is independent from both E and ( H t − 1 , Y t ) .
Revised: The former term I ( E ; X t | H t − 1 , Y t ) is equal tobecause X t is independent from both E and ( H t − 1 , Y t ) .

Original: The latter term I ( E ; Y t +1 | H t ) can be thought of as the level of surprise experienced by the agent upon observing Y t +1 .
Revised: The latter term I ( E ; Y t +1 | H t ) can be thought of as the level of surprise experienced by the agent upon observing Y t +1 .

Original: Surprise is associated with prediction error, and the following result formalizes the equivalence between error and information.
Revised: Surprise is associated with prediction error, and the following result formalizes the equivalence between error and information.

Original: Lemma 4.
Revised: Lemma 4.

Original: (expected prediction error equals information gain)
Revised: (expected prediction error equals information gain)

Original: For all t ∈ Z + ,
Revised: For all t ∈ Z + ,

Original: [Equation]
Revised: [Equation]

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: and R ( t )
Revised: and R ( t )

Original: =
Revised: =

Original: I ( E ; H t ) .
Revised: I ( E ; H t ) .

Original: The agent’s ability to predict tends to improve as it learns from experience.
Revised: The agent’s ability to predict tends to improve as it learns from experience.

Original: This is formalized by the following result, which establishes that expected prediction errors are monotonically nonincreasing.
Revised: This is formalized by the following result, which establishes that expected prediction errors are monotonically nonincreasing.

Original: Lemma 5.
Revised: Lemma 5.

Original: (expected prediction error is monotonically nonincreasing)
Revised: (expected prediction error is monotonically nonincreasing)

Original: For all t ∈ Z + ,
Revised: For all t ∈ Z + ,

Original: [Equation]
Revised: [Equation]

Original:  General Bounds via Rate-Distortion Theory
Revised:  General Bounds via Rate-Distortion Theory

Original: An environment proxy is a random variable ˜ E that provides information about the environment E but no additional information pertaining to inputs or outputs.
Revised: An environment proxy is a random variable ˜ E that provides information about the environment E but no additional information pertaining to inputs or outputs.

Original: In other words, ˜ E ⊥ ( X, Y ) |E .
Revised: In other words, ˜ E ⊥ ( X, Y ) |E .

Original: We will denote the set of environment proxies by ˜Θ .
Revised: We will denote the set of environment proxies by ˜Θ .

Original: While an infinite amount of information must be acquired to identify the environment when H ( E ) = ∞ , there can be a proxy ˜ E with H ( ˜ E ) < ∞ that enables accurate predictions.
Revised: While an infinite amount of information must be acquired to identify the environment when H ( E ) = ∞ , there can be a proxy ˜ E with H ( ˜ E ) < ∞ that enables accurate predictions.

Original: The minimal expected error attainable based on the proxy is achieved by a prediction ˜ P
Revised: The minimal expected error attainable based on the proxy is achieved by a prediction ˜ P

Original: = P ( Y ∈ ·| ˜ E , X t ) .
Revised: = P ( Y ∈ ·| ˜ E , X t ) .

Original: This results in expected error E
Revised: This results in expected error E

Original: [ d KL ( P ∗ ∥ ˜ P )] .
Revised: [ d KL ( P ∗ ∥ ˜ P )] .

Original: We will establish that the expected error E [ d KL ( P ∗ ∥ ˜ P )] equals the information gained, beyond that supplied by the proxy ˜ E , about the environment E from observing Y .
Revised: We will establish that the expected error E [ d KL ( P ∗ ∥ ˜ P )] equals the information gained, beyond that supplied by the proxy ˜ E , about the environment E from observing Y .

Original: This is intuitive: more is learned from Y if knowledge of E enables a better prediction of Y than does ˜ E .
Revised: This is intuitive: more is learned from Y if knowledge of E enables a better prediction of Y than does ˜ E .

Original: We quantify this information gain in terms of the difference H ( Y | ˜ E , X ) − H
Revised: We quantify this information gain in terms of the difference H ( Y | ˜ E , X ) − H

Original: ( Y |E , X ) between the uncertainty conditioned on ˜ E and that conditioned on E .
Revised: ( Y |E , X ) between the uncertainty conditioned on ˜ E and that conditioned on E .

Original: This is equal to the mutual information I ( E ; Y | ˜ E , X ) =
Revised: This is equal to the mutual information I ( E ; Y | ˜ E , X ) =

Original: H ( Y | ˜ E , X ) − H
Revised: H ( Y | ˜ E , X ) − H

Original: ( Y |E , X ) .
Revised: ( Y |E , X ) .

Original: The following result equates this with expected error.
Revised: The following result equates this with expected error.

Original: Lemma 6.
Revised: Lemma 6.

Original: (proxy error equals information gain)
Revised: (proxy error equals information gain)

Original: For all ˜ E ∈ ˜Θ ,
Revised: For all ˜ E ∈ ˜Θ ,

Original: [Equation]
Revised: [Equation]

Original: A proof may be found in Appendix C. E [ d KL ( P ∗ ∥ ˜ P )] is the distortion incurred by our estimate of
Revised: A proof may be found in Appendix C. E [ d KL ( P ∗ ∥ ˜ P )] is the distortion incurred by our estimate of

Original: Y given only ˜ E as opposed to E .
Revised: Y given only ˜ E as opposed to E .

Original: For example, ˜ E may be a quantization or lossy compression of E an E [ d KL ( P ∗ ∥ ˜ P )] is measuring how inaccurate our prediction of Y is under this compression ˜ E .
Revised: For example, ˜ E may be a quantization or lossy compression of E an E [ d KL ( P ∗ ∥ ˜ P )] is measuring how inaccurate our prediction of Y is under this compression ˜ E .

Original: Now we consider the following ϵ -optimal set:
Revised: Now we consider the following ϵ -optimal set:

Original: [Equation]
Revised: [Equation]

Original: ˜Θ ϵ denotes the set of proxies that produce predictions that incur a distortion of no more than an ϵ .
Revised: ˜Θ ϵ denotes the set of proxies that produce predictions that incur a distortion of no more than an ϵ .

Original: One can imagine that for environments with simple input distributions, such as those constrained to a subspace or low-dimensional manifold, this set ˜Θ ϵ will be much larger since the average error need only be below ϵ in a small subset of the domain.
Revised: One can imagine that for environments with simple input distributions, such as those constrained to a subspace or low-dimensional manifold, this set ˜Θ ϵ will be much larger since the average error need only be below ϵ in a small subset of the domain.

Original: With the distortion component of rate-distortion covered, the rate remains.
Revised: With the distortion component of rate-distortion covered, the rate remains.

Original: The rate is I ( E ; ˜ E ) , the amount of information about the environment conveyed by the proxy.
Revised: The rate is I ( E ; ˜ E ) , the amount of information about the environment conveyed by the proxy.

Original: For example, a finer quantization would result in a higher rate since ˜ E would capture E with more bits of precision.
Revised: For example, a finer quantization would result in a higher rate since ˜ E would capture E with more bits of precision.

Original: However, with this higher rate, the distortion incurred by ˜ E should in turn be lower .
Revised: However, with this higher rate, the distortion incurred by ˜ E should in turn be lower .

Original: A higher fidelity compression should cost more bits but result in less distortion.
Revised: A higher fidelity compression should cost more bits but result in less distortion.

Original: The rate-distortion function formalizes this trade-off concretely:
Revised: The rate-distortion function formalizes this trade-off concretely:

Original: Definition 7.
Revised: Definition 7.

Original: For all ϵ ≥ 0 and environments E , The rate-distortion function w.r.t distortion function E [ d KL ( P ∗ ∥ ˜ P )] is
Revised: For all ϵ ≥ 0 and environments E , The rate-distortion function w.r.t distortion function E [ d KL ( P ∗ ∥ ˜ P )] is

Original: [Equation]
Revised: [Equation]

Original: H ϵ characterizes the minimal amount of information that a proxy must convey about the environment in order to make ϵ -accurate predictions.
Revised: H ϵ characterizes the minimal amount of information that a proxy must convey about the environment in order to make ϵ -accurate predictions.

Original: Even when H ( E ) is infinite and ϵ is small, H ϵ ( E ) can be manageable.
Revised: Even when H ( E ) is infinite and ϵ is small, H ϵ ( E ) can be manageable.

Original: Once again, if we consider an environment with a simple input distribution, the ratedistortion would be lower since we would be taking an infimum over a larger set ˜Θ ϵ .
Revised: Once again, if we consider an environment with a simple input distribution, the ratedistortion would be lower since we would be taking an infimum over a larger set ˜Θ ϵ .

Original: Depending on the structure of the input distribution and the environment, one could imagine that the reduction could be significant.
Revised: Depending on the structure of the input distribution and the environment, one could imagine that the reduction could be significant.

Original: As we will see in the following section, both the regret and sample complexity of learning scales with H ϵ ( E ) .
Revised: As we will see in the following section, both the regret and sample complexity of learning scales with H ϵ ( E ) .

Original: 5.1 Bounding Regret and Sample Complexity via Rate-Distortion
Revised: 5.1 Bounding Regret and Sample Complexity via Rate-Distortion

Original: With rate-distortion in place, we produce tight bounds on regret and sample complexity.
Revised: With rate-distortion in place, we produce tight bounds on regret and sample complexity.

Original: These bounds are very general, applying to any data generating process.
Revised: These bounds are very general, applying to any data generating process.

Original: The results upper and lower bound error and sample complexity in terms of rate-distortion.
Revised: The results upper and lower bound error and sample complexity in terms of rate-distortion.

Original: As such, for any particular data generating process, bounds can be produced by characterizing the associated rate-distortion function.
Revised: As such, for any particular data generating process, bounds can be produced by characterizing the associated rate-distortion function.

Original: Proofs for the following two theorems can be found in Appendix C.
Revised: Proofs for the following two theorems can be found in Appendix C.

Original: The following result brackets the cumulative error of optimal predictions.
Revised: The following result brackets the cumulative error of optimal predictions.

Original: Theorem 8.
Revised: Theorem 8.

Original: (rate-distortion regret bounds)
Revised: (rate-distortion regret bounds)

Original: For all T ,
Revised: For all T ,

Original: [Equation]
Revised: [Equation]

Original: This upper bound is intuitive.
Revised: This upper bound is intuitive.

Original: Knowledge of a proxy ˜ E ∈ ˜Θ
Revised: Knowledge of a proxy ˜ E ∈ ˜Θ

Original: ϵ enables an agent to limit prediction error to ϵ per timestep.
Revised: ϵ enables an agent to limit prediction error to ϵ per timestep.

Original: Getting to that level of prediction error requires H ϵ ( E ) nats, and therefore, that much cumulative error.
Revised: Getting to that level of prediction error requires H ϵ ( E ) nats, and therefore, that much cumulative error.

Original: Hence, R ( T ) ≤ H ϵ ( E ) + ϵT .
Revised: Hence, R ( T ) ≤ H ϵ ( E ) + ϵT .

Original: To motivate the lower bound, note that an agent requires H ϵ ( E ) nats to attain per timestep error within ϵ .
Revised: To motivate the lower bound, note that an agent requires H ϵ ( E ) nats to attain per timestep error within ϵ .

Original: Obtaining those nats requires cumulative error at least H ϵ ( E ) .
Revised: Obtaining those nats requires cumulative error at least H ϵ ( E ) .

Original: So prior to obtaining that many nats, the agent must incur at least ϵ error per timestep, hence the ϵT term in the minimum.
Revised: So prior to obtaining that many nats, the agent must incur at least ϵ error per timestep, hence the ϵT term in the minimum.

Original: Meanwhile, if at time T , the agent is able to produce predictions with error less than ϵ it means that it has already accumulated at least H ϵ ( E ) nats of information about E (error).
Revised: Meanwhile, if at time T , the agent is able to produce predictions with error less than ϵ it means that it has already accumulated at least H ϵ ( E ) nats of information about E (error).

Original: Sample complexity bounds follow almost immediately from Theorem 8.
Revised: Sample complexity bounds follow almost immediately from Theorem 8.

Original: Theorem 9.
Revised: Theorem 9.

Original: (rate-distortion sample complexity bounds)
Revised: (rate-distortion sample complexity bounds)

Original: For all ϵ ≥ 0 ,
Revised: For all ϵ ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original:  Deep Neural Network Environments
Revised:  Deep Neural Network Environments

Original: We now study the rate-distortion function of multilayer environments.
Revised: We now study the rate-distortion function of multilayer environments.

Original: As we established, sample complexity is governed by the rate-distortion function H ϵ ( E ) .
Revised: As we established, sample complexity is governed by the rate-distortion function H ϵ ( E ) .

Original: In this section, we will focus on characterizing this rate-distortion function, and hence the sample complexity, of two prototypical multilayer environments.
Revised: In this section, we will focus on characterizing this rate-distortion function, and hence the sample complexity, of two prototypical multilayer environments.

Original: 6.1 Prototypical Multilayer Environments 6.1.1 Prototypical Environment 1: (ReLU Neural Networks)
Revised: 6.1 Prototypical Multilayer Environments 6.1.1 Prototypical Environment 1: (ReLU Neural Networks)

Original: Prototypical Environment 1 mirrors the architecture of fully-connected feed-forward neural networks (refer to Figure 2 left).
Revised: Prototypical Environment 1 mirrors the architecture of fully-connected feed-forward neural networks (refer to Figure 2 left).

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: where A (1) ∈ ℜ N × d , b (1) ∈
Revised: where A (1) ∈ ℜ N × d , b (1) ∈

Original: ℜ N both with independent elements each with variance 1 d .
Revised: ℜ N both with independent elements each with variance 1 d .

Original: For k ∈ { 2 , . . .
Revised: For k ∈ { 2 , . . .

Original: , K − 1 } , let
Revised: , K − 1 } , let

Original: [Equation]
Revised: [Equation]

Original: where A ( k ) ∈
Revised: where A ( k ) ∈

Original: ℜ N × N , b ( k ) ∈ ℜ N , both with independent elements each with variance 1 N .
Revised: ℜ N × N , b ( k ) ∈ ℜ N , both with independent elements each with variance 1 N .

Original: For the final layer, let
Revised: For the final layer, let

Original: [Equation]
Revised: [Equation]

Original: where A ( K ) ∈
Revised: where A ( K ) ∈

Original: ℜ N with independent elements, each with variance 1 N .
Revised: ℜ N with independent elements, each with variance 1 N .

Original: In this environment, E k is identified by ( A ( k ) , b ( k ) ) for k ∈ { 1 , . . .
Revised: In this environment, E k is identified by ( A ( k ) , b ( k ) ) for k ∈ { 1 , . . .

Original: , K − 1 } and E K is identified by A ( K ) .
Revised: , K − 1 } and E K is identified by A ( K ) .

Original: We will refer to this environment as the deep ReLU network .
Revised: We will refer to this environment as the deep ReLU network .

Original: 6.1.2 Prototypical Environment 2: (Deep Nonparametric Networks)
Revised: 6.1.2 Prototypical Environment 2: (Deep Nonparametric Networks)

Original: Prototypical Environment 2 considers a deep nonparametric neural network (refer to Figure 2 right).
Revised: Prototypical Environment 2 considers a deep nonparametric neural network (refer to Figure 2 right).

Original: For k ∈ { 1 , . . .
Revised: For k ∈ { 1 , . . .

Original: , K } , let
Revised: , K } , let

Original: [Equation]
Revised: [Equation]

Original: where c ∈ Z ++ , α ( k ) =
Revised: where c ∈ Z ++ , α ( k ) =

Original: ( α ( k ) 1 , . . .
Revised: ( α ( k ) 1 , . . .

Original: , α ( k ) N ) ∼ Dir (cid:0) N, (cid:2) cN , . . .
Revised: , α ( k ) N ) ∼ Dir (cid:0) N, (cid:2) cN , . . .

Original: , cN (cid:3)(cid:1) , and δ ( k ) n iid ∼ Rademacher .
Revised: , cN (cid:3)(cid:1) , and δ ( k ) n iid ∼ Rademacher .

Original: For k ∈ { 2 , . . .
Revised: For k ∈ { 2 , . . .

Original: , K − 1 } , the deterministic basis functions g ( k ) n are maps from ℜ M (cid:55)→ ℜ M .
Revised: , K − 1 } , the deterministic basis functions g ( k ) n are maps from ℜ M (cid:55)→ ℜ M .

Original: ( g (1) n : n ∈ { 1 , . . .
Revised: ( g (1) n : n ∈ { 1 , . . .

Original: , N } ) are maps from ℜ d (cid:55)→ ℜ M and ( g ( K ) n : n ∈ { 1 , . . .
Revised: , N } ) are maps from ℜ d (cid:55)→ ℜ M and ( g ( K ) n : n ∈ { 1 , . . .

Original: , N } ) are maps from ℜ M (cid:55)→ ℜ .
Revised: , N } ) are maps from ℜ M (cid:55)→ ℜ .

Original: For regularity, we will assume that for all n and k , the basis functions satisfy E [ g ( k ) n ( U k − 1 ) 2 ] ≤ 1 .
Revised: For regularity, we will assume that for all n and k , the basis functions satisfy E [ g ( k ) n ( U k − 1 ) 2 ] ≤ 1 .

Original: E k is identified by δ ( k ) ⊙ α ( k ) .
Revised: E k is identified by δ ( k ) ⊙ α ( k ) .

Original: We will refer to this environment ( E K :1 ) as the deep nonparametric network .
Revised: We will refer to this environment ( E K :1 ) as the deep nonparametric network .

Original: For intractably large N , f k is effectively nonparametric.
Revised: For intractably large N , f k is effectively nonparametric.

Original: In this regime, parameter count becomes a vacuous description of the data generating process’s complexity.
Revised: In this regime, parameter count becomes a vacuous description of the data generating process’s complexity.

Original: However, the complexity is still controlled by the fact that (cid:80) Nn =1 α n = 1 due to the Dirichlet prior.
Revised: However, the complexity is still controlled by the fact that (cid:80) Nn =1 α n = 1 due to the Dirichlet prior.

Original: We will establish rate-distortion and sample complexity bounds that depend only logarithmically on N and linearly in c .
Revised: We will establish rate-distortion and sample complexity bounds that depend only logarithmically on N and linearly in c .

Original: A natural example of such an environment is one in which the basis functions are g ( k ) n ( X ) =
Revised: A natural example of such an environment is one in which the basis functions are g ( k ) n ( X ) =

Original: ReLU( θ ⊤ n,k X ) where ∥ θ ∥ 2 = 

Revised: ReLU( θ ⊤ n,k X ) where ∥ θ ∥ 2 =

Original: and N may be prohibitively large i.e. exponential in the input dimension.
Revised: and N may be prohibitively large i.e. exponential in the input dimension.

Original: 6.2 From Single to Multilayer Environments
Revised: 6.2 From Single to Multilayer Environments

Original: We are now interested in analyzing the depth dependence of the rate-distortion and sample complexity of these environments.
Revised: We are now interested in analyzing the depth dependence of the rate-distortion and sample complexity of these environments.

Original: We will use E i : j and ˜ E i : j for i ≥ j to denote ( E i , E i − 1 , . . .
Revised: We will use E i : j and ˜ E i : j for i ≥ j to denote ( E i , E i − 1 , . . .

Original: , E j ) and ( ˜ E i , ˜ E
Revised: , E j ) and ( ˜ E i , ˜ E

Original: i − 1 , . . . , ˜ E j ) respectively.
Revised: i − 1 , . . . , ˜ E j ) respectively.

Original: For multilayer environments, the error is cumbersome to reason about.
Revised: For multilayer environments, the error is cumbersome to reason about.

Original: Figure 3 (left) depicts the error incurred by using a multilayer proxy E K :1 to approximate multilayer environment E K :1 .
Revised: Figure 3 (left) depicts the error incurred by using a multilayer proxy E K :1 to approximate multilayer environment E K :1 .

Original: Evidently, it seems difficult to manage the error propagation through the layers of the environment.
Revised: Evidently, it seems difficult to manage the error propagation through the layers of the environment.

Original: Many traditional lines of analysis struggle on this front and result in sample complexity bounds that are exponential in the depth Bartlett et al.
Revised: Many traditional lines of analysis struggle on this front and result in sample complexity bounds that are exponential in the depth Bartlett et al.

Original: [2017], Golowich et al.
Revised: [2017], Golowich et al.

Original: These papers consider a worst-case reasoning under which an ϵ error between the first outputs U 1 and ˜ U 1 may blow up to a λ K ϵ error when passed through remaining layers of the network (where λ is a spectral radius).
Revised: These papers consider a worst-case reasoning under which an ϵ error between the first outputs U 1 and ˜ U 1 may blow up to a λ K ϵ error when passed through remaining layers of the network (where λ is a spectral radius).

Original: It would be much simpler to instead analyzethe incremental errorincurred at stage network.
Revised: It would be much simpler to instead analyze incremental error network.

Original: Figure 3 (right) depicts this.
Revised: Figure 3 (right) depicts this.

Original: We consider that at each layer, we know the true input U k −and simply measure the immediate error incurred at the output U k as opposed to the error incurred at the final output of the network Y .
Revised: We consider that at each layer, we know the true input U k −and simply measure the immediate error incurred at the output U k as opposed to the error incurred at the final output of the network Y .

Original: Mathematically, the error incurred by the full system from using proxy ˜
Revised: Mathematically, the error incurred by the full system from using proxy ˜

Original: E K :1 can be expressed as I
Revised: E K :1 can be expressed as I

Original: [Equation]
Revised: [Equation]

Original: E K
Revised: E K

Original: [Equation]
Revised: [Equation]

Original: 1 | ˜
Revised: 1 | ˜

Original: E K :1 , X ) .
Revised: E K :1 , X ) .

Original: By the chain rule, we have that
Revised: By the chain rule, we have that

Original: [Equation]
Revised: [Equation]

Original: Therefore, the error incurred from layer k can be expressed as I ( Y ; E k |E K : k +1 , ˜
Revised: Therefore, the error incurred from layer k can be expressed as I ( Y ; E k |E K : k +1 , ˜

Original: E k :1 , X ) .
Revised: E k :1 , X ) .

Original: This is cumbersome since we are not given the true input
Revised: This is cumbersome since we are not given the true input

Original: U k − 1 but rather an approximation from input
Revised: U k − 1 but rather an approximation from input

Original: X and ˜ E k − 1:1 .
Revised: X and ˜ E k − 1:1 .

Original: Furthermore, we are measuring the error in the final output Y as opposed to the immediate output
Revised: Furthermore, we are measuring the error in the final output Y as opposed to the immediate output

Original: U k .
Revised: U k .

Original: It would be much more simple to analyze something like the following:
Revised: It would be much more simple to analyze something like the following:

Original: [Equation]
Revised: [Equation]

Original: where W is independent 0 -mean gaussian noise with variance σ 2 in each dimension.
Revised: where W is independent 0 -mean gaussian noise with variance σ 2 in each dimension.

Original: This sum is much easier to work with because the k th term only depends on E k ,
Revised: This sum is much easier to work with because the k th term only depends on E k ,

Original: ˜ E k , U k − 1 , and U k .
Revised: ˜ E k , U k − 1 , and U k .

Original: There is no inter-layer dependence.
Revised: There is no inter-layer dependence.

Original: The key insight is that in our two prototypical environments (and many others), something akin to the following will hold:
Revised: The key insight is that in our two prototypical environments (and many others), something akin to the following will hold:

Original: [Equation]
Revised: [Equation]

Original: As a result, the cumbersome sum 1 will be upper bounded by the nice sum 2.
Revised: As a result, the cumbersome sum 1 will be upper bounded by the nice sum 2.

Original: The condition in inequality 3 involves two parts:
Revised: The condition in inequality 3 involves two parts:

Original: I ( Y ; E k |E K : k +1 , ˜
Revised: I ( Y ; E k |E K : k +1 , ˜

Original: E
Revised: E

Original: k :1 , X ) ≤
Revised: k :1 , X ) ≤

Original: I ( Y ; E k |E K : k +1 , ˜
Revised: I ( Y ; E k |E K : k +1 , ˜

Original: E k , U k − 1 )
Revised: E k , U k − 1 )

Original: • Conditioning on the true input U k − 1 provides more information about E k than conditioning on an approximation ( ˜ E k − 1:1 , X ) .
Revised: • Conditioning on the true input U k − 1 provides more information about E k than conditioning on an approximation ( ˜ E k − 1:1 , X ) .

Original: I ( Y ; E k |E K : k +1 , ˜
Revised: I ( Y ; E k |E K : k +1 , ˜

Original: E k , U k − 1 ) ≤ I ( U k + W ; E k | ˜
Revised: E k , U k − 1 ) ≤ I ( U k + W ; E k | ˜

Original: E k , U k − 1 ) • The immediate output U k
Revised: E k , U k − 1 ) • The immediate output U k

Original: + W provides more information about E k than the final output Y does.
Revised: + W provides more information about E k than the final output Y does.

Original: 1) holds for all proxies of the form ˜ E
Revised: 1) holds for all proxies of the form ˜ E

Original: =
Revised: =

Original: ( ˜ E 1 , . .
Revised: ( ˜ E 1 , . .

Original: ˜ E K ) where ˜ E
Revised: ˜ E K ) where ˜ E

Original: i ⊥ ˜
Revised: i ⊥ ˜

Original: E j for i ̸
Revised: E j for i ̸

Original: =
Revised: =

Original: j .
Revised: j .

Original: We prove this result explicitly in Lemma 19 in Appendix
Revised: We prove this result explicitly in Lemma 19 in Appendix

Original: E.
Revised: E.

Original: It is rather intuitive that the pristine data pair ( U k − 1 , Y ) would provide more information about E k than ( U 0 ,
Revised: It is rather intuitive that the pristine data pair ( U k − 1 , Y ) would provide more information about E k than ( U 0 ,

Original: ˜ E k − 1:1 , Y ) would.
Revised: ˜ E k − 1:1 , Y ) would.

Original: We expect some information to be lost through the noisy reconstruction of U k − 1 from U 0 , ˜
Revised: We expect some information to be lost through the noisy reconstruction of U k − 1 from U 0 , ˜

Original: E k − 1:1 .
Revised: E k − 1:1 .

Original: 2) will not hold exactly for our two environments.
Revised: 2) will not hold exactly for our two environments.

Original: I ( Y ; E k |E K : k +1 , U k − 1 ) will not always be ≤
Revised: I ( Y ; E k |E K : k +1 , U k − 1 ) will not always be ≤

Original: I ( U k + W, E k |E k , U k − 1 ) .
Revised: I ( U k + W, E k |E k , U k − 1 ) .

Original: However, I ( Y ; E k |E K : k +1 , U k − 1 ) will be ≤ a very natural upper bound of I ( U k + W, E k |E k , U k − 1 ) , which we detail in Lemma 20 of Appendix E. Intuitively
Revised: However, I ( Y ; E k |E K : k +1 , U k − 1 ) will be ≤ a very natural upper bound of I ( U k + W, E k |E k , U k − 1 ) , which we detail in Lemma 20 of Appendix E. Intuitively

Original: U k
Revised: U k

Original: + W will provide more information about E k than Y so long as in expectation, the layers f k +1 , . . .
Revised: + W will provide more information about E k than Y so long as in expectation, the layers f k +1 , . . .

Original: , f K don’t amplify the scale of the output.
Revised: , f K don’t amplify the scale of the output.

Original: If they were to amplify the scale, then the signal to noise ratio of Y would look more favorable than that of U k + W , leading to potentially information.
Revised: If they were to amplify the scale, then the signal to noise ratio of Y would look more favorable than that of U k + W , leading to potentially information.

Original: Concretely, if
Revised: Concretely, if

Original: [Equation]
Revised: [Equation]

Original: then we will have the desired result.
Revised: then we will have the desired result.

Original: The expectation here is taken over the randomness in the function f ( k ) .
Revised: The expectation here is taken over the randomness in the function f ( k ) .

Original: For both of our prototypical data generating processes, this condition is met.
Revised: For both of our prototypical data generating processes, this condition is met.

Original: L can be thought of as an expected squared Lipschitz constant of f ( k ) .
Revised: L can be thought of as an expected squared Lipschitz constant of f ( k ) .

Original: The set of random functions for which L ≤ 1 is much broader than say 1-Lipschitz functions because of both the square and the expectation.
Revised: The set of random functions for which L ≤ 1 is much broader than say 1-Lipschitz functions because of both the square and the expectation.

Original: For example, f ( x ) =
Revised: For example, f ( x ) =

Original: Ax is ∥
Revised: Ax is ∥

Original: A ∥ 2 -Lipschitz
Revised: A ∥ 2 -Lipschitz

Original: but if say cov
Revised: but if say cov

Original: [ A ] = I n , then L = 1 .
Revised: [ A ] = I n , then L = 1 .

Original: For multilayer processes for which we can establish a relationship akin to inequality 3, we have the following relationship between the rate-distortion function H ϵ ( E K :1 ) and the individual ratedistortion functions H ϵ ( E k ) (proof in Appendix E).
Revised: For multilayer processes for which we can establish a relationship akin to inequality 3, we have the following relationship between the rate-distortion function H ϵ ( E K :1 ) and the individual ratedistortion functions H ϵ ( E k ) (proof in Appendix E).

Original: Theorem 10.
Revised: Theorem 10.

Original: multilayer rate-distortion bound For all K ∈ Z
Revised: multilayer rate-distortion bound For all K ∈ Z

Original: ++ , σ 2 , ϵ ≥ 0 , if E K :1 is a multilayer environment such that for all k ∈ { 1 , . .
Revised: ++ , σ 2 , ϵ ≥ 0 , if E K :1 is a multilayer environment such that for all k ∈ { 1 , . .

Original: , K } and δ ≥ 0 , there exist ˜ E k s.t.
Revised: , K } and δ ≥ 0 , there exist ˜ E k s.t.

Original: [Equation]
Revised: [Equation]

Original: where W ∼ N (0 , σ 2 I ) , then
Revised: where W ∼ N (0 , σ 2 I ) , then

Original: [Equation]
Revised: [Equation]

Original: 6.3 Sample Complexity Bounds for Multilayer Environments
Revised: 6.3 Sample Complexity Bounds for Multilayer Environments

Original: With Theorem 10 in place, we present the two main results (proofs can be found in Appendix F).
Revised: With Theorem 10 in place, we present the two main results (proofs can be found in Appendix F).

Original: Theorem 11.
Revised: Theorem 11.

Original: (relu neural network rate-distortion and sample complexity bound)
Revised: (relu neural network rate-distortion and sample complexity bound)

Original: For all d, N, K ∈ Z ++ and σ 2 , ϵ ≥ 0 , if multilayer environment E K :1 is the deep ReLU network with input X : Ω (cid:55)→ ℜ d s.t. V
Revised: For all d, N, K ∈ Z ++ and σ 2 , ϵ ≥ 0 , if multilayer environment E K :1 is the deep ReLU network with input X : Ω (cid:55)→ ℜ d s.t. V

Original: [ X ] = I d and output Y ∼ N ( U K , σ 2 ) , then
Revised: [ X ] = I d and output Y ∼ N ( U K , σ 2 ) , then

Original: [Equation]
Revised: [Equation]

Original: While this is a parameter-count based bound in nature, it improves upon existing sample complexity results based on VC Dimension Bartlett et al.
Revised: While this is a parameter-count based bound in nature, it improves upon existing sample complexity results based on VC Dimension Bartlett et al.

Original: [1998],Harvey et al.
Revised: [1998],Harvey et al.

Original: [2017] which under our notation would be ˜ O
Revised: [2017] which under our notation would be ˜ O

Original: ( K 2 N 2 + KdN ) .
Revised: ( K 2 N 2 + KdN ) .

Original: Note however that our result is average case while theirs holds with high probability.
Revised: Note however that our result is average case while theirs holds with high probability.

Original: Regardless, the reduction in sample complexity by a factor of K is notable.
Revised: Regardless, the reduction in sample complexity by a factor of K is notable.

Original: For nonparametric deep networks, we have the following result.
Revised: For nonparametric deep networks, we have the following result.

Original: Theorem 12.
Revised: Theorem 12.

Original: (nonparametric network rate-distortion and sample complexity bound)
Revised: (nonparametric network rate-distortion and sample complexity bound)

Original: For all c, d, N, M, K ∈ Z ++ , σ 2 , ϵ ≥ 0 , if multilayer environment E K :1 is the deep nonparametric network with input X : Ω (cid:55)→ ℜ d and output Y ∼ N ( U K , σ 2 )
Revised: For all c, d, N, M, K ∈ Z ++ , σ 2 , ϵ ≥ 0 , if multilayer environment E K :1 is the deep nonparametric network with input X : Ω (cid:55)→ ℜ d and output Y ∼ N ( U K , σ 2 )

Original: , then
Revised: , then

Original: [Equation]
Revised: [Equation]

Original: The nonparametric network’s sample complexity grows only quadratically in K not exponentially or high-order polynomially as in Bartlett et al.
Revised: The nonparametric network’s sample complexity grows only quadratically in K not exponentially or high-order polynomially as in Bartlett et al.

Original: [2017] and Wei and Ma [2019] respectively.
Revised: [2017] and Wei and Ma [2019] respectively.

Original:  Closing Remarks
Revised:  Closing Remarks

Original: We have introduced a novel and elegant information-theoretic framework for analyzing the sample complexity of data generating processes.
Revised: We have introduced a novel and elegant information-theoretic framework for analyzing the sample complexity of data generating processes.

Original: We demonstrate its usefulness by proving two new theoretical results that suggests that it is possible to learn efficiently from data generated by deep and nonparametric functions.
Revised: We demonstrate its usefulness by proving two new theoretical results that suggests that it is possible to learn efficiently from data generated by deep and nonparametric functions.

Original: Beyond the scope of this paper, we believe that the flexibility and simplicity of our framework will allow for the analysis of machine learning systems such as semi-supervised learning, multitask learning, bandits and reinforcement learning.
Revised: Beyond the scope of this paper, we believe that the flexibility and simplicity of our framework will allow for the analysis of machine learning systems such as semi-supervised learning, multitask learning, bandits and reinforcement learning.

Original: We also believe that many of the nuances of empirical deep learning such as batch-normalization, pooling, and structured input distributions can be analyzed through the average-case nature of information theory and powerful tools such as the data processing inequality.
Revised: We also believe that many of the nuances of empirical deep learning such as batch-normalization, pooling, and structured input distributions can be analyzed through the average-case nature of information theory and powerful tools such as the data processing inequality.

Original: Notably omitted from this paper is any analysis of practical algorithms for estimating the environment.
Revised: Notably omitted from this paper is any analysis of practical algorithms for estimating the environment.

Original: We have assumed perfect Bayesian inference, while in practice, particular neural network architectures are used together with stochastic gradient descent.
Revised: We have assumed perfect Bayesian inference, while in practice, particular neural network architectures are used together with stochastic gradient descent.

Original: Whether practical algorithms of this sort can achieve sample complexity bounds similar to what we have established for our multilayer environments remains an interesting subject for future research.
Revised: Whether practical algorithms of this sort can achieve sample complexity bounds similar to what we have established for our multilayer environments remains an interesting subject for future research.

Original: Our discussion of suboptimal algorithms and misspecification in section 3.1 and Appendix B provide some starting points for such a pursuit.
Revised: Our discussion of suboptimal algorithms and misspecification in section 3.1 and Appendix B provide some starting points for such a pursuit.

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: We thank Andrea Montari for suggesting that information-theoretic tools may be helpful to understanding the dependence of sample complexity on depth.
Revised: We thank Andrea Montari for suggesting that information-theoretic tools may be helpful to understanding the dependence of sample complexity on depth.

Original: Financial support from Army Research
Revised: Financial support from Army Research

Original: Office (ARO) grant W911NF2010055 and the National Science Foundation (NSF) Graduate Research Fellowships Program (GRFP) are gratefully acknowledged.
Revised: Office (ARO) grant W911NF2010055 and the National Science Foundation (NSF) Graduate Research Fellowships Program (GRFP) are gratefully acknowledged.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information on how to answer these questions.
Revised: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] or [N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] or [N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Section ?? .
Revised: [Yes] See Section ?? .

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [No] The code and the data are proprietary.
Revised: [No] The code and the data are proprietary.

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] in Section(c) Did you discuss any potential negative societal impacts of your work?
Revised: [Yes] in Section(c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: It is a math paper.
Revised: It is a math paper.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] Either in the theorem statement or in the respective section of the paper.
Revised: [Yes] Either in the theorem statement or in the respective section of the paper.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] In the Appendix. 3.
Revised: [Yes] In the Appendix. 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: [N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [N/A] 4.
Revised: [N/A] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether
Revised: [N/A] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 5.
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: A Relation to Other Notions of Error A.1 Connections to Cross-Entropy Loss

Original: We establish that in the classification setting, our notion of error is equivalent to cross-entropy loss up to translations.

Original: Example 1.

Original: (cross-entropy loss) Suppose the set Y of responses is finite.

Original: Then,

Original: [Equation]

Original: The first term does not depend on P , so minimizing KL-divergence is equivalent to minimizing the final term,

Original: [Equation]

Original: which is exactly the expected cross-entropy loss of P , as is commonly used to assess classifiers.

Original: A.2 Connections to Mean-Squared Error

Original: In the regression setting we first establish a direct link between KL-divergence and mean-squared error for the case in which P ∗ t and P t are Gaussian.

Original: Example 2.

Original: (Gaussian mean-squared error)

Original: Fix µ ∈ ℜ and σ 2 ∈ ℜ ++ .

Original: Let P ( Y t +1 ∈ ·|E , X t ) ∼

Original: N ( µ, σ 2 ) .

Original: Consider a point prediction ˆ µ t that is determined by H t and a distributional prediction

Original: P t ∼ N (ˆ µ t , σ 2 ) .

Original: Then,

Original: [Equation]

Original: Hence, KL-divergence grows monotonically the squared error E

Original: [( Y t +1 − ˆ µ t ) 2 |E , H t ] .

Original: However, while the minimal squared error E [( Y t +1 − µ ) 2 |E , H t ] = σ 2 that is attainable with full knowledge of the environment remains positive, the minimal KL-divergence, which is delivered by P t ∼ N ( µ, σ 2 ) , is zero.

Original: Now consider a distributional prediction P t ∼ N (ˆ µ t , ˆ σ 2 t ) , based on a variance estimate ˆ σ 2 t ̸ = σ 2 .

Original: Then,

Original: [Equation]

Original: Consider optimizing the choice of ˆ σ 2 t given H t :

Original: [Equation]

Original: The minimum is attained by

Original: [Equation]

Original: which differs from σ 2 .

Original: While σ 2 characterizes aleatoric uncertainty, the incremental variance ˆ σ 2 t − σ 2 accounts for epistemic uncertainty and bias.

Original: Now, for P ∗ t and P t that are not Gaussian, we have the following upper bound:

Original: Lemma 13.

Original: For all t ∈ Z + , if ˆ µ t = (cid:82) y ∈Y y dP t ( y ) , then

Original: [Equation]

Original: Therefore, decreasing the mean squared error will always decrease the expected KL-divergence.

Original: A corresponding lower bound holds for data generating processes for which Y t +1 satisfies a certain subgaussian condition:

Original: Lemma 14.

Original: For all t ∈ Z + , let ˆ µ t = (cid:82) y ∈Y y dP t ( y ) .

Original: If P t ( Y t +1 ∈ · ) is δ 2 t -subgaussian conditioned on H t w.p 1 , then

Original: [Equation]

Original: Therefore, for data generating processes that obey this subgaussian condition, we have both upper and lower bounds for expected KL divergence in terms of mean-squared error.

Original: B Proof of Information and Error Results

Original: Here, we provide a proof of Theorem 3.

Original: Theorem 3.

Original: (optimal prediction)

Original: For all t ≥ 0 ,

Original: [Equation]

Original: where P t = π ( H t , Z t ) .

Original: Proof.

Original: Let ˆ P t =

Original: P ( Y t +1 ∈ ·| H t ) .

Original: By Gibbs’ inequality,

Original: [Equation]

Original: Let P ∗ t =

Original: P ( Y t +1 ∈ ·|E , X t ) .

Original: Then, for all P t ,

Original: [Equation]

Original: [Table]

Original: It follows that

Original: [Equation]

Original: A direct corollary of this result is the following shortfall quantification for a misspecified/suboptimal prediction.

Original: Corollary 15.

Original: (misspecified/suboptimal prediction)

Original: For all t ≥ 0 and P t = π

Original: ( H t , Z t ) ,

Original: [Equation]

Original: An interpretation of the above result is that the error of a suboptimal agent with prediction P t is the sum of the error of an optimal agent ˆ P t and the expected KL divergence between ˆ P t and P t .

Original: Since all the results in this paper bound the first term, in order to understand the behavior of a suboptimal agent, techniques to bound the second term would be required.

Original: Next, we provide a proof for Lemma 4.

Original: Lemma 4.

Original: (expected prediction error equals information gain)

Original: For all t ∈ Z + ,

Original: [Equation]

Original: and R ( t )

Original: =

Original: I ( E ; H t ) .

Original: Proof.

Original: It is well known that the mutual information I ( A ; B ) between random variables A and B can be expressed in terms of the expected KL-divergence I ( A ; B ) =

Original: E [ d KL ( P ( A ∈ ·| B ) ∥

Original: P ( A ∈ · ))] .

Original: It follows that

Original: [Equation]

Original: where ( a ) follows from the fact that Y t +1 ⊥ H t | ( E , X t ) .

Original: We then have

Original: [Equation]

Original: where ( a ) follows from the chain rule of mutual information.

Original: Lastly, we provide a proof for Lemma 5.

Original: Lemma 5.

Original: (expected prediction error is monotonically nonincreasing)

Original: For all t ∈ Z + ,

Original: [Equation]

Original: Proof.

Original: We have

Original: [Equation]

Original: where ( a ) follows from Lemma 4, ( b ) follows from the the fact that Y t +2 ⊥ ( X t , Y t +1 )

Original: | ( E , X t +1 ) , ( c ) follows from the fact that conditioning reduces differential entropy, ( d ) follows from the fact that ( X t , Y t +1 ) and ( X t +1 , Y t +2 ) are independent and identically distributed conditioned on ( H t − 1 , Y t ) , and ( e ) follows from the equivalence between mutual information and expected KL-divergence.

Original: C Proofs of Regret and Sample Complexity Results

Original: In this section, we prove results pertaining to bounding optimal regret and sample complexity in terms of rate-distortion.

Original: Lemma 6.

Original: (proxy error equals information gain)

Original: For all ˜ E ∈ ˜Θ ,

Original: [Equation]

Original: Proof.

Original: It is well known that the mutual information I ( A ; B ) between random variables A and B can be expressed in terms of the expected KL-divergence I ( A ; B ) = E [ d KL ( P ( A ∈ ·| B ) ∥ P ( A ∈ · ))] . We therefore have
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [Equation]

Original: where the second equation follows from the fact that ( X, Y ) ⊥ ˜ E|E .

Original: Theorem 8.

Original: (rate-distortion regret bounds)

Original: For all T ,

Original: [Equation]

Original: Proof.

Original: We begin by establishing the upper bound.

Original: [Equation]

Original: where (a) follows from Lemma 4, ( b ) follows from the chain rule of mutual information, ( c ) follows from the chain rule of mutual information, ( d ) follows from the facts that h ( Y t +1 | ˜ E , H t )

Original: ≤ h ( Y t +1 | ˜

Original: E , X t ) and h ( Y t +1 |E , H t )

Original: =

Original: h ( Y t +1 |E , X t ) , ( e ) holds for any ˜ E ∈ ˜Θ ϵ , and ( f ) follows from the data processing inequality.

Original: Since the above inequality holds for all ϵ ≥ 0 and ˜ E ∈ ˜Θ ϵ , the result follows.

Original: Next, we establish the lower bound.

Original: Fix T ∈ Z + .

Original: Let ˜ E

Original: =

Original: ( ˜ H T − 2 , ˜ Y T − 1 ) be independent from but distributed identically with ( H T − 2 , Y T − 1 ) , conditioned on E .

Original: In other words, ˜ E ⊥ ( H T − 2 , Y T − 1 ) |E and P ( ˜ E ∈ ·|E ) =

Original: P (( H T − 2 , Y T − 1 ) ∈ ·|E ) .

Original: This implies that P (( E , ˜

Original: H T − 2 , ˜ Y T − 1 , X T − 1 , Y T ) ∈ · ) =

Original: P (( E , H T − 2 , Y T − 1 , X T − 1 , Y T ) ∈ · ) , and therefore, I ( E ; Y T | H T − 1 )

Original: =

Original: I ( E ; Y T | ˜ E , X T − 1 ) .

Original: Fix ϵ ≥ 0 .

Original: If R ( T ) <

Original: H ϵ ( E ) then,

Original: [Equation]

Original: where (a) follows from Lemma 4, (b) follows from the chain rule of mutual information, (c) follows from Lemma 5, (d) follows from Lemma 6, and (e) follows from the fact that ˜ E / ∈ ˜Θ ϵ because

Original: R ( T ) =

Original: I ( E ; H T ) =

Original: I ( E ; ˜ E ) <

Original: H ϵ ( E ) .

Original: Therefore,

Original: R ( T ) ≥ min { H ϵ ( E ) , ϵT } .

Original: Since this holds for any ϵ ≥ 0 , the result follows.

Original: Theorem 9.

Original: (rate-distortion sample complexity bounds)

Original: For all ϵ ≥ 0 ,

Original: [Equation]

Original: ϵ (cid:25) .

Original: Proof.

Original: We begin by showing the upper bound.

Original: Fix ϵ ≥ 0 and δ ∈ [0 , ϵ ] .

Original: Let

Original: T = (cid:24)

Original: H ϵ − δ ( E ) δ (cid:25) , so that H ϵ − δ ( E ) ≤ δT .

Original: We have that:

Original: [Equation]

Original: where ( a ) follows from the upper bound of Theorem 8 and ( b ) follows from our choice of T .

Original: Since

Original: T ϵ = min { T : R ( T )

Original: ≤ ϵ R (1) T }

Original: , it follows that T ≥ T ϵ .

Original: Since the above holds for arbitrary δ ∈ [0 , ϵ ] , the result follows.

Original: We now show the lower bound.

Original: Fix ϵ ≥ 0 .

Original: By the definition of T ϵ , we have

Original: [Equation]

Original: In the proof of the lower bound in Theorem 8, we show that for all ϵ ≥ 0 , R ( T ) <

Original: H ϵ ( E ) =

Original: ⇒

Original: R ( T ) > ϵT .

Original: Therefore, using the contrapositive and the above definition of T ϵ , we have that H ϵ ( E ) ≤ R ( T ϵ ) and therefore

Original: [Equation]

Original: The result follows.

Original: D Proofs of Single-Layer Rate-Distortion Bounds

Original: We have established in Theorem 10 that the rate-distortion function for a multilayer environment is a sum of the rate-distortion functions for each single-layer environment under tolerance ϵK .

Original: In this section, we show the following rate-distortion bounds for a single layer of the deep ReLU network and the deep nonparametric network.

Original: We begin by stating a well known result from information theory:

Original: Lemma 16.

Original: (maximum differential entropy) For all random vectors

Original: X : Ω (cid:55)→ ℜ d with covariance

Original: K ,

Original: [Equation]

Original: with equality iff Pr ( X ∈ · ) ∼ N ( µ, K ) for some µ ∈ ℜ d .

Original: Proof.

Original: Follows from Theorems 8.6.3 and 8.6.5 of Cover and Thomas [2006].

Original: Theorem 17.

Original: (single-layer relu neural network rate-distortion bound)

Original: For all N, k ∈ Z ++ and σ 2 , ϵ ≥ 0 , if E k is identified by ( A, b ) where A : Ω (cid:55)→ ℜ N × N , b : Ω (cid:55)→ ℜ N both consist of independent elements each with variance 1 N , X : Ω (cid:55)→ ℜ N is a random vector with covariance

Original: I N , and Y ∼ N (ReLU( AX + b ) , σ 2 I N ) , then

Original: [Equation]

Original: Proof.

Original: We use A i , b i to denote the i th rows of A, b respectively.

Original: Let ˜ E k

Original: = ( ˜ A, ˜ b )

Original: where

Original: ˜ A =

Original: A + V where V ⊥ A and V ∼ N (0 , δ 2 I d ) and likewise ˜

Original: b =

Original: b + Z where Z ⊥ b and Z ∼ N (0 , δ 2 ) .

Original: [Equation]

Original: N + .

Original: We have that

Original: [Equation]

Original: We now verify that our choice of ˜ E k satisfies the distortion constraint:

Original: I

Original: [Equation]

Original: Y

Original: [Equation]

Original: where in ( a ) , W i ∼ N (0 , σ 2 ) , ( b ) follows from the fact that conditioning reduces entropy, ( c ) follows from Lemma 16, ( d ) follows from Jensen’s Inequality, and ( e ) follows from the fact

Original: that for all x, y ∈ ℜ , (ReLU( x ) − ReLU( y )) 2 ≤ ( x − y ) 2 .

Original: Theorem 18.

Original: (single-layer nonparametric rate-distortion bound)

Original: For all N, M, c ∈ Z ++ and σ 2 , ϵ ≥ 0 , if E is identified by α where α ∼ Dir (cid:0) N, (cid:2) cN , . . .

Original: , cN (cid:3)(cid:1) and f ( X

Original: [Equation]

Original: = √ c

Original: [Equation]

Original: n

Original: [Equation]

Original: α n g n

Original: [Equation]

Original: X ) for deterministic basis functions ( g 1 , . . .

Original: , g N ) mapping ℜ M (cid:55)→ ℜ M that satisfy E [ g n ( X ) ⊤ g n ( X )] ≤ 1 for random vector X : Ω (cid:55)→ ℜ N , and Y ∼ N ( f ( X ) , σ 2 I M ) , then

Original: [Equation]

Original: Proof.

Original: Let ˜ E = √ cr (cid:80) ri =1 h i for r = c 2 σ 2 ϵ and where for all i ∈ { 1 , . . . , r } , h i iid ∼ (cid:8) δ n · g n w.p α n .
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: The distortion of ˜ E is

Original: I ( f ( X )

Original: [Equation]

Original: W

Original: [Equation]

Original: E| ˜ E

Original: [Equation]

Original: X )

Original: [Equation]

Original: [Equation]

Original: ( X )

Original: [Equation]

Original: W

Original: [Equation]

Original: X

Original: [Equation]

Original: X

Original: [Equation]

Original: X

Original: [Equation]

Original: ϵ

Original: [Equation]

Original: [-where ( a ) follows from conditioning reducing differential entropy, ( b ) follows from Lemma 16, and ( c ) follows from the fact that for all m, x ∈ ℜ ++ , m ln (cid:0) 
-]
Revised: [-where ( a ) follows from conditioning reducing differential entropy, ( b ) follows from Lemma 16, and ( c ) follows from the fact that for all m, x ∈ ℜ ++ , m ln (cid:0) 
-]

Original: + xm (cid:1) ≤ x .

Original: We now upper bound the rate of ˜ E .

Original: [Equation]

Original: The result follows.

Original: E Proof of Multilayer Results

Original: Lemma 19.

Original: (more is learned with the true input) Let

Original: ˜ E K

Original: :1 be a multilayer proxy.

Original: Then,

Original: [Equation]

Original: Proof.

Original: [Equation]

Original: where ( a ) follows from the fact that E k ⊥ ˜

Original: E k − 1:1 | ( X, Y, E k − 1:1 ) and the data processing inequality, ( b ) follows from the fact that I ( E k ; E k − 1:1 , X |E K : k +1 ,

Original: ˜ E k )

Original: = 0 , and ( c ) follows from the fact

Original: that

Original: Y ⊥ ( E k − 1:1 , X ) |

Original: U k − 1 .

Original: Lemma 19 states that we learn more information about E k when we are given the true input U k −than when we are given ( X, ˜

Original: E k − 1:1 ) and have to infer U k − 1 .

Original: This is intuitive as we should be able to recover more about E k when we observe its input exactly.

Original: To prove 2) , that the immediate output U k provides more information about E k than does Y , each prototypical environment will involve slightly different analysis techniques.

Original: In essence, we expect a result like 2) to hold as a result of the data-processing inequality.

Original: No amount of post-processing of

Original: U k into ( Y ) can increase the amount of available information conveyed about E k .

Original: Lemma 20.

Original: For all K ∈ Z ++ , ( N 0 , N 1 , . . .

Original: , N K ) ∈

Original: Z K +1++ , and σ 2 ≥ 0 , if N K = 1 , and multilayer environment E K :1 consists of single-layer environments E k that each identify a random function f ( k ) from ℜ N k − 1 (cid:55)→ ℜ N k for which

Original: [Equation]

Original: and

Original: U

Original: [Equation]

Original: with Y ∼ N ( U K , σ 2 ) , then for any proxy ˜ E k ,

Original: [Equation]

Original: Proof.

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: We now provide a proof of Theorem 10.

Original: Theorem 10.

Original: multilayer rate-distortion bound For all K ∈ Z

Original: ++ , σ 2 , ϵ ≥ 0 , if E K :1 is a multilayer environment such that for all k ∈ { 1 , . .

Original: , K } and δ ≥ 0 , there exist ˜ E k s.t.

Original: [Equation]

Original: where W ∼ N (0 , σ 2 I ) , then

Original: [Equation]

Original: Proof.

Original: Let

Original: [Equation]

Original: We have that

Original: [Equation]

Original: where ( a ) follows from the fact that ˜ Θ K :1 ϵ ⊂ ˜

Original: Θ ϵ , ( b ) follows from the fact that ˜ E

Original: i ⊥ E j for i ̸

Original: = j , and ( c ) follows from the fact that for ˜Θ ( k ) ϵ/K :

Original: = { ˜ E k ∈ ˜Θ : I ( U k + W ; E k | ˜

Original: E k , U k − 1 ) ≤ ϵK } , ˜Θ (1) ϵ/K × . . .

Original: × ˜Θ ( K ) ϵ/K

Original: ⊂ ˜Θ K :1 ϵ since we assumed

Original: that I ( Y ; E k |E K : k +1 , ˜ E k :1 , X ) ≤

Original: I ( U k + W ; E k | ˜

Original: E k , U k − 1 ) for all k .

Original: F Proof of Main Results

Original: Theorem 11.

Original: (relu neural network rate-distortion and sample complexity bound)

Original: For all d, N, K ∈ Z ++ and σ 2 , ϵ ≥ 0 , if multilayer environment E K :1 is the deep ReLU network with input X : Ω (cid:55)→ ℜ d s.t. V

Original: [ X ] = I d and output Y ∼ N ( U K , σ 2 ) , then

Original: [Equation]

Original: Proof.

Original: If we have a distortion function d ( E k , ˜ E k ) for which

Original: [Equation]

Original: then by the same proof as in Theorem 10,

Original: [Equation]
Revised: [N/A]

Original: where H ϵK ( E k , d ) denotes the rate-distortion function for random variable E k under distortion function d ( E k ,

Original: ˜ E k ) .

Original: By Lemma 20, we have the above condition for d ( E k , ˜ E k ) =

Original: [Equation]

Original: (cid:17) .

Original: Furthermore, distortion used in the proof of Theorem 17 is exactly d ( E k ; ˜ E k ) .

Original: As a result,

Original: [Equation]

Original: where ( a ) follows from Theorem 17.

Original: The sample complexity result follows from applying Theorem 9.

Original: Theorem 12.

Original: (nonparametric network rate-distortion and sample complexity bound)

Original: For all c, d, N, M, K ∈ Z ++ , σ 2 , ϵ ≥ 0 , if multilayer environment E K :1 is the deep nonparametric network with input X : Ω (cid:55)→ ℜ d and output Y ∼ N ( U K , σ 2 )

Original: , then

Original: [Equation]

Original: Proof.

Original: If we have a distortion function d ( E k , ˜ E k ) for which

Original: [Equation]

Original: then by the same proof as in Theorem 10,

Original: [Equation]

Original: where H ϵK ( E k , d ) denotes the rate-distortion function for random variable E k under distortion function d ( E k ,

Original: ˜ E k ) .

Original: By Lemma 20, we have the above condition for d ( E k , ˜ E k )

Original: =

Original: 1 2 ln

Original: [Equation]

Original: Furthermore, distortion used in the proof of Theorem 18 is exactly d ( E k ; ˜ E k ) .

Original: As a result,

Original: [Equation]

Original: where ( a ) follows from Theorem 18.

Original: The sample complexity result follows from applying Theorem 9.