Original: Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformerss
Revised: Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformers

Original: AfﬁliationAddress
Revised: AfﬁliationAddress

Original: Abstract
Revised: Abstract

Original: A common lens to theoretically study neural net architectures is to analyze the functions they can approximate.
Revised: A common lens to theoretically study neural net architectures is to analyze the functions they can approximate.

Original: However, the constructions from approximation theoryoften have unrealistic aspects, for example, reliance on inﬁnite precision to memorizetarget function values.
Revised: However, the constructions from approximation theoryoften have unrealistic aspects, for example, reliance on inﬁnite precision to memorizetarget function values.

Original: To address this issue, we propose a formal deﬁnition of statisticallymeaningfulapproximationwhichrequirestheapproximatingnetworktoexhibitgood statistical learnability.
Revised: To address this issue, we propose a formal deﬁnition of statisticallymeaningfulapproximationwhichrequirestheapproximatingnetworktoexhibitgood statistical learnability.

Original: We present case studies on statistically meaningful approximation for two classes of functions: boolean circuits and Turing machines.
Revised: We present case studies on statistically meaningful approximation for two classes of functions: boolean circuits and Turing machines.

Original: Weshow that overparameterized feedforward neural nets can statistically meaningfullyapproximate boolean circuits with sample complexity depending only polynomiallyon the circuit size, not the size of the approximating network.
Revised: Weshow that overparameterized feedforward neural nets can statistically meaningfullyapproximate boolean circuits with sample complexity depending only polynomiallyon the circuit size, not the size of the approximating network.

Original: In addition, we showthat transformers can statistically meaningfully approximate Turing machines withcomputation time bounded by T , requiring sample complexity polynomial in thealphabet size, state space size, and log p T q .
Revised: In addition, we showthat transformers can statistically meaningfully approximate Turing machines withcomputation time bounded by T , requiring sample complexity polynomial in thealphabet size, state space size, and log p T q .

Original: Our analysis introduces new tools for generalization bounds that provide much tighter sample complexity guarantees than thetypical VC-dimension or norm-based bounds, which may be of independent interest.
Revised: Our analysis introduces new tools for generalization bounds that provide much tighter sample complexity guarantees than thetypical VC-dimension or norm-based bounds, which may be of independent interest.

Original:  Introduction
Revised:  Introduction

Original: Dating back to the seminal works on universal approximation [15, 22, 35, 27], a common way totheoretically study neural nets has been through their expressivity, which measures the ability ofneural nets to approximate well-behaved functions.
Revised: Dating back to the seminal works on universal approximation [16, 25, 40, 31], a common way totheoretically study neural nets has been through their expressivity, which measures the ability ofneural nets to approximate well-behaved functions.

Original: This perspective has shaped how researchersperceive different types of deep learning architectures: a basic way to theoretically justify newarchitectures is to study their approximation capabilities.
Revised: This perspective has shaped how researchersperceive different types of deep learning architectures: a basic way to theoretically justify newarchitectures is to study their approximation capabilities.

Original: This has led to a number of analyses studyinguniversal approximation capabilities for various widely-used architectures, such as recurrent neuralnets (RNNs) [42], graph neural nets [41], convolutional networks [3, 58, 53], residual networks [28],transformers [55], and neural ODEs [46, 57].
Revised: This has led to a number of analyses studyinguniversal approximation capabilities for various widely-used architectures, such as recurrent neuralnets (RNNs) [47], graph neural nets [46], convolutional networks [3, 64, 59], residual networks [32],transformers [61], and neural ODEs [51, 63].

Original: However, approximation theoretic results often misalign with more meaningful end-to-end guarantees,because models constructed in the literature often exhibit unrealistic properties.
Revised: However, approximation theoretic results often misalign with more meaningful end-to-end guarantees, because models constructed in the literature often exhibit unrealistic properties.

Original: For example, acommon technique in the universal approximation literature is to rely strongly on inﬁnite-precisionweights and activations, or exponentially many parameters to encode the desired function values [22, 15, 27, 28, 55, 39].
Revised: For example, a common technique in the universal approximation literature is to rely strongly on inﬁniteprecision weights and activations, or exponentially many parameters to encode the desired functionvalues [25, 16, 31, 32, 61, 44].

Original: This issue even arises outside of universal approximation, e.g., variouspapers demonstrate the ability of RNNs and transformers to simulate various computational modelssuch as Turing machines and automata, but require strong reliance on arbitrary precision [43, 37, 25, 8].
Revised: This issue even arises outside of universal approximation, e.g., variouspapers demonstrate the ability of RNNs and transformers to simulate various computational modelssuch as Turing machines and automata, but require strong reliance on arbitrary precision [48, 42, 29, 9].

Original: Inﬁnite precision can inﬂate the expressivity of an architecture (function class) in a unrealistic andmisleading way: for example, ﬁnite width RNNs with inﬁnite precision can simulate Turing machines,but ﬁnite-precision, ﬁnite-width RNNs cannot, as implied by streaming lower bounds [1].
Revised: Inﬁnite precision can inﬂate the expressivity of an architecture (function class) in a unrealistic andmisleading way: for example, ﬁnite width RNNs with inﬁnite precision can simulate Turing machines,but ﬁnite-precision, ﬁnite-width RNNs cannot. This is implied by streaming lower bounds [1] – anyﬁnite-precision, ﬁnite-width RNN induces a ﬁnite-space streaming algorithm corresponding to runningthe RNN on the inputs. However, streaming lower bounds tell us that ﬁnite-space streaming algorithms are not powerful enough to simulate Turing machines, and hence ﬁnite-precision, ﬁnite-width RNNs cannot either.

Original: As anotherexample, Park et al.
Revised: As another example, Park et al.

Original: [36] exploit inﬁnite precision in the parameters to show that a neural net withparameter count sublinear in n can memorize n arbitrary input-label pairs.
Revised: [41] exploit inﬁnite precision in the parameters to showthat a neural net with parameter count sublinear in n can memorize n arbitrary input-label pairs.

Original: However, a simple counting argument reveals that this result cannot be proven using ﬁnite precision networks – there are 2 ninput-labeling pairs, but only 2 o p n q ﬁnite precision networks with o p n q parameters.
Revised: However, a simple counting argument reveals that this result cannot be proven using ﬁnite precision networks– there are 2 n input-labeling pairs, but only 2 o p n q ﬁnite precision networks with o p n q parameters.

Original: More broadly, the ideal theoretical perspective should consider not only whether target functionscan be expressed, but also whether the approximating functions can plausibly be obtained by ﬁttinga neural network to a ﬁnite training sample, as is the case in practical deep learning settings.
Revised: More broadly, the ideal theoretical perspective should consider not only whether target functionscan be expressed, but also whether the approximating functions can plausibly be obtained by ﬁttinga neural network to a ﬁnite training sample, as is the case in practical deep learning settings.

Original: The latterquestion can be decomposed into studying optimization and generalization.
Revised: The latterquestion can be decomposed into studying optimization and generalization.

Original: Unfortunately, a rigorousanalysis of optimization is unresolved even for simple two-layer nets [30].
Revised: Unfortunately, a rigorousanalysis of optimization is unresolved even for simple two-layer nets [35, 33].

Revised: Global optimizationanalyses such as NTK do exist [18, 26], but there is a large body of theoretical and empirical workshowing that neural networks can generalize much better than NTK analyses can hope to prove [20, 57].

Original: Generalization is moretractable, so we propose to study expressivity and generalization together.
Revised: Generalization is more tractable, so we propose to study expressivity and generalization together.

Original: Towards studying more meaningful notions of approximation, this work proposes statisticallymeaningful (SM) approximation .
Revised: Towards studying more meaningful notions of approximation, this work proposes statisticallymeaningful (SM) approximation .

Original: This deﬁnition requires not only the existence of an approximatingnetwork, but also that it has good statistical properties.
Revised: This deﬁnition requires not only the existence of an approximatingnetwork, but also that it has good statistical properties.

Original: Consider a setting where the aim is to ﬁtthe target function G using the approximating family F and a ﬁnite sample of training data.
Revised: Consider a setting where the aim is to ﬁtthe target function G using the approximating family F and a ﬁnite sample of training data.

Original: SMapproximation requires existence of a loss whose empirical risk minimizer in F leads to a model withlow approximation error in ﬁtting G .
Revised: SMapproximation requires existence of a loss whose empirical risk minimizer in F leads to a model withlow approximation error in ﬁtting G .

Original: We deﬁne the sample complexity of the approximation as thenumber of training samples needed to guarantee (cid:15) approximation error and study SM approximationwith low sample complexity bounds.
Revised: We deﬁne the sample complexity of the approximation as thenumber of training samples needed to guarantee (cid:15) approximation error and study SM approximationwith low sample complexity bounds.

Original: SM approximation essentially eliminates statistical concernsabout ﬁtting the target function with a ﬁnite sample (optimization concerns can remain).
Revised: SM approximation essentially eliminates statistical concernsabout ﬁtting the target function with a ﬁnite sample (optimization concerns can remain).

Original: We present two case studies on SM approximation.
Revised: We present two case studies on SM approximation.

Original: First, we demonstrate that overparameterizedfeedforward neural nets can SM approximate boolean circuits with a low sample complexity thatdepends only on the intrinsic circuit size.
Revised: First, we demonstrate that overparameterizedfeedforward neural nets can SM approximate boolean circuits with a low sample complexity thatdepends only on the intrinsic circuit size.

Original: Though it is simple to construct neural nets to approximateboolean circuits, bounding the sample complexity of the approximation is challenging.
Revised: Though it is simple to construct neural nets to approximateboolean circuits, bounding the sample complexity of the approximation is challenging.

Original: For example,standard norm-based generalization bounds for the naive construction scale exponentially in depth [5].
Revised: For example,standard norm-based generalization bounds for the naive construction scale exponentially indepth [5, 6].

Original: Furthermore, VC dimension-based bounds would scale polynomially in the number of parametersin the network [20], which is problematic because for practical optimization concerns, neural netsare typically overparameterized in terms of width [56].
Revised: Furthermore, VC dimension-based bounds would scale polynomially in the number ofparameters in the network [23], which is problematic because for practical optimization concerns,neural nets are typically overparameterized in terms of width [62].

Original: In contrast, our sample complexity boundfor SM approximation depends only on the intrinsic circuit size, up to logarithmic factors.
Revised: In contrast, our sample complexitybound for SM approximation depends only on the intrinsic circuit size, up to logarithmic factors.

Original: Our second case study is on SM approximating Turing machines with transformers.
Revised: Our second case study is on SM approximating Turing machines with transformers.

Original: We considera class of Turing machines with bounded computation time T and construct encoder-decodertransformers [48] which SM approximate these Turing machines.
Revised: We considera class of Turing machines with bounded computation time T and construct encoder-decodertransformers [53] which SM approximate these Turing machines.

Original: The sample complexity of theapproximation depends on a polynomial in log T and the sizes of the state space and the alphabet of the
Revised: The sample complexity of theapproximation depends on a polynomial in log T and the sizes of the state space and the alphabet of the

Original: Turing machine.
Revised: Turing machine.

Original: Though constructions for approximating Turing machines from prior work [43, 37, 8]have not been formally studied from a sample complexity perspective, existing bounds would dependat least linearly on T .
Revised: Though constructions for approximating Turing machines from prior work [48, 42, 9]have not been formally studied from a sample complexity perspective, existing bounds would dependat least linearly on T .

Original: Furthermore, our construction only uses loglog T precision, compared to atleast log T in prior works, resulting in the exponential improvement in the sample complexity.
Revised: Furthermore, our construction only uses loglog T precision, compared to atleast log T in prior works, resulting in the exponential improvement in the sample complexity.

Original: Proving sample complexity guarantees for our SM approximation results is nontrivial and requiresadditional insights.
Revised: Proving sample complexity guarantees for our SM approximation results is nontrivial and requiresadditional insights.

Original: To obtain our sample complexity bounds, we leverage a recent generalizationbound which depends on data-dependent Lipschitzness [51].
Revised: To obtain our sample complexity bounds, we leverage a recent generalizationbound which depends on data-dependent Lipschitzness [56].

Original: We develop theoretical tools to converta broad class of neural nets, with possibly large Lipschitzness, into ones with small Lipschitznesson the training data, by introducing a number of new layers that is linear in depth.
Revised: We develop theoretical tools to converta broad class of neural nets, with possibly large Lipschitzness, into ones with small Lipschitznesson the training data, by introducing a number of new layers that is linear in depth.

Original: Our result appliesto neural nets where each entry in the hidden representations on the training data takes values froma ﬁnite set (e.g., binary entries), and may be of independent interest.
Revised: Our result appliesto neural nets where each entry in the hidden representations on the training data takes values froma ﬁnite set (e.g., binary entries), and may be of independent interest.

Original: In summary, our conceptual contribution is to propose a new notion of statistically meaningfulapproximation, intended to provide more meaningful guarantees by requiring that the approximatingfamily have good statistical learnability.
Revised: In summary, our conceptual contribution is to propose a new notion of statistically meaningfulapproximation, intended to provide more meaningful guarantees by requiring that the approximatingfamily have good statistical learnability.

Original: Technically, 1) we prove that feedforward neural nets canmeaningfully approximate boolean circuits with sample complexity that depends polynomially onthe width and depth of the circuit; and 2) we show that transformers can meaningfully approximate
Revised: Technically, 1) we prove that feedforward neural nets canmeaningfully approximate boolean circuits with sample complexity that depends polynomially onthe width and depth of the circuit; and 2) we show that transformers can meaningfully approximate

Original: Turing machines with sample complexity logarithmic in the computation time.
Revised: Turing machines with sample complexity logarithmic in the computation time.

Original: 1.1 Related works
Revised: 1.1 Related works

Original: Classiﬁcal approximation theory for neural networks has a long history.
Revised: Classiﬁcal approximation theory for neural networks has a long history.

Original: Hornik et al.
Revised: Hornik et al.

Original: [22], Cybenko[15], and Leshno et al.
Revised: [25], Cybenko[16], and Leshno et al.

Original: [27] show that neural nets with one hidden layer are universal approximatorsbut require the hidden layer size to grow exponentially in input dimension.
Revised: [31] show that neural nets with one hidden layer are universal approximatorsbut require the hidden layer size to grow exponentially in input dimension.

Original: Barron [4] uses the Fouriertransform to write target functions as inﬁnite-width networks and subsamples neurons to obtain widthswhich depend only on target function properties.
Revised: Barron [4] uses the Fouriertransform to write target functions as inﬁnite-width networks and subsamples neurons to obtain widths which depend only on target function properties.

Original: Lee et al.
Revised: Lee et al.

Original: [26], Ji et al.
Revised: [30], Ji et al.

Original: [23] prove recent relateddevelopments in this direction of universal approximation.
Revised: [27] prove recent relateddevelopments in this direction of universal approximation.

Original: Many works study beneﬁts of deep networks over shallow ones [7, 2, 45, 17, 16, 10, 9].
Revised: Many works study beneﬁts of deep networks over shallow ones [8, 2, 50, 19, 17, 11, 10].

Original: Bengio and
Revised: Bengio and

Original: Delalleau [7] show separation for exact representation, whereas Telgarsky [45] shows separation for ap- proximaterepresentationswithunivariateinputs.
Revised: Delalleau [8] show separation for exact representation, whereas Telgarsky [50] shows separation for approximaterepresentationswithunivariateinputs.

Original: EldanandShamir[17]demonstratehigh-dimensionalfunctions that can be approximated by two-layer polynomial-sized neural networks, but cannot beapproximated by one-layer neural nets with subexponential hidden units.
Revised: EldanandShamir[19]demonstratehigh-dimensionalfunctions that can be approximated by two-layer polynomial-sized neural networks, but cannot beapproximated by one-layer neural nets with subexponential hidden units.

Original: Via reduction to certain complexity theoretic questions, Vardi and Shamir [47] show that proving constant depth separations may behard.
Revised: Via reduction to certain complexity theoretic questions, Vardi and Shamir [52] show that proving constant depth separations may behard.

Original: Malach et al.
Revised: Malach et al.

Original: [29] analyze the relationship between optimization and approximability, showing invarious settings that deeper networks cannot be optimized if shallow networks cannot approximate them.
Revised: [34] analyze the relationship between optimization and approximability, showing invarious settings that deeper networks cannot be optimized if shallow networks cannot approximate them.

Original: This demonstrates that depth separation results [45] from approximation theory can be misleading sincegradient descent anyways cannot optimize the deep networks used to construct the approximation.
Revised: This demonstrates that depth separation results [50] from approximation theory can be misleading sincegradient descent anyways cannot optimize the deep networks used to construct the approximation.

Original: Another area of study is on the ability of deep networks to memorize training data [56, 54, 36, 49].
Revised: Another area of study is on the ability of deep networks to memorize training data [62, 60, 41, 54].

Original: Yunetal.[54]showthat Θ p n q parametersaresufﬁcienttomemorize Θ p n q trainingpointsforReLUnetswithat least 3 layers, and Park et al.
Revised: Yunetal.[60]showthat Θ p n q parametersaresufﬁcienttomemorize Θ p n q trainingpointsforReLUnetswithat least 3 layers, and Park et al.

Original: [36] reduce the parameter requirement to sublinear in n .
Revised: [41] reduce the parameter requirement to sublinear in n .

Original: Similar resultshavebeenprovenforresidualarchitectures[19]andconvolutionalnets[34].Bartlettetal.[6]analyzethe
Revised: Similar resultshavebeenprovenforresidualarchitectures[22]andconvolutionalnets[39].Bartlettetal.[7]analyzethe

Original: VC-dimension of neural nets, leading to bounds on the parameter count needed to ﬁt training data.
Revised: VC-dimension of neural nets, leading to bounds on the parameter count needed to ﬁt training data.

Original: Otherworks study expressivity via connections to tensor approximation and sum-product networks [13, 14].
Revised: Otherworks study expressivity via connections to tensor approximation and sum-product networks [14, 15].

Original: There is a long line of work on studying the ability of neural nets to recognize and represent formallanguages.
Revised: There is a long line of work on studying the ability of neural nets to recognize and represent formallanguages.

Original: The seminal work of Siegelmann and Sontag [43] shows that RNNs are Turing completebut leverages inﬁnite precision in the hidden activations.
Revised: The seminal work of Siegelmann and Sontag [48] shows that RNNs are Turing completebut leverages inﬁnite precision in the hidden activations.

Original: Chen et al.
Revised: Chen et al.

Original: [11] extend this result to ReLUactivations and study implications in language modeling.
Revised: [12] extend this result to ReLUactivations and study implications in language modeling.

Original: Many variants of transformers are shownto be Turing-complete, but these constructions also rely on arbitrary precision [37, 8].
Revised: Many variants of transformers are shownto be Turing-complete, but these constructions also rely on arbitrary precision [42, 9].

Original: Recent workshave also proven results for generating or recognizing formal languages with ﬁnite-precision neuralnets [52, 25, 21], but these results do not consider Turing machines or analyze statistical properties oftheir constructions.
Revised: Recent workshave also proven results for generating or recognizing formal languages with ﬁnite-precision neuralnets [58, 29, 24], but these results do not consider Turing machines or analyze statistical properties oftheir constructions.

Original: Concurrent work [12] proves Turing completeness of RNNs with ﬁnite precision,relying on a dynamically growing memory module in the architecture (which serves the same purpose asthe long decoder sequences in our Transformer construction).
Revised: Concurrent work [13] proves Turing completeness of RNNs with ﬁnite precision,relying on a dynamically growing memory module in the architecture (which serves the same purpose asthe long decoder sequences in our Transformer construction).

Original: However, they do not analyze statisticalproperties, which requires additional complications in both the construction and statistical analysis.
Revised: However, they do not analyze statisticalproperties, which requires additional complications in both the construction and statistical analysis.

Original: 1.2 Notation
Revised: 1.2 Notation

Original: Let f ˝
Revised: Let f ˝

Original: g denote the composition of functions f and g .
Revised: g denote the composition of functions f and g .

Original: For a family of functions G , letf ˝ G ﬁ t
Revised: For a family of functions G , letf ˝ G ﬁ t

Original: f
Revised: f

Original: ˝
Revised: ˝

Original: g : g P G u denote the family of compositions between f and functions in G .
Revised: g : g P G u denote the family of compositions between f and functions in G .

Original: For a set Sand function f : S Ñ Y , let f p S q denote the set t f p s q : s P S uĎ Y .
Revised: For a set Sand function f : S Ñ Y , let f p S q denote the set t f p s q : s P S uĎ Y .

Original: We use 1 d to denote the all-one’svector in d dimensions, with the subscripted omitted if clear.
Revised: We use 1 d to denote the all-one’svector in d dimensions, with the subscripted omitted if clear.

Original: For i Pr d s , we let 1 d p i q denote the one-hotembedding in d -dimensions, which is 1 at index i and 0 everywhere else.
Revised: For i Pr d s , we let 1 d p i q denote the one-hotembedding in d -dimensions, which is 1 at index i and 0 everywhere else.

Original: We use the notation r O p¨q tohide poly-logarithmic factors in the argument.
Revised: We use the notation r O p¨q tohide poly-logarithmic factors in the argument.

Original: The notation À , Á indicates the existence of a constantfactor such that the inequality holds, and — denotes that the Á and À relations simultaneously hold.
Revised: The notation À , Á indicates the existence of a constantfactor such that the inequality holds, and — denotes that the Á and À relations simultaneously hold.

Original: Weuse poly p¨q to indicate the existence of a polynomial in the argument which makes the equation true.
Revised: Weuse poly p¨q to indicate the existence of a polynomial in the argument which makes the equation true.

Original: Fora set A (e.g., the set of alphabet symbols for a Turing machine) let A ˚ denote the set of all sequences ofelements of A , where sequence length can vary.
Revised: Fora set A (e.g., the set of alphabet symbols for a Turing machine) let A ˚ denote the set of all sequences ofelements of A , where sequence length can vary.

Original: Let P denote a distribution over a space of inputs X .
Revised: Let P denote a distribution over a space of inputs X .

Original: Letξ 1 ,...,ξ n be n i.i.d.
Revised: Letξ 1 ,...,ξ n be n i.i.d.

Original: Rademacher variables sampled from t´ 1 , ` 1 u .
Revised: Rademacher variables sampled from t´ 1 , ` 1 u .

Original: The expected n -sample Rademachercomplexity of F on P is as follows:
Revised: The expected n -sample Rademachercomplexity of F on P is as follows:

Original: [Equation]
Revised: Rad n,P p F q ﬁ E p x i q ni “ 1 i.i.d „ P “ E ξ 1 ,...,ξ n “ sup F P F 1 n ř ni “ 1 ξ i F p x

Original: where p x i q ni “ 1 denotes n i.i.d.
Revised: i q ‰‰ ,where p x i q ni “ 1 denotes n i.i.d.

Original: samples from P .
Revised: samples from P .

Original:  Statistically meaningful approximation
Revised:  Statistically meaningful approximation

Original: We consider settings where we wish to approximate every member G in a real-valued function class
Revised: We consider settings where we wish to approximate every member G in a real-valued function class

Original: G with some function F in function class F .
Revised: G with some function F in function class F .

Revised: Functions in both G and F map input space X to R .

Original: In this work, F is some family of neural networks.
Revised: Inthis work, F is some family of neural networks.

Original: Fixa loss (cid:96) :
Revised: Fix a loss (cid:96) :

Original: R ˆ R Ñr 0 , 1 s .
Revised: R ˆ R Ñr 0 , 1 s .

Original: The classical notion of (cid:15) -approximation [38] is as follows:
Revised: The classical notion of(cid:15) -approximation [43] is as follows:

Original: Deﬁnition 2.1 (Classical (cid:15) -approximation) .
Revised: Deﬁnition 2.1 (Classical (cid:15) -approximation) .

Original: We say a function class F (cid:15) -approximates a functionclass G with respect to loss (cid:96) and input distribution P , if for any given G P G , there exists F P F suchthat E x „ P r (cid:96) p F p x q ,G p x qqsď (cid:15) .
Revised: We say a function class F (cid:15) -approximates a functionclass G with respect to loss (cid:96) and input distribution P , if for any given G P G , there exists F P F suchthat E x „ P r (cid:96) p F p x q ,G p x qqsď (cid:15) .

Original: The issue with this classical notion of approximation is that it allows solutions which use inﬁnite precision (or other potential unrealistic characteristics).
Revised: The issue with this classical notion of approximation is that it allows solutions which use inﬁnite precision (or other potential unrealistic characteristics).

Original: Because of these drawbacks, even if F approximates
Revised: Because of these drawbacks, even if F approximates

Original: G , it does not mean that we can use F to ﬁt the target function from G with a good sample complexity.
Revised: G , it does not mean that we can use F to ﬁt the target function from G with a good sample complexity.

Original: This work studies a stronger notion of approximation, statistically meaningful (SM) approximation,to eliminate statistical issues with ﬁtting G on a ﬁnite sample.
Revised: This work studies a stronger notion of approximation, statistically meaningful (SM) approximation,to eliminate statistical issues with ﬁtting G on a ﬁnite sample.

Original: SM-approximation requires that G islearnable via empirical risk minimization using models from F , when data is generated from P .
Revised: SM-approximation requires that G islearnable via empirical risk minimization using models from F , when data is generated from P .

Original: Deﬁnition 2.2 ( (cid:15) -SM-approximation) .
Revised: Deﬁnition 2.2 ( (cid:15) -SM-approximation) .

Original: We say a function class F (cid:15) -SM-approximates a function class
Revised: We say a function class F (cid:15) -SM-approximates a function class

Original: G with respect to evaluation loss (cid:96) and input distribution P with sample complexity n if there existsa surrogate loss s (cid:96) : F ˆ X ˆ R Ñr 0 , 1 s such that for any given G P G , the following holds:
Revised: G with respect to evaluation loss (cid:96) and input distribution P with sample complexity n if there existsa surrogate loss s (cid:96) : F ˆ X ˆ R Ñr 0 , 1 s such that for any given G P G , the following holds:

Original: With probability 0.99 over the randomness of n examples p x i q ni “ 1 drawn from P , the empirical riskminimizer of s (cid:96) , p F ﬁ argmin F P F 1 n ř ni “ 1 s (cid:96) p F,x i ,G p x
Revised: With probability 0.99 over the randomness of n examples p x i q ni “ 1 drawn from P , the empirical riskminimizer of s (cid:96) , p F ﬁ argmin F P F 1 n ř ni “ 1 s (cid:96) p F,x i ,G p x

Original: i qq , approximates G : E x „ P r (cid:96)
Revised: i qq , approximates G : E x „ P r (cid:96)

Original: p p F p x q ,G p x qqsď (cid:15) .
Revised: p p F p x q ,G p x qqsď (cid:15) .

Original: Deﬁnition 2.2 requires that the empirical risk minimizer of s (cid:96) over F on a ﬁnite sample p x i ,G p x
Revised: Deﬁnition 2.2 requires that the empirical risk minimizer of s (cid:96) over F on a ﬁnite sample p x i ,G p x

Original: i qq ni “ 1 isguaranteed to (cid:15)
Revised: i qq ni “ 1 isguaranteed to (cid:15)

Original: -approximate
Revised: -approximate

Original: G on the population.
Revised: G on the population.

Original: Note that the surrogate loss s (cid:96) and evaluation loss (cid:96) candiffer, and that s (cid:96) takes the model F as an argument, allowing the empirical risk to include regularization.
Revised: Note that the surrogate loss s (cid:96) and evaluation loss (cid:96) candiffer, and that s (cid:96) takes the model F as an argument, allowing the empirical risk to include regularization.

Original: Though Deﬁnition 2.2 may be reminiscent of PAC-learnability, there is a major conceptual difference:
Revised: Though Deﬁnition 2.2 may be reminiscent of PAC-learnability, there is a major conceptual difference:

Original: SM approximation uniﬁes expressivity and generalization, whereas PAC-learnability is only concernedwith generalization.
Revised: SM approximation uniﬁes expressivity and generalization, whereas PAC-learnability is only concernedwith generalization.

Original: For example, in the realizable PAC-learning case, there is no notion of anapproximating family F – the setting only cares about fundamental learnability of G .
Revised: For example, in the realizable PAC-learning case, there is no notion of anapproximating family F – the setting only cares about fundamental learnability of G .

Original: Furthermore,in agnostic PAC-learning (non-realizable) settings, the main focus is achieving a low loss relative to thebest function in the hypothesis class.
Revised: Furthermore,in agnostic PAC-learning (non-realizable) settings, the main focus is achieving a low loss relative to thebest function in the hypothesis class.

Original: In contrast, SM approximation also requires proving that the bestfunction in F achieves near-zero loss, whereas there is no such requirement in PAC-learning settings.
Revised: In contrast, SM approximation also requires proving that the bestfunction in F achieves near-zero loss, whereas there is no such requirement in PAC-learning settings.

Original: 2.1 Background and tools
Revised: 2.1 Background and tools

Original: To prove SM-approximation guarantees, Deﬁnition 2.2 requires a loss surrogate s (cid:96) such that theempirical risk minimizer of s (cid:96) on the training data can approximate functions in G .
Revised: To prove SM-approximation guarantees, Deﬁnition 2.2 requires a loss surrogate s (cid:96) such that theempirical risk minimizer of s (cid:96) on the training data can approximate functions in G .

Original: The followingproposition, which is motivated by classical generalization theory, provides several conditions ons (cid:96) which lead to SM-approximation guarantees.
Revised: The followingproposition, which is motivated by classical generalization theory, provides several conditions ons (cid:96) which lead to SM-approximation guarantees.

Original: Proposition 2.3.
Revised: Proposition 2.3.

Original: For loss function (cid:96) : R ˆ R Ñ r 0 , 1 s and input distribution P , suppose there existsa surrogate loss s (cid:96) :
Revised: For loss function (cid:96) : R ˆ R Ñ r 0 , 1 s and input distribution P , suppose there existsa surrogate loss s (cid:96) :

Original: F ˆ X ˆ R Ñr 0 , 1 s satisfying the following properties:
Revised: F ˆ X ˆ R Ñr 0 , 1 s satisfying the following properties:

Original: For all F P F , x P X , y P R , s (cid:96) p F,x,y qě (cid:96)
Revised: For all F P F , x P X , y P R , s (cid:96) p F,x,y qě (cid:96)

Original: p F p x q ,y q .2)
Revised: p F p x q ,y q .

Original: For any G P G , consider the function class L G ﬁ t x ÞÑ s (cid:96) p F,x,G p x qq : F P F u .
Revised: For any G P G , consider the function class L G ﬁ t x ÞÑ s (cid:96) p F,x,G p x qq : F P F u .

Original: Then the n -sampleRademacher complexity of L G is bounded: Rad n,P p L G qď (cid:15) .3)
Revised: Then the n -sample Rademacher complexity of L G is bounded: Rad n,P p L G qď (cid:15) .

Original: For any G P G , there exists F P F with small surrogate loss: E x „ P r s (cid:96) p F,x,G p x qqsď (cid:15) .Then, the function class F O ´ (cid:15) ` 1 ? n ¯ -SM-approximates G with respect to loss (cid:96) and input distributionP with sample complexity n .
Revised: For any G P G , there exists F P F with small surrogate loss: E x „ P r s (cid:96) p F,x,G p x qqsď (cid:15) . Then, the function class F O ´ (cid:15) ` 1 ? n ¯ -SM-approximates G with respect to loss (cid:96) and input distribution P

Original:  By Proposition 2.3, it sufﬁces that s (cid:96) upper bounds the target loss (cid:96) and has low complexity, and
Revised: with sample complexity n By Proposition 2.3, it sufﬁces that s (cid:96) upper bounds the target loss (cid:96) and has low complexity, and

Original: F approximates G with respect to p s (cid:96),P q in the classical sense.
Revised: F approximates G with respect to p s (cid:96),P q in the classical sense.

Original: The proof follows from standardtechniques for bounding generalization based on Rademacher complexity and is provided in Section A.
Revised: The proof follows from standardtechniques for bounding generalization based on Rademacher complexity and is provided in Section A.

Original: All-layer margin loss.
Revised: All-layer margin loss.

Original: We introduce one particular construction for s (cid:96) used in subsequent sections,which is motivated by the all-layer margin generalization bound proposed by [51].
Revised: We introduce one particular construction for s (cid:96) used in subsequent sections,which is motivated by the all-layer margin generalization bound proposed by [56].

Original: This bound is basedon data-dependent Lipschitzness measures [31, 50], and can provide stronger guarantees than classicalnorm-based bounds [32, 5, 33, 18].
Revised: This bound is basedon data-dependent Lipschitzness measures [36, 55], and can provide stronger guarantees than classicalnorm-based bounds [37, 6, 38, 21].

Original: We focus on the binary classiﬁcation setting, where G p x q P t 0 , 1 u , and study approximation withrespect to the 0-1 loss (cid:96) 0-1 p z,y q ﬁ 1 pp y ´ 0 .
Revised: We focus on the binary classiﬁcation setting, where G p x qPt 0 , 1 u , and study approximation with respectto the 0-1 loss (cid:96) 0-1 p z,y q ﬁ 1 pp y ´ 0 .

Original: 5 q z ď 0 q where y P t 0 , 1 u is assumed to be a binary label.
Revised: 5 q z ď 0 q where y Pt 0 , 1 u is assumed to be a binary label, and the aimis to output a negative prediction z for y “ 0 and positive for y “ 1 .

Original: We consider a family of functions F parameterized by p -dimensional parameters θ P Θ Ď R p , suchthat F “ t x ÞÑ F p x,θ q : θ P Θ u , where we abuse notation and let F denote a general parameterizedfunction F : X ˆ R p Ñ R .
Revised: We consider a family of functions Fparameterized by p -dimensional parameters θ P Θ Ď R p , such that F “t x ÞÑ F p x,θ q : θ P Θ u , where weabuse notation and let F denote a general parameterized function F : X ˆ R p Ñ R .

Original: We sometimes use θ to identify an element of F .
Revised: We sometimes use θto identify an element of F .

Original: Throughout the paper,we deﬁne Θ as a set with }¨} 1 -norm bounded by α : } θ } 1 ď α, @
Revised: Throughout the paper, we deﬁne Θ as a set with }¨} 1 -norm bounded by α :} θ } 1 ď α, @

Original: θ P Θ .
Revised: θ P Θ .

Original: We deﬁne the parameter-basedall-layer margin ρ F : R p ˆ X ˆt 0 , 1 uÑ R as follows:
Revised: We deﬁne the parameter-based all-layer margin ρ F : R p ˆ X ˆt 0 , 1 uÑ R as follows:

Original: [Equation]
Revised: [Equation]

Original: We omit F from the subscript of ρ when it is clear from context.
Revised: We omit F from the subscript of ρ when it is clear from context.

Original: This quantity measures the stabilityof the model around an input x in parameter space.
Revised: This quantity measures the stabilityof the model around an input x in parameter space.

Original: As is the case for the standard output margin, alarger all-layer margin, or better stability, tends to imply better generalization.
Revised: As is the case for the standard output margin, alarger all-layer margin, or better stability, tends to imply better generalization.

Original: We modiﬁed the deﬁnition in [51] to consider perturbations δ in parameter space, whereas Wei and
Revised: We modiﬁed the deﬁnition in [56] to consider perturbations δ in parameter space, whereas Wei and

Original: Ma [51] consider perturbations to the hidden layers.
Revised: Ma [56] consider perturbations to the hidden layers.

Original: The parameter-space formulation is simpler and subsumes the results in [51].
Revised: The parameter-space formulation is simpler andsubsumes the results in [56].

Original: Our formulation also accounts for weight sharing, which is importantfor our Turing machine results, whereas the formulation of [51] could not.
Revised: Our formulation also accounts for weight sharing, which is importantfor our Turing machine results, whereas the formulation of [56] could not.

Original: A key and immediate property of the all-layer margin is that it is strictly positive if and only if F p x,θ qpredicts the correct label.
Revised: A key and immediate property of the all-layer margin is that it is strictly positive if and only if F p x,θ qpredicts the correct label.

Original: We can leverage this property to construct a surrogate loss.
Revised: We can leverage this property to construct a surrogate loss.

Original: For someparameter γ intended to lowerbound the all-layer margins, we deﬁne the loss s (cid:96) γ as follows:
Revised: For someparameter γ intended to lowerbound the all-layer margins, we deﬁne the loss s (cid:96) γ as follows:

Original: [Equation]
Revised: [Equation]

Original: Note that s (cid:96) γ composes the classical ramp loss, which is used to prove margin-based generalizationcomplexity bounds, with the value of the all-layer margin.
Revised: Note that s (cid:96) γ composes the classical ramp loss, which is used to prove margin-based generalizationcomplexity bounds, with the value of the all-layer margin.

Original: By our construction, it immediately followsthat s (cid:96) γ p θ,x,G p x qqě (cid:96) 0-1 p F p x,θ q ,G p x qq , as is required of a surrogate loss.
Revised: By our construction, it immediately followsthat s (cid:96) γ p θ,x,G p x qqě (cid:96) 0-1 p F p x,θ q ,G p x qq , as is required of a surrogate loss.

Original: We show that to obtain sample complexity bounds for SM-approximation of G in a classiﬁcation setting,it sufﬁces to prove that functions in F can ﬁt labels of G P G with large all-layer margin.
Revised: We show that to obtain sample complexity bounds for SM-approximation of G in a classiﬁcation setting,it sufﬁces to prove that functions in F can ﬁt labels of G P G with large all-layer margin.

Original: Our argumentuses s (cid:96) γ as the loss surrogate in the deﬁnition of SM approximation.
Revised: Our argumentuses s (cid:96) γ as the loss surrogate in the deﬁnition of SM approximation.

Original: Though s (cid:96) γ is computationallyintractable to optimize, Wei and Ma [51] demonstrate that heuristically minimizing s (cid:96) γ also leads toimproved generalization empirically.
Revised: Though s (cid:96) γ is computationallyintractable to optimize, Wei and Ma [56] demonstrate that heuristically minimizing s (cid:96) γ also leads toimproved generalization empirically.

Original: Lemma 2.4.
Revised: Lemma 2.4.

Original: Fix any parameterized function F : X ˆ R p Ñ R , and deﬁne F α ﬁ t x ÞÑ F p x,θ q : θ P Θ u ,where we assume Θ Ď R p is such that } θ } 1 ď α for all θ P Θ .
Revised: Fix any parameterized function F : X ˆ R p Ñ R , and deﬁne F α ﬁ t x ÞÑ F p x,θ q : θ P Θ u ,where we assume Θ Ď R p is such that } θ } 1 ď α for all θ P Θ .

Original: Fix (cid:15) ě 0 .
Revised: Fix (cid:15) ě 0 .

Original: Suppose that for all G P G , thereexists θ P Θ such that the following holds:
Revised: Suppose that for all G P G , thereexists θ P Θ such that the following holds:

Original: [Equation]
Revised: [Equation]

Original: Then, F α (cid:15) -SM-approximates G with respect to p (cid:96) 0-1 ,P q with sample complexity
Revised: Then, F α (cid:15) -SM-approximates G with respect to p (cid:96) 0-1 ,P q with sample complexity

Original: r O
Revised: r O

Original: [Equation]
Revised: [Equation]

Original: Here r O hides poly-logarithmic factors in the arguments, in this case, poly log p
Revised: Here r O hides poly-logarithmic factors in the arguments, in this case, poly log p

Original: α 2 log p p q γ 2 (cid:15) 2 q factors.
Revised: α 2 log p p q γ 2 (cid:15) 2 q factors.

Original: Theproof follows [51] and is deferred to Section A.
Revised: Theproof follows [56] and is deferred to Section A.

Original: In Section A, we also state a generalization boundfor 0-1 loss based on (2.1), which may be of independent interest.
Revised: In Section A, we also state a generalization boundfor 0-1 loss based on (2.1), which may be of independent interest.

Original: We use (2.2) and Lemma 2.4 toprove that neural nets can SM-approximate Boolean circuits and Turing machines.
Revised: We use (2.2) and Lemma 2.4 toprove that neural nets can SM-approximate Boolean circuits and Turing machines.

Original:  SM approximation of Boolean circuits with feedforward nets
Revised:  SM approximation of Boolean circuits with feedforward nets

Original: This section shows that feedforward neural nets can SM-approximate Boolean circuits with samplecomplexity that depends polynomially on the size of the circuit.
Revised: This section shows that feedforward neural nets can SM-approximate Boolean circuits with samplecomplexity that depends polynomially on the size of the circuit.

Original: A boolean circuit G : t 0 , 1 u m Ñt 0 , 1 uon m inputs bits is described by a directed acyclic graph, with vertices of this graph referred to as“gates”.
Revised: A boolean circuit G : t 0 , 1 u m Ñt 0 , 1 uon m inputs bits is described by a directed acyclic graph, with vertices of this graph referred to as“gates”.

Original: The graph contains m input gates of indegree 0, which are identiﬁed with the input bits.
Revised: The graph contains m input gates of indegree 0, which are identiﬁed with the input bits.

Original: The remaining gates each compute a boolean function taking values at their parents as arguments,and a designated output gate produces the output of the entire circuit.
Revised: The remaining gates each compute a boolean function taking values at their parents as arguments,and a designated output gate produces the output of the entire circuit.

Original: We consider boolean circuitsconsisting of AND, OR, and NOT gates, which compute the corresponding boolean functions on 2,2, and 1 inputs, respectively and are sufﬁcient to compute any boolean function [40].
Revised: We consider boolean circuitsconsisting of AND, OR, and NOT gates, which compute the corresponding boolean functions on 2,2, and 1 inputs, respectively and are sufﬁcient to compute any boolean function [45].

Original: We also allowidentity (ID) gates, which take 1 input and output the same value.
Revised: We also allowidentity (ID) gates, which take 1 input and output the same value.

Original: We consider layered circuits, where we can partition the gates into layers such that the only edgesin the graph occur from gates in layer i to gates in layer i ` 1 for some i .
Revised: We consider layered circuits, where we can partition the gates into layers such that the only edgesin the graph occur from gates in layer i to gates in layer i ` 1 for some i .

Original: Note that we can transformany boolean circuit into a layered one by adding ID gates.
Revised: Note that we can transformany boolean circuit into a layered one by adding ID gates.

Original: Letting q denote the number of layers andr the maximum number of gates in any layer, we say that the circuit has depth q and width r .
Revised: Letting q denote the number of layers andr the maximum number of gates in any layer, we say that the circuit has depth q and width r .

Original: We saythat a circuit with s total gates has size s .
Revised: We saythat a circuit with s total gates has size s .

Original: Our convention will be that the set of input gates is considereda layer, so r ě m .
Revised: Our convention will be that the set of input gates is considereda layer, so r ě m .

Original: We consider the following class of boolean circuits:
Revised: We consider the following class of boolean circuits:

Original: [Equation]
Revised: [Equation]

Original: We will approximate G q,r,s using a family of width w , depth d feedforward ReLU nets pa-  rameterized by linear weights and biases θ “ p W 0 , b 0 , ... , W d , b d q computed as follows:
Revised: We will approximate G q,r,s using a family of width w , depth d feedforward ReLU nets pa- 235 rameterized by linear weights and biases θ “ p W 0 , b 0 , ... , W d , b d q computed as follows:

Original: [Equation] , where all intermediate layers 231 have width w for simplicity and φ denotes the coordinate-wise ReLU activation.
Revised: F w,d p x, θ q “ W d φ p W d ´ 1 φ p¨¨¨ φ p W 0 x ` b 0 q ¨¨¨ q ` b d ´ 1 q ` b d , where all intermediate layers 237 have width w for simplicity and φ denotes the coordinate-wise ReLU activation.

Original: The weight parametersare set so that for 1 ď i ď d ´ 1 , W i P R w ˆ w , W 0 P R w ˆ m , and W d P R 1 ˆ w .
Revised: The weight parametersare set so that for 1 ď i ď d ´ 1 , W i P R w ˆ w , W 0 P R w ˆ m , and W d P R 1 ˆ w .

Original: The bias parametersare such that b i P R w for 0 ď i ď d ´ 1 , and b d P R .
Revised: The bias parametersare such that b i P R w for 0 ď i ď d ´ 1 , and b d P R .

Original: To control the sample complexity, we restrict ourattention to parameters with total }¨} 1 -norm bounded by α , giving the following function class:
Revised: To control the sample complexity, we restrict ourattention to parameters with total }¨} 1 -norm bounded by α , giving the following function class:

Original: [Equation]
Revised: [Equation]

Original: The following theorem states that feedforward neural nets can statistically meaningfully approximateboolean circuits with sample complexity polynomial in the circuit size.
Revised: The following theorem states that feedforward neural nets can statistically meaningfully approximateboolean circuits with sample complexity polynomial in the circuit size.

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: Consider the class G q,r,s of size- s ,width- r , and depth- q layered boolean circuits, and 238 the class F w,d,α of neural nets above.
Revised: Consider the class G q,r,s of size- s ,width- r , and depth- q layered boolean circuits, and 244 the class F w,d,α of neural nets above.

Original: Suppose w Á r , α — s , and d — q .
Revised: Suppose w Á r , α — s , and d — q .

Original: Then, for all (cid:15) ą 0 and any input distribution P over t 0 , 1 u m , F w,d,α (cid:15) -SM-approximates G with respectto p (cid:96) 0-1 ,P q with sample complexity poly p s q r O 
Revised: Then, for all (cid:15) ą 0 and any input distribution P over t 0 , 1 u m , F w,d,α (cid:15) -SM-approximates G with respectto p (cid:96) 0-1 ,P q with sample complexity poly p s q r O [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: ¯.
Revised: ¯

Original:  We note that the bound in Theorem 3.1 only scales logarithmically in the width w of the network, evenif w is arbitrarily greater than the circuit width r .
Revised: [Equation] We note that the bound in Theorem 3.1 only scales logarithmically in the width w of the network, evenif w is arbitrarily greater than the circuit width r .

Original: This ensures that even heavily overparameterizednets will have low sample complexity of the approximation.
Revised: This ensures that even heavily overparameterizednets will have low sample complexity of the approximation.

Original: For this setting, the all-layer margin loss in (2.2) is essential for proving tight sample complexitybounds, as other surrogate losses s (cid:96) would give weaker results.
Revised: For this setting, the all-layer margin loss in (2.2) is essential for proving tight sample complexitybounds, as other surrogate losses s (cid:96) would give weaker results.

Original: For example, if we choose (cid:96) 0-1 as thesurrogate loss, VC-dimension bounds [20] imply that F w,d,α statistically meaningfully approximates 247 G q,r,s with sample complexity scaling in poly p wq q under the conditions of Theorem 3.1.
Revised: For example, if we choose (cid:96) 0-1 as thesurrogate loss, VC-dimension bounds [23] imply that F w,d,α statistically meaningfully approximates 253 G q,r,s with sample complexity scaling in poly p wq q under the conditions of Theorem 3.1.

Original: This suffers 248 a polynomial dependence on the overparameterized width w , which is not ideal for realistic settings,where neural nets are often wider than necessary to facilitate optimization.
Revised: This suffers 254 a polynomial dependence on the overparameterized width w , which is not ideal for realistic settings,where neural nets are often wider than necessary to facilitate optimization.

Original: In contrast, our dependenceon w is logarithmic.
Revised: In contrast, our dependenceon w is logarithmic.

Original: Another possible surrogate loss is the output margin-based ramp loss, which canbe used to prove norm-based sample complexities [5].
Revised: Another possible surrogate loss is the output margin-based ramp loss, which canbe used to prove norm-based sample complexities [6].

Original: However, these bounds depend on ś d i “ 1 } W i } op(or related quantities), which would be exponentially large in d for the naive construction in Section 3.1.
Revised: However, these bounds depend on ś di “ 1 } W i } op(or related quantities), which would be exponentially large in d for the naive construction in Section 3.1.

Original: 3.1 Proof sketch for Theorem 3.1 
Revised: 3.1 Proof sketch for Theorem 3.

Original: There are two key steps in the proof.
Revised: There are two key steps in the proof.

Original: First, given any layered circuit G P G , we construct a neural netthat directly simulates G by computing the layers of G one-by-one, which is simple to do by directlyconstructing ReLU and linear layers to simulate the AND, OR, NOT, and ID gates.
Revised: First, given any layered circuit G P G , we construct a neural netthat directly simulates G by computing the layers of G one-by-one, which is simple to do by directlyconstructing ReLU and linear layers to simulate the AND, OR, NOT, and ID gates.

Original: Lemma 3.2.
Revised: Lemma 3.2.

Original: In the setting of Theorem 3.1, let G denote the layered boolean circuit, which we aim tocompute using a neural net.
Revised: In the setting of Theorem 3.1, let G denote the layered boolean circuit, which we aim tocompute using a neural net.

Original: Let g i : t 0 , 1 u r i ´ 1 Ñt 0 , 1 u r i denote function computed between the i ´ 1 -thand
Revised: Let g i : t 0 , 1 u r i ´ 1 Ñt 0 , 1 u r i denote function computed between the i ´ 1 -thand

Original: i -th layers of G , which we assume have r i ´ 1 and r i gates, respectively, so G “ g q ´ 1 ˝¨¨¨˝ g 1 .
Revised: i -th layers of G , which we assume have r i ´ 1 and r i gates, respectively, so G “ g q ´ 1 ˝¨¨¨˝ g 1 .

Original: Then there exist functions f 1 ,...,f q ´ 1 , where each f i is computed by a feedforward ReLU net with two 261 linear and activation layers, such that for all i Pr q ´ 1 s and x Pt 0 , 1 u m , f i ˝¨¨¨˝ f 1 p x q“ g i ˝¨¨¨˝ g 1 p x q .
Revised: Then there exist functions f 1 ,...,f q ´ 1 , where each f i is computed by a feedforward ReLU net with two 267 linear and activation layers, such that for all i Pr q ´ 1 s and x Pt 0 , 1 u m , f i ˝¨¨¨˝ f 1 p x q“ g i ˝¨¨¨˝ g 1 p x q .

Original: Thus, the composition F p¨ ,θ q ﬁ f q ´ 1 ˝¨¨¨˝ f 1 satisﬁes F p x,θ q “ G p x q for all x P t 0 , 1 u m .
Revised: Thus, the composition F p¨ ,θ q ﬁ f q ´ 1 ˝¨¨¨˝ f 1 satisﬁes F p x,θ q “ G p x q for all x P t 0 , 1 u m .

Original: Note that 263 we omitted the dependency of f q ´ 1 ,...,f 1 on parameters θ for simplicity.
Revised: Note that 269 we omitted the dependency of f q ´ 1 ,...,f 1 on parameters θ for simplicity.

Original: Lower bounding all-layer margin.
Revised: Lower bounding all-layer margin.

Original: The next step for proving SM-approximation is to construct aloss s (cid:96) so that the empirical risk minimizer of s (cid:96) on the training data has good sample complexity.
Revised: The next step for proving SM-approximation is to construct aloss s (cid:96) so that the empirical risk minimizer of s (cid:96) on the training data has good sample complexity.

Original: Thiscrucially requires the all-layer margin tool developed in Section 2.1, as other complexity measures(e.g.
Revised: Thiscrucially requires the all-layer margin tool developed in Section 2.1, as other complexity measures(e.g.

Original: norm-based) would not give good sample complexity bounds.
Revised: norm-based) would not give good sample complexity bounds.

Original: Recall that the all-layer margin ρ F p θ, x, G p x qq measures the stability of the output F p x, θ q toperturbations in to θ , and, by Lemma 2.4, it sufﬁces to show that F has large all-layer margin onx P t 0 , 1 u m .
Revised: Recall that the all-layer margin ρ F p θ, x, G p x qq measures the stability of the output F p x, θ q toperturbations in to θ , and, by Lemma 2.4, it sufﬁces to show that F has large all-layer margin onx P t 0 , 1 u m .

Original: Unfortunately, we cannot guarantee that the naive construction from Lemma 3.2 haslarge all-layer margin without further modiﬁcations.
Revised: Unfortunately, we cannot guarantee that the naive construction from Lemma 3.2 haslarge all-layer margin without further modiﬁcations.

Original: To remedy this issue, Theorem D.6 introducesa generic way to convert the model F p¨ ,θ q , with possibly small all-layer margin on x Pt 0 , 1 u m , intoa new architecture and parameter set F 1 p¨ ,θ 1 q , with provably large all-layer margin on x Pt 0 , 1 u m , suchthat F 1 p x,θ
Revised: To remedy this issue, Theorem D.6 introducesa generic way to convert the model F p¨ ,θ q , with possibly small all-layer margin on x Pt 0 , 1 u m , intoa new architecture and parameter set F 1 p¨ ,θ 1 q , with provably large all-layer margin on x Pt 0 , 1 u m , suchthat F 1 p x,θ

Original: 1 q “ F p x,θ q on all inputs x P t 0 , 1 u m .
Revised: 1 q “ F p x,θ q on all inputs x P t 0 , 1 u m .

Original: The construction relies on introducing new layersto F to obtain F 1 and increases the total number of layers by only a constant factor.
Revised: The construction relies on introducing new layersto F to obtain F 1 and increases the total number of layers by only a constant factor.

Original: This step of theproof is formally stated in the following lemma.
Revised: This step of theproof is formally stated in the following lemma.

Original: Lemma 3.3.
Revised: Lemma 3.3.

Original: In the setting of Lemma 3.2, let F p¨ ,θ q“ f q ´ 1 ˝¨¨¨˝ f 1 be the neural net with parameters 278 θ constructed to compute the circuit G .
Revised: In the setting of Lemma 3.2, let F p¨ ,θ q“ f q ´ 1 ˝¨¨¨˝ f 1 be the neural net with parameters 284 θ constructed to compute the circuit G .

Original: There exist “correction functions” ζ 1 ,...,ζ q ´ 2 , where ζ i iscomputed by a neural net with two activation and linear layers, such that the composition F 1 p¨ ,θ 1 q ﬁf q ´ 1 ˝ ζ q ´ 

Revised: There exist “correction functions” ζ 1 ,...,ζ q ´ 2 , where ζ i iscomputed by a neural net with two activation and linear layers, such that the composition F 1 p¨ ,θ 1 q ﬁf q ´ 1 ˝ ζ q ´

Original: ˝ f q ´ 2 ˝¨¨¨˝ ζ 1 ˝ f 1 has large all-layer margin: ρ F 1 p θ 1 ,x,G p x qqě1 poly p s q for all x Pt 0 , 1 u m .
Revised: ˝ f q ´ 2 ˝¨¨¨˝ ζ 1 ˝ f 1 has large all-layer margin: ρ F 1 p θ 1 ,x,G p x qqě

Original: Here θ 1 denotes the collection of all parameters, and dependency of f i ,ζ i on θ 1 is omitted for simplicity.
Revised: [Figure]

Original: We convey the core intuitions for Lemma 3.3 in a simpliﬁed toy setting as follows.
Revised: We convey the core intuitions for Lemma 3.3 in a simpliﬁed toy setting as follows.

Original: Consider thecase where we start with an initial architecture f
Revised: Consider thecase where we start with an initial architecture f

Original: computing f p x, p W 1 ,...,W d qq“ ´ś di “ 1 W i ¯ x ´ 0 .
Revised: computing f p x, p W 1 ,...,W d qq“ ´ś di “ 1 W i ¯ x ´ 0 .

Original: 5 ,where W i P R .
Revised: 5 ,where W i P R .

Original: In this simpliﬁed setting, we consider W i “ 1 @ i .
Revised: In this simpliﬁed setting, we consider W i “ 1 @ i .

Original: For input x “ 1 and target y “ 1 , theall-layer margin is small: ρ f pp 1 ,..., 1 q , 1 , 1 q À 1 ?
Revised: For input x “ 1 and target y “ 1 , theall-layer margin is small: ρ f pp 1 ,..., 1 q , 1 , 1 q À 1 ?

Original: d , where the architecture is in the subscript.
Revised: d , where the architecture is in the subscript.

Original: Indeed, choosing δ i “ 3 d , we have f p 1 , p 1 ´ 3 d ,..., 1 ´ 3 d qq“p 1 ´ 3 d q d ´ 0 .
Revised: Indeed,choosing δ i “ 3 d , we have f p 1 , p 1 ´ 3 d ,..., 1 ´ 3 d qq“p 1 ´ 3 d q d ´ 0 .

Original: 5 « exp p´ 3 q´ 0 .
Revised: 5 « exp p´ 3 q´ 0 .

Original: 5 ă 0 .
Revised: 5 ă 0 .

Original: Thus, by thedeﬁnition of all-layer margin, ρ f pp 1 ,..., 1 q , 1 , 1 qď aři δ 2 i À 1 ?
Revised: Thus, by thedeﬁnition of all-layer margin, ρ f pp 1 ,..., 1 q , 1 , 1 qď ař

Original: d .
Revised: [Equation]

Original: Now we will insert ReLU layers in f to increase the all-layer margin to Ω p 1 q .
Revised: Now we will insert ReLU layers in f to increase the all-layer margin to Ω p 1 q .

Original: We use ReLU layersto implement the round function, which has the key property that round p z q“ 1 @ z ě 2 { 3 .
Revised: We use ReLU layersto implement the round function, which has the key property that round p z q“ 1 @ z ě 2 { 3 .

Original: Proposition3.4.
Revised: Proposition3.4.

Original: Forany z P R , wecanimplementthefunction round p z q“ $& % if z ă 1 {3 x ´if 1 { 3 ď z ă 2 { if z ě 2 { via a feedforward ReLU net, as follows:
Revised: Forany z P R , wecanimplementthefunction round p z q“ $& % if z ă 1 {3 x ´if 1 { 3 ď z ă 2 { if z ě 2 { via a feedforward ReLU net, as follows:

Original: round p z q“ 3 φ p z ´ 1 { 3 q´ 3 φ p z ´ 2 { 3 q .
Revised: round p z q“ 3 φ p z ´ 1 { 3 q´ 3 φ p z ´ 2 { 3 q .

Original: We consider the following function r f , which inserts round between every layer in f :
Revised: We consider the following function r f , which inserts round between every layer in f :

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: For this demonstration, we ignore the parameters of round, though the actual proof considers them.
Revised: For this demonstration, we ignore the parameters of round, though the actual proof considers them.

Original: The following claim shows that (3.1) preserves the output of f while increasing the all-layer margin:
Revised: The following claim shows that (3.1) preserves the output of f while increasing the all-layer margin:

Original: Claim 3.5.
Revised: Claim 3.5.

Original: In the setting above, r f p 1 , p 1 ,..., 1 qq“ f p 1 , p 1 ,..., 1 qq and ρ r f pp 1 ,..., 1 q , 1 , 1 qě 13 .
Revised: In the setting above, r f p 1 , p 1 ,..., 1 qq“ f p 1 , p 1 ,..., 1 qq and ρ r f pp 1 ,..., 1 q , 1 , 1 qě 13 .

Original: This reﬂects a signiﬁcant increase in the all-layer margin, while only increasing depth by a constantfactor.
Revised: This reﬂects a signiﬁcant increase in the all-layer margin, while only increasing depth by a constantfactor.

Original: The proof is simple: we observe that if δ i ď 13 for all i , the function output will not changebecause round p z q“ 1 @ z ě 23 .
Revised: The proof is simple: we observe that if δ i ď 13 for all i , the function output will not changebecause round p z q“ 1 @ z ě 23 .

Original: This immediately gives the all-layer margin lower bound 13 .
Revised: This immediately gives the all-layer margin lower bound 13 .

Original: To apply this construction more generally, we note that round corrects errors in previous layers.
Revised: To apply this construction more generally, we note that round corrects errors in previous layers.

Original: In the more general setting, we insert “correction functions” ζ between each layer satisfying thekey property that ζ p h 1 q “ h if h is the intended output of the layer and h 1 is any perturbed valuesatisfying } h 1 ´ h } 2 ď 13 .
Revised: In the more general setting, we insert “correction functions” ζ between each layer satisfying thekey property that ζ p h 1 q “ h if h is the intended output of the layer and h 1 is any perturbed valuesatisfying } h 1 ´ h } 2 ď 13 .

Original: Since intended outputs of layers in the function constructed by Lemma 3.2are binary-valued in t 0 , 1 u w because F simulates a boolean circuit, we can simply apply the functionround constructed in Proposition 3.4 elementwise as the correction function.
Revised: Since intended outputs of layers in the function constructed by Lemma 3.2are binary-valued in t 0 , 1 u w because F simulates a boolean circuit, we can simply apply the functionround constructed in Proposition 3.4 elementwise as the correction function.

Original: By the construction,this can be implemented by adding two additional feedforward ReLU layers per correction function.
Revised: By the construction,this can be implemented by adding two additional feedforward ReLU layers per correction function.

Original: Following the intuition for Claim 3.5, we prove that inserting these correction functions guaranteesa large all-layer margin (Theorem D.6) on all x Pt 0 , 1 u m .
Revised: Following the intuition for Claim 3.5, we prove that inserting these correction functions guaranteesa large all-layer margin (Theorem D.6) on all x Pt 0 , 1 u m .

Original: This leads to the proof of Lemma 3.3.
Revised: This leads to the proof of Lemma 3.3.

Original: Wecan complete the proof of Theorem 3.1 by invoking Lemma 2.4, as shown in Section B.
Revised: Wecan complete the proof of Theorem 3.1 by invoking Lemma 2.4, as shown in Section B.

Original:  SM approximation of Turing machines with transformers
Revised:  SM approximation of Turing machines with transformers

Original: In this section, we show that transformers SM-approximate Turing machines with computation timebounded by T , using sample complexity polynomial in log p T q and the state space and alphabet sizes oftheTuringmachine.
Revised: In this section, we show that transformers SM-approximate Turing machines with computation timebounded by T , using sample complexity polynomial in log p T q and the state space and alphabet sizes oftheTuringmachine.

Original: Constructionsfrompriorworkwouldrequiretheapproximationsamplecomplexityto be linear in T [43, 11, 37, 8].
Revised: Constructionsfrompriorworkwouldrequiretheapproximationsamplecomplexityto be linear in T [48, 12, 42, 9].

Original: Thus, we obtain an exponential improvement in the dependency on T .
Revised: Thus, we obtain an exponential improvement in the dependency on T .

Original: We brieﬂy describe a Turing machine; see [44] for a more thorough survey.
Revised: We brieﬂy describe a Turing machine; see [49] for a more thorough survey.

Original: A Turing machine is amodel for computation speciﬁed by a tuple p Z , A ,S, Z term q containing a set of states Z , a tape alphabet
Revised: A Turing machine is amodel for computation speciﬁed by a tuple p Z , A ,S, Z term q containing a set of states Z , a tape alphabet

Original: A , a transition function S : Z ˆ A Ñ Z
Revised: A , a transition function S : Z ˆ A Ñ Z

Original: ˆ A ˆt´ 1 , ` 1 u , and set of terminal states
Revised: ˆ A ˆt´ 1 , ` 1 u , and set of terminal states

Original: Z term indicatingaccept or reject.
Revised: Z term indicatingaccept or reject.

Original: For simplicity, we assume the Turing machine has a single tape, as any single-tape
Revised: For simplicity, we assume the Turing machine has a single tape, as any single-tape

Original: Turing machine can simulate a multi-tape one with only quadratic increase in runtime [44].
Revised: Turing machine can simulate a multi-tape one with only quadratic increase in runtime [49].

Original: Givenan input x P
Revised: Givenan input x P

Original: A ˚ recorded on the left-most part of the tape, the Turing machine performs computationin a sequence of timesteps.
Revised: A ˚ recorded on the left-most part of the tape, the Turing machine performs computationin a sequence of timesteps.

Original: In each timestep, the machine determines the next state, symbol to write,and direction to move the head via the transition function.
Revised: In each timestep, the machine determines the next state, symbol to write,and direction to move the head via the transition function.

Original: We let TM p Z , A ,S, Z term q denote the function computed by the Turing machine, which produces anoutput in t 0 , 1 u (if the machine halts).
Revised: We let TM p Z , A ,S, Z term q denote the function computed by the Turing machine, which produces anoutput in t 0 , 1 u (if the machine halts).

Original: Fixing the alphabet A , we consider the class of binary functionscomputed by Turing machines with at most k states terminating in T steps:
Revised: Fixing the alphabet A , we consider the class of binary functionscomputed by Turing machines with at most k states terminating in T steps:

Original: G k,T ﬁ t x ÞÑ TM p Z , A ,S, Z term q p x q : | Z |ď k, and @ x P X ,
Revised: G k,T ﬁ t x ÞÑ TM p Z , A ,S, Z term q p x q : | Z |ď k, and @ x P X ,

Original: TM p Z , A ,S, Z term q terminates in T steps u (4.1)
Revised: TM p Z , A ,S, Z term q terminates in T steps u (4.1)

Revised: Note that we can assume the input sequences x also have length at most T , as this is the maximum computation time of the Turing machine and the maximum amount of symbols the Turing machine can read.

Original: 4.1 Transformer architecture for SM-approximating Turing machines
Revised: 4.1 Transformer architecture for SM-approximating Turing machines

Original: We study approximation of G with a family of architectures consisting of both an encoder and decodercomponent [48], described as follows.
Revised: We study approximation of G with a family of architectures consisting of both an encoder and decodercomponent [53], described as follows.

Original: The encoder architecture is simple and only performs an embedding of the input symbols, using learnable symbol embeddings E P R w ˆ| A | and ﬁxed positional encodings β p 1 q ,β p 2 q ,...
Revised: The encoder architecture is simple and only performs an embedding of the input symbols, using learnable symbol embeddings E P R w ˆ| A | and ﬁxed positional encodings β p 1 q ,β p 2 q ,...

Original: P R w .
Revised: P R w .

Original: Given input x P
Revised: Given input x P

Original: A ˚ with m symbols, the encoder produces m output vectorsin R w via Enc i p x,E q“ E : ,x i ` β p i q , where Enc i denotes the output of the encoder at the i -th position.
Revised: A ˚ with m symbols, the encoder produces m output vectorsin R w via Enc i p x,E q“ E : ,x i ` β p i q , where Enc i denotes the output of the encoder at the i -th position.

Original: The decoder iteratively computes an output, running for T steps.
Revised: The decoder iteratively computes an output, running for T steps.

Original: We deﬁne a transformer layer of thedecoder as a sequence of modules consisting of decoder self-attention, followed by encoder-decoderattention, followed by three feedforward ReLU layers.
Revised: We deﬁne a transformer layer of thedecoder as a sequence of modules consisting of decoder self-attention, followed by encoder-decoderattention, followed by three feedforward ReLU layers.

Original: Attention layers.
Revised: Attention layers.

Original: Attention layers consist of key, value, and query functions K,V,Q , each, computinga linear transformation.
Revised: Attention layers consist of key, value, and query functions K,V,Q , each, computinga linear transformation.

Original: We omit parameters here for simplicity.
Revised: We omit parameters here for simplicity.

Original: For a single decoder timestep, theattention layer takes two types of inputs: a sequence of previously-computed representations h 1 ,...,h i ,and a current input representation h 1 .
Revised: For a single decoder timestep, theattention layer takes two types of inputs: a sequence of previously-computed representations h 1 ,...,h i ,and a current input representation h 1 .

Original: The layer applies the key, value, and query functions as follows:
Revised: The layer applies the key, value, and query functions as follows:

Original: [Equation]
Revised: [Equation]

Original: where K 0 and V 0 are ﬁxed “null” key and value vectors which are learned parameters of the layer.
Revised: where K 0 and V 0 are ﬁxed “null” key and value vectors which are learned parameters of the layer.

Original: Letting J denote the set of indices t j : τ j “ max t τ 0 ,...,τ i uu , the attention layer performs hard-maxattention [37] to compute the output, as follows:
Revised: Letting J denote the set of indices t j : τ j “ max t τ 0 ,...,τ i uu , the attention layer performs hard-maxattention [42] to compute the output, as follows:

Original: Attnp h 1 , p
Revised: Attn

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: ,h i qq“ h 1 ` 1 | J | ř j P J v j .

Original: Our theory also applies to the standard softmax attention used in practice, but we focus on thehard-max case for a simpler proof.
Revised: Our theory also applies to the standard softmax attention used in practice, but we focus on thehard-max case for a simpler proof.

Original: Let h p
Revised: Let h p

Original: j q t denote the representation computed by the j -th layerof the decoder at timestep t .
Revised: j q t denote the representation computed by the j -th layerof the decoder at timestep t .

Original: At timestep i , decoder self-attention at the p j ` 1 q -th layer computes
Revised: At timestep i , decoder self-attention at the p j ` 1 q -th layer computes

Original: Attn p h p
Revised: Attn p h p

Original: j q i , p h p
Revised: j q i , p h p

Original: j q 1 ,...,h p
Revised: j q 1 ,...,h p

Original: j q i qq .
Revised: j q i qq .

Original: Letting e 1 ,...,
Revised: Letting e 1 ,...,

Original: e m denote the encoder outputs, encoder-decoder self-attentionat the p j ` 1 q -th layer
Revised: e m denote the encoder outputs, encoder-decoder self-attentionat the p j ` 1 q -th layer

Original: and i -th step would compute Attn p
Revised: and i -th step would compute Attn p

Original: h p
Revised: h p

Original: j q i , p e 1 ,...,e m qq .
Revised: j q i , p e 1 ,...,e m qq .

Original: Transformer layers.
Revised: Transformer layers.

Original: We use feedforward layers which apply 3 standard ReLU layers, as follows:
Revised: We use feedforward layers which apply 3 standard ReLU layers, as follows:

Original: [Equation]
Revised: [Equation]

Original:  q .
Revised: 3 q .

Original: Our theory also allows for residual feedforward layers, 348 and the architecture here is chosen mainly to simplify the construction.
Revised: Our theory also allows for residual feedforward layers, 356 and the architecture here is chosen mainly to simplify the construction.

Original: A transformer layer applies these constructions in sequence.
Revised: A transformer layer applies these constructions in sequence.

Original: Letting H p
Revised: Letting H p

Original: j q i “ [Equation] q
Revised: j q i “ p h p

Original: [Equation]

Original: ...,h p j q i q denotethe output after the j -th transformer layer for timesteps 1 ď t ď i , and θ p j q the parameters, we compute
Revised: j q 1 ,...,h p j q i q denotethe output after the j -th transformer layer for timesteps 1 ď t ď i , and θ p j q the parameters, we compute

Original: [Equation]
Revised: [Equation]

Revised: q ,θ p j+1,ff q q

Original: Note that we included the explicit dependence of the attention layers on the parameters forcompleteness.
Revised: Note that we included the explicit dependence of the attention layers on the parameters forcompleteness.

Original: We now set h p j ` 1 q i 
Revised: We now set h p j ` 1 q i “ Tr

Original: [Equation]
Revised: p h p j q i ,H p j q i , p e 1 ,..., e m q ,θ p j ` 1 q q .

Original: Decoderoutputs.
Revised: Decoderoutputs.

Original: We consider d -layerdecoders, so
Revised: We consider d -layerdecoders, so

Original: o i
Revised: o i

Original: ﬁ h p
Revised: ﬁ h p

Original: d q i denotestheoutput
Revised: d q i denotestheoutput

Original: ofthedecoderattime i ,which is also inputted to the decoder at time i ` 1 as follows: h p 0 q i ` 1 “ h p d q i ` β p i ` 1 q .
Revised: ofthedecoderattime i ,which is also inputted to the decoder at time i ` 1 as follows: h p 0 q i ` 1 “ h p d q i ` β p i ` 1 q .

Original: The initial decoderinput h p 0 q 0 is a trainable parameter.
Revised: The initial decoderinput h p 0 q 0 is a trainable parameter.

Original: The decoder runs for a ﬁxed number of timesteps T 1 and outputs prediction θ J cls
Revised: The decoder runs for a ﬁxed number of timesteps T 1 and outputs prediction θ J cls

Original: h p d q T 1 .
Revised: h p d q T 1 .

Original: For simplicity, we assume T 1 “ T , the computation time of the Turing machine family.
Revised: For simplicity, we assume T 1 “ T , the computation time of the Turing machine family.

Original: Note that our architecture allows long (length T ) decoding sequences, whereas typical architecturesin practice use decoding sequences with roughly the same length as the input [48].
Revised: Note that our architecture allows long (length T ) decoding sequences, whereas typical architecturesin practice use decoding sequences with roughly the same length as the input [53].

Original: The architecturewe study is similar to ones studied by [37, 8].
Revised: The architecturewe study is similar to ones studied by [42, 9].

Original: We use x ÞÑ F w,d,T p x, θ q to denote the described transformer architecture with parameters θ , 361 w -dimensional hidden layers, d transformer layers in the decoder, and T decoder steps.
Revised: We use x ÞÑ F w,d,T p x, θ q to denote the described transformer architecture with parameters θ , 369 w -dimensional hidden layers, d transformer layers in the decoder, and T decoder steps.

Original: This leads tothe following class of transformer functions: F w,d,α,T “t x ÞÑ F w,d,T p x,θ q : } θ } 1 ď α u .
Revised: This leads tothe following class of transformer functions: F w,d,α,T “t x ÞÑ F w,d,T p x,θ q : } θ } 1 ď α u .

Original: The following 363 theorem states that this class of transformers SM-approximates the Turing machine family G deﬁnedin (4.1) with sample complexity polynomial in log T , k and | A | .
Revised: The following 371 theorem states that this class of transformers SM-approximates the Turing machine family G deﬁnedin (4.1) with sample complexity polynomial in log T , k and | A | .

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: In the setting above, consider the class G of functions computed by Turing machineswith at most k states, alphabet A , and computation time bounded by T steps for inputs x P X .
Revised: In the setting above, consider the class G of functions computed by Turing machineswith at most k states, alphabet A , and computation time bounded by T steps for inputs x P X .

Original: Supposethat w Á k | A |` log T , d — log T , and α “ poly p k,| A | , log T q .
Revised: Supposethat w Á k | A |` log T , d — log T , and α “ poly p k,

Original: Then, for all (cid:15) ą 0 and any input distribution P over X , F w,d,α,T (cid:15) -SM-approximates G with respectto p (cid:96) 0-1 ,P q with sample complexity poly p k, | A | , log T q r O ´ log p wd q (cid:15)¯ .
Revised: Then, for all (cid:15) ą 0 and any input distribution P over X , F w,d,α,T (cid:15) -SM-approximates G with respectto p (cid:96) 0-1 ,P q with sample complexity poly p k, | A | , log T q r O ´ log p wd q (cid:15)¯ .

Original: As with Section 3, we set the surrogate loss s (cid:96) in Deﬁnition 2.2 to be the all-layer margin loss deﬁnedin Section 2.1.
Revised: As with Section 3, we set the surrogate loss s (cid:96) in Deﬁnition 2.2 to be the all-layer margin loss deﬁnedin Section 2.1.

Original: Commonly-used alternatives for the surrogate loss would not sufﬁce for either ourconstruction or ones in prior work [43, 11, 37, 8].
Revised: Commonly-used alternatives for the surrogate loss would not sufﬁce for either ourconstruction or ones in prior work [48, 12, 42, 9].

Original: First, the VC dimension of F w,d,α,T is at least  Ω p wT q .
Revised: First, the VC dimension of F w,d,α,T is at least 381 Ω p wT q .

Original: This is because transformer architectures which contain a decoder component can express
Revised: This is because transformer architectures which contain a decoder component can express

Original: RNNs, which by lower bounds have VC dimension at least wT [24].
Revised: RNNs, which by lower bounds have VC dimension at least wT [28].

Original: This indicates that using (cid:96) 0-1 asthe surrogate loss would lead to sample complexities that are suboptimal in both the overparameterizedwidth w and the computation T .
Revised: This indicates that using (cid:96) 0-1 asthe surrogate loss would lead to sample complexities that are suboptimal in both the overparameterizedwidth w and the computation T .

Original: Second, the correct norm-based Rademacher complexity bound touse for transformers is unclear; however, the RNN-based equivalent would scale with the T -th powerof some parameter norm, or exponentially in T .
Revised: Second, the correct norm-based Rademacher complexity bound touse for transformers is unclear; however, the RNN-based equivalent would scale with the T -th powerof some parameter norm, or exponentially in T .

Original: Thus, as in Section 3, the all-layer margin surrogateloss (2.2) is essential for obtaining our sample complexity bounds.
Revised: Thus, as in Section 3, the all-layer margin surrogateloss (2.2) is essential for obtaining our sample complexity bounds.

Original: 4.2 Proof sketch for Theorem 4.
Revised: 4.2 Proof sketch for Theorem 4.1

Original: Following Lemma 2.4, our goal is to construct a transformer which can simulate Turing machines withlarge all-layer margin, namely, Ω 
Revised: Following Lemma 2.4, our goal is to construct a transformer which can simulate Turing machines withlarge all-layer margin, namely, Ω ´ [Equation] p k, | A |

Original: [Equation]
Revised: [Equation]

Original:  ¯ .
Revised: log T q ¯ .

Original: The fundamental limitation of prior work [37]towards attaining this is that the positional embeddings are required to store values as small as
Revised: The fundamental limitation of prior work [42]towards attaining this is that the positional embeddings are required to store values as small as

Original: [Equation]
Revised: [Equation]

Original: Our construction cannot afford to rely on values this small – informally, if the construction relieson the exact values of these small entries, then the all layer margin would be at most 1 poly p T q becauseperturbing the layer by the small entries could change the prediction.
Revised: Our construction cannot afford to rely on values this small – informally, if the construction relieson the exact values of these small entries, then the all layer margin would be at most 1 poly p T q becauseperturbing the layer by the small entries could change the prediction.

Original: Instead, we propose using Bin p i q ,the binary encoding of i in r log T s bits, as the positional encoding for timestep i .
Revised: Instead, we propose using Bin p i q ,the binary encoding of i in r log T s bits, as the positional encoding for timestep i .

Original: This allows us touse unique positional encodings for each timestep which do not rely on arbitrary precision.
Revised: This allows us touse unique positional encodings for each timestep which do not rely on arbitrary precision.

Original: We describe the construction.
Revised: We describe the construction.

Original: Fix a Turing machine G P G .
Revised: Fix a Turing machine G P G .

Original: We ﬁrst require notation to describe thecomputation of G .
Revised: We ﬁrst require notation to describe thecomputation of G .

Original: For input x P X , let z i p x q , a i p x q denote the Turing machine state and symbol underthe tape head at the end of step i .
Revised: For input x P X , let z i p x q , a i p x q denote the Turing machine state and symbol underthe tape head at the end of step i .

Original: We let l i p x q denote the location of the Turing machine head at theconclusion of step i .
Revised: We let l i p x q denote the location of the Turing machine head at theconclusion of step i .

Original: During the timestep, the Turing machine computes 
Revised: During the timestep, the Turing machine computes S p

Revised: z i ´ 1 p x q ,a

Revised: i ´ 1 p x qq , writes a 401 new symbol under the head at location l

Revised: i ´ 1 p x q , and moves the head either left or right.

Revised: Let u i p x q denote 402 the symbol written during timestep i , and q i p x qPt left , right u the movement direction of the head.

Original: Following [37] with several key modiﬁcations, we simulate the Turing machine using the transformeras follows.
Revised: Following [42] with several key modiﬁcations, we simulate the Turing machine using the transformeras follows.

Original: Each timestep will maintain the invariance that
Revised: Each timestep will maintain the invariance that

Original: o i contains an encoding of z i p x q ,a i p x q ,and l i p x q .
Revised: o i contains an encoding of z i p x q ,a i p x q ,and l i p x q .

Original: Given that this invariance holds until timestep i , the transformer simulates timestep i ` 1of the Turing machine with the following steps:
Revised: Given that this invariance holds until timestep i , the transformer simulates timestep i ` 1of the Turing machine with the following steps:

Original: 1) Use feedforward layers to apply transition S on z i p x q and a i p x q , which can be read fromo i , to obtain z i ` 1 p x q , u i ` 1 p x q , and movement direction q i ` 1 p x qPt left, right u .2)
Revised: 1) Use feedforward layers to apply transition S on z i p x q and a i p x q , which can be read fromo i , to obtain z i ` 1 p x q , u i ` 1 p x q , and movement direction q i ` 1 p x qPt left, right u .2)

Original: Using feedforward layers, compute l i ` 1 p x q from q i ` 1 p x q and the encoding of l i p x q in o i .3) Compute a
Revised: Using feedforward layers, compute l i ` 1 p x q from q i ` 1 p x q and the encoding of l i p x q in o i . 3) Compute a

Original: i ` 1 p x q .
Revised: i ` 1 p x q .

Original: We use decoder self-attention to search over past timesteps which wroteto l i ` 1 p x q .
Revised: We use decoder self-attention to search over past timesteps which wroteto l i ` 1 p x q .

Original: Our aim is to ﬁnd u i 1 p x q , where i 1 “ max t j ď i ` 1 : l j ´ 1 p x q “ l i ` 1 p x qu .We
Revised: Our aim is to ﬁnd u i 1 p x q , where i 1 “ max t j ď i ` 1 : l j ´ 1 p x q “ l i ` 1 p x qu .

Original: implement a binary search over past timesteps j , which is needed to ﬁnd the largest j ď i ` where l j ´ 1 p x q “ l i ` 1 p x q 
Revised: Weimplement a binary search over past timesteps j , which is needed to ﬁnd the largest j ď i ` 1where l j ´ 1 p x q “ l i ` 1 p x q .

Original: The binary search  can be implemented with O p r log T s q decoder self-attention layers and the 
Revised: The binary search is performed over the bits of i 1 and can beimplemented with O p r log T s q decoder self-attention layers, and the construction ensureslarge all-layer margin.

Original: construction ensures large all-layer margin.

Original: If no such i 1 from the previous timestep existed, we check whether l i ` 1 p x q contained an inputsymbol using encoder-decoder attention and copy this input symbol if so.
Revised: If no such i 1 from the previous timestep existed, we check whether l i ` 1 p x q contained an input symbol using encoder-decoder attention and copy this input symbol if so.

Original: 5) If no symbols were found in 3) or 4), l i ` 1 p x q must contain the blank symbol (meaningit wasn’t visited yet by the head).
Revised: 5) If no symbols were found in 3) or 4), l i ` 1 p x q must contain the blank symbol (meaning it wasn’t visited yet by the head).

Original: Thus, we have computed a i ` 1 p x q , so we have all theinformation needed to compute the new embedding o i `1 .
Revised: Thus, we have computed a i ` 1 p x q , so we have all the information needed to compute the new embedding o i `

Original: To lower bound the all-layer margin of the constructed transformer, we use Theorem D.6, whichrequires existence of a “correction function” which can correct outputs in previous layers.
Revised: To lower bound the all-layer margin of the constructed transformer, we use Theorem D.6, whichrequires existence of a “correction function” which can correct outputs in previous layers.

Original: Since weconstruct a network with intermediate layer entries in t 0 , 1 u , we can use the same correction functionas Section 3.1, which rounds to the nearest bit.
Revised: Since weconstruct a network with intermediate layer entries in t 0 , 1 u , we can use the same correction functionas Section 3.1, which rounds to the nearest bit.

Original: The full proof is provided in Section C.
Revised: The full proof is provided in Section C.

Original:  Conclusion
Revised:  Conclusion

Original: This work proposes a new deﬁnition of approximation, statistically meaningful approximation, whichensures that the approximating family not only has sufﬁcient expressivity, but also exhibits goodstatistical learnability.
Revised: This work proposes a new deﬁnition of approximation, statistically meaningful approximation, whichensures that the approximating family not only has sufﬁcient expressivity, but also exhibits goodstatistical learnability.

Original: Towards a ﬁrst analysis with this deﬁnition, we show approximability of twofunction classes: boolean circuits and Turing machines, with strong sample complexity guaranteesdepending only on the intrinsic properties of these function classes.
Revised: Towards a ﬁrst analysis with this deﬁnition, we show approximability of twofunction classes: boolean circuits and Turing machines, with strong sample complexity guaranteesdepending only on the intrinsic properties of these function classes.

Original: There are several interestingdirections to extend our study of statistically meaningful approximation.
Revised: There are several interestingdirections to extend our study of statistically meaningful approximation.

Original: Examples include provingmore upper and lower bounds for statistically meaningful approximation for different target functionsand neural net architectures, and using our deﬁnition as a lens to compare architectures. 
Revised: Examples include provingmore upper and lower bounds for statistically meaningful approximation for different target functionsand neural net architectures, and using our deﬁnition as a lens to compare architectures. Limitations

Revised: One potential limitation is that the “correction function” machinery discussed in Lemma 3.3 relies onthe discrete nature of boolean circuits and Turing machines, and so additional work and insight wouldbe required to prove analogous SM-approximation results for continuous functions.

Revised: One importantproperty of discrete functions, which we suspect may be leveraged more generally, is that it is easy tocorrect errors in intermediate computations of discrete functions (by rounding).

Revised: It would be interestingto see whether this property has a continuous analog which can be analyzed.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for informationon how to answer these questions.
Revised: Please read the checklist guidelines carefully for informationon how to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .

Original: You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Section ?? .
Revised: [Yes] See Section ?? .

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [No] The code and the data are proprietary.
Revised: [No] The code and the data areproprietary.

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data and instructions needed to reproduce the main
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main

Original: experimental results (either in the supplemental material or as a URL)?
Revised: experimental results (either in the supplemental material or as a URL)?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they werechosen)?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experimentsmultiple times)?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [N/A]
Revised: [N/A]4.

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁableinformation or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review Board
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review Board

Original: (IRB) approvals, if applicable?
Revised: (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A](c)

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?

Original: [N/A]
Revised: [N/A]