Original: RCNNs Learn Succinct Learning Algorithms in Polynomial Time
Revised: RCNNs Learn Succinct Learning Algorithms in Polynomial Time

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Neural Networks (NNs) struggle to efﬁciently learn certain problems, such as parityproblems, even when there are simple learning algorithms for those problems.
Revised: Neural Networks (NNs) struggle to efﬁciently learn certain problems, such as parityproblems, even when there are simple learning algorithms for those problems.

Original: Can
Revised: Can

Original: NNs discover learning algorithms on their own?
Revised: NNs discover learning algorithms on their own?

Original: We exhibit a NN architecture that,in polynomial time, learns as well as any efﬁcient learning algorithm describableby a constant-sized learning algorithm.
Revised: We exhibit a NN architecture that,in polynomial time, learns as well as any efﬁcient learning algorithm describableby a constant-sized learning algorithm.

Original: For example, on parity problems, the
Revised: For example, on parity problems, the

Original: NN learns as well as row reduction, an efﬁcient algorithm that can be succinctlydescribed.
Revised: NN learns as well as row reduction, an efﬁcient algorithm that can be succinctlydescribed.

Original: Our architecture combines both recurrent weight-sharing between layersand convolutional weight-sharing to reduce the number of parameters down toa constant, even though the network itself may have trillions of nodes.
Revised: Our architecture combines both recurrent weight-sharing between layersand convolutional weight-sharing to reduce the number of parameters down toa constant, even though the network itself may have trillions of nodes.

Original: While inpractice the constants in our analysis are too large to be directly meaningful, ourwork suggests that the synergy of Recurrent and Convolutional NNs (RCNNs) maybe more powerful than either alone.
Revised: While inpractice the constants in our analysis are too large to be directly meaningful, ourwork suggests that the synergy of Recurrent and Convolutional NNs (RCNNs) maybe more powerful than either alone.

Original:  Introduction
Revised:  Introduction

Original: Neural Networks (NNs) can seem magical in what they can learn.
Revised: Neural Networks (NNs) can seem magical in what they can learn.

Original: Yet humans have designed simplelearning algorithms, even for binary classiﬁcation, which they cannot match.
Revised: Yet humans have designed simplelearning algorithms, even for binary classiﬁcation, which they cannot match.

Original: A well-known exampleis the class of parity functions over the d -dimensional hypercube, i.e., d -bit strings.
Revised: A well-known exampleis the class of parity functions over the d -dimensional hypercube, i.e., d -bit strings.

Original: In that problem,there is an unknown subset S of the d bits, and the label of each example x is 1 if x has an oddnumber of 1’s in S .
Revised: In that problem,there is an unknown subset S of the d bits, and the label of each example x is 1 if x has an oddnumber of 1’s in S .

Original: While gradient-based learning struggles to learn parity functions [25] even overuniformly random x , row reduction (i.e. Gaussian elimination) can be used to ﬁnd S using only O ( d )examples and O ( d 2 ) runtime.
Revised: While gradient-based learning struggles to learn parity functions [25] even overuniformly random x , row reduction (i.e. Gaussian elimination) can be used to ﬁnd S using only O ( d ) examples and O ( d 2 ) runtime.

Original: A tantalizing question is whether a NN can discover an efﬁcient learning algorithm itself , therebylearning classes such as parities.
Revised: A tantalizing question is whether a NN can discover an efﬁcient learning algorithm itself , therebylearning classes such as parities.

Original: We refer to this as Turing optimality , since algorithms can bedescribed by Turing machines.
Revised: We refer to this as Turing optimality , since algorithms can bedescribed by Turing machines.

Original: More speciﬁcally, we will give an example of a simple NN architecturethat achieves Turing optimality.
Revised: More speciﬁcally, we will give an example of a simple NN architecturethat achieves Turing optimality.

Original: In particular, this is the ﬁrst NN architecture that provably discovers aefﬁcient parity learning algorithm in polynomial time.
Revised: In particular, this is the ﬁrst NN architecture that provably discovers aefﬁcient parity learning algorithm in polynomial time.

Original: The parity learning algorithm is efﬁcient, likerow reduction, requiring O ( d ) examples and O ( d 2 ) runtime.
Revised: The parity learning algorithm is efﬁcient, likerow reduction, requiring O ( d ) examples and O ( d 2 ) runtime.

Original: Our learning architecture would be quitesimple to describe with a modern library such as pytorch.
Revised: Our learning architecture would be quitesimple to describe with a modern library such as pytorch.

Original: However, we do not expect our speciﬁcarchitecture to be especially good in practice, as the constants in our analysis are much too large tobe practical.
Revised: However, we do not expect our speciﬁcarchitecture to be especially good in practice, as the constants in our analysis are much too large tobe practical.

Original: Nonetheless, it does suggest that the ingredients used in the architecture, especiallythe combination of recurrent weight-sharing across layers and convolutional weight-sharing withinlayers, may be useful in designing practical architectures for NNs to learn algorithms.
Revised: Nonetheless, it does suggest that the ingredients used in the architecture, especiallythe combination of recurrent weight-sharing across layers and convolutional weight-sharing withinlayers, may be useful in designing practical architectures for NNs to learn algorithms.

Original: Figure 2 in the Appendix illustrates the difference between classiﬁers , such as NNs, and the learningalgorithms that learn their parameters, such as Stochastic Gradient Descent (SGD) with a givenarchitecture (we use the term architecture broadly to include other algorithmic features includinglearning random initialization, learning rate schedule, restarts, and hyperparameter search).
Revised: Figure 2 in the Appendix illustrates the difference between classiﬁers , such as NNs, and the learningalgorithms that learn their parameters, such as Stochastic Gradient Descent (SGD) with a givenarchitecture (we use the term architecture broadly to include other algorithmic features includinglearning random initialization, learning rate schedule, restarts, and hyperparameter search).

Original: Asclassiﬁers , two-layer NNs can compute any Boolean function on d binary inputs, including parity functions.
Revised: Asclassiﬁers , two-layer NNs can compute any Boolean function on d binary inputs, including parity functions.

Original: However, it is unclear whether these architectures can learn such functions efﬁcientlyusing gradient-based approaches without any priors encoded in the architectures.
Revised: However, it is unclear whether these architectures can learn such functions efﬁcientlyusing gradient-based approaches without any priors encoded in the architectures.

Original: More formally, a Turing-optimal learning algorithm is one which learns as well as any boundedlearning algorithm, speciﬁcally a constant-sized Turing machine that outputs a binary classiﬁer inpolynomial time, such as row reduction for parity learning.
Revised: More formally, a Turing-optimal learning algorithm is one which learns as well as any boundedlearning algorithm, speciﬁcally a constant-sized Turing machine that outputs a binary classiﬁer inpolynomial time, such as row reduction for parity learning.

Original: Our key innovation is a simple recurrentconvolutional (RCNN) architecture which combines recurrent weight-sharing across layers andconvolutional weight-sharing within each layer.
Revised: Our key contribution is a simple recurrentconvolutional (RCNN) architecture which combines recurrent weight-sharing across layers andconvolutional weight-sharing within each layer.

Original: The number of weights in the convolutional ﬁlter canbe very few, even a constant, but these weights can determine the activations of a very wide and deepnetwork.
Revised: The number of weights in the convolutional ﬁlter canbe very few, even a constant, but these weights can determine the activations of a very wide and deepnetwork.

Original: We show that any algorithm A represented by a constant-sized TM has a correspondingconstant-sized convolutional ﬁlter for which the RCNN computes the same function as A .
Revised: We show that any algorithm A represented by a constant-sized TM has a correspondingconstant-sized convolutional ﬁlter for which the RCNN computes the same function as A .

Original: Becausethe convolutional ﬁlter is constant-sized, with constant probability random initialization will ﬁnd it(or
Revised: Becausethe convolutional ﬁlter is constant-sized, with constant probability random initialization will ﬁnd it(or

Original: something even better, assuming our reduction is not optimal).
Revised: something even better, assuming our reduction is not optimal).

Original: Thus, using a validation set andrandom restarts, the RCNN will ﬁnd a ﬁlter which performs as well as A , with high probability.
Revised: Thus, using a validation set andrandom restarts, the RCNN will ﬁnd a ﬁlter which performs as well as A , with high probability.

Original: Unfortunately, the above argument would apply to an RCNN architecture that strangely takes theentire training set as input at once, and outputs a classiﬁer.
Revised: Unfortunately, the above argument would apply to an RCNN architecture that strangely takes theentire training set as input at once, and outputs a classiﬁer.

Original: Fortunately, Abbe and Sandon [2] showhow to use ause a few additional simple NN components and SGD updates to memorize relevantinformation in the weights of these components.
Revised: Fortunately, Abbe and Sandon [2]show how to use a few additional simple NN components and SGD updates to memorize relevantinformation in the weights of these components.

Original: Similarly, we add a few extra non-convolutionallayers to our architecture so that it can be learned “normally” with SGD rather than requiring theentire dataset at once.
Revised: Similarly, we add a few extra non-convolutionallayers to our architecture so that it can be learned “normally” with SGD rather than requiring theentire dataset at once.

Original: Fortunately, we can implement this functionality using the RCNN with only aconstant overhead in terms of size.
Revised: Fortunately, we can implement this functionality using the RCNN with only aconstant overhead in terms of size.

Original: Probably Algorithmically Optimal (PAO) learning.
Revised: Probably Algorithmically Optimal (PAO) learning.

Original: To deﬁne Turing-optimality, it is convenientto formalize a weaker requirement than PAC learning which we call PAO learning, that in some senseturns PAC learning on its head.
Revised: To deﬁne Turing-optimality, it is convenientto formalize a weaker requirement than PAC learning which we call PAO learning, that in some senseturns PAC learning on its head.

Original: Rather than requiring optimality among the space of all classiﬁersc ∈ C , it requires only optimality compared to classiﬁers output by learning algorithms A ∈ A .
Revised: Rather than requiring optimality among the space of all classiﬁersc ∈ C , it requires only optimality compared to classiﬁers output by learning algorithms A ∈ A .

Original: If|A| (cid:28) |C| , then matching the performance of the classiﬁer output by the best learning algorithm(within some family) may be more reasonable matching the performance of the best overall classiﬁer.
Revised: If|A| (cid:28) |C| , then matching the performance of the classiﬁer output by the best learning algorithm(within some family) may be more reasonable matching the performance of the best overall classiﬁer.

Original: Turing-optimality is the special case of A consisting of the set of succinct programs, speciﬁcallyconstant-sized time-bounded Turing machines.
Revised: Turing-optimality is the special case of A consisting of the set of succinct programs, speciﬁcallyconstant-sized time-bounded Turing machines.

Original: In Section 5, we also discuss how this approachcan be used across multiple problems to discover a learning algorithm that can be reused on futureproblems, so the search need not be repeated for each learning problem.
Revised: In Section 5, we also discuss how this approachcan be used across multiple problems to discover a learning algorithm that can be reused on futureproblems, so the search need not be repeated for each learning problem.

Original: Turing-optimality as a lens.
Revised: Turing-optimality as a lens.

Original: Like any asymptotic notion, Turing-optimality does not guarantee efﬁcient learning.
Revised: Like any asymptotic notion, Turing-optimality does not guaranteeefﬁcient learning.

Original: Just as a polynomial-time algorithm is not guaranteed to be faster than an exponentialtime algorithm on inputs of interest, such notions can still provide a useful lens to understand algorithms.
Revised: Just as a polynomial-time algorithm is not guaranteed to be faster than anexponential-time algorithm on inputs of interest, such notions can still provide a useful lens tounderstand algorithms.

Original: If learning algorithm L 1 is Turing-optimal and L 2 is not, then L 1 can nearly match (or exceed)the performance of L 2 on any distribution, with polynomial overhead.
Revised: If learning algorithm L 1 is Turing-optimal and L 2 is not, then L 1 can nearlymatch (or exceed) the performance of L 2 on any distribution, with polynomial overhead.

Original: Data distributions where L 2requires super-polynomially resources to match L 1 ’s performance would need to be examined to seeif they are important.
Revised: Datadistributions where L 2 requires super-polynomially resources to match L 1 ’s performance wouldneed to be examined to see if they are important.

Original: We show that a Recurrent Convolutional Neural Network (RCNN) architecture,with random initialization, is Turing-optimal.
Revised: We show that a Recurrent Convolutional Neural Network (RCNN) architecture, with random initialization, is Turing-optimal.

Original: The contribution of this work is showing that a deﬁnitionof Turing-optimality is achievable by a simple NN architecture.
Revised: The contribution ofthis work is showing that a deﬁnition of Turing-optimality is achievable by a simple NN architecture.

Original: In future work, it would be interestingto better understand which architectures, initializations, and learning rates are Turing-optimal.
Revised: In future work, it would be interesting to better understand which other combinations of architectures,initializations, and learning rates are Turing-optimal.

Original: 1.1 Related work
Revised: 1.1 Related work

Original: We review some prior lines of work which establish or use other notions of computational universality.
Revised: We review some prior lines of work which establish or use other notions of computational universality.

Original: We note that most of the notions deﬁned in these works apply to representations rather than algorithms.
Revised: We note that most of the notions deﬁned in these works apply to representations rather than algorithms.

Original: Universal function approximation.
Revised: Universal function approximation.

Original: The ﬁrst line of work relevant to our results is the basic theoryof universal function approximation, which quantiﬁes the ability to ﬁt any sufﬁciently well-behavedfunction for neural networks [21, 10, 16], nearest neighbors [13] and SVMs with RBF kernel [39].
Revised: The ﬁrst line of work relevant to our results is the basic theoryof universal function approximation, which quantiﬁes the ability to ﬁt any sufﬁciently well-behavedfunction for neural networks [21, 10, 16], nearest neighbors [13] and SVMs with RBF kernel [39].

Original: However, they lack statistical insight, e.g., lookup tables are universal over X =
Revised: However, they lack statistical insight, e.g., lookup tables are universal over X =

Original: − 1 , 1 } d but offerlittle statistical power.
Revised: − 1 , 1 } d but offerlittle statistical power.

Original: Further reﬁnements [5, 6, 27] consider Fourier-analytic criteria for functionsto be representable by smaller neural networks.
Revised: Further reﬁnements [5, 6, 27] consider Fourier-analytic criteria for functionsto be representable by smaller neural networks.

Original: The goal of subsequent lines of work described inthis section, as well as the present work, is to investigate the computationally efﬁcient approximationof functions—in other words, the ability of neural networks to emulate efﬁcient learning algorithms.
Revised: The goal of subsequent lines of work described inthis section, as well as the present work, is to investigate the computationally efﬁcient approximationof functions—in other words, the ability of neural networks to emulate efﬁcient learning algorithms.

Original: Turing-completeness of neural architectures.
Revised: Turing-completeness of neural architectures.

Original: Siegelmann and Sontag [35] establish that recurrentneural networks are Turing-complete, using a trick to store the entire TM tape in a single rationalnumber, therefore requiring an extreme amount of bit precision.
Revised: Siegelmann and Sontag [35] establish that recurrentneural networks are Turing-complete, using a trick to store the entire TM tape in a single rational number, therefore requiring an extreme amount of bit precision.

Original: More recently, [19] construct a differentiable TM-inspired architecture.
Revised: More recently, [19] construct adifferentiable TM-inspired architecture.

Original: A number of recent works establish Turing-completeness (aclassical and weaker notion) for variants of the Transformer architecture [12, 41, 7, 8], motivated byempirical advances in discrete reasoning tasks found in natural language processing, theorem proving,and program synthesis.
Revised: A number of recent works establish Turing-completeness (aclassical and weaker notion) for variants of the Transformer architecture [12, 41, 7, 8], motivated byempirical advances in discrete reasoning tasks found in natural language processing, theorem proving,and program synthesis.

Original: Recently [40] propose a notion of statistically meaningful (SM) approximationwhich requires the approximation to be statistically learnable as well.
Revised: Recently [40] propose a notion of statistically meaningful (SM) approximationwhich requires the approximation to be statistically learnable as well.

Original: They show that transformerarchitectures can SM approximate time bounded TMs with sample complexity logarithmic in thetime.
Revised: They show that transformerarchitectures can SM approximate time bounded TMs with sample complexity logarithmic in thetime.

Original: Unlike our notion, Turing-completeness does not take computational efﬁciency into account.
Revised: Unlike our notion, Turing-completeness does not take computational efﬁciency into account.

Original: Enumerative program search.
Revised: Enumerative program search.

Original: A folk-lore argument, similar to Levin’s classic universal search[28], states that one can achieve Turing-optimality by enumerating all Turing machines of a ﬁxedsize, run them all on a training set, and choose the one which performs best on a validation set.
Revised: A folk-lore argument, similar to Levin’s classic universal search[28], states that one can achieve Turing-optimality by enumerating all Turing machines of a ﬁxedsize, run them all on a training set, and choose the one which performs best on a validation set.

Original: Thealgorithm, however, is also completely infeasible in normal programming languages because theprobability of even generating a single program that compiles is minuscule.
Revised: Thealgorithm, however, is also completely infeasible in normal programming languages because theprobability of even generating a single program that compiles is minuscule.

Original: Efﬁcient universality of deep learning.
Revised: Efﬁcient universality of deep learning.

Original: Most closely related to our work is that of Abbe and Sandon[2], which shows how, given any circuit C , e.g., encoding a learning algorithm for parity, one caninitialize the weights of a NN so that it emulates C when the NN is trained by SGD.
Revised: Most closely related to our work is that of Abbe and Sandon[2], which shows how, given any circuit C , e.g., encoding a learning algorithm for parity, one caninitialize the weights of a NN so that it emulates C when the NN is trained by SGD.

Original: This emulatorrequires C to be given as input.
Revised: This emulatorrequires C to be given as input.

Original: Now, row reduction, like any polynomial-time algorithm, can beconverted to a circuit C .
Revised: Now, row reduction, like any polynomial-time algorithm, can beconverted to a circuit C .

Original: However, the size of this circuit is polynomial in the runtime of the algorithm.
Revised: However, the size of this circuit is polynomial in the runtime of the algorithm.

Original: This is why C is required as input, e.g., one has no hope of discovering the Gaussian eliminationalgorithm by random initialization as its probability would be exponentially small in the datasetsize.
Revised: This is why C is required as input, e.g., one has no hope of discovering the Gaussian eliminationalgorithm by random initialization as its probability would be exponentially small in the datasetsize.

Original: Thus, their algorithm does not “discover” the learning algorithm itself–it is hard-coded intothe network.
Revised: Thus, their algorithm does not “discover” the learning algorithm itself–it is hard-coded intothe network.

Original: As they discuss, they could encode in the circuit C an enumerative program search,but this is also a parity learning algorithm that needs to be encoded into the network (and is in factsigniﬁcantly more involved to encode as a circuit).
Revised: As they discuss, they could encode in the circuit C an enumerative program search,but this is also a parity learning algorithm that needs to be encoded into the network (and is in factsigniﬁcantly more involved to encode as a circuit).

Original: Their work was recently extended to mini-batch
Revised: Their work was recently extended to mini-batch

Original: SGD by Abbe et al.
Revised: SGD by Abbe et al.

Original: [1], and it would be interesting to see if our result could be similarly extended.
Revised: [1], and it would be interesting to see if our result could be similarly extended.

Original:  Preliminaries
Revised:  Preliminaries

Original: For simplicity, we focus on binary classiﬁcation with Y :
Revised: For simplicity, we focus on binary classiﬁcation with Y :

Original: = {− 1 , 1 } .
Revised: = {− 1 , 1 } .

Original: For domain X , a (binary)classiﬁer is a function c : X → Y .
Revised: For domain X , a (binary)classiﬁer is a function c : X → Y .

Original: For any distribution D over X × Y , the error of c is, err D ( c ) :=
Revised: For any distribution D over X × Y , the error of c is, err D ( c ) :=

Original: Pr ( x,y )
Revised: Pr ( x,y )

Original: ∼D
Revised: ∼D

Original: [ c ( x ) (cid:54) =
Revised: [ c ( x ) (cid:54) =

Original: y ] ∈ [0 , 1] .
Revised: y ] ∈ [0 , 1] .

Original: A learning algorithm takes as input m ≥ 1 labeled training examplesin ( X × Y )
Revised: A learning algorithm takes as input m ≥ 1 labeled training examplesin ( X × Y )

Original: + = (cid:83) m ≥ 1 ( X × Y ) m
Revised: + = (cid:83) m ≥ 1 ( X × Y ) m

Original: and outputs a classiﬁer.
Revised: and outputs a classiﬁer.

Original: For further simplicity, we focus on dataon the hypercube X d := {− 1 , 1 } d .
Revised: For further simplicity, we focus on dataon the hypercube X d := {− 1 , 1 } d .

Original: The powers of 2 less than 1 are denoted by 2 − N = { 2 − i | i ∈ N } .
Revised: The powers of 2 less than 1 are denoted by 2 − N = { 2 − i | i ∈ N } .

Original: We say an algorithm is polytime if it runs in time polynomial in its input length, which is poly( dm )for a learning algorithm when run on m examples in d dimensions.
Revised: We say an algorithm is polytime if it runs in time polynomial in its input length, which is poly( dm )for a learning algorithm when run on m examples in d dimensions.

Original: 2.1 Turing machines, circuits, and efﬁcient computability
Revised: 2.1 Turing machines, circuits, and efﬁcient computability

Original: Since our main results require the simulation of an arbitrary efﬁcient learning algorithm, we will needto establish formal notation for relevant concepts from the theory of computation.
Revised: Since our main results require the simulation of an arbitrary efﬁcient learning algorithm, we will needto establish formal notation for relevant concepts from the theory of computation.

Original: Various notions ofcomputational efﬁciency may be used.
Revised: Various notions ofcomputational efﬁciency may be used.

Original: To be concrete, we may use a 2 -tape Turing Machine (TM)where the input is on the ﬁrst tape and the 2nd tape is used for computation (e.g. see Hopcroft et al.[20] for a standard reference on Turing machines).
Revised: To be concrete, we may use a 2 -tape Turing Machine (TM)where the input is on the ﬁrst tape and the 2nd tape is used for computation (e.g. see Hopcroft et al.[20] for a standard reference on Turing machines).

Original: One issue that complicates runtime analysis of learning algorithms is that a classiﬁer may be veryslow to evaluate, even if the learning algorithm is fast.
Revised: One issue that complicates runtime analysis of learning algorithms is that a classiﬁer may be veryslow to evaluate, even if the learning algorithm is fast.

Original: There are two solutions to this issue, whichare equivalent up to polynomial time.
Revised: There are two solutions to this issue, which areequivalent up to polynomial time.

Original: The ﬁrst is learning algorithms that output classiﬁers, whichwe represent as Boolean circuits.
Revised: The ﬁrst is learning algorithms that output classiﬁers, which we represent as Boolean circuits.

Original: Circuits circumvent this technicality because they can be evaluatedin time nearly linear in the time it takes to output them.
Revised: Circuits circumvent this technicality because they can be evaluated in timenearly linear in the time it takes to output them.

Original: Thus time spent on classiﬁcation is foldedinto training time.
Revised: Thus time spent on classiﬁcation is folded into trainingtime.

Original: Moreover, any binary classiﬁer on X d can be represented as a circuit, andit is straightforward to convert a NN to a circuit with linear blowup.
Revised: Moreover, any binary classiﬁer on X d can be represented as a circuit, and it is straightforward toconvert a NN to a circuit with linear blowup.

Original: It is also well-known thatother universal representations such as (time-bounded) TMs can be converted to Boolean circuitsin polynomial time using unrolling.
Revised: It is also well-known that other universal representationssuch as (time-bounded) TMs can be converted to Boolean circuits in polynomial time using unrolling.

Original: Other succinct representations could be used, but this choicesimpliﬁes runtime analysis.
Revised: Other succinct representations could be used, but this choice simpliﬁes runtime analysis.

Original: Formally, we assume that each classiﬁer output by a learning algorithm c : X d → Y is represented as
Revised: Formally, we assume that each classiﬁer output by a learning algorithm c : X d → Y is represented as

Original: Boolean circuit, with False representing − 1 and True representing 1.
Revised: Boolean circuit, with False representing − 1 and True representing 1.

Original: If the output of the learner isnot a valid circuit classiﬁer, then by default we assume it classiﬁes everything as 1.
Revised: If the output of the learner isnot a valid circuit classiﬁer, then by default we assume it classiﬁes everything as 1.

Original: We also considerlearners that can be simulated by a TM with size ≤ s in time t using only m labeled examples.
Revised: We also considerlearners that can be simulated by a TM with size ≤ s in time t using only m labeled examples.

Original: Deﬁnition 1 ( ( s, m, t ) -bounded learner) .
Revised: Deﬁnition 1 ( ( s, m, t ) -bounded learner) .

Original: A learner A is a s -bounded learner which outputs aclassiﬁer circuit in at most t steps on any dataset consisting of at most m labeled examples.
Revised: A learner A is a s -bounded learner which outputs aclassiﬁer circuit in at most t steps on any dataset consisting of at most m labeled examples.

Original: 2.2 Components of deep learning
Revised: 2.2 Components of deep learning

Original: In this section, we establish some notation for the building blocks of common deep learning pipelines.
Revised: In this section, we establish some notation for the building blocks of common deep learning pipelines.

Original: Feedforward layers.
Revised: Feedforward layers.

Original: A fully-connected feedforward layer R d in → R d out , with activation functionσ : R → R is parameterized by a matrix W ∈ R d out × d in and bias b ∈ R d out , specifying the mapx (cid:55)→ σ ( Wx + b ) , where σ ( · ) is applied entrywise.
Revised: A fully-connected feedforward layer R d in → R d out , with activation functionσ : R → R is parameterized by a matrix W ∈ R d out × d in and bias b ∈ R d out , specifying the mapx (cid:55)→ σ ( Wx + b ) , where σ ( · ) is applied entrywise.

Original: A feedforward network is the iterative compositionof feedforward layers, possibly omitting an application of σ at the ﬁnal layer.
Revised: A feedforward network is the iterative compositionof feedforward layers, possibly omitting an application of σ at the ﬁnal layer.

Original: Convolutional layers.
Revised: Convolutional layers.

Original: Our main construction will apply the same constant depth feedforwardnetwork repeatedly to each 3 × 3 patch of a 2 -dimensional “image”.
Revised: Our main construction will apply the same constant depth feedforwardnetwork repeatedly to each 3 × 3 patch of a 2 -dimensional “image”.

Original: This can be viewed as applyingmulti-channel convolutional layers followed by non-linear activation consecutively.
Revised: This can be viewed as applyingmulti-channel convolutional layers followed by non-linear activation consecutively.

Original: Due to weightsharing across patches, the number of parameters do not depend on the input dimension but rather onthe patch dimension and the number of channels.
Revised: Due to weightsharing across patches, the number of parameters do not depend on the input dimension but rather onthe patch dimension and the number of channels.

Original: Often in practice, to ensure same output dimensionas input, it is common to add a constant padding (say p ) around the boundaries.
Revised: Often in practice, to ensure same output dimensionas input, it is common to add a constant padding (say p ) around the boundaries.

Original: This is crucial forour construction.
Revised: This is crucial forour construction.

Original: More formally, a convolutional layer speciﬁes a k × k patch-wise linear maps from 157 R k × k × C in to R C out ; in particular, when k = 1 , a convolutional layer speciﬁes a pixel-wise linearmap from R C in to R C out .
Revised: More formally, a convolutional layer speciﬁes a k × k patch-wise linear maps from 157 R k × k × C in to R C out ; in particular, when k = 1 , a convolutional layer speciﬁes a pixel-wise linearmap from R C in to R C out .

Original: We let Conv2D be the application of the linear maps extended to the entireinput.
Revised: We let Conv2D be the application of the linear maps extended to the entireinput.

Original: We overload Conv2D to also allow for patch-wise fully-connected feedforward layers.
Revised: We overload Conv2D to also allow for patch-wise fully-connected feedforward layers.

Original: Recurrent layers.
Revised: Recurrent layers.

Original: Finally, our construction will use recurrent weight sharing: for a functionf : Z × Θ → Z and a positive integer L , we use f ( L ) :
Revised: Finally, our construction will use recurrent weight sharing: for a functionf : Z × Θ → Z and a positive integer L , we use f ( L ) :

Original: Z × Θ → Z to denote the L -times iteratedcomposition of f , sharing the parameters θ ∈ Θ between iterations; for example,f (3) ( X ; θ ) := f
Revised: Z × Θ → Z to denote the L -times iteratedcomposition of f , sharing the parameters θ ∈ Θ between iterations; for example,f (3) ( X ; θ ) := f

Original: ( f ( f ( X ; θ ); θ ); θ ) .
Revised: ( f ( f ( X ; θ ); θ ); θ ) .

Original: The training pipeline: SGD with random initialization.
Revised: The training pipeline: SGD with random initialization.

Original: Finally, we establish some notation forstochastic gradient descent, whose variants form the predominant class of methods for training neuralnetworks.
Revised: Finally, we establish some notation forstochastic gradient descent, whose variants form the predominant class of methods for training neuralnetworks.

Original: Given a continuously differentiable 1 loss function (cid:96) :
Revised: Given a continuously differentiable 1 loss function (cid:96) :

Original: Y × Y → R and continuouslydifferentiable function f : X × Θ → Y where Θ = R d , a step of stochastic gradient descent (SGD)on a single example ( x, y ) ∈
Revised: Y × Y → R and continuouslydifferentiable function f : X × Θ → Y where Θ = R d , a step of stochastic gradient descent (SGD)on a single example ( x, y ) ∈

Original: X × Y , with learning rate η ∈ R , maps the current iterate θ toθ (cid:48) :
Revised: X × Y , with learning rate η ∈ R , maps the current iterate θ toθ (cid:48) :

Original: = θ − η ∇ θ (cid:96) ( f ( x, θ ) , y ) .
Revised: = θ − η ∇ θ (cid:96) ( f ( x, θ ) , y ) .

Original: SGD on a sequence of examples { ( x t , y t ) }
Revised: SGD on a sequence of examples { ( x t , y t ) }

Original: Tt =1 is deﬁned by applying this recurrence iteratively froman initialization θ 0 (usually selected randomly from a speciﬁed distribution), giving a sequence ofiterates { θ t } Tt =1 .
Revised: Tt =1 is deﬁned by applying this recurrence iteratively froman initialization θ 0 (usually selected randomly from a speciﬁed distribution), giving a sequence ofiterates { θ t } Tt =1 .

Original: It is routine to specify a subset S ⊆
Revised: It is routine to specify a subset S ⊆

Original: [ d ] of the parameters to be optimized; in thiscase, the parameters in S are updated according to Equation 2.2, while the rest are unchanged.
Revised: [ d ] of the parameters to be optimized; in thiscase, the parameters in S are updated according to the above equation, while the rest are unchanged.

Original:  Algorithm learning and Turing optimality
Revised:  Algorithm learning and Turing optimality

Original: In this section, we adopt a model of learning which turns PAC learning upside down.
Revised: In this section, we adopt a model of learning which turns PAC learning upside down.

Original: A learningalgorithm is a function
Revised: A learningalgorithm is a function

Original: A that, for any d, m ≥ 1 , outputs a classiﬁer A ( Z ) :
Revised: A that, for any d, m ≥ 1 , outputs a classiﬁer A ( Z ) :

Original: X d → Y for anydataset Z = { ( x ( i ) , y ( i ) ) } mi =1 ∈ ( X d × Y ) m of m ≥ 1 labeled d -dimensional examples.
Revised: X d → Y for anydataset Z = { ( x ( i ) , y ( i ) ) } mi =1 ∈ ( X d × Y ) m of m ≥ 1 labeled d -dimensional examples.

Original: Recall that
Revised: Recall that

Original: X d = {− 1 , 1 } d and Y = {− 1 , 1 } .
Revised: X d = {− 1 , 1 } d and Y = {− 1 , 1 } .

Original: The following deﬁnition captures efﬁcient learnability of a class of learning algorithms A .
Revised: The following deﬁnition captures efﬁcient learnability of a class of learning algorithms A .

Original: Therun-time of the algorithm is required to be polynomial in its input size poly( dm ) .
Revised: Therun-time of the algorithm is required to be polynomial in its input size poly( dm ) .

Original: An importantfeature of this deﬁnition is that it requires thea number of examples to be polynomial in the dimensiond , avoiding the curse of dimensionality.
Revised: An importantfeature of this deﬁnition is that it requires the number of examples to be polynomial in the dimensiond , avoiding the curse of dimensionality.

Original: Since we will soon consider (cid:15), δ as inputs, we consider onlypowers of 2 to avoid having to represent arbitrary real numbers.
Revised: Since we will soon consider (cid:15), δ as inputs, we consider onlypowers of 2 to avoid having to represent arbitrary real numbers.

Original: Deﬁnition 2 (PAO-learner) .
Revised: Deﬁnition 2 (PAO-learner) .

Original: Polytime learning algorithm A is a Probably Algorithmically Optimal(PAO) learner for family
Revised: Polytime learning algorithm A is a Probably Algorithmically Optimal(PAO) learner for family

Original: A if there is a polynomial p such that for any (cid:15), δ ∈ 2 − N , for any d ≥ 1 , anydistribution D over X d × Y , and any dataset sizes m ≥ 1 , M ≥ p ( d, m, 1 /(cid:15), 1 /δ ) ,
Revised: A if there is a polynomial p such that for any (cid:15), δ ∈ 2 − N , for any d ≥ 1 , anydistribution D over X d × Y , and any dataset sizes m ≥ 1 , M ≥ p ( d, m, 1 /(cid:15), 1 /δ ) ,

Original: [Equation]
Revised: [Equation]

Original: where Z ; Z (cid:48) is the concatenation of the two datasets Z, Z (cid:48) .
Revised: where Z ; Z (cid:48) is the concatenation of the two datasets Z, Z (cid:48) .

Original: We further assume that d, m and M canbe determined from the PAO-learner’s input.
Revised: We further assume that d, m and M canbe determined from the PAO-learner’s input.

Original: We now observe that one can equivalently design a learning algorithm that has (cid:15), δ > 0 as inputs.
Revised: We now observe that one can equivalently design a learning algorithm that has (cid:15), δ > 0 as inputs.

Original: Observation 1 ( (cid:15), δ -PAO-learner reduction) .
Revised: Observation 1 ( (cid:15), δ -PAO-learner reduction) .

Original: Let A (cid:15),δ be an “ (cid:15), δ -PAO learner” for A meaning thatit is a polytime learning algorithm that takes additional inputs (cid:15), δ , and there exists some constant ksuch that: for any (cid:15), δ ∈ 2 − N , any dataset sizes m ≥ 1 , M ≥ ( dm/(cid:15)δ [Equation] k ,
Revised: Let A (cid:15),δ be an “ (cid:15), δ -PAO learner” for A meaning thatit is a polytime learning algorithm that takes additional inputs (cid:15), δ , and there exists some constant ksuch that: for any (cid:15), δ ∈ 2 − N , any dataset sizes m ≥ 1 , M ≥ ( dm/(cid:15)δ ) k ,

Original: [Equation]
Revised: [Equation]

Original: Then, for r = 2 (cid:98)− 13 k log M (cid:99) , A r,r is a PAO-learner for A .
Revised: Then, for r = 2 (cid:98)− 13 k log M (cid:99) , A r,r is a PAO-learner for A .

Original: The proof is straightforward and can be found in Appendix A.
Revised: The proof is straightforward and can be found in Appendix A.

Original: The M ≥ ( dm/(cid:15)δ ) k requirement is aconvenient equivalent to a polynomial bound M ≥ p ( d, m, 1 /(cid:15), 1 /δ ) .
Revised: The M ≥ ( dm/(cid:15)δ ) k requirement is aconvenient equivalent to a polynomial bound M ≥ p ( d, m, 1 /(cid:15), 1 /δ ) .

Original: Although we only analyze PAO learning for the family A of bounded Turing machines, it can beanalyzed even for continuous classes A .
Revised: Although we only analyze PAO learning for the family A of bounded Turing machines, it can beanalyzed even for continuous classes A .

Original: For instance, it would be straightforward to show that gridsearch can PAO learn a constant number of bounded hyperparameters of a given algorithm if thealgorithm’s error is Lipschitz continuous in those hyperparameters, using a separate validation set tochoose the best hyperparameters.
Revised: For instance, it would be straightforward to show that gridsearch can PAO learn a constant number of bounded hyperparameters of a given algorithm if thealgorithm’s error is Lipschitz continuous in those hyperparameters, using a separate validation setto choose the best hyperparameters.

Original: PAO learning, as deﬁned, does not specify how classiﬁers are represented, and could apply to any classiﬁer representation.
Revised: PAO learning, as deﬁned, does not specify how classiﬁers arerepresented, and could apply to any classiﬁer representation.

Original: Recall that we represent classiﬁers by Boolean circuits as discussed in Section 2.
Revised: Recall that we represent classiﬁersby Boolean circuits as discussed in Section 2.

Original: We next deﬁne Turing-optimal learners, which are PAO-learners for the class of bounded TMs(constant size, run in polynomial time, and output a circuit classiﬁer).
Revised: We next deﬁne Turing-optimal learners, which are PAO-learners for the class of bounded TMs(constant size, run in polynomial time, and output a circuit classiﬁer).

Original: Deﬁnition 3 (Turing optimal) .
Revised: Deﬁnition 3 (Turing optimal) .

Original: Fix constants s, k ∈ N .
Revised: Fix constants s, k ∈ N .

Original: Let the set B s,k be the set of Turing machineswhich have ≤ s states and, run in time ≤ (2 dm ) k on a dataset Z ∈ ( X d × Y ) m and output a circuit.
Revised: Let the set B s,k be the set of Turing machineswhich have ≤ s states and, run in time ≤ (2 dm ) k on a dataset Z ∈ ( X d × Y ) m and output a circuit.

Original: Learning algorithm
Revised: Learning algorithm

Original: A is ( s, k ) -Turing-optimal if A PAO-learns (or equivalently (cid:15), δ -PAO learns)
Revised: A is ( s, k ) -Turing-optimal if A PAO-learns (or equivalently (cid:15), δ -PAO learns)

Original: B s,k .
Revised: B s,k .

Original: Learning algorithm
Revised: Learning algorithm

Original: A is Turing optimal if A is ( s, k )
Revised: A is Turing optimal if A is ( s, k )

Original: -Turing optimal for all constants s, k ∈ N .
Revised: -Turing optimal for all constants s, k ∈ N .

Original: Note that a Turing-optimal learner A must run in polytime, but the number of examples required tolearn each B s,k can be different, i.e., it will learn B s,k using M ≥ ( dm/(cid:15)δ )
Revised: Note that a Turing-optimal learner A must run in polytime, but the number of examples required tolearn each B s,k can be different, i.e., it will learn B s,k using M ≥ ( dm/(cid:15)δ )

Original: e sk additional examples,for a different constant e sk for each s and k .
Revised: e sk additional examples,for a different constant e sk for each s and k .

Original: Similar to Observation 1, a Turing-optimal learner canbe constructed from an ( s, k ) -Turing optimal learner.
Revised: Similar to Observation 1, a Turing-optimal learner canbe constructed from an ( s, k ) -Turing optimal learner.

Original: The claim below, together with Observation 1,imply that a ( (cid:15), δ, s, k ) -Turing-optimal learner can be converted to a Turing optimal learner.
Revised: The claim below, together with Observation 1,imply that a ( (cid:15), δ, s, k ) -Turing-optimal learner can be converted to a Turing optimal learner.

Original: Algorithm2 and its proof are presented in Appendix A.
Revised: Algorithm2 and its proof are presented in Appendix A.

Original: Claim 1 ( ( s, k ) -Turing-optimal reduction) .
Revised: Claim 1 ( ( s, k ) -Turing-optimal reduction) .

Original: Let A s,k be an algorithm that takes inputs s, k and is( s, k )
Revised: Let A s,k be an algorithm that takes inputs s, k and is( s, k )

Original: -Turing optimal for each pair of constants s, k ∈ N .
Revised: -Turing optimal for each pair of constants s, k ∈ N .

Original: Then, Algorithm 2 ( A s,k ) is Turing optimal.
Revised: Then, Algorithm 2 ( A s,k ) is Turing optimal.

Original: Finally, it is not difﬁcult to see that a Turing-optimal learner also PAC-learns any concept class Cthat is PAC-learnable.
Revised: Finally, it is not difﬁcult to see that a Turing-optimal learner also PAC-learns any concept class Cthat is PAC-learnable.

Original: Following standard conventions, the PAC learning algorithm is given targetaccuracy (cid:15) and failure probability δ as inputs.
Revised: Following standard conventions, the PAC learning algorithm is given targetaccuracy (cid:15) and failure probability δ as inputs.

Original: Also, say a distribution D is said to be consistent withset C of classiﬁers if there is some c ∈ C with err D ( c )
Revised: Also, say a distribution D is said to be consistent withset C of classiﬁers if there is some c ∈ C with err D ( c )

Original: = 0 .
Revised: = 0 .

Original: Deﬁnition 4 (PAC-learning) .
Revised: Deﬁnition 4 (PAC-learning) .

Original: Let C
Revised: Let C

Original: =
Revised: =

Original: (cid:83) d ≥ 1 C d , where c : X d → Y for each c ∈ C d .
Revised: (cid:83) d ≥ 1 C d , where c : X d → Y for each c ∈ C d .

Original: Polytimelearning algorithm A (cid:15),δ PAC-learns C if A (cid:15),δ
Revised: Polytimelearning algorithm A (cid:15),δ PAC-learns C if A (cid:15),δ

Original: and there is a polynomial p such that, for any (cid:15), δ ∈ 2 − N , d ∈ N , m ≥ p ( d, 1 /(cid:15), 1 /δ ) and distribution D consistent with C d :
Revised: and there is a polynomial p such that, for any (cid:15), δ ∈ 2 − N , d ∈ N , m ≥ p ( d, 1 /(cid:15), 1 /δ ) and distribution D consistent with C d :

Original: [Equation]
Revised: [Equation]

Original: The computational polynomial-time efﬁciency requirement on A (cid:15),δ means that its runtime is polynomial in its input size, poly( dm + log 1 /(cid:15)δ ) , because it takes O (log 1 /γ ) bits to describe γ ∈ 2 − N .
Revised: The computational polynomial-time efﬁciency requirement on A (cid:15),δ means that its runtime is polynomial in its input size, poly( dm + log 1 /(cid:15)δ ) , because it takes O (log 1 /γ ) bits to describe γ ∈ 2 − N .

Original: Claim 2.
Revised: Claim 2.

Original: Suppose there is some learning algorithm that PAC-learns C and suppose that A is apolynomial-time Turing-optimal learner.
Revised: Suppose there is some learning algorithm that PAC-learns C and suppose that A is apolynomial-time Turing-optimal learner.

Original: Then A PAC-learns C as well.
Revised: Then A PAC-learns C as well.

Original: We defer the proof to the Appendix A.
Revised: We defer the proof to the Appendix A.

Original: Input: training set S :
Revised: Input: training set S :

Original: = { ( x ( i ) , y ( i ) ) } mi =1 , size s , initialization set U s , depth L , learning rate η
Revised: = { ( x ( i ) , y ( i ) ) } mi =1 , size s , initialization set U s , depth L , learning rate η

Original: Create dummy sample ( x ( m +1) , y ( m +1) )
Revised: Create dummy sample ( x ( m +1) , y ( m +1) )

Original: = ( 1 d , 1)
Revised: = ( 1 d , 1)

Original: Initialize f ∈ F d +1 ,m +1 , 100 s, 100 ,L RCNN (see Deﬁnition 5) with parameters Θ (1) mem , Θ rc , Θ head such that
Revised: Initialize f ∈ F d +1 ,m +1 , 100 s, 100 ,L RCNN (see Deﬁnition 5) with parameters Θ (1) mem , Θ rc , Θ head such that

Original: W (1) = 0 , and all entries of V 1 , V 2 , V 3 , V 4 , V 5 , U 1 , U 2 are sampled uniformly from U s for i = 1 to m + 1 do
Revised: W (1) = 0 , and all entries of V 1 , V 2 , V 3 , V 4 , V 5 , U 1 , U 2 are sampled uniformly from U s for i = 1 to m + 1 do

Original: Update parameters in the memory layer:
Revised: Update parameters in the memory layer:

Original: [Equation]
Revised: [Equation]

Original: where (cid:96) :
Revised: where (cid:96) :

Original: R × R → R is the squared loss, that is, (cid:96) (ˆ y, y ) =
Revised: R × R → R is the squared loss, that is, (cid:96) (ˆ y, y ) =

Original: 12 ( y − ˆ y )end for
Revised: 12 ( y − ˆ y )

Original: [Figure]
Revised: [Figure]

Original:  Turing-optimality of SGD on randomly initialized RCNNs
Revised:  Turing-optimality of SGD on randomly initialized RCNNs

Original: In this section, we will design a Turing-optimal leaner in the form of a NN and a correspondingtraining pipeline.
Revised: In this section, we will design a Turing-optimal leaner in the form of a NN and a correspondingtraining pipeline.

Original: Our NN will be of the form of a RCNN (see Figure 1) with very few trainableparameters and our training pipeline (see Algorithm 1) will use random initialization, and randomrestarts to ﬁnd good parameters.
Revised: Our NN will be of the form of a RCNN (see Figure 1) with very few trainableparameters and our training pipeline (see Algorithm 1) will use random initialization, and randomrestarts to ﬁnd good parameters.

Original: Let us present our main result.
Revised: Let us present our main result.

Original: Theorem 1.
Revised: Theorem 1.

Original: There exists constants c 1 , c 2 , c 3 > 0 such that the following holds.
Revised: There exists constants c 1 , c 2 , c 3 > 0 such that the following holds.

Original: For any d, s, m, t ∈ 231 N ,δ ∈ (0 , 1) , there exists learning rate η ∈ R , L = c 1 ( t + m + d ) , and U s ⊆ R 3 where: forany probability measure D on X d × {− 1 , 1 } , ( s, m, t ) -computable learner A , and training set S ∈ ( X d × {− 1 , 1 } ) m drawn i.i.d.
Revised: For any d, s, m, t ∈ 231 N , there exists learning rate η ∈ R , L = c 1 ( t + m + d ) , and U s ⊆ R 3 where: for any probabilitymeasure D on X d ×{− 1 , 1 } , ( s, m, t ) -computable learner A , and training set S ∈ ( X d ×{− 1 , 1 } ) mdrawn i.i.d.

Original: from D , Algorithm 1 returns a function f such that withprobability at least s − c 2 s , [Equation]
Revised: from D , Algorithm 1 returns a function f such that with probability at least s − c 2 s 2 ,err D ( f ) ≤ err D ( A, S ) .

Original: The bit precision required by Algorithm 1 is (cid:100) log( s ) (cid:101) + c 3 .
Revised: The bit precision required by Algorithm 1 is (cid:100) log( s ) (cid:101) + c 3 .

Original: Remark.
Revised: Remark.

Original: Our Algorithm 1 sets the learning rate of 0 for the shared weights in the RCNN part of the
Revised: Our Algorithm 1 sets the learning rate of 0 for the shared weights in the RCNN part of the

Original: NN and updates only the dense memory layer.
Revised: NN and updates only the dense memory layer.

Original: Since our result is constructive, it is entirely possiblethat a search for an optimal learning rate may perform better in practice.
Revised: Since our result is constructive, it is entirely possiblethat a search for an optimal learning rate may perform better in practice.

Original: Finally, we will use Theorem 1 to create a ( s, k ) -Turing optimal learner with the help of randomrestarts and an additional validation set,
Revised: Finally, we will use Theorem 1 to create a ( s, k ) -Turing optimal learner with the help of randomrestarts and an additional validation set,

Original: Corollary 2.
Revised: Corollary 2.

Original: For ﬁxed constants c 1 , c 2 > 0 , Algorithm 1 can be converted to an ( (cid:15), δ, s, k ) -Turingoptimal learner for any ﬁxed s, k , time bound t =
Revised: For ﬁxed constants c 1 , c 2 > 0 , Algorithm 1 can be converted to an ( (cid:15), δ, s, k ) -Turingoptimal learner for any ﬁxed s, k , time bound t =

Original: (2 dm ) k and (cid:15), δ ∈ (0 , 1) by running it s c
Revised: (2 dm ) k and (cid:15), δ ∈ (0 , 1) by running it s c

Original: s log(1 /δ )
Revised: s log(1 /δ )

Original: times with random restarts, and selecting the classiﬁer that performs best on a validation set of sizec 2 s log s log(1 /δ ) /(cid:15) 2 .
Revised: times with random restarts, and selecting the classiﬁer that performs best on a validation set of sizec 2 s log s log(1 /δ ) /(cid:15) 2 .

Original: Claim 1 converts this to a Turing-optimal learner.
Revised: Claim 1 converts this to a Turing-optimal learner.

Original: Because our formal deﬁnition of Turing-optimalityapplies only to deterministic circuit classiﬁers, one must also convert the NN to a circuit andderandomize the algorithm (which can be done using random bits extracted from additional iidrandom labeled examples [26]).
Revised: Because our formal deﬁnition of Turing-optimalityapplies only to deterministic circuit classiﬁers, one must also convert the NN to a circuit andderandomize the algorithm (which can be done using random bits extracted from additional iidrandom labeled examples [26]).

Original: Next we give a detailed description of the architecture and a proof overview of Theorem 1.
Revised: Next we give a detailed description of the architecture and a proof overview of Theorem 1.

Original: The proofof Corollary 2 and the complete proof of Theorem 1can be found in Appendix B.
Revised: The proofof Corollary 2 and the complete proof of Theorem 1 can be found in Appendix B.

Original: 4.1 Network architecture: RCNN with a memory layer
Revised: 4.1 Network architecture: RCNN with a memory layer

Original: Let us ﬁrst describe the neural network architecture that Algorithm 1 uses in more detail.
Revised: Let us ﬁrst describe the neural network architecture that Algorithm 1 uses in more detail.

Original: Thearchitecture comprises of a dense layer of size linear in m  and d  as the ﬁrst layer.
Revised: The architecture comprises of a dense layer of size linear in m (the number of samples) and d (the dimensionof the input) as the ﬁrst layer.

Original: The output of thislayer is padded with 1 s on the right, and then fed into a RCNN (recurrent weight-sharing across depthand convolutional weight sharing across width).
Revised: The output of this layer is padded with 1 s on the right, and then fedinto a RCNN (recurrent weight-sharing across depth and convolutional weight sharing across width).

Original: The RCNN has only poly ( s ) shared parameters,due to its recurrent and convolutional nature.
Revised: The RCNN has only poly ( s ) shared parameters, due to its recurrent and convolutional nature.

Original: These shared parameters are in the form of a 5-layer NN applied as a convolutional layer to 3 × 3 patches of the input.
Revised: Theseshared parameters are in the form of a 5-layer NN applied as a convolutional layer to 3 × 3 patches ofthe input.

Original: This convolutional layer isapplied recurrently L = p ( t, d, m ) times for some polynomial p (with the same parameters) where tis the bound on the runtime of the TM.
Revised: This convolutional layer is applied recurrently L = p ( t, d, m ) times for some polynomial p(with the same parameters) where t is the bound on the runtime of the TM.

Original: The ﬁnal outputs from the RCNN are then passed through apixel-wise 2-layer NN and ﬁnally summed to give a scalar prediction.
Revised: The ﬁnal outputs from the RCNN are then passed through a pixel-wise 2-layer NN and ﬁnally summed to give a scalar prediction.

Original: Deﬁnition 5 (RCNN with a memory layer) .
Revised: Deﬁnition 5 (RCNN with a memory layer) .

Original: For d in , d mem , d rc , d head , l > 0 , deﬁne the function class
Revised: For d in , d mem , d rc , d head , l > 0 , deﬁne the function class

Original: F d in ,d mem ,d rc ,d head ,l RCNN of RCNNs where each f ∈ F d in ,d mem ,d rc ,d head ,l RCNN is parameterized by memory layerparameters Θ mem = { W } with W ∈ R d mem × d in , RC layer parameters Θ rc := { V 1 , V 2 , V 3 , V 4 , V 5 } 5with V 1 ∈ R d rc × 9 , V 2 , V 3 , V 4 ∈ R d rc × s rc , V 5 ∈ R 1 × d rc , and head layer parameters Θ head =
Revised: F d in ,d mem ,d rc ,d head ,l RCNN of RCNNs where each f ∈ F d in ,d mem ,d rc ,d head ,l RCNN is parameterized by memory layerparameters Θ mem = { W } with W ∈ R d mem × d in , RC layer parameters Θ rc := { V 1 , V 2 , V 3 , V 4 , V 5 } 5with V 1 ∈ R d rc × 9 , V 2 , V 3 , V 4 ∈ R d rc × d rc , V 5 ∈ R 1 × d rc , and head layer parameters Θ head =

Original: { U 1 , U 2 }with U 1 ∈ R
Revised: { U 1 , U 2 }with U 1 ∈ R

Original: d
Revised: d

Original: head × 1 , U 2 ∈ R 1 × d head ) .
Revised: head × 1 , U 2 ∈ R 1 × d head ) .

Original: The local convolutional operation is denoted by f Conv2D : 266 R 3 ×→ R and is applied on the 3 × 3 grid centered at each coordinate of the d mem × ( d in + l )input (with padding p for the edge coordinates).
Revised: The local convolutional operation is denoted by f Conv2D : 265 R 3 ×→ R and is applied on the 3 × 3 grid centered at each coordinate of the d mem × ( d in + l )input (with padding p for the edge coordinates).

Original: We overload the notation f Conv2D to denote thed mem × ( d in + l ) output post the local application on the input.
Revised: We overload the notation f Conv2D to denote thed mem × ( d in + l ) output post the local application on the input.f ( z ; Θ mem , Θ RCNN ; Θ head )

Original: [Equation]
Revised: = sum (cid:16) f head (cid:16) f

Original:  where f mem ( z ; Θ mem = { W } ) =
Revised: ( l ) Conv2D ( f mem ( z ; Θ mem ); Θ RCNN ) : , : d in (cid:17) ; Θ head (cid:17) where f mem ( z ; Θ mem = { W } ) =

Original: [ W diag( z ) 1 d mem × l ] f head ( z : θ = { U 1 , U 2 } )
Revised: [ W diag( z ) 1 d mem × l ] f head ( z : θ = { U 1 , U 2 } )

Original: =
Revised: =

Original: U 1 σ ( U 2 z ) f Conv2D ( z ; Θ
Revised: U 1 σ ( U 2 z ) f Conv2D ( z ; Θ

Original: = { V 1 , V 2 , V 3 , V 4 , V 5 } )
Revised: = { V 1 , V 2 , V 3 , V 4 , V 5 } )

Original: =
Revised: =

Original: V 1 σ ( V 2 V 3 σ ( V 4 V 5 vec ( z ))))) .
Revised: V 1 σ ( V 2 V 3 σ ( V 4 V 5 vec ( z ))))) .

Original: with σ being the ReLU activation.
Revised: with σ being the ReLU activation.

Original: Remark.
Revised: Remark.

Original: If our architecture did not have both recurrent and convolutional weight sharing, then thenumber of parameters would have dependence on d ∈ , d mem and/or l  .
Revised: If our architecture did not have both recurrent and convolutional weight sharing, then thenumber of parameters would have dependence on d in , d mem and l , which depend on m , d , and T .

Original: 4.2 Proof sketch for Theorem 1
Revised: 4.2 Proof sketch for Theorem 1

Original: Here we present a proof sketch for Theorem 1.
Revised: Here we present a proof sketch for Theorem 1.

Original: Our proof follows by construction, that is, we showthat for
Revised: Our proof follows by construction, that is, we showthat for

Original: each TM A of size s , there exists parameters Θ rc and Θ head that ensure that (1) for the ﬁrstm + 1 steps, when Θ mem is trained with SGD, the gradients assist with memorizing the training setin the values of W mem , and (2) given the memorized training set, the RCNN computes the roll-out of
Revised: each TM A of size s , there exists parameters Θ rc and Θ head that ensure that (1) for the ﬁrstm + 1 steps, when Θ mem is trained with SGD, the gradients assist with memorizing the training setin the values of W mem , and (2) given the memorized training set, the RCNN computes the roll-out of

Original: A with the input tape having the training set and the test example giving the prediction of A on thetest example as the output.
Revised: A with the input tape having the training set and the test example giving the prediction of A on thetest example as the output.

Original: We ﬁnally show that the parameters Θ rc and Θ head in our construction forall TM A of ﬁxed size and runtime belong to a ﬁxed ﬁnite set of size O ( s ) that can be constructedwith knowledge of only s .
Revised: We ﬁnally show that the parameters Θ rc and Θ head in our construction forall TM A of ﬁxed size and runtime belong to a ﬁxed ﬁnite set of size O ( s ) that can be constructedwith knowledge of only s .

Original: The following lemma summarizes the aforementioned properties:
Revised: The following lemma summarizes the aforementioned properties:

Original: Lemma 1.
Revised: Lemma 1.

Original: For any d, s, m, t ∈ N , δ ∈ (0 , 1) , and any ( s, m, t ) -computable learner A , there existsΘ rc , Θ head with each parameter belonging to a ﬁxed set U s of size O ( s ) (that can be constructed withthe knowledge of only s ) such that Algorithm 1  satisﬁes:
Revised: For any d, s, m, t ∈ N , δ ∈ (0 , 1) , and any ( s, m, t ) -computable learner A , there existsΘ rc , Θ head with each parameter belonging to a ﬁxed set U s of size O ( s ) (that can be constructed withthe knowledge of only s ) such that Algorithm 1 with L = satisﬁes:

Original: Memorization: For 1 ≤ i ≤ m + 1 , W ( i ) ab = 
Revised: Memorization: For 1 ≤ i ≤ m + 1 , W ( i ) ab = 

Original:  13 y ( a ) x ( a ) b if a ≤ i, b ≤ d 13 y ( a ) if a ≤ i, b = d + otherwise.
Revised:  13 y ( a ) x ( a ) b if a ≤ i, b ≤ d 13 y ( a ) if a ≤ i, b = d + otherwise.

Original: Computation:
Revised: Computation:

Original: For all x ∈ {± 1 } d , f (cid:18)(cid:20) x 1 (cid:21) ; W ( m +2) , Θ rc , Θ head (cid:19) =
Revised: For all x ∈ {± 1 } d , f (cid:18)(cid:20) x 1 (cid:21) ; W ( m +2) , Θ rc , Θ head (cid:19) =

Original: A ( S )
Revised: A ( S )

Original: [ x ] .
Revised: [ x ] .

Original: Theorem 1 follows from the above the computation property of Lemma 1, since it implies that theerror of Algorithm 1 will be exactly equal to the error of A .
Revised: Theorem 1 follows from the above the computation property of Lemma 1, since it implies that theerror of Algorithm 1 will be exactly equal to the error of A .

Original: What remains is to prove the existence of parameters that satisfy Lemma 1.
Revised: What remains is to prove the existence of parameters that satisfy Lemma 1.

Original: Let us now brieﬂydescribe the key functionality we require the RCNN to implement for this:
Revised: Let us now brieﬂydescribe the key functionality we require the RCNN to implement for this:

Original: Computation. Each roll-out step of the TM is a local update around the head of the TM.
Revised: Computation: Each roll-out step of the TM is a local update around the head of the TM.

Original: Toimplement one step of the TM, we need to compute the transition function of A at the location of thehead, update the new head and state, and copy the inputs of the rest of the tape.
Revised: To implementone step of the TM, we need to compute the transition function of A at the location of the head,update the new head and state, and copy the inputs of the rest of the tape.

Original: Our ﬁrst observationis that this local update can be implemented using a convolutional layer if we interpret the input asthe tape of A (input and working concatenated) with the head and state information stored along withthe tape value.
Revised: Our ﬁrst observation isthat this local update can be implemented using a convolutional layer if we interpret the input as thetape of A (input and working concatenated) with the head and state information stored along with thetape value.

Original: Composing these layers t times (with the same parameters) allows us to simulate tsteps of the TM.
Revised: Composing these layers t times (with the same parameters) allows us to simulate t stepsof the TM.

Original: More importantly, the convolutional layer requires only O ( s ) parameters since Aonly has s states.
Revised: More importantly, the convolutional layer requires only O ( s ) parameters since A onlyhas s states.

Original: In order to decode the tape content, head position, and state from the values fed tothe layers of the RCNN, we interpret the input in base 3 and use different positions to encode thedesired information (see Appendix B.1 for more details).
Revised: In order to decode the tape content, head position, and state from the values fed to thelayers of the RCNN, we interpret the input in base 3 and use different positions to encode the desiredinformation (see Appendix B.1 for more details).

Original: Memorization. Given that we can simulate the TM, we need to ensure that the input to the RCNNhas the training set and the test example encoded onto it.
Revised: Memorization: Given that we can simulate the TM, we need to ensure that the input to the RCNN hasthe training set and the test example encoded onto it.

Original: Similar to [2], we can use SGD to memorizethe training examples into the weights of the memory layer, with each row storing one example.
Revised: Similar to [2], we can use SGD to memorize thetraining examples into the weights of the memory layer, with each row storing one example.

Original: Wedo this by ensuring that the gradient at iteration i through the RCNN is 1 for exactly the i th row and 0for every other row.
Revised: We dothis by ensuring that the gradient at iteration i through the RCNN is 1 for exactly the i th row and 0for every other row.

Original: We also ensure that the output is 0 through the memorization phase.
Revised: We also ensure that the output is 0 through the memorization phase.

Original: By usingchain rule, this gives us a gradient of [ y ( i ) x
Revised: By usingchain rule, this gives us a gradient of [ y ( i ) x

Original: ( i ) , y ( i ) ] for the i th row of W mem and 0 otherwise.
Revised: ( i ) , y ( i ) ] for the i th row of W mem and 0 otherwise.

Original: We canthis in a local manner using the RCNN.
Revised: We canthis in a local manner using the RCNN.

Original: Note that the memory layer has O ( dm ) parameters, howeverwe can learn these parameters from 0 initialization.
Revised: Note that the memory layer has O ( dm ) parameters, howeverwe can learn these parameters from 0 initialization.

Original: Communication. Lastly, we need to ensure that the network can differentiate between memorizationphase (passing meaningful gradients) and computation phase (implementing the roll-out of the TM)using the local operations in the RCNN layers.
Revised: Communication: Lastly, we need to ensure that the network can differentiate between memorizationphase (passing meaningful gradients) and computation phase (implementing the roll-out of the TM)using the local operations in the RCNN layers.

Original: We do this by implementing a local communicationprotocol: we broadcast a message based on certain conditions, where each RCNN layer implements astep of the broadcast.
Revised: We do this by implementing a local communicationprotocol: we broadcast a message based on certain conditions, where each RCNN layer implements astep of the broadcast.

Original: In order to broadcast to the entire input, we require ≈ d + m overhead in termsof the depth of the network.
Revised: In order to broadcast to the entire input, we require ≈ d + m overhead in termsof the depth of the network.

Original: Finally, we show that the above mentioned functionality can be achieved by a 5-layer NN with O ( s  )parameters.
Revised: Finally, we show that the above mentioned functionality can be achieved by a 5-layer NN with O ( s 2 )parameters.

Original: To do so, we ﬁrst describe the exact function we require the network to compute andits Jacobian on all inputs that our bit precision allows (see Appendix B.2 for the exact function).
Revised: To do so, we ﬁrst describe the exact function we require the network to compute andits Jacobian on all inputs that our bit precision allows (see Appendix B.2 for the exact function).

Original: Given the function and its Jacobian on a ﬁnite set, we prove a general representational theorem (see
Revised: Given the function and its Jacobian on a ﬁnite set, we prove a general representational theorem (see

Original: Appendix B.4.1) that constructs a 5-layer NN with weights from a ﬁxed set that can be constructedwith knowledge of only s .
Revised: Appendix B.4.1) that constructs a 5-layer NN with weights from a ﬁxed set that can be constructedwith knowledge of only s .

Original: We refer the reader to the Appendix B for the complete proof.
Revised: We refer the reader to the Appendix B for the complete proof.

Original:  Discussion
Revised:  Discussion

Original: In this section, we discuss potential practical implications of Turing-optimality, and broadly discusscorroborations and tensions with empirical trends.
Revised: In this section, we discuss potential practical implications of Turing-optimality, and broadly discusscorroborations and tensions with empirical trends.

Original: Discovering reusable algorithms.
Revised: Discovering reusable algorithms.

Original: Our analysis is wasteful in that, if one has multiple learningproblems, e.g., multiple parity problems, one has to relearn the learning algorithm for each one.
Revised: Our analysis is wasteful in that, if one has multiple learningproblems, e.g., multiple parity problems, one has to relearn the learning algorithm for each one.

Original: Infact, arguably the NN may not have learned an algorithm for parity problems in general, but rathera specialized algorithm that works on just one.
Revised: In fact, arguably the NN may not have learned an algorithm for parity problems in general, butrather a specialized algorithm that works on just one.

Original: To ﬁnd a reusable algorithm, one needs multipleproblems, say drawn from a meta-distribution µ over learning problems.
Revised: To ﬁnd a reusable algorithm, one needsmultiple problems, say drawn from a meta-distribution µ over learning problems.

Original: The idea is simple: viewingthe constant number of weights of the RCNN ﬁlter as hyperparameters, one tries multiple such hyperparameters on log 1 /δ learning problems, and ﬁnally selecting the hyperparameters that performbest on average.
Revised: The idea is simple:viewing the constant number of weights of the RCNN ﬁlter as hyperparameters, one tries multiple such hyperparameters on log 1 /δ learning problems, and ﬁnally selecting the hyperparametersthat perform best on average.

Original: With a constant-sized random sample of hyperparameters, with high probability,one of them will perform nearly as well as the best constant-sized TM not only on these few trainingproblems, but also on future problems drawn from µ .
Revised: With a constant-sized random sample of hyperparameters, with highprobability, one of them will perform nearly as well as the best constant-sized TM not only on thesefew training problems, but also on future problems drawn from µ .

Original: We leave this and numerous other interestingdirections for future work.
Revised: We leave this and numerous otherinteresting directions for future work.

Original: Concise architectures.
Revised: Concise architectures.

Original: Many Turing-complete architectures have been proposed and used in practice.
Revised: Many Turing-complete architectures have been proposed and used in practice.

Original: The lens of Turing-optimality may help us understand what architectures are minimally adequate froma theoretical perspective.
Revised: The lens of Turing-optimality may help us understand what architectures are minimally adequate froma theoretical perspective.

Original: In particular, it has been popular to report ever-growing parameter countsfor state-of-the-art models in domains such as natural language processing [9, 15, 30].
Revised: In particular, it has been popular to report ever-growing parameter countsfor state-of-the-art models in domains such as natural language processing [9, 15, 30].

Original: Although theother beneﬁts of over-parameterization are at play, this work suggests that very parameter-efﬁcientarchitectures are sufﬁcient to simulate any computationally efﬁcient learning algorithm.
Revised: Although theother beneﬁts of over-parameterization are at play, this work suggests that very parameter-efﬁcientarchitectures are sufﬁcient to simulate any computationally efﬁcient learning algorithm.

Original: In lightof the above, one concrete direction for further investigation is to develop practical variants of our
Revised: In lightof the above, one concrete direction for further investigation is to develop practical variants of our

Original: RCNN construction, in domains dominated by other architectures.
Revised: RCNN construction, in domains dominated by other architectures.

Original: Although our analysis is toopessimistic to be of immediate practical use, it highlights the computational power of an architecturethat has occasionally appeared in applications-focused research [33, 29, 36, 3].
Revised: Although our analysis is toopessimistic to be of immediate practical use, it highlights the computational power of an architecturethat has occasionally appeared in applications-focused research [33, 29, 36, 3].

Original: Signiﬁcantly closerto our work, Schwarzschild et al.
Revised: Signiﬁcantly closerto our work, Schwarzschild et al.

Original: [34] conduct an empirical study on the ability of RCNNs toextrapolate from easier to harder tasks (thus “learn an algorithm”); our work shows that it is possiblefor these architectures to learn any computationally efﬁcient algorithm.
Revised: [34] conduct an empirical study on the ability of RCNNs toextrapolate from easier to harder tasks (thus “learn an algorithm”); our work shows that it is possiblefor these architectures to learn any computationally efﬁcient algorithm.

Original: Similarly, RCNNs have beeninvestigated for planning in RL [38]; other empirical works which take a “computation time” view ofdepth include [19, 4, 22].
Revised: Similarly, RCNNs have beeninvestigated for planning in RL [38]; other empirical works which take a “computation time” view ofdepth include [19, 4, 22].

Original: RCNNs have not seen widespread adoption in state-of-the-art deep learningcompared to their non-recurrent and/or non-convolutional counterparts.
Revised: RCNNs have not seen widespread adoption in state-of-the-art deep learningcompared to their non-recurrent and/or non-convolutional counterparts.

Original: Beyond local search for recurrent models.
Revised: Beyond local search for recurrent models.

Original: The proliferation of non-recurrent attention-basedmodels in domains previously dominated by recurrent networks, along with the under-representationof RCNNs, is perhaps due to instabilities in training recurrent networks with SGD [32].
Revised: The proliferation of non-recurrent attention-basedmodels in domains previously dominated by recurrent networks, along with the under-representationof RCNNs, is perhaps due to instabilities in training recurrent networks with SGD [32].

Original: Indeed, Kasaiet al.
Revised: Indeed, Kasaiet al.

Original: [23] demonstrate that a carefully designed training procedure can convert a trained Transformerinto a more parameter-efﬁcient RNN.
Revised: [23] demonstrate that a carefully designed training procedure can convert a trained Transformerinto a more parameter-efﬁcient RNN.

Original: There may be undiscovered practical training algorithms whichcan bridge the gap in favor of recurrent models.
Revised: There may be undiscovered practical training algorithms whichcan bridge the gap in favor of recurrent models.

Original: Although our Turing-complete algorithm uses SGD,it uses the gradients in a way that is far from making local greedy progress on an objective; theas-efﬁcient-as-possible search for the correct TM is implemented by random initialization.
Revised: Although our Turing-complete algorithm uses SGD,it uses the gradients in a way that is far from making local greedy progress on an objective; theas-efﬁcient-as-possible search for the correct TM is implemented by random initialization.

Original: Memory modules.
Revised: Memory modules.

Original: There have been many attempts to build practical memory modules into neuralnetworks [19, 37, 18, 11].
Revised: There have been many attempts to build practical memory modules into neuralnetworks [19, 37, 18, 11].

Original: Our construction proposes an integrated memory mechanism: use SGD tostore samples in the ﬁrst layer’s trainable parameters, ahead of the deep (RCNN) computation layers,by carefully ensuring that the gradient signal back-propagates through the RCNN layers correctly.
Revised: Our construction proposes an integrated memory mechanism: use SGD tostore samples in the ﬁrst layer’s trainable parameters, ahead of the deep (RCNN) computation layers,by carefully ensuring that the gradient signal back-propagates through the RCNN layers correctly.

Original: Limitations and broader impact.
Revised: Limitations and broader impact.

Original: The primary limitation of this work is that the constant factorsin our analysis are much too large to be meaningful in practice.
Revised: The primary limitation of this work is that the constant factorsin our analysis are much too large to be meaningful in practice.

Original: Nonetheless, we hope that idea ofcombining recurrent and convolutional weight sharing will have impact.
Revised: Nonetheless, we hope that idea ofcombining recurrent and convolutional weight sharing will have impact.

Original: Also, the algorithms that
Revised: Also, the algorithms that

Original: our
Revised: our

Original: NNs learn would be in a format that is largely uninterpretable.
Revised: NNs learn would be in a format that is largely uninterpretable.

Original: Using such an algorithm carries risks,especially if the algorithm is not doing what one expects it to do.
Revised: Using such an algorithm carries risks,especially if the algorithm is not doing what one expects it to do.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we present a simple NN architecture, combining recurrent and convolutional weightsharing, that achieves Turing-optimality.
Revised: In this paper, we present a simple NN architecture, combining recurrent and convolutional weightsharing, that achieves Turing-optimality.

Original: Among other things, it learns the well-studied class of parityfunctions in polynomial time, whereas prior NN analyses of parity require time exponential in thesize of the parity function (or require a parity learning algorithm to be initialized into the networksweights).
Revised: Among other things, it learns the well-studied class of parityfunctions in polynomial time, whereas prior NN analyses of parity require time exponential in thesize of the parity function (or require a parity learning algorithm to be initialized into the networksweights).

Original: Our proposed architecture has connections to the deep learning literature and observedempirical trends, discussed in Section 5.
Revised: Our proposed architecture has connections to the deep learning literature and observedempirical trends, discussed in Section 5.

Original: Immediate improvements to make the architecture moreconcise and natural include: (1) reducing the size of the dense parameters to depend on the algorithm’smemory usage instead of the training sample size, and (2) using SGD beyond memorization.
Revised: Immediate improvements to make the architecture moreconcise and natural include: (1) reducing the size of the dense parameters to depend on the algorithm’smemory usage instead of the training sample size, and (2) using SGD beyond memorization.

Original: Infuture work, it would be interesting to understand which other architectures are Turing-optimal,answering questions such as: are 2d convolutions necessary, and are there natural Transformer-basedarchitectures that are Turing optimal?
Revised: Infuture work, it would be interesting to understand which other architectures are Turing-optimal,answering questions such as: are 2d convolutions necessary, and are there natural Transformer-basedarchitectures that are Turing optimal?

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?

Original: [Yes](
Revised: [Yes](

Original: b
Revised: b

Original: ) Did you describe the limitations of your work?
Revised: ) Did you describe the limitations of your work?

Original: [Yes] Last paragraph of Section [-
-]
Revised: [Yes] Last paragraph of Section [-
-]5(

Original: (c)
Revised: c

Original:  Did you discuss any potential negative societal impacts of your work
Revised: ) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] Lastparagraph of Section 

Revised: [Yes] Lastparagraph of Section

Original: (d) Have you read the ethics review guidelines and ensured that
Revised: (d) Have you read the ethics review guidelines and ensured that

Original: your
Revised: your

Original: paper
Revised: paper

Original: conforms to
Revised: conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If
Revised: If

Original: you are including theoretical results
Revised: you are including theoretical results

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] Many proof are in the
Revised: [Yes] Many proof are in the

Original: appendix
Revised: appendix

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi-
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Revised: [Table]

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [N/A]
Revised: [N/A]

Original: [Table]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: [Table]
Revised: [Table]

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: [Table]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: [Table]
Revised: [Table]

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: [Table]

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: [Table]
Revised: [Table]

Original: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: [Table]
Revised: [Table]

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: [Table]

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A] [Table]
Revised: [N/A] [Table]