Original: Active Learning with Neural Networks: Insights from Nonparametric Statistics
Revised: Active Learning with Neural Networks: Insights from Nonparametric Statistics

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Deep neural networks have great representation power, but typically require largenumbers of training examples.
Revised: Deep neural networks have great representation power, but typically require large numbers of training examples.

Original: This motivates deep active learning methods thatcan signiﬁcantly reduce the amount of labeled training data.
Revised: This motivates deep active learning methods that can signiﬁcantly reduce the amount of labeled training data.

Original: Empirical successes ofdeep active learning have been recently reported in the literature, however, rigorouslabel complexity guarantees of deep active learning have remained elusive.
Revised: Empirical successes of deep active learning have been recently reported in the literature, however, rigorous label complexity guarantees of deep active learning have remained elusive.

Original: Thisconstitutes a signiﬁcant gap between theory and practice.
Revised: This constitutes a signiﬁcant gap between theory and practice.

Original: This paper tackles thisgap by providing the ﬁrst near minimax optimal label complexity guarantees fordeep active learning.
Revised: This paper tackles this gap by providing the ﬁrst near-optimal label complexity guarantees for deep active learning.

Original: The key insight is to study deep active learning from thenonparametric classiﬁcation perspective.
Revised: The key insight is to study deep active learning from the nonparametric classiﬁcation perspective.

Original: Under common low noise conditions, weshow that active learning with neural networks can provably achieve the minimaxlabel complexity, up to disagreement coefﬁcient and other logarithmic terms.
Revised: Under standard low noise conditions, we show that active learning with neural networks can provably achieve the minimax label complexity, up to disagreement coefﬁcient and other logarithmic terms.

Original: Whenequipped with an abstention option, we further develop a computationally efﬁcientdeep active learning algorithm that guarantees polylog ( 1 ε ) label complexity, withoutany low noise assumptions.
Revised: When equipped with an abstention option, we further develop an efﬁcient deep active learning algorithm that achieves polylog ( 1 ε ) label complexity, without any low noise assumptions.

Original: We also provide extensions of  results beyond thecommonly studied Sobolev/Hölder spaces and develop label complexity guaranteesfor learning in Radon BV 2 spaces, which have recently been proposed as the naturalfunction spaces associated with neural networks.
Revised: We also provide extensions of our results beyond the commonly studied Sobolev/Hölder spaces and develop label complexity guarantees for learning in Radon BV 2 spaces, which have recently been proposed as natural function spaces associated with neural networks.

Original:  Introduction
Revised:  Introduction

Original: We study active learning withhypothesis class being a set of neural networks, sometimes known asdeep active learning .
Revised: We study active learning with neural network hypothesis classes, sometimes known as deep active learning .

Original: Active learning agent proceeds by selecting the most informative data points to label:
Revised: Active learning agent proceeds by selecting the most informative data points to label:

Original: The goal of active learning is to achieve the same accuracy achievable by passive learning butwith much fewer labeling examples (Settles, 2009; Hanneke, 2014).
Revised: The goal of active learning is to achieve the same accuracy achievable by passive learning, but with much fewer label queries (Settles, 2009; Hanneke, 2014).

Original: When the hypothesis class isa set of neural networks, the learner further beneﬁts from the representation power of deep neuralnetworks, which has driven the success of passive learning in the past decade (Krizhevsky et al. ,2012; LeCun et al., 2015).
Revised: When the hypothesis class is a set of neural networks, the learner further beneﬁts from the representation power of deep neural networks, which has driven the successes of passive learning in the past decade (Krizhevsky et al., 2012; LeCun et al., 2015).

Original: With these added beneﬁts, deep active learning has become a popularresearch area, with empirical successes observed in many recent papers (Sener and Savarese, 2018; Ash et al., 2019; Citovsky et al., 2021; Ash et al., 2021; Kothawade et al., 2021; Emam et al., 2021;
Revised: With these added beneﬁts, deep active learning has become a popular research area, with empirical successes observed in many recent papers (Sener and Savarese, 2018; Ash et al., 2019; Citovsky et al., 2021; Ash et al., 2021; Kothawade et al., 2021; Emam et al., 2021;

Original: Ren et al., 2021).
Revised: Ren et al., 2021).

Original: However, due to the difﬁculty of analyzing a set of neural networks, rigorous labelcomplexity guarantees for deep active learning have remained largely elusive.
Revised: However, due to the difﬁculty of analyzing a set of neural networks, rigorous label complexity guarantees for deep active learning have remained largely elusive.

Original: To the best of our knowledge, there are only two papersthat have made the attempts (Karzand and Nowak, 2020; Wang et al., 2021)  at theoretically quantifying active learning gains in neural networksettings.
Revised: To the best of our knowledge, there are only two papers (Karzand and Nowak, 2020; Wang et al., 2021) that have made the attempts at theoretically quantifying active learning gains with neural networks.

Original: While insightful views are provided, these two works have their own limitations.
Revised: While insightful views are provided, these two works have their own limitations.

Original: Theguarantees provided in Karzand and Nowak (2020) only work in the 1 d case where data points areuniformly sampled from [0 , 1] and labeled by a well-seperated piece-wise constant function in anoise-free way (i.e., without any labeling noise).
Revised: The guarantees provided in Karzand and Nowak (2020) only work in the 1 d case where data points are uniformly sampled from [0 , 1] and labeled by a well-seperated piece-wise constant function in a noise-free way (i.e., without any labeling noise).

Original: Wang et al.
Revised: Wang et al.

Original: (2021) study deep active learning bylinearizing the neural network at its random initialization and then analyzing it as a linear function; moreover, as the authors agree, their error bounds and label complexity guarantees can in fact bevacuous in certain cases.
Revised: (2021) study deep active learning by 36th Conference on Neural Information Processing Systems (NeurIPS 2022). linearizing the neural network at its random initialization and then analyzing it as a linear function; moreover, as the authors agree, their error bounds and label complexity guarantees can in fact be vacuous in certain cases.

Original: Thus, it’s fair to say that up to now researchers have not identiﬁed caseswhere deep active learning are provably near minimax optimal (or even with provably non-vacuousguarantees), which constitutes a signiﬁcant gap between theory and practice.
Revised: Thus, it’s fair to say that up to now researchers have not identiﬁed cases where deep active learning are provably near minimax optimal (or even with provably non-vacuous guarantees), which constitutes a signiﬁcant gap between theory and practice.

Original: In this paper, we ﬁll this gap by providing the ﬁrst near minimax optimal label complexity guaranteesfor deep active learning.
Revised: In this paper, we bridge this gap by providing the ﬁrst near-optimal label complexity guarantees for deep active learning.

Original: We obtain insights from the nonparametric setting where the conditionalprobability is assumed to be a smooth function (Tsybakov, 2004; Audibert and Tsybakov, 2007).
Revised: We obtain insights from the nonparametric setting where the conditional probability (of taking a positive label) is assumed to be a smooth function (Tsybakov, 2004; Audibert and Tsybakov, 2007).

Original: Previous nonparametric active learning algorithms proceed by partitioning the action space intoexponentially many sub-regions (e.g., partitioning the unit cube [0 , 1] d into ε − d sub-cubes each withvolume ε d ), and then conducting local mean  estimation within each sub-region (Castro and Nowak,2008; Minsker, 2012; Locatelli et al., 2017, 2018; 
Revised: Previous nonparametric active learning algorithms proceed by partitioning the action space into exponentially many sub-regions (e.g., partitioning the unit cube [0 , 1] d into ε − d sub-cubes each with volume ε d ), and then conducting local mean (or some higher-order statistics) estimation within each sub-region (Castro and Nowak, 2008; Minsker, 2012; Locatelli et al., 2017, 2018; Shekhar et al., 2021;

Original: Shekhar et al., 2021; Kpotufe et al., 2021).
Revised: Kpotufe et al., 2021).

Original: Weshow that, with an appropriately chosen set of neural networks that globally approximates the smoothregression function, one can in fact recover the minimax label complexity for active learning (upto disagreement coefﬁcient (Hanneke, 2007, 2014) and other logarithmic factors).
Revised: We show that, with an appropriately chosen set of neural networks that globally approximates the smooth regression function, one can in fact recover the minimax label complexity for active learning, up to disagreement coefﬁcient (Hanneke, 2007, 2014) and other logarithmic factors.

Original: Our results areestablished by (1) identifying the “right tools” to study neural networks (ranging from approximationresults (Yarotsky, 2017, 2018) to complexity measure of neural networks (Bartlett et al., 2019)), and(2) developing novel extensions of agnostic active learning algorithms (Balcan et al., 2009; Hanneke,2014) to work with a set of neural networks.
Revised: Our results are established by (i) identifying the “right tools” to study neural networks (ranging from approximation results (Yarotsky, 2017, 2018) to complexity measure of neural networks (Bartlett et al., 2019)), and (ii) developing novel extensions of agnostic active learning algorithms (Balcan et al., 2006; Hanneke, 2007, 2014) to work with a set of neural networks.

Original: While matching the minimax label complexity in nonparametric active learning is existing, suchminimax results scale as Θ(poly( 1 ε ))
Revised: While matching the minimax label complexity in nonparametric active learning is existing, such minimax results scale as Θ(poly( 1 ε ))

Original: (Castro and Nowak, 2008; Locatelli et al., 2017) and do notresemble what is practically observed in deep active learning: A fairly accurate neural networkclassiﬁer can be obtained by training with only a few labeled data points.
Revised: (Castro and Nowak, 2008; Locatelli et al., 2017) and do not resemble what is practically observed in deep active learning: A fairly accurate neural network classiﬁer can be obtained by training with only a few labeled data points.

Original: Inspired by recent resultsin parametric active learning with abstention (Puchkin and Zhivotovskiy, 2021; Zhu and Nowak,2022), we develop an  algorithm showing that deep active learning provably achieves polylog( 1 ε ) labelcomplexity when equipped with an abstention option (Chow, 1970).
Revised: Inspired by recent results in parametric active learning with abstention (Puchkin and Zhivotovskiy, 2021; Zhu and Nowak, 2022), we develop an oracle-efﬁcient algorithm showing that deep active learning provably achieves polylog( 1 ε ) label complexity when equipped with an abstention option (Chow, 1970).

Original: Such learning paradigm not onlyguarantees an exponential improvement in label complexity ( without any low noise assumptions ), butis also highly practical: In real-world scenarios such as medical imaging, it makes more sense forthe classiﬁer to abstain from making prediction on hard examples (e.g., those that are close totheboundary), and leave the decision-making process to a medical expert.
Revised: Our algorithm not only achieves an exponential saving in label complexity ( without any low noise assumptions ), but is also highly practical: In real-world scenarios such as medical imaging, it makes more sense for the classiﬁer to abstain from making prediction on hard examples (e.g., those that are close to the boundary), and ask medical experts to make the judgments.

Original: 1.1 Problem setting
Revised: 1.1 Problem setting

Original: Let X denote the instance space and Y denote the label space.
Revised: Let X denote the instance space and Y denote the label space.

Original: We focus on the binary classiﬁcationproblem where Y =
Revised: We focus on the binary classiﬁcation problem where Y :=

Original:  +1 , − 1 } .
Revised: { +1 , − 1 } .

Original: The joint distribution over X × Y is denoted as D XY .
Revised: The joint distribution over X × Y is denoted as D XY .

Original: We use
Revised: We use

Original: D X to denote the marginal distribution over the instance space X , and use D Y| x to denote the69 conditional distribution of
Revised: D X to denote the marginal distribution over the instance space X , and use D Y| x to denote the conditional distribution of

Original: Y with respect to any x ∈ X .
Revised: Y with respect to any x ∈ X .

Original: We consider the standard active learning70 setup where x ∼ D X but its label y ∼ D Y| x is only observed after issuing a label query.
Revised: We consider the standard active learning setup where x ∼ D X but its label y ∼ D Y| x is only observed after issuing a label query.

Original: We71 deﬁne η ( x ) :=
Revised: We deﬁne η ( x ) :=

Original: P y ∼D Y| x
Revised: P y ∼D Y| x

Original: ( y = +1)
Revised: ( y = +1)

Original: as the conditional probability of taking a positive label.
Revised: as the conditional probability of taking a positive label.

Original: The
Revised: The

Original: Bayes optimal classiﬁer h ? can thus be expressed as h ?
Revised: Bayes optimal classiﬁer h ? can thus be expressed as h ?

Original: ( x ) :=
Revised: ( x ) :=

Original: sign(2 η ( x ) − 1) .
Revised: sign(2 η ( x ) − 1) .

Original: For any classiﬁerh : X → Y , its (standard) error is calculated as err( h ) :
Revised: For any classiﬁer h : X → Y , its (standard) error is calculated as err( h ) :

Original: = P ( x,y )
Revised: = P ( x,y )

Original: ∼D XY ( h ( x ) 6 = y ) ; and its (standard)excess error is deﬁned as excess ( h ) :
Revised: ∼D XY ( h ( x ) 6 = y ) ; and its (standard) excess error is deﬁned as excess ( h ) :

Original: = err( h ) − err( h ? ) .
Revised: = err( h ) − err( h ? ) .

Original: Our goal is to learn an accurate classiﬁerwith a small amount of label querying.
Revised: Our goal is to learn an accurate classiﬁer with a small number of label querying.

Original: The nonparametric setting.
Revised: The nonparametric setting.

Original: We consider the nonparametric setting where the conditional probability η is characterized by a smooth function.
Revised: We consider the nonparametric setting where the conditional probability η is characterized by a smooth function.

Original: Fix any α ∈ N + , the Sobolev norm of a functionf : X → R is deﬁned as k f k W α, ∞ := max α, | α |≤ α ess sup x ∈X | D α f ( x ) | , where α = ( α 1 , . . . , α d ) ,| α | =
Revised: Fix any α ∈ N + , the Sobolev norm of a function f : X → R is deﬁned as k f k W α, ∞ := max α, | α |≤ α ess sup x ∈X | D α f ( x ) | , where α = ( α 1 , . . . , α d ) , | α | =

Original: P di =

Revised: P di =

Original: α
Revised: α

Original: i and D α f denotes the standard α -th weak derivative of f .
Revised: i and D α f denotes the standard α -th weak derivative of f .

Original: The unit ball in the Sobolevspace is deﬁned as W α, ∞( X ) :
Revised: The unit ball in the Sobolev space is deﬁned as W α, ∞( X ) :

Original: = { f : k f k W α, ∞ ≤ 1 } .
Revised: = { f : k f k W α, ∞ ≤ 1 } .

Original: Following the convention of nonparametric81 active learning (Castro and Nowak, 2008; Minsker, 2012; Locatelli et al., 2017, 2018; Shekhar et al. ,2021; Kpotufe et al., 2021), we assume X = [0 , 1] d and η ∈ W α, ∞( X ) (except in Section 4).
Revised: Following the convention of nonparametric active learning (Castro and Nowak, 2008; Minsker, 2012; Locatelli et al., 2017, 2018; Shekhar et al., 2021; Kpotufe et al., 2021), we assume X = [0 , 1] d and η ∈ W α, ∞( X ) (except in Section 4).

Original: Neural Networks.
Revised: Neural Networks.

Original: We consider feedforward neural networks with ReLU (Rectiﬁed Linear Unit)activation function, which is deﬁned as ReLU ( x ) :
Revised: We consider feedforward neural networks with Rectiﬁed Linear Unit (ReLU) activation function, which is deﬁned as ReLU ( x ) :

Original: = max { x, 0 } .
Revised: = max { x, 0 } .

Original: Each neural network f dnn : X → Rconsists of several input units (which corresponds to the covariates of x ∈ X ), one output unit(which corresponds to the prediction in R ), and multiple hidden computational units.
Revised: Each neural network f dnn : X → R consists of several input units (which corresponds to the covariates of x ∈ X ), one output unit (which corresponds to the prediction in R ), and multiple hidden computational units.

Original: Each hiddencomputational unit takes inputs { x i } Ni =1 (which are outputs from previous layers) and perform thecomputation ReLU (
Revised: Each hidden computational unit takes inputs { x i } Ni =1 (which are outputs from previous layers) and perform the computation ReLU (

Original: P
Revised: P

Original: Ni =1 w i x i + b ) with adjustable parameters { w i } Ni =1 and b ; The output unit89 performs the same operation, but without the ReLU nonlinearity.
Revised: Ni =1 w i x i + b ) with adjustable parameters { w i } Ni =1 and b ; the output unit performs the same operation, but without the ReLU nonlinearity.

Original: We use W to denote the totalnumber of parameters of a neural network, and L to denote the depth of the neural network.
Revised: We use W to denote the total number of parameters of a neural network, and L to denote the depth of the neural network.

Original: 1.2 Contributions and paper organization
Revised: 1.2 Contributions and paper organization

Original: Neural networks are known to be universal approximators (Cybenko, 1989; Hornik, 1991).
Revised: Neural networks are known to be universal approximators (Cybenko, 1989; Hornik, 1991).

Original: In thispaper, we argue that, in both passive and active regimes, the universal approximatability makes neuralnetworks “universal classiﬁers” for classiﬁcation problems: With an appropriately chosen set ofneural networks, one can recover known minimax rates (up to disagreement coefﬁcient in the activesetting) in the rich nonparametric regimes.
Revised: In this paper, we argue that, in both passive and active regimes, the universal approximatability makes neural networks “universal classiﬁers” for classiﬁcation problems: With an appropriately chosen set of neural networks, one can recover known minimax rates (up to disagreement coefﬁcients in the active setting) in the rich nonparametric regimes.

Original: 1 We provide information statements of our main results inthe sequel, with detailed statements and associated deﬁnitions/algorithms deferred to later sections.
Revised: 1 We provide informal statements of our main results in the sequel, with detailed statements and associated deﬁnitions/algorithms deferred to later sections.

Original: In Section 2, we analyze the label complexity of deep active learning under the common Tsybakovnoise condition with smoothness parameter β ≥ 

Revised: In Section 2, we analyze the label complexity of deep active learning under the standard Tsybakov noise condition with smoothness parameter β ≥

Original: (Tsybakov, 2004).
Revised: (Tsybakov, 2004).

Original: Let H dnn be an appropriatelychosen set of neural network classiﬁers and denote θ H dnn ( ε ) as the disagreement coefﬁcient (Hanneke,2007, 2014) at level ε .
Revised: Let H dnn be an appropriately chosen set of neural network classiﬁers and denote θ H dnn ( ε ) as the disagreement coefﬁcient (Hanneke, 2007, 2014) at level ε .

Original: We develop the following label complexity guarantee for deep active learning.
Revised: We develop the following label complexity guarantees for deep active learning.

Original: Theorem 1 (Informal) .
Revised: Theorem 1 (Informal) .

Original: There exists an algorithm that returns a neural network classiﬁer
Revised: There exists an algorithm that returns a neural network classiﬁer

Original: [Equation]
Revised: [Equation]

Original: dnnwith excess error e
Revised: with excess error e

Original: O ( ε ) after querying
Revised: O ( ε ) after querying

Original: [Equation]
Revised: [Equation]

Original: The label complexity presented in Theorem 1 matches the active learning lower bound Ω( ε − d [-+
-]
Revised: The label complexity presented in Theorem 1 matches the active learning lower bound Ω( ε − d [-+
-]+

Original:  α α + αβ )(Locatelli et al., 2017) up to the dependence on the disagreement coefﬁcient (and other logarithmicfactors).
Revised: [Equation] α α + αβ ) (Locatelli et al., 2017) up to the dependence on the disagreement coefﬁcient (and other logarithmic factors).

Original: Since θ H dnn ( ε ) ≤1 ε  by deﬁnition, the label complexity presented in Theorem 1 is never worsethan the passive learning ratese Θ
Revised: Since θ H dnn ( ε ) ≤ ε − 1 by deﬁnition, the label complexity presented in Theorem 1 is never worse than the passive learning rates

Original: [Equation]
Revised: [Equation]

Revised: ( ε − d +2 α

Original:  ) (Audibert and Tsybakov, 2007).
Revised: + αβ α + αβ ) (Audibert and Tsybakov, 2007).

Original: We also discoverconditions under which the disagreement coefﬁcient with respect to a set of neural network classiﬁerscan be properly bounded: θ H dnn ( ε ) =
Revised: We also discover conditions under which the disagreement coefﬁcient with respect to a set of neural network classiﬁers can be properly bounded, i.e., θ H dnn ( ε ) =

Original: o ( ε  ) (implying strict improvement over passive learning) andθ H dnn ( ε ) =
Revised: o ( ε − 1 ) (implying strict improvement over passive learning) and θ H dnn ( ε ) =

Original: o (1) (implying matching active learning lower bound).
Revised: o (1) (implying matching active learning lower bound).

Original: In Section 3, we develop label complexity guarantee for deep active learning when an additionalabstention option is allowed (Chow, 1970; Puchkin and Zhivotovskiy, 2021; Zhu and Nowak, 2022).
Revised: In Section 3, we develop label complexity guarantees for deep active learning when an additional abstention option is allowed (Chow, 1970; Puchkin and Zhivotovskiy, 2021; Zhu and Nowak, 2022).

Original: Suppose a cost (e.g. 0 . 49 ) that is marginally smaller than random guessing (which has expected cost0 . 5 ) is incurred whenever the classiﬁer abstains from  predication, we have the following guaranteefor learning with a set of neural networks.
Revised: Suppose a cost (e.g. 0 . 49 ) that is marginally smaller than random guessing (which has expected cost 0 . 5 ) is incurred whenever the classiﬁer abstains from making a predication, we develop the following label complexity guarantees for deep active learning.

Original: Theorem 2 (Informal) .
Revised: Theorem 2 (Informal) .

Original: There exists an efﬁcient algorithm that constructs a neural network classiﬁerb h dnn with Chow’s excess error e
Revised: There exists an efﬁcient algorithm that constructs a neural network classiﬁer b h dnn with Chow’s excess error e

Original: O ( ε ) after querying polylog( 1 ε ) labels.
Revised: O ( ε ) after querying polylog( 1 ε ) labels.

Original: The above polylog( 1 ε ) label complexity bound is achieved without any low noise assumptions .
Revised: The above polylog( 1 ε ) label complexity bound is achieved without any low noise assumptions .

Original: Suchexponential improvement theoretically justify the great empirical performance of deep active learningobserved in practice (e.g., in Sener and Savarese (2018)):
Revised: Such exponential label savings theoretically justify the great empirical performances of deep active learning observed in practice (e.g., in Sener and Savarese (2018)):

Original: It sufﬁces to label a few data points toachieve a high accuracy level.
Revised: It sufﬁces to label a few data points to achieve a high accuracy level.

Original: Moreover,  our algorithm (Algorithm 2) developed for Theorem 2 canbe efﬁciently implemented in e
Revised: Moreover, apart from an initialization step, our algorithm (Algorithm 4) developed for Theorem 2 can be efﬁciently implemented in e

Original: O (1 ε  ) time, given a least square regression oracle over a set of neuralnetworks (which can be approximated by running stochastic gradient descent in practice).
Revised: O ( ε − 1 ) time, given a convex loss regression oracle over an appropriately chosen set of neural networks; in practice, the regression oracle can be approximated by running stochastic gradient descent.

Original: Technical contributions.
Revised: Technical contributions.

Original: Besides identifying the “right tools” (ranging from approximationresults (Yarotsky, 2017, 2018) to complexity analyses (Bartlett et al., 2019)) to analyze deep activelearning, our theoretical guarantees are empowered by novel extensions of active learning algorithmsunder neural network approximations .
Revised: Besides identifying the “right tools” (ranging from approximation results (Yarotsky, 2017, 2018) to complexity analyses (Bartlett et al., 2019)) to analyze deep active learning, our theoretical guarantees are empowered by novel extensions of active learning algorithms under neural network approximations .

Original: In particular, our results extend the understanding of howapproximation affects active learningto achieve fast rates (e.g., under Tsybakov noise in Theorem 1),and we identify conditions that greatly relax the(previous) approximation requirement in the learningwith abstention setup (to develop Theorem 2).
Revised: In particular, we deal with approximation error in active learning under Tsybakov noise, and identify conditions that greatly relax the approximation requirement in the learning with abstention setup; we also analyze the disagreement coefﬁcient, both classiﬁer-based and value function-based, with a set of neural networks. These analyses together lead to our main results for deep active learning (e.g., Theorem 1 and Theorem 2).

Original: We also provide our active learning guarantees(underapproximation) ingeneric forms (in Appendix B  and Appendix D), which can be of independentinterests. Other technical contributions including analysis of disagreement coefﬁcient with a setof neural networks (Appendix C.2) and an important deep active learning lower bound (Theorem 8). We defer all proofs to the appendix due to lack of space.
Revised: More generally, we establish a bridge between approximation theory and active learning; we provide these general guarantees in Appendix B (under Tsybakov noise) and Appendix D (with the abstention option), which can be of independent interests.

Original: Beneﬁted from our generic approaches/analysis of deep active learning, in Section 4, we extend ourresults into learning smooth functions in the Radon BV 2 space (Ongie et al., 2020; Parhi and Nowak,2021a,b; Unser, 2022), which is recently proposed as a natural space to analyze neural networks.
Revised: Beneﬁted from these generic algorithms and guarantees, in Section 4, we extend our results into learning smooth functions in the Radon BV 2 space (Ongie et al., 2020; Parhi and Nowak, 2021, 2022a,b; Unser, 2022), which is recently proposed as a natural space to analyze neural networks.

Original: 1.3 Related work
Revised: 1.3 Related work

Original: Active learning concerns about learning accurate classiﬁers without extensive human labeling.
Revised: Active learning concerns about learning accurate classiﬁers without extensive human labeling.

Original: Oneof the earliest work of active learning dates back to the CAL algorithm proposed by Cohn et al.(1994), which set the cornerstone for disagreement-based active learning.
Revised: One of the earliest work of active learning dates back to the CAL algorithm proposed by Cohn et al. (1994), which set the cornerstone for disagreement-based active learning.

Original: Since then, a long lineof work have been developed, either directly working with a set classiﬁer (Hanneke, 2007; Balcanet al., 2007, 2009; Zhang and Chaudhuri, 2014; Huang et al., 2015) or work with a set of regressionfunctions (Krishnamurthy et al., 2017, 2019).
Revised: Since then, a long line of work have been developed, either directly working with a set classiﬁer (Balcan et al., 2006; Hanneke, 2007; Dasgupta et al., 2007; Beygelzimer et al., 2009, 2010; Huang et al., 2015; Cortes et al., 2019) or work with a set of regression functions (Krishnamurthy et al., 2017, 2019).

Original: These work mainly focus on the parametric regime(e.g., learning with a set of linear classiﬁers), and their label complexities rely on the boundednessof the so-called disagreement coefﬁcient (Hanneke, 2007; Friedman, 2009; Hanneke, 2014).
Revised: These work mainly focus on the parametric regime (e.g., learning with a set of linear classiﬁers), and their label complexities rely on the boundedness of the so-called disagreement coefﬁcient (Hanneke, 2007, 2014; Friedman, 2009).

Original: Activelearning in the nonparametric regime has been analyzed in Castro and Nowak (2008); Minsker (2012); Locatelli et al.
Revised: Active learning in the nonparametric regime has been analyzed in Castro and Nowak (2008); Minsker (2012); Locatelli et al.

Original: ; Kpotufe et al.
Revised: ; Kpotufe et al.

Original: These algorithms rely on partitioning of the inputspace X ⊆ [0 , 1] d into exponentially (in dimension) many small cubes, and then conduct mean(orhigher-order statistics) estimation within each small cube.
Revised: These algorithms rely on partitioning of the input space X ⊆ [0 , 1] d into exponentially (in dimension) many small cubes, and then conduct local mean (or some higher-order statistics) estimation within each small cube.

Original: It is well known that, in the worst case, active learning exhibits no label complexity gains overthe passive counterpart (Kääriäinen, 2006).
Revised: It is well known that, in the worst case, active learning exhibits no label complexity gains over the passive counterpart (Kääriäinen, 2006).

Original: To bypass these worst-case scenarios, active learninghas been popularly analyzed under the  Tsybakov low noise conditions (Tsybakov, 2004).
Revised: To bypass these worst-case scenarios, active learning has been popularly analyzed under the so-called Tsybakov low noise conditions (Tsybakov, 2004).

Original: Under

Original:  Tsybakov noise conditions, active learning has been shown to be strictly superior than passivelearning in terms of label complexity (Castro and Nowak, 2008; Locatelli et al., 2017).
Revised: Under Tsybakov noise conditions, active learning has been shown to be strictly superior than passive learning in terms of label complexity (Castro and Nowak, 2008; Locatelli et al., 2017).

Original: Besidesanalyzing active learning under favorable low noise assumptions, more recently, researchersconsider active learning with an abstention option and analyze its label complexity under Chow’serror (Chow, 1970).
Revised: Besides analyzing active learning under favorable low noise assumptions, more recently, researchers consider active learning with an abstention option and analyze its label complexity under Chow’s error (Chow, 1970).

Original: More speciﬁcally, Puchkin and Zhivotovskiy (2021); Zhu and Nowak (2022)analyze active learning under Chow’s excess error in the parametric regime, and show that onecan alway polylog( 1 ε ) label complexity.
Revised: In particular, Puchkin and Zhivotovskiy (2021); Zhu and Nowak (2022) develop active learning algorithms with polylog( 1 ε ) label complexity when analyzed under Chow’s excess error.

Original: Shekhar et al.
Revised: Shekhar et al.

Original: (2021) study nonparametric active learningunder aslightly different version of the Chow’s excess error, and propose algorithms with poly( 1 ε )label complexity; their algorithm follows similar logic as previous nonparametric active learningalgorithms (e.g., Minsker (2012);Locatelli et al.
Revised: (2021) study nonparametric active learning under a different notion of the Chow’s excess error, and propose algorithms with poly( 1 ε ) label complexity; their algorithms follow similar procedures of those partitionbased nonparametric active learning algorithms (e.g., Minsker (2012);

Original: and is partition-based.
Revised: Locatelli et al.

Original: Inspired by the success of deep learning in the passive regime, active learning with deep neuralnetworks has been extensively explored in recent years (Sener and Savarese, 2018; Ash et al., 2019; Citovsky et al., 2021; Ash et al., 2021; Kothawade et al., 2021; Emam et al., 2021;
Revised: Inspired by the success of deep learning in the passive regime, active learning with neural networks has been extensively explored in recent years (Sener and Savarese, 2018; Ash et al., 2019; Citovsky et al., 2021; Ash et al., 2021; Kothawade et al., 2021; Emam et al., 2021;

Original: Ren et al., 2021).
Revised: Ren et al., 2021).

Original: Great empirical performances are observed in these papers, however, rigorous label complexityguarantees have largely remains elusive (except in Karzand and Nowak (2020);
Revised: Great empirical performances are observed in these papers, however, rigorous label complexity guarantees have largely remains elusive (except in Karzand and Nowak (2020);

Original: Wang et al.
Revised: Wang et al.

Original: (2021)with limitations discussed before).
Revised: (2021), with limitations discussed before).

Original: We ﬁll the gap between practice and theory by providing theﬁrst near minimax optimal label complexity guarantees for deep active learning.
Revised: We bridge the gap between practice and theory by providing the ﬁrst near-optimal label complexity guarantees for deep active learning.

Original: Our results are builtupon approximation results of deep neural networks (Yarotsky, 2017, 2018; Parhi and Nowak, 2021b)and VC/Pesudo dimension analyses of neural networks of a given structure (Bartlett et al., 2019).
Revised: Our results are built upon approximation results of deep neural networks (Yarotsky, 2017, 2018; Parhi and Nowak, 2022b) and VC/pseudo dimension analyses of neural networks with given structures (Bartlett et al., 2019).

Original:  Label complexity of deep active learning
Revised:  Label complexity of deep active learning

Original: Werigorously analyze the label complexity of deep active learning in this section.
Revised: We analyze the label complexity of deep active learning in this section.

Original: We ﬁrstintroduce the Tsybakov noise condition in Section 2.1, and then identify the “right tools” to analyzeclassiﬁcation problems with neural network classiﬁers in Section 2.2 (where we also provide passivelearning guarantees).
Revised: We ﬁrst introduce the Tsybakov noise condition in Section 2.1, and then identify the “right tools” to analyze classiﬁcation problems with neural network classiﬁers in Section 2.2 (where we also provide passive learning guarantees).

Original: Our main active learning guaranteesare provided in Section 2.3, where wealso discuss importance implications of our results.
Revised: We establish our main active learning guarantees in Section 2.3.

Original: 2.1 Tsybakov noise condition
Revised: 2.1 Tsybakov noise condition

Original: It is well known that active learning exhibit no label complexity gains over the passive counterpartwithout additional low noise assumptions (Kääriäinen, 2006).
Revised: It is well known that active learning exhibits no label complexity gains over the passive counterpart without additional low noise assumptions (Kääriäinen, 2006).

Original: We  introduce the Tsybokov low noisecondition (Tsybakov, 2004), which has been extensively analyzed in active learning literature.
Revised: We next introduce the Tsybokov low noise condition (Tsybakov, 2004), which has been extensively analyzed in active learning literature.

Original: Deﬁnition 1 (Tsybakov noise) .
Revised: Deﬁnition 1 (Tsybakov noise) .

Original: The marginal distribution D X satisﬁes the Tsybakov noise condition184 with parameter β ≥ 0 and universal constants c ≥ 1 if, ∀ t > 0 ,
Revised: The marginal distribution D X satisﬁes the Tsybakov noise condition with parameter β ≥ 0 and universal constants [Equation] if

Original: [Equation]
Revised: [Equation]

Original: The case with β = 0 corresponds to the general case without any low noise conditions, where noactive learning algorithm can outperform the passive counterpart (Audibert and Tsybakov, 2007;
Revised: The case with β = 0 corresponds to the general case without any low noise conditions, where no active learning algorithm can outperform the passive counterpart (Audibert and Tsybakov, 2007;

Original: Locatelli et al., 2017).
Revised: Locatelli et al., 2017).

Original: We use P ( α, β ) to denote the set of distributions satisfying: (1) the smoothnessconditions introduced in Section 1.1 with parameter α > 0 ; and (2) the Tsybakov low noise condition(i.e. , Deﬁnition 1) with parameter β ≥ 0 .
Revised: We use P ( α, β ) to denote the set of distributions satisfying: (i) the smoothness conditions introduced in Section 1.1 with parameter α > 0 ; and (ii) the Tsybakov low noise condition (i.e., Deﬁnition 1) with parameter β ≥ 0 .

Original: We assume D XY ∈ P ( α, β ) in the rest of Section 2.
Revised: We assume D XY ∈ P ( α, β ) in the rest of Section 2.

Original: As in Castro and Nowak (2008); Hanneke (2014), we assume the knowledge of noise/smoothnessparameters.
Revised: As in Castro and Nowak (2008); Hanneke (2014), we assume the knowledge of noise/smoothness parameters.

Original: 2.2 Approximation and expressiveness of neural networks
Revised: 2.2 Approximation and expressiveness of neural networks

Original: Neural networks are known to be universal approximators (Cybenko, 1989; Hornik, 1991):
Revised: Neural networks are known to be universal approximators (Cybenko, 1989; Hornik, 1991):

Original: For anycontinuous function g : X → R and any error tolerance κ > 0 , there exists a large enough neuralnetwork f dnn such that k f dnn − g k ∞ :
Revised: For any continuous function g : X → R and any error tolerance κ > 0 , there exists a large enough neural network f dnn such that k f dnn − g k ∞ :

Original: = sup x ∈X | f dnn ( x ) − g ( x ) | ≤ κ .
Revised: = sup x ∈X | f dnn ( x ) − g ( x ) | ≤ κ .

Original: Recently, non-asympototic196 approximation rates by ReLU neural networks have been developed for smooth functions in the
Revised: Recently, non-asympototic approximation rates by ReLU neural networks have been developed for smooth functions in the

Original: Sobolev space, which we re-state in the following.
Revised: Sobolev space, which we restate in the following.

Original: Theorem 3 (Yarotsky (2017)) .
Revised: Theorem 3 (Yarotsky (2017)) .

Original: Fix any κ > 0 .
Revised: Fix any κ > 0 .

Original: For any f ? ≡ η ∈ W α, ∞([ − 1 , 1] d ) , there exists aneural network f dnn with W
Revised: For any f ? = η ∈ W α, ∞([ − 1 , 1] d ) , there exists a neural network f dnn with W

Original: = O ( κ − dα log 1 κ ) total number of parameters arranged in L
Revised: = O ( κ − dα log 1 κ ) total number of parameters arranged in L

Original: = O (log 1 κ )layers such that k f dnn − f ? k ∞ ≤ κ .
Revised: = O (log 1 κ ) layers such that k f dnn − f ? k ∞ ≤ κ .

Original: The architecture of the neural network f dnn appearing in the above theorem is independent off ? .
Revised: The architecture of the neural network f dnn appearing in the above theorem only depends on the smooth function space W α, ∞ [Equation]

Revised: , but otherwise is independent of the true regression function f ? ; also see Yarotsky (2017) for details.

Original: Let F dnn denote the set of neural network regression functions with the same architecture.
Revised: Let F dnn denote the set of neural network regression functions with the same architecture.

Original: We construct a set of neural network classiﬁers by thresholding the regression function at 12 , i.e., 
Revised: We construct a set of neural network classiﬁers by thresholding the regression function at 12 , i.e., H dnn :=

Original: H dnn := { h f :=
Revised: { h f :=

Original: sign(2 f ( x ) − 1) : f ∈ F dnn } .
Revised: sign(2 f ( x ) − 1) : f ∈ F dnn } .

Original: The next results concerns about the expressiveness205 of the neural network classiﬁers, in terms of a well-known complexity measure: the VC dimension(Vapnik and Chervonenkis, 1971).
Revised: The next result concerns about the expressiveness of the neural network classiﬁers, in terms of a well-known complexity measure: the VC dimension (Vapnik and Chervonenkis, 1971).

Original: Theorem 4 (Bartlett et al.
Revised: Theorem 4 (Bartlett et al.

Original: Let H dnn be a set of neural network classiﬁers of the samearchitecture and with W parameters arranged in L layers.
Revised: Let H dnn be a set of neural network classiﬁers of the same architecture and with W parameters arranged in L layers.

Original: We then have
Revised: We then have

Original: [Equation]
Revised: [Equation]

Original: With theserecently established tools, we can construct a set of neural network classiﬁers H dnn suchthat (1) the best in-class classiﬁer ˇ h ∈ H dnn has small excess error; and (2) H dnn has  well-controlled VC dimension that is proportional to smooth/noise parameters.
Revised: With these tools, we can construct a set of neural network classiﬁers H dnn such that (i) the best in-class classiﬁer ˇ h ∈ H dnn has small excess error, and (ii) H dnn has a well-controlled VC dimension that is proportional to smooth/noise parameters.

Original: More speciﬁcally, we have thefollowing proposition.
Revised: More speciﬁcally, we have the following proposition.

Original: Proposition 1.
Revised: Proposition 1.

Original: Suppose D XY ∈ P
Revised: Suppose D XY ∈ P

Original: ( α, β ) .
Revised: ( α, β ) .

Original: One can construct a set of neural network classiﬁer H dnn214 such that the following two properties hold simultaneously:
Revised: One can construct a set of neural network classiﬁer H dnn such that the following two properties hold simultaneously:

Original: [Equation]
Revised: [Equation]

Original: With the approximation results obtained above, to learn a classiﬁer with O ( ε ) excess error, one onlyneeds to focus on a set of neural networks H dnn with  well-controlled VC dimension.
Revised: With the approximation results obtained above, to learn a classiﬁer with O ( ε ) excess error, one only needs to focus on a set of neural networks H dnn with a well-controlled VC dimension.

Original: As a warm-up,we ﬁrst analyze the label complexity of such procedure in the passive regime (with fast rate).
Revised: As a warm-up, we ﬁrst analyze the label complexity of such procedure in the passive regime (with fast rates).

Original: Theorem 5.
Revised: Theorem 5.

Original: Suppose D XY ∈ P
Revised: Suppose D XY ∈ P

Original: ( α, β ) .
Revised: ( α, β ) .

Original: Fix any ε, δ > 0 .
Revised: Fix any ε, δ > 0 .

Original: Let H dnn be the set of neural networkclassiﬁers constructed in Proposition 1.
Revised: Let H dnn be the set of neural network classiﬁers constructed in Proposition 1.

Original: Withn = e
Revised: With

Original: O ( ε − d +2 α + αβ α (1+ β ) )
Revised: [Equation]

Original: i.i.d.
Revised: i .d.

Original: sampled points, withprobability at least 1 − δ , the empirical risk minimizer b h ∈ H dnn achieves excess error O( ε ) .
Revised: sampled points, with probability at least 1 − δ , the empirical risk minimizer [Equation] achieves excess error O

Original:  The label complexity results obtained in Theorem 5 matches, up to logarithmic factors, the passivelearning lower bound Ω( ε − d +2 α + αβ α (1+ β ) ) established in Audibert and Tsybakov (2007), indicating our proposed learning procedure with a set of neural networks is  minimax optimal (up to logarithmic factors).
Revised: [Equation] ε ) . The label complexity results obtained in Theorem 5 matches, up to logarithmic factors, the passive learning lower bound Ω( ε − d +2 α + αβ α (1+ β ) ) established in Audibert and Tsybakov (2007), indicating our proposed learning procedure with a set of neural networks is near minimax optimal.

Original: 2.3 Active learning andlabel complexity guarantees
Revised: 2.3 Deep active learning and guarantees

Original: The passive learning procedure presented in the previous section treats every label as equally important, i.e., it requests the label of every data point.
Revised: The passive learning procedure presented in the previous section treats every data point equally, i.e., it requests the label of every data point.

Original: Active learning reduces the label complexity byonly querying labels of data points that are “more important”.
Revised: Active learning reduces the label complexity by only querying labels of data points that are “more important”.

Original: We present deep active learning resultswith the RobustCAL algorithm
Revised: We present deep active learning results in this section.

Original:  (Balcan et al., 2009; Hanneke, 2007, 2014), which is built upon theseminal CAL algorithm by Cohn et al.
Revised: Our algorithm (Algorithm 1) is inspired by RobustCAL (Balcan et al., 2006; Hanneke, 2007, 2014) and the seminal CAL algorithm (Cohn et al., 1994); we call our algorithm NeuralCAL to emphasize that it works with a set of neural networks. For any accuracy level ε > 0 , NeuralCAL ﬁrst initialize a set of neural network classiﬁers H 0 := H dnn such that (i) the best in-class classiﬁer ˇ h := arg min h ∈H dnn err( h ) has excess error at most O ( ε ) , and (ii) the VC dimension of H dnn is upper bounded by

Original: We now introduce RobustCAL (Algorithm 1), which runs in epochs of geometrically increasinglengths.
Revised: [Equation] ) (see Section 2.2 for more details). NeuralCAL then runs in epochs of geometrically increasing lengths.

Original: At the beginning of epoch m , based on previously labeled data points, RobustCAL updates aset of active classiﬁer H m such that, with high probability, the best classiﬁer
Revised: At the beginning of epoch m , based on previously labeled data points, NeuralCAL updates a set of active classiﬁer H m such that, with high probability, the best classiﬁer

Original: ˇ h := arg min h ∈H err( h )remains uneliminated .
Revised: ˇ h remains uneliminated .

Original: Within each epoch m , RobustCAL only queries the label y of a data pointx if it lies in the region of disagreement with respect to the current active set of classiﬁer H m , i.e., DIS ( H m ) :
Revised: Within each epoch m , NeuralCAL only queries the label y of a data point x if it lies in the region of disagreement with respect to the current active set of classiﬁer H m , i.e., DIS ( H m ) :

Original: = { x ∈ X : ∃ h 1 , h 2 ∈
Revised: = { x ∈ X : ∃ h 1 , h 2 ∈

Original: H m s.t.
Revised: H m s.t.

Original: h 1 ( x ) 6 =
Revised: h 1 ( x ) 6 =

Original: h 2 ( x ) } .
Revised: h 2 ( x ) } .

Original: RobustCAL returns any classiﬁerb h ∈ H m that remains uneliminated after M − 1 epoch.
Revised: NeuralCAL returns any classiﬁer b h ∈ H m that remains uneliminated after M − 1 epoch.

Original: Input: Accuracy level ε ∈ (0 , 1) , conﬁdence level δ ∈ (0 , 1) , and an appropriately chosen set of neural network classiﬁers H ≡ H dnn .
Revised: Input: Accuracy level ε ∈ (0 , 1) , conﬁdence level δ ∈ (0 , 1) . 1: Let H dnn be a set of neural networks classiﬁers constructed in Proposition 1.

Original: 1: Set T = ε − 2+ β 1+ β
Revised: 2: Deﬁne

Original: VCdim( H ) , and M =
Revised: [Equation]

Original:  d log 2 T e . 2: Deﬁne τ m := 2 m for m ≥ 1 and τ 0 = 0[-; H 
 = H-] .
Revised: := d log 2 T e , τ m := 2 m for m ≥ 1 and τ 0 := 0[-; H 
 = H-] .

Original: 3: Deﬁne ρ m :=
Revised: 3: Deﬁne ρ m :=

Original: O (cid:18)(cid:16)
Revised: O (cid:18)(cid:16)

Original: VCdim( H ) · log( τ m − 1 ) · log( M/δ )
Revised: VCdim( [Table] dnn ) · log( [Table]

Revised: m

Revised: [Table]

Revised: ) · log( M/

Original:  τ m −(cid:17) 1+ β[-
-]
Revised: [Table] ) τ m −(cid:17) [Table] β[-
-]

Original: + β (cid:19) for m ≥ 2 and ρ 1 := 1 .
Revised: [Table] β (cid:19) for m ≥ 2 and ρ 1 := 1 .

Original: 4: Deﬁne b
Revised: 4: Deﬁne b

Original: R m ( h ) :=
Revised: R m ( h ) :=

Original: P τ m − 1 t =1 Q t 1 ( h ( x t ) 6 = y t ) with the convention that
Revised: P τ m − 1 t =1 Q t 1 ( h ( x t ) 6 = y t ) with the convention that

Original: P 0 t =1 . . .
Revised: P 0 t =1 . . .

Original: = 0 .
Revised: = 0 .

Revised: 5: Initialize H 0 :=

Revised: H dnn .

Original: 5: for epoch m = 1 , 2 , . .
Revised: 6: for epoch m = 1 , 2 , . .

Original: , M do 6:
Revised: , M do 7:

Original: Update active set
Revised: Update active set

Original: H m := n h ∈ H m − 1 : b
Revised: H m := n [Equation] H m [Equation]

Original:  R m ( h )≤ inf h ∈H m −b
Revised: : b [Equation] m −b R m ( h )

Original: R m ( h ) + τ m − 1 · ρ m o 7: if epoch m =
Revised: [Equation] τ [Equation] m o 8: if epoch m =

Original: M then 8:
Revised: M then 9:

Original: Return any classiﬁer b h ∈ H M . 9: for time t =
Revised: Return any classiﬁer b h ∈ H M . 10: for time t =

Original: τ m − 1 + 1 , . .
Revised: τ m − 1 + 1 , . .

Original: , τ m do 10:
Revised: , τ m do 11:

Original: Observe x t ∼ D X .
Revised: Observe x t ∼ D X .

Original: Set Q t = 1 ( x t ∈ DIS ( H m )) .
Revised: Set Q t := 1 ( x t ∈ DIS ( H m )) .

Original: 11: if Q t = 1 then 12:
Revised: 12: if Q t = 1 then 13:

Original: Query the label y t of x t .
Revised: Query the label y t of x t .

Original: Since RobustCAL only queries labels of data points lying in the region of disagreement, its label complexity should intuitively be related to how fast the region of disagreement shrinks.
Revised: Since NeuralCAL only queries labels of data points lying in the region of disagreement, its label complexity should intuitively be related to how fast the region of disagreement shrinks.

Original: The rate of convergence is captured by the  disagreement coefﬁcient (Hanneke, 2007, 2014), which we deﬁned below.
Revised: More formally, the rate of collapse of the (probability measure of) region of disagreement is captured by the (classiﬁer-based) disagreement coefﬁcient (Hanneke, 2007, 2014), which we introduce next.

Original: Deﬁnition 2 (Disagreement coefﬁcient) .
Revised: Deﬁnition 2 (Classiﬁer-based disagreement coefﬁcient) .

Original: For any ε 0 and classiﬁer h ∈ H , the  disagreement coefﬁcientof h is deﬁned as[Equation] ∨
Revised: For any ε 0 and classiﬁer h ∈ H , the classiﬁer-based disagreement coefﬁcient of h is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where B H ( h, ε ) :
Revised: where B H ( h, ε ) :

Original: = { g ∈ H : P ( x ∈ X : g ( x ) 6 = h ( x )) ≤ ε } .
Revised: = { g ∈ H : P ( x ∈ X : g ( x ) 6 = h ( x )) ≤ ε } .

Original: We also deﬁne θ H ( ε 0 ) :
Revised: We also deﬁne θ H ( ε 0 ) :

Original: =243 sup h ∈H θ H ,h ( ε 0 ) .
Revised: = sup h ∈H θ H ,h ( ε 0 ) .

Original: While using the well-known RobustCAL algorithm, our results are established on novel analysis.
Revised: The guarantees of NeuralCAL follows from a more general analysis of RobustCAL under approximation.

Original: Inparticular, to achieve fast rate (under Tsybakov noise), previous analysis of RobustCAL requires thatthe Bayes classiﬁer is in the class (or a Bernstein condition for every h ∈ H )
Revised: In particular, to achieve fast rates (under Tsybakov noise), previous analysis of RobustCAL requires that the Bayes classiﬁer is in the class (or a Bernstein condition for every h ∈ H )

Original: (Hanneke, 2014).
Revised: (Hanneke, 2014).

Original: Theserequirements are stronger compared to what we have in the case with neural network approximations .
Revised: These requirements are stronger compared to what we have in the case with neural network approximations.

Original: Our analysis extends the understanding of RobustCAL under approximation (see Appendix B),which leads to  the following guarantees.
Revised: Our analysis extends the understanding of RobustCAL under approximation. We defer such general analysis to Appendix B, and present the following guarantees.

Original: Theorem 6.
Revised: Theorem 6.

Original: Suppose D XY ∈ P
Revised: Suppose D XY ∈ P

Original: ( α, β ) .
Revised: ( α, β ) .

Original: Fix any ε, δ > 0 .
Revised: Fix any ε, δ > 0 .

Original: Let H dnn be the set of neural network 251 classiﬁers constructed in Proposition 1.

Original: With probability at least− δ , Algorithm 1 returns aclassiﬁer b h ∈ H dnn with excess error e
Revised: With probability at least 1 − δ , Algorithmreturns a classiﬁer b h ∈ H dnn with excess error e

Original: O [Equation] ) after querying
Revised: O ( ε ) after querying

Original: [Equation]
Revised: [Equation]

Original: We next discuss  the label complexity of deep active learning achieved in Theorem 6.
Revised: We next discuss in detail the label complexity of deep active learning proved in Theorem 6.

Original: Ignoring the dependence on disagreement coefﬁcient, the label complexity appearing in Theorem 6matches, up to logarithmic factors, the lower bound Ω( ε − d +

Revised: Ignoring the dependence on disagreement coefﬁcient, the label complexity appearing in Theoremmatches, up to logarithmic factors, the lower bound Ω( ε − d +

Original: α α + αβ ) for active learning (Locatelliet al., 2017).
Revised: α α + αβ ) for active learning (Locatelli et al., 2017).

Original: At the same time, the label complexity appearing in Theorem 6 is never worse thanthe passive counterpart (i.e.,e
Revised: At the same time, the label complexity appearing in Theorem 6 is never worse than the passive counterpart (i.e.,

Original: [Equation]
Revised: [Equation]

Original: ε
Revised: θ

Original: [Equation]
Revised: [Equation]

Original: since θ H [Equation] [Equation] ε − β
Revised: dnn

Original: [Equation]
Revised: [Equation]

Original: β .• We also identify cases when θ H dnn ( ε β 1+ β )
Revised: • We also identify cases when θ H dnn ( ε β 1+ β )

Original: =
Revised: =

Original: o ( ε − β 1+ β ) , indicating strict improvement over passivelearning (e g., when D X is supported on countably many data points); and when θ H dnn ( ε β 1+ β ) =
Revised: o ( ε − β 1+ β ) , indicating strict improvement over passive learning (e.g., when D X is supported on countably many data points), and when θ H dnn ( ε β 1+ β ) =

Original: O (1) , indicating matching the minimax active lower bound (e.g., when D XY satisﬁes conditionssuch as decomposibility deﬁned in Deﬁnition 4.
Revised: O (1) , indicating matching the minimax active lower bound (e.g., when D XY satisﬁes conditions such as decomposibility deﬁned in Deﬁnition 4.

Original: See Appendix C.2 for detailed discussion).
Revised: See Appendix C.2 for detailed discussion).

Original: Our Theorem 6 has the following two important implications, which could beneﬁt both deep activelearning and nonparametric learning communities.
Revised: Our algorithm and theorems lead to the following results, which could beneﬁt both deep active learning and nonparametric learning communities.

Original: • Near minimax optimal label complexity for deep active learning.
Revised: • Near minimax optimal label complexity for deep active learning.

Original: While empirical successesof deep active learning have been observed, rigorous label complexity analysis remains elusiveexcept for two attempts made in Karzand and Nowak (2020);
Revised: While empirical successes of deep active learning have been observed, rigorous label complexity analysis remains elusive except for two attempts made in Karzand and Nowak (2020);

Original: Wang et al.
Revised: Wang et al.

Original: The guaranteesprovided in Karzand and Nowak (2020) only work in very special cases (i.e., data uniformlysampled from [0 , 1] and labeled by well-separated piece-constant functions in a noise-free way).Wang et al.
Revised: The guarantees provided in Karzand and Nowak (2020) only work in very special cases (i.e., data uniformly sampled from [0 , 1] and labeled by well-separated piece-constant functions in a noise-free way). Wang et al.

Original: (2021) study deep active learning in the NTK regime by linearizing the neural networkat its random initialization and analyzing it as a linear function; moreover, as the authors agree,their error bounds and label complexity guarantees are vacuous in certain cases.
Revised: (2021) study deep active learning in the NTK regime by linearizing the neural network at its random initialization and analyzing it as a linear function; moreover, as the authors agree, their error bounds and label complexity guarantees are vacuous in certain cases.

Original: On the other side,our guarantees are minimax optimal (up to disagreement coefﬁcient), which, for the ﬁrst time,close the gap between theory and practice inthe ﬁeld of deep active learning.• New perspective on nonparametric learning.
Revised: On the other hand, our guarantees are minimax optimal, up to disagreement coefﬁcient and other logarithmic factors, which bridge the gap between theory and practice in deep active learning. • New perspective on nonparametric learning.

Original: Nonparametric learning of smooth functions havebeen mainly approached by partitioning-based methods (Tsybakov, 2004; Audibert and Tsybakov,2007; Castro and Nowak, 2008; Minsker, 2012; Locatelli et al., 2017, 2018; Kpotufe et al., 2021) :
Revised: Nonparametric learning of smooth functions have been mainly approached by partitioning-based methods (Tsybakov, 2004; Audibert and Tsybakov, 2007; Castro and Nowak, 2008; Minsker, 2012; Locatelli et al., 2017, 2018; Kpotufe et al., 2021) :

Original: Partition the unit cube [0 , 1] d into exponentially (in dimension) many sub-cubes and conduct localmean estimation within each sub-cube (which additionally requires a strictly stronger membershipquerying oracle).
Revised: Partition the unit cube [0 , 1] d into exponentially (in dimension) many sub-cubes and conduct local mean estimation within each sub-cube (which additionally requires a strictly stronger membership querying oracle).

Original: Our results show that, in both passive and active settings, one can in fact learnwith a set of neural network to achieve near minimax optimal label complexities.
Revised: Our results show that, in both passive and active settings, one can learn globally with a set of neural networks and achieve near minimax optimal label complexities.

Original:  Deep active learning with abstention: Exponential speedups
Revised:  Deep active learning with abstention: Exponential speedups

Original: We analyze the label complexity  of  deep active learning under Chow’s excess error (Chow, 1970).
Revised: While the theoretical guarantees provided in Section 2 are near minimax optimal, the label complexity scales as poly( 1 ε ) , which doesn’t match the great empirical performance observed in deep active learning. In this section, we ﬁll in this gap by leveraging the idea of abstention and provide a deep active learning algorithm that achieves exponential label savings.

Original: Weﬁrst introduce the concepts of abstention and Chow’s excess error in Section 3.1, and then provideour label complexity guarantees in Section 3.2.
Revised: We introduce the concepts of abstention and Chow’s excess error in Section 3.1, and provide our label complexity guarantees in Section 3.2.

Original: 3.1 Active learning without low noise conditions
Revised: 3.1 Active learning without low noise conditions

Original: The previous section analyze active learning under Tsybakov noise, which has been extensivelystudied in the literature since Castro and Nowak (2008).
Revised: The previous section analyzes active learning under Tsybakov noise, which has been extensively studied in the literature since Castro and Nowak (2008).

Original: More recently, promising results are observedin active learning under Chow’s excess error, but otherwise without any low noise assumption (Puchkinand Zhivotovskiy, 2021; Zhu and Nowak, 2022).
Revised: More recently, promising results are observed in active learning under Chow’s excess error, but otherwise without any low noise assumption (Puchkin and Zhivotovskiy, 2021; Zhu and Nowak, 2022).

Original: We introduce such setting as follows.
Revised: We introduce this setting in the following.

Original: Abstention and Chow’s error (Chow, 1970).
Revised: Abstention and Chow’s error (Chow, 1970).

Original: We consider classiﬁer of the form b h : X → Y∪{⊥}291 where ⊥ denotes the action of abstention.
Revised: We consider classiﬁer of the form b h : X → Y∪{⊥} where ⊥ denotes the action of abstention.

Original: For any ﬁxed 0 < γ < 12 , the Chow’s error is deﬁned as
Revised: For any ﬁxed 0 < γ < 12 , the Chow’s error is deﬁned as

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: The parameter γ can be chosen as a small constant, e.g., γ = 0 .
Revised: The parameter γ can be chosen as a small constant, e.g., γ = 0 .

Original: 01 , to avoid excessive abstention: Theprice of abstention is only marginally smaller than random guess (which incurs cost 0 . 5 ).
Revised: 01 , to avoid excessive abstention: The price of abstention is only marginally smaller than random guess (which incurs cost 0 . 5 ).

Original: The Chow’sexcess error is then deﬁned as excess γ ( b h ) :=err γ (
Revised: The Chow’s excess error is then deﬁned as excess γ ( b h ) :=

Original: [Equation]

Original:  h ) − err( h [Equation]
Revised: err γ ( b h ) − err( h ? )

Original: ) (Puchkin and Zhivotovskiy, 2021).
Revised: (Puchkin and Zhivotovskiy, 2021).

Original: At a high level, analyzing with Chow’s excess error allows slackness in predications of hard examples(e.g. , data points whose η ( x ) is close to 12 ) by leveraging the power of abstention.
Revised: At a high level, analyzing with Chow’s excess error allows slackness in predications of hard examples (e.g., data points whose η ( x ) is close to 12 ) by leveraging the power of abstention.

Original: Puchkin and
Revised: Puchkin and

Original: Zhivotovskiy (2021); Zhu and Nowak (2022) show that polylog( 1 ε ) is always achievable in theparametric settings.
Revised: Zhivotovskiy (2021); Zhu and Nowak (2022) show that polylog( 1 ε ) is always achievable in the parametric settings.

Original: We generalize their results to the nonparametric setting and analyze activelearning with a set of neural networks.
Revised: We generalize their results to the nonparametric setting and analyze active learning with a set of neural networks.

Original: 3.2 Exponential speedups with abstention
Revised: 3.2 Exponential speedups with abstention

Original: In this section, we work with a set of neural network regression functions F dnn : X → [0 , 1] (thatapproximates η ) and then construct classiﬁers h : X → Y ∪ {⊥} with an additional abstention303 action . We construct
Revised: In this section, we work with a set of neural network regression functions F dnn : X → [0 , 1] (that approximates η ) and then construct classiﬁers h : X → Y ∪ {⊥} with an additional abstention action . To work with a set of regression functions

Original: F dnnbased on approximation results developed in Theorem 3, with additionalpreprocessing steps that: (1) clipping f dnn : X → R into f 0 dnn : X → [0 ,1] (since we obviously haveη ( x ) ∈ [0 , 1] ); and (2) ﬁltering out f dnn ∈ F dnn that are clearly not a good approximation of [-η . 
-]
Revised: F dnn , we analyze its “complexity” from the lenses of pseudo dimension Pdim( F dnn ) (Pollard, 1984; Haussler, 1989, 1995) and value function disagreement coefﬁcient θ val F dnn ( ι ) (for some ι > 0 ) (Foster et al., 2020). We defer detailed deﬁnitions of [-η . 
-]these complexity measures to Appendix D.1.

Original: Algorithm
Revised: Algorithm

Original: Input: Accuracy level ε ∈ ( , abstention parameter γ ∈ ( 1 / 2) , conﬁdence level δ
Revised: Input: Accuracy level ε ∈ (0 , 1) , conﬁdence level δ ∈ (0 , 1) , abstention parameter γ ∈ (0 , 1 / 2) .

Original: (0 , 1) , and an appropriately chosen set of neural network regression functions F
Revised: 1: Let F dnn be a set of neural network regression functions obtained by (i) applying Theoremwith an appropriate approximation level κ (which satisﬁes 1 κ = poly( 1 γ ) polylog( 1 εγ ) ), and (ii) applying a preprocessing step on the set of neural networks obtained from step (i). See Appendix E for details.

Original:  Deﬁne 
Revised: 2: Deﬁne T :

Original: − convention that{+= θ val F dnn ( γ/ 4) · Pdim( F dnn ) εγ , M := d log 2 T e , and C δ := O (Pdim( F dnn ) · log( T/δ )) . 3: Deﬁne τ m := 2 m for m ≥ 1 , τ 0 := 0 , and β m := 
+}
Revised: {+= θ val F dnn ( γ/ 4) · Pdim( F dnn ) εγ , M := d log 2 T e , and C δ := O (Pdim( F dnn ) · log( T/δ )) . 3: Deﬁne τ m := 2 m for m ≥ 1 , τ 0 := 0 , and β m := 
+}

Original:  P 
Revised: ( M − m + 1) C δ . 4: Deﬁne b R m ( f ) := P τ m − 1 t =1 Q t ( b f ( x t ) − y t ) 2 with the convention that

Original: 4: for epoch do
Revised: P 0 t =1 . . . = 0 . 5: for epoch m = 1 , 2 , . .

Original:  Get b f m = arg min f ∈F 
Revised: , M do 6: Get b f m := arg min f ∈F dnn

Original: P τ m − 1 t =1 Q t ( f ( x t ) − y t ) 2 .
Revised: P τ m − 1 t =1 Q t ( f ( x t ) − y t ) 2 .

Original: 6: (Implicitely) Construct active set F = n f ∈ F  b
Revised: 7: (Implicitely) Construct active set F m := n f ∈ F dnn : b

Original: R m ( f ) ≤ b
Revised: R m ( f ) ≤ b

Original: R m ( b f m ) + β m o .
Revised: R m ( b f m ) + β m o .

Original: Construct classiﬁer b h m : X → { +1 , − 1 , ⊥} as b h m ( x ) :
Revised: Construct classiﬁer b h m : X → { +1 , − 1 , ⊥} as b h m ( x ) :

Original: = ( ⊥ , if [ lcb ( x ; F m ) [Equation] ( x [Equation]
Revised: = ( ⊥ , if [ lcb ( x ; F m ) − γ 4 , ucb ( x ; F m )

Original: F

Original: [Equation]

Original: [Equation]

Original: (cid:2) [Equation] γ (cid:3) [Equation]
Revised: + γ 4 ] ⊆

Original:  sign( b f m ( x ) − 1) , o.w. and query function g m ( x ) :=(cid:0) 12 ∈ [Equation] x [Equation]
Revised: (cid:2) 12 − γ, 12 + γ (cid:3) ; sign( b f m ( x ) − 1) , o.w. and query function g m ( x ) :=(cid:0) 12 ∈ (cid:0) lcb ( x ; F m ) − γ 4 , ucb ( x ; F m ) + γ(cid:1)(cid:1) · 1 ( b h m ( x ) 6 = ⊥ ) .

Original: F

Original: [Equation]

Original: (cid:1)(cid:1)

Original: [Equation]

Original: x

Original: [Equation] ⊥ [Equation] 8: if epoch m =
Revised: 9: if epoch m =

Original: M then 9:
Revised: M then 10:

Original: Return classiﬁer b h M .
Revised: Return classiﬁer b h M .

Original: 10: for time t = τ m − 1 + 1 , . . .
Revised: 11: for time t = τ m − 1 + 1 , . . .

Original: , τ m do 11:
Revised: , τ m do 12:

Original: Observe x t ∼ D X .
Revised: Observe x t ∼ D X .

Original: Set Q t =
Revised: Set Q t :=

Original: g m ( x t ) .
Revised: g m ( x t ) .

Original: 12: if Q t = 1 then 13:
Revised: 13: if Q t = 1 then 14:

Original: Query the label y t of x t .
Revised: Query the label y t of x t .

Original: Our algorithm is presented in Algorithm 2, which is adapted from the algorithm developed in Zhu and
Revised: We now present NeuralCAL ++ (Algorithm 2), a deep active learning algorithm that leverages the power of abstention.

Original: Nowak (2022).
Revised: NeuralCAL ++ ﬁrst initialize a set of set of neural network regression functions F dnn by applying a preprocessing step on top of the set of regression functions obtained from Theorem 3 with a carefully chosen approximation level κ . The preprocessing step mainly contains two actions: (1) clipping f dnn : X → R into ˇ f dnn : X → [0 , 1] (since we obviously have η ( x ) ∈ [0 , 1] ); and (2) ﬁltering out f dnn ∈ F dnn that are clearly not a good approximation of η .

Original: Algorithm 2 runs in epochs of geometrically increasing lengths.
Revised: After initialization, NeuralCAL ++ runs in epochs of geometrically increasing lengths.

Original: At the beginning ofepoch m ∈ [ M ] , Algorithm 2 (implicitly) constructs an active set of regression functions F m that are“close” to the true conditional probability η .
Revised: At the beginning of epoch m ∈ [ M ] , NeuralCAL ++ (implicitly) constructs an active set of regression functions F m that are “close” to the true conditional probability η .

Original: For any x ∼ D X , Algorithm 2 constructs a lower bound310 lcb ( x ; F m ) 
Revised: For any x ∼ D X , NeuralCAL ++ constructs a lower bound lcb ( x ; F m ) :

Original: :=
Revised: =

Original: inf f ∈F m f ( x ) and an upper bound ucb ( x ; F m ) :
Revised: inf f ∈F m f ( x ) and an upper bound ucb ( x ; F m ) :

Original: = sup f ∈F m f ( x ) as a conﬁdencerange of η ( x ) (based on F m ).
Revised: = sup f ∈F m f ( x ) as a conﬁdence range of η ( x ) (based on F m ).

Original: An empirical classiﬁer  b h m : X → { +1 , − 1 , ⊥} and a query function312 g m : X → { 0 , 1 } are then constructed based on the conﬁdence range (and the abstention parameter313 γ ).
Revised: An empirical classiﬁer with an abstention option b h m : X → { +1 , − 1 , ⊥} and a query function g m : X → { 0 , 1 } are then constructed based on the conﬁdence range (and the abstention parameter γ ).

Original: For any time step t within epoch m , Algorithm 2 queries the label of the observed data point x t ifand only if g m ( x t )
Revised: For any time step t within epoch m , NeuralCAL ++ queries the label of the observed data point x t if and only if Q t :=

Original:  = 1 .
Revised: g m ( x t ) = 1 .

Original: Algorithm 2 returns b h M as the learned classiﬁer.
Revised: NeuralCAL ++ returns b h M as the learned classiﬁer.

Original: The algorithm provided in Zhu and Nowak (2022) 
Revised: NeuralCAL ++ is adapted from the algorithm developed in Zhu and Nowak (2022), but with novel extensions. In particular, the algorithm presented in Zhu and Nowak (2022) requires the existence of a f ∈ F

Original: requires the existence of a f ∈ F such thatk f − η k ∞ ≤ ε (to achieve ε Chow’s excess error). Such  approximation requirement directly leads to317 poly(ε ) label complexity in the nonparametric setting , which is unacceptable.
Revised: such that k f − η k ∞ ≤ ε (to achieve ε Chow’s excess error), Such an approximation requirement directly leads to poly(ε ) label complexity in the nonparametric setting , which is unacceptable.

Original: We identify conditionsto bypass such requirement and provide sharper analysis of concentration results (see Appendix D).
Revised: The initialization step of NeuralCAL ++ (line 1) is carefully chosen to ensure that Pdim( F dnn ) , θ val F dnn ( γ/ 4) =

Original: These novel extensions are important in the setting of active learning with neural networks, whichhelp us derive the following deep active learning guarantees.
Revised: e O (poly(1 /γ )) ; together with a sharper analysis of concentration results, these conditions help us derive the following deep active learning guarantees (also see Appendix D for a more general guarantee).

Original: Theorem 7.
Revised: Theorem 7.

Original: Fix any ε, δ, γ > 0 .
Revised: Fix any ε, δ, γ > 0 .

Original: Let F dnn be an appropriately chosen set of neural network regressionfunction.

Original: With probability at least 1 − δ , Algorithm 2  returns a classiﬁer b h with Chow’s excess errore
Revised: With probability at least 1 − δ , Algorithm 2 (with an appropriate initialization at line 1) returns a classiﬁer b h with Chow’s excess error e

Original: O ( ε ) after querying poly( 1 γ ) · polylog( 1 εδ ) labels.
Revised: O ( ε ) after querying poly( 1 γ ) · polylog( 1 εδ ) labels.

Original: We discuss two important aspects of Algorithm 2/Theorem 7, i.e., exponential savings and computational efﬁciency, as follows (more detailed discussionsdeferred to Appendix E.2, due to lack ofspace).
Revised: We discuss two important aspects of Algorithm 2/Theorem 7 in the following, i.e., exponential savings and computational efﬁciency. We defer more detailed discussions to Appendix F.1.

Original: • Exponential speedups.
Revised: • Exponential speedups.

Original: Theorem 7 shows that, equipped with an abstention option, deep activelearning enjoys polylog( 1 ε ) label complexity.
Revised: Theorem 7 shows that, equipped with an abstention option, deep active learning enjoys polylog( 1 ε ) label complexity.

Original: This provides theoretical justiﬁcations for greatempirical results of deep active learning observed in practice.
Revised: This provides theoretical justiﬁcations for great empirical results of deep active learning observed in practice.

Original: Moreover, Algorithm 2 outputsa classiﬁer that abstains properly , i e., it abstains only if abstention is the optimal choice; suchproperty further implies polylog( 1 ε ) label complexity under standard excess error and Massartnoise (Massart and Nédélec, 2006).• Computational efﬁciency.
Revised: Moreover, Algorithm 2 outputs a classiﬁer that abstains properly , i.e., it abstains only if abstention is the optimal choice; such a property further implies polylog( 1 ε ) label complexity under standard excess error and Massart noise (Massart and Nédélec, 2006). • Computational efﬁciency.

Original: Suppose one can efﬁciently implement a (weighted) square lossregression oracle over a set of neural networks, e
Revised: Suppose one can efﬁciently implement a (weighted) square loss regression oracle over the initialized set of neural networks F dnn : Given any set S of weighted examples ( w, x

Original: g.,
Revised: , y ) ∈ R + × X × Y as input, the regression oracle outputs

Original: [Equation]
Revised: [Equation]

Original: Algorithm 2 canbe then  efﬁciently implemented with poly( 1 γ ) · 1 ε oracle calls. 
Revised: Algorithm 2 can then be efﬁciently implemented with poly( 1 γ ) · 1 ε oracle calls. While the label complexity

Revised: obtained

Revised: in

Original: While the label complexity obtained in Theorem7 has desired dependence on polylog( 1 ε ) , itsdependence on γcan be of order
Revised: Theorem has desired dependence on polylog 1 ε ) , its dependence on γ

Revised: can

Original:  γ − poly( d ) .
Revised: be of order γ − poly( d ) .

Revised: Our next result shows

Revised: that, however,

Original: Our next result shows that, however, such dependence isunavoidable even in the case of learning a single ReLU function.
Revised: such dependence is unavoidable even in the case of learning a single ReLU function

Original: Theorem 8.
Revised: Theorem 8.

Original: Fix any γ ∈ (0 , 1 / 8) .
Revised: Fix any γ ∈ (0 , 1 / 8) .

Original: For any accuracy level ε sufﬁciently small, there exists a probleminstance such that (1) η ∈ W 1 , ∞( X ) and is of the form η ( x ) :
Revised: For any accuracy level ε sufﬁciently small, there exists a problem instance such that (1) η ∈ W 1 , ∞( X ) and is of the form η ( x ) :

Original: = ReLU ( h w, x i + a ) + b ; and (2) forany active learning algorithm, it takes at least γ − Ω( d ) labels to identify an ε -optimal classiﬁer, foreither standard excess error or Chow’s excess error (with parameter γ ).
Revised: = ReLU ( h w, x i + a ) + b ; and (2) for any active learning algorithm, it takes at least γ − Ω( d ) labels to identify an ε -optimal classiﬁer, for either standard excess error or Chow’s excess error (with parameter γ ).

Original:  Extensions
Revised:  Extensions

Original: Previous results are developed in the commonly studied Sobolev/Hölder spaces.
Revised: Previous results are developed in the commonly studied Sobolev/Hölder spaces.

Original: Our techniques,however, are generic and can be adapted to other function spaces, given neural network approximationresults.
Revised: Our techniques, however, are generic and can be adapted to other function spaces, given neural network approximation results.

Original: In this section, we provide extensions of our results to the Radon BV 2 space, which wasrecently proposed as the natural function space associated with ReLU neural networks (Ongie et al. ,2020; Parhi and Nowak, 2021a,b; Unser, 2022).
Revised: In this section, we provide extensions of our results to the Radon BV 2 space, which was recently proposed as the natural function space associated with ReLU neural networks (Ongie et al., 2020; Parhi and Nowak, 2021, 2022a,b; Unser, 2022).

Original: The Radon BV 2 space.
Revised: The Radon BV 2 space.

Original: The Radon BV 2 unit ball over domain X is deﬁned as R BV 21 ( X ) :
Revised: The Radon BV 2 unit ball over domain X is deﬁned as R BV 21 ( X ) :

Original: ={ f : k f k R BV 2 ( X ) ≤ 1 } , where k f k R BV 2 ( X ) denotes the Radon BV 2 norm of f over domain X .[-
-]
Revised: = { f : k f k R BV 2 ( X ) ≤ 1 } , where k f k R BV 2 ( X ) denotes the Radon BV 2 norm of f over domain X .[-
-]

Original: Following Parhi and Nowak (2021b), we assume X = { x ∈ R d : k x k 2 ≤ 1 } and η ∈ R BV 21 ( X ) .
Revised: Following Parhi and Nowak (2022b), we assume X = { x ∈ R d : k x k 2 ≤ 1 } and η ∈ R BV 2 1 ( X ) .

Original: The Radon BV 2 space naturally contains neural networks of the form f dnn ( x ) =
Revised: The Radon BV 2 space naturally contains neural networks of the form f dnn ( x ) =

Original: P
Revised: P

Original: Kk =1 v i ·353 ReLU ( w > i x + b i ) .
Revised: Kk =1 v i · ReLU ( w > i x + b i ) .

Original: On the contrary, such f dnn doesn’t lie in any Sobolev space of order α ≥ 2(since f dnn doesn’t have second order weak derivative).
Revised: On the contrary, such f dnn doesn’t lie in any Sobolev space of order α ≥(since f dnn doesn’t have second order weak derivative).

Original: Thus, if η takes the form of the aforementioned neural network (e.g., η ≡ f dnn ), approximating η up to κ from a Sobolev perspectiverequires e
Revised: Thus, if η takes the form of the aforementioned neural network (e.g., η = f dnn ), approximating η up to κ from a Sobolev perspective requires e

Original: O ( κ − d ) total parameters, which suffers from the curse of dimensionality.
Revised: O ( κ − d ) total parameters, which suffers from the curse of dimensionality.

Original: On the other side,however, such bad dependence on dimensionality goes away when approximating from a Radon BV 2perspective, as shown in the following theorem.
Revised: On the other side, however, such bad dependence on dimensionality goes away when approximating from a Radon BVperspective, as shown in the following theorem.

Original: Theorem 9 (Parhi and Nowak (2021b)) .
Revised: Theorem 9 (Parhi and Nowak (2022b)) .

Original: Fix any κ > 0 .
Revised: Fix any κ > 0 .

Original: For any f ? ∈ R BV 21 ( X ) , there exists aone-hidden layer neural network f dnn of width K
Revised: For any f ? ∈ R BV 21 ( X ) , there exists a one-hidden layer neural network f dnn of width K

Original: = O ( κ − 2 d d +3 ) such that k f ?
Revised: = O ( κ − 2 d d +3 ) such that k f ?

Original: − f dnn k ∞ ≤ κ .
Revised: − f dnn k ∞ ≤ κ .

Original: Equipped with this approximation result, we provide the active learning guarantees for learning asmooth function within the Radon BV 2 unit ball as follows.
Revised: Equipped with this approximation result, we provide the active learning guarantees for learning a smooth function within the Radon BV 2 unit ball as follows.

Original: Theorem 10.
Revised: Theorem 10.

Original: Suppose η ∈ R BV 21 ( X ) and the Tsybakov noise condition is satisﬁed with parameter364 β ≥ 0 .
Revised: Suppose η ∈ R BV 21 ( X ) and the Tsybakov noise condition is satisﬁed with parameter β ≥ 0 .

Original: Fix any ε, δ > 0 .
Revised: Fix any ε, δ > 0 .

Original: Let H dnn be an appropriately chosen set of neural networks.

Original: Withprobability at least 1 − δ , Algorithm 1 returns a classiﬁer b h ∈H dnn with excess error e
Revised: There exists an algorithm such that, with probability at least 1 − δ , it learns a classiﬁer b [Equation] ∈

Revised: [Equation]

Revised: dnn with excess error e

Original: O ( ε ) afterquerying
Revised: O ( ε ) after querying e

Original: [Equation]
Revised: [Equation]

Original:  Compared to the label complexity obtained in Theorem 6, the label complexity obtained in theabove theorem doesn’t suffer from the curse of dimensionality: For d large enough, the above labelcomplexity scales as ε − O (1) yet label complexity in Theorem 6 scales as ε − O
Revised: labels. Compared to the label complexity obtained in Theorem 6, the label complexity obtained in the above theorem doesn’t suffer from the curse of dimensionality: For d large enough, the above label complexity scales as ε − O (1) yet label complexity in Theorem 6 scales as ε − O

Original: ( d ) .
Revised: ( d ) .

Original: Active learningguarantees under Chow’s excess error in the Radon BV 2 space is similar to results presented in
Revised: Active learning guarantees under Chow’s excess error in the Radon BV 2 space are similar to results presented in

Original: Theorem 7, and is thus deferred to Appendix F.
Revised: Theorem 7, and are thus deferred to Appendix G.  Discussion

Revised: We provide the ﬁrst near-optimal deep active learning guarantees, under both standard excess error and Chow’s excess error.

Revised: Our results are powered by generic algorithms and analyses developed for active learning that bridge approximation guarantees into label complexity guarantees.

Revised: We outline some natural directions for future research below.

Original: Checklist
Revised: • Disagreement coefﬁcients for neural networks. While we have provided some results regarding the disagreement coefﬁcients for neural networks, we believe a comprehensive investigation on this topic is needed. For instance, can we discover more general settings where the classiﬁer-based disagreement coefﬁcient can be upper bounded by O (1) ? It is also interesting to explore sharper analyses on the value function disagreement coefﬁcient.

Revised: • Adaptivity in deep active learning.

Revised: Our current results are established with the knowledge of some problem-dependent parameters, e.g., the smoothness parameters regarding the function spaces and the noise levels.

Revised: It will be interesting to see if one can develop algorithms that can automatically adapt to unknown parameters, e.g., by leveraging techniques developed in Locatelli et al.

Original: For all authors...
Revised: The authors would like to thank Rahul Parhi for many helpful discussions regarding his papers. We also would like to thank anonymous NeurIPS 2022 reviewers for their constructive comments. This work is partially supported by NSF grant 1934612 and AFOSR grant FA9550-18-1-

Original:  (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes](b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] As discussed in Section 2.3, our activelearning label complexity only matches the lower bound up to the disagreement coefﬁcient. While we have provided examples (around Theorem 6 and in Appendix C.2) where thedisagreement coefﬁcient is well-bounded, we leave a more comprehensive analyses as futurework.(c)
Revised: [Yes] See Section 5 for discussions on limitations and directions for future work.

Original:  Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: Our paper istheoretical in nature, and there is no negative societal impact of our work in the foreseeablefuture.(d)
Revised: Our paper is theoretical in nature, and there is no negative societal impact of our work in the foreseeable future.

Original:  Have you read the ethics review guidelines and ensured that your paper conforms to them?[Yes]2.
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] Assumptions areclearly stated in the statement of each theorem.(b)
Revised: [Yes] Assumptions are clearly stated in the statement of each theorem.

Original:  Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] Complete proofs are providedin the Appendix.3.
Revised: [Yes] Complete proofs are provided in the Appendix.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimentalresults (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [N/A](b) Did you specify all the training details (e.g., data splits, hyperparameters, how they werechosen)?
Revised: [N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [N/A](c) Did you report error bars (e.g., with respect to the random seed after running experimentsmultiple times)?
Revised: [N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [N/A](d) Did you include the total amount of compute and the type of resources used (e.g., type ofGPUs, internal cluster, or cloud provider)?
Revised: [N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [N/A]4.
Revised: [N/A] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...[Table] (a) If your work uses existing assets, did you cite the creators?[N/A]
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Table]

Original:  (b) Did you mention the license of the assets?[N/A]
Revised: [N/A] (b) Did you mention the license of the assets?

Original: [Table] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A](d) Did you discuss whether
Revised: [N/A] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re[Table] using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A](e) Did you discuss whether the data you are using/curating contains personally identiﬁableinformation or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [N/A] [Table]
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects...[Table] (a) Did you include the full text of instructions given to participants and screenshots, if applicable?[Table]
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A]

Original: [Table] (b) Did you describe any potential participant risks, with links to Institutional Review Board[Table] (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A][Table] (c) Did you include the estimated hourly wage paid to participants and the total amount spent onparticipant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: [Table]