Original: Active Learning with Neural Networks: Insights from Nonparametric Statistics
Revised: Active Learning with Neural Networks: Insights from Nonparametric Statistics

Original: Abstract
Revised: Abstract

Original: Deep neural networks have great representation power, but typically require large numbers of training examples.
Revised: Deep neural networks have great representation power, but typically require large numbers of training examples.

Original: This motivates deep active learning methods that can signiﬁcantly reduce the amount of labeled training data.
Revised: This motivates deep active learning methods that can signiﬁcantly reduce the amount of labeled training data.

Original: Empirical successes of deep active learning have been recently reported in the literature, however, rigorous label complexity guarantees of deep active learning have remained elusive.
Revised: Empirical successes of deep active learning have been recently reported in the literature, however, rigorous label complexity guarantees of deep active learning have remained elusive.

Original: This constitutes a signiﬁcant gap between theory and practice.
Revised: This constitutes a signiﬁcant gap between theory and practice.

Original: This paper tackles this gap by providing the ﬁrst near-optimal label complexity guarantees for deep active learning.
Revised: This paper tackles this gap by providing the ﬁrst near-optimal label complexity guarantees for deep active learning.

Original: The key insight is to study deep active learning from the nonparametric classiﬁcation perspective.
Revised: The key insight is to study deep active learning from the nonparametric classiﬁcation perspective.

Original: Under standard low noise conditions, we show that active learning with neural networks can provably achieve the minimax label complexity, up to disagreement coefﬁcient and other logarithmic terms.
Revised: Under standard low noise conditions, we show that active learning with neural networks can provably achieve the minimax label complexity, up to disagreement coefﬁcient and other logarithmic terms.

Original: When equipped with an abstention option, we further develop an efﬁcient deep active learning algorithm that achieves polylog ( 1 ε ) label complexity, without any low noise assumptions.
Revised: When equipped with an abstention option, we further develop an efﬁcient deep active learning algorithm that achieves polylog ( 1 ε ) label complexity, without any low noise assumptions.

Original: We also provide extensions of our results beyond the commonly studied Sobolev/Hölder spaces and develop label complexity guarantees for learning in Radon BV 2 spaces, which have recently been proposed as natural function spaces associated with neural networks.
Revised: We also provide extensions of our results beyond the commonly studied Sobolev/Hölder spaces and develop label complexity guarantees for learning in Radon BV 2 spaces, which have recently been proposed as natural function spaces associated with neural networks.

Original:  Introduction
Revised:  Introduction

Original: We study active learning with neural network hypothesis classes, sometimes known as deep active learning .
Revised: We study active learning with neural network hypothesis classes, sometimes known as deep active learning .

Original: Active learning agent proceeds by selecting the most informative data points to label:
Revised: Active learning agent proceeds by selecting the most informative data points to label:

Original: The goal of active learning is to achieve the same accuracy achievable by passive learning, but with much fewer label queries (Settles, 2009; Hanneke, 2014).
Revised: The goal of active learning is to achieve the same accuracy achievable by passive learning, but with much fewer label queries (Settles, 2009; Hanneke, 2014).

Original: When the hypothesis class is a set of neural networks, the learner further beneﬁts from the representation power of deep neural networks, which has driven the successes of passive learning in the past decade (Krizhevsky et al., 2012; LeCun et al., 2015).
Revised: When the hypothesis class is a set of neural networks, the learner further beneﬁts from the representation power of deep neural networks, which has driven the successes of passive learning in the past decade (Krizhevsky et al., 2012; LeCun et al., 2015).

Original: With these added beneﬁts, deep active learning has become a popular research area, with empirical successes observed in many recent papers (Sener and Savarese, 2018; Ash et al., 2019; Citovsky et al., 2021; Ash et al., 2021; Kothawade et al., 2021;
Revised: With these added beneﬁts, deep active learning has become a popular research area, with empirical successes observed in many recent papers (Sener and Savarese, 2018; Ash et al., 2019; Citovsky et al., 2021; Ash et al., 2021; Kothawade et al., 2021;

Original: Emam et al., 2021;
Revised: Emam et al., 2021;

Original: Ren et al., 2021).
Revised: Ren et al., 2021).

Original: However, due to the difﬁculty of analyzing a set of neural networks, rigorous label complexity guarantees for deep active learning have remained largely elusive.
Revised: However, due to the difﬁculty of analyzing a set of neural networks, rigorous label complexity guarantees for deep active learning have remained largely elusive.

Original: To the best of our knowledge, there are only two papers (Karzand and Nowak, 2020; Wang et al., 2021) that have made the attempts at theoretically quantifying active learning gains with neural networks.
Revised: To the best of our knowledge, there are only two papers (Karzand and Nowak, 2020; Wang et al., 2021) that have made the attempts at theoretically quantifying active learning gains with neural networks.

Original: While insightful views are provided, these two works have their own limitations.
Revised: While insightful views are provided, these two works have their own limitations.

Original: The guarantees provided in Karzand and Nowak (2020) only work in the 1 d case where data points are uniformly sampled from [0 , 1] and labeled by a well-seperated piece-wise constant function in a noise-free way (i.e., without any labeling noise).
Revised: The guarantees provided in Karzand and Nowak (2020) only work in the 1 d case where data points are uniformly sampled from [0 , 1] and labeled by a well-seperated piece-wise constant function in a noise-free way (i.e., without any labeling noise).

Original: Wang et al.
Revised: Wang et al.

Original: (2021) study deep active learning by 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
Revised: (2021) study deep active learning by 36th Conference on Neural Information Processing Systems (NeurIPS 2022).

Original: linearizing the neural network at its random initialization and then analyzing it as a linear function; moreover, as the authors agree, their error bounds and label complexity guarantees can in fact be vacuous in certain cases.
Revised: linearizing the neural network at its random initialization and then analyzing it as a linear function; moreover, as the authors agree, their error bounds and label complexity guarantees can in fact be vacuous in certain cases.

Original: Thus, it’s fair to say that up to now researchers have not identiﬁed cases where deep active learning are provably near minimax optimal (or even with provably non-vacuous guarantees), which constitutes a signiﬁcant gap between theory and practice.
Revised: Thus, it’s fair to say that up to now researchers have not identiﬁed cases where deep active learning are provably near minimax optimal (or even with provably non-vacuous guarantees), which constitutes a signiﬁcant gap between theory and practice.

Original: In this paper, we bridge this gap by providing the ﬁrst near-optimal label complexity guarantees for deep active learning.
Revised: In this paper, we bridge this gap by providing the ﬁrst near-optimal label complexity guarantees for deep active learning.

Original: We obtain insights from the nonparametric setting where the conditional probability (of taking a positive label) is assumed to be a smooth function (Tsybakov, 2004; Audibert and Tsybakov, 2007).
Revised: We obtain insights from the nonparametric setting where the conditional probability (of taking a positive label) is assumed to be a smooth function (Tsybakov, 2004; Audibert and Tsybakov, 2007).

Original: Previous nonparametric active learning algorithms proceed by partitioning the action space into exponentially many sub-regions (e.g., partitioning the unit cube [0 , 1] d into ε − d sub-cubes each with volume ε d ), and then conducting local mean (or some higher-order statistics) estimation within each sub-region (Castro and Nowak, 2008; Minsker, 2012; Locatelli et al., 2017, 2018; Shekhar et al., 2021;
Revised: Previous nonparametric active learning algorithms proceed by partitioning the action space into exponentially many sub-regions (e.g., partitioning the unit cube [0 , 1] d into ε − d sub-cubes each with volume ε d ), and then conducting local mean (or some higher-order statistics) estimation within each sub-region (Castro and Nowak, 2008; Minsker, 2012; Locatelli et al., 2017, 2018; Shekhar et al., 2021;

Original: Kpotufe et al., 2021).
Revised: Kpotufe et al., 2021).

Original: We show that, with an appropriately chosen set of neural networks that globally approximates the smooth regression function, one can in fact recover the minimax label complexity for active learning, up to disagreement coefﬁcient (Hanneke, 2007, 2014) and other logarithmic factors.
Revised: We show that, with an appropriately chosen set of neural networks that globally approximates the smooth regression function, one can in fact recover the minimax label complexity for active learning, up to disagreement coefﬁcient (Hanneke, 2007, 2014) and other logarithmic factors.

Original: Our results are established by (i) identifying the “right tools” to study neural networks (ranging from approximation results (Yarotsky, 2017, 2018) to complexity measure of neural networks (Bartlett et al., 2019)), and (ii) developing novel extensions of agnostic active learning algorithms (Balcan et al., 2006; Hanneke, 2007, 2014) to work with a set of neural networks.
Revised: Our results are established by (i) identifying the “right tools” to study neural networks (ranging from approximation results (Yarotsky, 2017, 2018) to complexity measure of neural networks (Bartlett et al., 2019)), and (ii) developing novel extensions of agnostic active learning algorithms (Balcan et al., 2006; Hanneke, 2007, 2014) to work with a set of neural networks.

Original: While matching the minimax label complexity in nonparametric active learning is existing, such minimax results scale as Θ(poly( 1 ε ))
Revised: While matching the minimax label complexity in nonparametric active learning is existing, such minimax results scale as Θ(poly( 1 ε ))

Original: (Castro and Nowak, 2008; Locatelli et al., 2017) and do not resemble what is practically observed in deep active learning: A fairly accurate neural network classiﬁer can be obtained by training with only a few labeled data points.
Revised: (Castro and Nowak, 2008; Locatelli et al., 2017) and do not resemble what is practically observed in deep active learning: A fairly accurate neural network classiﬁer can be obtained by training with only a few labeled data points.

Original: Inspired by recent results in parametric active learning with abstention (Puchkin and Zhivotovskiy, 2021; Zhu and Nowak, 2022), we develop an oracle-efﬁcient algorithm showing that deep active learning provably achieves polylog( 1 ε ) label complexity when equipped with an abstention option (Chow, 1970).
Revised: Inspired by recent results in parametric active learning with abstention (Puchkin and Zhivotovskiy, 2021; Zhu and Nowak, 2022), we develop an oracle-efﬁcient algorithm showing that deep active learning provably achieves polylog( 1 ε ) label complexity when equipped with an abstention option (Chow, 1970).

Original: Our algorithm not only achieves an exponential saving in label complexity ( without any low noise assumptions ), but is also highly practical: In real-world scenarios such as medical imaging, it makes more sense for the classiﬁer to abstain from making prediction on hard examples (e.g., those that are close to the boundary), and ask medical experts to make the judgments.
Revised: Our algorithm not only achieves an exponential saving in label complexity ( without any low noise assumptions ), but is also highly practical: In real-world scenarios such as medical imaging, it makes more sense for the classiﬁer to abstain from making prediction on hard examples (e.g., those that are close to the boundary), and ask medical experts to make the judgments.

Original: 1.1 Problem setting
Revised: 1.1 Problem setting

Original: Let X denote the instance space and Y denote the label space.
Revised: Let X denote the instance space and Y denote the label space.

Original: We focus on the binary classiﬁcation problem where Y :=
Revised: We focus on the binary classiﬁcation problem where Y :=

Original: { +1 , − 1 } .
Revised: { +1 , − 1 } .

Original: The joint distribution over X × Y is denoted as D XY .
Revised: The joint distribution over X × Y is denoted as D XY .

Original: We use
Revised: We use

Original: D X to denote the marginal distribution over the instance space X , and use D Y| x to denote the conditional distribution of
Revised: D X to denote the marginal distribution over the instance space X , and use D Y| x to denote the conditional distribution of

Original: Y with respect to any x ∈ X .
Revised: Y with respect to any x ∈ X .

Original: We consider the standard active learning setup where x ∼ D X but its label y ∼ D Y| x is only observed after issuing a label query.
Revised: We consider the standard active learning setup where x ∼ D X but its label y ∼ D Y| x is only observed after issuing a label query.

Original: We deﬁne η ( x ) :=
Revised: We deﬁne η ( x ) :=

Original: P y ∼D Y| x
Revised: P y ∼D Y| x

Original: ( y = +1)
Revised: ( y = +1)

Original: as the conditional probability of taking a positive label.
Revised: as the conditional probability of taking a positive label.

Original: The
Revised: The

Original: Bayes optimal classiﬁer h ? can thus be expressed as h ?
Revised: Bayes optimal classiﬁer h ? can thus be expressed as h ?

Original: ( x ) :=
Revised: ( x ) :=

Original: sign(2 η ( x ) − 1) .
Revised: sign(2 η ( x ) − 1) .

Original: For any classiﬁer h : X → Y , its (standard) error is calculated as err( h ) :
Revised: For any classiﬁer h : X → Y , its (standard) error is calculated as err( h ) :

Original: = P ( x,y )
Revised: = P ( x,y )

Original: ∼D XY ( h ( x ) 6 = y ) ; and its (standard) excess error is deﬁned as excess ( h ) :
Revised: ∼D XY ( h ( x ) 6 = y ) ; and its (standard) excess error is deﬁned as excess ( h ) :

Original: = err( h ) − err( h ? ) .
Revised: = err( h ) − err( h ? ) .

Original: Our goal is to learn an accurate classiﬁer with a small number of label querying.
Revised: Our goal is to learn an accurate classiﬁer with a small number of label querying.

Original: The nonparametric setting.
Revised: The nonparametric setting.

Original: We consider the nonparametric setting where the conditional probability η is characterized by a smooth function.
Revised: We consider the nonparametric setting where the conditional probability η is characterized by a smooth function.

Original: Fix any α ∈ N + , the Sobolev norm of a function f : X → R is deﬁned as k f k W α, ∞ := max α, | α |≤ α ess sup x ∈X | D α f ( x ) | , where α = ( α 1 , . .
Revised: Fix any α ∈ N + , the Sobolev norm of a function f : X → R is deﬁned as k f k W α, ∞ := max α, | α |≤ α ess sup x ∈X | D α f ( x ) | , where α = ( α 1 , . .

Original: . , α d ) , | α | =
Revised: . , α d ) , | α | =

Original: P di =

Revised: P di =

Original: α
Revised: α

Original: i and D α f denotes the standard α -th weak derivative of f .
Revised: i and D α f denotes the standard α -th weak derivative of f .

Original: The unit ball in the Sobolev space is deﬁned as W α, ∞( X ) :
Revised: The unit ball in the Sobolev space is deﬁned as W α, ∞( X ) :

Original: = { f : k f k W α, ∞ ≤ 1 } .
Revised: = { f : k f k W α, ∞ ≤ 1 } .

Original: Following the convention of nonparametric active learning (Castro and Nowak, 2008; Minsker, 2012; Locatelli et al., 2017, 2018; Shekhar et al., 2021; Kpotufe et al., 2021), we assume X = [0 , 1] d and η ∈ W α, ∞( X ) (except in Section 4).
Revised: Following the convention of nonparametric active learning (Castro and Nowak, 2008; Minsker, 2012; Locatelli et al., 2017, 2018; Shekhar et al., 2021; Kpotufe et al., 2021), we assume X = [0 , 1] d and η ∈ W α, ∞( X ) (except in Section 4).

Original: Neural Networks.
Revised: Neural Networks.

Original: We consider feedforward neural networks with Rectiﬁed Linear Unit (ReLU) activation function, which is deﬁned as ReLU ( x ) :
Revised: We consider feedforward neural networks with Rectiﬁed Linear Unit (ReLU) activation function, which is deﬁned as ReLU ( x ) :

Original: = max { x, 0 } .
Revised: = max { x, 0 } .

Original: Each neural network f dnn : X → R consists of several input units (which corresponds to the covariates of x ∈ X ), one output unit (which corresponds to the prediction in R ), and multiple hidden computational units.
Revised: Each neural network f dnn : X → R consists of several input units (which corresponds to the covariates of x ∈ X ), one output unit (which corresponds to the prediction in R ), and multiple hidden computational units.

Original: Each hidden computational unit takes inputs { x i } Ni =1 (which are outputs from previous layers) and perform the computation ReLU (
Revised: Each hidden computational unit takes inputs { x i } Ni =1 (which are outputs from previous layers) and perform the computation ReLU (

Original: P
Revised: P

Original: Ni =1 w i x i + b ) with adjustable parameters { w i } Ni =1 and b ; the output unit performs the same operation, but without the ReLU nonlinearity.
Revised: Ni =1 w i x i + b ) with adjustable parameters { w i } Ni =1 and b ; the output unit performs the same operation, but without the ReLU nonlinearity.

Original: We use W to denote the total number of parameters of a neural network, and L to denote the depth of the neural network.
Revised: We use W to denote the total number of parameters of a neural network, and L to denote the depth of the neural network.

Original: 1.2 Contributions and paper organization
Revised: 1.2 Contributions and paper organization

Original: Neural networks are known to be universal approximators (Cybenko, 1989; Hornik, 1991).
Revised: Neural networks are known to be universal approximators (Cybenko, 1989; Hornik, 1991).

Original: In this paper, we argue that, in both passive and active regimes, the universal approximatability makes neural networks “universal classiﬁers” for classiﬁcation problems: With an appropriately chosen set of neural networks, one can recover known minimax rates (up to disagreement coefﬁcients in the active setting) in the rich nonparametric regimes.
Revised: In this paper, we argue that, in both passive and active regimes, the universal approximatability makes neural networks “universal classiﬁers” for classiﬁcation problems: With an appropriately chosen set of neural networks, one can recover known minimax rates (up to disagreement coefﬁcients in the active setting) in the rich nonparametric regimes.

Original: 1 We provide informal statements of our main results in the sequel, with detailed statements and associated deﬁnitions/algorithms deferred to later sections.
Revised: 1 We provide informal statements of our main results in the sequel, with detailed statements and associated deﬁnitions/algorithms deferred to later sections.

Original: In Section 2, we analyze the label complexity of deep active learning under the standard Tsybakov noise condition with smoothness parameter β ≥ 

Revised: In Section 2, we analyze the label complexity of deep active learning under the standard Tsybakov noise condition with smoothness parameter β ≥

Original: (Tsybakov, 2004).
Revised: (Tsybakov, 2004).

Original: Let H dnn be an appropriately chosen set of neural network classiﬁers and denote θ H dnn ( ε ) as the disagreement coefﬁcient (Hanneke, 2007, 2014) at level ε .
Revised: Let H dnn be an appropriately chosen set of neural network classiﬁers and denote θ H dnn ( ε ) as the disagreement coefﬁcient (Hanneke, 2007, 2014) at level ε .

Original: We develop the following label complexity guarantees for deep active learning.
Revised: We develop the following label complexity guarantees for deep active learning.

Original: Theorem 1 (Informal) .
Revised: Theorem 1 (Informal) .

Original: There exists an algorithm that returns a neural network classiﬁer
Revised: There exists an algorithm that returns a neural network classiﬁer

Original: [Equation]
Revised: [Equation]

Original: with excess error e
Revised: with excess error e

Original: O ( ε ) after querying
Revised: O ( ε ) after querying

Original: [Equation]
Revised: [Equation]

Original: The label complexity presented in Theorem 1 matches the active learning lower bound Ω( ε − d +
Revised: The label complexity presented in Theorem 1 matches the active learning lower bound Ω( ε − d +

Original: [Equation]
Revised: [Equation]

Original: α α + αβ ) (Locatelli et al., 2017) up to the dependence on the disagreement coefﬁcient (and other logarithmic factors).
Revised: α α + αβ ) (Locatelli et al., 2017) up to the dependence on the disagreement coefﬁcient (and other logarithmic factors).

Original: Since θ H dnn ( ε ) ≤ ε − 1 by deﬁnition, the label complexity presented in Theorem 1 is never worse than the passive learning rates
Revised: Since θ H dnn ( ε ) ≤ ε − 1 by deﬁnition, the label complexity presented in Theorem 1 is never worse than the passive learning rates

Original: [Equation]
Revised: [Equation]

Original: ( ε − d +2 α
Revised: ( ε − d +2 α

Original: + αβ α + αβ ) (Audibert and Tsybakov, 2007).
Revised: + αβ α + αβ ) (Audibert and Tsybakov, 2007).

Original: We also discover conditions under which the disagreement coefﬁcient with respect to a set of neural network classiﬁers can be properly bounded, i.e., θ H dnn ( ε ) =
Revised: We also discover conditions under which the disagreement coefﬁcient with respect to a set of neural network classiﬁers can be properly bounded, i.e., θ H dnn ( ε ) =

Original: o ( ε − 1 ) (implying strict improvement over passive learning) and θ H dnn ( ε ) =
Revised: o ( ε − 1 ) (implying strict improvement over passive learning) and θ H dnn ( ε ) =

Original: o (1) (implying matching active learning lower bound).
Revised: o (1) (implying matching active learning lower bound).

Original: In Section 3, we develop label complexity guarantees for deep active learning when an additional abstention option is allowed (Chow, 1970; Puchkin and Zhivotovskiy, 2021; Zhu and Nowak, 2022).
Revised: In Section 3, we develop label complexity guarantees for deep active learning when an additional abstention option is allowed (Chow, 1970; Puchkin and Zhivotovskiy, 2021; Zhu and Nowak, 2022).

Original: Suppose a cost (e.g. 0 . 49 ) that is marginally smaller than random guessing (which has expected cost 0 . 5 ) is incurred whenever the classiﬁer abstains from making a predication, we develop the following label complexity guarantees for deep active learning.
Revised: Suppose a cost (e.g. 0 . 49 ) that is marginally smaller than random guessing (which has expected cost 0 . 5 ) is incurred whenever the classiﬁer abstains from making a predication, we develop the following label complexity guarantees for deep active learning.

Original: Theorem 2 (Informal) .
Revised: Theorem 2 (Informal) .

Original: There exists an efﬁcient algorithm that constructs a neural network classiﬁer b h dnn with Chow’s excess error e
Revised: There exists an efﬁcient algorithm that constructs a neural network classiﬁer b h dnn with Chow’s excess error e

Original: O ( ε ) after querying polylog( 1 ε ) labels.
Revised: O ( ε ) after querying polylog( 1 ε ) labels.

Original: The above polylog( 1 ε ) label complexity bound is achieved without any low noise assumptions .
Revised: The above polylog( 1 ε ) label complexity bound is achieved without any low noise assumptions .

Original: Such exponential label savings theoretically justify the great empirical performances of deep active learning observed in practice (e.g., in Sener and Savarese (2018)):
Revised: Such exponential label savings theoretically justify the great empirical performances of deep active learning observed in practice (e.g., in Sener and Savarese (2018)):

Original: It sufﬁces to label a few data points to achieve a high accuracy level.
Revised: It sufﬁces to label a few data points to achieve a high accuracy level.

Original: Moreover, apart from an initialization step, our algorithm (Algorithm 4) developed for Theorem 2 can be efﬁciently implemented in e
Revised: Moreover, apart from an initialization step, our algorithm (Algorithm 4) developed for Theorem 2 can be efﬁciently implemented in e

Original: O ( ε − 1 ) time, given a convex loss regression oracle over an appropriately chosen set of neural networks; in practice, the regression oracle can be approximated by running stochastic gradient descent.
Revised: O ( ε − 1 ) time, given a convex loss regression oracle over an appropriately chosen set of neural networks; in practice, the regression oracle can be approximated by running stochastic gradient descent.

Original: Technical contributions.
Revised: Technical contributions.

Original: Besides identifying the “right tools” (ranging from approximation results (Yarotsky, 2017, 2018) to complexity analyses (Bartlett et al., 2019)) to analyze deep active learning, our theoretical guarantees are empowered by novel extensions of active learning algorithms under neural network approximations .
Revised: Besides identifying the “right tools” (ranging from approximation results (Yarotsky, 2017, 2018) to complexity analyses (Bartlett et al., 2019)) to analyze deep active learning, our theoretical guarantees are empowered by novel extensions of active learning algorithms under neural network approximations .

Original: In particular, we deal with approximation error in active learning under Tsybakov noise, and identify conditions that greatly relax the approximation requirement in the learning with abstention setup; we also analyze the disagreement coefﬁcient, both classiﬁer-based and value function-based, with a set of neural networks.
Revised: In particular, we deal with approximation error in active learning under Tsybakov noise, and identify conditions that greatly relax the approximation requirement in the learning with abstention setup; we also analyze the disagreement coefﬁcient, both classiﬁer-based and value function-based, with a set of neural networks.

Original: These analyses together lead to our main results for deep active learning (e.g., Theorem 1 and Theorem 2).
Revised: These analyses together lead to our main results for deep active learning (e.g., Theorem 1 and Theorem 2).

Original: More generally, we establish a bridge between approximation theory and active learning; we provide these general guarantees in
Revised: More generally, we establish a bridge between approximation theory and active learning; we provide these general guarantees in

Original: Appendix B (under Tsybakov noise) and Appendix D (with the abstention option), which can be of independent interests.
Revised: Appendix B (under Tsybakov noise) and Appendix D (with the abstention option), which can be of independent interests.

Original: Beneﬁted from these generic algorithms and guarantees, in Section 4, we extend our results into learning smooth functions in the Radon BV 2 space (Ongie et al., 2020; Parhi and Nowak, 2021, 2022a,b; Unser, 2022), which is recently proposed as a natural space to analyze neural networks.
Revised: Beneﬁted from these generic algorithms and guarantees, in Section 4, we extend our results into learning smooth functions in the Radon BV 2 space (Ongie et al., 2020; Parhi and Nowak, 2021, 2022a,b; Unser, 2022), which is recently proposed as a natural space to analyze neural networks.

Original: 1.3 Related work
Revised: 1.3 Related work

Original: Active learning concerns about learning accurate classiﬁers without extensive human labeling.
Revised: Active learning concerns about learning accurate classiﬁers without extensive human labeling.

Original: One of the earliest work of active learning dates back to the CAL algorithm proposed by Cohn et al. (1994), which set the cornerstone for disagreement-based active learning.
Revised: One of the earliest work of active learning dates back to the CAL algorithm proposed by Cohn et al. (1994), which set the cornerstone for disagreement-based active learning.

Original: Since then, a long line of work have been developed, either directly working with a set classiﬁer (Balcan et al., 2006; Hanneke, 2007;
Revised: Since then, a long line of work have been developed, either directly working with a set classiﬁer (Balcan et al., 2006; Hanneke, 2007;

Original: Dasgupta et al., 2007; Beygelzimer et al., 2009, 2010; Huang et al., 2015; Cortes et al., 2019) or work with a set of regression functions (Krishnamurthy et al., 2017, 2019).
Revised: Dasgupta et al., 2007; Beygelzimer et al., 2009, 2010; Huang et al., 2015; Cortes et al., 2019) or work with a set of regression functions (Krishnamurthy et al., 2017, 2019).

Original: These work mainly focus on the parametric regime (e.g., learning with a set of linear classiﬁers), and their label complexities rely on the boundedness of the so-called disagreement coefﬁcient (Hanneke, 2007, 2014; Friedman, 2009).
Revised: These work mainly focus on the parametric regime (e.g., learning with a set of linear classiﬁers), and their label complexities rely on the boundedness of the so-called disagreement coefﬁcient (Hanneke, 2007, 2014; Friedman, 2009).

Original: Active learning in the nonparametric regime has been analyzed in Castro and Nowak (2008); Minsker (2012); Locatelli et al.
Revised: Active learning in the nonparametric regime has been analyzed in Castro and Nowak (2008); Minsker (2012); Locatelli et al.

Original: ; Kpotufe et al.
Revised: ; Kpotufe et al.

Original: These algorithms rely on partitioning of the input space X ⊆ [0 , 1] d into exponentially (in dimension) many small cubes, and then conduct local mean (or some higher-order statistics) estimation within each small cube.
Revised: These algorithms rely on partitioning of the input space X ⊆ [0 , 1] d into exponentially (in dimension) many small cubes, and then conduct local mean (or some higher-order statistics) estimation within each small cube.

Original: It is well known that, in the worst case, active learning exhibits no label complexity gains over the passive counterpart (Kääriäinen, 2006).
Revised: It is well known that, in the worst case, active learning exhibits no label complexity gains over the passive counterpart (Kääriäinen, 2006).

Original: To bypass these worst-case scenarios, active learning has been popularly analyzed under the so-called Tsybakov low noise conditions (Tsybakov, 2004).
Revised: To bypass these worst-case scenarios, active learning has been popularly analyzed under the so-called Tsybakov low noise conditions (Tsybakov, 2004).

Original: Under Tsybakov noise conditions, active learning has been shown to be strictly superior than passive learning in terms of label complexity (Castro and Nowak, 2008; Locatelli et al., 2017).
Revised: Under Tsybakov noise conditions, active learning has been shown to be strictly superior than passive learning in terms of label complexity (Castro and Nowak, 2008; Locatelli et al., 2017).

Original: Besides analyzing active learning under favorable low noise assumptions, more recently, researchers consider active learning with an abstention option and analyze its label complexity under Chow’s error (Chow, 1970).
Revised: Besides analyzing active learning under favorable low noise assumptions, more recently, researchers consider active learning with an abstention option and analyze its label complexity under Chow’s error (Chow, 1970).

Original: In particular, Puchkin and Zhivotovskiy (2021); Zhu and Nowak (2022) develop active learning algorithms with polylog( 1 ε ) label complexity when analyzed under Chow’s excess error.
Revised: In particular, Puchkin and Zhivotovskiy (2021); Zhu and Nowak (2022) develop active learning algorithms with polylog( 1 ε ) label complexity when analyzed under Chow’s excess error.

Original: Shekhar et al.
Revised: Shekhar et al.

Original: (2021) study nonparametric active learning under a different notion of the Chow’s excess error, and propose algorithms with poly( 1 ε ) label complexity; their algorithms follow similar procedures of those partitionbased nonparametric active learning algorithms (e.g., Minsker (2012);
Revised: (2021) study nonparametric active learning under a different notion of the Chow’s excess error, and propose algorithms with poly( 1 ε ) label complexity; their algorithms follow similar procedures of those partitionbased nonparametric active learning algorithms (e.g., Minsker (2012);

Original: Locatelli et al.
Revised: Locatelli et al.

Original: Inspired by the success of deep learning in the passive regime, active learning with neural networks has been extensively explored in recent years (Sener and Savarese, 2018; Ash et al., 2019; Citovsky et al., 2021; Ash et al., 2021; Kothawade et al., 2021;
Revised: Inspired by the success of deep learning in the passive regime, active learning with neural networks has been extensively explored in recent years (Sener and Savarese, 2018; Ash et al., 2019; Citovsky et al., 2021; Ash et al., 2021; Kothawade et al., 2021;

Original: Emam et al., 2021;
Revised: Emam et al., 2021;

Original: Ren et al., 2021).
Revised: Ren et al., 2021).

Original: Great empirical performances are observed in these papers, however, rigorous label complexity guarantees have largely remains elusive (except in Karzand and Nowak (2020);
Revised: Great empirical performances are observed in these papers, however, rigorous label complexity guarantees have largely remains elusive (except in Karzand and Nowak (2020);

Original: Wang et al.
Revised: Wang et al.

Original: (2021), with limitations discussed before).
Revised: (2021), with limitations discussed before).

Original: We bridge the gap between practice and theory by providing the ﬁrst near-optimal label complexity guarantees for deep active learning.
Revised: We bridge the gap between practice and theory by providing the ﬁrst near-optimal label complexity guarantees for deep active learning.

Original: Our results are built upon approximation results of deep neural networks (Yarotsky, 2017, 2018; Parhi and Nowak, 2022b) and VC/pseudo dimension analyses of neural networks with given structures (Bartlett et al., 2019).
Revised: Our results are built upon approximation results of deep neural networks (Yarotsky, 2017, 2018; Parhi and Nowak, 2022b) and VC/pseudo dimension analyses of neural networks with given structures (Bartlett et al., 2019).

Original:  Label complexity of deep active learning
Revised:  Label complexity of deep active learning

Original: We analyze the label complexity of deep active learning in this section.
Revised: We analyze the label complexity of deep active learning in this section.

Original: We ﬁrst introduce the
Revised: We ﬁrst introduce the

Original: Tsybakov noise condition in Section 2.1, and then identify the “right tools” to analyze classiﬁcation problems with neural network classiﬁers in Section 2.2 (where we also provide passive learning guarantees).
Revised: Tsybakov noise condition in Section 2.1, and then identify the “right tools” to analyze classiﬁcation problems with neural network classiﬁers in Section 2.2 (where we also provide passive learning guarantees).

Original: We establish our main active learning guarantees in Section 2.3.
Revised: We establish our main active learning guarantees in Section 2.3.

Original: 2.1 Tsybakov noise condition
Revised: 2.1 Tsybakov noise condition

Original: It is well known that active learning exhibits no label complexity gains over the passive counterpart without additional low noise assumptions (Kääriäinen, 2006).
Revised: It is well known that active learning exhibits no label complexity gains over the passive counterpart without additional low noise assumptions (Kääriäinen, 2006).

Original: We next introduce the Tsybokov low noise condition (Tsybakov, 2004), which has been extensively analyzed in active learning literature.
Revised: We next introduce the Tsybokov low noise condition (Tsybakov, 2004), which has been extensively analyzed in active learning literature.

Original: Deﬁnition 1 (Tsybakov noise) .
Revised: Deﬁnition 1 (Tsybakov noise) .

Original: The marginal distribution D X satisﬁes the Tsybakov noise condition with parameter β ≥ 0 and  universal constants [Equation] if
Revised: A distribution D XY satisﬁes the Tsybakov noise condition with parameter β ≥ 0 and a universal constant c ≥ 1 if, ∀ τ > 0 ,

Original: [Equation]
Revised: [Equation]

Original: The case with β = 0 corresponds to the general case without any low noise conditions, where no active learning algorithm can outperform the passive counterpart (Audibert and Tsybakov, 2007;
Revised: The case with β = 0 corresponds to the general case without any low noise conditions, where no active learning algorithm can outperform the passive counterpart (Audibert and Tsybakov, 2007;

Original: Locatelli et al., 2017).
Revised: Locatelli et al., 2017).

Original: We use P ( α, β ) to denote the set of distributions satisfying: (i) the smoothness conditions introduced in Section 1.1 with parameter α > 0 ; and (ii) the Tsybakov low noise condition (i.e., Deﬁnition 1) with parameter β ≥ 0 .
Revised: We use P ( α, β ) to denote the set of distributions satisfying: (i) the smoothness conditions introduced in Section 1.1 with parameter α > 0 ; and (ii) the Tsybakov low noise condition (i.e., Deﬁnition 1) with parameter β ≥ 0 .

Original: We assume D XY ∈ P ( α, β ) in the rest of Section 2.
Revised: We assume D XY ∈ P ( α, β ) in the rest of Section 2.

Original: As in Castro and Nowak (2008); Hanneke (2014), we assume the knowledge of noise/smoothness parameters.
Revised: As in Castro and Nowak (2008); Hanneke (2014), we assume the knowledge of noise/smoothness parameters.

Original: 2.2 Approximation and expressiveness of neural networks
Revised: 2.2 Approximation and expressiveness of neural networks

Original: Neural networks are known to be universal approximators (Cybenko, 1989; Hornik, 1991):
Revised: Neural networks are known to be universal approximators (Cybenko, 1989; Hornik, 1991):

Original: For any continuous function g : X → R and any error tolerance κ > 0 , there exists a large enough neural network f dnn such that k f dnn − g k ∞ :
Revised: For any continuous function g : X → R and any error tolerance κ > 0 , there exists a large enough neural network f dnn such that k f dnn − g k ∞ :

Original: = sup x ∈X | f dnn ( x ) − g ( x ) | ≤ κ .
Revised: = sup x ∈X | f dnn ( x ) − g ( x ) | ≤ κ .

Original: Recently, non-asympototic approximation rates by ReLU neural networks have been developed for smooth functions in the
Revised: Recently, non-asympototic approximation rates by ReLU neural networks have been developed for smooth functions in the

Original: Sobolev space, which we restate in the following.
Revised: Sobolev space, which we restate in the following.

Original: Theorem 3 (Yarotsky (2017)) .
Revised: Theorem 3 (Yarotsky (2017)) .

Original: Fix any κ > 0 .
Revised: Fix any κ > 0 .

Original: For any f ?
Revised: For any f ?

Original: =
Revised: =

Original: η ∈ W α, ∞([ − 1 , 1] d ) , there exists a neural network f dnn with W 
Revised: η ∈ W α, ∞([0 , 1] d ) , there exists a neural network f dnn with W =

Original: = O ( κ − dα log 1 κ ) total number of parameters arranged in L
Revised: O ( κ − dα log 1 κ ) total number of parameters arranged in L

Original: = O (log 1 κ ) layers such that k f dnn − f ? k ∞ ≤ κ .
Revised: = O (log 1 κ ) layers such that k f dnn − f ? k ∞ ≤ κ .

Original: The architecture of the neural network f dnn appearing in the above theorem only depends on the smooth function space W α, ∞ [Equation] , but otherwise is independent of the true regression function f ? ; also see Yarotsky (2017) for details.
Revised: The architecture of the neural network f dnn appearing in the above theorem only depends on the smooth function space W α, ∞([0 , 1] d ) , but otherwise is independent of the true regression function f ? ; also see Yarotsky (2017) for details.

Original: Let F dnn denote the set of neural network regression functions with the same architecture.
Revised: Let F dnn denote the set of neural network regression functions with the same architecture.

Original: We construct a set of neural network classiﬁers by thresholding the regression function at 12 , i.e., H dnn :=
Revised: We construct a set of neural network classiﬁers by thresholding the regression function at 12 , i.e., H dnn :=

Original: { h f :=
Revised: { h f :=

Original: sign(2 f ( x ) − 1) : f ∈ F dnn } .
Revised: sign(2 f ( x ) − 1) : f ∈ F dnn } .

Original: The next result concerns about the expressiveness of the neural network classiﬁers, in terms of a well-known complexity measure: the VC dimension (Vapnik and Chervonenkis, 1971).
Revised: The next result concerns about the expressiveness of the neural network classiﬁers, in terms of a well-known complexity measure: the VC dimension (Vapnik and Chervonenkis, 1971).

Original: Theorem 4 (Bartlett et al.
Revised: Theorem 4 (Bartlett et al.

Original: Let H dnn be a set of neural network classiﬁers of the same architecture and with W parameters arranged in L layers.
Revised: Let H dnn be a set of neural network classiﬁers of the same architecture and with W parameters arranged in L layers.

Original: We then have
Revised: We then have

Original: [Equation]
Revised: [Equation]

Original: With these tools, we can construct a set of neural network classiﬁers H dnn such that (i) the best in-class classiﬁer ˇ h ∈ H dnn has small excess error, and (ii) H dnn has a well-controlled VC dimension that is proportional to smooth/noise parameters.
Revised: With these tools, we can construct a set of neural network classiﬁers H dnn such that (i) the best in-class classiﬁer ˇ h ∈ H dnn has small excess error, and (ii) H dnn has a well-controlled VC dimension that is proportional to smooth/noise parameters.

Original: More speciﬁcally, we have the following proposition.
Revised: More speciﬁcally, we have the following proposition.

Original: Proposition 1.
Revised: Proposition 1.

Original: Suppose D XY ∈ P
Revised: Suppose D XY ∈ P

Original: ( α, β ) .
Revised: ( α, β ) .

Original: One can construct a set of neural network classiﬁer H dnn such that the following two properties hold simultaneously:
Revised: One can construct a set of neural network classiﬁer H dnn such that the following two properties hold simultaneously:

Original: [Equation]
Revised: [Equation]

Original: With the approximation results obtained above, to learn a classiﬁer with O ( ε ) excess error, one only needs to focus on a set of neural networks H dnn with a well-controlled VC dimension.
Revised: With the approximation results obtained above, to learn a classiﬁer with O ( ε ) excess error, one only needs to focus on a set of neural networks H dnn with a well-controlled VC dimension.

Original: As a warm-up, we ﬁrst analyze the label complexity of such procedure in the passive regime (with fast rates).
Revised: As a warm-up, we ﬁrst analyze the label complexity of such procedure in the passive regime (with fast rates).

Original: Theorem 5.
Revised: Theorem 5.

Original: Suppose D XY ∈ P
Revised: Suppose D XY ∈ P

Original: ( α, β ) .
Revised: ( α, β ) .

Original: Fix any ε, δ > 0 .
Revised: Fix any ε, δ > 0 .

Original: Let H dnn be the set of neural network classiﬁers constructed in Proposition 1.
Revised: Let H dnn be the set of neural network classiﬁers constructed in Proposition 1.

Original: With 
Revised: With n = e O

Original: [Equation]
Revised: [Equation]

Original: i
Revised: ε

Original: .d.
Revised: [Equation] ) ) i.i.d.

Original: sampled points, with probability at least 1 − δ , the empirical risk minimizer
Revised: sampled points, with probability at least 1 − δ , the empirical risk minimizer

Original: [Equation]
Revised: [Equation]

Original: achieves excess error O [Equation] ε ) .
Revised: achieves excess error O ( ε ) .

Original: The label complexity results obtained in Theorem 5 matches, up to logarithmic factors, the passive learning lower bound Ω( ε − d +2 α
Revised: The label complexity results obtained in Theorem 5 matches, up to logarithmic factors, the passive learning lower bound Ω( ε − d +2 α

Original: + αβ α (1+ β ) ) established in Audibert and Tsybakov (2007), indicating  our proposed learning procedure with a set of neural networks is near minimax optimal.
Revised: + αβ α (1+ β ) ) established in Audibert and Tsybakov (2007), indicating that our proposed learning procedure with a set of neural networks is near minimax optimal.

Original: 2.3 Deep active learning and guarantees
Revised: 2.3 Deep active learning and guarantees

Original: The passive learning procedure presented in the previous section treats every data point equally, i.e., it requests the label of every data point.
Revised: The passive learning procedure presented in the previous section treats every data point equally, i.e., it requests the label of every data point.

Original: Active learning reduces the label complexity by only querying labels of data points that are “more important”.
Revised: Active learning reduces the label complexity by only querying labels of data points that are “more important”.

Original: We present deep active learning results in this section.
Revised: We present deep active learning results in this section.

Original: Our algorithm (Algorithm 1) is inspired by RobustCAL (Balcan et al., 2006; Hanneke, 2007, 2014) and the seminal CAL algorithm (Cohn et al., 1994); we call our algorithm NeuralCAL to emphasize that it works with a set of neural networks.
Revised: Our algorithm (Algorithm 1) is inspired by RobustCAL (Balcan et al., 2006; Hanneke, 2007, 2014) and the seminal CAL algorithm (Cohn et al., 1994); we call our algorithm NeuralCAL to emphasize that it works with a set of neural networks.

Original: For any accuracy level ε > 0 , NeuralCAL ﬁrst initialize a set of neural network classiﬁers H 0 :=
Revised: For any accuracy level ε > 0 , NeuralCAL ﬁrst initialize a set of neural network classiﬁers H 0 :=

Original: H dnn such that (i) the best in-class classiﬁer ˇ h :=
Revised: H dnn such that (i) the best in-class classiﬁer ˇ h :=

Original: arg min h ∈H dnn err( h ) has excess error at most O ( ε ) , and (ii) the VC dimension of H dnn is upper bounded by
Revised: arg min h ∈H dnn err( h ) has excess error at most O ( ε ) , and (ii) the VC dimension of H dnn is upper bounded by

Original: [Equation]
Revised: [Equation]

Original: ) (see Section 2.2 for more details).
Revised: ) (see Section 2.2 for more details).

Original: NeuralCAL then runs in epochs of geometrically increasing lengths.
Revised: NeuralCAL then runs in epochs of geometrically increasing lengths.

Original: At the beginning of epoch m , based on previously labeled data points, NeuralCAL updates a set of active classiﬁer H m such that, with high probability, the best classiﬁer
Revised: At the beginning of epoch m , based on previously labeled data points, NeuralCAL updates a set of active classiﬁer H m such that, with high probability, the best classiﬁer

Original: ˇ h remains uneliminated .
Revised: ˇ h remains uneliminated .

Original: Within each epoch m , NeuralCAL only queries the label y of a data point x if it lies in the region of disagreement with respect to the current active set of classiﬁer H m , i.e., DIS ( H m ) :
Revised: Within each epoch m , NeuralCAL only queries the label y of a data point x if it lies in the region of disagreement with respect to the current active set of classiﬁer H m , i.e., DIS ( H m ) :

Original: = { x ∈ X : ∃ h 1 , h 2 ∈
Revised: = { x ∈ X : ∃ h 1 , h 2 ∈

Original: H m s.t.
Revised: H m s.t.

Original: h 1 ( x ) 6 =
Revised: h 1 ( x ) 6 =

Original: h 2 ( x ) } .
Revised: h 2 ( x ) } .

Original: NeuralCAL returns any classiﬁer b h ∈ H m that remains uneliminated after M − 1 epoch.
Revised: NeuralCAL returns any classiﬁer b h ∈ H m that remains uneliminated after M − 1 epoch.

Original: Input: Accuracy level ε ∈ (0 , 1) , conﬁdence level δ ∈ (0 , 1) .
Revised: Input: Accuracy level ε ∈ (0 , 1) , conﬁdence level δ ∈ (0 , 1) .

Original: 1: Let H dnn be a set of neural networks classiﬁers constructed in Proposition 1.
Revised: 1: Let H dnn be a set of neural networks classiﬁers constructed in Proposition 1.

Original: 2: Deﬁne
Revised: 2: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: := d log 2 T e , τ m := 2 m for m ≥ 1 and τ 0 := 0 .
Revised: := d log 2 T e , τ m := 2 m for m ≥ 1 and τ 0 := 0 .

Original: 3: Deﬁne ρ m :=
Revised: 3: Deﬁne ρ m :=

Original: O (cid:18)(cid:16)
Revised: O (cid:18)(cid:16)

Original: VCdim(
Revised: VCdim(

Original: [Table]
Revised: [Table]

Original: dnn ) · log(
Revised: dnn ) · log(

Original: [Table]
Revised: [Table]

Original: m
Revised: m

Original: [Table]
Revised: [Table]

Original: ) · log( M/
Revised: ) · log( M/

Original: [Table]
Revised: [Table]

Original: ) τ m −(cid:17)
Revised: ) τ m −(cid:17)

Original: [Table]
Revised: [Table]

Original: β
Revised: β

Original: [Table]
Revised: [Table]

Original: β (cid:19) for m ≥ 2 and ρ 1 := 1 .
Revised: β (cid:19) for m ≥ 2 and ρ 1 := 1 .

Original: 4: Deﬁne b
Revised: 4: Deﬁne b

Original: R m ( h ) :=
Revised: R m ( h ) :=

Original: P τ m − 1 t =1 Q t 1 ( h ( x t ) 6 = y t ) with the convention that
Revised: P τ m − 1 t =1 Q t 1 ( h ( x t ) 6 = y t ) with the convention that

Original: P 0 t =1 . . .
Revised: P 0 t =1 . . .

Original: = 0 .
Revised: = 0 .

Original: 5: Initialize H 0 :=
Revised: 5: Initialize H 0 :=

Original: H dnn .
Revised: H dnn .

Original: 6: for epoch m
Revised: 6: for epoch m

Original: = 1 , 2 , . .
Revised: = 1 , 2 , . .

Original: , M do 7:
Revised: , M do 7:

Original: Update active set
Revised: Update active set

Original: H m := n
Revised: H m := n

Original: [Equation]
Revised: [Equation]

Original: H m
Revised: H m

Original: [Equation]
Revised: [Equation]

Original: : b
Revised: : b

Original: [Equation]
Revised: [Equation]

Original: m −b
Revised: m −b

Original: R m ( h )
Revised: R m ( h )

Original: [Equation]
Revised: [Equation]

Original: τ
Revised: τ

Original: [Equation]
Revised: [Equation]

Original: m o 8: if epoch m =
Revised: m o 8: if epoch m =

Original: M then 9:
Revised: M then 9:

Original: Return any classiﬁer b h ∈ H M . 10: for time t =
Revised: Return any classiﬁer b h ∈ H M . 10: for time t =

Original: τ m − 1 + 1 , . .
Revised: τ m − 1 + 1 , . .

Original: , τ m do 11:
Revised: , τ m do 11:

Original: Observe x t ∼ D X .
Revised: Observe x t ∼ D X .

Original: Set Q t := 1 ( x t ∈ DIS ( H m )) .
Revised: Set Q t := 1 ( x t ∈ DIS ( H m )) .

Original: 12: if Q t = 1 then 13:
Revised: 12: if Q t = 1 then 13:

Original: Query the label y t of x t .
Revised: Query the label y t of x t .

Original: Since NeuralCAL only queries labels of data points lying in the region of disagreement, its label complexity should intuitively be related to how fast the region of disagreement shrinks.
Revised: Since NeuralCAL only queries labels of data points lying in the region of disagreement, its label complexity should intuitively be related to how fast the region of disagreement shrinks.

Original: More formally, the rate of collapse of the (probability measure of) region of disagreement is captured by the (classiﬁer-based) disagreement coefﬁcient (Hanneke, 2007, 2014), which we introduce next.
Revised: More formally, the rate of collapse of the (probability measure of) region of disagreement is captured by the (classiﬁer-based) disagreement coefﬁcient (Hanneke, 2007, 2014), which we introduce next.

Original: Deﬁnition 2 (Classiﬁer-based disagreement coefﬁcient) .
Revised: Deﬁnition 2 (Classiﬁer-based disagreement coefﬁcient) .

Original: For any ε 0 and classiﬁer h ∈ H , the classiﬁer-based disagreement coefﬁcient of h is deﬁned as
Revised: For any ε 0 and classiﬁer h ∈ H , the classiﬁer-based disagreement coefﬁcient of h is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where B H ( h, ε ) :
Revised: where B H ( h, ε ) :

Original: = { g ∈ H : P ( x ∈ X : g ( x ) 6 = h ( x )) ≤ ε } .
Revised: = { g ∈ H : P ( x ∈ X : g ( x ) 6 = h ( x )) ≤ ε } .

Original: We also deﬁne θ H ( ε 0 ) :
Revised: We also deﬁne θ H ( ε 0 ) :

Original: = sup h ∈H θ H ,h ( ε 0 ) .
Revised: = sup h ∈H θ H ,h ( ε 0 ) .

Original: The guarantees of NeuralCAL follows from a more general analysis of RobustCAL under approximation.
Revised: The guarantees of NeuralCAL follows from a more general analysis of RobustCAL under approximation.

Original: In particular, to achieve fast rates (under Tsybakov noise), previous analysis of RobustCAL requires that the Bayes classiﬁer is in the class (or a Bernstein condition for every h ∈ H )
Revised: In particular, to achieve fast rates (under Tsybakov noise), previous analysis of RobustCAL requires that the Bayes classiﬁer is in the class (or a Bernstein condition for every h ∈ H )

Original: (Hanneke, 2014).
Revised: (Hanneke, 2014).

Original: These requirements are stronger compared to what we have in the case with neural network approximations.
Revised: These requirements are stronger compared to what we have in the case with neural network approximations.

Original: Our analysis extends the understanding of RobustCAL under approximation.
Revised: Our analysis extends the understanding of RobustCAL under approximation.

Original: We defer such general analysis to Appendix B, and present the following guarantees.
Revised: We defer such general analysis to Appendix B, and present the following guarantees.

Original: Theorem 6.
Revised: Theorem 6.

Original: Suppose D XY ∈ P
Revised: Suppose D XY ∈ P

Original: ( α, β ) .
Revised: ( α, β ) .

Original: Fix any ε, δ > 0 .
Revised: Fix any ε, δ > 0 .

Original: With probability at least 1 − δ , Algorithmreturns a classiﬁer b h ∈ H dnn with excess error e
Revised: With probability at least 1 − δ , Algorithmreturns a classiﬁer b h ∈ H dnn with excess error e

Original: O ( ε ) after querying
Revised: O ( ε ) after querying

Original: [Equation]
Revised: [Equation]

Original: We next discuss in detail the label complexity of deep active learning proved in Theorem 6.
Revised: We next discuss in detail the label complexity of deep active learning proved in Theorem 6.

Original: Ignoring the dependence on disagreement coefﬁcient, the label complexity appearing in Theoremmatches, up to logarithmic factors, the lower bound Ω( ε − d +

Revised: Ignoring the dependence on disagreement coefﬁcient, the label complexity appearing in Theoremmatches, up to logarithmic factors, the lower bound Ω( ε − d +

Original: α α + αβ ) for active learning (Locatelli et al., 2017).
Revised: α α + αβ ) for active learning (Locatelli et al., 2017).

Original: At the same time, the label complexity appearing in Theorem 6 is never worse than the passive counterpart (i.e.,
Revised: At the same time, the label complexity appearing in Theorem 6 is never worse than the passive counterpart (i.e.,

Original: [Equation]
Revised: [Equation]

Original: θ
Revised: θ

Original: [Equation]
Revised: [Equation]

Original: dnn
Revised: dnn

Original: [Equation]
Revised: [Equation]

Original: • We also identify cases when θ H dnn ( ε β 1+ β )
Revised: • We also identify cases when θ H dnn ( ε β 1+ β )

Original: =
Revised: =

Original: o ( ε − β 1+ β ) , indicating strict improvement over passive learning (e.g., when D X is supported on countably many data points), and when θ H dnn ( ε β 1+ β ) =
Revised: o ( ε − β 1+ β ) , indicating strict improvement over passive learning (e.g., when D X is supported on countably many data points), and when θ H dnn ( ε β 1+ β ) =

Original: O (1) , indicating matching the minimax active lower bound (e.g., when D XY satisﬁes conditions such as decomposibility deﬁned in Deﬁnition 4.
Revised: O (1) , indicating matching the minimax active lower bound (e.g., when D XY satisﬁes conditions such as decomposibility deﬁned in Deﬁnition 4.

Original: See Appendix C.2 for detailed discussion).
Revised: See Appendix C.2 for detailed discussion).

Original: Our algorithm and theorems lead to the following results, which could beneﬁt both deep active learning and nonparametric learning communities.
Revised: Our algorithm and theorems lead to the following results, which could beneﬁt both deep active learning and nonparametric learning communities.

Original: • Near minimax optimal label complexity for deep active learning.
Revised: • Near minimax optimal label complexity for deep active learning.

Original: While empirical successes of deep active learning have been observed, rigorous label complexity analysis remains elusive except for two attempts made in Karzand and Nowak (2020);
Revised: While empirical successes of deep active learning have been observed, rigorous label complexity analysis remains elusive except for two attempts made in Karzand and Nowak (2020);

Original: Wang et al.
Revised: Wang et al.

Original: The guarantees provided in Karzand and Nowak (2020) only work in very special cases (i.e., data uniformly sampled from [0 , 1] and labeled by well-separated piece-constant functions in a noise-free way).
Revised: The guarantees provided in Karzand and Nowak (2020) only work in very special cases (i.e., data uniformly sampled from [0 , 1] and labeled by well-separated piece-constant functions in a noise-free way).

Original: Wang et al.
Revised: Wang et al.

Original: (2021) study deep active learning in the NTK regime by linearizing the neural network at its random initialization and analyzing it as a linear function; moreover, as the authors agree, their error bounds and label complexity guarantees are vacuous in certain cases.
Revised: (2021) study deep active learning in the NTK regime by linearizing the neural network at its random initialization and analyzing it as a linear function; moreover, as the authors agree, their error bounds and label complexity guarantees are vacuous in certain cases.

Original: On the other hand, our guarantees are minimax optimal, up to disagreement coefﬁcient and other logarithmic factors, which bridge the gap between theory and practice in deep active learning.
Revised: On the other hand, our guarantees are minimax optimal, up to disagreement coefﬁcient and other logarithmic factors, which bridge the gap between theory and practice in deep active learning.

Original: • New perspective on nonparametric learning.
Revised: • New perspective on nonparametric learning.

Original: Nonparametric learning of smooth functions have been mainly approached by partitioning-based methods (Tsybakov, 2004; Audibert and Tsybakov, 2007; Castro and Nowak, 2008; Minsker, 2012; Locatelli et al., 2017, 2018;
Revised: Nonparametric learning of smooth functions have been mainly approached by partitioning-based methods (Tsybakov, 2004; Audibert and Tsybakov, 2007; Castro and Nowak, 2008; Minsker, 2012; Locatelli et al., 2017, 2018;

Original: Kpotufe et al., 2021) :
Revised: Kpotufe et al., 2021) :

Original: Partition the unit cube [0 , 1] d into exponentially (in dimension) many sub-cubes and conduct local mean estimation within each sub-cube (which additionally requires a strictly stronger membership querying oracle).
Revised: Partition the unit cube [0 , 1] d into exponentially (in dimension) many sub-cubes and conduct local mean estimation within each sub-cube (which additionally requires a strictly stronger membership querying oracle).

Original: Our results show that, in both passive and active settings, one can learn globally with a set of neural networks and achieve near minimax optimal label complexities.
Revised: Our results show that, in both passive and active settings, one can learn globally with a set of neural networks and achieve near minimax optimal label complexities.

Original:  Deep active learning with abstention: Exponential speedups
Revised:  Deep active learning with abstention: Exponential speedups

Original: While the theoretical guarantees provided in Section 2 are near minimax optimal, the label complexity scales as poly( 1 ε ) , which doesn’t match the great empirical performance observed in deep active learning.
Revised: While the theoretical guarantees provided in Section 2 are near minimax optimal, the label complexity scales as poly( 1 ε ) , which doesn’t match the great empirical performance observed in deep active learning.

Original: In this section, we ﬁll in this gap by leveraging the idea of abstention and provide a deep active learning algorithm that achieves exponential label savings.
Revised: In this section, we ﬁll in this gap by leveraging the idea of abstention and provide a deep active learning algorithm that achieves exponential label savings.

Original: We introduce the concepts of abstention and Chow’s excess error in Section 3.1, and provide our label complexity guarantees in Section 3.2.
Revised: We introduce the concepts of abstention and Chow’s excess error in Section 3.1, and provide our label complexity guarantees in Section 3.2.

Original: 3.1 Active learning without low noise conditions
Revised: 3.1 Active learning without low noise conditions

Original: The previous section analyzes active learning under Tsybakov noise, which has been extensively studied in the literature since Castro and Nowak (2008).
Revised: The previous section analyzes active learning under Tsybakov noise, which has been extensively studied in the literature since Castro and Nowak (2008).

Original: More recently, promising results are observed in active learning under Chow’s excess error, but otherwise without any low noise assumption (Puchkin and Zhivotovskiy, 2021; Zhu and Nowak, 2022).
Revised: More recently, promising results are observed in active learning under Chow’s excess error, but otherwise without any low noise assumption (Puchkin and Zhivotovskiy, 2021; Zhu and Nowak, 2022).

Original: We introduce this setting in the following.
Revised: We introduce this setting in the following.

Original: Abstention and Chow’s error (Chow, 1970).
Revised: Abstention and Chow’s error (Chow, 1970).

Original: We consider classiﬁer of the form b h : X → Y∪{⊥} where ⊥ denotes the action of abstention.
Revised: We consider classiﬁer of the form b h : X → Y∪{⊥} where ⊥ denotes the action of abstention.

Original: For any ﬁxed 0 < γ < 12 , the Chow’s error is deﬁned as
Revised: For any ﬁxed 0 < γ < 12 , the Chow’s error is deﬁned as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The parameter γ can be chosen as a small constant, e.g., γ = 0 .
Revised: The parameter γ can be chosen as a small constant, e.g., γ = 0 .

Original: 01 , to avoid excessive abstention: The price of abstention is only marginally smaller than random guess (which incurs cost 0 . 5 ).
Revised: 01 , to avoid excessive abstention: The price of abstention is only marginally smaller than random guess (which incurs cost 0 . 5 ).

Original: The Chow’s excess error is then deﬁned as excess γ ( b h ) :=
Revised: The Chow’s excess error is then deﬁned as excess γ ( b h ) :=

Original: err γ ( b h ) − err( h ? )
Revised: err γ ( b h ) − err( h ? )

Original: (Puchkin and Zhivotovskiy, 2021).
Revised: (Puchkin and Zhivotovskiy, 2021).

Original: At a high level, analyzing with Chow’s excess error allows slackness in predications of hard examples (e.g., data points whose η ( x ) is close to 12 ) by leveraging the power of abstention.
Revised: At a high level, analyzing with Chow’s excess error allows slackness in predications of hard examples (e.g., data points whose η ( x ) is close to 12 ) by leveraging the power of abstention.

Original: Puchkin and
Revised: Puchkin and

Original: Zhivotovskiy (2021); Zhu and Nowak (2022) show that polylog( 1 ε ) is always achievable in the parametric settings.
Revised: Zhivotovskiy (2021); Zhu and Nowak (2022) show that polylog( 1 ε ) is always achievable in the parametric settings.

Original: We generalize their results to the nonparametric setting and analyze active learning with a set of neural networks.
Revised: We generalize their results to the nonparametric setting and analyze active learning with a set of neural networks.

Original: 3.2 Exponential speedups with abstention
Revised: 3.2 Exponential speedups with abstention

Original: In this section, we work with a set of neural network regression functions F dnn : X → [0 , 1] (that approximates η ) and then construct classiﬁers h : X → Y ∪ {⊥} with an additional abstention action .
Revised: In this section, we work with a set of neural network regression functions F dnn : X → [0 , 1] (that approximates η ) and then construct classiﬁers h : X → Y ∪ {⊥} with an additional abstention action .

Original: To work with a set of regression functions
Revised: To work with a set of regression functions

Original: F dnn , we analyze its “complexity” from the lenses of pseudo dimension Pdim(
Revised: F dnn , we analyze its “complexity” from the lenses of pseudo dimension Pdim(

Original: F dnn )
Revised: F dnn )

Original: (Pollard, 1984; Haussler, 1989, 1995) and value function disagreement coefﬁcient θ val F dnn ( ι ) (for some ι > 0 ) (Foster et al., 2020).
Revised: (Pollard, 1984; Haussler, 1989, 1995) and value function disagreement coefﬁcient θ val F dnn ( ι ) (for some ι > 0 ) (Foster et al., 2020).

Original: We defer detailed deﬁnitions of these complexity measures to Appendix D.1.
Revised: We defer detailed deﬁnitions of these complexity measures to Appendix D.1.

Original: Algorithm
Revised: Algorithm

Original: Input: Accuracy level ε ∈ (0 , 1) , conﬁdence level δ ∈ (0 , 1) , abstention parameter γ ∈ (0 , 1 / 2) .
Revised: Input: Accuracy level ε ∈ (0 , 1) , conﬁdence level δ ∈ (0 , 1) , abstention parameter γ ∈ (0 , 1 / 2) .

Original: 1: Let F dnn be a set of neural network regression functions obtained by (i) applying Theoremwith an appropriate approximation level κ (which satisﬁes 1 κ = poly( 1 γ ) polylog( 1 εγ ) ), and (ii) applying a preprocessing step on the set of neural networks obtained from step (i).
Revised: 1: Let F dnn be a set of neural network regression functions obtained by (i) applying Theoremwith an appropriate approximation level κ (which satisﬁes 1 κ = poly( 1 γ ) polylog( 1 εγ ) ), and (ii) applying a preprocessing step on the set of neural networks obtained from step (i).

Original: See
Revised: See

Original: Appendix E for details.
Revised: Appendix E for details.

Original: 2: Deﬁne T :
Revised: 2: Deﬁne T :

Original: = θ val F dnn ( γ/ 4) · Pdim( F dnn ) εγ , M :=
Revised: = θ val F dnn ( γ/ 4) · Pdim( F dnn ) εγ , M :=

Original: d log 2 T e , and C δ :=
Revised: d log 2 T e , and C δ :=

Original: O (Pdim( F dnn ) · log( T/δ )) .
Revised: O (Pdim( F dnn ) · log( T/δ )) .

Original: 3: Deﬁne τ m := 2 m for m ≥ 1 , τ 0 := 0 , and β m := 

Revised: 3: Deﬁne τ m := 2 m for m ≥ 1 , τ 0 := 0 , and β m :=

Original: ( M − m + 1) C δ .
Revised: ( M − m + 1) C δ .

Original: 4: Deﬁne b
Revised: 4: Deﬁne b

Original: R m ( f ) :=
Revised: R m ( f ) :=

Original: P τ m − 1 t =1 Q t ( b f ( x t ) − y t ) 2 with the convention that
Revised: P τ m − 1 t =1 Q t ( b f ( x t ) − y t ) 2 with the convention that

Original: P 0 t =1 . . .
Revised: P 0 t =1 . . .

Original: = 0 .
Revised: = 0 .

Original: 5: for epoch m = 1 , 2 , . .
Revised: 5: for epoch m = 1 , 2 , . .

Original: , M do 6:
Revised: , M do 6:

Original: Get b f m := arg min f ∈F dnn
Revised: Get b f m := arg min f ∈F dnn

Original: P τ m − 1 t =1 Q t ( f ( x t ) − y t ) 2 .
Revised: P τ m − 1 t =1 Q t ( f ( x t ) − y t ) 2 .

Original: 7: (Implicitely) Construct active set F m := n f ∈ F dnn : b
Revised: 7: (Implicitely) Construct active set F m := n f ∈ F dnn : b

Original: R m ( f ) ≤ b
Revised: R m ( f ) ≤ b

Original: R m ( b f m ) + β m o .
Revised: R m ( b f m ) + β m o .

Original: Construct classiﬁer b h m : X → { +1 , − 1 , ⊥} as b h m ( x ) :
Revised: Construct classiﬁer b h m : X → { +1 , − 1 , ⊥} as b h m ( x ) :

Original: = ( ⊥ , if [ lcb ( x ; F m ) − γ 4 , ucb ( x ; F m )
Revised: = ( ⊥ , if [ lcb ( x ; F m ) − γ 4 , ucb ( x ; F m )

Original: + γ 4 ] ⊆
Revised: + γ 4 ] ⊆

Original: (cid:2) 12 − γ, 12 + γ (cid:3) ; sign( b f m ( x ) − 1) , o.w. and query function g m ( x ) :=(cid:0) 12 ∈ (cid:0) lcb ( x ; F m ) − γ 4 , ucb ( x ; F m ) + γ(cid:1)(cid:1) · 1 ( b h m ( x ) 6 = ⊥ ) .
Revised: (cid:2) 12 − γ, 12 + γ (cid:3) ; sign( b f m ( x ) − 1) , o.w. and query function g m ( x ) :=(cid:0) 12 ∈ (cid:0) lcb ( x ; F m ) − γ 4 , ucb ( x ; F m ) + γ(cid:1)(cid:1) · 1 ( b h m ( x ) 6 = ⊥ ) .

Original: 9: if epoch m =
Revised: 9: if epoch m =

Original: M then 10:
Revised: M then 10:

Original: Return classiﬁer b h M .
Revised: Return classiﬁer b h M .

Original: 11: for time t = τ m − 1 + 1 , . . .
Revised: 11: for time t = τ m − 1 + 1 , . . .

Original: , τ m do 12:
Revised: , τ m do 12:

Original: Observe x t ∼ D X .
Revised: Observe x t ∼ D X .

Original: Set Q t :=
Revised: Set Q t :=

Original: g m ( x t ) .
Revised: g m ( x t ) .

Original: 13: if Q t = 1 then 14:
Revised: 13: if Q t = 1 then 14:

Original: Query the label y t of x t .
Revised: Query the label y t of x t .

Original: We now present NeuralCAL ++ (Algorithm 2), a deep active learning algorithm that leverages the power of abstention.
Revised: We now present NeuralCAL ++ (Algorithm 2), a deep active learning algorithm that leverages the power of abstention.

Original: NeuralCAL ++ ﬁrst initialize a set of set of neural network regression functions F dnn by applying a preprocessing step on top of the set of regression functions obtained from Theorem 3 with a carefully chosen approximation level κ .
Revised: NeuralCAL ++ ﬁrst initialize a set of set of neural network regression functions F dnn by applying a preprocessing step on top of the set of regression functions obtained from Theorem 3 with a carefully chosen approximation level κ .

Original: The preprocessing step mainly contains two actions: (1) clipping f dnn : X → R into ˇ f dnn : X → [0 , 1] (since we obviously have η ( x ) ∈ [0 , 1] ); and (2) ﬁltering out f dnn ∈ F dnn that are clearly not a good approximation of η .
Revised: The preprocessing step mainly contains two actions: (1) clipping f dnn : X → R into ˇ f dnn : X → [0 , 1] (since we obviously have η ( x ) ∈ [0 , 1] ); and (2) ﬁltering out f dnn ∈ F dnn that are clearly not a good approximation of η .

Original: After initialization, NeuralCAL ++ runs in epochs of geometrically increasing lengths.
Revised: After initialization, NeuralCAL ++ runs in epochs of geometrically increasing lengths.

Original: At the beginning of epoch m ∈ [ M ] , NeuralCAL ++ (implicitly) constructs an active set of regression functions F m that are “close” to the true conditional probability η .
Revised: At the beginning of epoch m ∈ [ M ] , NeuralCAL ++ (implicitly) constructs an active set of regression functions F m that are “close” to the true conditional probability η .

Original: For any x ∼ D X , NeuralCAL ++ constructs a lower bound lcb ( x ; F m ) :
Revised: For any x ∼ D X , NeuralCAL ++ constructs a lower bound lcb ( x ; F m ) :

Original: =
Revised: =

Original: inf f ∈F m f ( x ) and an upper bound ucb ( x ; F m ) :
Revised: inf f ∈F m f ( x ) and an upper bound ucb ( x ; F m ) :

Original: = sup f ∈F m f ( x ) as a conﬁdence range of η ( x ) (based on F m ).
Revised: = sup f ∈F m f ( x ) as a conﬁdence range of η ( x ) (based on F m ).

Original: An empirical classiﬁer with an abstention option b h m : X → { +1 , − 1 , ⊥} and a query function g m : X → { 0 , 1 } are then constructed based on the conﬁdence range (and the abstention parameter γ ).
Revised: An empirical classiﬁer with an abstention option b h m : X → { +1 , − 1 , ⊥} and a query function g m : X → { 0 , 1 } are then constructed based on the conﬁdence range (and the abstention parameter γ ).

Original: For any time step t within epoch m , NeuralCAL ++ queries the label of the observed data point x t if and only if Q t :=
Revised: For any time step t within epoch m , NeuralCAL ++ queries the label of the observed data point x t if and only if Q t :=

Original: g m ( x t ) = 1 .
Revised: g m ( x t ) = 1 .

Original: NeuralCAL ++ returns b h M as the learned classiﬁer.
Revised: NeuralCAL ++ returns b h M as the learned classiﬁer.

Original: NeuralCAL ++ is adapted from the algorithm developed in Zhu and Nowak (2022), but with novel extensions.
Revised: NeuralCAL ++ is adapted from the algorithm developed in Zhu and Nowak (2022), but with novel extensions.

Original: In particular, the algorithm presented in Zhu and Nowak (2022) requires the existence of a f ∈ F
Revised: In particular, the algorithm presented in Zhu and Nowak (2022) requires the existence of a f ∈ F

Original: such that k f − η k ∞ ≤ ε (to achieve ε Chow’s excess error), Such an approximation requirement directly leads to poly(ε ) label complexity in the nonparametric setting , which is unacceptable.
Revised: such that k f − η k ∞ ≤ ε (to achieve ε Chow’s excess error), Such an approximation requirement directly leads to poly(ε ) label complexity in the nonparametric setting , which is unacceptable.

Original: The initialization step of NeuralCAL ++ (line 1) is carefully chosen to ensure that
Revised: The initialization step of NeuralCAL ++ (line 1) is carefully chosen to ensure that

Original: Pdim( F dnn ) , θ val F dnn ( γ/ 4) = e O (poly(1 /γ )) ; together with a sharper analysis of concentration results, these conditions help us derive the following deep active learning guarantees (also see Appendix D for a more general guarantee).
Revised: Pdim( F dnn ) , θ val F dnn ( γ 4 ) = poly( 1 γ ) · polylog( 1 ε ) ; together with a sharper analysis of concentration results, these conditions help us derive the following deep active learning guarantees (also see Appendix D for a more general guarantee).

Original: Theorem 7.
Revised: Theorem 7.

Original: Fix any ε, δ, γ > 0 .
Revised: Fix any ε, δ, γ > 0 .

Original: With probability at least 1 − δ , Algorithm 2 (with an appropriate initialization at line 1) returns a classiﬁer b h with Chow’s excess error e
Revised: With probability at least 1 − δ , Algorithm 2 (with an appropriate initialization at line 1) returns a classiﬁer b h with Chow’s excess error e

Original: O ( ε ) after querying poly( 1 γ ) · polylog( 1 εδ ) labels.
Revised: O ( ε ) after querying poly( 1 γ ) · polylog( 1 εδ ) labels.

Original: We discuss two important aspects of Algorithm 2/Theorem 7 in the following, i.e., exponential savings and computational efﬁciency.
Revised: We discuss two important aspects of Algorithm 2/Theorem 7 in the following, i.e., exponential savings and computational efﬁciency.

Original: We defer more detailed discussions to Appendix F.1.
Revised: We defer more detailed discussions to Appendix F.1.

Original: • Exponential speedups.
Revised: • Exponential speedups.

Original: Theorem 7 shows that, equipped with an abstention option, deep active learning enjoys polylog( 1 ε ) label complexity.
Revised: Theorem 7 shows that, equipped with an abstention option, deep active learning enjoys polylog( 1 ε ) label complexity.

Original: This provides theoretical justiﬁcations for great empirical results of deep active learning observed in practice.
Revised: This provides theoretical justiﬁcations for great empirical results of deep active learning observed in practice.

Original: Moreover, Algorithm 2 outputs a classiﬁer that abstains properly , i.e., it abstains only if abstention is the optimal choice; such a property further implies polylog( 1 ε ) label complexity under standard excess error and Massart noise (Massart and Nédélec, 2006).
Revised: Moreover, Algorithm 2 outputs a classiﬁer that abstains properly , i.e., it abstains only if abstention is the optimal choice; such a property further implies polylog( 1 ε ) label complexity under standard excess error and Massart noise (Massart and Nédélec, 2006).

Original: • Computational efﬁciency.
Revised: • Computational efﬁciency.

Original: Suppose one can efﬁciently implement a (weighted) square loss regression oracle over the initialized set of neural networks F dnn : Given any set S of weighted examples ( w, x
Revised: Suppose one can efﬁciently implement a (weighted) square loss regression oracle over the initialized set of neural networks F dnn : Given any set S of weighted examples ( w, x

Original: , y ) ∈ R + × X × Y as input, the regression oracle outputs
Revised: , y ) ∈ R + × X × Y as input, the regression oracle outputs

Original: [Equation]
Revised: [Equation]

Original: Algorithm 2 can then be efﬁciently implemented with poly( 1 γ ) · 1 ε oracle calls.While the label complexity
Revised: Algorithm 2 can then be efﬁciently implemented with poly( 1 γ ) · 1 ε oracle calls.

Original: obtained

Original: in

Original:  Theorem  has desired dependence on polylog 1 ε ) , its dependence on γ 
Revised: While the label complexity obtained in Theorem 7 has desired dependence on polylog( 1 ε ) , its dependence on γ can be of order

Original: can

Original: be of order γ − poly( d ) .
Revised: γ − poly( d ) .

Original: Our next result shows

Original: that, however,

Original:  such dependence is unavoidable even in the case of learning a single ReLU function
Revised: Our next result shows that, however, such dependence is unavoidable even in the case of learning a single ReLU function.

Original: Theorem 8.
Revised: Theorem 8.

Original: Fix any γ ∈ (0 , 1 / 8) .
Revised: Fix any γ ∈ (0 , 1 / 8) .

Original: For any accuracy level ε sufﬁciently small, there exists a problem instance such that (1) η ∈ W 1 , ∞( X ) and is of the form η ( x ) :
Revised: For any accuracy level ε sufﬁciently small, there exists a problem instance such that (1) η ∈ W 1 , ∞( X ) and is of the form η ( x ) :

Original: = ReLU ( h w, x i + a ) + b ; and (2) for any active learning algorithm, it takes at least γ − Ω( d ) labels to identify an ε -optimal classiﬁer, for either standard excess error or Chow’s excess error (with parameter γ ).
Revised: = ReLU ( h w, x i + a ) + b ; and (2) for any active learning algorithm, it takes at least γ − Ω( d ) labels to identify an ε -optimal classiﬁer, for either standard excess error or Chow’s excess error (with parameter γ ).

Original:  Extensions
Revised:  Extensions

Original: Previous results are developed in the commonly studied Sobolev/Hölder spaces.
Revised: Previous results are developed in the commonly studied Sobolev/Hölder spaces.

Original: Our techniques, however, are generic and can be adapted to other function spaces, given neural network approximation results.
Revised: Our techniques, however, are generic and can be adapted to other function spaces, given neural network approximation results.

Original: In this section, we provide extensions of our results to the Radon BV 2 space, which was recently proposed as the natural function space associated with ReLU neural networks (Ongie et al., 2020; Parhi and Nowak, 2021, 2022a,b; Unser, 2022).
Revised: In this section, we provide extensions of our results to the Radon BV 2 space, which was recently proposed as the natural function space associated with ReLU neural networks (Ongie et al., 2020; Parhi and Nowak, 2021, 2022a,b; Unser, 2022).

Original: The Radon BV 2 space.
Revised: The Radon BV 2 space.

Original: The Radon BV 2 unit ball over domain X is deﬁned as R BV 21 ( X ) :
Revised: The Radon BV 2 unit ball over domain X is deﬁned as R BV 21 ( X ) :

Original: = { f : k f k R BV 2 ( X ) ≤ 1 } , where k f k R BV 2 ( X ) denotes the Radon BV 2 norm of f over domain X .
Revised: = { f : k f k R BV 2 ( X ) ≤ 1 } , where k f k R BV 2 ( X ) denotes the Radon BV 2 norm of f over domain X .

Original: Following Parhi and Nowak (2022b), we assume X = { x ∈ R d : k x k 2 ≤ 1 } and η ∈ R BV 2 1 ( X ) .
Revised: Following Parhi and Nowak (2022b), we assume X = { x ∈ R d : k x k 2 ≤ 1 } and η ∈ R BV 2 1 ( X ) .

Original: The Radon BV 2 space naturally contains neural networks of the form f dnn ( x ) =
Revised: The Radon BV 2 space naturally contains neural networks of the form f dnn ( x ) =

Original: P
Revised: P

Original: Kk =1 v i · ReLU ( w > i x + b i ) .
Revised: Kk =1 v i · ReLU ( w > i x + b i ) .

Original: On the contrary, such f dnn doesn’t lie in any Sobolev space of order α ≥(since f dnn doesn’t have second order weak derivative).
Revised: On the contrary, such f dnn doesn’t lie in any Sobolev space of order α ≥(since f dnn doesn’t have second order weak derivative).

Original: Thus, if η takes the form of the aforementioned neural network (e.g., η = f dnn ), approximating η up to κ from a Sobolev perspective requires e
Revised: Thus, if η takes the form of the aforementioned neural network (e.g., η = f dnn ), approximating η up to κ from a Sobolev perspective requires e

Original: O ( κ − d ) total parameters, which suffers from the curse of dimensionality.
Revised: O ( κ − d ) total parameters, which suffers from the curse of dimensionality.

Original: On the other side, however, such bad dependence on dimensionality goes away when approximating from a Radon BVperspective, as shown in the following theorem.
Revised: On the other side, however, such bad dependence on dimensionality goes away when approximating from a Radon BVperspective, as shown in the following theorem.

Original: Theorem 9 (Parhi and Nowak (2022b)) .
Revised: Theorem 9 (Parhi and Nowak (2022b)) .

Original: Fix any κ > 0 .
Revised: Fix any κ > 0 .

Original: For any f ? ∈ R BV 21 ( X ) , there exists a one-hidden layer neural network f dnn of width K
Revised: For any f ? ∈ R BV 21 ( X ) , there exists a one-hidden layer neural network f dnn of width K

Original: = O ( κ − 2 d d +3 ) such that k f ?
Revised: = O ( κ − 2 d d +3 ) such that k f ?

Original: − f dnn k ∞ ≤ κ .
Revised: − f dnn k ∞ ≤ κ .

Original: Equipped with this approximation result, we provide the active learning guarantees for learning a smooth function within the Radon BV 2 unit ball as follows.
Revised: Equipped with this approximation result, we provide the active learning guarantees for learning a smooth function within the Radon BV 2 unit ball as follows.

Original: Theorem 10.
Revised: Theorem 10.

Original: Suppose η ∈ R BV 21 ( X ) and the Tsybakov noise condition is satisﬁed with parameter β ≥ 0 .
Revised: Suppose η ∈ R BV 21 ( X ) and the Tsybakov noise condition is satisﬁed with parameter β ≥ 0 .

Original: Fix any ε, δ > 0 .
Revised: Fix any ε, δ > 0 .

Original: There exists an algorithm such that, with probability at least 1 − δ , it learns a classiﬁer b
Revised: There exists an algorithm such that, with probability at least 1 − δ , it learns a classiﬁer b

Original: [Equation]
Revised: [Equation]

Original: ∈
Revised: ∈

Original: [Equation]
Revised: [Equation]

Original: dnn with excess error e
Revised: dnn with excess error e

Original: O ( ε ) after querying e
Revised: O ( ε ) after querying e

Original: [Equation]
Revised: [Equation]

Original: labels.
Revised: labels.

Original: Compared to the label complexity obtained in Theorem 6, the label complexity obtained in the above theorem doesn’t suffer from the curse of dimensionality: For d large enough, the above label complexity scales as ε − O (1) yet label complexity in Theorem 6 scales as ε − O
Revised: Compared to the label complexity obtained in Theorem 6, the label complexity obtained in the above theorem doesn’t suffer from the curse of dimensionality: For d large enough, the above label complexity scales as ε − O (1) yet label complexity in Theorem 6 scales as ε − O

Original: ( d ) .
Revised: ( d ) .

Original: Active learning guarantees under Chow’s excess error in the Radon BV 2 space are similar to results presented in
Revised: Active learning guarantees under Chow’s excess error in the Radon BV 2 space are similar to results presented in

Original: Theorem 7, and are thus deferred to Appendix G.
Revised: Theorem 7, and are thus deferred to Appendix G.

Original:  Discussion
Revised:  Discussion

Original: We provide the ﬁrst near-optimal deep active learning guarantees, under both standard excess error and Chow’s excess error.
Revised: We provide the ﬁrst near-optimal deep active learning guarantees, under both standard excess error and Chow’s excess error.

Original: Our results are powered by generic algorithms and analyses developed for active learning that bridge approximation guarantees into label complexity guarantees.
Revised: Our results are powered by generic algorithms and analyses developed for active learning that bridge approximation guarantees into label complexity guarantees.

Original: We outline some natural directions for future research below.
Revised: We outline some natural directions for future research below.

Original: • Disagreement coefﬁcients for neural networks.
Revised: • Disagreement coefﬁcients for neural networks.

Original: While we have provided some results regarding the disagreement coefﬁcients for neural networks, we believe a comprehensive investigation on this topic is needed.
Revised: While we have provided some results regarding the disagreement coefﬁcients for neural networks, we believe a comprehensive investigation on this topic is needed.

Original: For instance, can we discover more general settings where the classiﬁer-based disagreement coefﬁcient can be upper bounded by O (1) ?
Revised: For instance, can we discover more general settings where the classiﬁer-based disagreement coefﬁcient can be upper bounded by O (1) ?

Original: It is also interesting to explore sharper analyses on the value function disagreement coefﬁcient.
Revised: It is also interesting to explore sharper analyses on the value function disagreement coefﬁcient.

Original: • Adaptivity in deep active learning.
Revised: • Adaptivity in deep active learning.

Original: Our current results are established with the knowledge of some problem-dependent parameters, e.g., the smoothness parameters regarding the function spaces and the noise levels.
Revised: Our current results are established with the knowledge of some problem-dependent parameters, e.g., the smoothness parameters regarding the function spaces and the noise levels.

Original: It will be interesting to see if one can develop algorithms that can automatically adapt to unknown parameters, e.g., by leveraging techniques developed in Locatelli et al.
Revised: It will be interesting to see if one can develop algorithms that can automatically adapt to unknown parameters, e.g., by leveraging techniques developed in Locatelli et al.

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: The authors would like to thank Rahul Parhi for many helpful discussions regarding his papers.
Revised: The authors would like to thank Rahul Parhi for many helpful discussions regarding his papers.

Original: We also would like to thank anonymousNeurIPS 2022 reviewers for their constructive comments.
Revised: We also would like to thank anonymous reviewers for their constructive comments.

Original: This work is partially supported by NSF grant 1934612 and AFOSR grant FA9550-18-1-
Revised: This work is partially supported by NSF grant 1934612 and AFOSR grant FA9550-18-1-0166.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] See Section 5 for discussions on limitations and directions for future work.
Revised: [Yes] See Section 5 for discussions on limitations and directions for future work.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: Our paper is theoretical in nature, and there is no negative societal impact of our work in the foreseeable future.
Revised: Our paper is theoretical in nature, and there is no negative societal impact of our work in the foreseeable future.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] Assumptions are clearly stated in the statement of each theorem.
Revised: [Yes] Assumptions are clearly stated in the statement of each theorem.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] Complete proofs are provided in the Appendix.
Revised: [Yes] Complete proofs are provided in the Appendix.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: [N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [N/A] 4.
Revised: [N/A] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [N/A] (b) Did you mention the license of the assets?
Revised: [N/A] (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether
Revised: [N/A] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [N/A] 5.
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]