Original: Improved Algorithms for Neural Active Learning
Revised: Improved Algorithms for Neural Active Learning

Original: Abstract
Revised: Abstract

Original: We improve the theoretical and empirical performance of neural-network(NN)- based active learning algorithms for the non-parametric streaming setting.
Revised: We improve the theoretical and empirical performance of neural-network(NN)- based active learning algorithms for the non-parametric streaming setting.

Original: In particular, we introduce two regret metrics by minimizing the population loss that are more suitable in active learning than the one used in state-of-the-art (SOTA) related work.
Revised: In particular, we introduce two regret metrics by minimizing the population loss that are more suitable in active learning than the one used in state-of-the-art (SOTA) related work.

Original: Then, the proposed algorithm leverages the powerful representation of NNs for both exploitation and exploration, has the query decision-maker tailored for k -class classification problems with the performance guarantee, utilizes the full feedback, and updates parameters in a more practical and efficient manner.
Revised: Then, the proposed algorithm leverages the powerful representation of NNs for both exploitation and exploration, has the query decision-maker tailored for k -class classification problems with the performance guarantee, utilizes the full feedback, and updates parameters in a more practical and efficient manner.

Original: These careful designs lead to a better regret upper bound, improving by a multiplicative factor O (log T ) and removing the curse of input dimensionality.
Revised: These careful designs lead to a better regret upper bound, improving by a multiplicative factor O (log T ) and removing the curse of input dimensionality.

Original: Furthermore, we show that the algorithm can achieve the same performance as the Bayes-optimal classifier in the long run under the hard-margin setting in classification problems.
Revised: Furthermore, we show that the algorithm can achieve the same performance as the Bayes-optimal classifier in the long run under the hard-margin setting in classification problems.

Original: In the end, we use extensive experiments to evaluate the proposed algorithm and
Revised: In the end, we use extensive experiments to evaluate the proposed algorithm and

Original: SOTA baselines, to show the improved empirical performance.
Revised: SOTA baselines, to show the improved empirical performance.

Original:  Introduction
Revised:  Introduction

Original: The Neural Network (NN) is one of the indispensable paradigms in machine learning and is widely used in multifarious supervised-learning tasks [25].
Revised: The Neural Network (NN) is one of the indispensable paradigms in machine learning and is widely used in multifarious supervised-learning tasks [25].

Original: As more and more complicated NNs are developed, the requirement of the training procedure on the labeled data grows, incurring significant cost of label annotation.
Revised: As more and more complicated NNs are developed, the requirement of the training procedure on the labeled data grows, incurring significant cost of label annotation.

Original: Active learning investigates effective techniques on a much smaller labeled data set while attaining the comparable generalization performance to passive learning [20].
Revised: Active learning investigates effective techniques on a much smaller labeled data set while attaining the comparable generalization performance to passive learning [20].

Original: In this paper, we focus on the classification problem in the streaming setting of active learning with NN models.
Revised: In this paper, we focus on the classification problem in the streaming setting of active learning with NN models.

Original: At every round, the learner receives an instance and is compelled to decide on-the-fly whether or not to observe the label associated with this instance.
Revised: At every round, the learner receives an instance and is compelled to decide on-the-fly whether or not to observe the label associated with this instance.

Original: This problem seeks to maximize the generalization capability of learned NNs in a sequence of rounds, such that the model has robust performance on the unseen data from the same distribution [42]
Revised: This problem seeks to maximize the generalization capability of learned NNs in a sequence of rounds, such that the model has robust performance on the unseen data from the same distribution [42]

Original: In active learning, given access to the i.i.d.
Revised: In active learning, given access to the i.i.d.

Original: generated instances from a distribution D , suppose there exist a class of functions F that formulate the mapping from instances to theirs labels.
Revised: generated instances from a distribution D , suppose there exist a class of functions F that formulate the mapping from instances to theirs labels.

Original: In the parametric setting, i.e., F has finite VC-dimension [27], existing works [26, 15, 8] have shown that the active learning algorithms can achieve the convergence rate of (cid:101)
Revised: In the parametric setting, i.e., F has finite VC-dimension [27], existing works [26, 15, 8] have shown that the active learning algorithms can achieve the convergence rate of (cid:101)

Original: O
Revised: O

Original: (1 / √ N ) to the best population loss in F , where N is the number of label queries.
Revised: (1 / √ N ) to the best population loss in F , where N is the number of label queries.

Original: In the non-parametric setting, recent works [36, 37] provide the similar convergence results while suffering from the curse of input dimensionality.
Revised: In the non-parametric setting, recent works [36, 37] provide the similar convergence results while suffering from the curse of input dimensionality.

Original: Unfortunately, most of NN-based approaches to active learning do not come with the performance guarantee, despite having powerful empirical results.
Revised: Unfortunately, most of NN-based approaches to active learning do not come with the performance guarantee, despite having powerful empirical results.

Original: The first performance guarantee for neural active learning has been established in a recent work by [50], and the analysis is for over-parameterized neural networks with the assistance of Neural
Revised: The first performance guarantee for neural active learning has been established in a recent work by [50], and the analysis is for over-parameterized neural networks with the assistance of Neural

Original: Tangent Kernel (NTK).
Revised: Tangent Kernel (NTK).

Original: We carefully investigate the limitations of [50], which turn into the main motivations of our paper.
Revised: We carefully investigate the limitations of [50], which turn into the main motivations of our paper.

Original: First, [50] transforms the classification problem into a multi-armed bandit problem [57], to minimize a pseudo regret metric.
Revised: First, [50] transforms the classification problem into a multi-armed bandit problem [57], to minimize a pseudo regret metric.

Original: Yet, on the grounds that they seek to minimize the conditional population loss on a sequence of given data, it is dubious that the pseudo regret used in [50] can explicitly measure the generalization capability of given algorithms (see Remark 2.1).
Revised: Yet, on the grounds that they seek to minimize the conditional population loss on a sequence of given data, it is dubious that the pseudo regret used in [50] can explicitly measure the generalization capability of given algorithms (see Remark 2.1).

Original: Second, the proposed algorithm in [50] has an unknown prior complexity term S that reflects the complexity of function to be learned, brought by the NTK approximation (Lemma 1 in [50]).
Revised: Second, the proposed algorithm in [50] has an unknown prior complexity term S that reflects the complexity of function to be learned, brought by the NTK approximation (Lemma 1 in [50]).

Original: To estimate the magnitude of S , the authors introduce another complex model selection algorithm and make strong assumptions regarding the generated models.
Revised: To estimate the magnitude of S , the authors introduce another complex model selection algorithm and make strong assumptions regarding the generated models.

Original: Third, the training process for NN models is not efficient, as [50] uses vanilla gradient descent and starts from randomly initialized parameters in every round.
Revised: Third, the training process for NN models is not efficient, as [50] uses vanilla gradient descent and starts from randomly initialized parameters in every round.

Original: Fourth, although [50] removes the curse of input dimensionality d , the performance guarantee strongly suffers from another introduced term, the effective dimensionality (cid:101) d , which can be thought of as the non-linear dimensions of Hilbert space spanned by NTK.
Revised: Fourth, although [50] removes the curse of input dimensionality d , the performance guarantee strongly suffers from another introduced term, the effective dimensionality (cid:101) d , which can be thought of as the non-linear dimensionalities of Hilbert space spanned by NTK.

Original: In the worse case, the magnitude of (cid:101) d can be an unacceptably large number and thus the performance guarantee collapses.
Revised: In the worse case, the magnitude of (cid:101) d can be an unacceptably large number and thus the performance guarantee collapses.

Original: 1.1 Main contributions
Revised: 1.1 Main contributions

Original: In this paper, we propose a novel algorithm, I-NeurAL
Revised: In this paper, we propose a novel algorithm, I-NeurAL

Original: ( I mproved Algorithms for Neur al A ctive
Revised: ( I mproved Algorithms for Neur al A ctive

Original: L earning), to tackle the above limitations.
Revised: L earning), to tackle the above limitations.

Original: Our contributions can be summarized as follows: (1) We consider the k -class classification problem, and we introduce two new regret metrics to minimize the population loss, which can directly reflect the generalization capability of NN-based algorithms.
Revised: Our contributions can be summarized as follows: (1) We consider the k -class classification problem, and we introduce two new regret metrics to minimize the population loss, which can directly reflect the generalization capability of NN-based algorithms.

Original: I-NeurAL has a neural exploration strategy with a novel component to decide whether or not to query the label, coming with the performance guarantee.
Revised: I-NeurAL has a neural exploration strategy with a novel component to decide whether or not to query the label, coming with the performance guarantee.

Original: I-NeurAL does not have unknown complexity terms and the hyper-parameters are straightforward and intuitive.
Revised: I-NeurAL does not have unknown complexity terms and the hyper-parameters are straightforward and intuitive.

Original: (3) I-NeurAL is designed to support mini-batch Stochastic Gradient Descent (SGD).
Revised: (3) I-NeurAL is designed to support mini-batch Stochastic Gradient Descent (SGD).

Original: In particular, at every round, I-NeurAL does minibatch SGD starting with the parameters of the last round, i.e., with warm start, which is more efficient and practical compared to [50].
Revised: In particular, at every round, I-NeurAL does minibatch SGD starting with the parameters of the last round, i.e., with warm start, which is more efficient and practical compared to [50].

Original: (4) Without any noise assumption on the data distribution, we provide the performance guarantee of I-NeurAL for over-parameterized neural networks.
Revised: (4) Without any noise assumption on the data distribution, we provide the performance guarantee of I-NeurAL for over-parameterized neural networks.

Original: Compared to [50], we remove the curse of both the input dimensionality d and the effective dimensionality (cid:101) d ; Moreover, we improve the regret by a multiplicative factor log( T ) , where T is the number of rounds.
Revised: Compared to [50], we remove the curse of both the input dimensionality d and the effective dimensionality (cid:101) d ; Moreover, we improve the regret by a multiplicative factor log( T ) , where T is the number of rounds.

Original: (5) under a hard-margin assumption on the data distribution, to the best of our knowledge, we provide the first performance analysis that NN models can achieve the same generalization capability as Bayes-optimal classifier after a fixed number of label queries; (6) we conduct extensive experiments on real-world data sets to demonstrate the improved performance of I-NeurAL over state-of-the-art baselines including the closest work [50] which has not provided empirical validation of their proposed algorithms.
Revised: (5) under a hard-margin assumption on the data distribution, to the best of our knowledge, we provide the first performance analysis that NN models can achieve the same generalization capability as Bayes-optimal classifier after a fixed number of label queries; (6) we conduct extensive experiments on real-world data sets to demonstrate the improved performance of I-NeurAL over state-of-the-art baselines including the closest work [50] which has not provided empirical validation of their proposed algorithms.

Original: 1.2 Related Work
Revised: 1.2 Related Work

Original: Active learning has been extensively studied and applied to many essential applications [46].
Revised: Active learning has been extensively studied and applied to many essential applications [46].

Original: Bayesian active learning methods typically use a probabilistic regression model to estimate the improvement of each query [31, 43].
Revised: Bayesian active learning methods typically use a probabilistic regression model to estimate the improvement of each query [31, 43].

Original: In spite of effectiveness on the small or moderate data sets, the Bayesian-based approaches are difficult to scale to large-scale data sets because of the batch sampling [45].
Revised: In spite of effectiveness on the small or moderate data sets, the Bayesian-based approaches are difficult to scale to large-scale data sets because of the batch sampling [45].

Original: Another important class, margin algorithms or uncertainty sampling [35], obtains considerate performance improvement over passive learning and is further developed by many practitioners [21, 30, 39, 16].
Revised: Another important class, margin algorithms or uncertainty sampling [35], obtains considerate performance improvement over passive learning and is further developed by many practitioners [21, 30, 39, 16].

Original: Margin algorithms are flexible and can be adapted to both streaming and pool settings.
Revised: Margin algorithms are flexible and can be adapted to both streaming and pool settings.

Original: In the pool setting, a line of works utilize the neural networks in active learning to improve the empirical performance [38, 44, 6, 18, 32, 47, 49, 56, 5].
Revised: In the pool setting, a line of works utilize the neural networks in active learning to improve the empirical performance [38, 44, 6, 18, 32, 47, 49, 56, 5].

Original: However, they do not provide performance guarantee for NN-based active learning algorithms.
Revised: However, they do not provide performance guarantee for NN-based active learning algorithms.

Original: From the theoretical perspective, [53, 22, 7, 9, 54] provide the performance guarantee with the specific classes of functions and [28, 23] present the theoretical analysis of active learning algorithms with the surrogate loss functions for binary classification.
Revised: From the theoretical perspective, [53, 22, 7, 9, 54] provide the performance guarantee with the specific classes of functions and [28, 23] present the theoretical analysis of active learning algorithms with the surrogate loss functions for binary classification.

Original: However, their performance guarantee is restricted within hypothesis classes, i.e, the parametric setting.
Revised: However, their performance guarantee is restricted within hypothesis classes, i.e, the parametric setting.

Original: In contrast, our goal is to derive an NN-based algorithm in the non-parametric setting that performs well both empirically and theoretically.
Revised: In contrast, our goal is to derive an NN-based algorithm in the non-parametric setting that performs well both empirically and theoretically.

Original: Neural contextual bandits[57, 55, 14, 12, 13, 41] provide the principled method to balance between the exploitation and exploration [10, 11].
Revised: Neural contextual bandits[57, 55, 14, 12, 13, 41] provide the principled method to balance between the exploitation and exploration [10, 11].

Original: [50] transforms active learning into neural contextual bandit problem and obtains a performance guarantee, of which limitations are discussed above.
Revised: [50] transforms active learning into neural contextual bandit problem and obtains a performance guarantee, of which limitations are discussed above.

Original: As [50] is the closest related work to our paper, we emphasize the differences of our techniques from [50] throughout the paper.
Revised: As [50] is the closest related work to our paper, we emphasize the differences of our techniques from [50] throughout the paper.

Original: We introduce the problem definition and proposed algorithms in Section
Revised: We introduce the problem definition and proposed algorithms in Section

Original: 2 and Section 3 respectively.
Revised: 2 and Section 3 respectively.

Original: Then, we provide performance guarantees in Section 4 and empirical results in Section 5, ending with the conclusion in Section 6.
Revised: Then, we provide performance guarantees in Section 4 and empirical results in Section 5, ending with the conclusion in Section 6.

Original:  Problem Definition
Revised:  Problem Definition

Original: In this paper, we study the streaming setting of active learning in the k -class classification problem.
Revised: In this paper, we study the streaming setting of active learning in the k -class classification problem.

Original: Let X denote the input space over R d , Y = { 1 , 2 , . .
Revised: Let X denote the input space over R d , Y = { 1 , 2 , . .

Original: . , k } represent the label space, and D be some unknown distribution over X ×Y .
Revised: . , k } represent the label space, and D be some unknown distribution over X ×Y .

Original: At round t ∈ [ T ] = { 1 , 2 , . . .
Revised: At round t ∈ [ T ] = { 1 , 2 , . . .

Original: , T } , an instance x t is drawn from the marginal distribution D X and accordingly y t is drawn from the conditional distribution D Y| x t .
Revised: , T } , an instance x t is drawn from the marginal distribution D X and accordingly y t is drawn from the conditional distribution D Y| x t .

Original: Here, y t can be thought of as the index of the class that x t belongs to.
Revised: Here, y t can be thought of as the index of the class that x t belongs to.

Original: Inspired by [50], we first transform x t into k context vectors representing the k classes respectively: x t, 1 = ( x ⊤ t , 0 ⊤ , . . .
Revised: Inspired by [50], we first transform x t into k context vectors representing the k classes respectively: x t, 1 = ( x ⊤ t , 0 ⊤ , . . .

Original: , 0 ⊤ )
Revised: , 0 ⊤ )

Original: ⊤ , x t, 2 = ( 0 ⊤ , x ⊤ t , . . .
Revised: ⊤ , x t, 2 = ( 0 ⊤ , x ⊤ t , . . .

Original: , 0 ⊤ ) ⊤ , . . .
Revised: , 0 ⊤ ) ⊤ , . . .

Original: , x t,k
Revised: , x t,k

Original: = ( 0 ⊤ , 0 ⊤ , . . .
Revised: = ( 0 ⊤ , 0 ⊤ , . . .

Original: , x ⊤ t ) ⊤ and x t,
Revised: , x ⊤ t ) ⊤ and x t,

Original: i ∈
Revised: i ∈

Original: R dk , ∀ i ∈ [ k ] .
Revised: R dk , ∀ i ∈ [ k ] .

Original: In accordance with context vectors, we construct the k label vectors representing the k possible prediction: y t, 1 = (1 , 0 , . . . , 0) ⊤ , y t, 2 = (0 , 1 , . . . , 0) ⊤ , . . .
Revised: In accordance with context vectors, we construct the k label vectors representing the k possible prediction: y t, 1 = (1 , 0 , . . . , 0) ⊤ , y t, 2 = (0 , 1 , . . . , 0) ⊤ , . . .

Original: , y t,k
Revised: , y t,k

Original: = (0 , 0 , . .
Revised: = (0 , 0 , . .

Original: . , 1) ⊤ and y t,i ∈
Revised: . , 1) ⊤ and y t,i ∈

Original: R k , ∀ i ∈ [ k ] .
Revised: R k , ∀ i ∈ [ k ] .

Original: Thus, y t,y t is the ground-truth label vector for x t .
Revised: Thus, y t,y t is the ground-truth label vector for x t .

Original: Under the non-parametric setting of active learning, we define an unknown function h to formulate the conditional distribution D Y| x t : X k → [0 , 1] , such that
Revised: Under the non-parametric setting of active learning, we define an unknown function h to formulate the conditional distribution D Y| x t : X k → [0 , 1] , such that

Original: [Equation]
Revised: [Equation]

Original: which is subject to (cid:80) ki
Revised: which is subject to (cid:80) ki

Original: =1 h ( x t,i ) = 1 .
Revised: =1 h ( x t,i ) = 1 .

Original: For simplicity, we consider the k -class classification problem with 0-1 loss.
Revised: For simplicity, we consider the k -class classification problem with 0-1 loss.

Original: Given x t , i.e., x t,i , i ∈ [ k ] , let (cid:98) i be the index of the class predicted by some hypothesis f and
Revised: Given x t , i.e., x t,i , i ∈ [ k ] , let (cid:98) i be the index of the class predicted by some hypothesis f and

Original: thus y t, (cid:98) i is the prediction.
Revised: thus y t, (cid:98) i is the prediction.

Original: Then, we have the following loss:
Revised: Then, we have the following loss:

Original: [Equation]
Revised: [Equation]

Original: where 1 is the indicator function.
Revised: where 1 is the indicator function.

Original: Given the number of rounds T , at each round t ∈ [ T ] , the learner receives an instance x t drawn i.i.d.
Revised: Given the number of rounds T , at each round t ∈ [ T ] , the learner receives an instance x t drawn i.i.d.

Original: from D X .
Revised: from D X .

Original: Then, the learner needs to make a prediction y t, (cid:98) i , and at the same time, decide on-the-fly whether or not to query the label y t,y t where y t is drawn i.i.d.
Revised: Then, the learner needs to make a prediction y t, (cid:98) i , and at the same time, decide on-the-fly whether or not to query the label y t,y t where y t is drawn i.i.d.

Original: from D Y| x t .
Revised: from D Y| x t .

Original: As the goal of active learning tasks is often to minimize the population loss [42], we introduce the following two regret metrics.
Revised: As the goal of active learning tasks is often to minimize the population loss [42], we introduce the following two regret metrics.

Original: Definition 2.1 (Latest Population Regret) .
Revised: Definition 2.1 (Latest Population Regret) .

Original: Given the data distribution D , the number of rounds T , the Latest Population Regret is defined as
Revised: Given the data distribution D , the number of rounds T , the Latest Population Regret is defined as

Original: [Equation]
Revised: [Equation]

Original: where y T,i ∗ is the prediction the Bayes-optimal classifier would make on instance x T , i.e., i ∗ = arg max i ∈
Revised: where y T,i ∗ is the prediction the Bayes-optimal classifier would make on instance x T , i.e., i ∗ = arg max i ∈

Original: [ k ] h ( x T,i ) for y T,i ∗ .
Revised: [ k ] h ( x T,i ) for y T,i ∗ .

Original: Definition 2.2 (Cumulative Population Regret) .
Revised: Definition 2.2 (Cumulative Population Regret) .

Original: Given the data distribution D , the number of rounds
Revised: Given the data distribution D , the number of rounds

Original: T , the Cumulative Population Regret is defined as:
Revised: T , the Cumulative Population Regret is defined as:

Original: [Equation]
Revised: [Equation]

Original: where y t,i ∗ is the prediction the Bayes-optimal classifier would make on instance x t , i.e., i ∗ = arg max i ∈
Revised: where y t,i ∗ is the prediction the Bayes-optimal classifier would make on instance x t , i.e., i ∗ = arg max i ∈

Original: [ k ] h ( x t,i ) for y t,i ∗ .
Revised: [ k ] h ( x t,i ) for y t,i ∗ .

Original: R T measures the performance at the last round T only, and R T measures the overall performance in
Revised: R T measures the performance at the last round T only, and R T measures the overall performance in

Original: T rounds combined.
Revised: T rounds combined.

Original: Therefore, the goal of this problem is to minimize R T or R T , or both.
Revised: Therefore, the goal of this problem is to minimize R T or R T , or both.

Original: At the same time, we also aim to minimize the following expected query cost:
Revised: At the same time, we also aim to minimize the following expected query cost:

Original: [Equation]
Revised: [Equation]

Original: where I t is the indicator of the query decision in round t such that I t
Revised: where I t is the indicator of the query decision in round t such that I t

Original: = 1 if y t is observed; I t = 0 , otherwise.
Revised: = 1 if y t is observed; I t = 0 , otherwise.

Original: Remark 2.1.
Revised: Remark 2.1.

Original: Minimizing R T or R T shows the generalization capability of the learned hypothesis on the distribution D .
Revised: Minimizing R T or R T shows the generalization capability of the learned hypothesis on the distribution D .

Original: However, the problem defined in [50] is to minimize the cumulative conditional population regret as follows:
Revised: However, the problem defined in [50] is to minimize the cumulative conditional population regret as follows:

Original: [Equation]
Revised: [Equation]

Original: As E y t ∼D Y| x t [ L ( y t, (cid:98) i , y t,y t ) | x t ] is the population loss conditioned on x t , unfortunately, (cid:101) R T only measures the performance of the learned hypothesis on the collected data { x t } Tt =1 , and (cid:101) R T cannot directly measure the accuracy of the hypothesis on unseen data instances.
Revised: As E y t ∼D Y| x t [ L ( y t, (cid:98) i , y t,y t ) | x t ] is the population loss conditioned on x t , unfortunately, (cid:101) R T only measures the performance of the learned hypothesis on the collected data { x t } Tt =1 , and (cid:101) R T cannot directly measure the accuracy of the hypothesis on unseen data instances.

Original: Although (cid:101) R T follows the regret definition in multi-armed bandits [57], it is fair to say that (cid:101) R T may not be a good metric in active learning.
Revised: Although (cid:101) R T follows the regret definition in multi-armed bandits [57], it is fair to say that (cid:101) R T may not be a good metric in active learning.

Original:  Proposed Algorithms
Revised:  Proposed Algorithms

Original: In this section, we elaborate on the proposed algorithm I-NeurAL (Algorithm 1).
Revised: In this section, we elaborate on the proposed algorithm I-NeurAL (Algorithm 1).

Original: In contrast to the directly comparable work [50], I-NeurAL has the following novel and advantageous aspects: (1)
Revised: In contrast to the directly comparable work [50], I-NeurAL has the following novel and advantageous aspects: (1)

Original: I-NeurAL incorporates a neural-based exploration strategy (Line 6) inspired by recent advances in bandits [14] to solve the exploitation-exploration dilemma in the decision for whether or not to query labels; (2) I-NeurAL includes a novel component (Line 11) to decide whether or not to query labels in the k -class classification problem, which does not have the unknown complexity term S as in [50], thus avoiding the complicated model selection procedure (Algorithm 2 in [50]); (3) I-NeurAL infers and exploits the feedback of all the contexts (Lines 12-17), instead of only utilizing the feedback of the chosen context in [50]; (4) I-NeurAL conducts mini-batch SGD based on the parameters of the last round (Algorithm 2), which is more practical, as opposed to conducting vanilla gradient descent from the initialization at every round in [50].
Revised: I-NeurAL incorporates a neural-based exploration strategy (Line 6) inspired by recent advances in bandits [14] to solve the exploitation-exploration dilemma in the decision for whether or not to query labels; (2) I-NeurAL includes a novel component (Line 11) to decide whether or not to query labels in the k -class classification problem; (3) I-NeurAL infers and exploits the feedback of all the contexts (Lines 12-17), instead of only utilizing the feedback of the chosen context in [50]; (4) I-NeurAL conducts mini-batch SGD based on the parameters of the last round (Algorithm 2), which is more practical, as opposed to conducting vanilla gradient descent from the initialization at every round in [50].

Original: Next, we will present the details of I-NeurAL.
Revised: Next, we will present the details of I-NeurAL.

Original: Exploitation Network f 1 .
Revised: Exploitation Network f 1 .

Original: Given x t,i , i ∈ [ k ] , to learn the unknown function h (Eq. (2.1)), we use a fully-connected neural network f 1 with L -depth and m -width:
Revised: Given x t,i , i ∈ [ k ] , to learn the unknown function h (Eq. (2.1)), we use a fully-connected neural network f 1 with L -depth and m -width:

Original: [Equation]
Revised: [Equation]

Original: where W 11 ∈ R m × kd , W 1 l ∈ R m × m , for 2 ≤ l ≤ L − 1 , W 1 L ∈ R 1 × m , θ 1 =
Revised: where W 11 ∈ R m × kd , W 1 l ∈ R m × m , for 2 ≤ l ≤ L − 1 , W 1 L ∈ R 1 × m , θ 1 =

Original: [ vec ( W 11 ) ⊤ , . .
Revised: [ vec ( W 11 ) ⊤ , . .

Original: , vec ( W 1 L ) ⊤ ]
Revised: , vec ( W 1 L ) ⊤ ]

Original: ⊤ ∈ R p 1 , and σ is the ReLU activation function σ ( x ) = max { 0 , x } .
Revised: ⊤ ∈ R p 1 , and σ is the ReLU activation function σ ( x ) = max { 0 , x } .

Original: In round t , given x t,i , i ∈ [ k ] , f 1 ( x t,i ; θ 1 t − 1 ) is assigned to learn h ( x t,i ) .
Revised: In round t , given x t,i , i ∈ [ k ] , f 1 ( x t,i ; θ 1 t − 1 ) is assigned to learn h ( x t,i ) .

Original: Based on the fact h ( x t,i ) =
Revised: Based on the fact h ( x t,i ) =

Original: E y t ∼D Y| x t [1 − L ( y t,i , y t,y t )] , it is natural to regard 1 − L ( y t,i , y t,y t ) as the label for training f 1 .
Revised: E y t ∼D Y| x t [1 − L ( y t,i , y t,y t )] , it is natural to regard 1 − L ( y t,i , y t,y t ) as the label for training f 1 .

Original: Note that we take the basic fully-connected network as an example for the sake of analysis in over-parameterized networks and f 1 can be easily replaced with more complicated models depending on the tasks.
Revised: Note that we take the basic fully-connected network as an example for the sake of analysis in over-parameterized networks and f 1 can be easily replaced with more complicated models depending on the tasks.

Original: Exploration Network f 2 .
Revised: Exploration Network f 2 .

Original: In addition to the network f 1 , we assign another network f 2 to explore uncertain information contained in incoming instances.
Revised: In addition to the network f 1 , we assign another network f 2 to explore uncertain information contained in incoming instances.

Original: First, we carefully design the input of f 2 to incorporate the context vectors of the instance and the discrimination-ability of f 1 , to learn the error between the Bayes-optimal probability h ( x t,i ) and the prediction f 1 ( x t,i ; θ 1 ) .
Revised: First, we carefully design the input of f 2 to incorporate the context vectors of the instance and the discrimination-ability of f 1 , to learn the error between the Bayes-optimal probability h ( x t,i ) and the prediction f 1 ( x t,i ; θ 1 ) .

Original: Definition 3.1 (Derivative-Context (DC) Embedding) .
Revised: Definition 3.1 (Derivative-Context (DC) Embedding) .

Original: Given the exploitation network f 1 ( · ; θ 1 t − 1 ) and an input context x t,i , its DC embeding is defined as
Revised: Given the exploitation network f 1 ( · ; θ 1 t − 1 ) and an input context x t,i , its DC embeding is defined as

Original: [Equation]
Revised: [Equation]

Original: where ▽ x t,i f 1 is the partial derivative of f 1 ( x t,i ; θ 1 t − 1 ) with respect to x t,i .
Revised: where ▽ x t,i f 1 is the partial derivative of f 1 ( x t,i ; θ 1 t − 1 ) with respect to x t,i .

Original: ϕ
Revised: ϕ

Original: ( x t,i ) is normalized so that ∥ ϕ ( x t,i ) ∥ 2 = 1 .
Revised: ( x t,i ) is normalized so that ∥ ϕ ( x t,i ) ∥ 2 = 1 .

Original: Note that the input for f 2 in [14] is the gradient with respect to θ 1 , denoted by ▽ θ 1 f 1 ( x t,i ; θ
Revised: Note that the input for f 2 in [14] is the gradient with respect to θ 1 , denoted by ▽ θ 1 f 1 ( x t,i ; θ

Original: 1 t − 1 ) ∈ R p 1 .
Revised: 1 t − 1 ) ∈ R p 1 .

Original: Its dimensionality is much larger than ▽ x t,i f 1 ( x t,i ; θ 1 t − 1 ) in Definition 3.1, may causing significant computation cost.
Revised: Its dimensionality is much larger than ▽ x t,i f 1 ( x t,i ; θ 1 t − 1 ) in Definition 3.1, may causing significant computation cost.

Original: Given the input ϕ
Revised: Given the input ϕ

Original: ( x t,i ) , similarly, we choose the fully-connected network to build f 2 :[Equation]
Revised: ( x t,i ) , similarly, we choose the fully-connected network to build f 2 :

Original: [Equation]
Revised: [Equation]

Original: Input: T (number of rounds) f 1 , f 2 (neural networks), η 1 , η 2 (learning rate), γ (exploration parameter), b (batch size), δ (confidence level) 1:
Revised: Input: T (number of rounds) f 1 , f 2 (neural networks), η 1 , η 2 (learning rate), γ (exploration parameter), b (batch size), δ (confidence level) 1:

Original: Initialize θ 10 , θ 20 ; (cid:98) θ 1 

Revised: Initialize θ 10 , θ 20 ; (cid:98) θ 1

Original: =
Revised: =

Original: θ
Revised: θ

Original: 10 ; (cid:98) θ 2 0 = θ2: H 10 = ∅ ; H 20 = ∅ 3: for t = 1 , 2 , . . .
Revised: 10 ; (cid:98) θ 2 0 = θ2: H 10 = ∅ ; H 20 = ∅ 3: for t = 1 , 2 , . . .

Original: , T do 4:
Revised: , T do 4:

Original: Observe instance x t ∈ R d and build x t,i , ∀ i ∈
Revised: Observe instance x t ∈ R d and build x t,i , ∀ i ∈

Original: [ k ] 5: for each i ∈
Revised: [ k ] 5: for each i ∈

Original: [ k ] do 6: f ( x t,i ; θ t − 1 ) = (cid:16) f 1 ( x t,i ; θ 1 t − 1 ) Exploitation Score + f 2 ( ϕ ( x t,i ); θ 2 t − 1 ) Exploration Score (cid:17) 7: end for 8: (cid:98) i = arg max i ∈
Revised: [ k ] do 6: f ( x t,i ; θ t − 1 ) = (cid:16) f 1 ( x t,i ; θ 1 t − 1 ) Exploitation Score + f 2 ( ϕ ( x t,i ); θ 2 t − 1 ) Exploration Score (cid:17) 7: end for 8: (cid:98) i = arg max i ∈

Original: [ k ] f ( x t,i ; θ t − 1 ) 9: i ◦ = arg max i ∈ ([ k ] \{ (cid:98) i } ) f ( x t,i ; θ t − 1 ) 10:
Revised: [ k ] f ( x t,i ; θ t − 1 ) 9: i ◦ = arg max i ∈ ([ k ] \{ (cid:98) i } ) f ( x t,i ; θ t − 1 ) 10:

Original: Predict y t, (cid:98) i 11:
Revised: Predict y t, (cid:98) i 11:

Original: I t = 1 {| f ( x t, (cid:98) i ; θ t − 1 ) − f ( x t,i ◦ ; θ t − 1 ) | < 2 γ β t } ∈ { 0 , 1 } ; β t = (cid:113) 2 c 1 t + (cid:16) c 2 3 L √ 2 t (cid:17) + (cid:113) 2log
Revised: I t = 1 {| f ( x t, (cid:98) i ; θ t − 1 ) − f ( x t,i ◦ ; θ t − 1 ) | < 2 γ β t } ∈ { 0 , 1 } ; β t = (cid:113) 2 c 1 t + (cid:16) c 2 3 L √ 2 t (cid:17) + (cid:113) 2log

Original: ( c 3 tk ) /δ )
Revised: ( c 3 tk ) /δ )

Original: t 12: if I t = 1 then 13:
Revised: t 12: if I t = 1 then 13:

Original: Query x t and observe y t 14: for i ∈ [ k ] do 15: r 1 t,i = 1 − L ( y t,i , y t,y t ) (defined in E.q.
Revised: Query x t and observe y t 14: for i ∈ [ k ] do 15: r 1 t,i = 1 − L ( y t,i , y t,y t ) (defined in E.q.

Original: (2.2)) 16: r 2 t,i =
Revised: (2.2)) 16: r 2 t,i =

Original: r 1 t,i − f 1 ( x t,i ; θ 1 t − 1 ) 17: end for 18: else 19: for i ∈ [ k ] do 20: r 1 t,i = 1 − L ( y t,i , y t, (cid:98) i ) 21: r 2 t,i =
Revised: r 1 t,i − f 1 ( x t,i ; θ 1 t − 1 ) 17: end for 18: else 19: for i ∈ [ k ] do 20: r 1 t,i = 1 − L ( y t,i , y t, (cid:98) i ) 21: r 2 t,i =

Original: r 1 t,i − f 1 ( x t,i ; θ 1 t − 1 ) 22: end for 23: end if 24:
Revised: r 1 t,i − f 1 ( x t,i ; θ 1 t − 1 ) 22: end for 23: end if 24:

Original: H 1 t
Revised: H 1 t

Original: =
Revised: =

Original: H 1 t −
Revised: H 1 t −

Original: [Equation]
Revised: [Equation]

Original: ∪ { ( x t,i , r 1 t,i ) , i ∈ [ k ] } 25:
Revised: ∪ { ( x t,i , r 1 t,i ) , i ∈ [ k ] } 25:

Original: H 2 t
Revised: H 2 t

Original: = H 2 t − 1 ∪ { ( x t,i , r 2 t,i ) , i ∈
Revised: = H 2 t − 1 ∪ { ( x t,i , r 2 t,i ) , i ∈

Original: [ k ] } 26: θ 1 t , θ 2 t =
Revised: [ k ] } 26: θ 1 t , θ 2 t =

Original: Mini-Batch-SGD-Warm-Start ( f 1 , f 2 , H 1 t , H 2 t , b ) 27: end for where W 21 ∈ R m × 2 kd , W 2 l ∈ R m × m , for 2 ≤ l ≤ L − 1 , W 2 L = R 1 × m and θ 2 =
Revised: Mini-Batch-SGD-Warm-Start ( f 1 , f 2 , H 1 t , H 2 t , b ) 27: end for where W 21 ∈ R m × 2 kd , W 2 l ∈ R m × m , for 2 ≤ l ≤ L − 1 , W 2 L = R 1 × m and θ 2 =

Original: [ vec ( W 21 ) ⊤ , . .
Revised: [ vec ( W 21 ) ⊤ , . .

Original: , vec ( W 2 L ) ⊤ ]
Revised: , vec ( W 2 L ) ⊤ ]

Original: ⊤ ∈ R p 2 .
Revised: ⊤ ∈ R p 2 .

Original: In round t , given x t,i , ∀ i ∈ [ k ] , f 2 is to predict h ( x t,i ) − f 1 ( x t,i ; θ 1 t − 1 ) for exploration.
Revised: In round t , given x t,i , ∀ i ∈ [ k ] , f 2 is to predict h ( x t,i ) − f 1 ( x t,i ; θ 1 t − 1 ) for exploration.

Original: Because h ( x t,i ) − f 1 ( x t,i ; θ 1 t − 1 ) =
Revised: Because h ( x t,i ) − f 1 ( x t,i ; θ 1 t − 1 ) =

Original: E y
Revised: E y

Original: [Equation]
Revised: [Equation]

Original: ∼D Y| x t [1 − L ( y t,i , y t,y t ) − f 1 ( x t,i ; θ 1 t − 1 )] , we regard 1 − L ( y t,i , y t,y t ) − f 1 ( x t,i ; θ 1 t − 1 ) as the label for training f 2 .
Revised: ∼D Y| x t [1 − L ( y t,i , y t,y t ) − f 1 ( x t,i ; θ 1 t − 1 )] , we regard 1 − L ( y t,i , y t,y t ) − f 1 ( x t,i ; θ 1 t − 1 ) as the label for training f 2 .

Original: To sum up, in round t , given x t,i , ∀ i ∈ [ k ] , the prediction (cid:98) i ( y t, (cid:98) i ) is made based on the sum of exploitation and exploration scores, i.e., f 1 ( x t,i ; θ 1 t − 1 ) + f 2 ( ϕ ( x t,i ); θ 2 t − 1 ) (Lines 5-10).
Revised: To sum up, in round t , given x t,i , ∀ i ∈ [ k ] , the prediction (cid:98) i ( y t, (cid:98) i ) is made based on the sum of exploitation and exploration scores, i.e., f 1 ( x t,i ; θ 1 t − 1 ) + f 2 ( ϕ ( x t,i ); θ 2 t − 1 ) (Lines 5-10).

Original: Query Decision-maker (Line 11) .
Revised: Query Decision-maker (Line 11) .

Original: A label query is made when I-NeurAL is not confident enough to discriminate the Bayes-optimal class from other classes.
Revised: A label query is made when I-NeurAL is not confident enough to discriminate the Bayes-optimal class from other classes.

Original: 2 γ β t ( β t is also defined in Lemma 7.3) can be thought of as a confidence interval for the distance between the optimal class and second optimal class, where γ is the hyper-parameter to tune the sensitivity of the decision-maker in practice.
Revised: 2 γ β t ( β t is also defined in Lemma 7.3) can be thought of as a confidence interval for the distance between the optimal class and second optimal class, where γ is the hyper-parameter to tune the sensitivity of the decision-maker in practice.

Original: Given any γ ≥ 1 , δ ∈ (0 , 1) , with probability at least 1 − δ , based on our analysis (Lemma 7.5), E ( x t ,y t ) ∼D [ L ( y t, (cid:98) i , y t,y t )]
Revised: Given any γ ≥ 1 , δ ∈ (0 , 1) , with probability at least 1 − δ , based on our analysis (Lemma 7.5), E ( x t ,y t ) ∼D [ L ( y t, (cid:98) i , y t,y t )]

Original: =
Revised: =

Original: E ( x t ,y t )
Revised: E ( x t ,y t )

Original: ∼D
Revised: ∼D

Original: [ L ( y t,i ∗ , y t,y t )] when I t = 0 , i.e., I-NeurAL suffers no regret.
Revised: [ L ( y t,i ∗ , y t,y t )] when I t = 0 , i.e., I-NeurAL suffers no regret.

Original: Thus, we use y t, (cid:98) i as the pseudo-label in this case and we have the following update rules.
Revised: Thus, we use y t, (cid:98) i as the pseudo-label in this case and we have the following update rules.

Original: Utilize Full Feedback (Lines 14-25) .
Revised: Utilize Full Feedback (Lines 14-25) .

Original: Different from the bandit setting where the learner can only observe the reward of the selected context, we can infer the rewards of all contexts in active learning, as we know the specific class of the current instance.
Revised: Different from the bandit setting where the learner can only observe the reward of the selected context, we can infer the rewards of all contexts in active learning, as we know the specific class of the current instance.

Original: Thus, for each x t,i , i ∈ [ k ] , r 1 t,i = 1 −L ( y t,i , y t,y t ) is regarded as the "reward" of x t,i , predicted by f 1 , and r 2 t,i =
Revised: Thus, for each x t,i , i ∈ [ k ] , r 1 t,i = 1 −L ( y t,i , y t,y t ) is regarded as the "reward" of x t,i , predicted by f 1 , and r 2 t,i =

Original: r 1 t,i − f 1 ( x t,i ; θ 1 ) is regarded as the
Revised: r 1 t,i − f 1 ( x t,i ; θ 1 ) is regarded as the

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: from Ω
Revised: from Ω

Original: "residual reward" of x t,i , predicted by f 2 .
Revised: "residual reward" of x t,i , predicted by f 2 .

Original: In summary, in round t , when I t = 1 , y t,y t is observed to update r 1 t,i and r 2 t,i ; when I t = 0 , y t, (cid:98) i is regard as the pseudo-label to obtain r 1 t,i and r 2 t,i , ∀ i ∈ [ k ] .
Revised: In summary, in round t , when I t = 1 , y t,y t is observed to update r 1 t,i and r 2 t,i ; when I t = 0 , y t, (cid:98) i is regard as the pseudo-label to obtain r 1 t,i and r 2 t,i , ∀ i ∈ [ k ] .

Original: Therefore, we have the training data H 1 t for f 1 and H 2 t for f 2 .
Revised: Therefore, we have the training data H 1 t for f 1 and H 2 t for f 2 .

Original: Mini-Batch SGD with Warm-Start (Algorithm 3) .
Revised: Mini-Batch SGD with Warm-Start (Algorithm 3) .

Original: Unlike [50] that uses vanilla gradient descent from randomly initialized parameters in each round, causing unnecessarily expensive computation, we extend the training procedure to mini-batch SGD with warm start, i.e., we incrementally train the parameters θ t starting from the parameters of the last round θ t − 1 in each round t .
Revised: Unlike [50] that uses vanilla gradient descent from randomly initialized parameters in each round, causing unnecessarily expensive computation, we extend the training procedure to mini-batch SGD with warm start, i.e., we incrementally train the parameters θ t starting from the parameters of the last round θ t − 1 in each round t .

Original: Algorithm 1 depicts the workflow of I-NeurAL.
Revised: Algorithm 1 depicts the workflow of I-NeurAL.

Original: Lines 1-2 initialize the parameters where each entry of W l is drawn from the normal distribution N (0 , 2 /m ) and each entry of W L is drawn from N (0 , 1 /m ) for both f 1 and f 2 .
Revised: Lines 1-2 initialize the parameters where each entry of W l is drawn from the normal distribution N (0 , 2 /m ) and each entry of W L is drawn from N (0 , 1 /m ) for both f 1 and f 2 .

Original: H 10 , H 20 store the historical data for f 1 and f 2 respectively.
Revised: H 10 , H 20 store the historical data for f 1 and f 2 respectively.

Original: In each round t , Line 4 builds the k contexts for the observed instance x t , and Lines 5-7 calculate the exploitation-exploration score for each context.
Revised: In each round t , Line 4 builds the k contexts for the observed instance x t , and Lines 5-7 calculate the exploitation-exploration score for each context.

Original: (cid:98) i
Revised: (cid:98) i

Original: (Line 8) is the index of the optimal-predicted class and thus y t, (cid:98) i is the prediction.
Revised: (Line 8) is the index of the optimal-predicted class and thus y t, (cid:98) i is the prediction.

Original: i ◦ (Line 9) is the index of the second optimal-predicted class, which is used to decide whether to make a query.
Revised: i ◦ (Line 9) is the index of the second optimal-predicted class, which is used to decide whether to make a query.

Original: Line 11 is our decision component.
Revised: Line 11 is our decision component.

Original: When I t = 1 , it shows that we are not confident enough about our prediction, so that we make a query for x t and observe the rewards for each context (Lines 12-17).
Revised: When I t = 1 , it shows that we are not confident enough about our prediction, so that we make a query for x t and observe the rewards for each context (Lines 12-17).

Original: When I t = 0 , based on our analysis, with high confidence, the prediction y t, (cid:98) i matches the one predicted by the Bayes-optimal classifier.
Revised: When I t = 0 , based on our analysis, with high confidence, the prediction y t, (cid:98) i matches the one predicted by the Bayes-optimal classifier.

Original: Hence, we consider y t, (cid:98) i as the label and observe the reward for all contexts (Lines 18-23).
Revised: Hence, we consider y t, (cid:98) i as the label and observe the reward for all contexts (Lines 18-23).

Original: In the end, we update the networks f 1 and f 2 , based on the collected data (Lines 24-26).
Revised: In the end, we update the networks f 1 and f 2 , based on the collected data (Lines 24-26).

Original:  Regret Analysis
Revised:  Regret Analysis

Original: In this section, we provide the regret analysis of I-NeurAL in the over-parameterized neural networks.
Revised: In this section, we provide the regret analysis of I-NeurAL in the over-parameterized neural networks.

Original: First, we need the standard normalization and separateness restricted to the input instances.
Revised: First, we need the standard normalization and separateness restricted to the input instances.

Original: Assumption 4.1.
Revised: Assumption 4.1.

Original: For any t ∈ [ T ] , ∥ x t ∥ 2 = 1 .
Revised: For any t ∈ [ T ] , ∥ x t ∥ 2 = 1 .

Original: This is standard assumption in the analysis of over-parameterized neural networks[2, 24, 4], neural active learning [50] and neural bandits [57, 14].
Revised: This is standard assumption in the analysis of over-parameterized neural networks[2, 24, 4], neural active learning [50] and neural bandits [57, 14].

Original: Given a constant ν > 0 , we define the following ν -ball of θ 2 around the random initialization: B ( θ 20 , ν ) = { (cid:101) θ 2 : ∥ (cid:101) θ 2 − θ 20 ∥ 2 ≤ O ( ν √ m ) } .
Revised: Given a constant ν > 0 , we define the following ν -ball of θ 2 around the random initialization: B ( θ 20 , ν ) = { (cid:101) θ 2 : ∥ (cid:101) θ 2 − θ 20 ∥ 2 ≤ O ( ν √ m ) } .

Original: Recall that r 2 t, (cid:98) i =
Revised: Recall that r 2 t, (cid:98) i =

Original: r 1 t, (cid:98) i − f 1 ( x t, (cid:98) i ; (cid:98) θ 1 t − 1 ) .
Revised: r 1 t, (cid:98) i − f 1 ( x t, (cid:98) i ; (cid:98) θ 1 t − 1 ) .

Original: Let (cid:98) θ 1 , ∗ t − 1 represent the parameters trained on H 1 , ∗ t − 1 using Algorithmwith the Bayes-optimal classifier, where H 1 , ∗ t − 1 = { x τ,i ∗ , r 1 τ,i ∗ } t − 1 τ =1 are the historical Bayes-optimal pairs.
Revised: Let (cid:98) θ 1 , ∗ t − 1 represent the parameters trained on H 1 , ∗ t − 1 using Algorithmwith the Bayes-optimal classifier, where H 1 , ∗ t − 1 = { x τ,i ∗ , r 1 τ,i ∗ } t − 1 τ =1 are the historical Bayes-optimal pairs.

Original: We define r 2 , ∗ t,i ∗ =
Revised: We define r 2 , ∗ t,i ∗ =

Original: r 1 t,i ∗ − f 1 ( x t,i ∗ ; (cid:98) θ 1 , ∗ t − 1 ) .
Revised: r 1 t,i ∗ − f 1 ( x t,i ∗ ; (cid:98) θ 1 , ∗ t − 1 ) .

Original: Then, we provide the following regret bound that depends on the classification ability of exploration network class induced by B ( θ 2 0 , ν ) .
Revised: Then, we provide the following regret bound that depends on the classification ability of exploration network class induced by B ( θ 2 0 , ν ) .

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: Given the number of rounds T , for any δ ∈ (0 , 1) , γ > 1 , ν > 0 , suppose
Revised: Given the number of rounds T , for any δ ∈ (0 , 1) , γ > 1 , ν > 0 , suppose

Original: [Equation]
Revised: [Equation]

Original: inf
Revised: inf

Original: [Equation]
Revised: [Equation]

Original: ≤ µ .
Revised: ≤ µ .

Original: Then, with probability at least 1 − δ over the initialization of θ 10 , θ 20 , there exist a small enough constant ν ′ , such that Algorithm 1 achieves the following regret bound:
Revised: Then, with probability at least 1 − δ over the initialization of θ 10 , θ 20 , there exist a small enough constant ν ′ , such that Algorithm 1 achieves the following regret bound:

Original: [Equation]
Revised: [Equation]

Original: and at the same time N T ≤ O ( T ) .
Revised: and at the same time N T ≤ O ( T ) .

Original: Theorem 4.1 provides the regret bound of I-NeurAL for R T and R T respectively, R T ≤ O ( √ log T √ T ) and R T ≤ O ( √ T log T ) .
Revised: Theorem 4.1 provides the regret bound of I-NeurAL for R T and R T respectively, R T ≤ O ( √ log T √ T ) and R T ≤ O ( √ T log T ) .

Original: As [50] only provides the regret bound for (cid:101) R T , to show the advantages of
Revised: As [50] only provides the regret bound for (cid:101) R T , to show the advantages of

Original: I-NeurAL, we also provide the following lemma for fair comparison.
Revised: I-NeurAL, we also provide the following lemma for fair comparison.

Original: Lemma 4.1.
Revised: Lemma 4.1.

Original: Given the number of rounds T , for any δ ∈ (0 , 1) , γ > 1 , ν > 0 , suppose m ≥ (cid:101) Ω( poly ( T, k, L, ν )) , η 1 = η 2 =
Revised: Given the number of rounds T , for any δ ∈ (0 , 1) , γ > 1 , ν > 0 , suppose m ≥ (cid:101) Ω( poly ( T, k, L, ν )) , η 1 = η 2 =

Original: Θ
Revised: Θ

Original: ( ν ′ ν √ Tm ) , and µ satisfies the conditions in Theorem 4.1.
Revised: ( ν ′ ν √ Tm ) , and µ satisfies the conditions in Theorem 4.1.

Original: Then, with probability at least 1 − δ over the initialization of θ 10 , θ 20 , these exists a small enough constant ν ′ , such that Algorithm 1 can achieve the following regret bound:
Revised: Then, with probability at least 1 − δ over the initialization of θ 10 , θ 20 , these exists a small enough constant ν ′ , such that Algorithm 1 can achieve the following regret bound:

Original: [Equation]
Revised: [Equation]

Original: and at the same time N T ≤ O ( T ) .
Revised: and at the same time N T ≤ O ( T ) .

Original: Comparison with [50] .
Revised: Comparison with [50] .

Original: Lemma 4.1 shows that I-NeurAL can achieve the regret bound of same complexity for (cid:101) R T as R T .
Revised: Lemma 4.1 shows that I-NeurAL can achieve the regret bound of same complexity for (cid:101) R T as R T .

Original: Under the same assumption in the over-parameterized neural networks, without any assumption on D , Theorem 1 in [50] (i.e., the lower-noise condition with exponent α = 0 , and k = 2 is ignored in the binary classification) achieves the following regret bound: (cid:101) R T ≤
Revised: Under the same assumption in the over-parameterized neural networks, without any assumption on D , Theorem 1 in [50] (i.e., the lower-noise condition with exponent α = 0 , and k = 2 is ignored in the binary classification) achieves the following regret bound: (cid:101) R T ≤

Original: O (log det( I + H ) (cid:112) T (log det( I + H ) + S 2 )) where H is the NTK matrix [29, 4] formed by received instances of all T rounds, S = √
Revised: O (log det( I + H ) (cid:112) T (log det( I + H ) + S 2 )) where H is the NTK matrix [29, 4] formed by received instances of all T rounds, S = √

Original: h ⊤ H − 1 h is a complexity term, and
Revised: h ⊤ H − 1 h is a complexity term, and

Original: h =
Revised: h =

Original: ( h ( x 1 , (cid:98) i ) , . . .
Revised: ( h ( x 1 , (cid:98) i ) , . . .

Original: , h ( x T, (cid:98) i ))
Revised: , h ( x T, (cid:98) i ))

Original: ⊤ ∈ R T .
Revised: ⊤ ∈ R T .

Original: Note that I-NeurAL and [50] have the same trivial label complexity O ( T ) in this difficult case.
Revised: Note that I-NeurAL and [50] have the same trivial label complexity O ( T ) in this difficult case.

Original: According to the definition of effective dimension (cid:101) d in [57], the above regret bound obtained by [50] can be represented by:
Revised: According to the definition of effective dimension (cid:101) d in [57], the above regret bound obtained by [50] can be represented by:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Remark 4.1.
Revised: Remark 4.1.

Original: The complexity term µ reflects the possible minimal convergence error on the data caused by the functions induced the parameters in B ( θ 20 , ν ) controlled by ν .
Revised: The complexity term µ reflects the possible minimal convergence error on the data caused by the functions induced the parameters in B ( θ 20 , ν ) controlled by ν .

Original: Such complexity term is first introduced in [17].
Revised: Such complexity term is first introduced in [17].

Original: When ν is small, the corresponding ball B ( θ 20 , ν ) is small, so µ will be large; Otherwise, when ν is large, µ will be small.
Revised: When ν is small, the corresponding ball B ( θ 20 , ν ) is small, so µ will be large; Otherwise, when ν is large, µ will be small.

Original: In particular, when setting ν = O (1) , Theorem 4.1 and Lemma 4.1 suggests that if the data can be learned by a function in the function class formed by B ( θ 20 , O (1)) with the small training error, then I-NeurAL will have small regret.
Revised: In particular, when setting ν = O (1) , Theorem 4.1 and Lemma 4.1 suggests that if the data can be learned by a function in the function class formed by B ( θ 20 , O (1)) with the small training error, then I-NeurAL will have small regret.

Original: Note that [50] has the complexity term S as well, to reflect the boundary of optimal parameters specific to the data.
Revised: Note that [50] has the complexity term S as well, to reflect the boundary of optimal parameters specific to the data.

Original: Remark 4.2.
Revised: Remark 4.2.

Original: Theorem 4.1 and Lemma 4.1 do not depend on (cid:101) d .
Revised: Theorem 4.1 and Lemma 4.1 do not depend on (cid:101) d .

Original: The effective dimension (cid:101) d was first introduced in [48] and then used in [57], which can be thought of as the non-linear dimensionalities in the NTK kernel space.
Revised: The effective dimension (cid:101) d was first introduced in [48] and then used in [57], which can be thought of as the non-linear dimensionalities in the NTK kernel space.

Original: However, (cid:101) d can be p = m
Revised: However, (cid:101) d can be p = m

Original: + mkd + m 

Revised: + mkd + m

Original: ( L − 1) in the worst case, i.e., (cid:101)
Revised: ( L − 1) in the worst case, i.e., (cid:101)

Original: d ≫ T (see details in Appendix 9).
Revised: d ≫ T (see details in Appendix 9).

Original: Eq.(4.4) has the term O ( (cid:101) d ) and thus the regret bound obtained by [50] can explode due to (cid:101) d .
Revised: Eq.(4.4) has the term O ( (cid:101) d ) and thus the regret bound obtained by [50] can explode due to (cid:101) d .

Original: This is because the analysis of [50] closely depends on NTK, i.e., to apply Confidence Ellipsoid bound (Theorem 2 in [1]) to the NTK approximation.
Revised: This is because the analysis of [50] closely depends on NTK, i.e., to apply Confidence Ellipsoid bound (Theorem 2 in [1]) to the NTK approximation.

Original: This procedure inevitably bind their regret bound to the determinant of NTK that can have a very large magnitude.
Revised: This procedure inevitably bind their regret bound to the determinant of NTK that can have a very large magnitude.

Original: In contrast, Eq.(4.3) does not have the term (cid:101) d , because our analysis does not depend on the NTK approximation and I-NeurAL directly utilizes the property of over-parameterized neural networks, i.e., the convergence error µ and the generalization concentration bound (Lemma 7.6).
Revised: In contrast, Eq.(4.3) does not have the term (cid:101) d , because our analysis does not depend on the NTK approximation and I-NeurAL directly utilizes the property of over-parameterized neural networks, i.e., the convergence error µ and the generalization concentration bound (Lemma 7.6).

Original: These two terms are independent of (cid:101) d , which paves the way for I-NeurAL to remove the curse of (cid:101) d .
Revised: These two terms are independent of (cid:101) d , which paves the way for I-NeurAL to remove the curse of (cid:101) d .

Original: Remark 4.3.
Revised: Remark 4.3.

Original: Theorem 4.1 and Lemma 4.1 improve the regret by a multiplicative factor
Revised: Theorem 4.1 and Lemma 4.1 improve the regret by a multiplicative factor

Original: O (log T ) over [50].
Revised: O (log T ) over [50].

Original: Note that the analysis of [50] is built for binary classification and thus k = 2 in Theorem
Revised: Note that the analysis of [50] is built for binary classification and thus k = 2 in Theorem

Original: 4.1 and Lemma 4.1.
Revised: 4.1 and Lemma 4.1.

Original: This improvement stems from the different analysis workflow of I-NeurAL from [50].
Revised: This improvement stems from the different analysis workflow of I-NeurAL from [50].

Original: Again, our analysis does not rely on NTK approximation and it is built on the convergence and generalization bound of wide neural networks.
Revised: Again, our analysis does not rely on NTK approximation and it is built on the convergence and generalization bound of wide neural networks.

Original: Remark 4.4.
Revised: Remark 4.4.

Original: Our proof workflow of Theorem 4.1 and Lemma 4.1 is inspired by [14].
Revised: Our proof workflow of Theorem 4.1 and Lemma 4.1 is inspired by [14].

Original: Compared to [14], we provide the first regret bound supporting mini-batch SGD with warm-start and a more generic generalization bound (Lemma 7.3) that holds for every arm (class).
Revised: Compared to [14], we provide the first regret bound supporting mini-batch SGD with warm-start and a more generic generalization bound (Lemma 7.3) that holds for every arm (class).

Original: Moreover, we carry out the performance analysis of query decision-maker (Lemma 7.5), which is a new addition.
Revised: Moreover, we carry out the performance analysis of query decision-maker (Lemma 7.5), which is a new addition.

Original: For the label complexity, N T has the trivial O ( T ) complexity which is the same as Theorem 1 in [50] (with the exponent α = 0 ).
Revised: For the label complexity, N T has the trivial O ( T ) complexity which is the same as Theorem 1 in [50] (with the exponent α = 0 ).

Original: Because we have to consider the worst case where the unique Bayes-optimal class does not exist, i.e., given x t,i , i ∈ [ k ] , there does not exist i ∗ such that h ( x t,i ∗ )
Revised: Because we have to consider the worst case where the unique Bayes-optimal class does not exist, i.e., given x t,i , i ∈ [ k ] , there does not exist i ∗ such that h ( x t,i ∗ )

Original: > h ( x t,i ) , ∀ i ∈
Revised: > h ( x t,i ) , ∀ i ∈

Original: [ k ] \ { i ∗ } .
Revised: [ k ] \ { i ∗ } .

Original: Therefore, we provide the following analysis and show that R T and N T can be upper bounded by constants as long as there exists a unique Bayes-optimal class for the input instances, described by the following mild margin assumption.
Revised: Therefore, we provide the following analysis and show that R T and N T can be upper bounded by constants as long as there exists a unique Bayes-optimal class for the input instances, described by the following mild margin assumption.

Original: Assumption 4.2 ( ϵ -margin) .
Revised: Assumption 4.2 ( ϵ -margin) .

Original: In round t ∈ [ T ] , given an instance x t and the label y t , then x t has the ϵ -Unique optimal class if there exists ϵ > 0 such that
Revised: In round t ∈ [ T ] , given an instance x t and the label y t , then x t has the ϵ -Unique optimal class if there exists ϵ > 0 such that

Original: [Equation]
Revised: [Equation]

Original: where i ∗ = arg max i ∈ [ k ] h ( x t,i ) is the Bayes-optimal class and i ◦ = arg max i ∈
Revised: where i ∗ = arg max i ∈ [ k ] h ( x t,i ) is the Bayes-optimal class and i ◦ = arg max i ∈

Original: ([ k ] \{ i ∗ } ) h ( x t,i ) is the second Bayes-optimal class.
Revised: ([ k ] \{ i ∗ } ) h ( x t,i ) is the second Bayes-optimal class.

Original: Given any i ∈ [ k ] , let i be a fixed index, i.e., suppose there exist a policy π i which always select the i -th context ( x t,i , r 1 t,i ) for every round t ∈ [ T ] .
Revised: Given any i ∈ [ k ] , let i be a fixed index, i.e., suppose there exist a policy π i which always select the i -th context ( x t,i , r 1 t,i ) for every round t ∈ [ T ] .

Original: Then, in round t , we have the collected data by Ω i : H 1 ,it − 1 = { x τ,i , r 1 τ,i } t − 1 τ =1 .
Revised: Then, in round t , we have the collected data by Ω i : H 1 ,it − 1 = { x τ,i , r 1 τ,i } t − 1 τ =1 .

Original: Then, let (cid:98) θ 1 ,i t − 1 represent the parameters trained only on H 1 ,it − 1 using
Revised: Then, let (cid:98) θ 1 ,i t − 1 represent the parameters trained only on H 1 ,it − 1 using

Original: Algorithm 2 with π i and r 2 ,it,i =
Revised: Algorithm 2 with π i and r 2 ,it,i =

Original: r 1 t,i − f 1 ( x t,i ; (cid:98) θ 1 ,i t − 1 ) .
Revised: r 1 t,i − f 1 ( x t,i ; (cid:98) θ 1 ,i t − 1 ) .

Original: Theorem 4.2.
Revised: Theorem 4.2.

Original: Suppose the instances drawn from D satisfy Assumption 4.2.
Revised: Suppose the instances drawn from D satisfy Assumption 4.2.

Original: Then, given the number of rounds T , for any δ ∈ (0 , 1) , γ > 1 , ϵ ∈ (0 , 1) , ν > 0 , suppose m ≥ (cid:101) Ω( poly ( T, k, L, ν )) , η 1 = η 2 =
Revised: Then, given the number of rounds T , for any δ ∈ (0 , 1) , γ > 1 , ϵ ∈ (0 , 1) , ν > 0 , suppose m ≥ (cid:101) Ω( poly ( T, k, L, ν )) , η 1 = η 2 =

Original: Θ
Revised: Θ

Original: ( ν ′ ν √ Tm ) , and µ satisfies the conditions in Theorem 4.1 and max i
Revised: ( ν ′ ν √ Tm ) , and µ satisfies the conditions in Theorem 4.1 and max i

Original: [Equation]
Revised: [Equation]

Original: ≤ µ .
Revised: ≤ µ .

Original: Then, with probability at least 1 − δ over the initialization of θ 10 , θ 20 , there exists a constant ν ′ , such that Algorithm 1 achieves the following regret bound:
Revised: Then, with probability at least 1 − δ over the initialization of θ 10 , θ 20 , there exists a constant ν ′ , such that Algorithm 1 achieves the following regret bound:

Original: [Equation]
Revised: [Equation]

Original: where ξ = 2 log 2 √ 6( γ +1) ϵ + log( O ( k ))
Revised: where ξ = 2 log 2 √ 6( γ +1) ϵ + log( O ( k ))

Original: + 94 ( O ( L 2 ν 2 ) + 4 µ 9 ) − log δ.
Revised: + 94 ( O ( L 2 ν 2 ) + 4 µ 9 ) − log δ.

Original: Remark 4.5.
Revised: Remark 4.5.

Original: Theorem 4.2 provides the upper bound for R T that is independent of T .
Revised: Theorem 4.2 provides the upper bound for R T that is independent of T .

Original: When other parameters are fixed, this indicates R T is upper bounded by a constant when T → + ∞ .
Revised: When other parameters are fixed, this indicates R T is upper bounded by a constant when T → + ∞ .

Original: Moreover, the analysis of R T indicates that I-NeurAL can achieve the same performance as Bayesoptimal classifier with high confidence after a fixed number of rounds (i.e. T > 48( γ +1) 2 ϵξ ).
Revised: Moreover, the analysis of R T indicates that I-NeurAL can achieve the same performance as Bayesoptimal classifier with high confidence after a fixed number of rounds (i.e. T > 48( γ +1) 2 ϵξ ).

Original: To the best of our knowledge, this is the first analysis for neural active learning to show that an NNbased algorithm can achieve the performance as the Bayes-optimal classifier after a fixed number of rounds.
Revised: To the best of our knowledge, this is the first analysis for neural active learning to show that an NNbased algorithm can achieve the performance as the Bayes-optimal classifier after a fixed number of rounds.

Original: In Theorem 1 of [50] ( with the exponent α → + ∞ equivalent to Assumption 4.2), (cid:101) R T ≤ O ( (cid:101)
Revised: In Theorem 1 of [50] ( with the exponent α → + ∞ equivalent to Assumption 4.2), (cid:101) R T ≤ O ( (cid:101)

Original: d log(1 + T ))
Revised: d log(1 + T ))

Original: (cid:113) ( (cid:101)
Revised: (cid:113) ( (cid:101)

Original: d log(1 + T ) + S 2 ) that still is dependent on T because NTK depends on T and (cid:101) d .
Revised: d log(1 + T ) + S 2 ) that still is dependent on T because NTK depends on T and (cid:101) d .

Original:  Experiments
Revised:  Experiments

Original: In this section, we evaluate I-NeurAL on public classification data sets compared with state-of-theart (SOTA) baselines.
Revised: In this section, we evaluate I-NeurAL on public classification data sets compared with state-of-theart (SOTA) baselines.

Original: Due to the space limit, we only report the main results here and leave the implementation details and parameter sensitivity in the Appendix 10.
Revised: Due to the space limit, we only report the main results here and leave the implementation details and parameter sensitivity in the Appendix 10.

Original: [Figure]
Revised: [Figure]

Original: We report the experimental results on the following six data sets: Phishing 1 , IJCNN [40], Letter [19],
Revised: We report the experimental results on the following six data sets: Phishing 1 , IJCNN [40], Letter [19],

Original: Fashion [51], MNIST [34] and CIFAR-10 [33].
Revised: Fashion [51], MNIST [34] and CIFAR-10 [33].

Original: In each round, one instance is randomly drawn from the data set and the algorithm is compelled to make prediction on it.
Revised: In each round, one instance is randomly drawn from the data set and the algorithm is compelled to make prediction on it.

Original: Then, the regret is 1 if the prediction does not match the label; the regret is 0 , otherwise.
Revised: Then, the regret is 1 if the prediction does not match the label; the regret is 0 , otherwise.

Original: At the same time, if the algorithm decides to observe the label, it costs one query budget.
Revised: At the same time, if the algorithm decides to observe the label, it costs one query budget.

Original: As the algorithm may abusively make label queries, we restrict the query budget to 3% of the total number of instances in the data set for fair comparison.
Revised: As the algorithm may abusively make label queries, we restrict the query budget to 3% of the total number of instances in the data set for fair comparison.

Original: The compared baselines are described as follows.
Revised: The compared baselines are described as follows.

Original: (1) Random :
Revised: (1) Random :

Original: The NN classifier queries the label with a fixed probability p until the query budget is exhausted; (2) Margin : The NN classifier queries the label when the predicted probability is lower than a threshold.
Revised: The NN classifier queries the label with a fixed probability p until the query budget is exhausted; (2) Margin : The NN classifier queries the label when the predicted probability is lower than a threshold.

Original: These two baselines are used in [23].
Revised: These two baselines are used in [23].

Original: (3) NeuAL-NTK-F (Algorithm 1 in [50] : This model makes predictions based on the frozen
Revised: (3) NeuAL-NTK-F (Algorithm 1 in [50] : This model makes predictions based on the frozen

Original: NTK approximation coming with an Upper-Confidence-Bound(UCB)-based exploration strategy.
Revised: NTK approximation coming with an Upper-Confidence-Bound(UCB)-based exploration strategy.

Original: NeuAL-NTK-D (Algorithm 3 in [50]) :
Revised: NeuAL-NTK-D (Algorithm 3 in [50]) :

Original: The prediction is made based on the NN classifier with a
Revised: The prediction is made based on the NN classifier with a

Original: UCB while the NTK is updated accordingly.
Revised: UCB while the NTK is updated accordingly.

Original: (5) ALPS [23] : Given a class of pre-trained hypotheses, the hypothesis minimizing the logistic loss of labeled and pseudo-labeled data is chosen to make predictions and the label query is based on the disagreement of different hypotheses.
Revised: (5) ALPS [23] : Given a class of pre-trained hypotheses, the hypothesis minimizing the logistic loss of labeled and pseudo-labeled data is chosen to make predictions and the label query is based on the disagreement of different hypotheses.

Original: Results .
Revised: Results .

Original: The regret comparison on six data sets is shown in Table 1 and Figure 1.
Revised: The regret comparison on six data sets is shown in Table 1 and Figure 1.

Original: I-NeurAL consistently outperforms all baselines across all data sets.
Revised: I-NeurAL consistently outperforms all baselines across all data sets.

Original: In particular, I-NeurAL surpasses the best baseline by 31.3%, 45.6%, 52.2% on IJCNN, MNIST, Fashion respectively.
Revised: In particular, I-NeurAL surpasses the best baseline by 31.3%, 45.6%, 52.2% on IJCNN, MNIST, Fashion respectively.

Original: Since NeuALNTK-F uses frozen NTK approximation, the new knowledge of each round is barely utilized by the neural network and thus it turns into the worst baseline.
Revised: Since NeuALNTK-F uses frozen NTK approximation, the new knowledge of each round is barely utilized by the neural network and thus it turns into the worst baseline.

Original: NeuAL-NTK-D updates the network parameters with gradient descent and queries the label based on the uncertainty estimation.
Revised: NeuAL-NTK-D updates the network parameters with gradient descent and queries the label based on the uncertainty estimation.

Original: However, its upper confidence bound is still based on the confidence ellipsoid.
Revised: However, its upper confidence bound is still based on the confidence ellipsoid.

Original: Instead, I-NeurAL leverages the representation power of neural networks for both exploitation and exploration.
Revised: Instead, I-NeurAL leverages the representation power of neural networks for both exploitation and exploration.

Original: ALPS maintains a class of pre-trained hypotheses and tries to make the best decisions based on these hypotheses.
Revised: ALPS maintains a class of pre-trained hypotheses and tries to make the best decisions based on these hypotheses.

Original: Nevertheless, the model parameters are fixed before the online active learning process.
Revised: Nevertheless, the model parameters are fixed before the online active learning process.

Original: Hence, ALPS is not able to take the new knowledge obtained by queries into account and its performance is highly restricted by the hypothesis class.
Revised: Hence, ALPS is not able to take the new knowledge obtained by queries into account and its performance is highly restricted by the hypothesis class.

Original: Although Margin algorithm is simple and straightforward, it exhibits great empirical performance in practice.
Revised: Although Margin algorithm is simple and straightforward, it exhibits great empirical performance in practice.

Original: This observation is consistent with other studies [52] [23].
Revised: This observation is consistent with other studies [52] [23].

Original: However, Margin algorithm does not incorporate the exploitation portion and the query criterion is not adaptive to difference instances, thus still outperformed by I-NeurAL.
Revised: However, Margin algorithm does not incorporate the exploitation portion and the query criterion is not adaptive to difference instances, thus still outperformed by I-NeurAL.

Original: Phishing
Revised: Phishing

Original: [Table]
Revised: [Table]

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we introduce two regret metrics and propose a novel neural-based algorithm (I-NeurAL) tailored for the streaming setting of non-parametric active learning.
Revised: In this paper, we introduce two regret metrics and propose a novel neural-based algorithm (I-NeurAL) tailored for the streaming setting of non-parametric active learning.

Original: We carefully design its exploration strategy, query decision-maker, update rules, and training procedure, which lead to both the theoretical and empirical improvement compared to SOTA [50].
Revised: We carefully design its exploration strategy, query decision-maker, update rules, and training procedure, which lead to both the theoretical and empirical improvement compared to SOTA [50].

Original: In the regret analysis, we remove the dependence on either the input dimensionality or effective dimensionality, and improve over [50] by a multiplicative factor of O (log T ) .
Revised: In the regret analysis, we remove the dependence on either the input dimensionality or effective dimensionality, and improve over [50] by a multiplicative factor of O (log T ) .

Original: On the other hand, we empirically show that I-NeurAL consistently achieves better accuracy under the same query budget than the strong baselines including the SOTA work [50] and [23].
Revised: On the other hand, we empirically show that I-NeurAL consistently achieves better accuracy under the same query budget than the strong baselines including the SOTA work [50] and [23].

Original: Acknowledgements
Revised: Acknowledgements

Original: This work is supported by NSF (IIS-1947203, IIS-2117902, IIS-2137468, IIS-2002540, DMS2134079, IIS-2131335, OAC-2130835, and DBI-2021898), DARPA (HR001121C0165), ARO (W911NF2110088), and C3.ai.
Revised: This work is supported by NSF (IIS-1947203, IIS-2117902, IIS-2137468, IIS-2002540, DMS2134079, IIS-2131335, OAC-2130835, and DBI-2021898), DARPA (HR001121C0165), ARO (W911NF2110088), and C3.ai.

Original: The views and conclusions are those of the authors and should not be interpreted as representing the official policies of the funding agencies or the government.
Revised: The views and conclusions are those of the authors and should not be interpreted as representing the official policies of the funding agencies or the government.

Original:  Proofs of Theorem 4.1 and 4. 7.1 Proof of Theorem 4. 
Revised:  Proofs of Theorem 4.1 and 4. 7.1 Proof of Theorem 4.

Original: Proof.
Revised: Proof.

Original: Let f ( x ; θ ) =
Revised: Let f ( x ; θ ) =

Original: f 1 ( x ; θ 1 ) + f 2 ( ϕ ( x ); θ 2 ) and we use E x t ,y t to denote E ( x t ,y t ) ∼D for brevity.
Revised: f 1 ( x ; θ 1 ) + f 2 ( ϕ ( x ); θ 2 ) and we use E x t ,y t to denote E ( x t ,y t ) ∼D for brevity.

Original: For any t ∈ [ T ] ∧ ( I t = 1) , we have
Revised: For any t ∈ [ T ] ∧ ( I t = 1) , we have

Original: [Equation]
Revised: [Equation]

Original: where E 1 is based on the fact r t,i ∈ [0 , 1] , ∀ i ∈
Revised: where E 1 is based on the fact r t,i ∈ [0 , 1] , ∀ i ∈

Original: [ k ] , E 2 is due to f ( x t,i ∗ ; θ t − 1 ) ≤ f ( x t,i t ; θ t − 1 ) according to our selection criterion, and θ ∗ t − 1 in E 3 are intermediate parameters to bound errors.
Revised: [ k ] , E 2 is due to f ( x t,i ∗ ; θ t − 1 ) ≤ f ( x t,i t ; θ t − 1 ) according to our selection criterion, and θ ∗ t − 1 in E 3 are intermediate parameters to bound errors.

Original: For any t ∈ [ T ] ∧ ( I t = 0) , we have R t | ( I t = 0) =
Revised: For any t ∈ [ T ] ∧ ( I t = 0) , we have R t | ( I t = 0) =

Original: E x t ,y t
Revised: E x t ,y t

Original: [ L ( y t, (cid:98) i , y t
Revised: [ L ( y t, (cid:98) i , y t

Original: [Equation]
Revised: [Equation]

Original: y t ) − L ( y t,i ∗ , y t,y t )]
Revised: y t ) − L ( y t,i ∗ , y t,y t )]

Original: = 

Revised: =

Original: based on Lemma 7.5.
Revised: based on Lemma 7.5.

Original: Therefore, for any t ∈ [ T ] , we have
Revised: Therefore, for any t ∈ [ T ] , we have

Original: [Equation]
Revised: [Equation]

Original: Based on Lemma 7.6, Lemma 7.7, and Lemma 7.14, with probability at least 1 − δ , we have
Revised: Based on Lemma 7.6, Lemma 7.7, and Lemma 7.14, with probability at least 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original: t (cid:33)
Revised: t (cid:33)

Original: [Equation]
Revised: [Equation]

Original: (cid:33) , (7.3) where ξ 1 = O (cid:16) νL √ m (cid:17) + O (cid:16) L 2 √ log mν 4 / 3 m 1 /(cid:17) .
Revised: (cid:33) , (7.3) where ξ 1 = O (cid:16) νL √ m (cid:17) + O (cid:16) L 2 √ log mν 4 / 3 m 1 /(cid:17) .

Original: Applying the union bound over all the rounds, with probability at least 1 − δ , we have ∀ t ∈
Revised: Applying the union bound over all the rounds, with probability at least 1 − δ , we have ∀ t ∈

Original: [ T ] , R t ≤(cid:32)
Revised: [ T ] , R t ≤(cid:32)

Original: O (cid:18)
Revised: O (cid:18)

Original: [Equation]
Revised: [Equation]

Original: Lν +
Revised: Lν +

Original: [Equation]
Revised: [Equation]

Original: √ 2 t (cid:19)
Revised: √ 2 t (cid:19)

Original: + O (cid:32)(cid:114)
Revised: + O (cid:32)(cid:114)

Original: [Equation]
Revised: [Equation]

Original: log( O ( tk ) /δ ) t ) (cid:33) + ξ 1 .
Revised: log( O ( tk ) /δ ) t ) (cid:33) + ξ 1 .

Original: (cid:33) (7.4)
Revised: (cid:33) (7.4)

Original: When m is large enough, we have ξ 

Revised: When m is large enough, we have ξ

Original: =
Revised: =

Original: O
Revised: O

Original: ( 1 √ T ) .
Revised: ( 1 √ T ) .

Original: Therefore, in round T , we have
Revised: Therefore, in round T , we have

Original: [Equation]
Revised: [Equation]

Original: Finally, the regret of T rounds is
Revised: Finally, the regret of T rounds is

Original: [Equation]
Revised: [Equation]

Original: where I 1 is due to (cid:80)
Revised: where I 1 is due to (cid:80)

Original: Tt =1 1 √ t ≤ (cid:82) T 1 1 √ t dx + 1 = 2 √ T − 1 and I 2 is because of the choice of m .
Revised: Tt =1 1 √ t ≤ (cid:82) T 1 1 √ t dx + 1 = 2 √ T − 1 and I 2 is because of the choice of m .

Original: The proof is complete.
Revised: The proof is complete.

Original: 7.2 Proof of Theorem 4. 
Revised: 7.2 Proof of Theorem 4.

Original: Proof.
Revised: Proof.

Original: Given ¯ T = 48( γ +1) 2 ϵ
Revised: Given ¯ T = 48( γ +1) 2 ϵ

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: , suppose T > ¯ T , we have
Revised: , suppose T > ¯ T , we have

Original: [Equation]
Revised: [Equation]

Original: For I 1 , based on Eq.(7.4), for any µ ∈ (0 , 1) , t ∈ [ ¯ T ] , we have
Revised: For I 1 , based on Eq.(7.4), for any µ ∈ (0 , 1) , t ∈ [ ¯ T ] , we have

Original: [Equation]
Revised: [Equation]

Original: where E 1 is because of (cid:80)
Revised: where E 1 is because of (cid:80)

Original: Tt =1 1 √ t ≤ (cid:82) T 1 1 √ t dx + 1 = 2 √ T − 1 and the choice of m .
Revised: Tt =1 1 √ t ≤ (cid:82) T 1 1 √ t dx + 1 = 2 √ T − 1 and the choice of m .

Original: It is straight forward to show that R T also satisfies this upper bound when T ≤ ¯ T .
Revised: It is straight forward to show that R T also satisfies this upper bound when T ≤ ¯ T .

Original: For I 2 , based on the Lemma 7.1, we have E x t ∼D
Revised: For I 2 , based on the Lemma 7.1, we have E x t ∼D

Original: X
Revised: X

Original: [ h ( x t, (cid:98) i ) − h ( x t,i ∗ ))]
Revised: [ h ( x t, (cid:98) i ) − h ( x t,i ∗ ))]

Original: = 0 when t ≥ ¯ T .
Revised: = 0 when t ≥ ¯ T .

Original: This implies
Revised: This implies

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have I 2 = 0 .
Revised: Therefore, we have I 2 = 0 .

Original: Putting them together, we have
Revised: Putting them together, we have

Original: [Equation]
Revised: [Equation]

Original: According to Eq.(7.4) and Eq.(7.9), we have
Revised: According to Eq.(7.4) and Eq.(7.9), we have

Original: [Equation]
Revised: [Equation]

Original: else
Revised: else

Original: [Equation]
Revised: [Equation]

Original: Then, replace ¯ T and the proof is complete.
Revised: Then, replace ¯ T and the proof is complete.

Original: 7. Main Lemmas
Revised: 7. Main Lemmas

Original: Lemma 7.1.
Revised: Lemma 7.1.

Original: For any δ ∈ (0 , 1) , γ ≥ 1 , suppose T ≥ ¯ T .
Revised: For any δ ∈ (0 , 1) , γ ≥ 1 , suppose T ≥ ¯ T .

Original: Then, with probability at least 1 − δ , these exist constants C 1 , C 2 , such that the following two event E 1 , E 2 happens
Revised: Then, with probability at least 1 − δ , these exist constants C 1 , C 2 , such that the following two event E 1 , E 2 happens

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: According to Lemma 7.3 and Jensen’s inequality
Revised: According to Lemma 7.3 and Jensen’s inequality

Original: [Equation]
Revised: [Equation]

Original: for any i ∈ [ k ]
Revised: for any i ∈ [ k ]

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − δ , we have
Revised: with probability at least 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original: In round t , define the event
Revised: In round t , define the event

Original: [Equation]
Revised: [Equation]

Original: Then, applying the union bound over t and k , then, with probability at least 1 − δ , E happens, where
Revised: Then, applying the union bound over t and k , then, with probability at least 1 − δ , E happens, where

Original: [Equation]
Revised: [Equation]

Original: where we merge ξ τ into O (cid:16) 3 L √ 2 τ (cid:17) as a result of choice of m .
Revised: where we merge ξ τ into O (cid:16) 3 L √ 2 τ (cid:17) as a result of choice of m .

Original: Next, define the event
Revised: Next, define the event

Original: [Equation]
Revised: [Equation]

Original: When (cid:98) E 0 happens with probability at least 1 − δ , based on the fact h ( · ) ∈ [0 , 1] , we have 
Revised: When (cid:98) E 0 happens with probability at least 1 − δ , based on the fact h ( · ) ∈ [0 , 1] , we have 

Original: [Equation]
Revised: [Equation]

Original: Then, based on Lemma 7.2, with probability at least 1 − δ , when t > ¯ T , 2( γ + 1) β t ≤ ϵ ⇒ β t < 1 .
Revised: Then, based on Lemma 7.2, with probability at least 1 − δ , when t > ¯ T , 2( γ + 1) β t ≤ ϵ ⇒ β t < 1 .

Original: This implies
Revised: This implies

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: Suppose (cid:98) E 1 happens, we have
Revised: Suppose (cid:98) E 1 happens, we have

Original: [Equation]
Revised: [Equation]

Original: Then, based on Lemma 7.2, when t > ¯ T , 2( γ + 1) β t ≤ ϵ .
Revised: Then, based on Lemma 7.2, when t > ¯ T , 2( γ + 1) β t ≤ ϵ .

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: This contradicts Assumption 4.2, i.e., h ( x t,i ∗ ) − h ( x t,i ◦ ) ≥ ϵ .
Revised: This contradicts Assumption 4.2, i.e., h ( x t,i ∗ ) − h ( x t,i ◦ ) ≥ ϵ .

Original: Hence, (cid:98) E 1 will not happen.
Revised: Hence, (cid:98) E 1 will not happen.

Original: Accordingly, with probability at least 1 − δ , the following event will happen
Revised: Accordingly, with probability at least 1 − δ , the following event will happen

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have E [ f ( x t,i ∗ ; θ t − 1 )] > E [ f ( x t,i ◦ ; θ t − 1 )] .
Revised: Therefore, we have E [ f ( x t,i ∗ ; θ t − 1 )] > E [ f ( x t,i ◦ ; θ t − 1 )] .

Original: Recall that i ∗ = arg max i ∈
Revised: Recall that i ∗ = arg max i ∈

Original: [ k ] h ( x t,i ) and (cid:98) i =
Revised: [ k ] h ( x t,i ) and (cid:98) i =

Original: arg max i ∈
Revised: arg max i ∈

Original: [ k ] f ( x t,i ; θ t − 1 ) .
Revised: [ k ] f ( x t,i ; θ t − 1 ) .

Original: As ∀ i ∈ ([ k ] \ {
Revised: As ∀ i ∈ ([ k ] \ {

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Based on the definition of (cid:98) i , we have
Revised: Based on the definition of (cid:98) i , we have

Original: [Equation]
Revised: [Equation]

Original: This indicates E 2 happens with probability at least 1 − δ .
Revised: This indicates E 2 happens with probability at least 1 − δ .

Original: Therefore, based on (cid:98) E
Revised: Therefore, based on (cid:98) E

Original: [Equation]
Revised: [Equation]

Original: the following inferred event (cid:98) E 3 happens with probability at least 1 − δ :
Revised: the following inferred event (cid:98) E 3 happens with probability at least 1 − δ :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Then, based on Eq. 7.
Revised: Then, based on Eq. 7.

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where E 1 is because (cid:98) E 3 happened with probability at least 1 − δ .
Revised: where E 1 is because (cid:98) E 3 happened with probability at least 1 − δ .

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: Similarly, we can prove that
Revised: Similarly, we can prove that

Original: [Equation]
Revised: [Equation]

Original: Then, based on the definition of x t,i ∗ , we have
Revised: Then, based on the definition of x t,i ∗ , we have

Original: [Equation]
Revised: [Equation]

Original: Thus, the event E 1 happens with probability at least 1 − δ .
Revised: Thus, the event E 1 happens with probability at least 1 − δ .

Original: Lemma 7.2.
Revised: Lemma 7.2.

Original: When t >  T = 48( [Equation] +
Revised: When t > ¯ T = 48( γ +

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: , it has 2( γ + 1) β t ≤ ϵ .
Revised: , it has 2( γ + 1) β t ≤ ϵ .

Original: Proof.
Revised: Proof.

Original: To achieve 2( γ + 1) β t ≤ ϵ , there exist constants C 1 , C 2 , such that
Revised: To achieve 2( γ + 1) β t ≤ ϵ , there exist constants C 1 , C 2 , such that

Original: [Equation]
Revised: [Equation]

Original: By calculations, we have
Revised: By calculations, we have

Original: [Equation]
Revised: [Equation]

Original: Then, based on Lemme 8 in [3] and Lemma 8.1 in [11], we have, when t 
Revised: Then, based on Lemme 8 in [3] and Lemma 8.1 in [11], we have, when t ≥ 48(

Original: [Equation]
Revised: [Equation]

Revised: +

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 2( γ + 1) β t ≤ ϵ
Revised: 2( γ + 1) β t ≤ ϵ

Original: [Equation]
Revised: [Equation]

Original: Lemma 7.3.
Revised: Lemma 7.3.

Original: For any δ ∈ (0 , 1) , ν > 0 , suppose m satisfies the conditions in Theorem 4.1.
Revised: For any δ ∈ (0 , 1) , ν > 0 , suppose m satisfies the conditions in Theorem 4.1.

Original: Then, with probability at least 1 − δ , given any fixed index i ∈ [ k ] , it holds that
Revised: Then, with probability at least 1 − δ , given any fixed index i ∈ [ k ] , it holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Given any i ∈ [ k ] , let i be a fixed index, i.e., suppose there exist a policy Ω i which always select the i -th context ( x t,i , r 1 t,i ) for every round t ∈ [ T ] .
Revised: Given any i ∈ [ k ] , let i be a fixed index, i.e., suppose there exist a policy Ω i which always select the i -th context ( x t,i , r 1 t,i ) for every round t ∈ [ T ] .

Original: Then, in round t , we have the collected data by Ω i : H 1 ,it − 1 = { x τ,i , r 1 τ,i } t − 1 τ =1 .
Revised: Then, in round t , we have the collected data by Ω i : H 1 ,it − 1 = { x τ,i , r 1 τ,i } t − 1 τ =1 .

Original: Then, let θ 1 ,it − 1 , θ 2 ,it − 1 represent the parameters trained only on
Revised: Then, let θ 1 ,it − 1 , θ 2 ,it − 1 represent the parameters trained only on

Original: H it − 1 using Algorithm 2, satisfying ∥ θ 1 ,it − 1 − θ 10 ∥ 2 ≤ O ( ν √ m ) and ∥ θ 2 ,it − 1 − θ 20 ∥ 2 ≤ O ( ν √ m ) .
Revised: H it − 1 using Algorithm 2, satisfying ∥ θ 1 ,it − 1 − θ 10 ∥ 2 ≤ O ( ν √ m ) and ∥ θ 2 ,it − 1 − θ 20 ∥ 2 ≤ O ( ν √ m ) .

Original: Note that θ 1 ,it − 1 , θ 2 ,it − 1 are uniformly drawn from { (cid:98) θ 1 ,i τ − 1 , (cid:98) θ 2 ,i τ − 1 } t − 1 τ =0 and these parameters are unknown but introduced for the sake of analysis.
Revised: Note that θ 1 ,it − 1 , θ 2 ,it − 1 are uniformly drawn from { (cid:98) θ 1 ,i τ − 1 , (cid:98) θ 2 ,i τ − 1 } t − 1 τ =0 and these parameters are unknown but introduced for the sake of analysis.

Original: Then, for τ ∈ [ t ] , we define
Revised: Then, for τ ∈ [ t ] , we define

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Then, we have
Revised: Then, we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where F τ − 1 denotes the σ -algebra generated by the history H iτ − 1 .
Revised: where F τ − 1 denotes the σ -algebra generated by the history H iτ − 1 .

Original: Therefore, { V τ } tτ =1 are the martingale difference sequence.
Revised: Therefore, { V τ } tτ =1 are the martingale difference sequence.

Original: Then, using the similar proof method in Lemma 7.6, we have
Revised: Then, using the similar proof method in Lemma 7.6, we have

Original: [Equation]
Revised: [Equation]

Original: Let f ( x ; θ t − 1 ) =
Revised: Let f ( x ; θ t − 1 ) =

Original: f 1 ( x ; θ 1 t − 1 ) + f 2 ( ϕ ( x ); θ 2 t − 1 ) and f ( x ; θ it − 1 ) =
Revised: f 1 ( x ; θ 1 t − 1 ) + f [Equation] ϕ [Equation]

Original: f 1 ( x ; θ 1 ,it − 1 ) + f2 ( ϕ ( x ); θ 2 ,it − 1 ) .
Revised: x ); θ 2 t − 1 ) and f ( x ; θ it − 1 ) = f [Equation] x ; θ 1 ,

Original: Then, given i ∈ [ k ]
Revised: i [Equation] − [Equation]

Revised: +

Revised: [Equation]

Revised: ϕ

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: Then, given i ∈ [ k ]

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is the application of Lemma 7.14 and Eq.
Revised: where the last inequality is the application of Lemma 7.14 and Eq.

Original: The proof is complete.
Revised: The proof is complete.

Original: Lemma 7.4 (Label Complexity Analysis) .
Revised: Lemma 7.4 (Label Complexity Analysis) .

Original: For any δ ∈ (0 , 1) , γ ≥ 1 , suppose m satisfies the conditions in Theorem 4.1.
Revised: For any δ ∈ (0 , 1) , γ ≥ 1 , suppose m satisfies the conditions in Theorem 4.1.

Original: Then, with probability at least 1 − δ , we have
Revised: Then, with probability at least 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Recall that
Revised: Recall that

Original: [Equation]
Revised: [Equation]

Original: and 
Revised: and x

Original: [Equation]
Revised: [Equation]

Original: = 
Revised: = max [Equation] x [Equation]

Revised: i

Original: [Equation]
Revised: [Equation]

Original: θ 
Revised: θ t

Original: [Equation]
Revised: [Equation]

Original: With probability at least 1 − δ , according to Eq.
Revised: With probability at least 1 − δ , according to Eq.

Original: (7.16) the event (cid:98) E 0 = (cid:26) τ ∈
Revised: (7.16) the event (cid:98) E 0 = (cid:26) τ ∈

Original: [ t ] , i ∈ [ k ] ,
Revised: [ t ] , i ∈ [ k ] ,

Original: [Equation]
Revised: [Equation]

Original: (cid:27) happens.
Revised: (cid:27) happens.

Original: Therefore, we have  
Revised: Therefore, we have  

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Then, we have   
Revised: Then, we have  

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]

Original: Let ϵ t = |
Revised: Let ϵ t = |

Original: E x t ∼D X
Revised: E x t ∼D X

Original: [ h ( x t, (cid:98) i )] −
Revised: [ h ( x t, (cid:98) i )] −

Original: E x t ∼D X
Revised: E x t ∼D X

Original: [ h ( x t,i ◦ )] | .
Revised: [ h ( x t,i ◦ )] | .

Original: Then, based on Lemma 7.2, when t ≥ ¯ Tt , we have
Revised: Then, based on Lemma 7.2, when t ≥ ¯ T , we have

Original: [Equation]
Revised: [Equation]

Original: 44) where ¯

Original: [Equation]

Original: [Equation]

Original: For any t ∈ [ T ] and t < ¯ Tt , we have
Revised: For any t ∈ [ T ] and t < ¯ T , we have

Original: E x t ∼D X

Original: [ I t ] ≤ 1 .

Original: For the round t > ¯ T t , suppose

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: ∼D X [ h
Revised: [ I t ] ≤ 1 .

Original: [Equation] x t,
Revised: For the round t > ¯ T , suppose

Original: [Equation]
Revised: [Equation]

Original: ] −

Original: E x t ∼D X
Revised: E x t ∼D X

Original: [ h ( x t,i ◦ )]
Revised: [ h ( x t,i ◦ )]

Original: = − ϵ t , then, we have
Revised: = − ϵ t , then, we have

Revised: [Equation]

Revised: ◦

Original: [Equation]
Revised: [Equation]

Original: where E 1 is because of Eq.
Revised: where E 1 is because of Eq.

Original: (7.44) since γ ≥ 1 .
Revised: (7.44) since γ ≥ 1 .

Original: This contradicts the fact
Revised: This contradicts the fact

Original: [Equation]
Revised: [Equation]

Revised: X [

Revised: [Equation]

Original: f ( x t,i ◦ ; θ t − 1 )] ≥ 0 .
Revised: f ( x t,i ◦ ; θ t − 1 )] ≥ 0 .

Original: Therefore,
Revised: Therefore,

Original: E
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: ∼D

Original: [Equation] h [Equation] ] =
Revised: =

Original: ϵ t .
Revised: ϵ t .

Original: Then, based on Eq.(7.43), we have
Revised: Then, based on Eq.(7.43), we have

Original: [Equation]
Revised: [Equation]

Original: where E 2 is because of Eq.
Revised: where E 2 is because of Eq.

Original: According to Lemma 7.1, when t > ¯ T ,
Revised: According to Lemma 7.1, when t > ¯ T ,

Original: [Equation]
Revised: [Equation]

Original: Then, applying Eq.
Revised: Then, applying Eq.

Original: (7.47), for the round t > ¯ T , we have
Revised: (7.46), for the round t > ¯ T , we have

Original: [Equation]
Revised: [Equation]

Original: Then, assume T > ¯ T , we have
Revised: Then, assume T > ¯ T , we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have N T ≤ ¯ T .
Revised: Therefore, we have N T ≤ ¯ T .

Original: Lemma 7.5.
Revised: Lemma 7.5.

Original: For any δ ∈ (0 , 1) , γ ≥ 1 , suppose m satisfies the conditions in Theorem 4.1.
Revised: For any δ ∈ (0 , 1) , γ ≥ 1 , suppose m satisfies the conditions in Theorem 4.1.

Original: Then, with probability at least 1 − δ , when I t = 0 , we have
Revised: Then, with probability at least 1 − δ , when I t = 0 , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: As I t = 0 , we have
Revised: As I t = 0 , we have

Original: [Equation]
Revised: [Equation]

Original: When (cid:98) E 0 (Eq.
Revised: When (cid:98) E 0 (Eq.

Original: (7.16)) happens with probability at least 1 − δ , based on the fact h ( · ) ∈ [0 , 1] [Equation] we have  
Revised: (7.16)) happens with probability at least 1 − δ , based on the fact h ( · ) ∈ [0 , 1] , we have  

Original: [Equation]
Revised: [Equation]

Original: x

Original: [Equation]

Original: [Equation]

Original: Then, with probability at least 1 − δ ,we have
Revised: Then, with probability at least 1 − δ ,we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is because of γ ≥ 1 .
Revised: where the last inequality is because of γ ≥ 1 .

Original: Then, similarly, for any i ′ ∈ ([ k ] \ { (cid:98) i, i ◦ } ) , we have E x t
Revised: Then, similarly, for any i ′ ∈ ([ k ] \ { (cid:98) i, i ◦ } ) , we have E x t

Original: ∼D
Revised: ∼D

Original: X
Revised: X

Original: [ h ( x t, (cid:98) i ) − h ( x t,i ′ )] ≥ 0 .
Revised: [ h ( x t, (cid:98) i ) − h ( x t,i ′ )] ≥ 0 .

Original: Thus, based on the definition of h ( x t,i ∗ ) , we have
Revised: Thus, based on the definition of h ( x t,i ∗ ) , we have

Original: [Equation]
Revised: [Equation]

Original:  x t,
Revised: E x t ∼D X [ h ( x t,i ∗ )] . Because

Original: [Equation]
Revised: [Equation]

Original: i

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: E x t ∼D X [ h ( x t,i ∗ )] . Because
Revised: y t,

Original: [Equation]
Revised: [Equation]

Revised: t

Revised: [Equation]

Original:  we have
Revised: ] , we have

Original: [Equation]
Revised: [Equation]

Original: The proof is complete.
Revised: The proof is complete.

Original: Lemma 7.6.
Revised: Lemma 7.6.

Original: For any δ ∈ (0 , 1) , ν > 0 , suppose m satisfies the conditions in Theorem 4.1.
Revised: For any δ ∈ (0 , 1) , ν > 0 , suppose m satisfies the conditions in Theorem 4.1.

Original: In round t ∈ [ T ] , given ( x t , y t ) ∼ D , let
Revised: In round t ∈ [ T ] , given ( x t , y t ) ∼ D , let

Original: [Equation]
Revised: [Equation]

Original: Then, with probability at least 1 − δ , we have
Revised: Then, with probability at least 1 − δ [Equation] we have

Original: [Equation]
Revised: [Equation]

Original: 7.51) where H 1 t − 1 = { x τ, (cid:98) i , r 1 τ, (cid:98) i } t − 1 τ =1 is historical data and the expectation is taken over ( θ 1 t − 1 , θ 2 t − 1 ) .
Revised: where H 1 t − 1 = { x τ, [Equation] i , r 1 τ, (cid:98) i } t − 1 τ =1 is historical data and the expectation is taken over ( θ 1 t − 1 , θ 2 t − 1 ) .

Original: Proof.
Revised: Proof.

Original: This lemma is inspired by Lemma 5.1 in [14].
Revised: This lemma is inspired by Lemma 5.1 in [14].

Original: For any round τ ∈ [ t ] , define
Revised: For any round τ ∈ [ t ] , define

Original: [Equation]
Revised: [Equation]

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: where F τ − 1 denotes the σ -algebra generated by the history H 1 τ − 1 .
Revised: where F τ − 1 denotes the σ -algebra generated by the history H 1 τ − 1 .

Original: Therefore, { V τ } tτ =1 are the martingale difference sequence.
Revised: Therefore, { V τ } tτ =1 are the martingale difference sequence.

Original: Then, applying the Hoeffding-Azuma inequality [Equation] with probability at least 1 − δ [Equation]
Revised: Then, applying the Hoeffding-Azuma inequality, with probability at least 1 − δ , we have

Original: we have

Original: [Equation]
Revised: [Equation]

Original: As I 1 is equal to
Revised: As I 1 is equal to

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: + f 2 ( ϕ ( x τ, (cid:98) i ); (cid:98) θ 2 τ − 1 ) − r 1 τ, (cid:98) i | , 1 } (cid:105) ≤ min {| f 1 ( x τ, (cid:98) i ; (cid:98) θ 1 τ − 1 ) + f 2 ( ϕ ( x τ, (cid:98) i ); (cid:98) θ 2 τ − 1 ) − r 1 τ, (cid:98) i | , 1 } + (cid:114) 2 log(1 /δ )

Original: t ≤t t (cid:88) τ

Original: = (cid:12)(cid:12)(cid:12) f

Original: [Equation]

Original: (cid:16) x τ, (cid:98) i ; (cid:98) θ 2 τ − 1 (cid:17) − (cid:16) r 1 τ, (cid:98) i − f 1 ( x τ, (cid:98) i ; (cid:98) θ 1 τ − 1 ) (cid:17)(cid:12)(cid:12)(cid:12) + (cid:114) 2 log(1 /δ ) t .

Original: Based on the the definition of θ 1 t − 1 , θ 2 t − 1 , we have
Revised: Based on the the definition of θ 1 t − 1 , θ 2 t − 1 , we have

Original: [Equation]
Revised: [Equation]

Original: ; θ [Equation] + [Equation]
Revised: Therefore

Original: 1 ) − r

Original: [Equation]
Revised: [Equation]

Original: (cid:98) i | , 1 } (cid:105) = t t (cid:88) τ = E [Equation]
Revised: putting them together

Original: ∼D

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: x

Original: [Equation]
Revised: [Equation]

Original: (cid:98) [Equation] (cid:98) θ 2 τ − 1 ) − r 1 τ, (cid:98) i (cid:12)(cid:12)(cid:12) , 1 } (cid:105) . Therefore, putting them together, we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: For I 2 , based on Lemma 7. [Equation] we have
Revised: For I 2 , based on Lemma 7.8, we have

Original: [Equation]
Revised: [Equation]

Original: where I 3 is based onL ( (cid:101) θ [Equation] [Equation] 2 ≤ µ , the definition of µ .
Revised: where I 3 is based on the assumption of µ .

Original: Combining above Eq.
Revised: Combining above Eq.

Original: (7.57) and (7.58) together, with probability at least 1 − δ [Equation] we have
Revised: (7.56) and (7.57) together, with probability at least 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original: .59) where we apply union bound over δ to make above events occur concurrently.
Revised: ) where we apply union bound over δ to make above events occur concurrently.

Original: Then, based on Lemma 7.13 (2), it is sufficient to show that θ 1 t − 1 , θ 2 t − 1 are close to initialization for any t ∈
Revised: Then, based on Lemma 7.13 (2), it is sufficient to show that θ 1 t − 1 , θ 2 t − 1 are close to initialization for any t ∈

Original: [ T ] .
Revised: [ T ] .

Original: The proof is complete.
Revised: The proof is complete.

Original: Lemma 7.7.
Revised: Lemma 7.7.

Original: In round t ∈ [ T ] , given ( x t , y t ) ∼ D , let i ∗
Revised: In round t ∈ [ T ] , given ( x t , y t ) ∼ D , let i ∗

Original: = arg max i ∈
Revised: = arg max i ∈

Original: [ k ] h ( x τ,i ) .
Revised: [ k ] h ( x t,i ) .

Original: Let θ 1 , ∗ t − 1 , θ 2 , ∗ t − 1 are the parameters trained on H ∗ t − 1 using Algorithm 2.
Revised: Let θ 1 , ∗ t − 1 , θ 2 , ∗ t − 1 are the parameters trained on H ∗ t − 1 using Algorithm 2.

Original: For any ν > 0 , suppose inf
Revised: For any ν > 0 , suppose inf

Original: [Equation]
Revised: [Equation]

Revised: −

Revised: [Equation]

Original:  ≤ µ .
Revised: 2 ≤ µ .

Original: Then, with probability at least 1 − δ , we have
Revised: Then, with probability at least 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original: where H ∗ t − 1 = { x τ,i ∗ , r 1 τ,i ∗ } t − 1 τ =1 is optimal data of past rounds the expectation is taken over θ 1 , ∗ t − 1 , θ 2 , ∗ t − 1 .
Revised: where H ∗ t − 1 = { x τ,i ∗ , r 1 τ,i ∗ } t − 1 τ =1 is optimal data of past rounds the expectation is taken over θ 1 , ∗ t − 1 , θ 2 , ∗ t − 1 .

Original: Proof.
Revised: Proof.

Original: This lemma is a direct corollary of Lemma 7.6.
Revised: This lemma is a direct corollary of Lemma 7.6.

Original: For any τ ∈ [ t ] , define
Revised: For any τ ∈ [ t ] [Equation] define

Original: [Equation]
Revised: [Equation]

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: where F τ − 1 denotes the σ -algebra generated by the history H τ − 1 .
Revised: where F τ − 1 denotes the σ -algebra generated by the history H τ − 1 .

Original: Therefore, { V τ } tτ =1 are the martingale difference sequence.
Revised: Therefore, { V τ } tτ =1 are the martingale difference sequence.

Original: Then, applying the Hoeffding-Azuma inequality [Equation] with probability at least 1 − δ [Equation]
Revised: Then, applying the Hoeffding-Azuma inequality, with probability at least 1 − δ , we have

Original: we have

Original: [Equation]
Revised: [Equation]

Original: As I 1 is equal to
Revised: As I 1 is equal to

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: For I 2 , applying Lemma 7.
Revised: For I 2 , applying Lemma 7.8, for any (cid:101) θ 2 , ∗ satisfying ∥ (cid:101) θ 2 ,

Original: [Equation]
Revised: [Equation]

Original: for any (cid:101) θ 2 , ∗ satisfying ∥ (cid:101) θ 2 , ∗ − θ 20 ∥ 2 ≤ O (ν √ m )
Revised: − [Equation] ∥ 2 ≤ O (

Original: [Equation]
Revised: [Equation]

Original:  with probability at least 1 − 3 δ , we have
Revised: √ m ) , with probability at least 1 − 3 δ , we have

Original: I
Revised: I

Original: [Equation]
Revised: [Equation]

Original: where I 3 is because of the assumption of µ .
Revised: where I 3 is because of the assumption of µ .

Original: Combining above inequalities together, as µ ∈ (0 , 1] , with probability at least 1 − δ , we have
Revised: Combining above inequalities together, as µ ∈ (0 , 1] , with probability at least 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original: where we apply union bound over δ to make above events occur concurrently.
Revised: where we apply union bound over δ to make above events occur concurrently.

Original: Then, based on Lemma 7.13 (2), it is sufficient to show that θ 1 , ∗ t − 1 , θ 2 , ∗ t − 1 are close to initialization for any t ∈
Revised: Then, based on Lemma 7.13 (2), it is sufficient to show that θ 1 , ∗ t − 1 , θ 2 , ∗ t − 1 are close to initialization for any t ∈

Original: [ T ] .
Revised: [ T ] .

Original: Lemma 7.8.
Revised: Lemma 7.8.

Original: For any δ ∈ (0 , 1) , suppose m satisfies the condition in Theorem 4.1.
Revised: For any δ ∈ (0 , 1) , suppose m satisfies the condition in Theorem 4.1.

Original: Then, with probability at least 1 − δ , setting η 2 = ν ′ ν m √ t for algorithm 1, for ν > 0 and any (cid:101) θ 2 satisfying ∥ (cid:101) θ 2 − θ 20 ∥ 2 ≤ O ( ν √ m ) , and i ∈ [ k ] , there exists a small enough constant v ′ , such that 
Revised: Then, with probability at least 1 − δ , setting η 2 = ν ′ ν m √ t for algorithm 1, for ν > 0 and any (cid:101) θ 2 satisfying ∥ (cid:101) θ 2 − θ 20 ∥ 2 ≤ O ( ν √ m ) , and i ∈ [ k ] , there exists a small enough constant v ′ , such that [Equation] ≤

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: This is a direct application of Lemma 7.9 by setting ˆ ϵ = Lν √ 2 ν ′ t , and, where ν ′ is some small enough absolute constant.
Revised: This is a direct application of Lemma 7.9 by setting ˆ ϵ = Lν √ 2 ν ′ t , and, where ν ′ is some small enough absolute constant.

Original: We set L τ 
Revised: We set L τ (

Original: [Equation]
Revised: [Equation]

Original:  Then, for any (cid:101) θ 2 satisfying ∥ (cid:101) θ 2 − θ 20 ∥ 2 ≤ O ( ν √ m ) , there exist a small enough absolute constant ν ′ , such that
Revised: θ [Equation] (cid:17)(cid:12)(cid:12)(cid:12) . Then, for any (cid:101) θ 2 satisfying ∥ (cid:101) θ 2 − θ 20 ∥ 2 ≤ O ( ν √ m ) , there exist a small enough absolute constant ν ′ , such that

Original: [Equation]
Revised: [Equation]

Original: Then, replacing ˆ ϵ completes the proof.
Revised: Then, replacing ˆ ϵ completes the proof.

Original: Lemma 7.9.
Revised: Lemma 7.9.

Original: With probability at least 1 − δ over the randomness of θ 0 , given the convex loss L satisfying L ′ ≤ O (1) , for any ˆ ϵ, ν > 0 and (cid:101) θ satisfying ∥ (cid:98) θ − θ 

Revised: With probability at least 1 − δ over the randomness of θ 0 , given the convex loss L satisfying L ′ ≤ O (1) , for any ˆ ϵ, ν > 0 and (cid:101) θ satisfying ∥ (cid:98) θ − θ

Original: ∥ 2 ≤ O ( ν √ m ) , Algorithm 1 with η = ν ′
Revised: ∥ 2 ≤ O ( ν √ m ) , Algorithm 1 with η = ν ′

Original: ˆ ϵ Lm and t =
Revised: ˆ ϵ Lm and t =

Original: L 2 ν 2 2 ν ′
Revised: L 2 ν 2 2 ν ′

Original: ˆ ϵ 2 for some small enough constant ν ′ has the following bound:
Revised: ˆ ϵ 2 for some small enough constant ν ′ has the following bound:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Define B ( θ 0 , w ) = { θ ∈ R p :
Revised: Define B ( θ 0 , ν ) = { θ ∈ R p :

Original: ∥ θ − θ 

Revised: ∥ θ − θ

Original: ∥ 2 ≤ w } and L ( x ; r ) ( θ ) =
Revised: ∥ 2 ≤ O ( ν/ √ m ) } and L ( x ; r ) ( θ ) =

Original: | r − f ( x ; θ ) | .
Revised: | r − f ( x ; θ ) | .

Original: First, we need to show (cid:98) θ 1 , . . .
Revised: First, we need to show (cid:98) θ 1 , . . .

Original: , (cid:98) θ t also are in B ( θ 0 , w ) , where.
Revised: , (cid:98) θ t also are in B ( θ 0 , w ) , where.

Original: According to Lemma 7.14, when θ ∈ B ( θ 0 , w ) , we have
Revised: According to Lemma 7.14, when θ ∈ B ( θ 0 , w ) , we have

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: The proof follows a simple induction.
Revised: The proof follows a simple induction.

Original: Suppose that θ 0 , (cid:98) θ 1 , . . .
Revised: Suppose that θ 0 , (cid:98) θ 1 , . . .

Original: , (cid:98) θ t ∈ B ( θ 0 , w ) , by triangle inequality, we have
Revised: , (cid:98) θ t ∈ B ( θ 0 , w ) , by triangle inequality, we have

Original: [Equation]
Revised: [Equation]

Original: t − [Equation] (cid:88) τ = [Equation]
Revised: Because η = O ( 1 m )

Original: t

Original: [Equation]
Revised: [Equation]

Original: (cid:88) τ [Equation] (cid:88) ( x{+we have ∥ (cid:98) θ t − θ 
+}
Revised: {+we have ∥ (cid:98) θ t − θ 
+}

Original: [Equation] r ) [Equation]
Revised: ∥ 2 ≤ O ( ν/ √ m ) .

Original: (cid:98)

Original: [Equation] L ( x [Equation] r
Revised: In round τ ∈ [ t ]

Original: [Equation]
Revised: [Equation]

Original: ( Lηt

Original: [-[Equation] Because η = O ( 1 m ) , we have ∥ (cid:98) θ t − θ 
 ∥ 2 ≤ w . In round τ ∈ [ t ] ,-] recall that | (cid:98) H τ | =
Revised: [-[Equation] Because η = O ( 1 m ) , we have ∥ (cid:98) θ t − θ 
 ∥ 2 ≤ w . In round τ ∈ [ t ] ,-] recall that | (cid:98) H τ | =

Original: b and (cid:98) H τ ⊂ H τ .
Revised: b and (cid:98) H τ ⊂ H τ .

Original: Given the context xτ and its reward r , we have the fact
Revised: Given the context x and its reward r [Equation] we have the fact

Original: [Equation]
Revised: [Equation]

Original: where E 1 is because (cid:98) H τ is uniformly drawn from H τ and E 2 is duo to |H
Revised: where E 1 is because (cid:98) H τ is uniformly drawn from H τ and E 2 is duo to |H

Original: τ | = τ .
Revised: τ | = τ .

Original: Then, based on
Revised: Then, based on

Original: Lemma 7.12, for any ϵ >0 , we have
Revised: Lemma 7.12, for any ϵ >

Original: [Equation]
Revised: [Equation]

Original: Based on the fact 2 ⟨ A
Revised: we have

Original: [Equation]
Revised: [Equation]

Original:  B ⟩ = ∥
Revised: Based on the fact 2 ⟨ A , B ⟩ = ∥

Original: A ∥ 22 + ∥ B ∥ 22 − ∥A
Revised: A ∥ 22 + ∥ B ∥ 22 − ∥

Original: [Equation]

Original:  B ∥ [Equation] we have[Equation]
Revised: A , B ∥ 22 , we have

Original: [Equation]
Revised: [Equation]

Original: where E 3 is because of 7.71:
Revised: where E 3 is because of 7.70:

Original: [Equation]
Revised: [Equation]

Original: Therefore, we havet (cid:88)
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Then, for τ ∈ [ t ]
Revised: Then, for τ ∈ [ t ]

Original: [Equation]
Revised: [Equation]

Original: define
Revised: define

Original: V

Original: [Equation]

Original: [Equation]

Original: x ,r

Original: [Equation]
Revised: [Equation]

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: where where F τ − 1 denotes the σ -algebra generated by the history H τ − 1 .
Revised: where where F τ − 1 denotes the σ -algebra generated by the history H τ − 1 .

Original: Therefore, { V 0 , . . 
Revised: Therefore [Equation] { V 0 , . . .

Original: . , V t } is the martingale difference sequence.
Revised: , V t } is the martingale difference sequence.

Original: Then, applying the Hoeffding-Azuma inequality, with probability at least 1 − δ , we have
Revised: Then, applying the Hoeffding-Azuma inequality [Equation] with probability at least 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original: where E 4 be because of 7.75 and E 5 is by placing the parameter choice η = ν
Revised: where E 4 be because of 7.74 and E 5 is by placing the parameter choice η = ν

Original: ˆ ϵ Lm and t =
Revised: ˆ ϵ Lm and t =

Original: L 2 ν 2 2 ν ′
Revised: L 2 ν 2 2 ν ′

Original: ˆ
Revised: ˆ

Original: ϵ .
Revised: ϵ .

Original: The proof is completed
Revised: The proof is completed.

Original: [Equation]

Original: 7.4 Ancillary Lemmas
Revised: 7.4 Ancillary Lemmas

Original: Lemma 7.10 (Theorem 5, [2]) .
Revised: Lemma 7.10 (Theorem 5, [2]) .

Original: For any δ ∈ (0 , 1) , if w satisfies that
Revised: For any δ ∈ (0 , 1) , if w satisfies that

Original: [Equation]
Revised: [Equation]

Original: then, with probability at least 1 − δ [Equation] for all ∥ θ − θ 

Revised: then, with probability at least 1 − δ , for all ∥ θ − θ

Original: ∥ 2 ≤ w , we have
Revised: ∥ 2 ≤ w , we have

Original: [Equation]
Revised: [Equation]

Original: Lemma 7.11 (Lemma 4.1, [17]) .
Revised: Lemma 7.11 (Lemma 4.1, [17]) .

Original: For any δ ∈ (0 , 1) , if w satisfies
Revised: For any δ ∈ (0 , 1) , if w satisfies

Original: [Equation]
Revised: [Equation]

Original: then, with probability at least 1 − δ over randomness of θ 0 , for any t ∈ [ T ] ,
Revised: then, with probability at least 1 − δ over randomness of θ 0 , for any t ∈ [ T ] ,

Original: ∥ x ∥ 2 = 1 , and θ , θ ′ satisfying ∥ θ − θ 

Revised: ∥ x ∥ 2 = 1 , and θ , θ ′ satisfying ∥ θ − θ

Original: ∥ 2 ≤ w and ∥ θ ′ − θ 

Revised: ∥ 2 ≤ w and ∥ θ ′ − θ

Original: ∥ 2 ≤ w , it holds uniformly that
Revised: ∥ 2 ≤ w , it holds uniformly that

Original: [Equation]
Revised: [Equation]

Original: Lemma 7.12 (Lemma 4.2, [17]) .
Revised: Lemma 7.12 (Lemma 4.2, [17]) .

Original: For any δ ∈ (0 , 1) , ˆ ϵ > 0 , if w satisfies
Revised: For any δ ∈ (0 , 1) , ˆ ϵ > 0 , if w satisfies

Original: [Equation]
Revised: [Equation]

Original:  then, with probability at least 1 − δ over randomness of θ (0) , for any ˆ ϵ > 0 , i ∈ [ n ] , and θ , (cid:101) θ satisfying ∥ θ − θ (0) ∥ 2 ≤ w and ∥ (cid:101) θ − θ (0) ∥ 2 ≤ w , it holds uniformly that
Revised: [Equation] then, with probability at least 1 − δ over randomness of θ (0) , for any ˆ ϵ > 0 , i ∈ [ n ] [Equation] and θ , (cid:101) θ satisfying ∥ θ − θ (0) ∥ 2 ≤ w and ∥ (cid:101) θ − θ (0) ∥ 2 ≤ w , it holds uniformly that

Original: [Equation]
Revised: [Equation]

Original: Lemma 7.13.
Revised: Lemma 7.13.

Original: Given a constant 0 <
Revised: Given a constant 0 <

Original: ˆ ϵ < 1 , suppose m satisfies the conditions in Lemma 4.1, the learning rate η = Ω( ρ poly ( t,n,L ) m ) , the number of iterations K = Ω( poly ( t,n,L ) ρ· log ˆ ϵ − 1 ) .
Revised: ˆ ϵ < 1 , suppose m satisfies the conditions in Lemma 4.1, the learning rate η = Ω( ρ poly ( t,n,L ) m ) , the number of iterations K = Ω( poly ( t,n,L ) ρ· log ˆ ϵ − 1 ) .

Original: Then, with probability at least 1 − δ , starting from random initialization θ 0 , (1) (Theorem 1 in [2])
Revised: Then, with probability at least 1 − δ , starting from random initialization θ 0 , (1) (Theorem 1 in [2])

Original: In round t ∈ [ T ] , given the collected data { x τ , r τ } ti = τ , the loss function is defined as: L ( θ ) = 12 (cid:80) tτ
Revised: In round t ∈ [ T ] , given the collected data { x τ , r τ } ti = τ , the loss function is defined as: L ( θ ) = 12 (cid:80) tτ

Original: =

Revised: =

Original: ( f ( x τ ; θ ) − r τ ) 2 .
Revised: ( f ( x τ ; θ ) − r τ ) 2 .

Original: Then, there exists (cid:101) θ satisfying ∥ (cid:101) θ[-− θ 
-]
Revised: Then, there exists (cid:101) θ satisfying ∥ (cid:101) θ[-− θ 
-]

Original:  ∥ 2 ≤ O (cid:16) t3 ρ √ m log m (cid:17) , such that L ( (cid:101) θ ) ≤ ˆ ϵ in K = Ω( poly ( t,n,L ) ρ·
Revised: [Equation] ∥ [Equation] ≤ O (cid:16) t

Original: [Equation]
Revised: [Equation]

Original:  ˆ ϵ − 1 ) iterations; (2) For any t ∈ 
Revised: √ m log m (cid:17) , such that L ( (cid:101) θ ) ≤ ˆ ϵ in K = Ω( poly ( t,n,L ) ρ· log ˆ ϵ − 1 ) iterations; (2) For any t ∈ [ T

Original: [ T ] , it holds uniformly
Revised: , it holds uniformly

Original: that ∥ θ t − 1 − θ 

Revised: that ∥ θ t − 1 − θ

Original: ∥ 2 ≤ O (cid:16) t 3 ρ √ m log m (cid:17) ; (3) (Lemma C.4 in[14])
Revised: ∥ 2 ≤ O (cid:16) t 3 ρ √ m log m (cid:17) ; ( Lemma C. in

Revised: Following

Revised: the

Original: Following the initialization, given ∥ x ∥2 = 1 , it holdsthat
Revised: initialization, given ∥ x ∥ = 1 , it holds

Revised: that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: (2) is a corollary of Theorem 1 in [2].
Revised: (2) is a corollary of Theorem 1 in [2].

Original: Suppose θ τ = θ τ − 1 − ηb
Revised: Suppose θ τ = θ τ − 1 − ηb

Original: (cid:80) ( x ,r ) ∈ (cid:98)
Revised: (cid:80) ( x ,r ) ∈ (cid:98)

Original: H
Revised: H

Original: τ ▽ θ L [( x , r ); θ τ − 1 ] .
Revised: τ ▽ θ L [( x , r ); θ τ − 1 ] .

Original: The proof is based on the following induction.
Revised: The proof is based on the following induction.

Original: Let w = O (cid:16) t 3 δ √ m log m (cid:17) .
Revised: Let w = O (cid:16) t 3 δ √ m log m (cid:17) .

Original: Then, based on the
Revised: Then, based on the

Original: Theorem 1 in [2], we have
Revised: Theorem 1 in [2], we have

Original: [Equation]
Revised: [Equation]

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: Lemma 7.14 (Lemma C.2 [14]) .
Revised: Lemma 7.14 (Lemma C.2 [14]) .

Original: For any δ ∈ (0 , 1) , ν > 0 , suppose m satisfies the conditions in
Revised: For any δ ∈ (0 , 1) , ν > 0 , suppose m satisfies the conditions in

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: Then, with probability at least 1 − δ , in each round t ∈ [ T ] , for any x satisfying ∥ x ∥ 2 = 1 , θ 1 , ∗ t − 1 , θ 1 t − 1 satisfying ∥ θ 1 , ∗ t − 1 − θ 1 t − 1 ∥ 2 ≤ O (cid:16) ν √ m (cid:17) , and θ 2 , ∗ t − 1 , θ 2 t − 1 satisfying ∥ θ 2 , ∗ t − 1 − θ 2 t − 1 ∥ 2 ≤O (cid:16) ν √ m (cid:17) , we have [Equation]
Revised: Then, with probability at least 1 − δ , in each round t ∈ [ T ] , for any x satisfying ∥ x ∥ 2 = 1 , θ 1 , ∗ t − 1 , θ 1 t − 1 satisfying ∥ θ 1 , ∗ t − 1 − θ 1 t − 1 ∥ 2 ≤ O (cid:16) ν √ m (cid:17) , and θ 2 , ∗ t − 1 , θ 2 t − 1 satisfying ∥ θ 2 , ∗ t − 1 − θ 2 t − 1 ∥ 2 ≤

Original: νL √ m (cid:19) + O (cid:18) [Equation] log
Revised: O (cid:16) ν √ m (cid:17) , we have

Original: [Equation]
Revised: [Equation]

Original: /m 1 /(cid:19) ; (7.84) (3)

Original: [Equation]
Revised: [Equation]

Original: (cid:0)

Original: [Equation]

Original: [Equation]

Original: (cid:1) ∥≤O ( L ) .

Original:  Proof of Lemma 4. 
Revised:  Proof of Lemma 4.

Original: [Table] [Table] [Table] [Table]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: t

Original: [Table]
Revised: [Table]

Original:  L 
Revised: H 2 t ▽ θ 2 L 2 [( ϕ ( x ) , r 2 ); (cid:98)

Original: [Table]
Revised: [Equation]

Original: [Table] [Table] [Table] [Table]
Revised: 2 t − 1 ] 7: Return ( (cid:98) θ 1 t , (cid:98) θ 2 t )

Original: Proof.
Revised: Proof.

Original: For any t ∈ [ T ] ∧ ( I t = 1) , the regret of one round can be bounded as:
Revised: For any t ∈ [ T ] ∧ ( I t = 1) , the regret of one round can be bounded as:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: where E 1 is because of f ( x t,i ∗ ; θ t − 1 ) ≤ f ( x t, (cid:98) i ; θ t − 1 ) .
Revised: where E 1 is because of f ( x t,i ∗ ; θ t − 1 ) ≤ f ( x t, (cid:98) i ; θ t − 1 ) .

Original: For any t ∈ [ T ] ∧ ( I t = 0) , we have
Revised: For any t ∈ [ T ] ∧ ( I t = 0) , we have

Original: R t | ( I t = 0) =
Revised: R t | ( I t = 0) =

Original: E x t ,y t
Revised: E x t ,y t

Original: [ L ( y t, (cid:98) i , y t,y t ) − L ( y t,i ∗ , y t,y t )]
Revised: [ L ( y t, [Equation] i , y t,y t ) − L ( y t,i ∗ , y t,y t )]

Original: = 

Revised: =

Original: based on Lemma 7.5.
Revised: based on Lemma 7.5.

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]

Original: R

Original: [Equation]
Revised: [Equation]

Original: where I 1 is because of Lemma 8.2, I 3 is due to Lemma [Equation] 1, I 2 is the application of Lemma 7.14 and
Revised: where I 1 is because of Lemma 8.2, I 3 is due to Lemma 8.1, I 2 is the application of Lemma 7.14 and

Original: E 2 is the result of choice of m 
Revised: E 2 is the result of choice of m .

Original: [Equation]

Original: Lemma 8.1.
Revised: Lemma 8.1.

Original: For any δ ∈ (0 , 1) , ν > 0 , γ ≥ 1 , suppose m satisfies the conditions in Lemma 4.1.
Revised: For any δ ∈ (0 , 1) , ν > 0 , γ ≥ 1 , suppose m satisfies the conditions in Lemma 4.1.

Original: In round t ∈ [ T ] , given ( x t , y t ) ∼ D 
Revised: In round t ∈ [ T ] , given ( x t , y t ) ∼ D , let

Original: [Equation]
Revised: [Equation]

Original: let
Revised: Then, with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: Then, with probability at least 1 − δ , we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For any τ ∈ [ t ]
Revised: For any τ ∈ [ t ]

Original: [Equation]
Revised: [Equation]

Original: define
Revised: define

Original: [Equation]
Revised: [Equation]

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: where F τ − 1 denotes the σ -algebra generated by the history H τ − 1 .
Revised: where F τ − 1 denotes the σ -algebra generated by the history H τ − 1 .

Original: Therefore, { V τ } tτ =1 are the martingale difference sequence.
Revised: Therefore, { V τ } tτ =1 are the martingale difference sequence.

Original: Applying the Hoeffding-Azuma inequality
Revised: Applying the Hoeffding-Azuma inequality

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − δ
Revised: with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: As I 1 is equal to [Equation] we have
Revised: As I 1 is equal to 0 , we have

Original: [Equation]
Revised: [Equation]

Original: For I 3 , based on Lemma 7.8, for any (cid:101) θ 2 satisfying ∥ (cid:101) θ 2 − θ 20 ∥ 2 ≤ O ( ν √ m ) , with probability at least 1 − δ , we have 
Revised: For I 3 , based on Lemma 7.8, for any (cid:101) θ 2 satisfying ∥ (cid:101) θ 2 − θ 20 ∥ 2 ≤ O ( ν √ m ) , with probability at least 1 − δ , we have I

Original: [Equation]
Revised: [Equation]

Original: where I 4 is by the definition of µ .
Revised: where I 4 is by the definition of µ .

Original: Combining above inequalities together, with probability at least 1 − δ , we have
Revised: Combining above inequalities together, with probability at least 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original: where we applied union bound over δ to make above events occur concurrently.
Revised: where we applied union bound over δ to make above events occur concurrently.

Original: Lemma 8.2.
Revised: Lemma 8.2.

Original: For any δ ∈ (0 , 1) , ν > 0 , γ ≥ 1 , suppose m satisfies the conditions in Lemma [Equation]
Revised: For any δ ∈ (0 , 1) , ν > 0 , γ ≥ 1 , suppose m satisfies the conditions in Lemma 4.1.

Original: In round t ∈ [ T ] [Equation] given ( x t , y t ) ∼ D , let i ∗ = arg max i ∈
Revised: In round t ∈ [ T ] , given ( x t , y t ) ∼ D , let i ∗ = arg max i ∈

Original: [ k ] h ( x τ,i ) .
Revised: [ k ] h ( x t,i ) .

Original: Then, with probability at least 1 − δ , there exists θ 1 , ∗ t − 1 , θ 2 , ∗ t − 1 , such that
Revised: Then, with probability at least 1 − δ , there exists θ 1 , ∗ t − 1 , θ 2 , ∗ t − 1 , such that

Original: [Equation]
Revised: [Equation]

Original: where H t − 1 = { x τ,i ∗ , r 1 τ,i ∗ } t − 1 τ =1 is historical data.
Revised: where H t − 1 = { x τ,i ∗ , r 1 τ,i ∗ } t − 1 τ =1 is historical data

Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For any τ ∈ [ t ] , define
Revised: For any τ ∈ [ t ] [Equation] define

Original: [Equation]
Revised: [Equation]

Original: Then, we have
Revised: Then [Equation] we have

Original: [Equation]
Revised: [Equation]

Original:  where F τ − 1 denotes the σ -algebra generated by the history H τ − 1 .
Revised: ∗ | , 1 } (cid:3) = (8.12) where F τ − 1 denotes the σ -algebra generated by the history H τ − 1 .

Original: Therefore, { V τ } tτ =1 are the martingale difference sequence.
Revised: Therefore, { V τ } tτ =1 are the martingale difference sequence.

Original: Applying the Hoeffding-Azuma inequality, with probability at least 1 − δ , we have
Revised: Applying the Hoeffding-Azuma inequality, with probability at least 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original: As I 1 is equal to 0 , we have
Revised: As I 1 is equal to 0 , we have

Original: [Equation]
Revised: [Equation]

Original: For I 3 , based on Lemma7.8, for any (cid:101) θ 2 satisfying ∥ [Equation] θ 
Revised: For I 3 , based on Lemma7.8, for any (cid:101) θ 2 satisfying ∥ (cid:101) θ 2 − θ 20 ∥ 2 ≤ O ( ν √ m ) , with probability at least 1 − 3 δ , we have

Original: [Equation]
Revised: [Equation]

Original: − θ 20 ∥ 2 ≤ O ( ν √ m ) , with probability at least 1 − 3 δ , we have

Original: [Equation]
Revised: [Equation]

Original: where I 4 is by the definition of µ .
Revised: where I 4 is by the assumption of µ .

Original: Combining above inequalities together, with probability at least 1 − δ , we have 
Revised: Combining above inequalities together, with probability at least 1 − δ , we have t

Original: [Equation]
Revised: [Equation]

Original: where we applied union bound over δ to make above events occur concurrently.
Revised: where we applied union bound over δ to make above events occur concurrently.

Original:  Bounds for Effective Dimension (cid:101) d
Revised:  Bounds for Effective Dimension (cid:101) d

Original: Let { x t, (cid:98) i } Tt =1 be the selected contexts in T rounds, then we have the following definition of NTK.
Revised: Let { x t, (cid:98) i } Tt =1 be the selected contexts in T rounds, then we have the following definition of NTK.

Original: Definition 9.1 ( NTK [29, 4]) .
Revised: Definition 9.1 ( NTK [29, 4]) .

Original: Let N denote the normal distribution.
Revised: Let N denote the normal distribution.

Original: Define
Revised: Define

Original: [Equation]
Revised: [Equation]

Original: Then, over the contexts { x t, (cid:98) i } Tt =1 , the Neural Tangent Kernel (NTK) is defined as H =
Revised: Then, over the contexts { x t, (cid:98) i } Tt =1 , the Neural Tangent Kernel (NTK) is defined as H =

Original: ( H L + Σ L ) / 2 .
Revised: ( H L + Σ L ) / 2 .

Original: Then, we define the following gram matrix G .
Revised: Then, we define the following gram matrix G .

Original: Let g ( x ; θ 0 )
Revised: Let g ( x ; θ 0 )

Original: =
Revised: =

Original: ▽ θ f ( x ; θ 0 ) ∈
Revised: ▽ θ f ( x ; θ 0 ) ∈

Original: R p and G =
Revised: R p and G =

Original: [ g ( x 1 , (cid:98) i ; θ 0 ) / √ m, . .
Revised: [ g ( x 1 , (cid:98) i ; θ 0 ) / √ m, . .

Original: , g ( x T, (cid:98) i ; θ 0 ) / √ m ] ∈ R p × T where p
Revised: , g ( x T, (cid:98) i ; θ 0 ) / √ m ] ∈ R p × T where p

Original: = m + mkd + m 

Revised: = m + mkd + m

Original: ( L − 1) .
Revised: ( L − 1) .

Original: Therefore, we have G = G ⊤ G .
Revised: Therefore, we have G = G ⊤ G .

Original: Based on Theorem 3.1 in [4], when m ≥ O
Revised: Based on Theorem 3.1 in [4], when m ≥ O

Original: ( T 4 k 6 log(2 Tk/δ ) /λ 40 ) where λ 0 is the smallest eigenvalue of H , with probability at least 1 − δ , we have[Equation] Then
Revised: ( T 4 k 6 log(2 Tk/δ ) /λ 40 ) where λ 0 is the smallest eigenvalue of H , with probability at least 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original:  we have the following bound:
Revised: Then, we have the following bound:

Original: [Equation]
Revised: [Equation]

Original: where the first inequality is because of the concavity of log det( · ) and the third inequality is by
Revised: where the first inequality is because of the concavity of log det( · ) and the third inequality is by

Original: Lemma B.1 in [57] with the choice of m .
Revised: Lemma B.1 in [57] with the choice of m .

Original: Then, the effective dimension (cid:101) d can be bounded by:
Revised: Then, the effective dimension (cid:101) d can be bounded by:

Original: [Equation]
Revised: [Equation]

Original: where E 1 is because of det( I + G ⊤ G )
Revised: where E 1 is because of det( I + G ⊤ G )

Original: = det( I + GG ⊤ ) and E 2 is due to det( GG ⊤ ) =
Revised: = det( I + GG ⊤ ) and E 2 is due to det( GG ⊤ ) =

Original: ∥
Revised: ∥

Original: GG ⊤ ∥ p( GG ⊤ ∈
Revised: GG ⊤ ∥ p( GG ⊤ ∈

Original: R p × p ) and E 3 is according to
Revised: R p × p ) and E 3 is according to

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is as the result of ∥ g ( x t, (cid:98) i ; θ 0 ) / √ m ∥ 2 ≤ 1 (Lemma B.3 in [17]).Therefore, we have [Equation] d ≤
Revised: where the last inequality is as the result of ∥ g ( x t, (cid:98) i ; θ 0 ) / √ m ∥ 2 ≤ 1 (Lemma B.3 in [17]).

Original: [Equation]

Original: [Equation]

Original: T

Original: [Equation] p = m + [Equation] + m 2 ( L − 1) .
Revised: Therefore, we have (cid:101) d ≤ p + log(1 + T ) and p = m + mkd + m 2 ( L − 1) .

Original:  Further Details in Experiments
Revised:  Further Details in Experiments

Original: In this section, we report the specific configurations in the experiments, the sensitivity study of the core hyperparameter γ for I-NeurAL, and the ablation study for label budget.
Revised: In this section, we report the specific configurations in the experiments, the sensitivity study of the core hyperparameter γ for I-NeurAL, and the ablation study for label budget.

Original: Table 2 exhibits the details of using datasets.
Revised: Table 2 exhibits the details of using datasets.

Original: [Table]
Revised: [Table]

Original: Implementation Details .
Revised: Implementation Details .

Original: We use PyTorch as our backend, and all experiments were conducted on a server with NVIDIA Tesla V100 SXM2 GPU.
Revised: We use PyTorch as our backend, and all experiments were conducted on a server with NVIDIA Tesla V100 SXM2 GPU.

Original: The classification model in all methods is the same 2-layer fully-connected network with 100-width for the fair comparison.
Revised: The classification model in all methods is the same 2-layer fully-connected network with 100-width for the fair comparison.

Original: We use Adam optimizer to train the classification model with the fixed learning rate is 0 . 001 , and the batch size is 64 , since these are model-agnostic hyperparameters.
Revised: We use Adam optimizer to train the classification model with the fixed learning rate is 0 . 001 , and the batch size is 64 , since these are model-agnostic hyperparameters.

Original: As NeuAL-NTK-F and NeuAL-NTK-D only work on the binary classification problem, we transformed the k -class classification problem into the binary classification problem when k > 2 .
Revised: As NeuAL-NTK-F and NeuAL-NTK-D only work on the binary classification problem, we transformed the k -class classification problem into the binary classification problem when k > 2 .

Original: In detail, given k class, we regard the k/ 2 classes as one class and remaining classes as another class.
Revised: In detail, given k class, we regard the k/ 2 classes as one class and remaining classes as another class.

Original: For Random algorithm, the query probability p is set as 0 . 1 .
Revised: For Random algorithm, the query probability p is set as 0 . 1 .

Original: To find the best performance of each method, we conduct the grid search over all hyperparameters.
Revised: To find the best performance of each method, we conduct the grid search over all hyperparameters.

Original: In Margin algorithm, the query threshold is searched over { 0 . 3 , 0 . 5 , 0 . 7 , 0 . 9 , 0 .
Revised: In Margin algorithm, the query threshold is searched over { 0 . 3 , 0 . 5 , 0 . 7 , 0 . 9 , 0 .

Original: 95 } for all datasets.
Revised: 95 } for all datasets.

Original: For NeuAL-NTK-F and NeuAL-NTK-D, there is also an exploration parameter γ to determine the query aggressiveness and we conduct the grid search over { 0 . 1 , 0 . 3 , 0 . 5 , 0 . 8 , 1 . 0 } for it.
Revised: For NeuAL-NTK-F and NeuAL-NTK-D, there is also an exploration parameter γ to determine the query aggressiveness and we conduct the grid search over { 0 . 1 , 0 . 3 , 0 . 5 , 0 . 8 , 1 . 0 } for it.

Original: For ALPS, following the method in [23], we form the hypothesis class by generating 20 hypotheses on the 3% of total data samples (as same as the query budget) with different random seeds, and we conduct the grid search { 0 . 1 , 0 . 25 , 0 . 5 , 0 . 75 , 0 . 9 } over the two slack terms in ALPS.
Revised: For ALPS, following the method in [23], we form the hypothesis class by generating 20 hypotheses on the 3% of total data samples (as same as the query budget) with different random seeds, and we conduct the grid search { 0 . 1 , 0 . 25 , 0 . 5 , 0 . 75 , 0 . 9 } over the two slack terms in ALPS.

Original: We have tried to generate more hypotheses in the experiments, but the performance of ALPS does not improve accordingly.
Revised: We have tried to generate more hypotheses in the experiments, but the performance of ALPS does not improve accordingly.

Original: For I-NeurAL, the only hyperparameter γ is searched over { 1 , 2 , 5 , 6 , 7 , 10 } for all datasets ( c 1 and c 2 in β t is set as 1 ).
Revised: For I-NeurAL, the only hyperparameter γ is searched over { 1 , 2 , 5 , 6 , 7 , 10 } for all datasets ( c 1 and c 2 in β t is set as 1 ).

Original: The confidence level δ is set as 0 .
Revised: The confidence level δ is set as 0 .

Original: 1 for all the needed methods.
Revised: 1 for all the needed methods.

Original: In the end, we report the average results of 5 runs for all methods.
Revised: In the end, we report the average results of 5 runs for all methods.

Original: [Figure]
Revised: [Figure]

Original: Sensitivity study for γ .
Revised: Sensitivity study for γ .

Original: As γ is closely related to the query threshold of I-NeurAL, we test the sensitivity of I-NeurAL with regard to γ .
Revised: As γ is closely related to the query threshold of I-NeurAL, we test the sensitivity of I-NeurAL with regard to γ .

Original: Based on our analysis, it is required that γ ≥ 1 .
Revised: Based on our analysis, it is required that γ ≥ 1 .

Original: When γ is the smallest number (e.g., γ = 1 ), I-NeurAL queries the labels only if the difference between the top two classes is very small (i.e., the confidence level is very low).
Revised: When γ is the smallest number (e.g., γ = 1 ), I-NeurAL queries the labels only if the difference between the top two classes is very small (i.e., the confidence level is very low).

Original: In this manner, I-NeurAL will save more query budget but take more risks on many instances, incurring more regret.
Revised: In this manner, I-NeurAL will save more query budget but take more risks on many instances, incurring more regret.

Original: This explains why the red line ( γ = 1 ) is above all other lines.
Revised: This explains why the red line ( γ = 1 ) is above all other lines.

Original: In contrast, if γ is a large number, I-NeurAL will be more aggressive in making queries and thus obtain satisfactory performance.
Revised: In contrast, if γ is a large number, I-NeurAL will be more aggressive in making queries and thus obtain satisfactory performance.

Original: However, if γ is too large,
Revised: However, if γ is too large,

Original: I-NeurAL tends to query on these instances even when our model is very confident to the predictions, wasting the query budget that could have been used on these uncertain instances.
Revised: I-NeurAL tends to query on these instances even when our model is very confident to the predictions, wasting the query budget that could have been used on these uncertain instances.

Original: Therefore, we expect that γ is neither too small nor too large.
Revised: Therefore, we expect that γ is neither too small nor too large.

Original: The experiments verify our assumption, when γ isor 7 , I-NeurAL almost achieves the best performance throughout all datasets and configurations.
Revised: The experiments verify our assumption, when γ isor 7 , I-NeurAL almost achieves the best performance throughout all datasets and configurations.

Original: Ablation study for label budget .
Revised: Ablation study for label budget .

Original: To examine the final performance of each algorithm, we conduct new experiments with different percentages of label budget: 3%, 10%, 20%, 50%.
Revised: To examine the final performance of each algorithm, we conduct new experiments with different percentages of label budget: 3%, 10%, 20%, 50%.

Original: After T rounds, we evaluate the latest model on the test (unseen) data to calculate the accuracy, which evaluates the population accuracy.
Revised: After T rounds, we evaluate the latest model on the test (unseen) data to calculate the accuracy, which evaluates the population accuracy.

Original: For all the datasets, T is set as 10000 , except that T = 2000 for Phishing because Phishing has fewer data instances.
Revised: For all the datasets, T is set as 10000 , except that T = 2000 for Phishing because Phishing has fewer data instances.

Original: Table 3 - 6 reports the results.
Revised: Table 3 - 6 reports the results.

Original: To sum up, I-NeurAL still achieves the best accuracy with different label budget.
Revised: To sum up, I-NeurAL still achieves the best accuracy with different label budget.

Original: With a small amount of label budget (3%, 10%), I-NeurAL can make smart decisions to query labels on these instances with big uncertainty and leverage the full feedback to exploit the past knowledge, which enable I-NeurAL to outperform all the baselines.
Revised: With a small amount of label budget (3%, 10%), I-NeurAL can make smart decisions to query labels on these instances with big uncertainty and leverage the full feedback to exploit the past knowledge, which enable I-NeurAL to outperform all the baselines.

Original: With the larger label budget (20%, 50 %), all methods have enough labels to train.
Revised: With the larger label budget (20%, 50 %), all methods have enough labels to train.

Original: Thus, the advantages of I-NeurAL is less significant and the gap between I-NeurAL and baselines is decreasing.
Revised: Thus, the advantages of I-NeurAL is less significant and the gap between I-NeurAL and baselines is decreasing.

Original: Nevertheless, I-NeurAl still has the best performance benefiting from smart query choices.
Revised: Nevertheless, I-NeurAl still has the best performance benefiting from smart query choices.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]