Original: D EEP A CTIVE L EARNING BY L EVERAGING T RAINING D YNAMICS
Revised: D EEP A CTIVE L EARNING BY L EVERAGING T RAINING D YNAMICS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Active learning theories and methods have been extensively studied in classical statistical learning settings.
Revised: Active learning theories and methods have been extensively studied in classical statistical learning settings.

Original: However, deep active learning, i.e., active learning with deep learning models, is usually based on empirical criteria without solid theoretical justiﬁcation, thus suffering from heavy doubts when some of those fail to provide beneﬁts in applications.
Revised: However, deep active learning, i.e., active learning with deep learning models, is usually based on empirical criteria without solid theoretical justiﬁcation, thus suffering from heavy doubts when some of those fail to provide beneﬁts in applications.

Original: In this paper, by exploring the connection between the generalization performance and the training dynamics, we propose a theory-driven deep active learning method ( dynamicAL ) which selects samples to maximize training dynamics.
Revised: In this paper, by exploring the connection between the generalization performance and the training dynamics, we propose a theory-driven deep active learning method ( dynamicAL ) which selects samples to maximize training dynamics.

Original: In particular, we prove that convergence speed of training and the generalization performance is positively correlated under the ultra-wide condition and show that maximizing the training dynamics leads to a better generalization performance.
Revised: In particular, we prove that convergence speed of training and the generalization performance is positively correlated under the ultra-wide condition and show that maximizing the training dynamics leads to a better generalization performance.

Original: Further on, to scale up to large deep neural networks and data sets, we introduce two relaxations for the subset selection problem and reduce the time complexity from polynomial to constant.
Revised: Further on, to scale up to large deep neural networks and data sets, we introduce two relaxations for the subset selection problem and reduce the time complexity from polynomial to constant.

Original: Empirical results show that dynamicAL not only outperforms the other baselines consistently but also scales well on large deep learning models.
Revised: Empirical results show that dynamicAL not only outperforms the other baselines consistently but also scales well on large deep learning models.

Original: We hope our work inspires more attempts in bridging the theoretical ﬁndings of deep networks and practical impacts in deep active learning applications.
Revised: We hope our work inspires more attempts in bridging the theoretical ﬁndings of deep networks and practical impacts in deep active learning applications.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Training deep learning (DL) models usually requires a large amount of high-quality labeled data (Zhang et al., 2017) to optimize a model with a massive number of parameters.
Revised: Training deep learning (DL) models usually requires a large amount of high-quality labeled data (Zhang et al., 2017) to optimize a model with a massive number of parameters.

Original: The acquisition of such annotated data is usually time-consuming and expensive, making it unaffordable in the ﬁelds that require high domain expertise.
Revised: The acquisition of such annotated data is usually time-consuming and expensive, making it unaffordable in the ﬁelds that require high domain expertise.

Original: A promising approach for minimizing the labeling effort is active learning (AL), which aims to identify and label the maximally informative samples, so that a high-performing classiﬁer can be trained with minimal labeling effort (Settles, 2009).
Revised: A promising approach for minimizing the labeling effort is active learning (AL), which aims to identify and label the maximally informative samples, so that a high-performing classiﬁer can be trained with minimal labeling effort (Settles, 2009).

Original: Under classical statistical learning settings, theories of active learning have been extensively studied from the perspective of VC dimension (Hanneke et al., 2014).
Revised: Under classical statistical learning settings, theories of active learning have been extensively studied from the perspective of VC dimension (Hanneke et al., 2014).

Original: As a result, a variety of methods have been proposed, such as (i) the version-space-based approach, which requires maintaining a set of models (Cohn et al., 1994; Balcan et al., 2009), and (ii) the clustering-based approach, which assumes the data within the same cluster have pure labels (Dasgupta & Hsu, 2008).
Revised: As a result, a variety of methods have been proposed, such as (i) the version-space-based approach, which requires maintaining a set of models (Cohn et al., 1994; Balcan et al., 2009), and (ii) the clustering-based approach, which assumes the data within the same cluster have pure labels (Dasgupta & Hsu, 2008).

Original: However, the theoretical analyses for these classical settings may not hold for over-parameterized deep neural networks where the traditional wisdom is ineffective (Zhang et al., 2017).
Revised: However, the theoretical analyses for these classical settings may not hold for over-parameterized deep neural networks where the traditional wisdom is ineffective (Zhang et al., 2017).

Original: For example, marginbased methods select the labeling examples in the vicinity of the learned decision boundary (Balcan et al., 2007; Balcan & Long, 2013).
Revised: For example, marginbased methods select the labeling examples in the vicinity of the learned decision boundary (Balcan et al., 2007; Balcan & Long, 2013).

Original: However, in the over-parameterized regime, every labeled example could potentially be near the learned decision boundary (Karzand & Nowak, 2019).
Revised: However, in the over-parameterized regime, every labeled example could potentially be near the learned decision boundary (Karzand & Nowak, 2019).

Original: As a result, theoretically, such analysis can hardly guide us to design practical active learning methods.
Revised: As a result, theoretically, such analysis can hardly guide us to design practical active learning methods.

Original: Besides, empirically, multiple deep active learning works, borrowing observations and insights from the classical theories and methods, have been observed unable to outperform their passive learning counterparts in a few application scenarios (Kirsch et al., 2019; Ash et al., 2020).
Revised: Besides, empirically, multiple deep active learning works, borrowing observations and insights from the classical theories and methods, have been observed unable to outperform their passive learning counterparts in a few application scenarios (Kirsch et al., 2019; Ash et al., 2020).

Original: On the other hand, the analysis of neural network’s optimization and generalization performance has witnessed several exciting developments in recent years in terms of the deep learning theory (Jacot et al., 2018; Arora et al., 2019b; Lee et al., 2019).
Revised: On the other hand, the analysis of neural network’s optimization and generalization performance has witnessed several exciting developments in recent years in terms of the deep learning theory (Jacot et al., 2018; Arora et al., 2019b; Lee et al., 2019).

Original: It is shown that the training dynamics of deep neural networks using gradient descent can be characterized by the Neural Tangent Kernel (NTK) of inﬁnite (Jacot et al., 2018) or ﬁnite (Hanin & Nica, 2020) width networks.
Revised: It is shown that the training dynamics of deep neural networks using gradient descent can be characterized by the Neural Tangent Kernel (NTK) of inﬁnite (Jacot et al., 2018) or ﬁnite (Hanin & Nica, 2020) width networks.

Original: This is further leveraged to characterize the generalization of over-parameterized networks through Rademacher complexity analysis (Arora et al., 2019b; Cao & Gu, 2019).
Revised: This is further leveraged to characterize the generalization of over-parameterized networks through Rademacher complexity analysis (Arora et al., 2019b; Cao & Gu, 2019).

Original: We are therefore inspired to ask: How can we design a practical and generic active learning method for deep neural networks with theoretical justiﬁcations?
Revised: We are therefore inspired to ask: How can we design a practical and generic active learning method for deep neural networks with theoretical justiﬁcations?

Original: To answer this question, we ﬁrstly explore the connection between the model performance on testing data and the convergence speed on training data for the over-parameterized deep neural networks.
Revised: To answer this question, we ﬁrstly explore the connection between the model performance on testing data and the convergence speed on training data for the over-parameterized deep neural networks.

Original: Based on the NTK framework (Jacot et al., 2018; Arora et al., 2019b), we theoretically show that if a deep neural network converges faster (“Train Faster”), then it tends to have a better generalization performance (“Generalize Better”), which matches with the existing observations (Hardt et al., 2016;
Revised: Based on the NTK framework (Jacot et al., 2018; Arora et al., 2019b), we theoretically show that if a deep neural network converges faster (“Train Faster”), then it tends to have a better generalization performance (“Generalize Better”), which matches with the existing observations (Hardt et al., 2016;

Original: Liu et al., 2017; Lyle et al., 2020; Ru et al., 2020; Xu et al., 2021).
Revised: Liu et al., 2017; Lyle et al., 2020; Ru et al., 2020; Xu et al., 2021).

Original: Motivated by the aforementioned connection, we ﬁrst introduce Training Dynamics , the derivative of training loss with respect to iteration, as a proxy to quantitatively describe the training process.
Revised: Motivated by the aforementioned connection, we ﬁrst introduce Training Dynamics , the derivative of training loss with respect to iteration, as a proxy to quantitatively describe the training process.

Original: On top of it, we formally propose our generic and theoretically-motivated deep active learning method, dynamicAL , which will query labels for a subset of unlabeled samples that maximally increases the training dynamics.
Revised: On top of it, we formally propose our generic and theoretically-motivated deep active learning method, dynamicAL , which will query labels for a subset of unlabeled samples that maximally increases the training dynamics.

Original: In order to compute the training dynamics by merely using the unlabeled samples, we leverage two relaxations
Revised: In order to compute the training dynamics by merely using the unlabeled samples, we leverage two relaxations

Original: Pseudo-labeling and Subset Approximation to solve this non-trivial subset selection problem.
Revised: Pseudo-labeling and Subset Approximation to solve this non-trivial subset selection problem.

Original: Our relaxed approaches are capable of effectively estimating the training dynamics as well as efﬁciently solving the subset selection problem by reducing the complexity from O ( N b ) to O ( b ) .
Revised: Our relaxed approaches are capable of effectively estimating the training dynamics as well as efﬁciently solving the subset selection problem by reducing the complexity from O ( N b ) to O ( b ) .

Original: In theory, we coin a new term Alignment to measure the length of the label vector’s projection on the neural tangent kernel space.
Revised: In theory, we coin a new term Alignment to measure the length of the label vector’s projection on the neural tangent kernel space.

Original: Then, we demonstrate that higher alignment usually comes with a faster convergence speed and a lower generalization bound.
Revised: Then, we demonstrate that higher alignment usually comes with a faster convergence speed and a lower generalization bound.

Original: Furthermore, we extend the previous analysis to an active learning setting, where the i.i.d.
Revised: Furthermore, we extend the previous analysis to an active learning setting, where the i.i.d.

Original: assumption may not hold with the help of the maximum mean discrepancy (Borgwardt et al., 2006).
Revised: assumption may not hold with the help of the maximum mean discrepancy (Borgwardt et al., 2006).

Original: Finally, we show that alignment is positively correlated with our active learning goal, training dynamics, which implies that maximizing training dynamics will lead to a better generalization performance.
Revised: Finally, we show that alignment is positively correlated with our active learning goal, training dynamics, which implies that maximizing training dynamics will lead to a better generalization performance.

Original: Regarding experiments, we have empirically veriﬁed our theory by conducting extensive experiments on three datasets, CIFAR10 (Krizhevsky et al., 2009), SVHN (Netzer et al., 2011), and
Revised: Regarding experiments, we have empirically veriﬁed our theory by conducting extensive experiments on three datasets, CIFAR10 (Krizhevsky et al., 2009), SVHN (Netzer et al., 2011), and

Original: Caltech101 (Fei-Fei et al., 2004) using three types of network structure: vanilla CNN, ResNet (He et al., 2016), and VGG (Simonyan & Zisserman, 2015).
Revised: Caltech101 (Fei-Fei et al., 2004) using three types of network structure: vanilla CNN, ResNet (He et al., 2016), and VGG (Simonyan & Zisserman, 2015).

Original: We ﬁrst show that result of the subset selection problem delivered by the subset approximation is close to the global optimum solution.
Revised: We ﬁrst show that result of the subset selection problem delivered by the subset approximation is close to the global optimum solution.

Original: Furthermore, under active learning setting, our method not only outperforms the other baselines but also scales well on large deep learning models.
Revised: Furthermore, under active learning setting, our method not only outperforms the other baselines but also scales well on large deep learning models.

Original: The main contributions of our paper can be summarized as follows:
Revised: The main contributions of our paper can be summarized as follows:

Original: • We propose a theory-driven deep active learning method, dynamicAL , inspired by the results of “train faster, generalize better”.
Revised: • We propose a theory-driven deep active learning method, dynamicAL , inspired by the results of “train faster, generalize better”.

Original: To this end, we introduce the Training Dynamics, as a proxy to describe the training process and embody the inspiration.
Revised: To this end, we introduce the Training Dynamics, as a proxy to describe the training process and embody the inspiration.

Original: • We demonstrate that the convergence speed of training and the generalization performance is (positively) strongly correlated under the ultra-wide condition; we also show that maximizing the training dynamics will lead to a lower generalization error in the scenario of active learning.
Revised: • We demonstrate that the convergence speed of training and the generalization performance is (positively) strongly correlated under the ultra-wide condition; we also show that maximizing the training dynamics will lead to a lower generalization error in the scenario of active learning.

Original: • Our method is easy to implement.
Revised: • Our method is easy to implement.

Original: We conduct extensive experiments to evaluate the effectiveness of dynamicAL and empirically show that our method consistently outperforms other methods in a wide range of active learning settings.
Revised: We conduct extensive experiments to evaluate the effectiveness of dynamicAL and empirically show that our method consistently outperforms other methods in a wide range of active learning settings.

Original:  B ACKGROUND 2. N OTATIONS
Revised:  B ACKGROUND 2. N OTATIONS

Original: We use the random variable x ∈ X to represent input data feature and y ∈ Y as label where K is the number of classes and [ K ] := { 1 , 2 , ..., K } .
Revised: We use the random variable x ∈ X to represent input data feature and y ∈ Y as label where K is the number of classes and [ K ] := { 1 , 2 , ..., K } .

Original: We are given a data source D with unknown distribution p ( x, y ) .
Revised: We are given a data source D with unknown distribution p ( x, y ) .

Original: We further denote the concatenation of x as X =
Revised: We further denote the concatenation of x as X =

Original: [ x 1 , x 2 , ..., x M ] (cid:62) and that of y as Y =
Revised: [ x 1 , x 2 , ..., x M ] (cid:62) and that of y as Y =

Original: [ y 1 , y 2 , ..., y M ] (cid:62) .
Revised: [ y 1 , y 2 , ..., y M ] (cid:62) .

Original: We consider a deep learning classiﬁer h
Revised: We consider a deep learning classiﬁer h

Original: θ ( x ) = argmax σ ( f ( x ; θ )) :
Revised: θ ( x ) = argmax σ ( f ( x ; θ )) :

Original: x → y parameterized by θ ∈ R p , where σ ( · ) is the softmax function and f is a neural network.
Revised: x → y parameterized by θ ∈ R p , where σ ( · ) is the softmax function and f is a neural network.

Original: Let ⊗ be the
Revised: Let ⊗ be the

Original: Kronecker Product and I
Revised: Kronecker Product and I

Original: K ∈ R
Revised: K ∈ R

Original: K × K be an identity matrix.
Revised: K × K be an identity matrix.

Original: 2. A CTIVE L EARNING
Revised: 2. A CTIVE L EARNING

Original: The goal of active learning is to improve the learning efﬁciency of a model with a limited labeling budget.
Revised: The goal of active learning is to improve the learning efﬁciency of a model with a limited labeling budget.

Original: In this work, we consider the pool-based AL setup, where a ﬁnite data set S =
Revised: In this work, we consider the pool-based AL setup, where a ﬁnite data set S =

Original: { ( x l , y l ) } M l =
Revised: { ( x l , y l ) } M l =

Original: 2. N EURAL T ANGENT K ERNEL
Revised: 2. N EURAL T ANGENT K ERNEL

Original: The Neural Tangent Kernel (Jacot et al., 2018) has been widely applied to analyze the dynamics of neural networks.
Revised: The Neural Tangent Kernel (Jacot et al., 2018) has been widely applied to analyze the dynamics of neural networks.

Original: If the neural network is sufﬁciently wide, properly initialized, and trained by gradient descent with inﬁnitesimal step size (a.k.a.
Revised: If the neural network is sufﬁciently wide, properly initialized, and trained by gradient descent with inﬁnitesimal step size (a.k.a.

Original: gradient ﬂow), then the neural network is equivalent to kernel regression predictor with a deterministic kernel Θ ( · , · ) , called Neural Tangent
Revised: gradient ﬂow), then the neural network is equivalent to kernel regression predictor with a deterministic kernel Θ ( · , · ) , called Neural Tangent

Original: Kernel (NTK).
Revised: Kernel (NTK).

Original: When minimizing the mean squared error loss, at the iteration t , the dynamics of the neural network f has a closed-form expression:
Revised: When minimizing the mean squared error loss, at the iteration t , the dynamics of the neural network f has a closed-form expression:

Original: [Equation]
Revised: [Equation]

Original: where θ ( t ) denotes the parameter of the neural network at iteration t , K t ( X , X ) ∈ R |X|× K ×|X|× K is called the empirical NTK and K i,jt ( x, x (cid:48) )
Revised: where θ ( t ) denotes the parameter of the neural network at iteration t , K t ( X , X ) ∈ R |X|× K ×|X|× K is called the empirical NTK and K i,jt ( x, x (cid:48) )

Original: = ∇ θ f i ( x ; θ ( t )) (cid:62) ∇ θ f j ( x
Revised: = ∇ θ f i ( x ; θ ( t )) (cid:62) ∇ θ f j ( x

Original: [Equation]
Revised: [Equation]

Original: ; θ ( t )) , for two samples x, x (cid:48) ∈ X
Revised: ; θ ( t )) , for two samples x, x (cid:48) ∈ X

Original: and i, j ∈
Revised: and i, j ∈

Original: [ K ] .
Revised: [ K ] .

Original: The time-variant kernel K t ( · , · ) is equivalent to the (time-invariant) NTK with high probability, that is, if the neural network is sufﬁciently wide and properly initialized, then:
Revised: The time-variant kernel K t ( · , · ) is equivalent to the (time-invariant) NTK with high probability, that is, if the neural network is sufﬁciently wide and properly initialized, then:

Original: [Equation]
Revised: [Equation]

Original: The ﬁnal learned neural network at iteration t , is equivalent to the kernel regression solution with respect to the NTK (Lee et al., 2019).
Revised: The ﬁnal learned neural network at iteration t , is equivalent to the kernel regression solution with respect to the NTK (Lee et al., 2019).

Original: For any input x and training data { X, Y } we have,
Revised: For any input x and training data { X, Y } we have,

Original: [Equation]
Revised: [Equation]

Original: where η is the learning rate, Θ ( x, X ) is the NTK matrix between input x and all samples in training data X .
Revised: where η is the learning rate, Θ ( x, X ) is the NTK matrix between input x and all samples in training data X .

Original:  M ETHOD
Revised:  M ETHOD

Original: In section 3.1, we introduce the notion of training dynamics which can be used to describe the training process.
Revised: In section 3.1, we introduce the notion of training dynamics which can be used to describe the training process.

Original: Then, in section 3.2, based on the training dynamics, we proposed dynamicAL .
Revised: Then, in section 3.2, based on the training dynamics, we proposed dynamicAL .

Original: In section 3.3, we discuss the connection between dynamicAL and existing deep active learning methods.
Revised: In section 3.3, we discuss the connection between dynamicAL and existing deep active learning methods.

Original: 3. T RAINING D YNAMICS
Revised: 3. T RAINING D YNAMICS

Original: In this section, we introduce the notion of training dynamics.
Revised: In this section, we introduce the notion of training dynamics.

Original: The cross-entropy loss over labeled set
Revised: The cross-entropy loss over labeled set

Original: S is deﬁned as:
Revised: S is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: (4) where σ i ( f ( x ; θ )) =
Revised: (4) where σ i ( f ( x ; θ )) =

Original: exp( f i ( x ; θ )) (cid:80) j exp( f j ( x ; θ )) .
Revised: exp( f i ( x ; θ )) (cid:80) j exp( f j ( x ; θ )) .

Original: We ﬁrst analyze the dynamics of the training loss, with respect to iteration t , on one labeled sample (derivation is in Appendix A.1):
Revised: We ﬁrst analyze the dynamics of the training loss, with respect to iteration t , on one labeled sample (derivation is in Appendix A.1):

Original: [Equation]
Revised: [Equation]

Original: For neural networks trained by gradient descent, if learning rate η is small, then ∇ t θ = θ t +1 − θ t = − η
Revised: For neural networks trained by gradient descent, if learning rate η is small, then ∇ t θ = θ t +1 − θ t = − η

Original: [Equation]
Revised: [Equation]

Original: ( xl
Revised: ( xl

Original: [Equation]
Revised: [Equation]

Original: Taking the partial derivative of the training loss with respect to the parameters
Revised: Taking the partial derivative of the training loss with respect to the parameters

Original: [Equation]
Revised: [Equation]

Original: we have (the derivation of the following equation can be found in Appendix A.2):
Revised: we have (the derivation of the following equation can be found in Appendix A.2):

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we can further get the following result for the dynamics of training loss,
Revised: we can further get the following result for the dynamics of training loss,

Original: [Equation]
Revised: [Equation]

Original: Furthermore, we deﬁne d i ( X, Y )
Revised: Furthermore, we deﬁne d i ( X, Y )

Original: =
Revised: =

Original: σ i ( f ( X ; θ ))
Revised: σ i ( f ( X ; θ ))

Original: − Y i and Y
Revised: − Y i and Y

Original: i is the label vector of all samples for i -th class.
Revised: i is the label vector of all samples for i -th class.

Original: Then, the training dynamics (dynamics of training loss) over training set S is denoted by
Revised: Then, the training dynamics (dynamics of training loss) over training set S is denoted by

Original: G ( S ) ∈
Revised: G ( S ) ∈

Original: R :
Revised: R :

Original: [Equation]
Revised: [Equation]

Original: 3. A CTIVE L EARNING BY A CTIVATING T RAINING D YNAMICS
Revised: 3. A CTIVE L EARNING BY A CTIVATING T RAINING D YNAMICS

Original: Before we present dynamicAL , we state the Proposition 1, which serves as theoretical guidance for dynamicAL and will be proved in Section 4.
Revised: Before we present dynamicAL , we state the Proposition 1, which serves as theoretical guidance for dynamicAL and will be proved in Section 4.

Original: Proposition 1.
Revised: Proposition 1.

Original: For deep neural networks, converge faster leads to a lower worst-case generalization error.
Revised: For deep neural networks, converge faster leads to a lower worst-case generalization error.

Original: Motivated by the connection between convergence speed and generalization performance, we propose the general-purpose active learning method, dynamicAL , which aims to accelerate the convergence by querying labels for unlabeled samples.
Revised: Motivated by the connection between convergence speed and generalization performance, we propose the general-purpose active learning method, dynamicAL , which aims to accelerate the convergence by querying labels for unlabeled samples.

Original: As we described in the previous section, the training dynamics can be used to describe the training process.
Revised: As we described in the previous section, the training dynamics can be used to describe the training process.

Original: Therefore, we employ the training dynamics as a proxy to design an active learning method.
Revised: Therefore, we employ the training dynamics as a proxy to design an active learning method.

Original: Speciﬁcally, at each query round, dynamicAL will query labels for samples which maximize the training dynamics G ( S ) , i.e. ,
Revised: Speciﬁcally, at each query round, dynamicAL will query labels for samples which maximize the training dynamics G ( S ) , i.e. ,

Original: [Equation]
Revised: [Equation]

Original: ) where Q is the corresponding data set for Q with ground-truth labels.
Revised: ) where Q is the corresponding data set for Q with ground-truth labels.

Original: Notice that when applying the above objective in practice, we are facing two major challenges.
Revised: Notice that when applying the above objective in practice, we are facing two major challenges.

Original: First, G ( S ∪ Q ) cannot be directly computed, because the label information of unlabeled examples is not available before the query.
Revised: First, G ( S ∪ Q ) cannot be directly computed, because the label information of unlabeled examples is not available before the query.

Original: Second, the subset selection problem can be computationally prohibitive if enumerating all possible sets with size b .
Revised: Second, the subset selection problem can be computationally prohibitive if enumerating all possible sets with size b .

Original: Therefore, we employ the following two relaxations to make this maximization problem to be solved with constant time complexity.
Revised: Therefore, we employ the following two relaxations to make this maximization problem to be solved with constant time complexity.

Original: Pseudo Labeling.
Revised: Pseudo Labeling.

Original: To estimate the training dynamics, we use the predicted label ˆ y u for sample x u in the unlabeled data set U to compute G .
Revised: To estimate the training dynamics, we use the predicted label ˆ y u for sample x u in the unlabeled data set U to compute G .

Original: Note, the effectiveness of this adaptation has been demonstrated in the recent gradient-based methods (Ash et al., 2020; Mu et al., 2020), which compute the gradient as if the model’s current prediction on the example is the true label.
Revised: Note, the effectiveness of this adaptation has been demonstrated in the recent gradient-based methods (Ash et al., 2020; Mu et al., 2020), which compute the gradient as if the model’s current prediction on the example is the true label.

Original: Therefore, the maximization problem in Equation (9) is changed to,
Revised: Therefore, the maximization problem in Equation (9) is changed to,

Original: [Equation]
Revised: [Equation]

Original: where (cid:98) Q is the corresponding data set for Q with peseudo labels, (cid:98) Y Q .
Revised: where (cid:98) Q is the corresponding data set for Q with peseudo labels, (cid:98) Y Q .

Original: Subset Approximation.
Revised: Subset Approximation.

Original: The subset selection problem of Equation (10) still requires enumerating all possible subsets of U with size b , which is O ( n b ) .
Revised: The subset selection problem of Equation (10) still requires enumerating all possible subsets of U with size b , which is O ( n b ) .

Original: We simplify the selection problem to the following problem without causing any change on the result,
Revised: We simplify the selection problem to the following problem without causing any change on the result,

Original: [Equation]
Revised: [Equation]

Original: where ∆( (cid:98) Q | S ) =
Revised: where ∆( (cid:98) Q | S ) =

Original: G ( S ∪ (cid:98) Q ) − G
Revised: G ( S ∪ (cid:98) Q ) − G

Original: ( S ) is deﬁned as the change of training dynamics.
Revised: ( S ) is deﬁned as the change of training dynamics.

Original: We approximate the change of training dynamics caused by query set Q using the summation of the change of training dynamics caused by each sample in the query set.
Revised: We approximate the change of training dynamics caused by query set Q using the summation of the change of training dynamics caused by each sample in the query set.

Original: Then the maximization problem can be converted to Equation (12) which can be solved by a greedy algorithm with O ( b ) .
Revised: Then the maximization problem can be converted to Equation (12) which can be solved by a greedy algorithm with O ( b ) .

Original: [Equation]
Revised: [Equation]

Original: To further show the approximated result is reasonably good, we decompose the change of training dynamics as (derivation in Appendix A.4) ,
Revised: To further show the approximated result is reasonably good, we decompose the change of training dynamics as (derivation in Appendix A.4) ,

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst term in the right hand side is the approximated change of training dynamics.
Revised: The ﬁrst term in the right hand side is the approximated change of training dynamics.

Original: Then, we further deﬁne the Approximation Ratio (14) which measures the approximation quality,
Revised: Then, we further deﬁne the Approximation Ratio (14) which measures the approximation quality,

Original: [Equation]
Revised: [Equation]

Original: We empirically measure the expectation of the Approximation Ratio on three data sets with three different neural networks.
Revised: We empirically measure the expectation of the Approximation Ratio on three data sets with three different neural networks.

Original: As shown in Figure 4, the expectation, E Q ∼ U R ( (cid:98) Q | S )
Revised: As shown in Figure 4, the expectation, E Q ∼ U R ( (cid:98) Q | S )

Original: ≈ 1 when the model is converged.
Revised: ≈ 1 when the model is converged.

Original: Therefore, the approximated result delivered by greedy algorithm is close to the global optimal solution of the original maximization problem, Equation (10), especially when the model is converged.
Revised: Therefore, the approximated result delivered by greedy algorithm is close to the global optimal solution of the original maximization problem, Equation (10), especially when the model is converged.

Original: Based on the above two approximations, we present the proposed method dynamicAL in Algorithm 1.
Revised: Based on the above two approximations, we present the proposed method dynamicAL in Algorithm 1.

Original: As described below, the algorithm starts by training a neural network f ( · ; θ ) on initial labeled set S until converge.
Revised: As described below, the algorithm starts by training a neural network f ( · ; θ ) on initial labeled set S until converge.

Original: Then, for every unlabeled sample x u , we compute pseudo label ˆ y u and the change of training dynamics ∆( { ( x u , (cid:98) y u ) }| S ) .
Revised: Then, for every unlabeled sample x u , we compute pseudo label ˆ y u and the change of training dynamics ∆( { ( x u , (cid:98) y u ) }| S ) .

Original: After that, dynamicAL will query labels for top- b samples causing maximal change on training dynamics.
Revised: After that, dynamicAL will query labels for top- b samples causing maximal change on training dynamics.

Original: Then train the neural network (without re-initialization) on the extended labeled set, and repeat the process.
Revised: Then train the neural network (without re-initialization) on the extended labeled set, and repeat the process.

Original: Algorithm
Revised: Algorithm

Original: Input: Neural network f ( · ; θ ) , unlabeled sample set U , initial labeled set S , number of query round R , query batch size b .
Revised: Input: Neural network f ( · ; θ ) , unlabeled sample set U , initial labeled set S , number of query round R , query batch size b .

Original: for r = 1 to R do
Revised: for r = 1 to R do

Original: Train f ( · ; θ ) on S with cross-entropy loss until convergence.
Revised: Train f ( · ; θ ) on S with cross-entropy loss until convergence.

Original: for x u ∈ U do
Revised: for x u ∈ U do

Original: Compute its pseudo label ˆ y u = argmax f
Revised: Compute its pseudo label ˆ y u = argmax f

Original: ( x u ; θ ) .
Revised: ( x u ; θ ) .

Original: Compute ∆ ( { ( x u , (cid:98) y u ) }| S ) .
Revised: Compute ∆ ( { ( x u , (cid:98) y u ) }| S ) .

Original: end for
Revised: end for

Original: Select b query samples Q with the highest ∆ values, and request their labels from the oracle.
Revised: Select b query samples Q with the highest ∆ values, and request their labels from the oracle.

Original: Update the labeled data set S = S ∪ Q .
Revised: Update the labeled data set S = S ∪ Q .

Original: end for return Final model f ( · ; θ ) .
Revised: end for return Final model f ( · ; θ ) .

Original: 3. R ELATION TO E XISTING M ETHODS
Revised: 3. R ELATION TO E XISTING M ETHODS

Original: Although existing deep active learning methods are usually designed based on heuristic criteria, some of them have empirically shown their effectiveness (Ash et al., 2020; Liu et al., 2021; Huang et al., 2016).
Revised: Although existing deep active learning methods are usually designed based on heuristic criteria, some of them have empirically shown their effectiveness (Ash et al., 2020; Liu et al., 2021; Huang et al., 2016).

Original: We surprisingly found that our theoretically-motivated method dynamicAL has some connections with those existing methods from the perspective of active learning criterion.
Revised: We surprisingly found that our theoretically-motivated method dynamicAL has some connections with those existing methods from the perspective of active learning criterion.

Original: The proposed active learning criterion in Equation (12) can be explicitly written as (derivation in Appendix A.5) :
Revised: The proposed active learning criterion in Equation (12) can be explicitly written as (derivation in Appendix A.5) :

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst term of right-hand side can be interpreted as the square of gradient length (2-norm) which reﬂects the uncertainty of the model on the example and has been wildly used as an active learning criterion in some existing works (Huang et al., 2016; Ash et al., 2020; Shukla, 2021).
Revised: The ﬁrst term of right-hand side can be interpreted as the square of gradient length (2-norm) which reﬂects the uncertainty of the model on the example and has been wildly used as an active learning criterion in some existing works (Huang et al., 2016; Ash et al., 2020; Shukla, 2021).

Original: The second term can be regarded as the inﬂuence function (Koh & Liang, 2017) with identity hessian matrix.
Revised: The second term can be regarded as the inﬂuence function (Koh & Liang, 2017) with identity hessian matrix.

Original: And recently, Liu et al.
Revised: And recently, Liu et al.

Original: (2021) has empirically shown that the effectiveness of using it as the active learning criterion.
Revised: (2021) has empirically shown that the effectiveness of using it as the active learning criterion.

Original: We hope our theoretical analysis can also shed some light on the interpretation of previous methods.
Revised: We hope our theoretical analysis can also shed some light on the interpretation of previous methods.

Original:  T HEORETICAL A NALYSIS
Revised:  T HEORETICAL A NALYSIS

Original: In this section, we study the correlation between the convergence rate of the training loss and the generalization error under the ultra-wide condition (Jacot et al., 2018; Arora et al., 2019b).
Revised: In this section, we study the correlation between the convergence rate of the training loss and the generalization error under the ultra-wide condition (Jacot et al., 2018; Arora et al., 2019b).

Original: We deﬁne a measure named alignment to quantify the convergence rate and further show the connection with generalization bound.
Revised: We deﬁne a measure named alignment to quantify the convergence rate and further show the connection with generalization bound.

Original: The analysis provides a theoretical guarantee for the phenomenon of “Train
Revised: The analysis provides a theoretical guarantee for the phenomenon of “Train

Original: Faster, Generalization Better” as well as our active learning method dynamicAL with a rigorous treatment.
Revised: Faster, Generalization Better” as well as our active learning method dynamicAL with a rigorous treatment.

Original: Finally, we show that the our active learning proxy, training dynamics, is correlated with alignment, which indicates that increasing the training dynamics leads to larger convergence rate and better generalization performance.
Revised: Finally, we show that the our active learning proxy, training dynamics, is correlated with alignment, which indicates that increasing the training dynamics leads to larger convergence rate and better generalization performance.

Original: We leave all proofs of theorems and details of veriﬁcation experiments in Appendix B and D respectively.
Revised: We leave all proofs of theorems and details of veriﬁcation experiments in Appendix B and D respectively.

Original: 4. T RAIN FASTER PROVABLY GENERALIZE BETTER
Revised: 4. T RAIN FASTER PROVABLY GENERALIZE BETTER

Original: Given an ultra-wide neural network, the gradient descent can achieve a near-zero training error (Jacot et al., 2018; Arora et al., 2019a) and its generalization ability in unseen data can be bounded (Arora et al., 2019b).
Revised: Given an ultra-wide neural network, the gradient descent can achieve a near-zero training error (Jacot et al., 2018; Arora et al., 2019a) and its generalization ability in unseen data can be bounded (Arora et al., 2019b).

Original: It is shown that both convergence and generalization have a connection with the
Revised: It is shown that both convergence and generalization have a connection with the

Original: NTK (Arora et al., 2019b).
Revised: NTK (Arora et al., 2019b).

Original: However, the question what is the relation between convergence rate and generalization bound has not been answered.
Revised: However, the question what is the relation between convergence rate and generalization bound has not been answered.

Original: We formally give a solution by introducing the concept of alignment , which is deﬁned as follows,
Revised: We formally give a solution by introducing the concept of alignment , which is deﬁned as follows,

Original: Deﬁnition 1 (Alignment) .
Revised: Deﬁnition 1 (Alignment) .

Original: Given a data set S =
Revised: Given a data set S =

Original: { X, Y } , the alignment is a measure of correlation between X and Y projected in the NTK space.
Revised: { X, Y } , the alignment is a measure of correlation between X and Y projected in the NTK space.

Original: In particular, the alignment can be computed by
Revised: In particular, the alignment can be computed by

Original: [Equation]
Revised: [Equation]

Original: In the following, we will demonstrate why train faster generalizes better through alignment.
Revised: In the following, we will demonstrate why train faster generalizes better through alignment.

Original: In particular, the relation of convergence rate and generalization bound with alignment are analyzed.
Revised: In particular, the relation of convergence rate and generalization bound with alignment are analyzed.

Original: The convergence rate of gradient descent for ultra-wide networks is presented in following lemma,
Revised: The convergence rate of gradient descent for ultra-wide networks is presented in following lemma,

Original: Lemma 1 (Convergence Analysis with NTK, Theorem 4.1 of (Arora et al., 2019b)) .
Revised: Lemma 1 (Convergence Analysis with NTK, Theorem 4.1 of (Arora et al., 2019b)) .

Original: Suppose λ 0 = λ min ( Θ ) > 0 for all subsets of data samples.
Revised: Suppose λ 0 = λ min ( Θ ) > 0 for all subsets of data samples.

Original: For δ ∈ (0 , 1) , if m = Ω( n 7 λ 40 δ 4 (cid:15) 2 ) and η = O ( λ 0 n 2 ) .
Revised: For δ ∈ (0 , 1) , if m = Ω( n 7 λ 40 δ 4 (cid:15) 2 ) and η = O ( λ 0 n 2 ) .

Original: Then with probability at least 1 − δ , the network can achieve near-zero training error,
Revised: Then with probability at least 1 − δ , the network can achieve near-zero training error,

Original: [Equation]
Revised: [Equation]

Original: where n denotes the number of training samples, m the width of hidden layers.
Revised: where n denotes the number of training samples, m the width of hidden layers.

Original: The NTK Θ = V (cid:62) Λ V with Λ
Revised: The NTK Θ = V (cid:62) Λ V with Λ

Original: = { λ i } ni
Revised: = { λ i } ni

Original: =1 a diagonal matrix of eigenvalues and V = { (cid:126)v i } ni
Revised: =1 a diagonal matrix of eigenvalues and V = { (cid:126)v i } ni

Original: =1 a unitary matrix.
Revised: =1 a unitary matrix.

Original: In this lemma, we take mean square error (MSE) loss as an example for the convenience of illustration.
Revised: In this lemma, we take mean square error (MSE) loss as an example for the convenience of illustration.

Original: The conclusion can be extended to other loss functions such as cross-entropy loss (see Appendix B. in (Lee et al., 2019)).
Revised: The conclusion can be extended to other loss functions such as cross-entropy loss (see Appendix B. in (Lee et al., 2019)).

Original: From the lemma, we ﬁnd convergence rate is governed by the dominant term (16) as E t ( X, Y ) =
Revised: From the lemma, we ﬁnd convergence rate is governed by the dominant term (16) as E t ( X, Y ) =

Original: (cid:113)(cid:80)
Revised: (cid:113)(cid:80)

Original: Kk =1 (cid:80) ni
Revised: Kk =1 (cid:80) ni

Original: =1 (1 − ηλ i ) 2 t ( (cid:126)v (cid:62) i Y k ) 2 , which is correlated with the alignment :
Revised: =1 (1 − ηλ i ) 2 t ( (cid:126)v (cid:62) i Y k ) 2 , which is correlated with the alignment :

Original: Theorem 1 (Relationship between convergence rate and alignment) .
Revised: Theorem 1 (Relationship between convergence rate and alignment) .

Original: Under the same assumptions as in Lemma 1.
Revised: Under the same assumptions as in Lemma 1.

Original: The convergence rate described by E t satisﬁes,
Revised: The convergence rate described by E t satisﬁes,

Original: [Equation]
Revised: [Equation]

Original: Remark 1.
Revised: Remark 1.

Original: In the above theorem, we demonstrate that the alignment can measure the convergence rate.
Revised: In the above theorem, we demonstrate that the alignment can measure the convergence rate.

Original: Especially, we ﬁnd that both the upper bound and the lower bound of error E t ( X, Y ) are inversely proportional to the alignment, which implies that higher alignment will lead to achieving faster convergence.
Revised: Especially, we ﬁnd that both the upper bound and the lower bound of error E t ( X, Y ) are inversely proportional to the alignment, which implies that higher alignment will lead to achieving faster convergence.

Original: Now we analyze the generalization performance of the proposed method through complexity analysis.
Revised: Now we analyze the generalization performance of the proposed method through complexity analysis.

Original: We demonstrate that the ultra-wide networks can achieve a reasonable generalization bound.
Revised: We demonstrate that the ultra-wide networks can achieve a reasonable generalization bound.

Original: Lemma 2 (Generalization bound with NTK, Theorem 5.1 of (Arora et al., 2019b)) .
Revised: Lemma 2 (Generalization bound with NTK, Theorem 5.1 of (Arora et al., 2019b)) .

Original: Suppose data S
Revised: Suppose data S

Original: =
Revised: =

Original: ( x i , y i ) } n i =1 are i.i.d.
Revised: ( x i , y i ) } n i =1 are i.i.d.

Original: samples from a non-degenerate distribution p ( x, y ) , and m ≥ poly( n, λ − 1 0 , δ − 1 ) .
Revised: samples from a non-degenerate distribution p ( x, y ) , and m ≥ poly( n, λ − 1 0 , δ − 1 ) .

Original: Consider any loss function (cid:96) :
Revised: Consider any loss function (cid:96) :

Original: R × R → [0 , 1] that is 1 -Lipschitz.
Revised: R × R → [0 , 1] that is 1 -Lipschitz.

Original: Then with probability at least 1 − δ over the random initialization, the network trained by gradient descent for
Revised: Then with probability at least 1 − δ over the random initialization, the network trained by gradient descent for

Original: T ≥ Ω( 1 ηλ 0 log nδ ) iterations has population risk L p =
Revised: T ≥ Ω( 1 ηλ 0 log nδ ) iterations has population risk L p =

Original: E ( x,y ) ∼ p ( x,y ) [ (cid:96) ( f T ( x ; θ ) , y )] that is bounded as follows:
Revised: E ( x,y ) ∼ p ( x,y ) [ (cid:96) ( f T ( x ; θ ) , y )] that is bounded as follows:

Original: [Equation]
Revised: [Equation]

Original: In this lemma, we show that the dominant term in the generalization upper bound is B ( X, Y
Revised: In this lemma, we show that the dominant term in the generalization upper bound is B ( X, Y

Original: [Equation]
Revised: [Equation]

Original: = (cid:113)
Revised: = (cid:113)

Original: [Equation]
Revised: [Equation]

Original: n .
Revised: n .

Original: In the following theorem, we further prove that this bound is inversely proportional to the alignment
Revised: In the following theorem, we further prove that this bound is inversely proportional to the alignment

Original: A ( X, Y ) .
Revised: A ( X, Y ) .

Original: Theorem 2 (Relationship between the generalization bound and alignment) .
Revised: Theorem 2 (Relationship between the generalization bound and alignment) .

Original: Under the same assumptions as in Lemma 2.
Revised: Under the same assumptions as in Lemma 2.

Original: If we deﬁne the generalization upper bound as B ( X, Y ) = (cid:113) 2Tr
Revised: If we deﬁne the generalization upper bound as B ( X, Y ) = (cid:113) 2Tr

Original: [ Y (cid:62) Θ
Revised: [ Y (cid:62) Θ

Original: [Equation]
Revised: [Equation]

Original: 1 Y ] n , then it can be bounded with the alignment as follows,
Revised: 1 Y ] n , then it can be bounded with the alignment as follows,

Original: [Equation]
Revised: [Equation]

Original: Remark 2.
Revised: Remark 2.

Original: Theorems 1 and 2 reveal that the cause for the correlated phenomenons “Train Faster” and “Generalize Better” is the projection of label vector on the NTK space (alignment).
Revised: Theorems 1 and 2 reveal that the cause for the correlated phenomenons “Train Faster” and “Generalize Better” is the projection of label vector on the NTK space (alignment).

Original: 4. “T RAIN F ASTER , G ENERALIZE B ETTER ” FOR A CTIVE L EARNING
Revised: 4. “T RAIN F ASTER , G ENERALIZE B ETTER ” FOR A CTIVE L EARNING

Original: In the NTK framework (Arora et al., 2019b), the empirical average requires data in S is i.i.d.
Revised: In the NTK framework (Arora et al., 2019b), the empirical average requires data in S is i.i.d.

Original: samples,
Revised: samples,

Original: Lemma 2.
Revised: Lemma 2.

Original: However, this assumption may not hold in the active learning setting with multiple query rounds, because the training data is composed by i.i.d.
Revised: However, this assumption may not hold in the active learning setting with multiple query rounds, because the training data is composed by i.i.d.

Original: sampled initial label set and samples queried by active learning policy.
Revised: sampled initial label set and samples queried by active learning policy.

Original: To extend the previous analysis principle to active learning, we follow (Wang & Ye, 2015) to reformulate the Lemma 2 as:
Revised: To extend the previous analysis principle to active learning, we follow (Wang & Ye, 2015) to reformulate the Lemma 2 as:

Original: [Equation]
Revised: [Equation]

Original: where L q = E ( x,y ) ∼ q ( x,y ) [ (cid:96) ( f ( x ; θ ) , y )] , q ( x, y ) denotes the data distribution after query, and X, Y includes initial training samples and samples after query.
Revised: where L q = E ( x,y ) ∼ q ( x,y ) [ (cid:96) ( f ( x ; θ ) , y )] , q ( x, y ) denotes the data distribution after query, and X, Y includes initial training samples and samples after query.

Original: There is a new term in the upper bound, which is the difference between the true risk under different data distributions.
Revised: There is a new term in the upper bound, which is the difference between the true risk under different data distributions.

Original: [Equation]
Revised: [Equation]

Original: Though in active learning the data distribution for the labeled samples may be different from the original distribution.
Revised: Though in active learning the data distribution for the labeled samples may be different from the original distribution.

Original: However, they share the same conditional probability p ( y | x ) .
Revised: However, they share the same conditional probability p ( y | x ) .

Original: We deﬁne g ( x ) =
Revised: We deﬁne g ( x ) =

Original: (cid:82) y (cid:96) ( f ( x ; θ ) , y ) p ( y | x ) dy .
Revised: (cid:82) y (cid:96) ( f ( x ; θ ) , y ) p ( y | x ) dy .

Original: Then we have,
Revised: Then we have,

Original: [Equation]
Revised: [Equation]

Original: To measure the distance between two distribution, we employ the Maximum Mean Discrepancy (MMD) with neural tangent kernel (Jia et al., 2021)
Revised: To measure the distance between two distribution, we employ the Maximum Mean Discrepancy (MMD) with neural tangent kernel (Jia et al., 2021)

Original: (derivation in Appendix B.3).
Revised: (derivation in Appendix B.3).

Original: [Equation]
Revised: [Equation]

Original: Slightly overloading the notation, we denote the initial labeled set as S 0 , H Θ as the associated reproducing kernel
Revised: Slightly overloading the notation, we denote the initial labeled set as S 0 , H Θ as the associated reproducing kernel

Original: Note, MMD ( S 0 , S, H Θ ) as the empirical measure for
Revised: Note, MMD ( S 0 , S, H Θ ) as the empirical measure for

Original: MMD ( p ( x ) , q ( x ) , H Θ ) .
Revised: MMD ( p ( x ) , q ( x ) , H Θ ) .

Original: We empirically compute MMD and the dominant term of the generalization upper bound B under the active learning setting with our method dynamicAL .
Revised: We empirically compute MMD and the dominant term of the generalization upper bound B under the active learning setting with our method dynamicAL .

Original: As shown in Figure 1, on CIFAR10 with a CNN target model (three convolutional layers with global average pooling), the initial labeled set size | S | = 500 , query round R = 1 and budget size b ∈ { 250 , 500 , 1000 } , we observe that, under different active learning setting, the MMD is always much smaller than the B .
Revised: As shown in Figure 1, on CIFAR10 with a CNN target model (three convolutional layers with global average pooling), the initial labeled set size | S | = 500 , query round R = 1 and budget size b ∈ { 250 , 500 , 1000 } , we observe that, under different active learning setting, the MMD is always much smaller than the B .

Original: Therefore, the lemma 2 still holds for the target model with dynamicAL .
Revised: Therefore, the lemma 2 still holds for the target model with dynamicAL .

Original: The result for R ≥ 2 is in Appendix E.4.
Revised: The result for R ≥ 2 is in Appendix E.4.

Original: The computation detail of MMD and NTK is in Appendix D.1.
Revised: The computation detail of MMD and NTK is in Appendix D.1.

Original: 4. A LIGNMENT AND T RAINING D YNAMICS IN A CTIVE L EARNING S ETTING
Revised: 4. A LIGNMENT AND T RAINING D YNAMICS IN A CTIVE L EARNING S ETTING

Original: In this section, we show the relationship between the alignment and the training dynamics.
Revised: In this section, we show the relationship between the alignment and the training dynamics.

Original: To be consistent with the previous theoretical analysis (Theorem 1 and 2), we use the training dynamics with mean square error under the ultrawidth condition, which can be expressed as G MSE ( S ) =
Revised: To be consistent with the previous theoretical analysis (Theorem 1 and 2), we use the training dynamics with mean square error under the ultrawidth condition, which can be expressed as G MSE ( S ) =

Original: Tr (cid:2) ( f ( X ; θ ) − Y ) (cid:62) Θ ( X, X )( f ( X ; θ ) − Y ) (cid:3) .
Revised: Tr (cid:2) ( f ( X ; θ ) − Y ) (cid:62) Θ ( X, X )( f ( X ; θ ) − Y ) (cid:3) .

Original: Due to the limited space, we leave the derivation in Appendix A.3.
Revised: Due to the limited space, we leave the derivation in Appendix A.3.

Original: To further quantitatively evaluate the correlation between G MSE ( S ∪ Q ) and
Revised: To further quantitatively evaluate the correlation between G MSE ( S ∪ Q ) and

Original: A ( X (cid:107) X Q , Y (cid:107) Y Q ) , we utilize the Kendall τ coefﬁcient (Kendall, 1938) to empirically measure their relation.
Revised: A ( X (cid:107) X Q , Y (cid:107) Y Q ) , we utilize the Kendall τ coefﬁcient (Kendall, 1938) to empirically measure their relation.

Original: As shown in Figure 2, for CNN on CIFAR10 with active learning setting, where | S | =and | Q | = 250 , there is a strong agreement between G MSE ( S ∪ Q ) and A ( X (cid:107) X Q , Y (cid:107) Y Q ) , which further indicates that increasing the training dynamics will lead to a faster convergence and better generalization performance.
Revised: As shown in Figure 2, for CNN on CIFAR10 with active learning setting, where | S | =and | Q | = 250 , there is a strong agreement between G MSE ( S ∪ Q ) and A ( X (cid:107) X Q , Y (cid:107) Y Q ) , which further indicates that increasing the training dynamics will lead to a faster convergence and better generalization performance.

Original: More details about this veriﬁcation experiment are in Appendix D.2.
Revised: More details about this veriﬁcation experiment are in Appendix D.2.

Original:  E XPERIMENTS 5. E XPERIMENT S ETUP
Revised:  E XPERIMENTS 5. E XPERIMENT S ETUP

Original: Baselines.
Revised: Baselines.

Original: We compare dynamicAL with the following eight baselines: Random, Corset, Conﬁdence
Revised: We compare dynamicAL with the following eight baselines: Random, Corset, Conﬁdence

Original: Sampling (Conf), Margin Sampling (Marg), Entropy, and Active Learning by Learning (ALBL),
Revised: Sampling (Conf), Margin Sampling (Marg), Entropy, and Active Learning by Learning (ALBL),

Original: Batch Active learning by Diverse Gradient Embeddings (BADGE).
Revised: Batch Active learning by Diverse Gradient Embeddings (BADGE).

Original: Description of baseline methods is in Appendix E.1.
Revised: Description of baseline methods is in Appendix E.1.

Original: Data set and Target Model.
Revised: Data set and Target Model.

Original: We evaluate all the methods on three benchmark data sets, namely,
Revised: We evaluate all the methods on three benchmark data sets, namely,

Original: CIFAR10 (Krizhevsky et al., 2009), SVHN (Netzer et al., 2011), and Caltech101 (Fei-Fei et al., 2004).
Revised: CIFAR10 (Krizhevsky et al., 2009), SVHN (Netzer et al., 2011), and Caltech101 (Fei-Fei et al., 2004).

Original: We use the accuracy as the evaluation metric and report the mean value of 5 runs.
Revised: We use the accuracy as the evaluation metric and report the mean value of 5 runs.

Original: We consider three neural network architectures: vanilla CNN, ResNet

Revised: We consider three neural network architectures: vanilla CNN, ResNet

Original: (He et al., 2016), and VGG11 (Simonyan & Zisserman, 2015).
Revised: (He et al., 2016), and VGG11 (Simonyan & Zisserman, 2015).

Original: For each model, we keep the hyper-parameters used in their ofﬁcial implementations.
Revised: For each model, we keep the hyper-parameters used in their ofﬁcial implementations.

Original: More information about the implementation is in Appendix C.1.
Revised: More information about the implementation is in Appendix C.1.

Original: Active Learning Protocol.
Revised: Active Learning Protocol.

Original: We evaluate all those active learning methods in a batch-mode setup with an initial set size M = 500 for all those three data sets, batch size b varying from { 250 , 500 , 1000 } .
Revised: We evaluate all those active learning methods in a batch-mode setup with an initial set size M = 500 for all those three data sets, batch size b varying from { 250 , 500 , 1000 } .

Original: And, after each query, no re-initialization is used.
Revised: And, after each query, no re-initialization is used.

Original: 5. R ESULTS AND A NALYSIS
Revised: 5. R ESULTS AND A NALYSIS

Original: The main experimental results have been provided as plots due to the limited space.
Revised: The main experimental results have been provided as plots due to the limited space.

Original: We also provide tables in which we report the mean and standard deviation for each plot in Appendix E.3.
Revised: We also provide tables in which we report the mean and standard deviation for each plot in Appendix E.3.

Original: Overall Results.
Revised: Overall Results.

Original: The average test accuracy at each query round is shown in Figure 3.
Revised: The average test accuracy at each query round is shown in Figure 3.

Original: Our method dynamicAL can consistently outperform other methods for all query rounds.
Revised: Our method dynamicAL can consistently outperform other methods for all query rounds.

Original: This suggests that dynamicAL is a good choice regardless of the labeling budget.
Revised: This suggests that dynamicAL is a good choice regardless of the labeling budget.

Original: Besides, we notice dynamicAL can work on data sets with a large class number, such as Caltech101.
Revised: Besides, we notice dynamicAL can work on data sets with a large class number, such as Caltech101.

Original: However, the baseline method,
Revised: However, the baseline method,

Original: BADGE, cannot be scaled up to those data sets, because the required memory is linear with the number of classes.
Revised: BADGE, cannot be scaled up to those data sets, because the required memory is linear with the number of classes.

Original: Note, dynamicAL depends on the quality of pseudo labeling, thus a proper initial labeled set is important.
Revised: Note, dynamicAL depends on the quality of pseudo labeling, thus a proper initial labeled set is important.

Original: This ﬁgure shows that dynamicAL is able to work well with a relatively small initial labeled set ( M = 500 ).
Revised: This ﬁgure shows that dynamicAL is able to work well with a relatively small initial labeled set ( M = 500 ).

Original: Due to the limited space, we only show the result under three different setting in Figure 3.
Revised: Due to the limited space, we only show the result under three different setting in Figure 3.

Original: More evaluation results are in Appendix E.2.
Revised: More evaluation results are in Appendix E.2.

Original: [Figure]

Original: T e s t

Original: A cc u r a c y ( % )

Original: CIFAR10, CNN,

Original: [Figure]

Original: Size:

Original: Random Marg Entropy

Original: Coreset

Original: Conf ALBL BADGE dynamicAL 0 1 2 3 4 5 6 7 8 9 # Query Round

Original: T e s t

Original: A cc u r a c y ( % )

Original: SVHN

Original: [Figure]
Revised: [Figure]

Original: Effect of Query Size and Query Round.
Revised: Effect of Query Size and Query Round.

Original: Given the total label budget B , the increasing of query size always leads to the decreasing of query round.
Revised: Given the total label budget B , the increasing of query size always leads to the decreasing of query round.

Original: We study the inﬂuence of different query size and query round on dynamicAL from two perspectives.
Revised: We study the inﬂuence of different query size and query round on dynamicAL from two perspectives.

Original: First, we study the expected approximation ratio with different query batch size on different data sets.
Revised: First, we study the expected approximation ratio with different query batch size on different data sets.

Original: As shown in Figure 4, under different settings the expected approximation ratio always converges to 1 with the increase of training epochs, which further indicates that the query set selected by using the approximated change of training dynamics is a reasonably good result for the query set selection problem.
Revised: As shown in Figure 4, under different settings the expected approximation ratio always converges to 1 with the increase of training epochs, which further indicates that the query set selected by using the approximated change of training dynamics is a reasonably good result for the query set selection problem.

Original: Second, we study inﬂuence of query round for actual performance of target models.
Revised: Second, we study inﬂuence of query round for actual performance of target models.

Original: The performance for different target models on different data sets with total budge size B = 1000 is shown in Table 1.
Revised: The performance for different target models on different data sets with total budge size B = 1000 is shown in Table 1.

Original: For certain query budget, our active learning algorithm can be further improved if more query rounds is allowed.
Revised: For certain query budget, our active learning algorithm can be further improved if more query rounds is allowed.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Comparison with Different Variants.
Revised: Comparison with Different Variants.

Original: The active learning criterion of dynamicAL can be written as (cid:80) ( x,y )
Revised: The active learning criterion of dynamicAL can be written as (cid:80) ( x,y )

Original: [Equation]
Revised: [Figure]

Original: S (cid:107)∇ θ (cid:96) ( f ( x ; θ u ) , ˆ y u ) (cid:107) 2 + γ ∇ θ (cid:96) ( f ( x u ; θ ) , ˆ y u ) (cid:62) ∇ θ (cid:96) ( f ( x ; θ ) , y ) .
Revised: S (cid:107)∇ θ (cid:96) ( f ( x ; θ u ) , ˆ y u ) (cid:107) 2 + γ ∇ θ (cid:96) ( f ( x u ; θ ) , ˆ y u ) (cid:62) ∇ θ (cid:96) ( f ( x ; θ ) , y ) .

Original: We empirically show the performance for γ ∈ { 0 , 1 , 2 , ∞} in Figure 5.
Revised: We empirically show the performance for γ ∈ { 0 , 1 , 2 , ∞} in Figure 5.

Original: As the prediction of our analysis (Equation 15), for γ = 2 , the model achieves the best performance.
Revised: As the prediction of our analysis (Equation 15), for γ = 2 , the model achieves the best performance.

Original: The result conﬁrms the importance of theoretical analysis for the design of deep active learning methods.
Revised: The result conﬁrms the importance of theoretical analysis for the design of deep active learning methods.

Original: [-5 # Query Round 
-]
Revised: [-5 # Query Round 
-][Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: 5 # Query Round
Revised: 5 # Query Round

Original: T e s t
Revised: T e s t

Original: A cc u r a c y ( % )
Revised: A cc u r a c y ( % )

Original: CIFAR10, ResNet, Batch Size:γ=0 γ=1 γ=∞ γ=
Revised: CIFAR10, ResNet, Batch Size:γ=0 γ=1 γ=∞ γ=

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Neural Tangent Kernel (NTK): Recent study has shown that under proper conditions, an inﬁnitewidth neural network can be simpliﬁed as a linear model with Neural Tangent Kernel (NTK)
Revised: Neural Tangent Kernel (NTK): Recent study has shown that under proper conditions, an inﬁnitewidth neural network can be simpliﬁed as a linear model with Neural Tangent Kernel (NTK)

Original: (Jacot et al., 2018).
Revised: (Jacot et al., 2018).

Original: Since then, NTK has become a powerful theoretical tool to analyze the behavior of deep learning model (CNN, GNN, RNN) from its output dynamics (Arora et al., 2019a; Du et al., 2019;
Revised: Since then, NTK has become a powerful theoretical tool to analyze the behavior of deep learning model (CNN, GNN, RNN) from its output dynamics (Arora et al., 2019a; Du et al., 2019;

Original: Alemohammad et al., 2020) and to characterize the convergence and generalization error (Arora et al., 2019b).
Revised: Alemohammad et al., 2020) and to characterize the convergence and generalization error (Arora et al., 2019b).

Original: Besides, (Hanin & Nica, 2020) studies the ﬁnite-width NTK, aiming at making the NTK more practical.
Revised: Besides, (Hanin & Nica, 2020) studies the ﬁnite-width NTK, aiming at making the NTK more practical.

Original: Active Learning: Active learning aims at interactively query labels for unlabeled data points to maximize model performances (Settles, 2009).
Revised: Active Learning: Active learning aims at interactively query labels for unlabeled data points to maximize model performances (Settles, 2009).

Original: Among others, there are two popular strategies for active learning, i.e. , diversity sampling (Du et al., 2015; Volpi et al., 2018; Zhdanov, 2019), uncertainty sampling (Roy & McCallum, 2001; Zhu & Ma, 2012; Yang & Loog, 2016; Ash et al., 2020; Yoo & Kweon, 2019;
Revised: Among others, there are two popular strategies for active learning, i.e. , diversity sampling (Du et al., 2015; Volpi et al., 2018; Zhdanov, 2019), uncertainty sampling (Roy & McCallum, 2001; Zhu & Ma, 2012; Yang & Loog, 2016; Ash et al., 2020; Yoo & Kweon, 2019;

Original: Settles et al., 2007;
Revised: Settles et al., 2007;

Original: Liu et al., 2021).
Revised: Liu et al., 2021).

Original: Recently, there are several papers proposed to used gradient to measure uncertainty (Settles et al., 2007; Ash et al., 2020; Liu et al., 2021).
Revised: Recently, there are several papers proposed to used gradient to measure uncertainty (Settles et al., 2007; Ash et al., 2020; Liu et al., 2021).

Original: However, those methods need to compute gradient for each class, they can hardly be applied on data sets with a large class number.
Revised: However, those methods need to compute gradient for each class, they can hardly be applied on data sets with a large class number.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we bridge the gap between the theoretic ﬁndings of deep neural networks and realworld deep active learning applications.
Revised: In this work, we bridge the gap between the theoretic ﬁndings of deep neural networks and realworld deep active learning applications.

Original: By exploring the connection between the generalization performance and the training dynamics, we propose a theory-driven method, dynamicAL , which selects samples to maximize training dynamics.
Revised: By exploring the connection between the generalization performance and the training dynamics, we propose a theory-driven method, dynamicAL , which selects samples to maximize training dynamics.

Original: We prove that the convergence speed of training and the generalization performance is (positively) strongly correlated under the ultra-wide condition and we show that maximizing the training dynamics will lead to a lower generalization error.
Revised: We prove that the convergence speed of training and the generalization performance is (positively) strongly correlated under the ultra-wide condition and we show that maximizing the training dynamics will lead to a lower generalization error.

Original: Empirically our work show that dynamicAL not only consistently outperforms strong baselines across various setting but also scales well on large deep learning models
Revised: Empirically, our work show that dynamicAL not only consistently outperforms strong baselines across various setting, but also scales well on large deep learning models.

Revised: We provide the discussion of the limitation of future work in Appendix F.

Original:  R EPRODUCIBILITY
Revised:  R EPRODUCIBILITY

Original: To ensure the results and conclusions of our paper are reproducible, we make the following efforts:
Revised: To ensure the results and conclusions of our paper are reproducible, we make the following efforts:

Original: Empirically, we provide the URL for the code, and instructions needed to reproduce the main experimental results.
Revised: Empirically, we provide the URL for the code, and instructions needed to reproduce the main experimental results.

Original: And we specify all the training and implementation details in Section 5 and
Revised: And we specify all the training and implementation details in Section 5 and

Original: Appendix C.1.
Revised: Appendix C.1.

Original: Besides, we independently run experiments ﬁve times and report the mean and standard deviation in Appendix E.3.
Revised: Besides, we independently run experiments ﬁve times and report the mean and standard deviation in Appendix E.3.

Original: Theoretically, we state the full set of assumptions and include complete proofs of our theoretical results in Section 4 and
Revised: Theoretically, we state the full set of assumptions and include complete proofs of our theoretical results in Section 4 and

Original: Appendix B.
Revised: Appendix B.

Original: A APPENDIX: D ERIVATION OF O BJECTIVES
Revised: A APPENDIX: D ERIVATION OF O BJECTIVES

Original: For the notational convenience, we use f ( x ) to represent f ( x ; θ ) in the Appendix.
Revised: For the notational convenience, we use f ( x ) to represent f ( x ; θ ) in the Appendix.

Original: A. T RAINING D YNAMICS FOR C ROSS -E NTROPY L OSS
Revised: A. T RAINING D YNAMICS FOR C ROSS -E NTROPY L OSS

Original: The partial derivative for softmax function can be deﬁned with the following,
Revised: The partial derivative for softmax function can be deﬁned with the following,

Original: [Equation]
Revised: [Equation]

Original: Then, we have:
Revised: Then, we have:

Original: [Equation]
Revised: [Equation]

Original: A. D ERIVATION FOR C ROSS -E NTROPY L OSS
Revised: A. D ERIVATION FOR C ROSS -E NTROPY L OSS

Original: [Equation]
Revised: [Equation]

Original: A. APPENDIX: T RAINING D YNAMICS FOR M EAN S QUARED E RROR
Revised: A. APPENDIX: T RAINING D YNAMICS FOR M EAN S QUARED E RROR

Original: For the labeled data set S
Revised: For the labeled data set S

Original: [Equation]
Revised: [Equation]

Original: we deﬁne the Mean Squared Error(MSE) as:
Revised: we deﬁne the Mean Squared Error(MSE) as:

Original: [Equation]
Revised: [Equation]

Original: Then the training loss dynamics for each sample can deﬁne as:
Revised: Then the training loss dynamics for each sample can deﬁne as:

Original: [Equation]
Revised: [Equation]

Original: Because neural networks are optimized by gradient descent
Revised: Because neural networks are optimized by gradient descent

Original: [Equation]
Revised: [Equation]

Original: thus:
Revised: thus:

Original: [Equation]
Revised: [Equation]

Original: ∂θ
Revised: ∂θ

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: the training dynamics of MSE loss can be expressed as:
Revised: the training dynamics of MSE loss can be expressed as:

Original: [Equation]
Revised: [Equation]

Original: f ( X ) − Y )
Revised: f ( X ) − Y )

Original: A. APPENDIX: D ECOMPOSITION OF THE C HANGE OF T RAINING D YNAMICS
Revised: A. APPENDIX: D ECOMPOSITION OF THE C HANGE OF T RAINING D YNAMICS

Original: According to the deﬁnition of training dynamics ( Equation (8) ), we have,
Revised: According to the deﬁnition of training dynamics ( Equation (8) ), we have,

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: The change of training dynamics, ∆( (cid:98) Q | S ) =
Revised: The change of training dynamics, ∆( (cid:98) Q | S ) =

Original: G ( S ∪ (cid:98) Q ) − G ( S ) , can be further simpliﬁed as:
Revised: G ( S ∪ (cid:98) Q ) − G ( S ) , can be further simpliﬁed as:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: A. APPENDIX: S IMPLIFICATION OF THE C HANGE OF T RAINING D YNAMICS
Revised: A. APPENDIX: S IMPLIFICATION OF THE C HANGE OF T RAINING D YNAMICS

Original: [Equation]
Revised: [Equation]

Original: The derivative of loss with respect to model parameters can be written as:
Revised: The derivative of loss with respect to model parameters can be written as:

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: the change of training dynamics caused by { ( x u , (cid:98) y u ) } can be written as:
Revised: the change of training dynamics caused by { ( x u , (cid:98) y u ) } can be written as:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 2 (cid:88) ( x,y ) ∈ S ∇ θ (cid:96) ( f ( x u ; θ ) , ˆ y u ) (cid:62) ∇ θ (cid:96) ( f ( x ; θ ) , y )
Revised: 2 (cid:88) ( x,y ) ∈ S ∇ θ (cid:96) ( f ( x u ; θ ) , ˆ y u ) (cid:62) ∇ θ (cid:96) ( f ( x ; θ ) , y )

Original: B APPENDIX: P ROOFS FOR T HEORETICAL A NALYSIS B. P ROOFS FOR T HEOREM
Revised: B APPENDIX: P ROOFS FOR T HEORETICAL A NALYSIS B. P ROOFS FOR T HEOREM

Original: Lemma 1 (Convergence Analysis with NTK, Theorem 4.1 of Arora et al. (2019b)) .
Revised: Lemma 1 (Convergence Analysis with NTK, Theorem 4.1 of Arora et al. (2019b)) .

Original: Suppose λ 0 = λ min ( Θ ) > 0 for all subsets of data samples.
Revised: Suppose λ 0 = λ min ( Θ ) > 0 for all subsets of data samples.

Original: For δ ∈ (0 , 1) , if m = Ω( n 7 λ 40 δ 4 (cid:15) 2 ) and η = O ( λ 0 n 2 ) .
Revised: For δ ∈ (0 , 1) , if m = Ω( n 7 λ 40 δ 4 (cid:15) 2 ) and η = O ( λ 0 n 2 ) .

Original: Then with probability at least 1 − δ , the network can achieve near-zero training error,
Revised: Then with probability at least 1 − δ , the network can achieve near-zero training error,

Original: [Equation]
Revised: [Equation]

Original: (27) where n denotes the number of training samples, m the width of hidden layers.
Revised: (27) where n denotes the number of training samples, m the width of hidden layers.

Original: The NTK Θ = V (cid:62) Λ V with Λ
Revised: The NTK Θ = V (cid:62) Λ V with Λ

Original: = { λ i } n i =

Revised: = { λ i } n i =

Original: a diagonal matrix of eigenvalues and V = { (cid:126)v i } n i =1 a unitary matrix.
Revised: a diagonal matrix of eigenvalues and V = { (cid:126)v i } n i =1 a unitary matrix.

Original: Proof.
Revised: Proof.

Original: According to Arora et al. (2019b), if m = Ω( n 7 λ 40 δ 4 (cid:15) 2 ) and learning ratio η
Revised: According to Arora et al. (2019b), if m = Ω( n 7 λ 40 δ 4 (cid:15) 2 ) and learning ratio η

Original: = O ( λ 0 n 2 ) , then with probability at least 1 − δ over the random initialization, we have, (cid:107) Y l − f t
Revised: = O ( λ 0 n 2 ) , then with probability at least 1 − δ over the random initialization, we have, (cid:107) Y l − f t

Original: ( X ; θ ( t )) (cid:107) 2 = (cid:113)(cid:80)
Revised: ( X ; θ ( t )) (cid:107) 2 = (cid:113)(cid:80)

Original: Kk =1 (cid:80) ni
Revised: Kk =1 (cid:80) ni

Original: =1 (1 − ηλ i ) 2 t ( v (cid:62) i Y kl ) 2 ± (cid:15) .
Revised: =1 (1 − ηλ i ) 2 t ( v (cid:62) i Y kl ) 2 ± (cid:15) .

Original: We decompose the NTK using Θ = V (cid:62) Λ V with Λ =
Revised: We decompose the NTK using Θ = V (cid:62) Λ V with Λ =

Original: { λ i } ni =1 a diagonal matrix of eigenvalues and V = { v i } ni
Revised: { λ i } ni =1 a diagonal matrix of eigenvalues and V = { v i } ni

Original: =1 a unitary matrix.
Revised: =1 a unitary matrix.

Original: At each training step in active learning, the labeled samples will be updated by S = S ∪ Q .
Revised: At each training step in active learning, the labeled samples will be updated by S = S ∪ Q .

Original: We can apply the convergence result in each of this step and achieve near zero error.
Revised: We can apply the convergence result in each of this step and achieve near zero error.

Original: Theorem 1 (Relationship between convergence rate and alignment) .
Revised: Theorem 1 (Relationship between convergence rate and alignment) .

Original: Under the same assumptions as in Lemma 1.
Revised: Under the same assumptions as in Lemma 1.

Original: The convergence rate described by E t satisﬁes,
Revised: The convergence rate described by E t satisﬁes,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We ﬁrst prove the inequality on the right hand side.
Revised: We ﬁrst prove the inequality on the right hand side.

Original: It is easy to see that (1 − ηλ i ) 2 t ≤ (1 − ηλ i ) for each λ
Revised: It is easy to see that (1 − ηλ i ) 2 t ≤ (1 − ηλ i ) for each λ

Original: i and t ≥ 1 , based on the fact that ∀ λ i , 0 ≤ 1 − ηλ i ≤ 1 .
Revised: i and t ≥ 1 , based on the fact that ∀ λ i , 0 ≤ 1 − ηλ i ≤ 1 .

Original: Then we can obtain,
Revised: Then we can obtain,

Original: [Equation]
Revised: [Equation]

Original: Then we use Bernoulli’s inequality to prove the inequality on the left hand side.
Revised: Then we use Bernoulli’s inequality to prove the inequality on the left hand side.

Original: Bernoulli’s inequality states that, (1 + x ) r ≥ 1 + rx , for every integer r ≥ 0 and every real number x ≥ − 1 .
Revised: Bernoulli’s inequality states that, (1 + x ) r ≥ 1 + rx , for every integer r ≥ 0 and every real number x ≥ − 1 .

Original: It is easy to check that ( − ηλ i ) ≥ − 1 , ∀ λ i .
Revised: It is easy to check that ( − ηλ i ) ≥ − 1 , ∀ λ i .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: B. P ROOF FOR T HEOREM
Revised: B. P ROOF FOR T HEOREM

Original: Lemma 2 (Generalization bound with NTK, Theorem 5.1 of Arora et al. (2019b)) .
Revised: Lemma 2 (Generalization bound with NTK, Theorem 5.1 of Arora et al. (2019b)) .

Original: Suppose data S
Revised: Suppose data S

Original: =
Revised: =

Original: ( x i , y i ) } ni =1 are i.i
Revised: ( x i , y i ) } ni =1 are i.i

Original: [Equation]
Revised: [Equation]

Original: d. samples from a non-degenerate distribution p ( x, y )
Revised: d. samples from a non-degenerate distribution p ( x, y )

Original: [Equation]
Revised: [Equation]

Original: and m ≥ poly( n, λ − 1 0 , δ − 1 ) .
Revised: and m ≥ poly( n, λ − 1 0 , δ − 1 ) .

Original: Consider any loss function (cid:96) :
Revised: Consider any loss function (cid:96) :

Original: R × R → [0 , 1] that is 1 -Lipschitz.
Revised: R × R → [0 , 1] that is 1 -Lipschitz.

Original: Then with probability at least 1 − δ over the random initialization, the network trained by gradient descent for
Revised: Then with probability at least 1 − δ over the random initialization, the network trained by gradient descent for

Original: T ≥ Ω( 1 ηλ 0 log nδ ) iterations has population risk L p =
Revised: T ≥ Ω( 1 ηλ 0 log nδ ) iterations has population risk L p =

Original: E ( x,y ) ∼ p ( x,y ) [ (cid:96) ( f T ( x ) , y )] that is bounded as follows:
Revised: E ( x,y ) ∼ p ( x,y ) [ (cid:96) ( f T ( x ) , y )] that is bounded as follows:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We ﬁrst show that the generalization bound regrading our method on ultra-wide networks.
Revised: We ﬁrst show that the generalization bound regrading our method on ultra-wide networks.

Original: The distance between weights of trained networks and their initialization values can be bounded as, (cid:107) w r ( t ) − w r (0) (cid:107) =
Revised: The distance between weights of trained networks and their initialization values can be bounded as, (cid:107) w r ( t ) − w r (0) (cid:107) =

Original: O ( n √ mλ 0 √ δ ) .
Revised: O ( n √ mλ 0 √ δ ) .

Original: We then give a bound on the (cid:107) W ( t ) − W (0) (cid:107) F , where
Revised: We then give a bound on the (cid:107) W ( t ) − W (0) (cid:107) F , where

Original: W = { w 1 , w 2 , . . . } is the set of all parameters.
Revised: W = { w 1 , w 2 , . . . } is the set of all parameters.

Original: We deﬁnite Z = ∂f ( t ) ∂W ( t ) , then the update function is given by W ( t + 1) =
Revised: We deﬁnite Z = ∂f ( t ) ∂W ( t ) , then the update function is given by W ( t + 1) =

Original: W ( t ) − ηZ ( Z (cid:62) W ( t ) − Y ) .
Revised: W ( t ) − ηZ ( Z (cid:62) W ( t ) − Y ) .

Original: Summing over all the time step t = 0 , 1 , . .
Revised: Summing over all the time step t = 0 , 1 , . .

Original: , we can obtain that W ( ∞ ) − W (0) =
Revised: , we can obtain that W ( ∞ ) − W (0) =

Original: (cid:80) ∞ t =0 ηZ ( I − η Θ ) y
Revised: (cid:80) ∞ t =0 ηZ ( I − η Θ ) y

Original: =
Revised: =

Original: Z Θ − 1 Y .
Revised: Z Θ − 1 Y .

Original: Thus the distance can be measured by (cid:107) W ( ∞ ) − W (0) (cid:107) 2 F
Revised: Thus the distance can be measured by (cid:107) W ( ∞ ) − W (0) (cid:107) 2 F

Original: =
Revised: =

Original: Tr[ Y (cid:62) Θ − 1 Y ] .
Revised: Tr[ Y (cid:62) Θ − 1 Y ] .

Original: Then the key step is to apply Rademacher complexity.
Revised: Then the key step is to apply Rademacher complexity.

Original: Given R > 0 , with probability at least 1 − δ , simultaneously for every B > 0 , the function class F B,R = { f : (cid:107) w r ( t ) − w r (0) (cid:107) ≤ R ( ∀ r ∈ m ) , (cid:107) W ( ∞ ) − W (0) (cid:107) 2 F ≤ B } has empirical Rademacher complexity bounded as,
Revised: Given R > 0 , with probability at least 1 − δ , simultaneously for every B > 0 , the function class F B,R = { f : (cid:107) w r ( t ) − w r (0) (cid:107) ≤ R ( ∀ r ∈ m ) , (cid:107) W ( ∞ ) − W (0) (cid:107) 2 F ≤ B } has empirical Rademacher complexity bounded as,

Original: R S ( F B,R ) = 1 n E (cid:15)
Revised: R S ( F B,R ) = 1 n E (cid:15)

Original: [Table]
Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where B = (cid:113)
Revised: where B = (cid:113)

Original: Tr[ Y (cid:62) Θ − 1 ( X, X ) Y ] , and R =
Revised: Tr[ Y (cid:62) Θ − 1 ( X, X ) Y ] , and R =

Original: [Equation]
Revised: [Equation]

Original: Finally, Rademacher complexity directly gives an upper bound on generalization error (Mohri et al., 2018), sup
Revised: Finally, Rademacher complexity directly gives an upper bound on generalization error (Mohri et al., 2018), sup

Original: [Equation]
Revised: [Equation]

Original: n , where L S ( f ) ≤ 1 √ n .
Revised: n , where L S ( f ) ≤ 1 √ n .

Original: Based on this, we apply a union bound over a ﬁnite set of different i ’s.
Revised: Based on this, we apply a union bound over a ﬁnite set of different i ’s.

Original: Then with probability at least 1 − δ/over the sample S , we have sup
Revised: Then with probability at least 1 − δ/over the sample S , we have sup

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , O ( nλ 0 ) } .
Revised: , O ( nλ 0 ) } .

Original: Taking a union bound, we know that with probability at least 1 − 23 δ over the sample S , we have, f T ∈ F B ∗ i ,R for some i ∗ ,
Revised: Taking a union bound, we know that with probability at least 1 − 23 δ over the sample S , we have, f T ∈ F B ∗ i ,R for some i ∗ ,

Original: R S ( F B ∗ i ,R ) ≤ (cid:113)
Revised: R S ( F B ∗ i ,R ) ≤ (cid:113)

Original: [Equation]
Revised: [Equation]

Original: These together can imply,
Revised: These together can imply,

Original: [Equation]
Revised: [Equation]

Original: More proof details can be found in Arora et al.
Revised: More proof details can be found in Arora et al.

Original: (2019b).
Revised: (2019b).

Original: Theorem 2 (Relationship between the generalization bound and alignment) .
Revised: Theorem 2 (Relationship between the generalization bound and alignment) .

Original: Under the same assumptions as in Lemma (2) .
Revised: Under the same assumptions as in Lemma (2) .

Original: If we deﬁne the generalization upper bound as B ( X, Y ) = (cid:113) 2Tr[
Revised: If we deﬁne the generalization upper bound as B ( X, Y ) = (cid:113) 2Tr[

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ] n
Revised: ] n

Original: [Equation]
Revised: [Equation]

Original: then it can be bounded with the alignment as follows,
Revised: then it can be bounded with the alignment as follows,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We ﬁrst expand the following expression:
Revised: We ﬁrst expand the following expression:

Original: [Equation]
Revised: [Equation]

Original: Then we use this expansion to prove the inequality on the left hand side,
Revised: Then we use this expansion to prove the inequality on the left hand side,

Original: K (cid:88) k
Revised: K (cid:88) k

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: ≥
Revised: ≥

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: n (cid:88)
Revised: n (cid:88)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The second line is due to quadratic mean is greater or equal to geometric mean.
Revised: The second line is due to quadratic mean is greater or equal to geometric mean.

Original: Finally, we prove the inequality on the right hand side,
Revised: Finally, we prove the inequality on the right hand side,

Original: [Equation]
Revised: [Equation]

Original: B. D ERIVATION FOR M AXIMUM M EAN D ISCREPANCY
Revised: B. D ERIVATION FOR M AXIMUM M EAN D ISCREPANCY

Original: The difference between truth risk over p ( x ) and q ( x ) can be deﬁned as,
Revised: The difference between truth risk over p ( x ) and q ( x ) can be deﬁned as,

Original: [Equation]
Revised: [Equation]

Original: where g ( x ) = (cid:82) y (cid:96) ( f ( x ; θ ) , y ) p ( y | x ) dy .
Revised: where g ( x ) = (cid:82) y (cid:96) ( f ( x ; θ ) , y ) p ( y | x ) dy .

Original: Follow Wang & Ye (2015), we assume that the prediction functions have bounded norm (cid:107) f (cid:107) F .
Revised: Follow Wang & Ye (2015), we assume that the prediction functions have bounded norm (cid:107) f (cid:107) F .

Original: Thus, the function g is bounded.
Revised: Thus, the function g is bounded.

Original: By given the loss function, g is also measurable.
Revised: By given the loss function, g is also measurable.

Original: Then, ∃ ˆ g ∈ C ( x ) , such that,
Revised: Then, ∃ ˆ g ∈ C ( x ) , such that,

Original: [Equation]
Revised: [Equation]

Original: where C ( x ) is the function class of bounded and continuous functions of x .
Revised: where C ( x ) is the function class of bounded and continuous functions of x .

Original: To make the MMD term be measurable, we empirically restrict the MMD on a reproducing kernel Hilbert space (RKHS) with the characteristic kernel H Θ .
Revised: To make the MMD term be measurable, we empirically restrict the MMD on a reproducing kernel Hilbert space (RKHS) with the characteristic kernel H Θ .

Original: Following Gretton et al.
Revised: Following Gretton et al.

Original: [Equation]
Revised: [Equation]

Original: we know that the relationship between the true MMD and the empirical MMD is,
Revised: we know that the relationship between the true MMD and the empirical MMD is,

Original: [Equation]
Revised: [Equation]

Original: where MMD ( S 0 , S, H Θ ) is the empirical measure for MMD ( p ( x ) , q ( x ) , H Θ ) .
Revised: where MMD ( S 0 , S, H Θ ) is the empirical measure for MMD ( p ( x ) , q ( x ) , H Θ ) .

Original: Slightly overloading the notation, we denote S ∼ q ( x ) , which may not be i.i.d., and the initial label set S 0 ∼ p ( x ) .
Revised: Slightly overloading the notation, we denote S ∼ q ( x ) , which may not be i.i.d., and the initial label set S 0 ∼ p ( x ) .

Original: Then, in the active learning setting, S 0 ⊆ S .
Revised: Then, in the active learning setting, S 0 ⊆ S .

Original: Further, we denote | S 0 | = n 0 , | S | = n and ∀ x, x (cid:48) ∈
Revised: Further, we denote | S 0 | = n 0 , | S | = n and ∀ x, x (cid:48) ∈

Original: S, Θ ( x, x (cid:48) ) ≤ C .
Revised: S, Θ ( x, x (cid:48) ) ≤ C .

Original: Therefore, we have, (cid:113)
Revised: Therefore, we have, (cid:113)

Original: Cn + (cid:113)
Revised: Cn + (cid:113)

Original: Cn 0 ≥ 2 (cid:113)
Revised: Cn 0 ≥ 2 (cid:113)

Original: Cn .
Revised: Cn .

Original: For constant factor γ = MM + B , we have the following inequality,
Revised: For constant factor γ = MM + B , we have the following inequality,

Original: [Equation]
Revised: [Equation]

Original: Denoting 2 e − γ(cid:15) 2 n 4 C
Revised: Denoting 2 e − γ(cid:15) 2 n 4 C

Original: = δ/ 2 , then we have (cid:15) =
Revised: = δ/ 2 , then we have (cid:15) =

Original: (cid:113) 4 C ln
Revised: (cid:113) 4 C ln

Original: [Equation]
Revised: [Equation]

Original: 4 /δ )
Revised: 4 /δ )

Original: γn .
Revised: γn .

Original: Combining all the above results, we show that with probability at least 1 − δ , the following inequality holds:
Revised: Combining all the above results, we show that with probability at least 1 − δ , the following inequality holds:

Original: [Equation]
Revised: [Equation]

Original: Then, we can get,
Revised: Then, we can get,

Original: [Equation]
Revised: [Equation]

Original: C APPENDIX: M ORE DETAILS OF EXPERIMENTAL SETTINGS C. I MPLEMENTATION D ETAIL
Revised: C APPENDIX: M ORE DETAILS OF EXPERIMENTAL SETTINGS C. I MPLEMENTATION D ETAIL

Original: For simple CNN model, we utilize the same architecture used in Pytorch CIFAR10 Image Classiﬁcation Tutorial 1 .
Revised: For simple CNN model, we utilize the same architecture used in Pytorch CIFAR10 Image Classiﬁcation Tutorial 1 .

Original: For ResNet model, we use the Pytorch Ofﬁcal implementation of ResNet-18and set the output dimension to the number of classes.
Revised: For ResNet model, we use the Pytorch Ofﬁcal implementation of ResNet-18and set the output dimension to the number of classes.

Original: For VGG model, we use the Pytorch Ofﬁcal implementation of VGG-11 3 .
Revised: For VGG model, we use the Pytorch Ofﬁcal implementation of VGG-11 3 .

Original: Besides, we leverage the library BackPACK (Dangel et al., 2020) to collect the gradient of samples in batch.
Revised: Besides, we leverage the library BackPACK (Dangel et al., 2020) to collect the gradient of samples in batch.

Original: We keep a constant learning rate of 0.001 for all three datasets and all three models.
Revised: We keep a constant learning rate of 0.001 for all three datasets and all three models.

Original: All the codes mentioned above use the MIT license.
Revised: All the codes mentioned above use the MIT license.

Original: All experiments are done with four Tesla V100 SXM2 GPUs and a 12-core 2.2GHz CPU.
Revised: All experiments are done with four Tesla V100 SXM2 GPUs and a 12-core 2.2GHz CPU.

Original: D APPENDIX: V ERIFICATION E XPERIMENTS UNDER U LTRA - WIDE C ONDITION D. E XPERIMENT S ETTING AND C OMPUTATIONAL D ETAIL FOR THE E MPIRICAL C OMPARISON BETWEEN NTK AND MMD
Revised: D APPENDIX: V ERIFICATION E XPERIMENTS UNDER U LTRA - WIDE C ONDITION D. E XPERIMENT S ETTING AND C OMPUTATIONAL D ETAIL FOR THE E MPIRICAL C OMPARISON BETWEEN NTK AND MMD

Original: Experiment Setting For the veriﬁcation experiment shown in Figure 1, we employ a simple CNN as the target model, in which there are three convolutional layers following with global average pooling layer, on the CFAIR10 data set.
Revised: Experiment Setting For the veriﬁcation experiment shown in Figure 1, we employ a simple CNN as the target model, in which there are three convolutional layers following with global average pooling layer, on the CFAIR10 data set.

Original: Note, this CNN architecture is widely used in NTK analysis works (Arora et al., 2019a; Zandieh et al., 2021).
Revised: Note, this CNN architecture is widely used in NTK analysis works (Arora et al., 2019a; Zandieh et al., 2021).

Original: To make the veriﬁcation experiment close to the application setting, we keep size of initial labeled set and query batch size same as what we used in
Revised: To make the veriﬁcation experiment close to the application setting, we keep size of initial labeled set and query batch size same as what we used in

Original: Section 5.
Revised: Section 5.

Original: Computational Detail We follow (Jia et al., 2021) to compute the MMD with NTK kernel.
Revised: Computational Detail We follow (Jia et al., 2021) to compute the MMD with NTK kernel.

Original: The
Revised: The

Original: MMD term, MMD ( p ( x ) , q ( x ) , H Θ ) , can be simpliﬁed into the following form:
Revised: MMD term, MMD ( p ( x ) , q ( x ) , H Θ ) , can be simpliﬁed into the following form:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where x i , x j ∼ p ( x ) and x (cid:48)
Revised: where x i , x j ∼ p ( x ) and x (cid:48)

Original: i , x (cid:48)
Revised: i , x (cid:48)

Original: j ∼ q ( x ) .
Revised: j ∼ q ( x ) .

Original: Then, we deﬁne set S 0 as { x 1 , ..., x n 0 } ∼ p ( x ) and set S as { x (cid:48) 1 , ..., x (cid:48) n } ∼ q ( x ) , where n 0 ≤ n .
Revised: Then, we deﬁne set S 0 as { x 1 , ..., x n 0 } ∼ p ( x ) and set S as { x (cid:48) 1 , ..., x (cid:48) n } ∼ q ( x ) , where n 0 ≤ n .

Original: The MMD 2 ( S 0 , S ) is an unbiased estimation for
Revised: The MMD 2 ( S 0 , S ) is an unbiased estimation for

Original: MMD 2 ( p ( x ) , q ( x )) , can we explicitly computed by:
Revised: MMD 2 ( p ( x ) , q ( x )) , can we explicitly computed by:

Original: [Equation]
Revised: [Equation]

Original: Therefore, the MMD term of Equation (23), MMD ( S 0 , S, H Θ ) , can be empirically approximated by (cid:113) MMD 2 ( S 0 , S ) .
Revised: Therefore, the MMD term of Equation (23), MMD ( S 0 , S, H Θ ) , can be empirically approximated by (cid:113) MMD 2 ( S 0 , S ) .

Original: D. E XPERIMENT S ETTING FOR THE C ORRELATION S TUDY BETWEEN T RAINING D YNAMICS AND A LIGNMENT
Revised: D. E XPERIMENT S ETTING FOR THE C ORRELATION S TUDY BETWEEN T RAINING D YNAMICS AND A LIGNMENT

Original: Experiment Setting For the veriﬁcation experiment shown in Figure 2, we also use the simple
Revised: Experiment Setting For the veriﬁcation experiment shown in Figure 2, we also use the simple

Original: CNN on CIFAR10.
Revised: CNN on CIFAR10.

Original: And to keep consistent with the application setting, we set | S | = 500 and | Q | = 250 .
Revised: And to keep consistent with the application setting, we set | S | = 500 and | Q | = 250 .

Original: The Q is randomly sampled from the unlabeled set and the labeled set S is ﬁxed.
Revised: The Q is randomly sampled from the unlabeled set and the labeled set S is ﬁxed.

Original: We independently sample Q 150 times to compute the correlation between between G MSE ( S ∪ Q ) and
Revised: We independently sample Q 150 times to compute the correlation between between G MSE ( S ∪ Q ) and

Original: A ( X (cid:107) X Q , Y (cid:107) Y Q ) .
Revised: A ( X (cid:107) X Q , Y (cid:107) Y Q ) .

Original: E APPENDIX: M ORE DETAILS OF EXPERIMENTAL RESULTS E. B ASELINES
Revised: E APPENDIX: M ORE DETAILS OF EXPERIMENTAL RESULTS E. B ASELINES

Original: Random:
Revised: Random:

Original: Unlabeled data are randomly selected at each round.
Revised: Unlabeled data are randomly selected at each round.

Original: Coreset:
Revised: Coreset:

Original: This method performs a clustering over the last hidden representations in the network, and calculates the minimum distance
Revised: This method performs a clustering over the last hidden representations in the network, and calculates the minimum distance

Original: and embeddings of labeled samples.
Revised: and embeddings of labeled samples.

Original: Then data samples with the maximum distances are selected.
Revised: Then data samples with the maximum distances are selected.

Original: (Sener & Savarese, 2018).
Revised: (Sener & Savarese, 2018).

Original: Conﬁdence Sampling (Conf): The method selects b examples with smallest predicted class probability max Ki f i ( x ; θ ) (Wang & Shang, 2014).
Revised: Conﬁdence Sampling (Conf): The method selects b examples with smallest predicted class probability max Ki f i ( x ; θ ) (Wang & Shang, 2014).

Original: Margin Sampling (Marg): The bottom b examples sorted according to the example’s multi-class margin are selected.
Revised: Margin Sampling (Marg): The bottom b examples sorted according to the example’s multi-class margin are selected.

Original: The margin is deﬁned as f i ( x ; θ ) − f j
Revised: The margin is deﬁned as f i ( x ; θ ) − f j

Original: ( x ; θ ) , where i and j are the indices of the largest and second largest entries of f ( x ; θ ) (Roth & Small, 2006).
Revised: ( x ; θ ) , where i and j are the indices of the largest and second largest entries of f ( x ; θ ) (Roth & Small, 2006).

Original: Entropy: Top b samples are selected according to the entropy of the example’s predictive class probability distribution, the entropy is deﬁned as H (( f i ( x ; θ ))
Revised: Entropy: Top b samples are selected according to the entropy of the example’s predictive class probability distribution, the entropy is deﬁned as H (( f i ( x ; θ ))

Original: Ki =1 ) , where H ( p ) =
Revised: Ki =1 ) , where H ( p ) =

Original: (cid:80)
Revised: (cid:80)

Original: Ki p
Revised: Ki p

Original: i ln 1 p i (Wang & Shang, 2014).
Revised: i ln 1 p i (Wang & Shang, 2014).

Original: Active Learning by Learning (ALBL):
Revised: Active Learning by Learning (ALBL):

Original: The bandit-style meta-active learning algorithm combines Coreset and Conf (Hsu & Lin, 2015).
Revised: The bandit-style meta-active learning algorithm combines Coreset and Conf (Hsu & Lin, 2015).

Original: Batch Active learning by Diverse Gradient Embeddings (BADGE): b samples are selected by using k-means++ seeding on the gradients of the ﬁnal layer, in order to query by uncertainty and diversity.
Revised: Batch Active learning by Diverse Gradient Embeddings (BADGE): b samples are selected by using k-means++ seeding on the gradients of the ﬁnal layer, in order to query by uncertainty and diversity.

Original: (Ash et al., 2020).
Revised: (Ash et al., 2020).

Original: E. E XPERIMENT R ESULTS
Revised: E. E XPERIMENT R ESULTS

Original: The results for ResNet18, VGG11, and vanilla CNN on CIFAR10, SVHN, and Caltech101 with different batch sizes have been shown in the Figure 3 and 6.
Revised: The results for ResNet18, VGG11, and vanilla CNN on CIFAR10, SVHN, and Caltech101 with different batch sizes have been shown in the Figure 3 and 6.

Original: Note, for the large batch size setting ( b = 1000 ) on Caltech101, we set the number of query round R = 4 , in which 49.2% images will be labeled after 4 rounds.
Revised: Note, for the large batch size setting ( b = 1000 ) on Caltech101, we set the number of query round R = 4 , in which 49.2% images will be labeled after 4 rounds.

Original: E. N UMERICAL R ESULT OF M AIN E XPERIMENTS
Revised: E. N UMERICAL R ESULT OF M AIN E XPERIMENTS

Original: For the the main experiments, we report the means and standard deviations of active learning performance under different settings in the the following tables.
Revised: For the the main experiments, we report the means and standard deviations of active learning performance under different settings in the the following tables.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 9 # Query Round
Revised: 9 # Query Round

Original: T e s t
Revised: T e s t

Original: A cc u r a c y ( % )
Revised: A cc u r a c y ( % )

Original: CIFAR10, CNN,  Batch  Size:
Revised: CIFAR10, CNN, Query Batch Size:250, Initial Set Size:

Original: Random Marg
Revised: Random Marg

Original: Entropy Coreset
Revised: Entropy Coreset

Original: Conf ALBL
Revised: Conf ALBL

Original: BADGE dynamicAL         9 # Query Round
Revised: BADGE dynamicAL         9 # Query Round

Original: T e s t
Revised: T e s t

Original: A cc u r a c y ( % )
Revised: A cc u r a c y ( % )

Original: CIFAR10, ResNet,  Batch  Size:
Revised: CIFAR10, ResNet, Query Batch Size:500, Initial Set Size:

Original: Random Marg
Revised: Random Marg

Original: Entropy Coreset
Revised: Entropy Coreset

Original: Conf ALBL
Revised: Conf ALBL

Original: BADGE dynamicAL         9 # Query Round
Revised: BADGE dynamicAL         9 # Query Round

Original: T e s t
Revised: T e s t

Original: A cc u r a c y ( % )
Revised: A cc u r a c y ( % )

Original: CIFAR10, ResNet,  Batch  Size:
Revised: CIFAR10, ResNet, Query Batch Size:1000, Initial Set Size:

Original: Random Marg
Revised: Random Marg

Original: Entropy Coreset
Revised: Entropy Coreset

Original: Conf ALBL
Revised: Conf ALBL

Original: BADGE dynamicAL         9 # Query Round
Revised: BADGE dynamicAL         9 # Query Round

Original: T e s t
Revised: T e s t

Original: A cc u r a c y ( % )
Revised: A cc u r a c y ( % )

Original: SVHN, VGG,  Batch  Size:
Revised: SVHN, VGG, Query Batch Size:1000, Initial Set Size:

Original: Random Marg
Revised: Random Marg

Original: Entropy Coreset
Revised: Entropy Coreset

Original: Conf ALBL
Revised: Conf ALBL

Original: BADGE dynamicAL    4 # Query Round
Revised: BADGE dynamicAL    4 # Query Round

Original: T e s t
Revised: [Figure]

Original: A cc ur a c y ( % ) Caltech101,
Revised: u

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Random Marg
Revised: Random Marg

Original: Entropy Coreset
Revised: Entropy Coreset

Original: Conf ALBL dynamicAL
Revised: Conf ALBL dynamicAL

Original: [Figure]
Revised: Random Marg

Original: [Table]
Revised: Entropy Coreset Conf ALBL

Revised: BADGE dynamicAL

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: E. M AXIMUM M EAN D ISCREPANCY FOR M ULTIPLE R OUNDS
Revised: E. M AXIMUM M EAN D ISCREPANCY FOR M ULTIPLE R OUNDS

Original: As shown in Figure 1, the MMD term is much smaller than the B at the ﬁrst query round.
Revised: As shown in Figure 1, the MMD term is much smaller than the B at the ﬁrst query round.

Original: To better understand the relation between MMD and B for multiple query setting, we measure the MMD / B for R ≥ 2 .
Revised: To better understand the relation between MMD and B for multiple query setting, we measure the MMD / B for R ≥ 2 .

Original: As shown in Figure 7, B is much larger than MMD even multiple query rounds.
Revised: As shown in Figure 7, B is much larger than MMD even multiple query rounds.

Original: Besides, we notice that, for the ﬁrst round, the larger query batch always leads to larger MMD / B , because the sampling bias introduced by the query policy will be ampliﬁed by using large batch size.
Revised: Besides, we notice that, for the ﬁrst round, the larger query batch always leads to larger MMD / B , because the sampling bias introduced by the query policy will be ampliﬁed by using large batch size.

Original: # Query Round
Revised: # Query Round

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: MM
Revised: MM

Original: [Figure]
Revised: [Figure]

Original: / B
Revised: / B

Original: [Figure]

Original: Furthermore, we measure the MMD / B with a constant total budget size but different query rounds.
Revised: Furthermore, we measure the MMD / B with a constant total budget size but different query rounds.

Original: The result is shown in Table 10.
Revised: The result is shown in Table 10.

Original: As our expectation, spending the total query budget in one query round will induce the largest MMD / B .
Revised: As our expectation, spending the total query budget in one query round will induce the largest MMD / B .

Original: And, with more query rounds, the MMD / B will be lower.
Revised: And, with more query rounds, the MMD / B will be lower.

Original: [Table]
Revised: [Table]

Revised: E. E XPERIMENT R ESULT IN R ETRAINING S ETTING

Revised: In Section 5, we mainly focus on the non-retraining setting, in which the weights of the network obtained in the previous will be used for the next AL cycle.

Revised: To further study the effectiveness of dynamicAL in the retraining setting, in which the network will be reinitialized after each query, we compare dynamicAL with the strong baseline in the retraining active learning setting, such as

Revised: Coreset (Sener & Savarese, 2018).

Revised: We follow Ash et al.

Revised: (2020) to query samples when training accuracy be greater than 99%

Revised: and the results are shown in Table 11 and 12.

Revised: The results show that dynamicAL can still be better than or competitive with the commonly used active learning methods.

Revised: Besides, we notice that the improvement in the non-retraining setting is more signiﬁcant, because the dynamic analysis (Equation (8)) considers the change of dynamics according to the model’s current parameters

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: F L IMITATION AND F UTURE W ORK

Revised: Our proposed active learning method, dynamicAL , is designed for the small budget, non-retraining setting in active learning, which is commonly seen in practice

Revised: [Table]

Revised: and tends to be more challenging than other settings (Mittal et al., 2019).

Revised: However, as shown in Appendix E.5, if retraining from scratch is affordable in a certain application, the performance of our method is comparable with state-of-the-art techniques, such as Coreset (Sener & Savarese, 2018).

Revised: Moreover, there are some works trying to combine the techniques of self-supervised and semi-supervised learning with active learning (Mittal et al., 2019; Chan et al., 2021).

Revised: As demonstrated in the recent semi-supervised learning works (Sohn et al., 2020; Rizve et al., 2021), the data augmentation, consistency regularization and uncertaintyaware pseudo-label selection will boost the quality of the pseudo labeling that dynamicAL relies on.

Revised: Then, potentially, they can improve the performance of dynamicAL .

Revised: Besides, large initial set size and properly selected initial set (Yuan et al., 2011) also potentially improve the quality of the pseudo labeling.

Revised: Therefore, future research directions include theoretically analyzing the effect of semi-supervised learning technologies for active learning and empirically verifying the effectiveness of combining dynamicAL and semi-supervised learning technologies in retraining and non-retraining active learning settings.