Original: D ATA V ALUATION W ITHOUT T RAINING OF A M ODEL
Revised: D ATA V ALUATION W ITHOUT T RAINING OF A M ODEL

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Many recent works on understanding deep learning try to quantify how much individual data instances inﬂuence the optimization and generalization of a model, either by analyzing the behavior of the model during training or by measuring the performance gap of the model when the instance is removed from the dataset.
Revised: Many recent works on understanding deep learning try to quantify how much individual data instances inﬂuence the optimization and generalization of a model, either by analyzing the behavior of the model during training or by measuring the performance gap of the model when the instance is removed from the dataset.

Original: Such approaches reveal characteristics and importance of individual instances, which may provide useful information in diagnosing and improving deep learning.
Revised: Such approaches reveal characteristics and importance of individual instances, which may provide useful information in diagnosing and improving deep learning.

Original: However, most of the existing works on data valuation require actual training of a model, which often demands high-computational cost.
Revised: However, most of the existing works on data valuation require actual training of a model, which often demands high-computational cost.

Original: In this paper, we provide a training-free data valuation score, called complexity-gap score , which is a datacentric score to quantify the inﬂuence of individual instances in generalization of two-layer overparameterized neural networks.
Revised: In this paper, we provide a training-free data valuation score, called complexity-gap score , which is a datacentric score to quantify the inﬂuence of individual instances in generalization of two-layer overparameterized neural networks.

Original: The proposed score can quantify irregularity of the instances and measure how much each data instance contributes in the total movement of the network parameters during training.
Revised: The proposed score can quantify irregularity of the instances and measure how much each data instance contributes in the total movement of the network parameters during training.

Original: We theoretically analyze and empirically demonstrate the effectiveness of the complexity-gap score in ﬁnding ‘irregular or mislabeled’ data instances, and also provide applications of the score in analyzing datasets and diagnosing training dynamics.
Revised: We theoretically analyze and empirically demonstrate the effectiveness of the complexity-gap score in ﬁnding ‘irregular or mislabeled’ data instances, and also provide applications of the score in analyzing datasets and diagnosing training dynamics.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Creation of large datasets has driven recent development of deep learning in diverse applications including computer vision (Krizhevsky et al., 2012; Dosovitskiy et al., 2021), natural language processing (Vaswani et al., 2017; Brown et al., 2020) and reinforcement learning (Mnih et al., 2015;
Revised: Creation of large datasets has driven recent development of deep learning in diverse applications including computer vision (Krizhevsky et al., 2012; Dosovitskiy et al., 2021), natural language processing (Vaswani et al., 2017; Brown et al., 2020) and reinforcement learning (Mnih et al., 2015;

Original: Silver et al., 2016).
Revised: Silver et al., 2016).

Original: To utilize the dataset in a more efﬁcient and effective manner, many recent works have attempted to understand the role of individual data instances in training and generalization of neural networks.
Revised: To utilize the dataset in a more efﬁcient and effective manner, many recent works have attempted to understand the role of individual data instances in training and generalization of neural networks.

Original: As an example, in (Ghorbani & Zou, 2019), a metric to quantify the contribution of each training instance in achieving a high test accuracy was analyzed under the assumption that not only the training data but also the test data is available.
Revised: As an example, in (Ghorbani & Zou, 2019), a metric to quantify the contribution of each training instance in achieving a high test accuracy was analyzed under the assumption that not only the training data but also the test data is available.

Original: Jiang et al.
Revised: Jiang et al.

Original: (2021) deﬁned a score to identify irregular examples that need to be memorized during training, in order for the model to accurately predict the class of the example.
Revised: (2021) deﬁned a score to identify irregular examples that need to be memorized during training, in order for the model to accurately predict the class of the example.

Original: Swayamdipta et al.
Revised: Swayamdipta et al.

Original: (2020), on the other hand, analyzed the characteristics of data instances with respect to their role in out-of-distribution generalizations.
Revised: (2020), on the other hand, analyzed the characteristics of data instances with respect to their role in out-of-distribution generalizations.

Original: All these previous methods for data valuation require actual training of a model to quantify the role of individual instances at the model.
Revised: All these previous methods for data valuation require actual training of a model to quantify the role of individual instances at the model.

Original: Thus, the valuation itself often requires high-computational cost, which may contradict some motivations of data valuation.
Revised: Thus, the valuation itself often requires high-computational cost, which may contradict some motivations of data valuation.

Original: For example, in (Ghorbani & Zou, 2019; Jiang et al., 2021), to examine the effect of individual data instances in training, one needs to train a model repeatedly while eliminating each instance or subsets of instances, which requires training of a model at least the number of times proportional to the number of training instances.
Revised: For example, in (Ghorbani & Zou, 2019; Jiang et al., 2021), to examine the effect of individual data instances in training, one needs to train a model repeatedly while eliminating each instance or subsets of instances, which requires training of a model at least the number of times proportional to the number of training instances.

Original: In (Swayamdipta et al., 2020; Toneva et al., 2019), on the other hand, training dynamics–the behavior of a model on each instance throughout the training–is analyzed to categorize data instances, which also requires the training of a model with the full dataset.
Revised: In (Swayamdipta et al., 2020; Toneva et al., 2019), on the other hand, training dynamics–the behavior of a model on each instance throughout the training–is analyzed to categorize data instances, which also requires the training of a model with the full dataset.

Original: When the motivation for data valuation lies in pruning less important examples to save computational cost for training, the previous valuation methods might not be suitable, since they require the training of a model with the full dataset before one can ﬁgure out ‘important’ instances and possibly prune the rest of the examples.
Revised: When the motivation for data valuation lies in pruning less important examples to save computational cost for training, the previous valuation methods might not be suitable, since they require the training of a model with the full dataset before one can ﬁgure out ‘important’ instances and possibly prune the rest of the examples.

Original: In this paper, our main contribution is on deﬁning a training-free data valuation score, which can be directly computed from data and can effectively quantify the impact of individual instances in optimization and generalization of neural networks.
Revised: In this paper, our main contribution is on deﬁning a training-free data valuation score, which can be directly computed from data and can effectively quantify the impact of individual instances in optimization and generalization of neural networks.

Original: The proposed score, called complexity-gap score , measures the gap in data complexity where a certain data instance is removed from the full dataset.
Revised: The proposed score, called complexity-gap score , measures the gap in data complexity where a certain data instance is removed from the full dataset.

Original: The data complexity measure was originally introduced in Arora et al.
Revised: The data complexity measure was originally introduced in Arora et al.

Original: (2019) to quantify the com- plexity of the full dataset, which was used in bounding the generalization error of overparameterized two-layer neural networks trained by gradient descent.
Revised: (2019) to quantify the com- plexity of the full dataset, which was used in bounding the generalization error of overparameterized two-layer neural networks trained by gradient descent.

Original: Different from that work, where the complexity of the full dataset was main concern, our focus is on decomposing the effect of individual data instances in the training, and thus we newly introduce a complexity gap score (CG-score).
Revised: Different from that work, where the complexity of the full dataset was main concern, our focus is on decomposing the effect of individual data instances in the training, and thus we newly introduce a complexity gap score (CG-score).

Original: We theoretically analyze and empirically demonstrate that the CG-score can quantify ‘irregularity’ of instances within each class, and thus can be used in identifying atypical examples, either due to the inherent irregularity of the instance or mislabeled classiﬁcation.
Revised: We theoretically analyze and empirically demonstrate that the CG-score can quantify ‘irregularity’ of instances within each class, and thus can be used in identifying atypical examples, either due to the inherent irregularity of the instance or mislabeled classiﬁcation.

Original: We also demonstrate that the proposed score has a close relation to ‘learning difﬁcult’ of the instances by analyzing the training dynamics of data instances.
Revised: We also demonstrate that the proposed score has a close relation to ‘learning difﬁculty’ of the instances by analyzing the training dynamics of data instances.

Original: Our key contributions are as below:
Revised: Our key contributions are as below:

Original: • Training-free data valuation: Different from previous methods for data valuation, most of which leverage the information from training itself, we provide a training-free data valuation score, CG-score, which is the data-centric score to quantify the effect of individual data instances in optimization and generalization of neural networks.
Revised: • Training-free data valuation: Different from previous methods for data valuation, most of which leverage the information from training itself, we provide a training-free data valuation score, CG-score, which is the data-centric score to quantify the effect of individual data instances in optimization and generalization of neural networks.

Original: • Geometric interpretation: We provide theoretical analysis that the CG-score can measure irregularity of instances within each class,
Revised: • Geometric interpretation: We provide theoretical analysis that the CG-score can measure irregularity of instances within each class,

Original: i
Revised: i

Original: e., how much each instance represents the instances of the same class and how much it is different from those of other classes.
Revised: e., how much each instance represents the instances of the same class and how much it is different from those of other classes.

Original: • Effectiveness of the score : We empirically demonstrate the effectiveness of the CG-score in data valuation.
Revised: • Effectiveness of the score : We empirically demonstrate the effectiveness of the CG-score in data valuation.

Original: We show that pruning data instances with small CG-score do not significantly degrade the generalization capability of a model, e
Revised: We show that pruning data instances with small CG-score do not significantly degrade the generalization capability of a model, e

Original: g., for CIFAR-10 we can prune 40% of the data with only 1% of drop in test accuracy.
Revised: g., for CIFAR-10 we can prune 40% of the data with only 1% of drop in test accuracy.

Original: Our scoring method is especially useful in data pruning, since different from other scores, which require the training with the full dataset, our method does not require any training of a model.
Revised: Our scoring method is especially useful in data pruning, since different from other scores, which require the training with the full dataset, our method does not require any training of a model.

Original: • Application of the score
Revised: • Application of the score

Original: : We provide potential applications of the CG-score in analyzing datasets and training dynamics.
Revised: : We provide potential applications of the CG-score in analyzing datasets and training dynamics.

Original: We analyze the histograms of the CG-score for various datasets to demonstrate that
Revised: We analyze the histograms of the CG-score for various datasets to demonstrate that

Original: the CG-score can measure irregularity of the instances.
Revised: the CG-score can measure irregularity of the instances.

Original: We also demonstrate that the instances with higher CG-score are ‘difﬁcult’ examples, which are learned slowly by the models, by comparing the loss and test accuracy curves and the evolution of Neural Tangent Kernel (NTK) submatrices of lowest/highest-scoring groups.
Revised: We also demonstrate that the instances with higher CG-score are ‘difﬁcult’ examples, which are learned slowly by the models, by comparing the loss and test accuracy curves and the evolution of Neural Tangent Kernel (NTK) submatrices of lowest/highest-scoring groups.

Original: Related Works Different from many existing works on deep learning analysis, which compute the impact of the training data as a whole, some recent works have focused on understanding the impact of individual data instances in training of neural networks.
Revised: Related Works Different from many existing works where the impact of datasets on model training is analyzed as a whole, some recent works have focused on understanding the impact of individual data instances.

Original: Ghorbani & Zou (2019) deﬁned a method for data valuation, called Data Shapley, to evaluate the value of each data instance, by measuring the average gap in performances when an instance is held-out from any subsets of a given training data.
Revised: Ghorbani & Zou (2019) deﬁned a method for data valuation, called Data Shapley, to evaluate the value of each data instance, by measuring the average gap in performances when an instance is held-out from any subsets of a given training data.

Original: Jiang et al.
Revised: Jiang et al.

Original: (2021) deﬁned consistency score (C-score) of each instance by estimating the prediction accuracy of the instance attained by the model trained with the full dataset except the instance.
Revised: (2021) deﬁned consistency score (C-score) of each instance by estimating the prediction accuracy of the instance attained by the model trained with the full dataset except the instance.

Original: Both Data Shapley and C-score require multiple trainings of a model to compute the scores.
Revised: Both Data Shapley and Cscore require multiple trainings of a model to compute the scores.

Original: Another main stream of methodsfor data valuation uses theinformation from training dynamics to identify ‘difﬁcult’ instances for classiﬁcation, either due to irregularity or mislabeling, by measuring different forms of conﬁdence or stability  in the decision of the networks throughout the training (Toneva et al., 2019; Swayamdipta et al.,  2020).
Revised: Another main stream of methods uses the training dynamics to identify ‘difﬁcult’ instances for classiﬁcation, either due to irregularity or mislabeling, by measuring different forms of conﬁdence, stability or inﬂuence in the decision of the networks throughout the training (Toneva et al., 2019; Swayamdipta et al., 2020; Pruthi et al., 2020).

Original: Some recent work also uses the information from the network during training, by deﬁning the computational difﬁculty of an instance  as the number of hidden layers after which the networks’ prediction coincides with the prediction at the output (Baldock et al., 2021).
Revised: In (Baldock et al., 2021), the computational difﬁculty of an instance is deﬁned as the number of hidden layers after which the networks’ prediction coincides with the prediction at the output.

Original: With application of robust learning, there also exist some works that quantify the difﬁculty of each instance by a ‘margin’ from the decision boundary, which can be measured by the amount of noise sufﬁcient to alter the decision of the network when the noise is added to the instance (Zhang et al., 2021). 
Revised: With application of robust learning, there also exist some works that quantify the difﬁculty of each instance by a ‘margin’ from the decision boundary (Zhang et al., 2021). CRAIG (Mirzasoleiman et al., 2020) ﬁnds valuable subsets of data as coresets that preserve the gradient of the total loss.

Original: All these previous methods are demonstrated to be effective in at least one or more applications of data valuation, including data pruning (Paul et al., 2021; Swayamdipta et al., 2020; Agarwal et al., 2022;Feldman & Zhang, 2020), importance-based weighted sampling (Chang et al., 2017; Koh & Liang, 2017), noise ﬁltering (Li et al., 2020; Lee et al., 2019b;
Revised: All these previous methods are demonstrated to be effective in at least one or more applications of data valuation, including data pruning (Paul et al., 2021; Swayamdipta et al., 2020; Agarwal et al., 2022;

Original:  Kim et al., 2021), robust learning (Ren et al., 2018; Pleiss et al., 2020), or out-of-distribution generalizations (Swayamdipta et al., 2020).
Revised: Feldman & Zhang, 2020), importance-based weighted sampling (Chang et al., 2017; Koh & Liang, 2017), noise ﬁltering (Li et al., 2020; Lee et al., 2019b; Kim et al., 2021), robust learning (Ren et al., 2018; Pleiss et al., 2020), or out-of-distribution generalizations (Swayamdipta et al., 2020).

Original: However, all these methods require the training of a model with the full dataset (at least for a few optimization steps) or possibly for as many times as the number of instances in the training dataset.
Revised: However, all these methods require the training of a model with the full dataset (at least for a few optimization steps). Some recent works do the data valuation at the initialization of models (Wu et al., 2022) or by assuming data distributions (Kwon et al., 2021), but they often require relatively high computational cost or additional assumptions on data distributions.

Original: Our data valuation method, on the other hand, is a data-centric method, which does not require training of a model.
Revised: Our data valuation method, on the other hand, is a data-centric method that can be efﬁciently calculated from data only without training of a model.

Original: We demonstrate that with a simple scoring method,which can be efﬁciently calculated from data only, we can effectively analyze the impact of individual data instance in optimization and generalization of neural networks.
Revised: We demonstrate that with our scoring method, we can effectively analyze the impact of individual data instance in optimization and generalization of neural networks.

Original:  C OMPLEXITY -G AP S CORE : D ATA V ALUATION WITHOUT T RAINING
Revised:  C OMPLEXITY -G AP S CORE : D ATA V ALUATION WITHOUT T RAINING

Original: In this section, we introduce a new data valuation score, called complexity-gap score , based on the analysis of overparameterized two-layer neural networks from Arora et al.
Revised: In this section, we introduce a new data valuation score, called complexity-gap score , based on the analysis of overparameterized two-layer neural networks from Arora et al.

Original: 2. P RELIMINARIES : DATA COMPLEXITY MEASURE IN TWO - LAYER NEURAL NETWORKS
Revised: 2. P RELIMINARIES : DATA COMPLEXITY MEASURE IN TWO - LAYER NEURAL NETWORKS

Original: We ﬁrst review the result from Arora et al.
Revised: We ﬁrst review the result from Arora et al.

Original: (2019), where a two-layer neural network trained by randomly initialized gradient descent is analyzed.
Revised: (2019), where a two-layer neural network trained by randomly initialized gradient descent is analyzed.

Original: Following the notations from Arora et al.
Revised: Following the notations from Arora et al.

Original: (2019), we consider a two-layer ReLU activated neural network having m neurons in the hidden layer, of which the output is f W , a ( x ) = 1 √ m (cid:80) mr =1 a r σ ( w (cid:62) r x ) where x ∈ R
Revised: (2019), we consider a two-layer ReLU activated neural network having m neurons in the hidden layer, of which the output is f W , a ( x ) = 1 √ m (cid:80) mr =1 a r σ ( w (cid:62) r x ) where x ∈ R

Original: d is the input, w 1 , . . . , w m ∈ R d are weight vectors in the ﬁrst layer, a 1 , . .
Revised: d is the input, w 1 , . . . , w m ∈ R d are weight vectors in the ﬁrst layer, a 1 , . .

Original: . , a m ∈ R are weights in the second layer, and σ ( x ) = max(0 , x ) is the ReLU activation function.
Revised: . , a m ∈ R are weights in the second layer, and σ ( x ) = max(0 , x ) is the ReLU activation function.

Original: We denote W = ( w 1 , . . .
Revised: We denote W = ( w 1 , . . .

Original: , w m ) ∈ R
Revised: , w m ) ∈ R

Original: d × m and a = ( a 1 , . .
Revised: d × m and a = ( a 1 , . .

Original: . , a m ) (cid:62) ∈ R m .
Revised: . , a m ) (cid:62) ∈ R m .

Original: It is assumed that the network parameters are randomly initialized as w r (0) ∼ N (0 , κ 2 I d × d ) and a r ∼ unif ( {− 1 , 1 } ) , ∀ r ∈ [ m ] , where κ ∈ (0 , 1] is the size of random initialization.
Revised: It is assumed that the network parameters are randomly initialized as w r (0) ∼ N (0 , κ 2 I d × d ) and a r ∼ unif ( {− 1 , 1 } ) , ∀ r ∈ [ m ] , where κ ∈ (0 , 1] is the size of random initialization.

Original: The second layer a is then ﬁxed and only the ﬁrst layer W is optimized through gradient descent (GD) to minimize the quadratic loss, Φ( W ) = 12 (cid:80) n i =1 ( y i − u i ) 2 where u i = f W , a ( x i ) and { ( x i , y i ) } ni =1 is the dataset drawn i.i.d.
Revised: The second layer a is then ﬁxed and only the ﬁrst layer W is optimized through gradient descent (GD) to minimize the quadratic loss, Φ( W ) = 12 (cid:80) n i =1 ( y i − u i ) 2 where u i = f W , a ( x i ) and { ( x i , y i ) } ni =1 is the dataset drawn i.i.d.

Original: from an underlying distribution D .
Revised: from an underlying distribution D .

Original: The GD update rule can be written as w r ( k +1)
Revised: The GD update rule can be written as w r ( k +1)

Original: [Equation]
Revised: [Equation]

Original: where η > 0 is the learning rate.
Revised: where η > 0 is the learning rate.

Original: The output of the network for the input x i at the k -th iteration is denoted by u i ( k ) =
Revised: The output of the network for the input x i at the k -th iteration is denoted by u i ( k ) =

Original: f W ( k ) , a ( x i ) .
Revised: f W ( k ) , a ( x i ) .

Original: For simplicity, it is assumed that (cid:107) x (cid:107) 2 = 

Revised: For simplicity, it is assumed that (cid:107) x (cid:107) 2 =

Original: and | y | ≤ 1 .
Revised: and | y | ≤ 1 .

Original: The data complexity measure governing the training of the two-layer ReLU activated neural network is deﬁned in terms of the following Gram matrix H ∞ ∈ R n × n associated with ReLU activation:
Revised: The data complexity measure governing the training of the two-layer ReLU activated neural network is deﬁned in terms of the following Gram matrix H ∞ ∈ R n × n associated with ReLU activation:

Original: [Equation]
Revised: [Equation]

Original: In Arora et al.
Revised: In Arora et al.

Original: (2019), a complexity measure of data was deﬁned as y (cid:62) ( H ∞ ) − 1 y where
Revised: (2019), a complexity measure of data was deﬁned as y (cid:62) ( H ∞ ) − 1 y where

Original: y =
Revised: y =

Original: ( y 1 , . .
Revised: ( y 1 , . .

Original: y n ) , and it was shown that this measure bounds the total movement of all neurons in W from their random initialization.
Revised: y n ) , and it was shown that this measure bounds the total movement of all neurons in W from their random initialization.

Original: Moreover, the data complexity measure bounds the generalization error by restricting the Rademacher complexity of the resulting function class.
Revised: Moreover, the data complexity measure bounds the generalization error by restricting the Rademacher complexity of the resulting function class.

Original: In the following, we write the eigen-decomposition of H ∞ as H ∞ =
Revised: In the following, we write the eigen-decomposition of H ∞ as H ∞ =

Original: (cid:80) ni
Revised: (cid:80) ni

Original: =

Revised: =

Original: λ i v i v (cid:62) i where λ i ’s are ordered such that λ 1 ≥ λ 2 ≥ . . .
Revised: λ i v i v (cid:62) i where λ i ’s are ordered such that λ 1 ≥ λ 2 ≥ . . .

Original: λ n .
Revised: λ n .

Original: Further, assuming λ n ≥ λ 0 > 0 , we can write ( H ∞ ) − 1 = (cid:80) ni
Revised: Further, assuming λ n ≥ λ 0 > 0 , we can write ( H ∞ ) − 1 = (cid:80) ni

Original: =

Revised: =

Original: ( λ i ) − 1 v i v (cid:62) i .
Revised: ( λ i ) − 1 v i v (cid:62) i .

Original: Theorem 1 (Informal version of (Arora et al., 2019)) .
Revised: Theorem 1 (Informal version of (Arora et al., 2019)) .

Original: Assume that λ min ( H ∞ )
Revised: Assume that λ min ( H ∞ )

Original: =
Revised: =

Original: λ n ≥ λ 0 > 0 .
Revised: λ n ≥ λ 0 > 0 .

Original: For sufﬁciently large width m , sufﬁciently small learning rate η > 0 and sufﬁciently small random initialization κ > 0 , with probability at least 1 − δ over the random initialization, we have
Revised: For sufﬁciently large width m , sufﬁciently small learning rate η > 0 and sufﬁciently small random initialization κ > 0 , with probability at least 1 − δ over the random initialization, we have

Original: a) Bound in loss : (cid:107) y − u ( k ) (cid:107) 2 =
Revised: a) Bound in loss : (cid:107) y − u ( k ) (cid:107) 2 =

Original: [Equation]
Revised: [Equation]

Original: n (cid:88)
Revised: n (cid:88)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:62)
Revised: (cid:62)

Original: [Equation]
Revised: [Equation]

Original: y ) 

Revised: y )

Original: + small constant , b) Bound in total movement of neurons: (cid:107) W ( k ) − W (0) (cid:107) F ≤ (cid:113) y (cid:62) ( H ∞ ) − 1 y + small constant , c)
Revised: + small constant , b) Bound in total movement of neurons: (cid:107) W ( k ) − W (0) (cid:107) F ≤ (cid:113) y (cid:62) ( H ∞ ) − 1 y + small constant , c)

Original: Bound in population loss: E ( x ,y )
Revised: Bound in population loss: E ( x ,y )

Original: ∼D [ l ( f W ( k ) , a ( x ) , y )] ≤
Revised: ∼D [ l ( f W ( k ) , a ( x ) , y )] ≤

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: a)
Revised: a)

Original: and b hold for all iteration k ≥ 0 of GD, and c holds for k ≥ Ω (
Revised: and b hold for all iteration k ≥ 0 of GD, and c holds for k ≥ Ω (

Original: ( ηλ
Revised: ( ηλ

Original: ) log n δ )) .
Revised: ) log n δ )) .

Original: The above theorem shows that if the label vector y is aligned with top eigenvectors of H ∞ , i.e., ( v (cid:62) i y ) is large for large λ i , then the loss decreases quickly and the total movement of neurons from their random initialization is small, which implies a small generalization error.
Revised: The above theorem shows that if the label vector y is aligned with top eigenvectors of H ∞ , i.e., ( v (cid:62) i y ) is large for large λ i , then the loss decreases quickly and the total movement of neurons from their random initialization is small, which implies a small generalization error.

Original: Thus, the data complexity measure y (cid:62) ( H ∞ ) − 1 y captures the complexity of data governing both the optimization and generalization of the overparameterized two-layer neural networks.
Revised: Thus, the data complexity measure y (cid:62) ( H ∞ ) − 1 y captures the complexity of data governing both the optimization and generalization of the overparameterized two-layer neural networks.

Original: However, this quantity captures the complexity of the overall data.
Revised: However, this quantity captures the complexity of the overall data.

Original: To decompose the individual data instances, in the next section, we newly deﬁne a complexity-gap score .
Revised: To decompose the individual data instances, in the next section, we newly deﬁne a complexity-gap score .

Original: 2. C OMPLEXITY - GAP SCORE AND TRAINING DYNAMICS
Revised: 2. C OMPLEXITY - GAP SCORE AND TRAINING DYNAMICS

Original: We deﬁne the complexity-gap score (CG-score) of ( x i , y i ) as the difference between the data complexity measure when ( x i , y i ) is removed from a given dataset { ( x i , y i ) } ni =1 :
Revised: We deﬁne the complexity-gap score (CG-score) of ( x i , y i ) as the difference between the data complexity measure when ( x i , y i ) is removed from a given dataset { ( x i , y i ) } ni =1 :

Original: [Equation]
Revised: [Equation]

Original: where y − i is the label vector except the i -th sample point and H ∞− i is the ( n − 1) ×
Revised: where y − i is the label vector except the i -th sample point and H ∞− i is the ( n − 1) ×

Original: ( n − 1) matrix obtained by removing the i -th row and column of H ∞ .
Revised: ( n − 1) matrix obtained by removing the i -th row and column of H ∞ .

Original: We ﬁrst emphasize that the proposed score can be easily calculated from given data without the need of training neural networks, as opposed to other data valuation scores requiring either a trained neural network or statistics calculated from training dynamics.
Revised: We ﬁrst emphasize that the proposed score can be easily calculated from given data without the need of training neural networks, as opposed to other data valuation scores requiring either a trained neural network or statistics calculated from training dynamics.

Original: Yet, the proposed score captures two important properties on the training and generalization of data instance, implied by Theorem 1:
Revised: Yet, the proposed score captures two important properties on the training and generalization of data instance, implied by Theorem 1:

Original: An instance ( x i , y i ) with a large CG-score is a ‘difﬁcult’ example, in the sense that removing it from the dataset reduces the generalization error bound by a large amount, which implies that the dataset without ( x i , y i ) is much easier to be learned and generalized.
Revised: An instance ( x i , y i ) with a large CG-score is a ‘difﬁcult’ example, in the sense that removing it from the dataset reduces the generalization error bound by a large amount, which implies that the dataset without ( x i , y i ) is much easier to be learned and generalized.

Original: An instance ( x i , y i ) with a larger CG-score contributes more on the optimization and drives the total movement of neurons by a larger amount, measured by (cid:107) W ( k ) − W (0) (cid:107) F .
Revised: An instance ( x i , y i ) with a larger CG-score contributes more on the optimization and drives the total movement of neurons by a larger amount, measured by (cid:107) W ( k ) − W (0) (cid:107) F .

Original: We next discuss the computational complexity of calculating the CG-score.
Revised: We next discuss the computational complexity of calculating the CG-score.

Original: To calculate { CG( i ) } ni =1 , we need to have the inversion of matrices H ∞ and H ∞− i for all i ∈ [ n ] , which requires O ( n 4 ) complexity when we use general O ( n 3 ) -complexity algorithm for the matrix inversion.
Revised: To calculate { CG( i ) } ni =1 , we need to have the inversion of matrices H ∞ and H ∞− i for all i ∈ [ n ] , which requires O ( n 4 ) complexity when we use general O ( n 3 ) -complexity algorithm for the matrix inversion.

Original: By using Schur complement, however, we can reduce this complexity to O ( n 3 ) .
Revised: By using Schur complement, however, we can reduce this complexity to O ( n 3 ) .

Original: Without loss of generality, we can assume i = n .
Revised: Without loss of generality, we can assume i = n .

Original: Denote H ∞ and ( H ∞ ) − 1 by
Revised: Denote H ∞ and ( H ∞ ) − 1 by

Original: [Equation]
Revised: [Equation]

Original: where H ∞ n − 1 , ( H ∞ ) − 1 n − 1 ∈ R ( n − 1) × ( n − 1) , g i , h i ∈ R n − 1 and c i , d i ∈ R .
Revised: where H ∞ n − 1 , ( H ∞ ) − 1 n − 1 ∈ R ( n − 1) × ( n − 1) , g i , h i ∈ R n − 1 and c i , d i ∈ R .

Original: From H ∞ ( H ∞ ) − 1 =
Revised: From H ∞ ( H ∞ ) − 1 =

Original: I n , we have g (cid:62) i ( H ∞ ) − 1 n − 

Revised: I n , we have g (cid:62) i ( H ∞ ) − 1 n −

Original: + c i h (cid:62) i = 0 , i.e., h (cid:62) i =
Revised: + c i h (cid:62) i = 0 , i.e., h (cid:62) i =

Original: − c − 

Revised: − c −

Original: i g (cid:62) i ( H ∞ ) − 1 n − 1 .
Revised: i g (cid:62) i ( H ∞ ) − 1 n − 1 .

Original: By Shur complement, ( H ∞− i ) − 1 , which is equal to ( H ∞ n − 1 ) − 1 for i = n , can be calculated as
Revised: By Schur complement, ( H ∞− i ) − 1 , which is equal to ( H ∞ n − 1 ) − 1 for i = n , can be calculated as

Original: [Equation]
Revised: [Equation]

Original: Since we have
Revised: Since we have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: the CG-score, CG( i ) in equation 2, can be calculated by
Revised: the CG-score, CG( i ) in equation 2, can be calculated by

Original: [Equation]
Revised: [Equation]

Original: Thus, CG( i ) can be calculated by the n -th column ( h i , d i ) of ( H ∞ ) − 1 , without the need of calculating ( H ∞− i ) − 1 when i = n .
Revised: Thus, CG( i ) can be calculated by the n -th column ( h i , d i ) of ( H ∞ ) − 1 , without the need of calculating ( H ∞− i ) − 1 when i = n .

Original: The case for general i (cid:54) = n can also be readily solved by permuting the i -th rwo and column of H ∞ into the last positions.
Revised: The case for general i (cid:54) = n can also be readily solved by permuting the i -th row and column of H ∞ into the last positions.

Original: Correlation to other scores We show relation between our score and other data valuation scores that require the training of neural networks.
Revised: Correlation to other scores We show relation between our score and other data valuation scores that require the training of neural networks.

Original: Toneva et al.
Revised: Toneva et al.

Original: (2019) deﬁne ‘the forgetting score’ for each training example as the number of times during training the decision of that sample switches from a correct one to incorrect one.
Revised: (2019) deﬁne ‘the forgetting score’ for each training example as the number of times during training the decision of that sample switches from a correct one to incorrect one.

Original: Paul et al.
Revised: Paul et al.

Original: (2021), on the other hand, suggest the GraNd score, which is the expected loss gradient norm E [ (cid:107)∇ W ( k ) l ( u ( k ) , y ) (cid:107) ] , to bound the contribution of each training example to the decrease of loss on any other example over a single gradient step.
Revised: (2021), on the other hand, suggest the GraNd score, which is the expected loss gradient norm E [ (cid:107)∇ W ( k ) l ( u ( k ) , y ) (cid:107) ] , to bound the contribution of each training example to the decrease of loss on any other example over a single gradient step.

Original: The
Revised: The

Original: GraNd score is further approximated (under some assumptions) by the EL2N score, deﬁned to be E [ | y − u ( k ) | ] where u ( k ) is the output of the neural network for the sample ( x, y ) at the k -th step.
Revised: GraNd score is further approximated (under some assumptions) by the EL2N score, deﬁned to be E [ | y − u ( k ) | ] where u ( k ) is the output of the neural network for the sample ( x, y ) at the k -th step.

Original: Since | y − u ( k ) | , if rescaled, is an upper bound on 0–1 loss, (cid:80) k | y − u ( k ) | upper bounds forgetting score after rescaling.
Revised: Since | y − u ( k ) | , if rescaled, is an upper bound on 0–1 loss, (cid:80) k | y − u ( k ) | upper bounds forgetting score after rescaling.

Original: Thus, an example with a high forgetting score will also have a high GraND score and high EL2N score averaged over multiple time steps.
Revised: Thus, an example with a high forgetting score will also have a high GraND score and high EL2N score averaged over multiple time steps.

Original: We next relate our complexity-gap score to (cid:80) k
Revised: We next relate our complexity-gap score to (cid:80) k

Original: ( y − u ( k )) , and thus to all the three previous scores deﬁned using training dynamics.
Revised: ( y − u ( k )) , and thus to all the three previous scores deﬁned using training dynamics.

Original: [Table]
Revised: [Table]

Original: Arora et al.
Revised: Arora et al.

Original: (2019) show that for the overparameterized two-layer networks trained by GD, the gap between the label vector and the network output at the step k can be approximated as y − u ( k )
Revised: (2019) show that for the overparameterized two-layer networks trained by GD, the gap between the label vector and the network output at the step k can be approximated as y − u ( k )

Original: ≈ ( I − η H ∞ )
Revised: ≈ ( I − η H ∞ )

Original: k y .
Revised: k y .

Original: By summing both sides over k , we get (cid:80) 
Revised: By summing both sides over k , we get (cid:80) ∞ k

Original: [Table]
Revised: [Equation]

Revised: y − u ( k ) ≈ (cid:80) ∞

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: H ∞ )

Original: [Table] [Equation] [Table] [Equation]
Revised: k y

Original: [Table]
Revised: =

Original: η ( H ∞ ) − 1 y .
Revised: η ( H ∞ ) − 1 y .

Original: Without loss of generality, consider i = n .
Revised: Without loss of generality, consider i = n .

Original: Then, the accumulated difference between y i and u i ( k ) over k can be approximated as
Revised: Then, the accumulated difference between y i and u i ( k ) over k can be approximated as

Original: [Equation] 
Revised: [Equation] [Equation]

Original: Note that both the right-hand side of equation 7 and CG( i ) in equation 6 depend on the term (cid:0) y
Revised: Note that both the right-hand side of equation 7 and CG( i ) in equation 6 depend on the term (cid:0) y

Original: (cid:62)− i h i / √ d i + y i √
Revised: (cid:62)− i h i / √ d i + y i √

Original: d i (cid:1) .
Revised: d i (cid:1) .

Original: Thus, we can expect that CG( i ) will be correlated with the scores related to training dynamics, including the forgetting score, GraND score and EL2N score.
Revised: Thus, we can expect that CG( i ) will be correlated with the scores related to training dynamics, including the forgetting score, GraND score and EL2N score.

Original: Different from those scores, our score can be directly calculated from the data without training of a model.
Revised: Different from those scores, our score can be directly calculated from the data without training of a model.

Original: Inversion of H ∞ is an effective step In the deﬁnition of CG( i ) in equation 2, we use the inverse of H ∞ to measure the alignment of the eigenvectors of H ∞ with the label vector y .
Revised: Inversion of H ∞ is an effective step In the deﬁnition of CG( i ) in equation 2, we use the inverse of H ∞ to measure the alignment of the eigenvectors of H ∞ with the label vector y .

Original: One could suggest another score that directly uses H ∞ instead of ( H ∞ ) − 1 , which can save the computation for inversion.
Revised: One could suggest another score that directly uses H ∞ instead of ( H ∞ ) − 1 , which can save the computation for inversion.

Original: However, we argue that the score calculated by using ( H ∞ ) − 1 includes more information than the score calculated by H ∞ due to the reason described below.
Revised: However, we argue that the score calculated by using ( H ∞ ) − 1 includes more information than the score calculated by H ∞ due to the reason described below.

Original: Let us deﬁne CG (cid:48)
Revised: Let us deﬁne CG (cid:48)

Original: ( i ) :
Revised: ( i ) :

Original: =
Revised: =

Original: y (cid:62) H ∞ y − y (cid:62)− i H ∞− i y − i .
Revised: y (cid:62) H ∞ y − y (cid:62)− i H ∞− i y − i .

Original: Without loss of generality, assume i = n .
Revised: Without loss of generality, assume i = n .

Original: Then, CG (cid:48)
Revised: Then, CG (cid:48)

Original: ( i ) =
Revised: ( i ) =

Original: y i ( g (cid:62) i
Revised: y i ( g (cid:62) i

Original: y − i )
Revised: y − i )

Original: + y 

Revised: + y

Original: i c i where g i =
Revised: i c i where g i =

Original: ( H ∞ ) 1:( n − 1) ,n and c i =
Revised: ( H ∞ ) 1:( n − 1) ,n and c i =

Original: H ∞ n,n as deﬁned in equation 3.
Revised: H ∞ n,n as deﬁned in equation 15.

Original: Since c i = 1 / 2 for all i ∈ [ n ] from the deﬁnition of H ∞ in equation 1 and y 2 i = 1 for y i = ± 1 , we have CG (cid:48)
Revised: Since c i = 1 / 2 for all i ∈ [ n ] from the deﬁnition of H ∞ in equation 1 and y 2 i = 1 for y i = ± 1 , we have CG (cid:48)

Original: ( i ) =
Revised: ( i ) =

Original: ( y i ( H ∞ y )
Revised: ( y i ( H ∞ y )

Original: i − 1 / 2) + 1 / 2 .
Revised: i − 1 / 2) + 1 / 2 .

Original: By using the approximation y − u ( k )
Revised: By using the approximation y − u ( k )

Original: ≈ ( I − η H ∞ )
Revised: ≈ ( I − η H ∞ )

Original: k y from Arora et al.
Revised: k y from Arora et al.

Original: (2019), we have y − u (1)
Revised: (2019), we have y − u (1)

Original: ≈ y − η H ∞
Revised: ≈ y − η H ∞

Original: y
Revised: y

Original: when k = 1 , which implies u (1) ≈ η H ∞ y .
Revised: when k = 1 , which implies u (1) ≈ η H ∞ y .

Original: Thus,
Revised: Thus,

Original: CG (cid:48)
Revised: CG (cid:48)

Original: ( i ) =
Revised: ( i ) =

Original: 2 η y i u
Revised: 2 η y i u

Original: i (1) + 1 / 2 .
Revised: i (1) + 1 / 2 .

Original: Note that an instance ( x i , y i ) has a large CG (cid:48)
Revised: Note that an instance ( x i , y i ) has a large CG (cid:48)

Original: ( i ) if y i u
Revised: ( i ) if y i u

Original: i (1) is large, i.e., the network output u i (1) after 1-time step has the same sign as the targeted label y i ∈ {− 1 , 1 } and has a large magnitude.
Revised: i (1) is large, i.e., the network output u i (1) after 1-time step has the same sign as the targeted label y i ∈ {− 1 , 1 } and has a large magnitude.

Original: Thus, CG (cid:48)
Revised: Thus, CG (cid:48)

Original: ( i ) measures how fast the training instance can be learned by the neural network.
Revised: ( i ) measures how fast the training instance can be learned by the neural network.

Original: Different from CG (cid:48) ( i ) , our original score CG( i ) is correlated with the accumulated error between y i and u i ( k ) averaged over the training as shown in equation 7.
Revised: Different from CG (cid:48) ( i ) , our original score CG( i ) is correlated with the accumulated error between y i and u i ( k ) averaged over the training as shown in equation 7.

Original: Thus, our original score CG( i ) , using the inverse of H ∞ , reﬂects the statistics averaged over the whole training steps.
Revised: Thus, our original score CG( i ) , using the inverse of H ∞ , reﬂects the statistics averaged over the whole training steps.

Original: In Table 1, we compare the Spearman’a rank correlation between CG-score/CG’-score and other data valuation scores including C-score (Jiang et al., 2021), forgetting score (Toneva et al., 2019) and EL2N score (Paul et al., 2021) for CIFAR-10/100 datasets 1 .{+
+}
Revised: In Table 1, we compare the Spearman’a rank correlation between CG-score/CG’-score and other data valuation scores including C-score (Jiang et al., 2021), forgetting score (Toneva et al., 2019) and EL2N score (Paul et al., 2021) for CIFAR-10/100 datasets {+
+}

Original: We can observe that CG-score has higher correlations with the previous scores compared to those of CG’-score.
Revised: We can observe that CG-score has higher correlations with the previous scores compared to those of CG’-score.

Original: In the rest of this paper, we focus on the CG-score for our data valuation score.
Revised: In the rest of this paper, we focus on the CG-score for our data valuation score.

Original: 2. G EOMETRIC INTERPRETATION OF THE COMPLEXITY - GAP SCORE
Revised: 2. G EOMETRIC INTERPRETATION OF THE COMPLEXITY - GAP SCORE

Original: We next provide geometric interpretation for the CG-score.
Revised: We next provide geometric interpretation for the CG-score.

Original: For the sake of simplicity, we consider binary dataset with n 2 samples having y i = 1 and n 2 samples having y i = − 1 .
Revised: For the sake of simplicity, we consider binary dataset with n 2 samples having y i = 1 and n 2 samples having y i = − 1 .

Original: We further assume that E [ H ∞ ij ] =
Revised: We further assume that E [ H ∞ ij ] =

Original: p if y i =
Revised: p if y i =

Original: y j and E
Revised: y j and E

Original: [ H ∞ ij ] =
Revised: [ H ∞ ij ] =

Original: q if y i (cid:54) =
Revised: q if y i (cid:54) =

Original: y j for some | p | > | q | .
Revised: y j for some | p | > | q | .

Original: Note that the diagonal entires H ∞ ii = 1 / 2 for all i ∈ [ n ] .
Revised: Note that the diagonal entires H ∞ ii = 1 / 2 for all i ∈ [ n ] .

Original: Thus, E [ H ∞ ] can be decomposed as E [ H ∞ ]
Revised: Thus, E [ H ∞ ] can be decomposed as E [ H ∞ ]

Original: = (cid:0) 12 − p (cid:1) I n + S where S is a block matrix with S = (cid:18) p q q p (cid:19) ⊗ I n/ 2 , and the resulting eigenvalues of E [ H ∞ ] are ( p + q ) n+ (cid:0) 12 − p (cid:1) , ( p − q )
Revised: = (cid:0) 1 2 − p (cid:1) I n + S where S is a block matrix with S = (cid:18) p q q p (cid:19) ⊗ I n/ 2 , and the resulting eigenvalues of E [ H ∞ ] are ( p + q ) n+ (cid:0) 12 − p (cid:1) , ( p − q )

Original: n+ (cid:0)
Revised: n+ (cid:0)

Original: 12 − p (cid:1) and (cid:0) 12 − p (cid:1) with multiplicity ( n − 2) .
Revised: 12 − p (cid:1) and (cid:0) 12 − p (cid:1) with multiplicity ( n − 2) .

Original: When p =
Revised: When p =

Original: Θ( q )  and p =
Revised: Θ( q ) [Figure] and p =

Original: o (1 /n ) , the matrix E [ H ∞ ]
Revised: o (1 /n ) , the matrix E [ H ∞ ]

Original: ≈ (1 / 2) I n .
Revised: ≈ (1 / 2) I n .

Original: From the representations of H ∞ and ( H ∞ ) − 1 in[Figure] equation 3 and the implied relation h (cid:62) i = − c − 1 i g (cid:62) i ( H ∞ ) − 1 n − 1 , assuming H ∞ ≈ (1 / 2) I n and using c i = 1 / 2 , we can write h (cid:62) i = − 4 g (cid:62) i .
Revised: From the representations of H ∞ and ( H ∞ ) − 1 in equation 15 and the implied relation h (cid:62) i = − c − 1 i g (cid:62) i ( H ∞ ) − 1 n − 1 , assuming H ∞ ≈ (1 / 2) I n and using c i = 1 / 2 , we can write h (cid:62) i = − 4 g (cid:62) i .

Original: By using this approximation, our CG-score in equation 6 can be approximated as
Revised: By using this approximation, our CG-score in equation 6 can be approximated as

Original: [Equation]
Revised: [Equation]

Original: (8) since d i ≈ 2 and y 2 i = 1 .
Revised: since d i ≈ 2 and y 2 i = 1 .

Original: From this approximation, we can ﬁrst notice that − 8 y i
Revised: From this approximation, we can ﬁrst notice that − 8 y i

Original: ( y (cid:62)− i g i ) is the main term that determines the order of { CG( i ) } ni =1 , since y i
Revised: ( y (cid:62)− i g i ) is the main term that determines the order of { CG( i ) } ni =1 , since y i

Original: ( y (cid:62)− i g i ) =
Revised: ( y (cid:62)− i g i ) =

Original: (cid:80) j [Equation]
Revised: (cid:80) j ∈{ [ n ]: y i =

Original:  − [Equation]
Revised: y j } H ∞ ij − (cid:80) j ∈{ [ n ]: y i (cid:54) =

Original:  and thus E [ y i ( y (cid:62)− i g i )]
Revised: y j } H ∞ ij , and thus E [ y i ( y (cid:62)− i g i )]

Original: = ( p − q ) n=
Revised: = ( p − q ) n=

Original: o (1) , assuming p =
Revised: o (1) , assuming p =

Original: Θ( q ) and p =
Revised: Θ( q ) and p =

Original: o (1 /n ) , which is much larger than the ﬁrst term, proportional to ( y i ( y (cid:62)− i g i ))
Revised: o (1 /n ) , which is much larger than the ﬁrst term, proportional to ( y i ( y (cid:62)− i g i ))

Original: Note that y i
Revised: Note that y i

Original: ( y (cid:62)− i g i ) measures the gap between the summation of H ∞ ij over the samples of the same class as ( x i , y i ) and that over the samples of the different class from ( x i , y i ) .
Revised: ( y (cid:62)− i g i ) measures the gap between the summation of H ∞ ij over the samples of the same class as ( x i , y i ) and that over the samples of the different class from ( x i , y i ) .

Original: Since H ∞ ij = x (cid:62) i x j ( π − arccos( x (cid:62) i x j ))
Revised: Since H ∞ ij = x (cid:62) i x j ( π − arccos( x (cid:62) i x j ))

Original: 2 π , y i ( y (cid:62)− i g i ) measures the gap between the average similarities of x i with other samples { x j } of the same class y j
Revised: 2 π , y i ( y (cid:62)− i g i ) measures the gap between the average similarities of x i with other samples { x j } of the same class y j

Original: = y i
Revised: = y i

Original: and that with samples of the different class y j (cid:54) =
Revised: and that with samples of the different class y j (cid:54) =

Original: y i , where the similarity is measured by the cosine between the two samples ( x i , x j ) multiplied by the chance that ReLU is activated for both the samples.
Revised: y i , where the similarity is measured by the cosine between the two samples ( x i , x j ) multiplied by the chance that ReLU is activated for both the samples.

Original: Thus, we can expect two important trends regarding the CG-score:
Revised: Thus, we can expect two important trends regarding the CG-score:

Original: Regular vs. irregular samples: If a sample ( x i , y i ) is a ‘regular’ sample representing the class y i in the sense that it has a large similarity withthe other samples of the same class, much larger than the similarity with the samples from the different class, then the sample will have a large y i ( y (cid:62)− i g i ) , resulting in a lower CG-score due to the minus sign.
Revised: Regular vs. irregular samples: If a sample ( x i , y i ) is a ‘regular’ sample representing the class y i in the sense that it has a larger similarity with other samples of the same class than to samples from the different class, then it will have a large y i ( y (cid:62)− i g i ) , resulting in a lower CG-score due to the minus sign.

Original: On the other hand, if a sample ( x i , y i ) is ‘irregular’ in the sense that it does not represent the class and has a small similarity with the samples of the same class, then y i ( y (cid:62) − i g i ) will be small, resulting in a higher CG-score.
Revised: On the other hand, if a sample ( x i , y i ) is ‘irregular’ in the sense that it does not represent the class and has a small similarity with the samples of the same class, then y i ( y (cid:62)− i g i ) will be small, resulting in a higher CG-score.

Original: Clean label vs. noisy label:
Revised: Clean label vs. noisy label:

Original: A sample with label noise tend to have a large CG-score since y i
Revised: A sample with label noise tend to have a large CG-score since y i

Original: ( y (cid:62)− i g i ) is negative for a sample with label noise, while it is positive for a sample with clean label
Revised: ( y (cid:62)− i g i ) is negative for a sample with label noise, while it is positive for a sample with clean label.

Original: We empirically demonstrate the above two trends by a controlled experiment.
Revised: We empirically demonstrate the above two trends by a controlled experiment.

Original: Consider two 3000dimensional Gaussian distributions with a ﬁxed covariance matrix 0 .
Revised: Consider two 3000dimensional Gaussian distributions with a ﬁxed covariance matrix 0 .

Original: 25 I 3000 and different means (1 , 0 , . . .
Revised: 25 I 3000 and different means (1 , 0 , . . .

Original: and ( − 1 , 0 , . .
Revised: and ( − 1 , 0 , . .

Original: . , 0) , representing class +1 and -1, respectively.
Revised: . , 0) , representing class +1 and -1, respectively.

Original: We generate 1000 samples from each class.
Revised: We generate 1000 samples from each class.

Original: We ﬁrst check whether the structural assumptions on H ∞ hold with this synthetic dataset.
Revised: We ﬁrst check whether the structural assumptions on H ∞ hold with this synthetic dataset.

Original: In Fig.
Revised: In Fig.

Original: 1a, we observe that ( H ∞ ) − 1 n − 1 can be well approximated by 2 I n − 1 , and thus the approximation for CG( i ) in equation 8 may hold well.
Revised: 1a, we observe that ( H ∞ ) − 1 n − 1 can be well approximated by 2 I n − 1 , and thus the approximation for CG( i ) in equation 8 may hold well.

Original: We then examine the correlation between the CG-score and the sample value at the ﬁrst dimension.
Revised: We then examine the correlation between the CG-score and the sample value at the ﬁrst dimension.

Original: In Fig.
Revised: In Fig.

Original: 1b, we can observe that instances near the decision boundary x 1 = 0 have higher CG-scores compared to those located further from the boundary.
Revised: 1b, we can observe that instances near the decision boundary x 1 = 0 have higher CG-scores compared to those located further from the boundary.

Original: This shows that irregular samples, e.g., samples near the boundary of the two classes, indeed have higher CG-scores.
Revised: This shows that irregular samples, e.g., samples near the boundary of the two classes, indeed have higher CG-scores.

Original: We further modify the generated samples by randomly ﬂipping the labels of 10% samples and we re-calculate the CG-scores of all the samples.
Revised: We further modify the generated samples by randomly ﬂipping the labels of 10% samples and we re-calculate the CG-scores of all the samples.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 1c, the samples with label noise tend to have higher CG-scores, agreeing with our intuition. 
Revised: 1c, the samples with label noise tend to have higher CG-scores, agreeing with our intuition. [Figure]

Revised: Only CG-Score does not require any training of the model to calculate the score for examples, but it achieves competitive performances compared to other state-of-the-art data valuation scores and signiﬁcantly outperforms the performance of random baseline.

Revised: In (a), CG-score maintains the test accuracy up to signiﬁcant removing portions; in (b), the test accuracy drops most rapidly for CGscore since the examples with high CG-score are essential in generalization of the model.

Original:  D ATA V ALUATION THROUGH C OMPLEXITY G AP S CORE
Revised:  D ATA V ALUATION THROUGH C OMPLEXITY G AP S CORE

Original: In this section, we show diverse applications of the CG-score in data valuation for real datasets.
Revised: In this section, we show diverse applications of the CG-score in data valuation for real datasets.

Original: 3. D ATA PRUNING EXPERIMENTS
Revised: 3. D ATA PRUNING EXPERIMENTS

Original: To evaluate the ability of the CG-score in identifying important examples, we design data pruning experiments, similar to those in Ghorbani & Zou (2019); Paul et al.
Revised: To evaluate the ability of the CG-score in identifying important examples, we design data pruning experiments, similar to those in Ghorbani & Zou (2019); Paul et al.

Original: We evaluate our score on three public datasets, FMNIST, CIFAR-10/100 and train ResNet networks (He et al., 2016) of different depths, ResNet18 for FMNIST and CIFAR-10 and ResNet50 for CIFAR-100 dataset, respectively.
Revised: We evaluate our score on three public datasets, FMNIST, CIFAR-10/100 and train ResNet networks (He et al., 2016), ResNet18 for FMNIST and CIFAR-10 and ResNet50 for CIFAR-100 dataset, respectively.

Original: As baseline methods for data valuation, we use three state-of-the-art scores, C-score (Jiang et al., 2021), EL2N (Paul et al., 2021), and Forgetting score (Toneva et al., 2019), all of which require training of models for computation and thus are dependent on the models at which the scores are calculated.
Revised: As baseline methods for data valuation, we use three state-of-the-art scores, C-score (Jiang et al., 2021), EL2N (Paul et al., 2021), and Forgetting score (Toneva et al., 2019), all of which require training of models for computation.

Original: On the contrary, our score is a data-centric measure and independent on the model.
Revised: On the contrary, our score is a data-centric measure and independent on the model.

Original: More details on the baselines and experiments are summarized in Appendix § B.
Revised: More details on the baselines and experiments are summarized in Appendix § C.

Original: In Figure 2, the ﬁrst row shows the test accuracy of the model trained with different subset sizes, when low-scoring (regular) examples are removed ﬁrst, and the second row shows the similar result but when high-scoring (irregular) examples are removed ﬁrst.
Revised: In Figure 2, the ﬁrst row shows the test accuracy of the model trained with different subset sizes, when low-scoring (regular) examples are removed ﬁrst, and the second row shows the similar result but when high-scoring (irregular) examples are removed ﬁrst.

Original: We report the mean of the result after three independent runs, and the shaded regions indicate the standard deviation.
Revised: We report the mean of the result after three independent runs, and the shaded regions indicate the standard deviation.

Original: The red-curve (random) is the result when randomly ordered examples are used.
Revised: The red-curve (random) is the result when randomly ordered examples are used.

Original: We can observe that when removing low-scoring examples ﬁrst, networks maintain the test accuracy (with less than 1% drop) compared to the case of training with the full dataset up to signiﬁcant removing portions, for example, 40% for
Revised: We can observe that when removing low-scoring examples ﬁrst, networks maintain the test accuracy (with less than 1% drop) compared to the case of training with the full dataset up to signiﬁcant removing portions, for example, 40% for

Original: CIFAR-10 and 20% for CIFAR-100 with the CG-score.
Revised: CIFAR-10 and 20% for CIFAR-100 with the CG-score.

Original: Especially, our CG-score, which does not require any training of a model, can achieve competitive performances as the other baseline methods and it also signiﬁcantly outperforms the random baseline.
Revised: Especially, our CG-score, which does not require any training of a model, can achieve competitive performances as the other baseline methods and it also signiﬁcantly outperforms the random baseline.

Original: When we remove the high-scoring examples ﬁrst, the test accuracy drops most rapidly for the CG-score curve, implying that examples with the high CG-score are essential part of the data governing the generalization of the model.
Revised: When we remove the high-scoring examples ﬁrst, the test accuracy drops most rapidly for the CG-score curve, implying that examples with the high CG-score are essential part of the data governing the generalization of the model.

Original: 3. D ETECTING LABEL NOISE
Revised: 3. D ETECTING LABEL NOISE

Original: In Section 2.3, we analyzed the CG-score and showed that the examples with label noise tend to have higher CG-score.
Revised: In Section 2.3, we analyzed the CG-score and showed that the examples with label noise tend to have higher CG-score.

Original: We next empirically investigate the ability of the CG-score in detecting label noiseby artiﬁcially corrupting 20% of instances in FMNIST and CIFAR-100 with random label noise. We
Revised: We next empirically investigate the ability of the CG-score in detecting label noise

Original: [Figure]
Revised: [Figure]

Revised: by artiﬁcially corrupting 20% of instances in FMNIST and CIFAR-100 with random label noise.

Original:  ﬁrst examine the distribution of the CG-scores for each dataset after the corruption in Figure 3a.
Revised: We ﬁrst examine the distribution of the CG-scores for each dataset after the corruption in Figure 3a.

Original: We can observe that for both datasets, the examples with corrupted labels (orange) tend to have higher
Revised: We can observe that for both datasets, the examples with corrupted labels (orange) tend to have higher

Original: CG-scores than those with clean labels (blue).
Revised: CG-scores than those with clean labels (blue).

Original: For a relatively simpler dataset, FMNIST (left), the
Revised: For a relatively simpler dataset, FMNIST (left), the

Original: CG-score histograms can be more clearly separable between the clean and noisy groups, compared to a more complex dataset, CIFAR-100 (right).
Revised: CG-score histograms can be more clearly separable between the clean and noisy groups, compared to a more complex dataset, CIFAR-100 (right).

Original: With this observation, we can anticipate that the
Revised: With this observation, we can anticipate that the

Original: CG-score can have a better detectability of label noise for relatively simpler datasets.
Revised: CG-score can have a better detectability of label noise for relatively simpler datasets.

Original: We next evaluate the noise detectability of the CG-score by checking a varying portion of the examples ordered by the CG-score (highest ﬁrst) in Figure 3b.
Revised: We next evaluate the noise detectability of the CG-score by checking a varying portion of the examples ordered by the CG-score (highest ﬁrst) in Figure 3b.

Original: The curves indicate the fraction of noisy examples included in the examined subset.
Revised: The curves indicate the fraction of noisy examples included in the examined subset.

Original: In this experiment, we compare our score with two other scores, Forgetting score and EL2N, as well as a random baseline and the oracle.
Revised: In this experiment, we compare our score with two other scores, Forgetting score and EL2N, as well as a random baseline and the oracle.

Original: We can observe that the CG-score achieves the best performance, near that of the oracle, for FMINST, and competitive performances for CIFAR-100.
Revised: We can observe that the CG-score achieves the best performance, near that of the oracle, for FMINST, and competitive performances for CIFAR-100.

Original: In the plot, we also compare the performance of the CG-score with that of ‘Partial CG-score,’ which is a new score deﬁned by a sub-term of the CG-score.
Revised: In the plot, we also compare the performance of the CG-score with that of ‘Partial CG-score,’ which is a new score deﬁned by a sub-term of the CG-score.

Original: In the
Revised: In the

Original: CG-score in equation 6, we have three terms, but only the second term 

Revised: CG-score in equation 6, we have three terms, but only the second term

Original: y i
Revised: y i

Original: ( y (cid:62)− i h i ) uses the label information of the data ( x i , y i ) .
Revised: ( y (cid:62)− i h i ) uses the label information of the data ( x i , y i ) .

Original: Thus, we examined the ability of this term only, deﬁned as the ‘Partial CG-score’, in detecting the label noise, and found that the CG-score and partial CG-score have similar performances in detecting label noise, which implies that the label-noise detectability of the CG-score mainly comes from the term
Revised: Thus, we examined the ability of this term only, deﬁned as the ‘Partial CG-score’, in detecting the label noise, and found that the CG-score and partial CG-score have similar performances in detecting label noise, which implies that the label-noise detectability of the CG-score mainly comes from the term

Original: 2 y i
Revised: 2 y i

Original: ( y (cid:62)− i h i ) .
Revised: ( y (cid:62)− i h i ) .

Original:  C OMPLEXITY -G AP S CORE AND D EEP L EARNING P HENOMENA
Revised:  C OMPLEXITY -G AP S CORE AND D EEP L EARNING P HENOMENA

Original: In this section, we empirically demonstrate the relation between the CG-score and the ‘learning difﬁculty’ ofeach data instance for CIFAR-10 dataset trained in ResNet18.
Revised: We next demonstrate that the CG-score can capture ‘learning difﬁculty’ of data instances.

Original: 4. C OMPLEXITY GAP SCORE CAPTURES LEARNING DIFFICULTY
Revised: 4. C OMPLEXITY GAP SCORE CAPTURES LEARNING DIFFICULTY

Original: It has been widely observed that there exists an order of instances in which a model learns to classify the data correctly and this order is robust within model types (Wu et al., 2021).
Revised: It has been widely observed that there exists an order of instances in which a model learns to classify the data correctly and this order is robust within model types (Wu et al., 2021).

Original: Many previous scoring functions use this type of observation and deﬁne the ‘difﬁculty’ of a given instance based on the speed at which the model’s prediction converges (Toneva et al., 2019; Swayamdipta et al., 2020).
Revised: Many previous scoring functions use this type of observation and deﬁne the ‘difﬁculty’ of a given instance based on the speed at which the model’s prediction converges (Toneva et al., 2019; Swayamdipta et al., 2020).

Original: Our data-centric CG-score is originally deﬁned based on the gap in the generalization error bounds as in equation 2, but it also reﬂects the convergence speed of the instance, as analyzed in equation 7.
Revised: Our data-centric CG-score is originally deﬁned based on the gap in the generalization error bounds as in equation 2, but it also reﬂects the convergence speed of the instance, as analyzed in equation 7.

Original: We empirically demonstrate this relation by analyzing the training dynamics of CIFAR-10 dataset  in ResNet18 network.
Revised: We empirically demonstrate this relation by analyzing the training dynamics of CIFAR-10 dataset trained in ResNet18 network.

Original: We ﬁrst sort the data instances in ascending order using the CG-score, and divide the data into 10 equal-sized subgroups.
Revised: We ﬁrst sort the data instances in ascending order using the CG-score, and divide the data into 10 equal-sized subgroups.

Original: We then measure the mean of loss and training accuracy for the 10 subgroups as training progresses.
Revised: We then measure the mean of loss and training accuracy for the 10 subgroups as training progresses.

Original: Fig. 4a and Fig.
Revised: Fig. 4a and Fig.

Original: 4b show the mean of loss and the training accuracy throughout the training for the 10 subgroups, respectively, where the blue color represents the low-scoring groups and the red color represents the high-scoring groups.
Revised: 4b show the mean of loss and the training accuracy throughout the training for the 10 subgroups, respectively, where the blue color represents the low-scoring groups and the red color represents the high-scoring groups.

Original: We can observe that the mean loss and accuracy converge faster for low-scoring groups, while it takes longer to converge for high-scoring groups.
Revised: We can observe that the mean loss and accuracy converge faster for low-scoring groups, while it takes longer to converge for high-scoring groups.

Original: This indicates that the CG-score is highly correlated with the ‘difﬁculty’ of an example measured by the learning speed at a model.
Revised: This indicates that the CG-score is highly correlated with the ‘difﬁculty’ of an example measured by the learning speed at a model.

Original: [Figure]
Revised: [Figure]

Original: 4. D ATA SAMPLE DRIVING MOVEMENT OF NEURONS
Revised: 4. D ATA SAMPLE DRIVING MOVEMENT OF NEURONS

Original: We next investigate the relation between the CG-score and the evolution velocity of the datadependent Neural Tangent Kernel (NTK) (Fort et al., 2020).
Revised: We next investigate the relation between the CG-score and the evolution velocity of the datadependent Neural Tangent Kernel (NTK) (Fort et al., 2020).

Original: NTK has been widely used to approximate the evolution of an inﬁnite-width deep learning model via linearization around initial weights, when the network is trained by gradient descent with a sufﬁciently small learning rate (Jacot et al., 2018).
Revised: NTK has been widely used to approximate the evolution of an inﬁnite-width deep learning model via linearization around initial weights, when the network is trained by gradient descent with a sufﬁciently small learning rate (Jacot et al., 2018).

Original: The main idea is that in the limit of an inﬁnite width, the network parameters do not move very far from its initialization throughout the training, so that the learning process can be approximated by a linear process along the tangent space to the manifold of the model’s function class at the initialization (Lee et al., 2019a).
Revised: The main idea is that in the limit of an inﬁnite width, the network parameters do not move very far from its initialization throughout the training, so that the learning process can be approximated by a linear process along the tangent space to the manifold of the model’s function class at the initialization (Lee et al., 2019a).

Original: However, as observed in Fort et al.
Revised: However, as observed in Fort et al.

Original: (2020), for a ﬁnite-width network, the tangent kernel is not constant but it rather rapidly changes over time, especially at the beginning of the training.
Revised: (2020), for a ﬁnite-width network, the tangent kernel is not constant but it rather rapidly changes over time, especially at the beginning of the training.

Original: To quantify this change, Fort et al.
Revised: To quantify this change, Fort et al.

Original: (2020) addressed the data-dependent
Revised: (2020) addressed the data-dependent

Original: Kernel Gram matrix, which is the Gram matrix of the logit Jacobian, and deﬁned the Kernel velocity as the cosine distance between two NTK Gram matrices before and after one epoch of training.
Revised: Kernel Gram matrix, which is the Gram matrix of the logit Jacobian, and deﬁned the Kernel velocity as the cosine distance between two NTK Gram matrices before and after one epoch of training.

Original: In
Revised: In

Original: Paul et al.
Revised: Paul et al.

Original: (2021), the Kernel velocity was used to evaluate the subgroups of data instances to ﬁgure out the subgroup driving the learning and the change in the NTK feature space.
Revised: (2021), the Kernel velocity was used to evaluate the subgroups of data instances to ﬁgure out the subgroup driving the learning and the change in the NTK feature space.

Original: We conduct similar experiments for subgroups of data, divided according to our CG-score.
Revised: We conduct similar experiments for subgroups of data, divided according to our CG-score.

Original: Fig.
Revised: Fig.

Original: 4c shows the Kernel velocities for 10 different subgroups of CIFAR-10 data trained in ResNet18.
Revised: 4c shows the Kernel velocities for 10 different subgroups of CIFAR-10 data trained in ResNet18.

Original: Each group is composed of 125 consecutive instances from each level, where the level is deﬁned by dividing the full dataset, sorted in ascending order by the CG-score, into 10 groups.
Revised: Each group is composed of 125 consecutive instances from each level, where the level is deﬁned by dividing the full dataset, sorted in ascending order by the CG-score, into 10 groups.

Original: The higher level (red) is composed of instances having higher CG-scores, while the lower level (blue) is composed of instances having lower CG-scores.
Revised: The higher level (red) is composed of instances having higher CG-scores, while the lower level (blue) is composed of instances having lower CG-scores.

Original: We can observe that the samples with high CG-score (red) maintains higher Kernel velocity throughout the training, which means that NTK Gram matrix evolves by a larger amount for the samples of high CG-score.
Revised: We can observe that the samples with high CG-score (red) maintains higher Kernel velocity throughout the training, which means that NTK Gram matrix evolves by a larger amount for the samples of high CG-score.

Original: Thus, we can hypothesize that the instances with high CG-score are ‘difﬁcult’ examples the network may struggle to optimize and try to ﬁt throughout the training.
Revised: Thus, we can hypothesize that the instances with high CG-score are ‘difﬁcult’ examples the network may struggle to optimize and try to ﬁt throughout the training.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We proposed the CG-score, a data-centric valuation score, to quantify the effect of individual data instances in optimization and generalization of overparameterized two-layer neural networks trained by gradient descent.
Revised: We proposed the CG-score, a data-centric valuation score, to quantify the effect of individual data instances in optimization and generalization of overparameterized two-layer neural networks trained by gradient descent.

Original: We theoretically and empirically demonstrated that the CG-score can identify ‘irregular’ instances within each class, and can be used as a score to select instances essential for generalization of a model or in ﬁltering instances with label noise.
Revised: We theoretically and empirically demonstrated that the CG-score can identify ‘irregular’ instances within each class, and can be used as a score to select instances essential for generalization of a model or in ﬁltering instances with label noise.

Original: We also showed the close relation between the CG-score and learning difﬁculty of instances by analyzing training dynamics.
Revised: We also showed the close relation between the CG-score and learning difﬁculty of instances by analyzing training dynamics.

Original: Interesting open problems related to the CG-score include 1) providing theoretical justiﬁcation of the score for more general deep neural networks and 2) improving the score by modifying the deﬁnition in terms of ‘features’ of data instances.
Revised: Interesting open problems related to the CG-score include 1) providing theoretical justiﬁcation of the score for more general deep neural networks and 2) improving the score by modifying the deﬁnition in terms of ‘features’ of data instances.

Original: A E XTENSIONS TO M ULTI - CLASS C OMPLEXITY G AP SCORE
Revised: A E XTENSIONS TO M ULTI - CLASS C OMPLEXITY G AP SCORE

Original: In this section, we explain the details of how we calculated the CG-scores for multi-label public datasets.
Revised: In this section, we explain the details of how we calculated the CG-scores for multi-label public datasets.

Original: A. C ALCULATION OF CG- SCORE FOR MULTI - LABEL DATASETS
Revised: A. C ALCULATION OF CG- SCORE FOR MULTI - LABEL DATASETS

Original: In deﬁning the CG-score, we assumed the binary datasets y ∈ {± 1 } with inputs having a ﬁxed norm (cid:107) x (cid:107) 2 = 1 .
Revised: In deﬁning the CG-score, we assumed the binary datasets y ∈ {± 1 } with inputs having a ﬁxed norm (cid:107) x (cid:107) 2 = 1 .

Original: To calculate the CG-score for multi-label ( k -class) public datasets, we normalize all the inputs to have (cid:107) x (cid:107) 2 = 1 .
Revised: To calculate the CG-score for multi-label ( k -class) public datasets, we normalize all the inputs to have (cid:107) x (cid:107) 2 = 1 .

Original: We then calculate the CG-score for examples of each class j ∈ [ k ] , assuming that all the examples from class j have label +1 and the rest of the examples from any other classes have label − 1 .
Revised: We then calculate the CG-score for examples of each class j ∈ [ k ] , assuming that all the examples from class j have label +1 and the rest of the examples from any other classes have label − 1 .

Original: In detail, let y =
Revised: In detail, let y =

Original: ( y 1 , y 2 , ..., y n ) ∈ { 1 , 2 , . . .
Revised: ( y 1 , y 2 , ..., y n ) ∈ { 1 , 2 , . . .

Original: k } n be the label vector for n data instances and, without loss of generality, assume that x 1 , x 2 , . . .
Revised: k } n be the label vector for n data instances and, without loss of generality, assume that x 1 , x 2 , . . .

Original: , x l belong to class 1, i.e. y 1 , y 2 , ..., y l = 1 .
Revised: , x l belong to class 1, i.e. y 1 , y 2 , ..., y l = 1 .

Original: Then, to calculate the CG-score for x 1 , x 2 , . . .
Revised: Then, to calculate the CG-score for x 1 , x 2 , . . .

Original: x l , we generate the gram matrix
Revised: x l , we generate the gram matrix

Original: H ∞ with ( x 1 , 1) , ( x 2 , 1) , . . .
Revised: H ∞ with ( x 1 , 1) , ( x 2 , 1) , . . .

Original: , ( x l , 1) , ( x l +1 , − 1) , ( x l +2 , − 1) , . . .
Revised: , ( x l , 1) , ( x l +1 , − 1) , ( x l +2 , − 1) , . . .

Original: , ( x n , − 1) and calculate the CGscores of x 1 , x 2 , . . .
Revised: , ( x n , − 1) and calculate the CGscores of x 1 , x 2 , . . .

Original: , x l as the two-label case.
Revised: , x l as the two-label case.

Original: A. S TOCHASTIC METHOD TO CALCULATE CG- SCORES
Revised: A. S TOCHASTIC METHOD TO CALCULATE CG- SCORES

Original: Since the calculation of the CG-score requires the inversion of n -dimensional matrix H ∞ where n is the number of total samples, it would demand expensive memory and computational cost.
Revised: Since the calculation of the CG-score requires the inversion of n -dimensional matrix H ∞ where n is the number of total samples, it would demand expensive memory and computational cost.

Original: To lower the complexity, we sub-sample the examples with class − 1 so that the ratio between class + and class -1 is reduced from 1 : ( k − 1) to 1 : 3 for MNIST and FMINST, 1 : 4 for CIFAR-10 and
Revised: To lower the complexity, we sub-sample the examples with class − 1 so that the ratio between class + and class -1 is reduced from 1 : ( k − 1) to 1 : 3 for MNIST and FMINST, 1 : 4 for CIFAR-10 and

Original: CIFAR-100.
Revised: CIFAR-100.

Original: We repeat this process ten times by randomly sampling examples of label − 1 and then average out the calculated CG-score of each example from class +1 .
Revised: We repeat this process ten times by randomly sampling examples of label − 1 and then average out the calculated CG-score of each example from class +1 .

Original: A. J USTIFICATION OF STOCHASTIC METHOD TO CALCULATE CG- SCORES
Revised: A. J USTIFICATION OF STOCHASTIC METHOD TO CALCULATE CG- SCORES

Original: To justify the stochastic method in calculating the CG-scores, we conduct an experiment to check whether the CG-scores calculated by the stochastic method converges well to the true CG-scores utilizing the full dataset.
Revised: To justify the stochastic method in calculating the CG-scores, we conduct an experiment to check whether the CG-scores calculated by the stochastic method converges well to the true CG-scores utilizing the full dataset.

Original: We created a subset of the CIFAR-10 dataset, the Small-CIFAR-10 dataset, which consists of 1000 instances for each label (a total of 10000 instances).
Revised: We created a subset of the CIFAR-10 dataset, the Small-CIFAR-10 dataset, which consists of 1,000 instances for each label (a total of 10,000 instances).

Original: Then, we compared the CG-score calculated by 10000x10000 Gram matrix H ∞ of the full dataset (true CG-score) with the CG-score calculated by the stochastic method, where the stochastic CG-score for the instances from a class (class 1) are calculated by subsampling the samples from any other classes (class -1) with the ratio between class 1 and -1 equal to 1 : 1 (size 2000x2000), 1 : 2 (size 3000x3000), 1 :(size 4000x4000) and 1 : 4 (size 5000x5000) instead of 1 : 9 .
Revised: Then, we compared the CG-score calculated by 10,000x10,000 Gram matrix H ∞ of the full dataset (true CG-score) with the CG-score calculated by the stochastic method, where the stochastic CG-score for the instances from a class (class 1) are calculated by subsampling the samples from any other classes (class -1) with the ratio between class 1 and -1 equal to 1 : 1 (size 2,000x2,000), 1 : 2 (size 3,000x3,000), 1 : 3 (size 4,000x4,000) and 1 : 4 (size 5,000x5,000) instead of 1 : 9 .

Original: We repeat this process multiple times by randomly sampling examples of label − 1 and then average out the calculated CG-score of each example from class +1 .
Revised: We repeat this process multiple times by randomly sampling examples of label − 1 and then average out the calculated CG-score of each example from class +1 .

Original: In Figure 5, we plot the  rank correlation  between the true CG-score and the stochastic CG-score for each ratio (different colors) as the number of random sampling increases.
Revised: In Figure 5, we plot the Spearman’s rank correlation and Pearson correlation between the true CG-score and the stochastic CG-score for each ratio (different colors) as the number of random sampling increases.

Original: We can observe that the rank correlation converges to a certain number as the number of random sampling increases, and the amount of correlation increases as the size of the matrix (the number of instances included in deﬁning H ∞ ) increases. 
Revised: We can observe that the correlations converge to a certain number as the number of random sampling increases, and the amount of correlation increases as the size of the matrix (the number of instances included in deﬁning H ∞ ) increases. The values of correlations obtained after 20 independent runs are 0.829, 0.914, 0.953, and 0.973 for Recommended number of runs for stochastic calculation.

Original: B I MPLEMENTATION D ETAILS B. T RAINING DETAILS
Revised: The proper number of runs in stochastic calculation of the CG-score may need to be determined by the size and the number of classes of the datasets. We recommend the number of runs to include at least half of the whole dataset in calculation of the CG-score for each class. As an example, for the CIFAR-100 dataset, where each class includes 500 images, when the sampling ratio between the class of interest and the rest of classes is 1:4, we calculate the score for 500 images from the class of interest by using 2,000 images from the rest of 99 classes. Then, about 20 images are selected from each of theclasses.

Revised: To cover at least a half of the images per class (250), we need to repeat the runs abouttimes (ignoring overlap of samples in each run).

Revised: For the FMNIST/CIFAR-10 datasets, the similar calculation shows that only 2-3 runs will be enough.

Revised: B D ISCRIMINATING MISLABELED DATA FROM ATYPICAL ( BUT USEFUL ) DATA BY CG- SCORES

Revised: Discriminating mislabeled data and atypical (but useful) data is a major challenge in data valuation, since both the mislabeled data and atypical data are irregular in the data distribution and tend to have high CG-scores.

Revised: The same challenge has been observed with previous valuation scores such as forgetting score (Toneva et al., 2019) and EL2N score (Paul et al., 2021).

Revised: However, we ﬁnd that mislabeled data usually has a higher CG-score than atypical data, and this tendency gives us the possibility to separate mislabeled data from the rest of the clean data.

Revised: To check the tendency, we perform a data window experiment.

Revised: The data instances are sorted in ascending order by the CG-score, and we compare the test accuracy of a neural network trained with 50% of training instances selected from offset% to (offset+50)% scoring group, for different offset points of { 0 , 5 , 10 , . . .

Revised: For example, when the offset is 20%, we select the data instances from 20% to 70% scoring examples.

Revised: When the training instances do not include mislabeled data, we can expect that the window experiment will show higher accuracy as the offset increases up to 50%.

Revised: We add 20% of random label noise to FMNIST and CIFAR-100 datasets to see how the trend changes when the dataset includes mislabeled instances.

Revised: As shown in Fig.

Revised: 6 (a) and (b), the test accuracy increases until the offset reaches 30% and then drops after the point.

Revised: When the offset is 30%, the 50%-width window includes 30% to 80% scoring examples.

Revised: Since 20% mislabeled data are mainly located within the 80% to 100% scoring group, as the offset increases above 30%, the 50%-width window starts to include mislabeled instances and this causes the rapid drop of the test accuracy.

Revised: Thus, from the window experiments we can see that the mislabeled data has the highest CG-score and can be separable from the rest of the clean examples by the CG-score.

Revised: Then, the next reasonable question is how to set the threshold on CG-score to detect the mislabeled data when the portion of mislabeled data is unknown.

Revised: We show that the sign of the partial CGscore 2 y i ( y (cid:62)− i h i ) , which is a sub-term of the CG-score including the label information y i , can be used in this purpose.

Revised: As explained in Sec. 2.3, the partial CG-score measures the gap between the average similarity of the data instance with other samples of the same class compared to that with the samples of the different classes.

Revised: Thus, by checking the sign of the partial CG-score, we can discriminate mislabeled samples from clean samples.

Revised: In Fig.

Revised: 6 (c), we show the scatter plot of clean (blue) and mislabeled data (orange), where one can ﬁnd that mislabeled examples tend to have positive partial CG-score.

Revised: Furthermore, as shown in Table 2, we can check that for FMNIST with 20% label noise (12,000 mislabeled instances and 48,000 clean instances), 98%(=11796/12000) of

Revised: [Table]

Revised: [Figure]

Revised: mislabeled data ends up having positive partial CG-score, while only 7%(=3462/44538) of clean data has positive partial CG-score.

Revised: Thus, even when the portion of label noise is unknown, our partial CG-score can effectively detect the mislabeled data by the sign information.

Revised: This tendency was less clear for CIFAR-10 dataset due to the increased dataset complexity, but still the tendency existed.

Original: In Section 3 and4, we evaluate our score on three public datasets, FMNIST, CIFAR-10/100, by training ResNet networks (He et al., 2016) of different depths.
Revised: In Section 3 and we evaluate our score on three public datasets, FMNIST, CIFAR-10/100, by training ResNet networks (He et al., 2016) of different depths.

Original: ResNet18 is used for FMNIST and
Revised: ResNet18 is used for FMNIST and

Original: CIFAR-10 dataset and ResNet50 is used for CIFAR-100 dataset.
Revised: CIFAR-10 dataset and ResNet50 is used for CIFAR-100 dataset.

Original: Implementation of the ResNet is based on the ResNet network in torchvision (Paszke et al., 2019).
Revised: Implementation of the ResNet is based on the ResNet network in torchvision (Paszke et al., 2019).

Original: Since FMNIST and CIFAR images are smaller than Imagenet (Deng et al., 2009) images, we replace the front parts of the
Revised: Since FMNIST and CIFAR images are smaller than Imagenet (Deng et al., 2009) images, we replace the front parts of the

Original: ResNet (convolution layer with 7x7 kernel and 2x2 stride, max pooling layer with 3x3 kernel and 2x2 stride) with a single convolution layer with 3x3 kernel and 1x1 stride for small size image.
Revised: ResNet (convolution layer with 7x7 kernel and 2x2 stride, max pooling layer with 3x3 kernel and 2x2 stride) with a single convolution layer with 3x3 kernel and 1x1 stride for small size image.

Original: The details on hyperparameters and optimization methods used in training are summarized in the Table 2.
Revised: The details on hyperparameters and optimization methods used in training are summarized in the Table 3.

Original: [Table]
Revised: Table 4 provides computational time (in seconds) to obtain CG-scores with different sampling ratio 1:1, 1:2, 1:3, and 1:4 for FMNIST and CIFAR-10/100 dataset. We also report the time to compute the baseline scores, Forgetting (Toneva et al., 2019), EL2N (Paul et al., 2021), TracIn (Pruthi et al., 2020), and CRAIG (Pruthi et al., 2020).

Original: annealing schedule (Loshchilov & Hutter, 2017)
Revised: We do not calculate the CG-score of FMNIST dataset with sampling ratio 1:4 since it requires a inversion for 30,000x30,000 matrix, which exceeds the limit of the device memory. Every score including ours and baselines needs to be calculated by averaging the results of independent multiple runs. For a fair comparison, we compare the time to get each score for a single run. Cost to compute CG-score depends on the size of the Gram matrix H ∞ , since we need to calculate the inverse of H ∞ to get the CG-score and the computational complexity to

Original: [Table] 
Revised: [Table] [Table]

Original: dataset’s
Revised: conduct an inversion of n × n matrix is O ( n 3 ) . Therefore

Original: [Table]
Revised: [Table]

Original: Random
Revised: for a dataset of which each class includes a large number of instances (e

Original: [Table]
Revised: [Table]

Original: Cropping
Revised: g., FMNIST and CIFAR-10)

Original: [Table]
Revised: [Table]

Original: pixels
Revised: taking the inverse of a Gram matrix with large sampling ratio may cause expensive computational cost

Original: [Table]
Revised: [Table]

Original: all sides
Revised: while computing the score for a dataset of which each class includes relatively small number of instances (CIFAR-100) can be done in a short time. For example, calculating CG-score of CIFAR-10 dataset (5,000 data in a class) takes1.2 hours with sampling ratio 1:4, while that for CIFAR-100(500 data in a class) takes justminute. EL2N score is time-efﬁcient overall because it is calculate at the relatively early stage of training. Forgetting and TracIn, on the other hand, take relatively longer time since they require at least one full train of the model.

Original: [Table]
Revised: In addition, we argue that our method has another computational advantage that we do not need to search networks and hyperparameters which would work well for the target dataset.

Original: B. E XPERIMENTAL DETAILS
Revised: C. E XPERIMENTAL DETAILS

Original: Baseline scores for data valuation We use three state-of-the-art scores, C-score (Jiang et al., 2021), EL2N (Paul et al., 2021) [Table] and Forgetting score (Toneva et al., 2019) as baselines with which our CG-score is compared.
Revised: Baseline scores for data valuation We use three state-of-the-art scores, C-score (Jiang et al., 2021), EL2N (Paul et al., 2021), and Forgetting score (Toneva et al., 2019) as baselines with which our CG-score is compared.

Original: We use pre-calculated C-score for CIFAR-10 and CIFAR-100 from Jiang et al.
Revised: We use pre-calculated C-score for CIFAR-10 and CIFAR-100 from Jiang et al.

Original: and calculate EL2N and Forgetting score by averaging the score across ﬁve independent training using the full dataset.
Revised: and calculate EL2N and Forgetting score by averaging the score across ﬁve independent training using the full dataset.

Original: We  use the same network architectures to calculate EL2N and Forgetting score: ResNet18 for FMNIST and CIFAR-10 dataset and ResNet50 for CIFARdataset.
Revised: We obtain EL2N scores at 20th epochs of the training. We use the same network architectures to calculate EL2N and Forgetting score: ResNet18 for FMNIST and CIFAR-10 dataset and ResNet50 for CIFAR-100 dataset.

Original: Detailed deﬁnitions of the scores are as follows:
Revised: Detailed deﬁnitions of the scores are as follows:

Original: • Consistency score (C-score): C-score of each instance is calculated by estimating the prediction accuracy of the instance attained by the model trained with the full dataset except the instance.
Revised: • Consistency score (C-score): C-score of each instance is calculated by estimating the prediction accuracy of the instance attained by the model trained with the full dataset except the instance.

Original: [Equation] 
Revised: [Equation] [Equation]

Original: where f ( x i ; S ) is trained network using subset S , and ˆ E r denotes empirical averaging with r i.i.d.
Revised: where f ( x i ; S ) is trained network using subset S , and ˆ E r denotes empirical averaging with r i.i.d.

Original: samples of such subsets.
Revised: samples of such subsets.

Original: • Error L2-Norm (EL2N): The EL2N score of a training sample ( x i , y i ) is deﬁned to be E [ (cid:107) f
Revised: • Error L2-Norm (EL2N): The EL2N score of a training sample ( x i , y i ) is deﬁned to be E [ (cid:107) f

Original: ( W ( t ) , x i ) − y
Revised: ( W ( t ) , x i ) − y

Original: i
Revised: i

Original: (cid:107) 2 ] where f ( W ( t ) , x ) is the output of the neural network for the sample ( x , y ) at the t -th step.
Revised: (cid:107) 2 ] where f ( W ( t ) , x ) is the output of the neural network for the sample ( x , y ) at the t -th step.

Original: • Forgetting score: Forgetting score is deﬁned as the number of times during training (until time step T ) the decision of that sample switches from a correct one to an incorrect one: Forgetting ( x i , y i ) is deﬁned as
Revised: • Forgetting score: Forgetting score is deﬁned as the number of times during training (until time step T ) the decision of that sample switches from a correct one to an incorrect one: Forgetting ( x i , y i ) is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Data pruning experiment We report the mean of the results after three independent runs.
Revised: Data pruning experiment We report the mean of the results after three independent runs.

Original: The shaded regions indicate the standard deviation.
Revised: The shaded regions indicate the standard deviation.

Original: We acquire each result by training a network using a dataset pruned by speciﬁed portions, where the training instances are ordered by each score.
Revised: We acquire each result by training a network using a dataset pruned by speciﬁed portions, where the training instances are ordered by each score.

Original: We compute the number of iterations at which all data can be used in one epoch, and use the same number of iterations in all pruning experiments for a fair comparison.
Revised: We compute the number of iterations at which all data can be used in one epoch, and use the same number of iterations in all pruning experiments for a fair comparison.

Original: When we prune the dataset, we remove the instances from each class by the same amount, so as to preserve the original proportion between classes.
Revised: When we prune the dataset, we remove the instances from each class by the same amount, so as to preserve the original proportion between classes.

Original: As will be shown in Section E, the distribution of scores is different among classes, so an imbalance problem between classes may occur if the data pruning is performed without considering the portion of classes within the dataset. 
Revised: As will be shown in Section H, the distribution of scores is different among classes, so an imbalance problem between classes may occur if the data pruning is performed without considering the portion of classes within the dataset. D A DDITIONAL E XPERIMENTS WITH T WO M ORE B ASELINES D. A NOTHER DIRECTIONS OF DATA VALUATION There are two additional branches of related works for data valuation, in addition to the methodologies described in the Section 1. The ﬁrst branch uses the inﬂuence function.

Original: C D ETAILS OF KERNEL VELOCITY
Revised: The inﬂuence function approximates the degree of change of parameters when speciﬁc data enters and leaves the training dataset, so it determines which data is valuable by calculating the effect of the data on learning. The second branch uses coreset selection. Coresets are weighted subsets of the data selected to resemble the model training using the full dataset. Coresets may need to be updated as training progresses.

Revised: Baseline algorithms for data valuation We use representative scores in each branch, TracIn

Revised: (Pruthi et al., 2020) for inﬂuence function and CRAIG (Mirzasoleiman et al., 2020) for coreset selection, as additional baselines.

Revised: Detailed deﬁnitions of the scores are described below:

Revised: • TracIn: TracIn

Revised: CheckPoint (TracInCP) value between two data points is deﬁned as the weighted sum of dot products of the loss gradients calculated at the two data points.

Revised: The gradients are obtained from the k checkpoints { t 1 , . .

Revised: t k } of the model in the middle of training and the weight η t i is the learning rate at each checkpoint t i :

Revised: [Equation]

Revised: where l ( w t i , z ) is loss function at t i -th step with model parameter w t i .

Revised: • CRAIG:

Revised: CoResets for Accelerating Incremental Gradient descent (CRAIG) is an algorithm that solves the optimization problem, which ﬁnds a subset that preserves the gradient of the total loss:

Revised: [Equation]

Revised: where f i ( w ) =

Revised: l ( w, ( x i , y i )) is loss for the data ( x i , y i ) with model parameter w , V is the full dataset and S is the coresets with size r ,

Revised: and p i be the softmax output for data ( x i , y i ) .

Revised: In CRAIG, the gradient f i ( w ) is approximated by p i − y

Revised: i when cross entropy loss is used with soft-max at the last layer.

Revised: [Figure]

Revised: D. D ATA P RUNING EXPERIMENTS

Revised: In this section, we conduct data pruning experiments, similar to Section 3.1.

Revised: We conduct the experiments using the above two additional baselines, TracIn and CRAIG, separately from the experiments of the main text, since the two algorithms require additional assumptions/resources that have not been used for the baselines considered in the main text: TracIn requires a validation set to calculate the data values; CRAIG does not select a ﬁxed subset of data to be used throughout the training, but keeps updating the subset of the data (coresets) to be used for training every 10 epochs.

Revised: Experimental details As described in the Pruthi et al.

Revised: (2020), we calculate the TracIn score by using the gradients of the parameters of the network’s last layer.

Revised: The check points are set at every 20 epochs, starting from the end of the ﬁrst 20th epoch.

Revised: We use 5 checkpoints for FMNIST andcheckpoints for CIFAR-10 and CIFAR-100.

Revised: We create a validation set composed of 1,000 samples by taking a part of the test dataset, and calculate TracIn score with this validation set.

Revised: As TracIn score is deﬁned between two data points ( z, z (cid:48) ) as in equation 11, we set the score of each training sample z by averaging TracIn scores

Revised: TracInCP( z, z (cid:48) ) over all samples z (cid:48) in the validation set.

Revised: In CRAIG, the subset selection is performed every 10 epochs.

Revised: We only test CRAIG in pruning lowvalued data but not in the reverse order (pruning high-valued data), since CRAIG extracts coresets to be used with per element weights for preserving the gradient of total loss but does not give what are the high-valued (equal weight) samples.

Revised: TracIn score and CRAIG are calculated at the networks same as those used in the experiment in

Revised: Section 3.1: ResNet18 for FMNIST and CIFAR-10, and ResNet50 for CIFAR-100 dataset.

Revised: The other experimental details are the same as Table 3 in Appendix § C.

Revised: Experimental results Similar to data pruning experiment in Section 3.1, in Figure 7, the ﬁrst row shows the test accuracy of the model trained with different subset sizes, when low-scoring (regular) examples are removed ﬁrst, and the second row shows the similar result but when high-scoring (irregular) examples are removed ﬁrst.

Revised: We report the mean of the results after three independent runs, and the shaded regions indicate the standard deviation.

Revised: The red-curve (random) is the result when randomly ordered examples are used.

Revised: When pruning low-scoring data ﬁrst, it is preferable to maintain the accuracy up to a large removing portion (a small training set); when pruning highscoring data ﬁrst, the rapid drop of performance is preferable since it means that the score can detect high-value samples, necessary for generalization of the model.

Revised: Our CG-score can achieve better performances than the Tracin score and competitive performances compared to the CRAIG algorithm.

Revised: Since CRAIG updates the coresets over the training, it can choose different semantics of the data suitable for each phase of the training, which results in better performance.

Revised: This result may imply the effectiveness of the scheduled batch selection, e.g., curriculum learning, in training neural networks.

Revised: We inspect that the performance of TracIn may heavily depend on the size of the validation set and also the possible domain discrepancy between training and test datasets.

Revised: In our test, there is no domain discrepancy, but if there exists a domain shift between the test dataset and the training dataset, TracIn may perform better than other methods with the help of validation set.

Revised: D. D ETECTING L ABEL N OISE

Revised: We also compared the performance of our CG-score in detecting mislabeled data with that of TracIn.

Revised: As suggested in Pruthi et al.

Revised: (2020), we use ‘self-inﬂuence’ of each training example, i.e., the inﬂuence of a training point on its own loss during the training process, TracInCP

Revised: ( z, z ) , to identify mislabeled data.

Revised: In Pruthi et al.

Revised: (2020), it was shown that mislabeled examples tend to have higher

Revised: TracIn values, and thus TracIn values can be effectively used in identifying mislabeled examples.

Revised: In Fig. 8, we show the comparison of our method with TracIn in identifying mislabeled examples in FMNIST and CIFAR-100 datasets, respectively, each of which includes 20% label noise.

Revised: TracIn achieves better performance in CIFAR-100, but ours outperformed TracIn in FMNIST.

Revised: Since TraIn measures the ‘self-inﬂuence’ of each training example over the training, starting from 20th epoch, for relatively simpler dataset such as FMNIST, some mislabeled instances could have already been memorized at the network, which makes them not detectable by the TracIn values.

Revised: On the other hand, our method better detects mislabeled data for easier datasets as discussed in Sec. 3.2.

Revised: Thus, we can conclude that depending on data complexity, the outperforming method can be changing.

Revised: E R OBUSTNESS OF CG- SCORE OVER MODEL VARIANTS

Revised: Our CG-score is derived based on the theoretical analysis of the generalization error bounds on overparameterized two-layer ReLU activated neural network.

Revised: To demonstrate that our CG-score is effective in more complicated networks, in the main text (Section 3.1), we used ResNets to evaluate our score for data pruning experiments.

Revised: To further demonstrate the robustness of our score against model changes, in this section, we report the results of data pruning experiments on CIFARdataset using DenseNetBC-100, a more complicated convolutional network.

Revised: The implementation details of the DenseNet follows that of Huang et al.

Revised: We use the same hyperparameter and optimization method summarized in Table 3.

Revised: [Figure]

Revised: In Figure 9, the left ﬁgure shows the test accuracy of the model trained with different subset sizes, when low-scoring (regular) examples are removed ﬁrst, and the right ﬁgure shows the similar result but when high-scoring (irregular) examples are removed ﬁst.

Revised: We report the mean of the result after two independent runs, and the shaded regions indicate the standard deviation.

Revised: The red-curve (random) is the result when randomly ordered examples are used.

Revised: We can observe a similar trend as in

Revised: Fig.

Revised: 2, the experimental results using ResNets, in spite of the model change.

Revised: Our CG-score achieve competitive performances as the other baselines.

Revised: The reason that we observe a rapid performance degradation at a smaller training data portion in the left ﬁgure (removing low value data ﬁrst) is due to the characteristics of leave-one-out method we used in the calculation of the CG-score.

Original: We calculate the kernel velocity of 10 groups of instances, where each group is composed ofsamples.
Revised: Removing a typical sample from a dataset does not change the generalization error bounds much, since similar samples already exist in the dataset. Thus, typical samples tend to have low CG-scores when the score is measured by the leave-one-out method. However, when we remove 50% of instances, samples sharing the typicality can be excluded simultaneously from the dataset, which might cause severe degradation of the generalization capability of the neural network. Thus, for a smaller training data portion, it can be better to make sure at least a small portion of typical samples is indeed included in the training.

Revised: Similar observations have been made in Swayamdipta et al.

Original:  The following is how we construct each group: Sort CIFAR-10 examples in ascending order by the CG-score, divide the examples into 10 groups, and select 125 consecutive samples from the beginning of each group.
Revised: We calculate the kernel velocity of 10 groups of instances, where each group is composed ofsamples. The following is how we construct each group: Sort CIFAR-10 examples in ascending order by the CG-score, divide the examples into 10 groups, and select 125 consecutive samples from the beginning of each group.

Original: We calculate the NTK kernel velocity as described in Paul et al.
Revised: We calculate the NTK kernel velocity as described in Paul et al.

Original: Let C be the number of classes.
Revised: Let C be the number of classes.

Original: Let f ( c ) t ( x i ) be the c -th logit value for input x i at the t -epoch.
Revised: Let f ( c ) t ( x i ) be the c -th logit value for input x i at the t -epoch.

Original: When W ( t ) is the parameters of the model, the c -th logit gradient at input x i is ψ ( c ) t ( x i )
Revised: When W ( t ) is the parameters of the model, the c -th logit gradient at input x i is ψ ( c ) t ( x i )

Original: =
Revised: =

Original: ∇ W ( t ) f
Revised: ∇ W ( t ) f

Original: ( c ) t ( x i ) ∈ R N .
Revised: ( c ) t ( x i ) ∈ R N .

Original: Then, the datadependent NTK submatrix of a group of m samples S :
Revised: Then, the datadependent NTK submatrix of a group of m samples S :

Original: = { x a 1 , . .
Revised: = { x a 1 , . .

Original: , x a m } is deﬁned as K t ( S ) =
Revised: , x a m } is deﬁned as K t ( S ) =

Original: Ψ t ( S )
Revised: Ψ t ( S )

Original: Ψ t ( S ) (cid:62) , where Ψ t ( S ) ∈
Revised: Ψ t ( S ) (cid:62) , where Ψ t ( S ) ∈

Original: R mC × N is constructed by placing { ψ ( c ) t ( x a i ) }
Revised: R mC × N is constructed by placing { ψ ( c ) t ( x a i ) }

Original: c ∈ [ C ] ,i ∈
Revised: c ∈ [ C ] ,i ∈

Original: [ m ] in rows of Ψ t ( S ) .
Revised: [ m ] in rows of Ψ t ( S ) .

Original: The kernel velocity is deﬁned as
Revised: The kernel velocity is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Lastly, we provide some explanations of what the kernel velocity measures if we deﬁne it for ﬁnite-width ReLU activated 2-layer neural network.
Revised: Lastly, we provide some explanations of what the kernel velocity measures if we deﬁne it for ﬁnite-width ReLU activated 2-layer neural network.

Original: Remind that our analysis uses the
Revised: Remind that our analysis uses the

Original: Gram matrix H ∞ deﬁned in equation 1, which is derived for an overparameterized ReLU activated 2-layer neural network.
Revised: Gram matrix H ∞ deﬁned in equation 1, which is derived for an overparameterized ReLU activated 2-layer neural network.

Original: We can generalize the deﬁnition of the Gram matrix assuming a ﬁnite-width network, similar to K t ( S ) , as follows.
Revised: We can generalize the deﬁnition of the Gram matrix assuming a ﬁnite-width network, similar to K t ( S ) , as follows.

Original: The output of the network isf
Revised: The output of the network is

Original: [Equation]
Revised: [Figure]

Original: [Equation] x )
Revised: f

Original: [Equation]
Revised: [Equation]

Original:  x ) , and the gradient with respect to W = ( w 1 , . . . , w m ) ∈
Revised: σ ( w (cid:62) r x ) , and the gradient with respect to W = ( w 1 , . . . , w m ) ∈

Original: R d [Equation] is ∇ 
Revised: R d × m is ∇ W f

Original: [Equation] x [Equation]
Revised: ( x i )

Original: [Equation]
Revised: [Equation]

Original: 0 } , a 2 1 { w (cid:62) 2 x i ≥ 0 } , . .
Revised: 0 } , a 2 1 { w (cid:62) 2 x i ≥ 0 } , . .

Original: , a m 1 { w (cid:62) m x i ≥ 0 } ] .
Revised: , a m 1 { w (cid:62) m x i ≥ 0 } ] .

Original: Denoting the network parameters at the t -th step as W ( t ) , we can deﬁne the Gram matrix H t as ( H t ) ij = ∇ W ( t ) f ( x [Equation] ∇ W [Equation]
Revised: Denoting the network parameters at the t -th step as W ( t ) , we can deﬁne the Gram matrix H t as ( H t ) ij = ∇ W ( t ) f ( x i ) ∇ W ( t ) f ( x j ) (cid:62) =

Original: = [Equation] { w (cid:62) r x i ≥ 0 , w (cid:62) r x j ≥ 0 } , which is proportion to the number of neurons in the hidden layer activated both for x i and x j at the epoch.
Revised: x (cid:62) i x j 1 m (cid:80) mr =1 1 { w (cid:62) r x i ≥ 0 , w (cid:62) r x j ≥ 0 } , which is proportion to the number of neurons in the hidden layer activated both for x i and x j at the epoch.

Original: We can deﬁne the kernel velocity similar to equation 11 by calculating H t for a subset of data, and replacing K t ( S ) by H t ( S ) .
Revised: We can deﬁne the kernel velocity similar to equation 13 by calculating H t for a subset of data, and replacing K t ( S ) by H t ( S ) .

Original: For this case, the high kernel velocity implies that H t differs much from H t +1 , i.e., the portion of neurons activated for pairs of instances changes rapidly during training.
Revised: For this case, the high kernel velocity implies that H t differs much from H t +1 , i.e., the portion of neurons activated for pairs of instances changes rapidly during training.

Original: D V ISUALIZATION OF EXAMPLES SORTED BY CG- SCORE
Revised: G V ISUALIZATION OF EXAMPLES SORTED BY CG- SCORE

Original:  In Fig. 6, we show examples of MNIST (Fig.6a) and FMNIST (Fig.
Revised: Analysis of MNIST and FMINST by CG-score In Fig. 10, we show examples of MNIST (Fig.

Original: 6b) images sorted by CGscore.
Revised: 10a) and FMNIST (Fig. 10b) images sorted by CG-score.

Original: The top/bottom three rows show the top-/bottom-ranked examples, respectively.
Revised: The top/bottom three rows show the top-/bottom-ranked examples, respectively.

Original: We can observe that the examples with the lowest CG-score are regular examples representing each class and they look similar to each other, while the examples with the highest scores are irregular and they look different among themselves, among which we can identify (possibly) mislabeled instances, marked with red rectangles.
Revised: We can observe that the examples with the lowest CGscore are regular examples representing each class and they look similar to each other, while the examples with the highest scores are irregular and they look different among themselves, among which we can identify (possibly) mislabeled instances, marked with red rectangles.

Original: Similarly, inCIFAR-10 images (Fig. 6c), we observe that low-scoring [Figure]
Revised: Similarly, in

Original:  examples (bottom two rows) are regular ones sharing similar features representing the class while high-scoring examples (top two rows) are irregular ones.
Revised: CIFAR-10 images (Fig. 10c), we observe that low-scoring examples (bottom two rows) are regular ones sharing similar features representing the class while high-scoring examples (top two rows) are irregular ones.

Original:  Fig.
Revised: Analysis of CIFAR-100 by CG-score Fig.

Original: 7 shows the examples of CIFAR-100 dataset.
Revised: 11 shows the examples of CIFAR-100 dataset.

Original: Fig.
Revised: Fig.

Original: 7a shows the means and standard variations of 100 classes in CIFAR-100 dataset.
Revised: 11a shows the means and standard variations of 100 classes in CIFAR-100 dataset.

Original: Among 100 classes, ‘orange’ and ’plain’ classes have small CG-score means and variances, while ‘bottle’ and ’ﬂatﬁsh’ classes have larger CG-score means and variances.
Revised: Amongclasses, ‘orange’ and ’plain’ classes have small CG-score means and variances, while ‘bottle’ and ’ﬂatﬁsh’ classes have larger CG-score means and variances.

Original: We display ten top-/bottom-ranked examples of these four classes, with the histograms of the scores in Fig. 
Revised: We display ten top-/bottom-ranked examples of these four classes, with the histograms of the scores in Fig. 11b. [Figure]

Revised: Analysis of Tiny ImageNet by CG-score To check the effectiveness of CG-score in analyzing more complicated dataset, we compute CG-score on the high-resolution tiny ImageNet dataset ( by 64 resolution, 100,000 samples of 200 classes) as well.

Revised: The CG-score is computed with sampling ratio of 1:14 by averaging the results from 10 independents runs.

Original: 7b.
Revised: Fig.

Revised: 12 show the examples of Tiny

Revised: Imagenet datset.

Revised: Fig.

Revised: 12a show the means and standard variations of classes.

Revised: We examine two easiest classes, ‘Sulfur butterﬂy’ and ‘Jellyﬁsh’, having relatively smaller mean and std of CGscore, and two difﬁcult classes, ‘Pill bottle’ and ‘Syringe’, having higher mean and std of CG-score, and display ten top-/bottom-ranked examples of these four classes with CG-score histograms in Fig.

Revised: 12b.

Revised: For the two easy classes, low-scoring examples look very much similar to each other and share some typical attributes (color and shape).

Revised: On the other hand, for two difﬁcult classes, even lowscoring examples do not look very similar to each other but rather diverse.

Revised: From the analysis, we can see that our CG-score is still effective in examining high-resolution complicated dataset, and the score reﬂects the instance-wise structural regularities, which can be used in analyzing or improving learning algorithms.

Revised: [Figure]

Original: E S CORE DISTRIBUTION FOR PUBLIC DATASETS
Revised: H S CORE DISTRIBUTION FOR PUBLIC DATASETS

Revised: Score distributions

Original: In Fig.
Revised: In Fig.

Original: 8, we compare the CG-score distributions of three public datasets,FMNIST, CIFAR-10 and
Revised: 13, we compare the CG-score distributions of three public datasets,

Original:  CIFAR-100 before and after we artiﬁcially corrupt 20% of instances with random label noise.
Revised: FMNIST, CIFAR-10 and CIFAR-100 before and after we artiﬁcially corrupt 20% of instances with random label noise.

Original: The top row shows the distributions before the corruption, and the bottom row shows the distributions after the corruption, where the orange color displays the distribution of the CG-score for 20% instances with label noise and the blue displays that for 80% instances with clean label.
Revised: The top row shows the distributions before the corruption, and the bottom row shows the distributions after the corruption, where the orange color displays the distribution of the CG-score for 20% instances with label noise and the blue displays that for 80% instances with clean label.

Original: The gap between scores of clean data and noisy data, which enables us to detect the noise by the CG-score, is larger for relatively simpler dataset, FMINST, than those for CIFAR-10/100. 
Revised: The gap between scores of clean data and noisy data, which enables us to detect the noise by the CG-score, is larger for relatively simpler dataset, FMINST, than those for CIFAR-10/100. Detecting mislabeled instances at high noise rate We conduct additional experiments to check the detectability of mislabeled instances by our CG-score, when the noise ratio increases to 50% and even to 80% for FMNIST and CIFAR-10 dataset. In the main text, we considered a mild noise ratio of 20% and reported the result in Fig. The results for higher noise cases are shown in Fig.

Revised: In the CIFAR-10 dataset, when the noise ratio is 80%, each class includes 1,000 correctly labeled images and 4,000 mislabeled images, composed of 445 samples coming from each of the other nine classes.

Revised: Even for such an extremely noisy case, our CG-score can effectively detect the mislabeled data, since our score can discover samples that have a relatively lower correlation to the majority of the samples of the same class as analyzed in Sec. 2.3.

Original: F C ORRELATIONS BETWEEN CG- SCORE AND P ARTIAL CG- SCORES
Revised: I C ORRELATIONS BETWEEN CG- SCORE AND P ARTIAL CG- SCORES

Original: From the CG-score, which is expressed as d − 1 i ( y (cid:62)− i h i ) 2 + 2 y i ( y (cid:62)− i h i ) + y 2 i d i at the equation 6, we can deﬁne three partial CG-scores, d − 1 i ( y (cid:62) − i h i ) 2 , 2 y i ( y (cid:62) − i h i ) , and y 2 i d i , respectively.
Revised: From the CG-score, which is expressed as d − 1 i ( y (cid:62)− i h i ) 2 + 2 y i ( y (cid:62)− i h i ) + y 2 i d i at the equation 6, we can deﬁne three partial CG-scores, d − 1 i ( y (cid:62)− i h i ) 2 , 2 y i ( y (cid:62)− i h i ) , and y 2 i d i , respectively.

Original: In this section, we analyze which term dominates the order of CG-scores for two public datasets, CIFAR10/100, by analyzing the correlation between the CG-score and the three partial CG-scores.
Revised: In this section, we analyze which term dominates the order of CG-scores for two public datasets, CIFAR10/100, by analyzing the correlation between the CG-score and the three partial CG-scores.

Original: We also examine whether the correlations vary when we change the size of the Gram matrix H ∞ , whose dimension changes according to the level of sub-sampling, explained in Sec.
Revised: We also examine whether the correlations vary when we change the size of the Gram matrix H ∞ , whose dimension changes according to the level of sub-sampling, explained in Sec.

Original: A.2.
Revised: A.2.

Original: We get H ∞ with different subsampling ratios, 1:1, 1:2, 1:3, and 1:4, and then calculate the CG-score and partial CGscores, deﬁned by the three partial terms.
Revised: We get H ∞ with different subsampling ratios, 1:1, 1:2, 1:3, and 1:4, and then calculate the CG-score and partial CGscores, deﬁned by the three partial terms.

Original: Table 3 shows the Spearman rank correlations between [Figure] [Table]
Revised: Table 5 shows the Spearman rank correlations between the CG-scores and the partial CG-scores.

Original: [Table] the CG-scores and the partial CG-scores. We can see that 2 y i ( y (cid:62) − i h i ) , which was utilized at noise detection experiments
Revised: We can see that 2 y i ( y (cid:62)− i h i ) , which was utilized at noise detection experiments, is the partial score having the largest correlation with the CG-score across all subsampling ratios.

Revised: On the other hand, correlations to other partial-scores are relatively low, and the correlations change much as the subsampling ratio changes.

Original: [Table]
Revised: [Table]

Original: is the partial score having the largest correlation with the CG-score across all subsampling ratios.

Original: On the other hand, correlations to other partial-scores are relatively low, and the correlations change much as the subsampling ratio changes.

Original: G F EATURE - SPACE CG- SCORE
Revised: J F EATURE - SPACE CG- SCORE

Original: Our original CG-score can be calculated by dataonly without any trained network.
Revised: Our original CG-score can be calculated by data without any trained network.

Original: In this section we check the validity of a new CG-score, which is deﬁned in terms of feature of data.
Revised: In this section we check the validity of a new CG-score, which is deﬁned in terms of feature of data [Table] and compare its characteristic with that of the original data-centric CG-score.

Original: We train ResNet using the full CIFAR-10 dataset to obtain each data’s feature, which is deﬁned at the output of the penultimate layer value (512 dimensions).
Revised: We train ResNet18 using the full CIFAR-10 dataset to obtain each data’s feature at various epochs, where the feature is deﬁned at the output of the penultimate layer value (512 dimensions).

Original: Then we calculate the feature-space CG score, which is calculated by  feature instead of the data itself, and check the Spearman rank correlation between the feature-space CG-score and the previous training-based scores including C-score, Forgetting score, and EL2N.
Revised: Then we calculate the feature-space CG scores, which are calculated by the feature instead of the data itself [Table] and compare how the Spearman rank correlation between the feature-space CG-scores and other scores, including the original CGscore and the previous training-based scores such as C-score, Forgetting score, and EL2N, change over the epochs at which the feature space CG-scores are calculated.

Original:  In Table 4, we can see that there exist correlations between the feature-space CG-score and previous training-based scores.
Revised: Figure 15 reports the result. In Figure 15, we can ﬁrst observe that the feature space CG-score and the original CG-score has the highest correlation at the very beginning of the training (epoch 1), but as the training progresses, the correlation decreases. This trend can be explained by the fact that the feature space CG-score computes the value of data in the learned embedding space, while our score computes the value of the original data without embedding it into a latent space. As training of a model progresses, the embedded data may incur bias in the data valuation, depending on a particular model or training algorithm, and thus the correlation between the feature space CG-score and the original data-centric

Original: The correlation between the original (data-centric) CG-score and feature-space CG-score is relatively low.
Revised: CG-score may decrease. On the other hand, correlation between feature space CG-score and other training-based scores (EL2N, Forgetting, and C-score) increases during the initial stage, and then decreases gradually. More speciﬁcally, for both EL2N and Forgetting score, which are computed at the same network as that of the feature space CG-score, the correlation increases as the epoch increases and then slightly drops and becomes saturated at a certain value.

Original:  This result implies that the feature-space CG score includes meaningful information for data valuation, correlated with other training-based scores, but the context of information might be different from that of the original CG-score.
Revised: The C-score, which was calculated in a different CNN network, shows a slightly different tendency compared to EL2N or forgetting score, and attains its peak at an earlier epoch, epoch 5. This result implies that the feature-space CG score includes meaningful information for data valuation, correlated with other training-based scores for overall epochs.

Original: The feature-space
Revised: However, the feature space

Revised: CG-score may incur some bias in data valuation as the training progresses.

Revised: Understanding the effectiveness of the feature-space CG score in diverse applications can be an interesting future research direction.

Revised: K S CHUR COMPLEMENT

Original: CG score can also be easily calculated if a pretrained network exists.
Revised: Calculating CG-score (equation 2) of n data instances requires the inversion of n × n matrix, which may cause expensive computational cost for a large n . Therefore, we provided computationally efﬁcient way to obtain the CG-score by using Schur complement equation 4. Here we provide some details to derive the inverse of a sub-block matrix using Schur complement. By Schur complement, we have

Original: Understanding and evaluating the feature-space CG score can be an interesting future research direction.
Revised: [Equation] Remind that H ∞ and ( H ∞ ) − 1 are denoted by [Equation] where H ∞ n − 1 , ( H ∞ ) − 1 n − 1 ∈ R ( n − 1) × ( n − 1) , g i , h i ∈ R n − 1 and c i , d i ∈ R , i.e.,

Revised: [Equation]

Revised: By substituting A , B , C , and D in equation 14 with ( H ∞ ) − 1 n − 1 , h i , h (cid:62) i , and d i respectively, we obtain that

Revised: [Equation]

Revised: Thus, ( H ∞ n − 1 ) − 1 =

Revised: ( H ∞ ) − 1 n − 1 − h

Revised: i h (cid:62) i

Revised: /d i .