Original: D ATA V ALUATION W ITHOUT T RAINING OF A M ODEL
Revised: D ATA V ALUATION W ITHOUT T RAINING OF A M ODEL

Original: A BSTRACT
Revised: A BSTRACT

Original: Many recent works on understanding deep learning try to quantify how much individual data instances influence the optimization and generalization of a model.
Revised: Many recent works on understanding deep learning try to quantify how much individual data instances influence the optimization and generalization of a model.

Original: Such attempts reveal characteristics and importance of individual instances, which may provide useful information in diagnosing and improving deep learning.
Revised: Such attempts reveal characteristics and importance of individual instances, which may provide useful information in diagnosing and improving deep learning.

Original: However, most of the existing works on data valuation require actual training of a model, which often demands high-computational cost.
Revised: However, most of the existing works on data valuation require actual training of a model, which often demands high-computational cost.

Original: In this paper, we provide a training-free data valuation score, called complexity-gap score , which is a datacentric score to quantify the influence of individual instances in generalization of two-layer overparameterized neural networks.
Revised: In this paper, we provide a training-free data valuation score, called complexity-gap score , which is a datacentric score to quantify the influence of individual instances in generalization of two-layer overparameterized neural networks.

Original: The proposed score can quantify irregularity of the instances and measure how much each data instance contributes in the total movement of the network parameters during training.
Revised: The proposed score can quantify irregularity of the instances and measure how much each data instance contributes in the total movement of the network parameters during training.

Original: We theoretically analyze and empirically demonstrate the effectiveness of the complexity-gap score in finding ‘irregular or mislabeled’ data instances, and also provide applications of the score in analyzing datasets and diagnosing training dynamics.
Revised: We theoretically analyze and empirically demonstrate the effectiveness of the complexity-gap score in finding ‘irregular or mislabeled’ data instances, and also provide applications of the score in analyzing datasets and diagnosing training dynamics.

Original: Our code is publicly available at https://github.com/JJchy/CG_score .
Revised: Our code is publicly available at https://github.com/JJchy/CG_score .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Creation of large datasets has driven development of deep learning in diverse applications including computer vision (Krizhevsky et al., 2012; Dosovitskiy et al., 2021), natural language processing (Vaswani et al., 2017; Brown et al., 2020) and reinforcement learning (Mnih et al., 2015; Silver et al., 2016).
Revised: Creation of large datasets has driven development of deep learning in diverse applications including computer vision (Krizhevsky et al., 2012; Dosovitskiy et al., 2021), natural language processing (Vaswani et al., 2017; Brown et al., 2020) and reinforcement learning (Mnih et al., 2015; Silver et al., 2016).

Original: To utilize the dataset in a more efficient and effective manner, some recent works have attempted to understand the role of individual data instances in training and generalization of neural networks.
Revised: To utilize the dataset in a more efficient and effective manner, some recent works have attempted to understand the role of individual data instances in training and generalization of neural networks.

Original: In (Ghorbani & Zou, 2019), a metric to quantify the contribution of each training instance in achieving a high test accuracy was analyzed under the assumption that not only the training data but also the test data is available.
Revised: In (Ghorbani & Zou, 2019), a metric to quantify the contribution of each training instance in achieving a high test accuracy was analyzed under the assumption that not only the training data but also the test data is available.

Original: Jiang et al.
Revised: Jiang et al.

Original: (2021) defined a score to identify irregular examples that need to be memorized during training, in order for the model to accurately classify the example.
Revised: (2021) defined a score to identify irregular examples that need to be memorized during training, in order for the model to accurately classify the example.

Original: All these previous methods for data valuation require actual training of a model to quantify the role of individual instances at the model.
Revised: All these previous methods for data valuation require actual training of a model to quantify the role of individual instances at the model.

Original: Thus, the valuation itself often requires high-computational cost, which may contradict some motivations of data valuation.
Revised: Thus, the valuation itself often requires high-computational cost, which may contradict some motivations of data valuation.

Original: For example, in (Ghorbani & Zou, 2019; Jiang et al., 2021), to examine the effect of individual data instances in training, one needs to train a model repeatedly while eliminating each instance or subsets of instances.
Revised: For example, in (Ghorbani & Zou, 2019; Jiang et al., 2021), to examine the effect of individual data instances in training, one needs to train a model repeatedly while eliminating each instance or subsets of instances.

Original: In (Swayamdipta et al., 2020; Toneva et al., 2019), on the other hand, training dynamics–the behavior of a model on each instance throughout the training–is analyzed to categorize data instances.
Revised: In (Swayamdipta et al., 2020; Toneva et al., 2019), on the other hand, training dynamics–the behavior of a model on each instance throughout the training–is analyzed to categorize data instances.

Original: When the motivation for data valuation lies at finding a subset of data that can approximate the full-dataset training to save the computational cost for training, the previous valuation methods might not be suitable, since they already require the training with the full dataset before one can figure out ‘important’ instances.
Revised: When the motivation for data valuation lies at finding a subset of data that can approximate the full-dataset training to save the computational cost for training, the previous valuation methods might not be suitable, since they already require the training with the full dataset before one can figure out ‘important’ instances.

Original: In this paper, our main contribution is on defining a training-free data valuation score, which can be directly computed from data and can effectively quantify the impact of individual instances in optimization and generalization of neural networks.
Revised: In this paper, our main contribution is on defining a training-free data valuation score, which can be directly computed from data and can effectively quantify the impact of individual instances in optimization and generalization of neural networks.

Original: The proposed score, called complexity-gap score , measures the gap in data complexity where a certain data instance is removed from the full dataset.
Revised: The proposed score, called complexity-gap score , measures the gap in data complexity where a certain data instance is removed from the full dataset.

Original: The data complexity measure was originally introduced in Arora et al.
Revised: The data complexity measure was originally introduced in Arora et al.

Original: (2019) to quantify the complexity of the full dataset, which was used in bounding the generalization error of overparameterized ∗ two-layer neural networks trained by gradient descent.
Revised: (2019) to quantify the complexity of the full dataset, which was used in bounding the generalization error of overparameterized ∗ two-layer neural networks trained by gradient descent.

Original: Different from that work, where the complexity of the full dataset was of main concern, our focus is on decomposing the effect of individual data instances in the training, and thus we newly introduce a complexity gap score (CG-score).
Revised: Different from that work, where the complexity of the full dataset was of main concern, our focus is on decomposing the effect of individual data instances in the training, and thus we newly introduce a complexity gap score (CG-score).

Original: We theoretically analyze and empirically demonstrate that the CG-score can quantify ‘irregularity’ of instances within each class, and thus can be used in identifying atypical examples, either due to the inherent irregularity of the instance or mislabeled classification.
Revised: We theoretically analyze and empirically demonstrate that the CG-score can quantify ‘irregularity’ of instances within each class, and thus can be used in identifying atypical examples, either due to the inherent irregularity of the instance or mislabeled classification.

Original: We also demonstrate that the proposed score has a close relation to ‘learning difficulty’ of the instances by analyzing the training dynamics of data instances.
Revised: We also demonstrate that the proposed score has a close relation to ‘learning difficulty’ of the instances by analyzing the training dynamics of data instances.

Original: Our key contributions are as below:
Revised: Our key contributions are as below:

Original: • Training-free data valuation: Different from previous methods for data valuation, most of which leverage the information from training itself, we provide a training-free data valuation score, CG-score, which is the data-centric score to quantify the effect of individual data instances in optimization and generalization of neural networks.
Revised: • Training-free data valuation: Different from previous methods for data valuation, most of which leverage the information from training itself, we provide a training-free data valuation score, CG-score, which is the data-centric score to quantify the effect of individual data instances in optimization and generalization of neural networks.

Original: • Geometric interpretation: We provide analysis that the CG-score can measure irregularity of each instance,
Revised: • Geometric interpretation: We provide analysis that the CG-score can measure irregularity of each instance,

Original: i
Revised: i

Original: e., it measures the average ‘similarity’ of an instance to the instances of the same class and the average ‘dissimilarity’ to the instances of different classes.
Revised: e., it measures the average ‘similarity’ of an instance to the instances of the same class and the average ‘dissimilarity’ to the instances of different classes.

Original: • Effectiveness of the score : We empirically demonstrate the effectiveness of the CG-score in data valuation.
Revised: • Effectiveness of the score : We empirically demonstrate the effectiveness of the CG-score in data valuation.

Original: We show that pruning data instances with small CG-score does not significantly degrade the generalization capability of a model, e
Revised: We show that pruning data instances with small CG-score does not significantly degrade the generalization capability of a model, e

Original: g., for CIFAR-10 we can prune 40% of the data with less than 1% of drop in test accuracy.
Revised: g., for CIFAR-10 we can prune 40% of the data with less than 1% of drop in test accuracy.

Original: Our scoring method is especially useful in data pruning, since different from other scores, which require the training with the full dataset, our method does not require any training of a model.
Revised: Our scoring method is especially useful in data pruning, since different from other scores, which require the training with the full dataset, our method does not require any training of a model.

Original: • Application of the score
Revised: • Application of the score

Original: : We provide potential applications of the CG-score in analyzing datasets and training dynamics.
Revised: : We provide potential applications of the CG-score in analyzing datasets and training dynamics.

Original: We analyze the histograms of the CG-score for various datasets to demonstrate that the CG-score can measure irregularity of the instances.
Revised: We analyze the histograms of the CG-score for various datasets to demonstrate that the CG-score can measure irregularity of the instances.

Original: We also demonstrate that the instances with higher CG-score are
Revised: We also demonstrate that the instances with higher CG-score are

Original: ‘difficult’ examples, which are learned slowly by the models, by comparing the loss and test accuracy curves and the evolution of Neural Tangent Kernel (NTK) submatrices of lowest/highest-scoring groups.
Revised: ‘difficult’ examples, which are learned slowly by the models, by comparing the loss and test accuracy curves and the evolution of Neural Tangent Kernel (NTK) submatrices of lowest/highest-scoring groups.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Different from many existing works where the effect of datasets on model training is analyzed as a whole, some recent works have focused on understanding the impact of individual data instances.
Revised: Different from many existing works where the effect of datasets on model training is analyzed as a whole, some recent works have focused on understanding the impact of individual data instances.

Original: Ghorbani & Zou (2019); Kwon et al. (2021) defined ‘Data Shapley’, to evaluate the value of each data instance, by measuring the average gap in performances when an instance is held-out from any subsets of a given training data.
Revised: Ghorbani & Zou (2019); Kwon et al. (2021) defined ‘Data Shapley’, to evaluate the value of each data instance, by measuring the average gap in performances when an instance is held-out from any subsets of a given training data.

Original: Jiang et al.
Revised: Jiang et al.

Original: (2021) defined consistency score (C-score) of each instance by estimating the prediction accuracy of the instance attained by the model trained with the full dataset except the instance.
Revised: (2021) defined consistency score (C-score) of each instance by estimating the prediction accuracy of the instance attained by the model trained with the full dataset except the instance.

Original: Both Data Shapley and C-score require multiple trainings of a model to compute the scores.
Revised: Both Data Shapley and C-score require multiple trainings of a model to compute the scores.

Original: Another main branch uses the training dynamics to identify ‘difficult’ instances for classification, either due to irregularity or mislabeling, by measuring different forms of confidence, stability or influence in the decision of the networks throughout the training (Toneva et al., 2019; Swayamdipta et al., 2020; Pruthi et al., 2020).
Revised: Another main branch uses the training dynamics to identify ‘difficult’ instances for classification, either due to irregularity or mislabeling, by measuring different forms of confidence, stability or influence in the decision of the networks throughout the training (Toneva et al., 2019; Swayamdipta et al., 2020; Pruthi et al., 2020).

Original: In (Baldock et al., 2021), the computational difficulty of an instance is defined as the number of hidden layers after which the networks’ prediction coincides with the prediction at the output.
Revised: In (Baldock et al., 2021), the computational difficulty of an instance is defined as the number of hidden layers after which the networks’ prediction coincides with the prediction at the output.

Original: With application of robust learning, some works quantify the difficulty of each instance by a ‘margin’ from the decision boundary (Zhang et al., 2021).
Revised: With application of robust learning, some works quantify the difficulty of each instance by a ‘margin’ from the decision boundary (Zhang et al., 2021).

Original: CRAIG (Mirzasoleiman et al., 2020) finds valuable subsets of data as coresets that preserve the gradient of the total loss.
Revised: CRAIG (Mirzasoleiman et al., 2020) finds valuable subsets of data as coresets that preserve the gradient of the total loss.

Original: All these previous methods are demonstrated to be effective in at least one or more applications of data valuation, including data pruning (Paul et al., 2021; Swayamdipta et al., 2020; Agarwal et al., 2022; Feldman & Zhang, 2020), importancebased weighted sampling (Chang et al., 2017; Koh & Liang, 2017), noise filtering (Li et al., 2020;
Revised: All these previous methods are demonstrated to be effective in at least one or more applications of data valuation, including data pruning (Paul et al., 2021; Swayamdipta et al., 2020; Agarwal et al., 2022; Feldman & Zhang, 2020), importancebased weighted sampling (Chang et al., 2017; Koh & Liang, 2017), noise filtering (Li et al., 2020;

Original: Lee et al., 2019b; Kim et al., 2021), robust learning (Ren et al., 2018; Pleiss et al., 2020), out-ofdistribution generalizations (Swayamdipta et al., 2020) or diverse sample generalization (Lee et al., 2021).
Revised: Lee et al., 2019b; Kim et al., 2021), robust learning (Ren et al., 2018; Pleiss et al., 2020), out-ofdistribution generalizations (Swayamdipta et al., 2020) or diverse sample generalization (Lee et al., 2021).

Original: However, all these methods require the training of a model with the full dataset (at least for a few optimization steps).
Revised: However, all these methods require the training of a model with the full dataset (at least for a few optimization steps).

Original: Recently, some works try to evaluate different subsets of training data by utilizing generalization error bound, induced by the NTK theory of neural networks (Wu et al., 2022;
Revised: Recently, some works try to evaluate different subsets of training data by utilizing generalization error bound, induced by the NTK theory of neural networks (Wu et al., 2022;

Original: Zhang & Zhang, 2022).
Revised: Zhang & Zhang, 2022).

Original: However, these methods do not provide a computationally-efficient way to quantify the effect of individual data instances.
Revised: However, these methods do not provide a computationally-efficient way to quantify the effect of individual data instances.

Original: Our data valuation method, on the other hand, is a data-centric method that can be efficiently calculated from data only without training of a model.
Revised: Our data valuation method, on the other hand, is a data-centric method that can be efficiently calculated from data only without training of a model.

Original:  C OMPLEXITY -G AP S CORE : D ATA V ALUATION WITHOUT T RAINING
Revised:  C OMPLEXITY -G AP S CORE : D ATA V ALUATION WITHOUT T RAINING

Original: In this section, we introduce a new data valuation score, called complexity-gap score , based on the analysis of overparameterized two-layer neural networks from Arora et al.
Revised: In this section, we introduce a new data valuation score, called complexity-gap score , based on the analysis of overparameterized two-layer neural networks from Arora et al.

Original: 3. P RELIMINARIES : DATA COMPLEXITY MEASURE IN TWO - LAYER NEURAL NETWORKS
Revised: 3. P RELIMINARIES : DATA COMPLEXITY MEASURE IN TWO - LAYER NEURAL NETWORKS

Original: We first review the result from Arora et al.
Revised: We first review the result from Arora et al.

Original: (2019), where a two-layer neural network trained by randomly initialized gradient descent is analyzed.
Revised: (2019), where a two-layer neural network trained by randomly initialized gradient descent is analyzed.

Original: Consider a two-layer ReLU activated neural network having m neurons in the hidden layer, of which the output is f W , a ( x ) = 1 √ m (cid:80)
Revised: Consider a two-layer ReLU activated neural network having m neurons in the hidden layer, of which the output is f W , a ( x ) = 1 √ m (cid:80)

Original: mr =1 a r σ ( w ⊤ r x ) where x ∈ R
Revised: mr =1 a r σ ( w ⊤ r x ) where x ∈ R

Original: d is the input, w 1 , . . . , w m ∈ R d are weight vectors in the first layer, a 1 , . .
Revised: d is the input, w 1 , . . . , w m ∈ R d are weight vectors in the first layer, a 1 , . .

Original: . , a m ∈ R are weights in the second layer, and σ ( x ) = max(0 , x ) is the ReLU activation function.
Revised: . , a m ∈ R are weights in the second layer, and σ ( x ) = max(0 , x ) is the ReLU activation function.

Original: Let W = ( w 1 , . . .
Revised: Let W = ( w 1 , . . .

Original: , w m ) ∈ R
Revised: , w m ) ∈ R

Original: d × m and a = ( a 1 , . .
Revised: d × m and a = ( a 1 , . .

Original: . , a m )
Revised: . , a m )

Original: ⊤ ∈ R m .
Revised: ⊤ ∈ R m .

Original: Assume that the network parameters are randomly initialized as w r (0) ∼ N (0 , κ 2 I d × d ) and a r ∼ unif ( {− 1 , 1 } ) , ∀ r ∈ [ m ] , where κ ∈ (0 , 1] is the size of random initialization.
Revised: Assume that the network parameters are randomly initialized as w r (0) ∼ N (0 , κ 2 I d × d ) and a r ∼ unif ( {− 1 , 1 } ) , ∀ r ∈ [ m ] , where κ ∈ (0 , 1] is the size of random initialization.

Original: The second layer a is then fixed and only the first layer W is optimized through gradient descent (GD) to minimize the quadratic loss, Φ( W )
Revised: The second layer a is then fixed and only the first layer W is optimized through gradient descent (GD) to minimize the quadratic loss, Φ( W )

Original: = 12 (cid:80) ni
Revised: = 12 (cid:80) ni

Original: =

Revised: =

Original: ( y i − u i )where u
Revised: ( y i − u i )where u

Original: i = f W , a ( x i ) and { ( x i , y i ) } ni =1 is the dataset drawn i.i.d.
Revised: i = f W , a ( x i ) and { ( x i , y i ) } ni =1 is the dataset drawn i.i.d.

Original: from an underlying distribution
Revised: from an underlying distribution

Original: D .
Revised: D .

Original: The GD update rule can be written as w r ( k + 1) − w r ( k ) =
Revised: The GD update rule can be written as w r ( k + 1) − w r ( k ) =

Original: − η ∂ Φ( W ( k ))
Revised: − η ∂ Φ( W ( k ))

Original: ∂ w r where η > 0 is the learning rate.
Revised: ∂ w r where η > 0 is the learning rate.

Original: The output of the network for the input x i at the k -th iteration is denoted by u i ( k ) =
Revised: The output of the network for the input x i at the k -th iteration is denoted by u i ( k ) =

Original: f W ( k ) , a ( x i ) .
Revised: f W ( k ) , a ( x i ) .

Original: For simplicity, it is assumed that ∥ x ∥ 2 = 1 and | y | ≤ 1 .
Revised: For simplicity, it is assumed that ∥ x ∥ 2 = 1 and | y | ≤ 1 .

Original: The data complexity measure governing the training of the two-layer ReLU activated neural network is defined in terms of the following Gram matrix H ∞ ∈ R n × n associated with ReLU activation:
Revised: The data complexity measure governing the training of the two-layer ReLU activated neural network is defined in terms of the following Gram matrix H ∞ ∈ R n × n associated with ReLU activation:

Original: [Equation]
Revised: [Equation]

Original: In Arora et al.
Revised: In Arora et al.

Original: (2019), a complexity measure of data was defined as y ⊤ ( H ∞ ) − 1 y where
Revised: (2019), a complexity measure of data was defined as y ⊤ ( H ∞ ) − 1 y where

Original: y =
Revised: y =

Original: ( y 1 , . .
Revised: ( y 1 , . .

Original: y n ) , and it was shown that this measure bounds the total movement of all neurons in W from their random initialization.
Revised: y n ) , and it was shown that this measure bounds the total movement of all neurons in W from their random initialization.

Original: Moreover, the data complexity measure bounds the generalization error by restricting the Rademacher complexity of the resulting function class.
Revised: Moreover, the data complexity measure bounds the generalization error by restricting the Rademacher complexity of the resulting function class.

Original: In the following, we write the eigen-decomposition of H ∞ as H ∞ =
Revised: In the following, we write the eigen-decomposition of H ∞ as H ∞ =

Original: (cid:80) ni
Revised: (cid:80) ni

Original: =

Revised: =

Original: λ i v i v ⊤ i where λ i ’s are ordered such that λ 1 ≥ λ 2 ≥ . . .
Revised: λ i v i v ⊤ i where λ i ’s are ordered such that λ 1 ≥ λ 2 ≥ . . .

Original: λ n .
Revised: λ n .

Original: Further, assuming λ n ≥ λ 0 > 0 , we can write ( H ∞ ) − 1 = (cid:80) ni
Revised: Further, assuming λ n ≥ λ 0 > 0 , we can write ( H ∞ ) − 1 = (cid:80) ni

Original: =

Revised: =

Original: ( λ i ) − 1 v i v ⊤ i .
Revised: ( λ i ) − 1 v i v ⊤ i .

Original: Theorem 1 (Informal version of (Arora et al., 2019)) .
Revised: Theorem 1 (Informal version of (Arora et al., 2019)) .

Original: Assume that λ min ( H ∞ )
Revised: Assume that λ min ( H ∞ )

Original: =
Revised: =

Original: λ n ≥ λ 0 > 0 .
Revised: λ n ≥ λ 0 > 0 .

Original: For sufficiently large width m , sufficiently small learning rate η > 0 and sufficiently small random initialization κ > 0 , with probability at least 1 − δ over the random initialization, we have
Revised: For sufficiently large width m , sufficiently small learning rate η > 0 and sufficiently small random initialization κ > 0 , with probability at least 1 − δ over the random initialization, we have

Original: a) Bound in loss : ∥ y − u ( k ) ∥ 

Revised: a) Bound in loss : ∥ y − u ( k ) ∥

Original: = (cid:118)(cid:117)(cid:117)(cid:116) n (cid:88)
Revised: = (cid:118)(cid:117)(cid:117)(cid:116) n (cid:88)

Original: [Equation]
Revised: [Equation]

Original: ) 2 k ( v ⊤ i y ) 2 + small constant , b) Bound in total movement
Revised: ) 2 k ( v ⊤ i y ) 2 + small constant , b) Bound in total movement

Original: of neurons
Revised: of neurons

Original: : ∥ W ( k ) − W (0) ∥ F ≤ (cid:113) y ⊤ ( H ∞ ) − 1 y + small constant , c) Bound in population loss: E ( x ,y )
Revised: : ∥ W ( k ) − W (0) ∥ F ≤ (cid:113) y ⊤ ( H ∞ ) − 1 y + small constant , c) Bound in population loss: E ( x ,y )

Original: ∼D [ l ( f W ( k ) , a ( x ) , y )] ≤ (cid:114) y ⊤ ( H ∞ ) − 1 y n + O   (cid:115) log nλ 0 δ n 
Revised: ∼D [ l ( f W ( k ) , a ( x ) , y )] ≤ (cid:114) y ⊤ ( H ∞ ) − 1 y n + O   (cid:115) log nλ 0 δ n 

Original:  , where a) and b) hold for all iteration k ≥ 0 of GD, and c) holds for k ≥ Ω (1 / ( ηλ 0 ) log( n/δ )) .
Revised:  , where a) and b) hold for all iteration k ≥ 0 of GD, and c) holds for k ≥ Ω (1 / ( ηλ 0 ) log( n/δ )) .

Original: This theorem shows that if the label vector y is aligned with top eigenvectors of H ∞ , i.e., ( v ⊤ i y ) is large for large λ i , then the loss decreases quickly and the total movement of neurons as well as the generalization error is small.
Revised: This theorem shows that if the label vector y is aligned with top eigenvectors of H ∞ , i.e., ( v ⊤ i y ) is large for large λ i , then the loss decreases quickly and the total movement of neurons as well as the generalization error is small.

Original: Thus, the data complexity measure y ⊤ ( H ∞ )
Revised: Thus, the data complexity measure y ⊤ ( H ∞ )

Original: − 1 y captures the complexity of data governing both the optimization and generalization of the overparameterized two-layer neural networks.
Revised: − 1 y captures the complexity of data governing both the optimization and generalization of the overparameterized two-layer neural networks.

Original: However, this quantity captures the complexity of the whole data.
Revised: However, this quantity captures the complexity of the whole data.

Original: To decompose the effect of individual instances, we newly define a complexity-gap score .
Revised: To decompose the effect of individual instances, we newly define a complexity-gap score .

Original: 3. C OMPLEXITY - GAP SCORE AND TRAINING DYNAMICS
Revised: 3. C OMPLEXITY - GAP SCORE AND TRAINING DYNAMICS

Original: We define the complexity-gap score (CG-score) of ( x i , y i ) as the difference between the data complexity measure when ( x i , y i ) is removed from a given dataset { ( x i , y i ) } ni =1 :
Revised: We define the complexity-gap score (CG-score) of ( x i , y i ) as the difference between the data complexity measure when ( x i , y i ) is removed from a given dataset { ( x i , y i ) } ni =1 :

Original: [Equation]
Revised: [Equation]

Original: where y − i is the label vector except the i -th sample point and H ∞− i is the ( n − 1) ×
Revised: where y − i is the label vector except the i -th sample point and H ∞− i is the ( n − 1) ×

Original: ( n − 1) matrix obtained by removing the i -th row and column of H ∞ .
Revised: ( n − 1) matrix obtained by removing the i -th row and column of H ∞ .

Original: We first emphasize that the proposed score can be easily calculated from given data without the need of training neural networks, as opposed to other data valuation scores requiring either a trained neural network or statistics calculated from training dynamics.
Revised: We first emphasize that the proposed score can be easily calculated from given data without the need of training neural networks, as opposed to other data valuation scores requiring either a trained neural network or statistics calculated from training dynamics.

Original: Yet, the proposed score captures two important properties on the training and generalization of data instance, implied by Theorem 1:
Revised: Yet, the proposed score captures two important properties on the training and generalization of data instance, implied by Theorem 1:

Original: An instance ( x i , y i ) with a large CG-score is a ‘difficult’ example, in the sense that removing it from the dataset reduces the generalization error bound by a large amount, which implies that the dataset without ( x i , y i ) is much easier to be learned and generalized.
Revised: An instance ( x i , y i ) with a large CG-score is a ‘difficult’ example, in the sense that removing it from the dataset reduces the generalization error bound by a large amount, which implies that the dataset without ( x i , y i ) is much easier to be learned and generalized.

Original: An instance ( x i , y i ) with a larger CG-score contributes more on the optimization and drives the total movement of neurons by a larger amount, measured by ∥ W ( k ) − W (0) ∥ F .
Revised: An instance ( x i , y i ) with a larger CG-score contributes more on the optimization and drives the total movement of neurons by a larger amount, measured by ∥ W ( k ) − W (0) ∥ F .

Original: We next discuss the computational complexity of calculating the CG-score.
Revised: We next discuss the computational complexity of calculating the CG-score.

Original: To calculate { CG( i ) } ni =1 , we need to take the inverse of matrices H ∞ and H ∞− i for all i ∈ [ n ] , which requires
Revised: To calculate { CG( i ) } ni =1 , we need to take the inverse of matrices H ∞ and H ∞− i for all i ∈ [ n ] , which requires

Original: O ( n 4 ) complexity when we use general O ( n 3 ) -complexity algorithm for the matrix inversion.
Revised: O ( n 4 ) complexity when we use general O ( n 3 ) -complexity algorithm for the matrix inversion.

Original: By using Schur complement, however, we can reduce this complexity to O ( n 3 ) .
Revised: By using Schur complement, however, we can reduce this complexity to O ( n 3 ) .

Original: Without loss of generality, we can assume i = n .
Revised: Without loss of generality, we can assume i = n .

Original: Denote H ∞ and ( H ∞ ) − 1 by
Revised: Denote H ∞ and ( H ∞ ) − 1 by

Original: [Equation]
Revised: [Equation]

Original: where H ∞ n − 1 , ( H ∞ ) − 1 n − 1 ∈ R ( n − 1) × ( n − 1) , g i , h i ∈ R n − 1 and c i , d i ∈ R .
Revised: where H ∞ n − 1 , ( H ∞ ) − 1 n − 1 ∈ R ( n − 1) × ( n − 1) , g i , h i ∈ R n − 1 and c i , d i ∈ R .

Original: From H ∞ ( H ∞ ) − 1 =
Revised: From H ∞ ( H ∞ ) − 1 =

Original: I n , we have g ⊤ i ( H ∞ ) − 1 n − 

Revised: I n , we have g ⊤ i ( H ∞ ) − 1 n −

Original: + c i h
Revised: + c i h

Original: ⊤ i = 0 , i.e., h ⊤ i = − c − 1 i g ⊤ i ( H ∞ ) − 1 n − 1 .
Revised: ⊤ i = 0 , i.e., h ⊤ i = − c − 1 i g ⊤ i ( H ∞ ) − 1 n − 1 .

Original: By Schur complement, ( H ∞− i ) − 1 , which is equal to ( H ∞ n − 1 ) − 1 for i = n , can be calculated as
Revised: By Schur complement, ( H ∞− i ) − 1 , which is equal to ( H ∞ n − 1 ) − 1 for i = n , can be calculated as

Original: [Equation]
Revised: [Equation]

Original: Since we have
Revised: Since we have

Original: [Equation]
Revised: [Equation]

Original: the CG-score, CG( i ) in equation 2, can be calculated by
Revised: the CG-score, CG( i ) in equation 2, can be calculated by

Original: [Equation]
Revised: [Equation]

Original: Thus, CG( i ) can be calculated by the n -th column ( h i , d i ) of ( H ∞ ) − 1 , without the need of calculating ( H ∞ − i ) − 1 when i = n .
Revised: Thus, CG( i ) can be calculated by the n -th column ( h i , d i ) of ( H ∞ ) − 1 , without the need of calculating ( H ∞ − i ) − 1 when i = n .

Original: The case for general i ̸ = n can also be readily solved by permuting the i -th row and column of H ∞ into the last positions.
Revised: The case for general i ̸ = n can also be readily solved by permuting the i -th row and column of H ∞ into the last positions.

Original: Correlation to other scores We show relation between our score and other data valuation scores that require the training of neural networks.
Revised: Correlation to other scores We show relation between our score and other data valuation scores that require the training of neural networks.

Original: Toneva et al.
Revised: Toneva et al.

Original: (2019) define ‘the forgetting score’ for each training example as the number of times during training the decision of that sample switches from a correct one to incorrect one.
Revised: (2019) define ‘the forgetting score’ for each training example as the number of times during training the decision of that sample switches from a correct one to incorrect one.

Original: Paul et al.
Revised: Paul et al.

Original: (2021), on the other hand, suggest the GraNd score, which is the expected loss gradient norm E [ ∥∇ W ( k ) l ( u ( k ) , y ) ∥ ] , to bound the contribution of each training example to the decrease of loss on any other example over a single gradient step.
Revised: (2021), on the other hand, suggest the GraNd score, which is the expected loss gradient norm E [ ∥∇ W ( k ) l ( u ( k ) , y ) ∥ ] , to bound the contribution of each training example to the decrease of loss on any other example over a single gradient step.

Original: The
Revised: The

Original: GraNd score is further approximated (under some assumptions) by the EL2N score, defined to be E [ | y − u ( k ) | ] where u ( k ) is the output of the neural network for the sample ( x, y ) at the k -th step.
Revised: GraNd score is further approximated (under some assumptions) by the EL2N score, defined to be E [ | y − u ( k ) | ] where u ( k ) is the output of the neural network for the sample ( x, y ) at the k -th step.

Original: Since | y − u ( k ) | , if rescaled, is an upper bound on 0–1 loss, (cid:80) k | y − u ( k ) | upper bounds forgetting score after rescaling.
Revised: Since | y − u ( k ) | , if rescaled, is an upper bound on 0–1 loss, (cid:80) k | y − u ( k ) | upper bounds forgetting score after rescaling.

Original: Thus, an example with a high forgetting score will also have a high GraND score and high EL2N score averaged over multiple time steps.
Revised: Thus, an example with a high forgetting score will also have a high GraND score and high EL2N score averaged over multiple time steps.

Original: We next relate our complexity-gap score to (cid:80) k
Revised: We next relate our complexity-gap score to (cid:80) k

Original: ( y − u ( k )) , and thus to all the three previous scores defined using training dynamics.
Revised: ( y − u ( k )) , and thus to all the three previous scores defined using training dynamics.

Original: It was shown in (Arora et al., 2019) that for the overparameterized two-layer networks trained by
Revised: It was shown in (Arora et al., 2019) that for the overparameterized two-layer networks trained by

Original: GD, the gap between the label vector and the network output at the step k can be approximated as y − u ( k )
Revised: GD, the gap between the label vector and the network output at the step k can be approximated as y − u ( k )

Original: ≈
Revised: ≈

Original: ( I − η H ∞ )
Revised: ( I − η H ∞ )

Original: k y .
Revised: k y .

Original: Thus, we get (cid:80) ∞
Revised: Thus, we get (cid:80) ∞

Original: k =

Revised: k =

Original: y − u ( k ) ≈ (cid:80) ∞ k =

Revised: y − u ( k ) ≈ (cid:80) ∞ k =

Original: ( I − η H ∞ )
Revised: ( I − η H ∞ )

Original: k y
Revised: k y

Original: = 1 η ( H ∞ ) − 1 y .
Revised: = 1 η ( H ∞ ) − 1 y .

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Without loss of generality, consider i = n .
Revised: Without loss of generality, consider i = n .

Original: Then, the accumulated difference between y i and u i ( k ) over k can be approximated as
Revised: Then, the accumulated difference between y i and u i ( k ) over k can be approximated as

Original: [Equation]
Revised: [Equation]

Original: Note that both the right-hand side of equation 7 and CG( i ) in equation 6 depend on the term (cid:0)
Revised: Note that both the right-hand side of equation 7 and CG( i ) in equation 6 depend on the term (cid:0)

Original: y ⊤− i h i / √ d
Revised: y ⊤− i h i / √ d

Original: i + y i √
Revised: i + y i √

Original: d i (cid:1) .
Revised: d i (cid:1) .

Original: Thus, we can expect that CG( i ) will be correlated with the scores related to training dynamics, including the forgetting score, GraND score and EL2N score.
Revised: Thus, we can expect that CG( i ) will be correlated with the scores related to training dynamics, including the forgetting score, GraND score and EL2N score.

Original: Different from those scores, our score can be directly calculated from the data without training of a model.
Revised: Different from those scores, our score can be directly calculated from the data without training of a model.

Original: Inversion of H ∞ is an effective step In the definition of CG( i ) in equation 2, we use the inverse of H ∞ to measure the alignment of the eigenvectors of H ∞ with the label vector y .
Revised: Inversion of H ∞ is an effective step In the definition of CG( i ) in equation 2, we use the inverse of H ∞ to measure the alignment of the eigenvectors of H ∞ with the label vector y .

Original: One could suggest another score that directly uses H ∞ instead of ( H ∞ ) − 1 , which can save the computation for inversion.
Revised: One could suggest another score that directly uses H ∞ instead of ( H ∞ ) − 1 , which can save the computation for inversion.

Original: However, we argue that the score calculated by using ( H ∞ ) − 1 includes more information than the score calculated by H ∞ due to the reason described below.
Revised: However, we argue that the score calculated by using ( H ∞ ) − 1 includes more information than the score calculated by H ∞ due to the reason described below.

Original: Let us define CG ′
Revised: Let us define CG ′

Original: ( i ) :
Revised: ( i ) :

Original: = y ⊤ H ∞
Revised: = y ⊤ H ∞

Original: y − y ⊤−
Revised: y − y ⊤−

Original: i H ∞−
Revised: i H ∞−

Original: i y − i .
Revised: i y − i .

Original: Without loss of generality, assume i = n .
Revised: Without loss of generality, assume i = n .

Original: Then, CG ′ ( i ) =
Revised: Then, CG ′ ( i ) =

Original: y i ( g ⊤ i y − i )
Revised: y i ( g ⊤ i y − i )

Original: + y 

Revised: + y

Original: i c i where g i =
Revised: i c i where g i =

Original: ( H ∞ ) 1:( n − 1) ,n and c i =
Revised: ( H ∞ ) 1:( n − 1) ,n and c i =

Original: H ∞ n,n as defined in equation 3.
Revised: H ∞ n,n as defined in equation 3.

Original: Since c i = 1 / 2 for all i ∈ [ n ] from the definition of H ∞ in equation 1 and y 2 i = 1 for y i = ± 1 , we have CG ′
Revised: Since c i = 1 / 2 for all i ∈ [ n ] from the definition of H ∞ in equation 1 and y 2 i = 1 for y i = ± 1 , we have CG ′

Original: ( i ) =
Revised: ( i ) =

Original: ( y i ( H ∞ y )
Revised: ( y i ( H ∞ y )

Original: i − 1 / 2) + 1 / 2 .
Revised: i − 1 / 2) + 1 / 2 .

Original: By using the approximation y − u ( k )
Revised: By using the approximation y − u ( k )

Original: ≈ ( I − η H ∞ )
Revised: ≈ ( I − η H ∞ )

Original: k y from Arora et al.
Revised: k y from Arora et al.

Original: (2019), we have y − u (1)
Revised: (2019), we have y − u (1)

Original: ≈ y − η H ∞
Revised: ≈ y − η H ∞

Original: y
Revised: y

Original: when k = 1 , which implies u (1) ≈ η H ∞ y .
Revised: when k = 1 , which implies u (1) ≈ η H ∞ y .

Original: Thus,
Revised: Thus,

Original: CG ′
Revised: CG ′

Original: ( i ) =
Revised: ( i ) =

Original: 2 η y i u
Revised: 2 η y i u

Original: i (1) + 1 / 2 .
Revised: i (1) + 1 / 2 .

Original: Note that an instance ( x i , y i ) has a large CG ′ ( i ) if y i u i (1) is large, i.e., the network output u i (1) after 1-time step has the same sign as the targeted label y i ∈ {− 1 , 1 } and has a large magnitude.
Revised: Note that an instance ( x i , y i ) has a large CG ′ ( i ) if y i u i (1) is large, i.e., the network output u i (1) after 1-time step has the same sign as the targeted label y i ∈ {− 1 , 1 } and has a large magnitude.

Original: Thus, CG ′ ( i ) measures how fast the training instance can be learned by the neural network.
Revised: Thus, CG ′ ( i ) measures how fast the training instance can be learned by the neural network.

Original: Different from CG ′
Revised: Different from CG ′

Original: ( i ) , our original score CG( i ) is correlated with the accumulated error between y i and u i ( k ) averaged over the training as shown in equation 7.
Revised: ( i ) , our original score CG( i ) is correlated with the accumulated error between y i and u i ( k ) averaged over the training as shown in equation 7.

Original: Thus, our original score CG( i ) , using the inverse of H ∞ , reflects the statistics averaged over the whole training steps.
Revised: Thus, our original score CG( i ) , using the inverse of H ∞ , reflects the statistics averaged over the whole training steps.

Original: In Table 1, we compare the Spearman’a rank correlation between CG-score/CG’-score and other data valuation scores including C-score (Jiang et al., 2021), forgetting score (Toneva et al., 2019) and EL2N score (Paul et al., 2021) for CIFAR-10/100 datasets.
Revised: In Table 1, we compare the Spearman’a rank correlation between CG-score/CG’-score and other data valuation scores including C-score (Jiang et al., 2021), forgetting score (Toneva et al., 2019) and EL2N score (Paul et al., 2021) for CIFAR-10/100 datasets.

Original: 1 We can observe that CG-score has higher correlations with the previous scores compared to those of CG’-score.
Revised: 1 We can observe that CG-score has higher correlations with the previous scores compared to those of CG’-score.

Original: In the rest of this paper, we focus on the CG-score for our data valuation score.
Revised: In the rest of this paper, we focus on the CG-score for our data valuation score.

Original: 3. G EOMETRIC INTERPRETATION OF THE COMPLEXITY - GAP SCORE
Revised: 3. G EOMETRIC INTERPRETATION OF THE COMPLEXITY - GAP SCORE

Original: We next provide geometric interpretation for the CG-score.
Revised: We next provide geometric interpretation for the CG-score.

Original: For the sake of simplicity, we consider binary dataset with n 2 samples having y i = 1 and n 2 samples having y i = − 1 .
Revised: For the sake of simplicity, we consider binary dataset with n 2 samples having y i = 1 and n 2 samples having y i = − 1 .

Original: We further assume that E [ H ∞ ij ] =
Revised: We further assume that E [ H ∞ ij ] =

Original: p if y i =
Revised: p if y i =

Original: y j and E
Revised: y j and E

Original: [ H ∞ ij ] =
Revised: [ H ∞ ij ] =

Original: q if y i ̸
Revised: q if y i ̸

Original: = y j for some | p | > | q | .
Revised: = y j for some | p | > | q | .

Original: Note that the diagonal entires H ∞ ii = 1 / 2 for all i ∈ [ n ] .
Revised: Note that the diagonal entires H ∞ ii = 1 / 2 for all i ∈ [ n ] .

Original: Thus, E [ H ∞ ] can be decomposed as E [ H ∞ ]
Revised: Thus, E [ H ∞ ] can be decomposed as E [ H ∞ ]

Original: = (cid:0) 12 − p (cid:1) I n + S where S is a block matrix with S = (cid:18) p q q p (cid:19) ⊗ I n/ 2 , and the resulting eigenvalues of E [ H ∞ ] are ( p + q ) n+ (cid:0) 1 2 − p (cid:1) , ( p − q )
Revised: = (cid:0) 12 − p (cid:1) I n + S where S is a block matrix with S = (cid:18) p q q p (cid:19) ⊗ I n/ 2 , and the resulting eigenvalues of E [ H ∞ ] are ( p + q ) n+ (cid:0) 1 2 − p (cid:1) , ( p − q )

Original: n+ (cid:0) 

Revised: n+ (cid:0)

Original: 2 − p (cid:1) and (cid:0) 1 2 − p (cid:1) with multiplicity ( n − 2) .
Revised: 2 − p (cid:1) and (cid:0) 1 2 − p (cid:1) with multiplicity ( n − 2) .

Original: When p =
Revised: When p =

Original: Θ( q ) and p =
Revised: Θ( q ) and p =

Original: o (1 /n ) , the matrix E [ H ∞ ]
Revised: o (1 /n ) , the matrix E [ H ∞ ]

Original: ≈ (1 / 2) I n .
Revised: ≈ (1 / 2) I n .

Original: From the representations of H ∞ and ( H ∞ ) − 1 in equation 3 and the implied relation h ⊤ i = − c − 1 i g ⊤ i ( H ∞ ) − 1 n − 1 , assuming H ∞ ≈ (1 / 2) I n and using c i = 1 / 2 , we can write h ⊤ i = − 4 g ⊤ i .
Revised: From the representations of H ∞ and ( H ∞ ) − 1 in equation 3 and the implied relation h ⊤ i = − c − 1 i g ⊤ i ( H ∞ ) − 1 n − 1 , assuming H ∞ ≈ (1 / 2) I n and using c i = 1 / 2 , we can write h ⊤ i = − 4 g ⊤ i .

Original: By using this approximation, our CG-score in equation 6 can
Revised: By using this approximation, our CG-score in equation 6 can

Original: [Figure]
Revised: [Figure]

Original: be approximated as
Revised: be approximated as

Original: [Equation]
Revised: [Equation]

Original: since d i ≈ 2 and y 2 i = 1 .
Revised: since d i ≈ 2 and y 2 i = 1 .

Original: Since y i ( y ⊤− i g i )
Revised: Since y i ( y ⊤− i g i )

Original: = (cid:80) j ∈{ [ n ]: y i =
Revised: = (cid:80) j ∈{ [ n ]: y i =

Original: y j } H ∞ ij − (cid:80) j ∈{ [ n ]: y i ̸
Revised: y j } H ∞ ij − (cid:80) j ∈{ [ n ]: y i ̸

Original: = y j } H ∞ ij and thus
Revised: = y j } H ∞ ij and thus

Original: E [ y i ( y ⊤− i g i )]
Revised: E [ y i ( y ⊤− i g i )]

Original: = ( p − q ) n=
Revised: = ( p − q ) n=

Original: o (1) , assuming p =
Revised: o (1) , assuming p =

Original: Θ( q ) and p =
Revised: Θ( q ) and p =

Original: o (1 /n ) , the second term − 

Revised: o (1 /n ) , the second term −

Original: y i ( y ⊤− i g i ) becomes much larger than the first term, which is proportional to ( y i ( y ⊤− i g i )) 2 .
Revised: y i ( y ⊤− i g i ) becomes much larger than the first term, which is proportional to ( y i ( y ⊤− i g i )) 2 .

Original: Thus, − 8 y i ( y ⊤− i g i ) is the main term that determines the order of { CG( i ) } ni =1 .
Revised: Thus, − 8 y i ( y ⊤− i g i ) is the main term that determines the order of { CG( i ) } ni =1 .

Original: Note that y i ( y ⊤−
Revised: Note that y i ( y ⊤−

Original: i g i ) measures the gap between (cid:80) j ∈{ [ n ]: y i =
Revised: i g i ) measures the gap between (cid:80) j ∈{ [ n ]: y i =

Original: y j } H ∞ ij and (cid:80) j ∈{ [ n ]: y i ̸
Revised: y j } H ∞ ij and (cid:80) j ∈{ [ n ]: y i ̸

Original: = y j } H ∞ ij .
Revised: = y j } H ∞ ij .

Original: Since H ∞ ij = x
Revised: Since H ∞ ij = x

Original: ⊤ i x j ( π − arccos( x ⊤ i x j )) 2 π , y i
Revised: ⊤ i x j ( π − arccos( x ⊤ i x j )) 2 π , y i

Original: ( y ⊤− i g i ) measures the gap between the average similarities of x i with other samples { x j } of the same class y j
Revised: ( y ⊤− i g i ) measures the gap between the average similarities of x i with other samples { x j } of the same class y j

Original: = y i
Revised: = y i

Original: and that with samples of the different class y j ̸ =
Revised: and that with samples of the different class y j ̸ =

Original: y i , where the similarity is measured by the cosine between the two samples ( x i , x j ) multiplied by the chance that ReLU is activated for both the samples.
Revised: y i , where the similarity is measured by the cosine between the two samples ( x i , x j ) multiplied by the chance that ReLU is activated for both the samples.

Original: Thus, we can expect two important trends regarding the CG-score:
Revised: Thus, we can expect two important trends regarding the CG-score:

Original: Regular vs. irregular samples: If a sample ( x i , y i ) is a ‘regular’ sample representing the class y i in the sense that it has a larger similarity with other samples of the same class than to samples from the different class, then it will have a large y i
Revised: Regular vs. irregular samples: If a sample ( x i , y i ) is a ‘regular’ sample representing the class y i in the sense that it has a larger similarity with other samples of the same class than to samples from the different class, then it will have a large y i

Original: ( y ⊤− i g i ) , resulting in a lower CG-score due to the minus sign.
Revised: ( y ⊤− i g i ) , resulting in a lower CG-score due to the minus sign.

Original: On the other hand, if a sample ( x i , y i ) is ‘irregular’ in the sense that it does not represent the class and has a small similarity with the samples of the same class, then y i ( y ⊤− i g i ) will be small, resulting in a higher CG-score.
Revised: On the other hand, if a sample ( x i , y i ) is ‘irregular’ in the sense that it does not represent the class and has a small similarity with the samples of the same class, then y i ( y ⊤− i g i ) will be small, resulting in a higher CG-score.

Original: Clean label vs. noisy label:
Revised: Clean label vs. noisy label:

Original: A sample with label noise tends to have a large CGscore since the order of two terms in subtraction at y i ( y ⊤ − i g i ) =
Revised: A sample with label noise tends to have a large CGscore since the order of two terms in subtraction at y i ( y ⊤ − i g i ) =

Original: (cid:80) j ∈{ [ n ]: y i =
Revised: (cid:80) j ∈{ [ n ]: y i =

Original: y j } H ∞ ij − (cid:80) j ∈{ [ n ]: y i ̸
Revised: y j } H ∞ ij − (cid:80) j ∈{ [ n ]: y i ̸

Original: = y j } H ∞ ij is effectively switched for label-noise samples.
Revised: = y j } H ∞ ij is effectively switched for label-noise samples.

Original: Thus, y i ( y ⊤− i g i ) tend to be negative for a sample with label noise, while it is positive for clean-label samples.
Revised: Thus, y i ( y ⊤− i g i ) tend to be negative for a sample with label noise, while it is positive for clean-label samples.

Original: We empirically demonstrate the above two trends by a controlled experiment.
Revised: We empirically demonstrate the above two trends by a controlled experiment.

Original: Consider two 3000dimensional Gaussian distributions with a fixed covariance matrix 0 .
Revised: Consider two 3000dimensional Gaussian distributions with a fixed covariance matrix 0 .

Original: 25 I 3000 and different means (1 , 0 , . . .
Revised: 25 I 3000 and different means (1 , 0 , . . .

Original: and ( − 1 , 0 , . .
Revised: and ( − 1 , 0 , . .

Original: . , 0) , representing class +1 and -1, respectively.
Revised: . , 0) , representing class +1 and -1, respectively.

Original: We generate 1000 samples from each class.
Revised: We generate 1000 samples from each class.

Original: We first check whether the structural assumptions on H ∞ hold with this synthetic dataset.
Revised: We first check whether the structural assumptions on H ∞ hold with this synthetic dataset.

Original: In Fig.
Revised: In Fig.

Original: 1a, we observe that ( H ∞ ) − 1 n − 1 can be well approximated by 2 I n − 1 , and thus the approximation for CG( i ) in equation 8 may hold well.
Revised: 1a, we observe that ( H ∞ ) − 1 n − 1 can be well approximated by 2 I n − 1 , and thus the approximation for CG( i ) in equation 8 may hold well.

Original: We then examine the correlation between the CG-score and the sample value at the first dimension.
Revised: We then examine the correlation between the CG-score and the sample value at the first dimension.

Original: In Fig.
Revised: In Fig.

Original: 1b, we can observe that instances near the decision boundary x 1 = 0 have higher CG-scores.
Revised: 1b, we can observe that instances near the decision boundary x 1 = 0 have higher CG-scores.

Original: This shows that irregular samples, e.g., samples near the boundary of the two classes, indeed have higher CG-scores.
Revised: This shows that irregular samples, e.g., samples near the boundary of the two classes, indeed have higher CG-scores.

Original: We further modify the generated samples by randomly flipping the labels of 10% samples and re-calculate the CG-scores of all the samples.
Revised: We further modify the generated samples by randomly flipping the labels of 10% samples and re-calculate the CG-scores of all the samples.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 1c, the samples with label noise tend to have higher CG-scores, agreeing with our intuition.
Revised: 1c, the samples with label noise tend to have higher CG-scores, agreeing with our intuition.

Original: In Appendix §B, we further discuss discriminating mislabeled data and atypical (but useful) data by using CG-score, and also show high/low-scoring examples for public datasets including MNIST, FMNIST, CIFAR-10 and ImageNet in Appendix §C.
Revised: In Appendix §B, we further discuss discriminating mislabeled data and atypical (but useful) data by using CG-score, and also show high/low-scoring examples for public datasets including MNIST, FMNIST, CIFAR-10 and ImageNet in Appendix §C.

Original: [Figure]
Revised: [Figure]

Original: Only CG-Score does not require any training of the model to calculate the scores, but it achieves competitive performances and outperforms the random baseline.
Revised: Only CG-Score does not require any training of the model to calculate the scores, but it achieves competitive performances and outperforms the random baseline.

Original: In (a), CG-score maintains the test accuracy up to significant removing portions; in (b), the test accuracy drops most rapidly for CGscore since the examples with high CG-score are essential in generalization of the model.
Revised: In (a), CG-score maintains the test accuracy up to significant removing portions; in (b), the test accuracy drops most rapidly for CGscore since the examples with high CG-score are essential in generalization of the model.

Original:  D ATA V ALUATION THROUGH C OMPLEXITY G AP S CORE 4. D ATA PRUNING EXPERIMENTS
Revised:  D ATA V ALUATION THROUGH C OMPLEXITY G AP S CORE 4. D ATA PRUNING EXPERIMENTS

Original: To evaluate the ability of the CG-score in identifying important examples, we design data pruning experiments, similar to those in Ghorbani & Zou (2019); Paul et al.
Revised: To evaluate the ability of the CG-score in identifying important examples, we design data pruning experiments, similar to those in Ghorbani & Zou (2019); Paul et al.

Original: We evaluate our score on three public datasets, FMNIST, CIFAR-10/100 and train ResNet networks (He et al., 2016),
Revised: We evaluate our score on three public datasets, FMNIST, CIFAR-10/100 and train ResNet networks (He et al., 2016),

Original: ResNet18 for FMNIST and CIFAR-10 and ResNet50 for CIFAR-100 dataset, respectively.
Revised: ResNet18 for FMNIST and CIFAR-10 and ResNet50 for CIFAR-100 dataset, respectively.

Original: As baseline methods for data valuation, we use three state-of-the-art scores, C-score (Jiang et al., 2021),
Revised: As baseline methods for data valuation, we use three state-of-the-art scores, C-score (Jiang et al., 2021),

Original: EL2N (Paul et al., 2021), and Forgetting score (Toneva et al., 2019), all of which require training of models for computation.
Revised: EL2N (Paul et al., 2021), and Forgetting score (Toneva et al., 2019), all of which require training of models for computation.

Original: On the contrary, our score is a data-centric measure and independent on the model.
Revised: On the contrary, our score is a data-centric measure and independent on the model.

Original: More details on the baselines and experiments are summarized in Appendix §D.
Revised: More details on the baselines and experiments are summarized in Appendix §D.

Original: In Figure 2, the first row shows the test accuracy of the model trained with different subset sizes when low-scoring (regular) examples are removed first, while the second row shows the similar result but when high-scoring (irregular) examples are removed first.
Revised: In Figure 2, the first row shows the test accuracy of the model trained with different subset sizes when low-scoring (regular) examples are removed first, while the second row shows the similar result but when high-scoring (irregular) examples are removed first.

Original: We report the mean of the result after five independent runs, and the shaded regions indicate the standard deviation.
Revised: We report the mean of the result after five independent runs, and the shaded regions indicate the standard deviation.

Original: The gray line (full dataset) represents the results when all data is used, while the red curve (random) represents the results when examples are randomly removed.
Revised: The gray line (full dataset) represents the results when all data is used, while the red curve (random) represents the results when examples are randomly removed.

Original: We can observe that when removing low-scoring examples first, networks maintain the test accuracy up to significant removing portions, i.e., 40% for CIFAR-10 and 20% for CIFAR-100 with less than 1% test accuracy drop.
Revised: We can observe that when removing low-scoring examples first, networks maintain the test accuracy up to significant removing portions, i.e., 40% for CIFAR-10 and 20% for CIFAR-100 with less than 1% test accuracy drop.

Original: Our CG-score, which does not require any training of a model, can achieve competitive performances as the other baseline methods and it also significantly outperforms the random baseline.
Revised: Our CG-score, which does not require any training of a model, can achieve competitive performances as the other baseline methods and it also significantly outperforms the random baseline.

Original: When removing the high-scoring examples first, the test accuracy drops most rapidly for the CG-score curve, implying that examples with the high CG-score are essential part of the data governing the generalization of the model.
Revised: When removing the high-scoring examples first, the test accuracy drops most rapidly for the CG-score curve, implying that examples with the high CG-score are essential part of the data governing the generalization of the model.

Original: We show the effectiveness of CG-score in more complicated models and also in fine-tuning by reporting the data pruning experiments at DenseNetBC-100 (Huang et al., 2017) and Vision Transformer (ViT)
Revised: We show the effectiveness of CG-score in more complicated models and also in fine-tuning by reporting the data pruning experiments at DenseNetBC-100 (Huang et al., 2017) and Vision Transformer (ViT)

Original: (Dosovitskiy et al., 2021) in Appendix
Revised: (Dosovitskiy et al., 2021) in Appendix

Original: §E. We also compare CG-score with two more baselines, TracIn (Pruthi et al., 2020) and CRAIG (Mirzasoleiman et al., 2020), in Appendix §F.
Revised: §E. We also compare CG-score with two more baselines, TracIn (Pruthi et al., 2020) and CRAIG (Mirzasoleiman et al., 2020), in Appendix §F.

Original: D ETECTING LABEL NOISE
Revised: D ETECTING LABEL NOISE

Original: In Section 3.3, we analyzed the CG-score and showed that the examples with label noise tend to have higher CG-score.
Revised: In Section 3.3, we analyzed the CG-score and showed that the examples with label noise tend to have higher CG-score.

Original: We next empirically investigate the ability of the CG-score in detecting label noise
Revised: We next empirically investigate the ability of the CG-score in detecting label noise

Original: [Figure]
Revised: [Figure]

Original: by artificially corrupting 20% of instances in FMNIST and CIFAR-100 with random label noise.
Revised: by artificially corrupting 20% of instances in FMNIST and CIFAR-100 with random label noise.

Original: We first examine the distribution of the CG-scores for each dataset after the corruption in Figure 3a.
Revised: We first examine the distribution of the CG-scores for each dataset after the corruption in Figure 3a.

Original: We can observe that for both datasets, the examples with corrupted labels (orange) tend to have higher
Revised: We can observe that for both datasets, the examples with corrupted labels (orange) tend to have higher

Original: CG-scores than those with clean labels (blue).
Revised: CG-scores than those with clean labels (blue).

Original: For a relatively simpler dataset, FMNIST (left), the
Revised: For a relatively simpler dataset, FMNIST (left), the

Original: CG-score histograms can be more clearly separable between the clean and noisy groups, compared to a more complex dataset, CIFAR-100 (right).
Revised: CG-score histograms can be more clearly separable between the clean and noisy groups, compared to a more complex dataset, CIFAR-100 (right).

Original: With this observation, we can anticipate that the
Revised: With this observation, we can anticipate that the

Original: CG-score can have a better detectability of label noise for relatively simpler datasets.
Revised: CG-score can have a better detectability of label noise for relatively simpler datasets.

Original: We evaluate the noise detectability of the CG-score by checking a varying portion of the examples sorted by the CG-score (highest first) in Fig.
Revised: We evaluate the noise detectability of the CG-score by checking a varying portion of the examples sorted by the CG-score (highest first) in Fig.

Original: 3b.
Revised: 3b.

Original: The curves indicate the fraction of noisy examples included in the examined subset.
Revised: The curves indicate the fraction of noisy examples included in the examined subset.

Original: We compare our score with two other scores, Forgetting score and
Revised: We compare our score with two other scores, Forgetting score and

Original: EL2N, as well as a random baseline and the oracle.
Revised: EL2N, as well as a random baseline and the oracle.

Original: We can observe that the CG-score achieves the best performance, near that of the oracle, for FMINST, and competitive performances for CIFAR100.
Revised: We can observe that the CG-score achieves the best performance, near that of the oracle, for FMINST, and competitive performances for CIFAR100.

Original: In the plot, we also compare the performance of the CG-score with that of ‘Partial CG-score,’ which is a new score defined by a sub-term of the CG-score.
Revised: In the plot, we also compare the performance of the CG-score with that of ‘Partial CG-score,’ which is a new score defined by a sub-term of the CG-score.

Original: In the CG-score in equation 6, we have three terms, but only the second term 

Revised: In the CG-score in equation 6, we have three terms, but only the second term

Original: y i ( y ⊤−
Revised: y i ( y ⊤−

Original: i h i ) uses the label information of the data ( x i , y i ) .
Revised: i h i ) uses the label information of the data ( x i , y i ) .

Original: Thus, we examined the ability of this term only, defined as the ‘Partial CG-score’, in detecting the label noise, and found that the CG-score and partial CG-score have similar performances in detecting label noise, which implies that the label-noise detectability of the CG-score mainly comes from the term 2 y i
Revised: Thus, we examined the ability of this term only, defined as the ‘Partial CG-score’, in detecting the label noise, and found that the CG-score and partial CG-score have similar performances in detecting label noise, which implies that the label-noise detectability of the CG-score mainly comes from the term 2 y i

Original: ( y ⊤− i h i ) .
Revised: ( y ⊤− i h i ) .

Original: More experiments with higher noise rate are reported in Appendix §G.
Revised: More experiments with higher noise rate are reported in Appendix §G.

Original:  C OMPLEXITY -G AP S CORE AND D EEP L EARNING P HENOMENA
Revised:  C OMPLEXITY -G AP S CORE AND D EEP L EARNING P HENOMENA

Original: We next demonstrate that the CG-score can capture ‘learning difficulty’ of data instances.
Revised: We next demonstrate that the CG-score can capture ‘learning difficulty’ of data instances.

Original: 5. C OMPLEXITY GAP SCORE CAPTURES LEARNING DIFFICULTY
Revised: 5. C OMPLEXITY GAP SCORE CAPTURES LEARNING DIFFICULTY

Original: It has been widely observed that there exists an order of instances in which a model learns to classify the data correctly and this order is robust within model types (Wu et al., 2021).
Revised: It has been widely observed that there exists an order of instances in which a model learns to classify the data correctly and this order is robust within model types (Wu et al., 2021).

Original: Many previous scoring functions use this type of observation and define the ‘difficulty’ of a given instance based on the speed at which the model’s prediction converges (Toneva et al., 2019; Swayamdipta et al., 2020).
Revised: Many previous scoring functions use this type of observation and define the ‘difficulty’ of a given instance based on the speed at which the model’s prediction converges (Toneva et al., 2019; Swayamdipta et al., 2020).

Original: Our data-centric CG-score is originally defined based on the gap in the generalization error bounds as in equation 2, but it also reflects the convergence speed of the instance, as analyzed in equation 7.
Revised: Our data-centric CG-score is originally defined based on the gap in the generalization error bounds as in equation 2, but it also reflects the convergence speed of the instance, as analyzed in equation 7.

Original: We empirically demonstrate this relation by analyzing the training dynamics of CIFAR-10 dataset trained in ResNet18 network.
Revised: We empirically demonstrate this relation by analyzing the training dynamics of CIFAR-10 dataset trained in ResNet18 network.

Original: We first sort the data instances in ascending order using the CG-score, and divide the data into 10 equal-sized subgroups.
Revised: We first sort the data instances in ascending order using the CG-score, and divide the data into 10 equal-sized subgroups.

Original: We then measure the mean of loss and training accuracy for the 10 subgroups as training progresses.
Revised: We then measure the mean of loss and training accuracy for the 10 subgroups as training progresses.

Original: Fig. 4a and Fig.
Revised: Fig. 4a and Fig.

Original: 4b show the mean of loss and the training accuracy throughout the training for the 10 subgroups, respectively, where the blue color represents the low-scoring groups and the red color represents the high-scoring groups.
Revised: 4b show the mean of loss and the training accuracy throughout the training for the 10 subgroups, respectively, where the blue color represents the low-scoring groups and the red color represents the high-scoring groups.

Original: We can observe that the mean loss and accuracy converge faster for low-scoring groups, while it takes longer to converge for high-scoring groups.
Revised: We can observe that the mean loss and accuracy converge faster for low-scoring groups, while it takes longer to converge for high-scoring groups.

Original: This indicates that the CG-score is highly correlated with the ‘difficulty’
Revised: This indicates that the CG-score is highly correlated with the ‘difficulty’

Original: [Figure]
Revised: [Figure]

Original: D ATA SAMPLES DRIVING MOVEMENT OF NEURONS
Revised: D ATA SAMPLES DRIVING MOVEMENT OF NEURONS

Original: We next investigate the relation between the CG-score and the evolution velocity of the datadependent Neural Tangent Kernel (NTK) (Fort et al., 2020).
Revised: We next investigate the relation between the CG-score and the evolution velocity of the datadependent Neural Tangent Kernel (NTK) (Fort et al., 2020).

Original: NTK has been widely used to approximate the evolution of an infinite-width deep learning model via linearization around initial weights, when the network is trained by gradient descent with a sufficiently small learning rate (Jacot et al., 2018).
Revised: NTK has been widely used to approximate the evolution of an infinite-width deep learning model via linearization around initial weights, when the network is trained by gradient descent with a sufficiently small learning rate (Jacot et al., 2018).

Original: The main idea is that in the limit of an infinite width, the network parameters do not move very far from its initialization throughout the training, so that the learning process can be approximated by a linear process along the tangent space to the manifold of the model’s function class at the initialization (Lee et al., 2019a).
Revised: The main idea is that in the limit of an infinite width, the network parameters do not move very far from its initialization throughout the training, so that the learning process can be approximated by a linear process along the tangent space to the manifold of the model’s function class at the initialization (Lee et al., 2019a).

Original: However, as observed in Fort et al.
Revised: However, as observed in Fort et al.

Original: (2020), for a finite-width network, the tangent kernel is not constant but it rather rapidly changes over time, especially at the beginning of the training.
Revised: (2020), for a finite-width network, the tangent kernel is not constant but it rather rapidly changes over time, especially at the beginning of the training.

Original: To quantify this change, Fort et al.
Revised: To quantify this change, Fort et al.

Original: (2020) addressed the data-dependent
Revised: (2020) addressed the data-dependent

Original: Kernel Gram matrix, which is the Gram matrix of the logit Jacobian, and defined the Kernel velocity as the cosine distance between two NTK Gram matrices before and after one epoch of training.
Revised: Kernel Gram matrix, which is the Gram matrix of the logit Jacobian, and defined the Kernel velocity as the cosine distance between two NTK Gram matrices before and after one epoch of training.

Original: In
Revised: In

Original: Paul et al.
Revised: Paul et al.

Original: (2021), the Kernel velocity was used to evaluate the subgroups of data instances to figure out the subgroup driving the learning and the change in the NTK feature space.
Revised: (2021), the Kernel velocity was used to evaluate the subgroups of data instances to figure out the subgroup driving the learning and the change in the NTK feature space.

Original: We conduct similar experiments for subgroups of data, divided according to our CG-score.
Revised: We conduct similar experiments for subgroups of data, divided according to our CG-score.

Original: Fig.
Revised: Fig.

Original: 4c shows the Kernel velocities for 10 different subgroups of CIFAR-10 data trained in ResNet18.
Revised: 4c shows the Kernel velocities for 10 different subgroups of CIFAR-10 data trained in ResNet18.

Original: Each group is composed of 125 consecutive instances from each level, where the level is defined by dividing the full dataset, sorted in ascending order by the CG-score, into 10 groups.
Revised: Each group is composed of 125 consecutive instances from each level, where the level is defined by dividing the full dataset, sorted in ascending order by the CG-score, into 10 groups.

Original: The higher level (red) is composed of instances having higher CG-scores, while the lower level (blue) is composed of instances having lower CG-scores.
Revised: The higher level (red) is composed of instances having higher CG-scores, while the lower level (blue) is composed of instances having lower CG-scores.

Original: We can observe that the samples with high CG-score (red) maintains higher Kernel velocity throughout the training, which means that NTK Gram matrix evolves by a larger amount for the samples of high CG-score.
Revised: We can observe that the samples with high CG-score (red) maintains higher Kernel velocity throughout the training, which means that NTK Gram matrix evolves by a larger amount for the samples of high CG-score.

Original: Thus, we can hypothesize that the instances with high CG-score are ‘difficult’ examples the network may struggle to optimize and try to fit throughout the training.
Revised: Thus, we can hypothesize that the instances with high CG-score are ‘difficult’ examples the network may struggle to optimize and try to fit throughout the training.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We proposed the CG-score, a data-centric valuation score, to quantify the effect of individual data instances in optimization and generalization of overparameterized two-layer neural networks trained by gradient descent.
Revised: We proposed the CG-score, a data-centric valuation score, to quantify the effect of individual data instances in optimization and generalization of overparameterized two-layer neural networks trained by gradient descent.

Original: We theoretically and empirically demonstrated that the CG-score can identify ‘irregular’ instances within each class, and can be used as a score to select instances essential for generalization of a model or in filtering instances with label noise.
Revised: We theoretically and empirically demonstrated that the CG-score can identify ‘irregular’ instances within each class, and can be used as a score to select instances essential for generalization of a model or in filtering instances with label noise.

Original: We also showed the close relation between the CG-score and learning difficulty of instances by analyzing training dynamics.
Revised: We also showed the close relation between the CG-score and learning difficulty of instances by analyzing training dynamics.

Original: Interesting open problems related to the CG-score include 1) providing theoretical justification of the score for more general deep neural networks and 2) improving the score by modifying the definition in terms of ‘features’ of data instances.
Revised: Interesting open problems related to the CG-score include 1) providing theoretical justification of the score for more general deep neural networks and 2) improving the score by modifying the definition in terms of ‘features’ of data instances.

Original: In Appendix §H, we provide further discussion for ‘featurespace CG-score’ and report experimental results, demonstrating the effectiveness of the score.
Revised: In Appendix §H, we provide further discussion for ‘featurespace CG-score’ and report experimental results, demonstrating the effectiveness of the score.

Original: A CKNOWLEDGEMENT
Revised: A CKNOWLEDGEMENT

Original: This research was supported by the National Research Foundation of Korea under grant 2021R1C1C11008539, and by the Ministry of Science and ICT, Korea, under the IITP (Institute for Information and Communications Technology Panning and Evaluation) grant No.2020-
Revised: This research was supported by the National Research Foundation of Korea under grant 2021R1C1C11008539, and by the Ministry of Science and ICT, Korea, under the IITP (Institute for Information and Communications Technology Panning and Evaluation) grant No.2020-

Original: A E XTENSIONS TO M ULTI - CLASS C OMPLEXITY G AP SCORE
Revised: A E XTENSIONS TO M ULTI - CLASS C OMPLEXITY G AP SCORE

Original: In this section, we explain the details of how we calculate the CG-scores for multi-label public datasets.
Revised: In this section, we explain the details of how we calculate the CG-scores for multi-label public datasets.

Original: A. C ALCULATION OF CG- SCORE FOR MULTI - LABEL DATASETS
Revised: A. C ALCULATION OF CG- SCORE FOR MULTI - LABEL DATASETS

Original: In defining the CG-score as equation 2, we assumed the binary datasets y ∈ {± 1 } with inputs having a fixed norm ∥ x ∥ 2 = 1 .
Revised: In defining the CG-score as equation 2, we assumed the binary datasets y ∈ {± 1 } with inputs having a fixed norm ∥ x ∥ 2 = 1 .

Original: To calculate the CG-score for multi-label ( k -class) public datasets, we first normalize all the inputs to have ∥ x ∥ 2 = 1 .
Revised: To calculate the CG-score for multi-label ( k -class) public datasets, we first normalize all the inputs to have ∥ x ∥ 2 = 1 .

Original: We then calculate the CG-score for examples of each class j ∈ [ k ] , assuming that all the examples from class j have label +1 and the rest of the examples from any other classes have label − 1 .
Revised: We then calculate the CG-score for examples of each class j ∈ [ k ] , assuming that all the examples from class j have label +1 and the rest of the examples from any other classes have label − 1 .

Original: In detail, let y =
Revised: In detail, let y =

Original: ( y 1 , y 2 , ..., y n ) ∈ { 1 , 2 , . . .
Revised: ( y 1 , y 2 , ..., y n ) ∈ { 1 , 2 , . . .

Original: k } n be the label vector for n data instances and, without loss of generality, assume that x 1 , x 2 , . . .
Revised: k } n be the label vector for n data instances and, without loss of generality, assume that x 1 , x 2 , . . .

Original: , x l belong to class 1, i.e. y 1 , y 2 , ..., y l = 1 .
Revised: , x l belong to class 1, i.e. y 1 , y 2 , ..., y l = 1 .

Original: Then, to calculate the CG-score for x 1 , x 2 , . . .
Revised: Then, to calculate the CG-score for x 1 , x 2 , . . .

Original: x l , we generate the gram matrix H ∞ with ( x 1 , 1) , ( x 2 , 1) , . . .
Revised: x l , we generate the gram matrix H ∞ with ( x 1 , 1) , ( x 2 , 1) , . . .

Original: , ( x l , 1) , ( x l +1 , − 1) , ( x l +2 , − 1) , . . .
Revised: , ( x l , 1) , ( x l +1 , − 1) , ( x l +2 , − 1) , . . .

Original: , ( x n , − 1) and calculate the CG-scores of x 1 , x 2 , . .
Revised: , ( x n , − 1) and calculate the CG-scores of x 1 , x 2 , . .

Original: , x l as the two-label case.
Revised: , x l as the two-label case.

Original: A. S TOCHASTIC METHOD TO CALCULATE CG- SCORES
Revised: A. S TOCHASTIC METHOD TO CALCULATE CG- SCORES

Original: Since the calculation of the CG-score requires taking the inverse of a n × n -dimensional matrix H ∞ where n is the number of total samples, it would demand expensive memory and computational cost for large n .
Revised: Since the calculation of the CG-score requires taking the inverse of a n × n -dimensional matrix H ∞ where n is the number of total samples, it would demand expensive memory and computational cost for large n .

Original: To lower the complexity, we sub-sample the examples with class − 1 so that the ratio between class +1 and class -1 is reduced from 1 : ( k − 1) to 1 : 3 for MNIST and FMINST, 1 :for CIFAR-10 and CIFAR-100.
Revised: To lower the complexity, we sub-sample the examples with class − 1 so that the ratio between class +1 and class -1 is reduced from 1 : ( k − 1) to 1 : 3 for MNIST and FMINST, 1 :for CIFAR-10 and CIFAR-100.

Original: We repeat this process 10 times by randomly sampling examples of label − 1 and then average out the calculated CG-score of each example from class +1 .
Revised: We repeat this process 10 times by randomly sampling examples of label − 1 and then average out the calculated CG-score of each example from class +1 .

Original: A. J USTIFICATION OF STOCHASTIC METHOD TO CALCULATE CG- SCORES
Revised: A. J USTIFICATION OF STOCHASTIC METHOD TO CALCULATE CG- SCORES

Original: To justify the stochastic method in calculating the CG-scores, we conduct an experiment to check whether the CG-scores calculated by the stochastic method converges well to the true CG-scores utilizing the full dataset.
Revised: To justify the stochastic method in calculating the CG-scores, we conduct an experiment to check whether the CG-scores calculated by the stochastic method converges well to the true CG-scores utilizing the full dataset.

Original: We created a subset of the CIFAR-10 dataset, the Small-CIFAR-10 dataset, which consists of 1,000 instances for each label (a total of 10,000 instances).
Revised: We created a subset of the CIFAR-10 dataset, the Small-CIFAR-10 dataset, which consists of 1,000 instances for each label (a total of 10,000 instances).

Original: Then, we compared the CG-score calculated by 10,000x10,000 Gram matrix H ∞ of the full dataset (true CG-score) with the CG-score calculated by the stochastic method, where the stochastic CG-score for the instances from a class (class 1) are calculated by subsampling the samples from any other classes (class -1) with the ratio between class 1 and -1 equal to 1 : 1 (size 2,000x2,000), 1 : 2 (size 3,000x3,000), 1 :(size 4,000x4,000) and 1 : 4 (size 5,000x5,000) instead of 1 : 9 .
Revised: Then, we compared the CG-score calculated by 10,000x10,000 Gram matrix H ∞ of the full dataset (true CG-score) with the CG-score calculated by the stochastic method, where the stochastic CG-score for the instances from a class (class 1) are calculated by subsampling the samples from any other classes (class -1) with the ratio between class 1 and -1 equal to 1 : 1 (size 2,000x2,000), 1 : 2 (size 3,000x3,000), 1 :(size 4,000x4,000) and 1 : 4 (size 5,000x5,000) instead of 1 : 9 .

Original: We repeat this process multiple times by randomly sampling examples of label − 1 and then average out the calculated CG-score of each example from class +1 .
Revised: We repeat this process multiple times by randomly sampling examples of label − 1 and then average out the calculated CG-score of each example from class +1 .

Original: In Figure 5, we plot the Spearman’s rank correlation and Pearson correlation between the true CGscore and the stochastic CG-score for each ratio (different colors) as the number of random sampling increases.
Revised: In Figure 5, we plot the Spearman’s rank correlation and Pearson correlation between the true CGscore and the stochastic CG-score for each ratio (different colors) as the number of random sampling increases.

Original: We can observe that the correlations converge to a certain number as the number of random sampling increases, and the amount of correlation increases as the size of the matrix (the number of instances included in defining H ∞ ) increases.
Revised: We can observe that the correlations converge to a certain number as the number of random sampling increases, and the amount of correlation increases as the size of the matrix (the number of instances included in defining H ∞ ) increases.

Original: The values of correlations obtained after 20 independent runs are 0.829, 0.914, 0.953, and 0.973 for Spearman rank correlations and 0.796, 0.898, 0.943, and 0.967 for Pearson rank correlations.
Revised: The values of correlations obtained after 20 independent runs are 0.829, 0.914, 0.953, and 0.973 for Spearman rank correlations and 0.796, 0.898, 0.943, and 0.967 for Pearson rank correlations.

Original: Recommended number of runs for stochastic calculation.
Revised: Recommended number of runs for stochastic calculation.

Original: The proper number of runs in stochastic calculation of the CG-score may need to be determined by the size and the number of classes of the datasets.
Revised: The proper number of runs in stochastic calculation of the CG-score may need to be determined by the size and the number of classes of the datasets.

Original: We recommend the number of runs large enough to include at least a half of the whole dataset in calculating the CG-score for samples of each class.
Revised: We recommend the number of runs large enough to include at least a half of the whole dataset in calculating the CG-score for samples of each class.

Original: As an example, for the
Revised: As an example, for the

Original: CIFAR-100 dataset, where each class includes 500 images, when the sampling ratio between the class of interest and the rest of classes is 1:4, we calculate the score for 500 images from the class of interest by using 2,000 images from the rest of 99 classes.
Revised: CIFAR-100 dataset, where each class includes 500 images, when the sampling ratio between the class of interest and the rest of classes is 1:4, we calculate the score for 500 images from the class of interest by using 2,000 images from the rest of 99 classes.

Original: Then, about 20 images are selected from each of the 99 classes.
Revised: Then, about 20 images are selected from each of the 99 classes.

Original: To cover at least a half of the images per class (250), we need to repeat the runs about 10 times (ignoring overlap of samples in each run).
Revised: To cover at least a half of the images per class (250), we need to repeat the runs about 10 times (ignoring overlap of samples in each run).

Original: For the FMNIST/CIFAR-10 datasets, the similar calculation shows that only 2-3 runs will be enough.
Revised: For the FMNIST/CIFAR-10 datasets, the similar calculation shows that only 2-3 runs will be enough.

Original: B D ISCRIMINATING MISLABELED DATA FROM ATYPICAL ( BUT USEFUL ) DATA BY CG- SCORES
Revised: B D ISCRIMINATING MISLABELED DATA FROM ATYPICAL ( BUT USEFUL ) DATA BY CG- SCORES

Original: In Section 3.3, we showed that irregular samples, either from input itself or mislabelling, tend to have high CG-scores.
Revised: In Section 3.3, we showed that irregular samples, either from input itself or mislabelling, tend to have high CG-scores.

Original: Discriminating mislabeled data and atypical (but useful) data is a major challenge in data valuation, since both the mislabeled data and atypical data are irregular in the data distribution and tend to have high CG-scores.
Revised: Discriminating mislabeled data and atypical (but useful) data is a major challenge in data valuation, since both the mislabeled data and atypical data are irregular in the data distribution and tend to have high CG-scores.

Original: The same challenge has been observed with previous valuation scores such as forgetting score (Toneva et al., 2019) and EL2N score (Paul et al., 2021).
Revised: The same challenge has been observed with previous valuation scores such as forgetting score (Toneva et al., 2019) and EL2N score (Paul et al., 2021).

Original: However, we find that mislabeled data usually has a higher CG-score than atypical data, and this tendency gives us the possibility to separate mislabeled data from the rest of the clean data.
Revised: However, we find that mislabeled data usually has a higher CG-score than atypical data, and this tendency gives us the possibility to separate mislabeled data from the rest of the clean data.

Original: To check the tendency, we perform a data window experiment.
Revised: To check the tendency, we perform a data window experiment.

Original: The data instances are sorted in ascending order by the CG-score, and we compare the test accuracy of a neural network trained with 50% of training instances selected from an offset% to (offset+50)% scoring group, for different offset points of { 0 , 5 , 10 , . . .
Revised: The data instances are sorted in ascending order by the CG-score, and we compare the test accuracy of a neural network trained with 50% of training instances selected from an offset% to (offset+50)% scoring group, for different offset points of { 0 , 5 , 10 , . . .

Original: For example, when the offset is 20%, we select the data instances from 20% to 70% scoring examples.
Revised: For example, when the offset is 20%, we select the data instances from 20% to 70% scoring examples.

Original: When the training instances do not include mislabeled data, we can expect that the window experiment will show higher accuracy as the offset increases up to 50%.
Revised: When the training instances do not include mislabeled data, we can expect that the window experiment will show higher accuracy as the offset increases up to 50%.

Original: We add 20% of random label noise to FMNIST and CIFAR-10 datasets to see how the trend changes when the dataset includes mislabeled instances.
Revised: We add 20% of random label noise to FMNIST and CIFAR-10 datasets to see how the trend changes when the dataset includes mislabeled instances.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 6 (a) and (b), the test accuracy increases until the offset reaches 30% and then drops after the point.
Revised: 6 (a) and (b), the test accuracy increases until the offset reaches 30% and then drops after the point.

Original: When the offset is 30%, the 50%-width window includes 30% to 80% scoring examples.
Revised: When the offset is 30%, the 50%-width window includes 30% to 80% scoring examples.

Original: Since 20% mislabeled data are mainly located within the 80% to 100% scoring group, as the offset increases above 30%, the 50%-width window starts to include mislabeled instances and this causes the rapid drop of the test accuracy.
Revised: Since 20% mislabeled data are mainly located within the 80% to 100% scoring group, as the offset increases above 30%, the 50%-width window starts to include mislabeled instances and this causes the rapid drop of the test accuracy.

Original: Thus, from the window experiments we can see that the mislabeled data has the highest CG-score and can be separable from the rest of the clean examples by the CG-score.
Revised: Thus, from the window experiments we can see that the mislabeled data has the highest CG-score and can be separable from the rest of the clean examples by the CG-score.

Original: Then, the next reasonable question is how to set the threshold on CG-score to detect the mislabeled data when the portion of mislabeled data is unknown.
Revised: Then, the next reasonable question is how to set the threshold on CG-score to detect the mislabeled data when the portion of mislabeled data is unknown.

Original: We show that the sign of the partial CGscore 2 y i ( y ⊤ − i h i ) , which is a sub-term of the CG-score in equation 6 including the label information y i , can be used in this purpose.
Revised: We show that the sign of the partial CGscore 2 y i ( y ⊤ − i h i ) , which is a sub-term of the CG-score in equation 6 including the label information y i , can be used in this purpose.

Original: As explained in Sec.
Revised: As explained in Sec.

Original: 3.3, the partial CG-score measures the gap between the average similarity of the data instance with other samples of the same class compared to that with the samples of the different classes.
Revised: 3.3, the partial CG-score measures the gap between the average similarity of the data instance with other samples of the same class compared to that with the samples of the different classes.

Original: Thus, by checking the sign of the partial CG-score, we can discriminate mislabeled samples from clean samples.
Revised: Thus, by checking the sign of the partial CG-score, we can discriminate mislabeled samples from clean samples.

Original: In Fig.
Revised: In Fig.

Original: 6 (c), we show the scatter plot of clean (blue) and mislabeled data (orange), where one can find that mislabeled examples tend to have positive partial CG-score (x-axis).
Revised: 6 (c), we show the scatter plot of clean (blue) and mislabeled data (orange), where one can find that mislabeled examples tend to have positive partial CG-score (x-axis).

Original: As shown in Table 2, we can check that for FMNIST with
Revised: As shown in Table 2, we can check that for FMNIST with

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: 20% label noise (12,000 mislabeled instances and 48,000 clean instances), 98%(=11796/12000) of mislabeled data ends up having positive partial CG-score, while only 7%(=3462/48000) of clean data has positive partial CG-score.
Revised: 20% label noise (12,000 mislabeled instances and 48,000 clean instances), 98%(=11796/12000) of mislabeled data ends up having positive partial CG-score, while only 7%(=3462/48000) of clean data has positive partial CG-score.

Original: Thus, even when the portion of label noise is unknown
Revised: Thus, even when the portion of label noise is unknown

Original: [Table]
Revised: [Table]

Original: our partial CG-score can effectively detect the mislabeled data by the sign information.
Revised: our partial CG-score can effectively detect the mislabeled data by the sign information.

Original: This tendency was less clear for CIFAR-10 dataset due to the increased dataset complexity
Revised: This tendency was less clear for CIFAR-10 dataset due to the increased dataset complexity

Original: [Table]
Revised: [Table]

Original: but still the tendency existed.
Revised: but still the tendency existed.

Original: C V ISUALIZATION OF EXAMPLES SORTED BY CG- SCORE
Revised: C V ISUALIZATION OF EXAMPLES SORTED BY CG- SCORE

Original: Analysis of MNIST and FMINST by CG-score In Fig.
Revised: Analysis of MNIST and FMINST by CG-score In Fig.

Original: 7, we show examples of MNIST (Fig.
Revised: 7, we show examples of MNIST (Fig.

Original: 7a) and FMNIST (Fig.
Revised: 7a) and FMNIST (Fig.

Original: 7b) images sorted by CG-score.
Revised: 7b) images sorted by CG-score.

Original: The top/bottom three rows show the top- /bottom-ranked examples, respectively.
Revised: The top/bottom three rows show the top- /bottom-ranked examples, respectively.

Original: We can observe that the examples with the lowest CG-score are regular examples representing each class and they look similar to each other, while the examples with the highest scores are irregular and they look different among themselves, among which we can identify (possibly) mislabeled instances, marked with red rectangles.
Revised: We can observe that the examples with the lowest CG-score are regular examples representing each class and they look similar to each other, while the examples with the highest scores are irregular and they look different among themselves, among which we can identify (possibly) mislabeled instances, marked with red rectangles.

Original: Similarly, in CIFAR-10 images (Fig. 7c), we observe that low-scoring examples (bottom two rows) are regular ones sharing similar features representing the class while high-scoring examples (top two rows) are irregular ones.
Revised: Similarly, in CIFAR-10 images (Fig. 7c), we observe that low-scoring examples (bottom two rows) are regular ones sharing similar features representing the class while high-scoring examples (top two rows) are irregular ones.

Original: Analysis of CIFAR-100 by CG-score Fig.
Revised: Analysis of CIFAR-100 by CG-score Fig.

Original: 8 shows the examples of CIFAR-100 dataset.
Revised: 8 shows the examples of CIFAR-100 dataset.

Original: Fig.
Revised: Fig.

Original: 8a shows the means and standard variations of 100 classes in CIFAR-100 dataset.
Revised: 8a shows the means and standard variations of 100 classes in CIFAR-100 dataset.

Original: Among 100 classes, ‘orange’ and ‘plain’ classes have small CG-score means and variances, while ‘bottle’ and ‘flatfish’ classes have large CG-score means and variances.
Revised: Among 100 classes, ‘orange’ and ‘plain’ classes have small CG-score means and variances, while ‘bottle’ and ‘flatfish’ classes have large CG-score means and variances.

Original: We display ten top-/bottom-ranked examples of these four classes, with the histograms of the scores in Fig.
Revised: We display ten top-/bottom-ranked examples of these four classes, with the histograms of the scores in Fig.

Original: 8b.
Revised: 8b.

Original: Analysis of ImageNet by CG-score To check the effectiveness of CG-score in analyzing more complicated dataset, we compute CG-score on the ImageNet dataset as well.
Revised: Analysis of ImageNet by CG-score To check the effectiveness of CG-score in analyzing more complicated dataset, we compute CG-score on the ImageNet dataset as well.

Original: The CG-score is computed with the sampling ratio of 1:4 by averaging the results from 10 independents runs.
Revised: The CG-score is computed with the sampling ratio of 1:4 by averaging the results from 10 independents runs.

Original: Fig.shows the distribution of CG-score and shows examples of ImageNet dataset with CG-score his-
Revised: Fig.shows the distribution of CG-score and shows examples of ImageNet dataset with CG-score his-

Original: [Figure]
Revised: [Figure]

Original: togram.
Revised: togram.

Original: ‘Jack-o-lantern’ and ‘Rapeseed’ classes have relatively smaller mean and std of CG-score and the examples from these classes share some typical attributes (color and shape).
Revised: ‘Jack-o-lantern’ and ‘Rapeseed’ classes have relatively smaller mean and std of CG-score and the examples from these classes share some typical attributes (color and shape).

Original: On the other hand, ‘Ladybug’ has high standard deviation and we can observe clear differences between the lowscoring examples and high-scoring examples.
Revised: On the other hand, ‘Ladybug’ has high standard deviation and we can observe clear differences between the lowscoring examples and high-scoring examples.

Original: Examples of ‘Spatula’, which has the biggest mean and relatively high standard deviation, do not look similar to each other but rather diverse.
Revised: Examples of ‘Spatula’, which has the biggest mean and relatively high standard deviation, do not look similar to each other but rather diverse.

Original: From the analysis, we can see that our CG-score is effective in examining high-resolution complicated dataset such as ImageNet, and the score reflects the instance-wise structural regularities, which can be used in analyzing or improving learning algorithms.
Revised: From the analysis, we can see that our CG-score is effective in examining high-resolution complicated dataset such as ImageNet, and the score reflects the instance-wise structural regularities, which can be used in analyzing or improving learning algorithms.

Original: D I MPLEMENTATION D ETAILS AND COMPUTATIONAL COST D. T RAINING DETAILS
Revised: D I MPLEMENTATION D ETAILS AND COMPUTATIONAL COST D. T RAINING DETAILS

Original: In Section 4 and 5, we evaluate our score on three public datasets, FMNIST, CIFAR-10/100, by training ResNet networks (He et al., 2016) of different depths.
Revised: In Section 4 and 5, we evaluate our score on three public datasets, FMNIST, CIFAR-10/100, by training ResNet networks (He et al., 2016) of different depths.

Original: ResNet18 is used for FMNIST and
Revised: ResNet18 is used for FMNIST and

Original: CIFAR-10 dataset and ResNet50 is used for CIFAR-100 dataset.
Revised: CIFAR-10 dataset and ResNet50 is used for CIFAR-100 dataset.

Original: Implementation of the ResNet is based on the ResNet network in torchvision (Paszke et al., 2019).
Revised: Implementation of the ResNet is based on the ResNet network in torchvision (Paszke et al., 2019).

Original: Since FMNIST and CIFAR images are smaller than ImageNet (Deng et al., 2009)
Revised: Since FMNIST and CIFAR images are smaller than ImageNet (Deng et al., 2009)

Original: images, we replace the front parts of the
Revised: images, we replace the front parts of the

Original: ResNet (convolution layer with 7x7 kernel and 2x2 stride, max pooling layer with 3x3 kernel and 2x2 stride) with a single convolution layer with 3x3 kernel and 1x1 stride for small size image.
Revised: ResNet (convolution layer with 7x7 kernel and 2x2 stride, max pooling layer with 3x3 kernel and 2x2 stride) with a single convolution layer with 3x3 kernel and 1x1 stride for small size image.

Original: The details on hyperparameters and optimization methods used in training are summarized in the Table 3.
Revised: The details on hyperparameters and optimization methods used in training are summarized in the Table 3.

Original: D. C OMPUTATIONAL TIME
Revised: D. C OMPUTATIONAL TIME

Original: Table 4 provides computational time (in seconds) to obtain CG-scores with different sampling ratio 1:1, 1:2, 1:3, and 1:4 for FMNIST and CIFAR-10/100 dataset.
Revised: Table 4 provides computational time (in seconds) to obtain CG-scores with different sampling ratio 1:1, 1:2, 1:3, and 1:4 for FMNIST and CIFAR-10/100 dataset.

Original: We also report the time to compute the baseline scores, Forgetting (Toneva et al., 2019), EL2N (Paul et al., 2021), TracIn
Revised: We also report the time to compute the baseline scores, Forgetting (Toneva et al., 2019), EL2N (Paul et al., 2021), TracIn

Original: (Pruthi et al., 2020), and CRAIG (Pruthi et al., 2020).
Revised: (Pruthi et al., 2020), and CRAIG (Pruthi et al., 2020).

Original: We do not calculate the CG-score of FMNIST dataset with sampling ratio 1:4 since it requires a inversion for 30,000x30,000 matrix, which exceeds the limit of
Revised: We do not calculate the CG-score of FMNIST dataset with sampling ratio 1:4 since it requires a inversion for 30,000x30,000 matrix, which exceeds the limit of

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Cosine annealing schedule (Loshchilov & Hutter, 2017)
Revised: Cosine annealing schedule (Loshchilov & Hutter, 2017)

Original: [Table]
Revised: [Table]

Original: Normalize
Revised: Normalize

Original: [Table]
Revised: [Table]

Original: dataset’s
Revised: dataset’s

Original: [Table]
Revised: [Table]

Original: the device memory.
Revised: the device memory.

Original: Every score including ours and baselines needs to be calculated by averaging the results of independent multiple runs.
Revised: Every score including ours and baselines needs to be calculated by averaging the results of independent multiple runs.

Original: For a fair comparison
Revised: For a fair comparison

Original: [Table]
Revised: [Table]

Original: we compare the time to get each score for a single run.
Revised: we compare the time to get each score for a single run.

Original: Cost to compute CG-score depends on the size of the Gram matrix H ∞ , since we need to calculate the inverse of H ∞ to get the CG-score and the computational complexity to
Revised: Cost to compute CG-score depends on the size of the Gram matrix H ∞ , since we need to calculate the inverse of H ∞ to get the CG-score and the computational complexity to

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: conduct an inversion of n × n matrix is O ( n 3 ) .
Revised: conduct an inversion of n × n matrix is O ( n 3 ) .

Original: Therefore, for a dataset of which each class includes a large number of instances (e
Revised: Therefore, for a dataset of which each class includes a large number of instances (e

Original: [Table]
Revised: [Table]

Original: g., FMNIST and CIFAR-10)
Revised: g., FMNIST and CIFAR-10)

Original: [Table]
Revised: [Table]

Original: taking the inverse of a Gram matrix with large sampling ratio may cause expensive computational cost
Revised: taking the inverse of a Gram matrix with large sampling ratio may cause expensive computational cost

Original: [Table]
Revised: [Table]

Original: while computing the score for a dataset of which each class includes relatively small number of instances (CIFAR-100) can be done in a short time.
Revised: while computing the score for a dataset of which each class includes relatively small number of instances (CIFAR-100) can be done in a short time.

Original: For example, calculating CG-score of CIFAR-10 dataset (5,000 data in a class) takes 1.2 hours with sampling ratio 1:4, while that for CIFAR-100(500 data in a class) takes just 1 minute.
Revised: For example, calculating CG-score of CIFAR-10 dataset (5,000 data in a class) takes 1.2 hours with sampling ratio 1:4, while that for CIFAR-100(500 data in a class) takes just 1 minute.

Original: EL2N score is time-efficient overall because it is calculate at the relatively early stage of training.
Revised: EL2N score is time-efficient overall because it is calculate at the relatively early stage of training.

Original: Forgetting and TracIn, on the other hand, take relatively longer time since they require at least one full train of the model.
Revised: Forgetting and TracIn, on the other hand, take relatively longer time since they require at least one full train of the model.

Original: In addition, we argue that our method has another computational advantage that we do not need to search networks and hyperparameters which would work well for the target dataset.
Revised: In addition, we argue that our method has another computational advantage that we do not need to search networks and hyperparameters which would work well for the target dataset.

Original: D. E XPERIMENTAL DETAILS
Revised: D. E XPERIMENTAL DETAILS

Original: Baseline scores for data valuation We use three state-of-the-art scores, C-score (Jiang et al., 2021), EL2N (Paul et al., 2021), and Forgetting score (Toneva et al., 2019) as baselines with which our CG-score is compared.
Revised: Baseline scores for data valuation We use three state-of-the-art scores, C-score (Jiang et al., 2021), EL2N (Paul et al., 2021), and Forgetting score (Toneva et al., 2019) as baselines with which our CG-score is compared.

Original: We use pre-calculated C-score for CIFAR-10 and CIFAR-100 from Jiang et al.
Revised: We use pre-calculated C-score for CIFAR-10 and CIFAR-100 from Jiang et al.

Original: and calculate EL2N and Forgetting score by averaging the score across five independent training using the full dataset.
Revised: and calculate EL2N and Forgetting score by averaging the score across five independent training using the full dataset.

Original: We obtain EL2N scores at 20th epochs of the training.
Revised: We obtain EL2N scores at 20th epochs of the training.

Original: We use the same network architectures to calculate EL2N and Forgetting score: ResNet18 for FMNIST and
Revised: We use the same network architectures to calculate EL2N and Forgetting score: ResNet18 for FMNIST and

Original: CIFAR-10 dataset and ResNet50 for CIFAR-100 dataset.
Revised: CIFAR-10 dataset and ResNet50 for CIFAR-100 dataset.

Original: Detailed definitions of the scores are as follows:
Revised: Detailed definitions of the scores are as follows:

Original: • Consistency score (C-score): C-score of each instance is calculated by estimating the prediction accuracy of the instance attained by the model trained with the full dataset except the instance.
Revised: • Consistency score (C-score): C-score of each instance is calculated by estimating the prediction accuracy of the instance attained by the model trained with the full dataset except the instance.

Original: [Equation]
Revised: [Equation]

Original: where f ( x i ; S ) is trained network using subset S , and ˆ E r denotes empirical averaging with r i.i
Revised: where f ( x i ; S ) is trained network using subset S , and ˆ E r denotes empirical averaging with r i.i

Original: d. samples of such subsets.
Revised: d. samples of such subsets.

Original: • Error L2-Norm (EL2N): The EL2N score of a training sample ( x i , y i ) is defined to be E [ ∥ f ( W ( t ) , x i ) − y i ∥ 2 ] where f ( W ( t ) , x ) is the output of the neural network for the sample ( x , y ) at the t -th step.
Revised: • Error L2-Norm (EL2N): The EL2N score of a training sample ( x i , y i ) is defined to be E [ ∥ f ( W ( t ) , x i ) − y i ∥ 2 ] where f ( W ( t ) , x ) is the output of the neural network for the sample ( x , y ) at the t -th step.

Original: • Forgetting score: Forgetting score is defined as the number of times during training (until time step T ) the decision of that sample switches from a correct one to an incorrect one: Forgetting ( x i , y i ) is defined as
Revised: • Forgetting score: Forgetting score is defined as the number of times during training (until time step T ) the decision of that sample switches from a correct one to an incorrect one: Forgetting ( x i , y i ) is defined as

Original: [Equation]
Revised: [Equation]

Original: Data pruning experiment We report the mean of the results after three independent runs.
Revised: Data pruning experiment We report the mean of the results after five independent runs.

Original: The shaded regions indicate the standard deviation.
Revised: The shaded regions indicate the standard deviation.

Original: We acquire each result by training a network using a dataset pruned by specified portions, where the training instances are ordered by each score.
Revised: We acquire each result by training a network using a dataset pruned by specified portions, where the training instances are ordered by each score.

Original: We compute the number of iterations at which all data can be used in one epoch, and use the same number of iterations in all pruning experiments for a fair comparison.
Revised: We compute the number of iterations at which all data can be used in one epoch, and use the same number of iterations in all pruning experiments for a fair comparison.

Original: When we prune the dataset, we remove the instances from each class by the same amount, so as to preserve the original proportion between classes.
Revised: When we prune the dataset, we remove the instances from each class by the same amount, so as to preserve the original proportion between classes.

Original: As will be shown in Section G, the distribution of scores is different among classes, so an imbalance problem between classes may occur if the data pruning is performed without considering the portion of classes within the dataset.
Revised: As will be shown in Section G, the distribution of scores is different among classes, so an imbalance problem between classes may occur if the data pruning is performed without considering the portion of classes within the dataset.

Original: E R OBUSTNESS OF CG- SCORE OVER MODEL VARIANTS
Revised: E R OBUSTNESS OF CG- SCORE OVER MODEL VARIANTS

Original: Our CG-score is derived based on the theoretical analysis of the generalization error bounds on overparameterized two-layer ReLU activated neural network.
Revised: Our CG-score is derived based on the theoretical analysis of the generalization error bounds on overparameterized two-layer ReLU activated neural network.

Original: To demonstrate that our CG-score is effective in more complicated networks, in the main text (Section 4.1), we used ResNets to evaluate our score for data pruning experiments.
Revised: To demonstrate that our CG-score is effective in more complicated networks, in the main text (Section 4.1), we used ResNets to evaluate our score for data pruning experiments.

Original: To further demonstrate the robustness of our score against model changes, in this section, we report the results of data pruning experiments on CIFAR10 dataset using DenseNetBC-100 Huang et al.
Revised: To further demonstrate the robustness of our score against model changes, in this section, we report the results of data pruning experiments on CIFAR10 dataset using DenseNetBC-100 Huang et al.

Original: (2017), a more complicated convolutional network, and Vision Transformer (ViT) (Dosovitskiy et al., 2021) pretrained by ImageNet dataset.
Revised: (2017), a more complicated convolutional network, and Vision Transformer (ViT) (Dosovitskiy et al., 2021) pretrained by ImageNet dataset.

Original: The implementation details of DenseNet follows that of Huang et al.
Revised: The implementation details of DenseNet follows that of Huang et al.

Original: We use the same hyperparameter and optimization method summarized in Table 3 for training DenseNet.
Revised: We use the same hyperparameter and optimization method summarized in Table 3 for training DenseNet.

Original: To fine-tune ViT, we follow the implementation details outlined in Dosovitskiy et al.
Revised: To fine-tune ViT, we follow the implementation details outlined in Dosovitskiy et al.

Original: Specifically, we download a ViT model pretrained on the ImageNet dataset using the timm module in PyTorch, and then fine-tune
Revised: Specifically, we download a ViT model pretrained on the ImageNet dataset using the timm module in PyTorch, and then fine-tune

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: the model on the CIFAR-10 dataset for 10 epochs using the ADAM optimizer.
Revised: the model on the CIFAR-10 dataset for 10 epochs using the ADAM optimizer.

Original: We set the initial learning rate to 1e-5 and do not use a learning rate scheduler or weight decay.
Revised: We set the initial learning rate to 1e-5 and do not use a learning rate scheduler or weight decay.

Original: In Figure 10, the left figure shows the test accuracy of the model trained with different subset sizes, when low-scoring (regular) examples are removed first, and the right figure shows the similar result but when high-scoring (irregular) examples are removed fist.
Revised: In Figure 10, the left figure shows the test accuracy of the model trained with different subset sizes, when low-scoring (regular) examples are removed first, and the right figure shows the similar result but when high-scoring (irregular) examples are removed fist.

Original: We report the mean of the result after two independent runs, and the shaded regions indicate the standard deviation.
Revised: We report the mean of the result after two independent runs, and the shaded regions indicate the standard deviation.

Original: The red-curve (random) is the result when randomly ordered examples are used.
Revised: The red-curve (random) is the result when randomly ordered examples are used.

Original: We can observe a similar trend as in
Revised: We can observe a similar trend as in

Original: Fig.
Revised: Fig.

Original: 2, the experimental results using ResNets, in spite of the model change.
Revised: 2, the experimental results using ResNets, in spite of the model change.

Original: Our CG-score achieves competitive performances as the other baselines.
Revised: Our CG-score achieves competitive performances as the other baselines.

Original: The reason that we observe a rapid performance degradation at a smaller training data portion in the left figure (removing low value data first) is due to the characteristics of leave-one-out method we used in the calculation of the CG-score.
Revised: The reason that we observe a rapid performance degradation at a smaller training data portion in the left figure (removing low value data first) is due to the characteristics of leave-one-out method we used in the calculation of the CG-score.

Original: Removing a typical sample from a dataset does not change the generalization error bounds much, since similar samples already exist in the dataset.
Revised: Removing a typical sample from a dataset does not change the generalization error bounds much, since similar samples already exist in the dataset.

Original: Thus, typical samples tend to have low CG-scores when the score is measured by the leave-one-out method.
Revised: Thus, typical samples tend to have low CG-scores when the score is measured by the leave-one-out method.

Original: However, when we remove 50% of instances, samples sharing the typicality can be excluded simultaneously from the dataset, which might cause severe degradation of the generalization capability of the neural network.
Revised: However, when we remove 50% of instances, samples sharing the typicality can be excluded simultaneously from the dataset, which might cause severe degradation of the generalization capability of the neural network.

Original: Thus, for a smaller training data portion, it can be better to make sure at least a small portion of typical samples is indeed included in the training.
Revised: Thus, for a smaller training data portion, it can be better to make sure at least a small portion of typical samples is indeed included in the training.

Original: Similar observations have been made in Swayamdipta et al.
Revised: Similar observations have been made in Swayamdipta et al.

Original: As shown in Sorscher et al. (2022), data scoring can also be effective in reducing the amount of data for fine-tuning pre-trained models.
Revised: As shown in Sorscher et al. (2022), data scoring can also be effective in reducing the amount of data for fine-tuning pre-trained models.

Original: Therefore, we examine whether our CG-score can be effective for fine-tuning the transformer-based models, in particular, ViT.
Revised: Therefore, we examine whether our CG-score can be effective for fine-tuning the transformer-based models, in particular, ViT.

Original: In Figure 11, the left figure shows the test accuracy of ViT fine-tuned with different subset sizes of CIFAR-10 when low-scoring (regular) examples are removed first, while the right figure shows the results when high-scoring (irregular) examples are removed first.
Revised: In Figure 11, the left figure shows the test accuracy of ViT fine-tuned with different subset sizes of CIFAR-10 when low-scoring (regular) examples are removed first, while the right figure shows the results when high-scoring (irregular) examples are removed first.

Original: We report the mean of the results after five independent runs and indicate the standard deviation with shaded regions.
Revised: We report the mean of the results after five independent runs and indicate the standard deviation with shaded regions.

Original: The red curve (random) represents the results when examples are randomly removed.
Revised: The red curve (random) represents the results when examples are randomly removed.

Original: We observe that when low-scoring examples are removed first, similar to Figure 2, the networks maintain test accuracy even when up to 90% of the dataset is removed.
Revised: We observe that when low-scoring examples are removed first, similar to Figure 2, the networks maintain test accuracy even when up to 90% of the dataset is removed.

Original: Our CG-score achieves competitive performance compared to other baselines and significantly outperforms the random baseline.
Revised: Our CG-score achieves competitive performance compared to other baselines and significantly outperforms the random baseline.

Original: F A DDITIONAL E XPERIMENTS WITH T WO M ORE B ASELINES F. O THER DIRECTIONS OF DATA VALUATION
Revised: F A DDITIONAL E XPERIMENTS WITH T WO M ORE B ASELINES F. O THER DIRECTIONS OF DATA VALUATION

Original: There are two additional branches of related works for data valuation, in addition to the methodologies described in the Section 2.
Revised: There are two additional branches of related works for data valuation, in addition to the methodologies described in the Section 2.

Original: The first branch uses the influence function.
Revised: The first branch uses the influence function.

Original: The influence function approximates the degree of change of parameters when specific data enters or leaves the training dataset, so it determines which data is valuable by calculating the effect of the data on learning.
Revised: The influence function approximates the degree of change of parameters when specific data enters or leaves the training dataset, so it determines which data is valuable by calculating the effect of the data on learning.

Original: The second branch uses coreset selection.
Revised: The second branch uses coreset selection.

Original: Coresets are weighted subsets of the data selected to resemble the model training using the full dataset.
Revised: Coresets are weighted subsets of the data selected to resemble the model training using the full dataset.

Original: Coresets may need to be updated as training progresses.
Revised: Coresets may need to be updated as training progresses.

Original: Baseline algorithms for data valuation We use representative scores in each branch, TracIn
Revised: Baseline algorithms for data valuation We use representative scores in each branch, TracIn

Original: (Pruthi et al., 2020) for influence function and CRAIG (Mirzasoleiman et al., 2020) for coreset selection, as additional baselines.
Revised: (Pruthi et al., 2020) for influence function and CRAIG (Mirzasoleiman et al., 2020) for coreset selection, as additional baselines.

Original: Detailed definitions of the scores are described below:
Revised: Detailed definitions of the scores are described below:

Original: • TracIn: TracIn
Revised: • TracIn: TracIn

Original: CheckPoint (TracInCP) value between two data points is defined as the weighted sum of dot products of the loss gradients calculated at the two data points.
Revised: CheckPoint (TracInCP) value between two data points is defined as the weighted sum of dot products of the loss gradients calculated at the two data points.

Original: The gradients are obtained from the k checkpoints { t 1 , . .
Revised: The gradients are obtained from the k checkpoints { t 1 , . .

Original: t k } of the model in the middle of training and the weight η t i is the learning rate at each checkpoint t i :
Revised: t k } of the model in the middle of training and the weight η t i is the learning rate at each checkpoint t i :

Original: [Equation]
Revised: [Equation]

Original: where l ( w t i , z ) is loss function at t i -th step with model parameter w t i .
Revised: where l ( w t i , z ) is loss function at t i -th step with model parameter w t i .

Original: • CRAIG:
Revised: • CRAIG:

Original: CoResets for Accelerating Incremental Gradient descent (CRAIG) is an algorithm that solves the optimization problem, which finds a subset that preserves the gradient of the total loss:
Revised: CoResets for Accelerating Incremental Gradient descent (CRAIG) is an algorithm that solves the optimization problem, which finds a subset that preserves the gradient of the total loss:

Original: [Equation]
Revised: [Equation]

Original: where f i ( w ) =
Revised: where f i ( w ) =

Original: l ( w, ( x i , y i )) is loss for the data ( x i , y i ) with model parameter w , V is the full dataset and S is the coresets with size r , and p i be the softmax output for data ( x i , y i ) .
Revised: l ( w, ( x i , y i )) is loss for the data ( x i , y i ) with model parameter w , V is the full dataset and S is the coresets with size r , and p i be the softmax output for data ( x i , y i ) .

Original: In CRAIG, the gradient f i ( w ) is approximated by p i − y
Revised: In CRAIG, the gradient f i ( w ) is approximated by p i − y

Original: i when cross entropy loss is used with soft-max at the last layer.
Revised: i when cross entropy loss is used with soft-max at the last layer.

Original: F. D ATA P RUNING EXPERIMENTS
Revised: F. D ATA P RUNING EXPERIMENTS

Original: In this section, we conduct data pruning experiments, similar to Section 4.1.
Revised: In this section, we conduct data pruning experiments, similar to Section 4.1.

Original: We conduct the experiments using the above two additional baselines, TracIn and CRAIG, separately from the experiments of the main text, since the two algorithms require additional assumptions/resources that have not been used for the baselines considered in the main text: TracIn requires a validation set to calculate the data values; CRAIG does not select a fixed subset of data to be used throughout the training, but keeps updating the subset of the data (coresets) to be used for training every 10 epochs.
Revised: We conduct the experiments using the above two additional baselines, TracIn and CRAIG, separately from the experiments of the main text, since the two algorithms require additional assumptions/resources that have not been used for the baselines considered in the main text: TracIn requires a validation set to calculate the data values; CRAIG does not select a fixed subset of data to be used throughout the training, but keeps updating the subset of the data (coresets) to be used for training every 10 epochs.

Original: [Figure]
Revised: [Figure]

Original: Experimental details As described in the Pruthi et al.
Revised: Experimental details As described in the Pruthi et al.

Original: (2020), we calculate the TracIn score by using the gradients of the parameters of the network’s last layer.
Revised: (2020), we calculate the TracIn score by using the gradients of the parameters of the network’s last layer.

Original: The check points are set at every 20 epochs, starting from the end of the first 20th epoch.
Revised: The check points are set at every 20 epochs, starting from the end of the first 20th epoch.

Original: We use 5 checkpoints for FMNIST andcheckpoints for CIFAR-10 and CIFAR-100.
Revised: We use 5 checkpoints for FMNIST andcheckpoints for CIFAR-10 and CIFAR-100.

Original: We create a validation set composed of 1,000 samples by taking a part of the test dataset, and calculate TracIn score with this validation set.
Revised: We create a validation set composed of 1,000 samples by taking a part of the test dataset, and calculate TracIn score with this validation set.

Original: As TracIn score is defined between two data points ( z, z ′ ) as in equation 11, we set the score of each training sample z by averaging TracIn scores
Revised: As TracIn score is defined between two data points ( z, z ′ ) as in equation 11, we set the score of each training sample z by averaging TracIn scores

Original: TracInCP( z, z ′ ) over all samples z ′ in the validation set.
Revised: TracInCP( z, z ′ ) over all samples z ′ in the validation set.

Original: In CRAIG, the subset selection is performed every 10 epochs.
Revised: In CRAIG, the subset selection is performed every 10 epochs.

Original: We only test CRAIG in pruning lowvalued data but not in the reverse order (pruning high-valued data), since CRAIG extracts coresets to be used with per element weights for preserving the gradient of total loss but does not give what are the high-valued (equal weight) samples.
Revised: We only test CRAIG in pruning lowvalued data but not in the reverse order (pruning high-valued data), since CRAIG extracts coresets to be used with per element weights for preserving the gradient of total loss but does not give what are the high-valued (equal weight) samples.

Original: TracIn score and CRAIG are calculated at the networks same as those used in the experiment in
Revised: TracIn score and CRAIG are calculated at the networks same as those used in the experiment in

Original: Section 4.1: ResNet18 for FMNIST and CIFAR-10, and ResNet50 for CIFAR-100 dataset.
Revised: Section 4.1: ResNet18 for FMNIST and CIFAR-10, and ResNet50 for CIFAR-100 dataset.

Original: The other experimental details are the same as Table 3 in Appendix §D.
Revised: The other experimental details are the same as Table 3 in Appendix §D.

Original: Experimental results Similar to data pruning experiment in Section 4.1, in Figure 12, the first row shows the test accuracy of the model trained with different subset sizes, when low-scoring (regular) examples are removed first, and the second row shows the similar result but when high-scoring (irregular) examples are removed first.
Revised: Experimental results Similar to data pruning experiment in Section 4.1, in Figure 12, the first row shows the test accuracy of the model trained with different subset sizes, when low-scoring (regular) examples are removed first, and the second row shows the similar result but when high-scoring (irregular) examples are removed first.

Original: We report the mean of the results after three independent runs, and the shaded regions indicate the standard deviation.
Revised: We report the mean of the results after five independent runs, and the shaded regions indicate the standard deviation.

Original: The red-curve (random) is the result when randomly ordered examples are used.
Revised: The red-curve (random) is the result when randomly ordered examples are used.

Original: When pruning low-scoring data first, it is preferable to maintain the accuracy up to a large removing portion (a small training set); when pruning highscoring data first, the rapid drop of performance is preferable since it means that the score can detect high-value samples, necessary for generalization of the model.
Revised: When pruning low-scoring data first, it is preferable to maintain the accuracy up to a large removing portion (a small training set); when pruning highscoring data first, the rapid drop of performance is preferable since it means that the score can detect high-value samples, necessary for generalization of the model.

Original: Our CG-score can achieve better performances than the Tracin score and competitive performances compared to the CRAIG algorithm.
Revised: Our CG-score can achieve better performances than the Tracin score and competitive performances compared to the CRAIG algorithm.

Original: Since CRAIG updates the coresets over the training, it can choose different semantics of the data suitable for each phase of the training, which results in better performance.
Revised: Since CRAIG updates the coresets over the training, it can choose different semantics of the data suitable for each phase of the training, which results in better performance.

Original: This result may imply the effectiveness of the scheduled batch selection, e.g., curriculum learning, in training neural networks.
Revised: This result may imply the effectiveness of the scheduled batch selection, e.g., curriculum learning, in training neural networks.

Original: We inspect that the performance of TracIn may heavily depend on the size of the validation set and also the possible domain discrepancy between training and test datasets.
Revised: We inspect that the performance of TracIn may heavily depend on the size of the validation set and also the possible domain discrepancy between training and test datasets.

Original: In our test, there is no domain discrepancy, but if there exists a domain shift between the test dataset and the training dataset, TracIn may perform better than other methods with the help of validation set.
Revised: In our test, there is no domain discrepancy, but if there exists a domain shift between the test dataset and the training dataset, TracIn may perform better than other methods with the help of validation set.

Original: F. D ETECTING L ABEL N OISE
Revised: F. D ETECTING L ABEL N OISE

Original: We also compared the performance of our CG-score in detecting mislabeled data with that of TracIn.
Revised: We also compared the performance of our CG-score in detecting mislabeled data with that of TracIn.

Original: As suggested in Pruthi et al.
Revised: As suggested in Pruthi et al.

Original: (2020), we use ‘self-influence’ of each training example, i.e., the influence of a training point on its own loss during the training process, TracInCP ( z, z ) , to identify mislabeled data.
Revised: (2020), we use ‘self-influence’ of each training example, i.e., the influence of a training point on its own loss during the training process, TracInCP ( z, z ) , to identify mislabeled data.

Original: In Pruthi et al.
Revised: In Pruthi et al.

Original: (2020), it was shown that mislabeled examples tend to have higher
Revised: (2020), it was shown that mislabeled examples tend to have higher

Original: TracIn values, and thus TracIn values can be effectively used in identifying mislabeled examples.
Revised: TracIn values, and thus TracIn values can be effectively used in identifying mislabeled examples.

Original: In Fig. 13, we show the comparison of our method with TracIn in identifying mislabeled examples in FMNIST and CIFAR-100 datasets, respectively, each of which includes 20% label noise.
Revised: In Fig. 13, we show the comparison of our method with TracIn in identifying mislabeled examples in FMNIST and CIFAR-100 datasets, respectively, each of which includes 20% label noise.

Original: TracIn achieves better performance in CIFAR-100, but ours outperformed TracIn in FMNIST.
Revised: TracIn achieves better performance in CIFAR-100, but ours outperformed TracIn in FMNIST.

Original: Since TracIn measures the ‘self-influence’ of each training example over the training, starting from 20th epoch, for relatively simpler dataset such as FMNIST, some mislabeled instances could have already been memorized at the network, which makes them not detectable by the TracIn values.
Revised: Since TracIn measures the ‘self-influence’ of each training example over the training, starting from 20th epoch, for relatively simpler dataset such as FMNIST, some mislabeled instances could have already been memorized at the network, which makes them not detectable by the TracIn values.

Original: On the other hand, our method better detects mislabeled data for easier datasets as discussed in Sec. 4.2.
Revised: On the other hand, our method better detects mislabeled data for easier datasets as discussed in Sec. 4.2.

Original: Thus, we can conclude that depending on data complexity, the outperforming method can be changing.
Revised: Thus, we can conclude that depending on data complexity, the outperforming method can be changing.

Original: G S CORE DISTRIBUTION FOR PUBLIC DATASETS
Revised: G S CORE DISTRIBUTION FOR PUBLIC DATASETS

Original: Score distributions
Revised: Score distributions

Original: In Fig.
Revised: In Fig.

Original: 14, we compare the CG-score distributions of three public datasets,
Revised: 14, we compare the CG-score distributions of three public datasets,

Original: FMNIST, CIFAR-10 and CIFAR-100 before and after we artificially corrupt 20% of instances with random label noise.
Revised: FMNIST, CIFAR-10 and CIFAR-100 before and after we artificially corrupt 20% of instances with random label noise.

Original: The top row shows the distributions before the corruption, and the bottom row shows the distributions after the corruption, where the orange color displays the distribution of the
Revised: The top row shows the distributions before the corruption, and the bottom row shows the distributions after the corruption, where the orange color displays the distribution of the

Original: CG-score for 20% instances with label noise and the blue displays
Revised: CG-score for 20% instances with label noise and the blue displays

Original: that for 80% instances with clean label.
Revised: that for 80% instances with clean label.

Original: The gap between scores of clean data and noisy data, which enables us to detect the noise by the CG-score, is larger for relatively simpler dataset, FMINST, than those for CIFAR-10/100.
Revised: The gap between scores of clean data and noisy data, which enables us to detect the noise by the CG-score, is larger for relatively simpler dataset, FMINST, than those for CIFAR-10/100.

Original: Detecting mislabeled instances at high noise rate We conduct additional experiments to check the detectability of mislabeled instances by our CG-score, when the noise ratio increases to 50% and even to 80% for FMNIST and CIFAR-10 dataset.
Revised: Detecting mislabeled instances at high noise rate We conduct additional experiments to check the detectability of mislabeled instances by our CG-score, when the noise ratio increases to 50% and even to 80% for FMNIST and CIFAR-10 dataset.

Original: In the main text, we considered a mild noise ratio of 20% and reported the result in Fig.
Revised: In the main text, we considered a mild noise ratio of 20% and reported the result in Fig.

Original: The results for higher noise cases are shown in Fig.
Revised: The results for higher noise cases are shown in Fig.

Original: In the CIFAR-10 dataset, when the noise ratio is 80%, each class includes 1,000 correctly labeled images and 4,000 mislabeled images, composed of 445 samples coming from each of the other nine classes.
Revised: In the CIFAR-10 dataset, when the noise ratio is 80%, each class includes 1,000 correctly labeled images and 4,000 mislabeled images, composed of 445 samples coming from each of the other nine classes.

Original: Even for such an extremely noisy case, our CG-score can effectively detect the mislabeled data, since our score can discover samples that have a relatively lower correlation to the majority of the samples of the same class as analyzed in Sec. 3.3.
Revised: Even for such an extremely noisy case, our CG-score can effectively detect the mislabeled data, since our score can discover samples that have a relatively lower correlation to the majority of the samples of the same class as analyzed in Sec. 3.3.

Original: H F EATURE - SPACE CG- SCORE
Revised: H F EATURE - SPACE CG- SCORE

Original: Our original CG-score can be calculated by data without any trained network.
Revised: Our original CG-score can be calculated by data without any trained network.

Original: In this section we check the validity of a new CG-score, which is defined in terms of feature of data, and compare its characteristic with that of the original data-centric CG-score.
Revised: In this section we check the validity of a new CG-score, which is defined in terms of feature of data, and compare its characteristic with that of the original data-centric CG-score.

Original: We train ResNet18 using FMNIST and CIFAR-10 dataset and ResNet50 using CIFAR-100 dataset to obtain each data’s feature at 10th epoch, where the feature is defined at the output of the penultimate layer (512 and 2048 dimensions for ResNet18 and ResNet50, respectively).
Revised: We train ResNet18 using FMNIST and CIFAR-10 dataset and ResNet50 using CIFAR-100 dataset to obtain each data’s feature at 10th epoch, where the feature is defined at the output of the penultimate layer (512 and 2048 dimensions for ResNet18 and ResNet50, respectively).

Original: Then we calculate the feature-space CG scores, which are calculated by the feature instead of the data itself.
Revised: Then we calculate the feature-space CG scores, which are calculated by the feature instead of the data itself.

Original: Unlike CG-score, calculating feature-space
Revised: Unlike CG-score, calculating feature-space

Original: CG-score requires training of a model for a few epochs, which implies that the score becomes dependent to the model and we need additional computational efforts.
Revised: CG-score requires training of a model for a few epochs, which implies that the score becomes dependent to the model and we need additional computational efforts.

Original: However, feature CG-score achieves superior performances at data valuation as the results of data pruning experiment (Fig. 16) and noise detection experiment (Fig. 17) show even though the features were extracted at early stage of the training (10th epoch).
Revised: However, feature CG-score achieves superior performances at data valuation as the results of data pruning experiment (Fig. 16) and noise detection experiment (Fig. 17) show even though the features were extracted at early stage of the training (10th epoch).

Original: Furthermore, we compare how the Spearman rank correlation between the feature-space CG-scores and other scores, including the original CG-score and the previous training-based scores such as
Revised: Furthermore, we compare how the Spearman rank correlation between the feature-space CG-scores and other scores, including the original CG-score and the previous training-based scores such as

Original: [Figure]
Revised: [Figure]

Original: C-score, Forgetting score, and EL2N, change over the epochs at which the feature space CG-scores are calculated.
Revised: C-score, Forgetting score, and EL2N, change over the epochs at which the feature space CG-scores are calculated.

Original: Figure 18 reports the result.
Revised: Figure 18 reports the result.

Original: In Figure 18, we can first observe that the feature space
Revised: In Figure 18, we can first observe that the feature space

Original: CG-score and the original CG-score has the highest correlation at the very beginning of the training (epoch 1) but as the training progresses, the correlation decreases.
Revised: CG-score and the original CG-score has the highest correlation at the very beginning of the training (epoch 1) but as the training progresses, the correlation decreases.

Original: This trend can be explained by the fact that the feature space CG-score computes the value of data in the learned embedding space, while our score computes the value of the original data without embedding it into a latent space.
Revised: This trend can be explained by the fact that the feature space CG-score computes the value of data in the learned embedding space, while our score computes the value of the original data without embedding it into a latent space.

Original: As training of a model progresses, the embedded data may incur bias in the data valuation, depending on a particular model or training algorithm and thus the correlation between the feature space CG-score and the original data-centric CG-score may decrease.
Revised: As training of a model progresses, the embedded data may incur bias in the data valuation, depending on a particular model or training algorithm and thus the correlation between the feature space CG-score and the original data-centric CG-score may decrease.

Original: On the other hand, correlation between feature space CG-score and other training-based scores (EL2N, Forgetting, and C-score) increases during the initial stage and then decreases gradually.
Revised: On the other hand, correlation between feature space CG-score and other training-based scores (EL2N, Forgetting, and C-score) increases during the initial stage and then decreases gradually.

Original: More specifically, for both EL2N and Forgetting score, which are computed at the same network as that of the feature space CG-score, the correlation increases as the epoch increases and then slightly drops and becomes saturated at a certain value.
Revised: More specifically, for both EL2N and Forgetting score, which are computed at the same network as that of the feature space CG-score, the correlation increases as the epoch increases and then slightly drops and becomes saturated at a certain value.

Original: The C-score, which was calculated in a different CNN network, shows a slightly different tendency compared to EL2N or forgetting score, and attains its peak at an earlier epoch, epoch 5.
Revised: The C-score, which was calculated in a different CNN network, shows a slightly different tendency compared to EL2N or forgetting score, and attains its peak at an earlier epoch, epoch 5.

Original: This result implies that the feature-space CG score includes meaningful information for data valuation, correlated with other training-based scores for overall epochs.
Revised: This result implies that the feature-space CG score includes meaningful information for data valuation, correlated with other training-based scores for overall epochs.

Original: However, the feature space CG-score may incur some bias in data valuation as the training progresses.
Revised: However, the feature space CG-score may incur some bias in data valuation as the training progresses.

Original: Understanding the effectiveness of the feature-space CG score in diverse applications can be an interesting future research direction.
Revised: Understanding the effectiveness of the feature-space CG score in diverse applications can be an interesting future research direction.

Original: I D ETAILS OF KERNEL VELOCITY
Revised: I D ETAILS OF KERNEL VELOCITY

Original: We calculate the kernel velocity of 10 groups of instances, where each group is composed ofsamples.
Revised: We calculate the kernel velocity of 10 groups of instances, where each group is composed ofsamples.

Original: The following is how we construct each group: Sort CIFAR-10 examples in ascending order by the CG-score, divide the examples into 10 groups, and select 125 consecutive samples from the beginning of each group.
Revised: The following is how we construct each group: Sort CIFAR-10 examples in ascending order by the CG-score, divide the examples into 10 groups, and select 125 consecutive samples from the beginning of each group.

Original: We calculate the NTK kernel velocity as described in Paul et al.
Revised: We calculate the NTK kernel velocity as described in Paul et al.

Original: Let C be the number of classes.
Revised: Let C be the number of classes.

Original: Let f ( c ) t ( x i ) be the c -th logit value for input x i at the t -epoch.
Revised: Let f ( c ) t ( x i ) be the c -th logit value for input x i at the t -epoch.

Original: When W ( t ) is the parameters of the model, the c -th logit gradient at input x i is ψ ( c ) t ( x i )
Revised: When W ( t ) is the parameters of the model, the c -th logit gradient at input x i is ψ ( c ) t ( x i )

Original: =
Revised: =

Original: ∇ W ( t ) f
Revised: ∇ W ( t ) f

Original: ( c ) t ( x i ) ∈ R N .
Revised: ( c ) t ( x i ) ∈ R N .

Original: Then, the datadependent NTK submatrix of a group of m samples S :
Revised: Then, the datadependent NTK submatrix of a group of m samples S :

Original: = { x a 1 , . .
Revised: = { x a 1 , . .

Original: , x a m } is defined as K t ( S ) =
Revised: , x a m } is defined as K t ( S ) =

Original: Ψ t ( S )
Revised: Ψ t ( S )

Original: Ψ t ( S ) ⊤ , where Ψ t ( S ) ∈
Revised: Ψ t ( S ) ⊤ , where Ψ t ( S ) ∈

Original: R mC × N is constructed by placing { ψ ( c ) t ( x a i ) }
Revised: R mC × N is constructed by placing { ψ ( c ) t ( x a i ) }

Original: c ∈ [ C ] ,i ∈
Revised: c ∈ [ C ] ,i ∈

Original: [ m ] in rows of Ψ t ( S ) .
Revised: [ m ] in rows of Ψ t ( S ) .

Original: The kernel velocity is defined as
Revised: The kernel velocity is defined as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Lastly, we provide some explanations of what the kernel velocity measures if we define it for finite-width ReLU activated 2-layer neural network.
Revised: Lastly, we provide some explanations of what the kernel velocity measures if we define it for finite-width ReLU activated 2-layer neural network.

Original: Remind that our analysis uses the
Revised: Remind that our analysis uses the

Original: Gram matrix H ∞ defined in equation 1, which is derived for an overparameterized ReLU activated 2-layer neural network.
Revised: Gram matrix H ∞ defined in equation 1, which is derived for an overparameterized ReLU activated 2-layer neural network.

Original: We can generalize the definition of the Gram matrix assuming a finite-width network, similar to K t ( S ) , as follows.
Revised: We can generalize the definition of the Gram matrix assuming a finite-width network, similar to K t ( S ) , as follows.

Original: The output of the network is f W , a ( x ) = 1 √ m (cid:80) mr =1 a r σ ( w ⊤ r x ) , and the gradient with respect to W = ( w 1 , . . .
Revised: The output of the network is f W , a ( x ) = 1 √ m (cid:80) mr =1 a r σ ( w ⊤ r x ) , and the gradient with respect to W = ( w 1 , . . .

Original: , w m ) ∈
Revised: , w m ) ∈

Original: R d × m is ∇ W f
Revised: R d × m is ∇ W f

Original: ( x i )
Revised: ( x i )

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: ⊤
Revised: ⊤

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: ≥ 0 } , . .
Revised: ≥ 0 } , . .

Original: [Equation]
Revised: [Equation]

Original: { w ⊤ m x i ≥ 0 } ] .
Revised: { w ⊤ m x i ≥ 0 } ] .

Original: Denoting the network parameters at the t -th step as W ( t ) , we can define the Gram matrix H t as ( H t ) ij = ∇ W ( t ) f ( x i ) ∇ W ( t
Revised: Denoting the network parameters at the t -th step as W ( t ) , we can define the Gram matrix H t as ( H t ) ij = ∇ W ( t ) f ( x i ) ∇ W ( t

Original: [Equation]
Revised: [Equation]

Original: f
Revised: f

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: ⊤
Revised: ⊤

Original: [Equation]
Revised: [Equation]

Original: x ⊤ i x
Revised: x ⊤ i x

Original: j 1 m (cid:80) mr =1 1 { w ⊤ r x i ≥ 0 , w ⊤ r x j ≥ 0 } , which is proportion to the number of neurons in the hidden layer activated both for x i and x j at the epoch.
Revised: j 1 m (cid:80) mr =1 1 { w ⊤ r x i ≥ 0 , w ⊤ r x j ≥ 0 } , which is proportion to the number of neurons in the hidden layer activated both for x i and x j at the epoch.

Original: We can define the kernel velocity similar to equation 13 by calculating H t for a subset of data, and replacing K t ( S ) by H t ( S ) .
Revised: We can define the kernel velocity similar to equation 13 by calculating H t for a subset of data, and replacing K t ( S ) by H t ( S ) .

Original: For this case, the high kernel velocity implies that H t differs much from H t +1 , i.e., the portion of neurons activated for pairs of instances changes rapidly during training.
Revised: For this case, the high kernel velocity implies that H t differs much from H t +1 , i.e., the portion of neurons activated for pairs of instances changes rapidly during training.

Original: [Table]
Revised: [Table]

Original: J C ORRELATIONS BETWEEN CG- SCORE AND P ARTIAL CG- SCORES
Revised: J C ORRELATIONS BETWEEN CG- SCORE AND P ARTIAL CG- SCORES

Original: From the CG-score, which is expressed as d − 1 i ( y ⊤− i h i ) 2 + 

Revised: From the CG-score, which is expressed as d − 1 i ( y ⊤− i h i ) 2 +

Original: y i
Revised: y i

Original: ( y ⊤− i h i )
Revised: ( y ⊤− i h i )

Original: + y 2 i
Revised: + y 2 i

Original: d i at the equation 6, we can define three partial CG-scores, d − 1 i ( y ⊤−
Revised: d i at the equation 6, we can define three partial CG-scores, d − 1 i ( y ⊤−

Original: i h i ) 2 , 2 y i ( y ⊤− i h i ) , and y 2 i d i , respectively.
Revised: i h i ) 2 , 2 y i ( y ⊤− i h i ) , and y 2 i d i , respectively.

Original: In this section
Revised: In this section

Original: [Table]
Revised: [Table]

Original: we analyze which term dominates the order of CG-scores for two public datasets, CIFAR10/100, by analyzing the correlation between the CG-score and the three partial CG-scores.
Revised: we analyze which term dominates the order of CG-scores for two public datasets, CIFAR10/100, by analyzing the correlation between the CG-score and the three partial CG-scores.

Original: We also examine whether the correlations vary when we change the size of the Gram matrix H ∞ , whose dimension changes according to the level of sub-sampling, explained in Sec.
Revised: We also examine whether the correlations vary when we change the size of the Gram matrix H ∞ , whose dimension changes according to the level of sub-sampling, explained in Sec.

Original: A.2.
Revised: A.2.

Original: We get H ∞ with different subsampling ratios, 1:1, 1:2, 1:3, and 1:4, and then calculate the CG-score and partial CGscores, defined by the three partial terms.
Revised: We get H ∞ with different subsampling ratios, 1:1, 1:2, 1:3, and 1:4, and then calculate the CG-score and partial CGscores, defined by the three partial terms.

Original: Table 5 shows the Spearman rank correlations between the CG-scores and the partial CG-scores.
Revised: Table 5 shows the Spearman rank correlations between the CG-scores and the partial CG-scores.

Original: We can see that 2 y i ( y ⊤− i h i ) , which was utilized at noise detection experiments, is the partial score having the largest correlation with the CG-score across all subsampling ratios.
Revised: We can see that 2 y i ( y ⊤− i h i ) , which was utilized at noise detection experiments, is the partial score having the largest correlation with the CG-score across all subsampling ratios.

Original: On the other hand, correlations to other partial-scores are relatively low, and the correlations change much as the subsampling ratio changes.
Revised: On the other hand, correlations to other partial-scores are relatively low, and the correlations change much as the subsampling ratio changes.

Original: K S CHUR COMPLEMENT
Revised: K S CHUR COMPLEMENT

Original: Calculating CG-score (equation 2) of n data instances requires the inversion of n × n matrix, which may cause expensive computational cost for a large n .
Revised: Calculating CG-score (equation 2) of n data instances requires the inversion of n × n matrix, which may cause expensive computational cost for a large n .

Original: Therefore, we provided computationally efficient way to obtain the CG-score by using Schur complement equation 4.
Revised: Therefore, we provided computationally efficient way to obtain the CG-score by using Schur complement equation 4.

Original: Here we provide some details to derive the inverse of a sub-block matrix using Schur complement.
Revised: Here we provide some details to derive the inverse of a sub-block matrix using Schur complement.

Original: By Schur complement, we have
Revised: By Schur complement, we have

Original: [Equation]
Revised: [Equation]

Original: Remind that H ∞ and ( H ∞ ) − 1 are denoted by
Revised: Remind that H ∞ and ( H ∞ ) − 1 are denoted by

Original: [Equation]
Revised: [Equation]

Original: where H ∞ n − 1 , ( H ∞ ) − 1 n − 1 ∈ R ( n − 1) × ( n − 1) , g i , h i ∈ R n − 1 and c i , d i ∈ R , i.e.,
Revised: where H ∞ n − 1 , ( H ∞ ) − 1 n − 1 ∈ R ( n − 1) × ( n − 1) , g i , h i ∈ R n − 1 and c i , d i ∈ R , i.e.,

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: By substituting A , B , C , and D in equation 14 with ( H ∞ ) − 1 n − 1 , h i , h ⊤ i , and d i respectively, we obtain that
Revised: By substituting A , B , C , and D in equation 14 with ( H ∞ ) − 1 n − 1 , h i , h ⊤ i , and d i respectively, we obtain that

Original: [Equation]
Revised: [Equation]

Original: Thus, ( H ∞ n − 1 ) − 1 =
Revised: Thus, ( H ∞ n − 1 ) − 1 =

Original: ( H ∞ ) − 1 n − 1 − h i
Revised: ( H ∞ ) − 1 n − 1 − h i

Original: h ⊤ i /d i .
Revised: h ⊤ i /d i .