Sample Relationships through the Lens of Learning Dynamics with Label Information
Abstract
Although much research has been done on proposing new models or loss functions to improve the generalisation of artificial neural networks (ANNs), less attention has been directed to the data, which is also an important factor for training ANNs.
In this work, we start from approximating the interaction between two samples, i.e. how learning one sample would modify the model’s prediction on the other sample.
Through analysing the terms involved in weight updates in supervised learning, we find that the signs of labels influence the interactions between samples.
Therefore, we propose the labelled pseudo Neural Tangent Kernel (lpNTK) which takes label information into consideration when measuring the interactions between samples.
We first prove that lpNTK would asymptotically converge to the well-known empirical Neural Tangent Kernel in terms of the Frobenius norm under certain assumptions.
Secondly, we illustrate how lpNTK helps to understand learning phenomena identified in previous work, specifically the learning difficulty of samples and forgetting events during learning.
Moreover, we also show that lpNTK can help to improve the generalisation performance of ANNs in image classification tasks, compared with the original whole training sets.
Introduction
In the past decade, artificial neural networks (ANNs) have achieved great successes with the help of large models and very large datasets (Silver et al., 2016; Krizhevsky et al., 2017; Vaswani et al., 2017).
There are usually three components involved in training an ANN: 1) the model, i e. the ANN itself; 2) a loss function; and 3) a dataset, usually labelled for supervised learning.
Previous work has shown that generalisation performance can be boosted by i) changing the model architecture, e g.
ResNet (He et al., 2016) and ViT (Dosovitskiy et al., 2020); or ii) introducing new loss functions, e.g. focal loss (Lin et al., 2017) and regularisation (Goodfellow et al., 2016).
Researchers have also explored methods that allow models to choose data, e.g. active learning (Settles, 2012), to improve the generalisation performance and reduce laborious data annotation.
In this work, we focus on selecting data in order to achieve greater generalisation performance.
Although the samples in datasets are usually assumed to be independent and identically distributed (i.i.d, Goodfellow et al., 2016), ANNs do not learn them independently: the parameter update on one labelled sample will influence the predictions for many other samples.
This dependency is a double-edged sword.
On the plus side, the dependency between seen and unseen samples makes it possible for ANNs to output reliable predictions on held-out data (Koh & Liang, 2017).
However, dependencies between samples can cause catastrophic forgetting (Kirkpatrick et al., 2017), i.e. updates on a sample appearing later might erase the correct predictions for previous samples.
While researchers have proposed statistics to describe such forgetting events (e.g. Toneva et al., 2018; Paul et al., 2021), it is an open question to identify the causes of such events.
We argue that it is necessary to investigate the relationships between samples in order to understand how data selection affects the learning and further improve generalisation performance through manipulating data.
We first decompose the learning dynamics of ANN models in classification tasks, and show that the interactions between training samples can be well-captured by combining the empirical neural tangent kernel (eNTK, Jacot et al., 2018) with label information.
Inspired by the pseudo NTK (pNTK) proposed by Mohamadi & Sutherland (2022) which can represent the relationship between two input samples using one scalar, we propose the labelled pseudo NTK (lpNTK) to measure the interactions between samples, i.e. how the update on one sample influences the predictions on others.
As shown in Section 2.3, our lpNTK can be viewed as a linear kernel on a feature space representing each sample as a vector derived from the lpNTK.
Since the angles between vectors can be obtuse/right/acute, we point out that there are three types of relationships between two samples: interchangeable, unrelated, and contradictory.
Through experiments, we verify that two concepts from previous work can be connected to and explained by the above three types of relationships: 1) the forgetting events during ANN learning found by Toneva et al.
(2018); 2) the learning difficulty of data discussed by Paul et al.
Inspired by the discussion about the connections between learning difficulty and generalisation from Sorscher et al.
(2022), we show that the generalisation performance of ANNs can be improved by removing part of the samples in the largest cluster obtained through farthest point clustering (FPC) with lpNTK as the similarity metric, compared with the whole original dataset.
In summary, we make three contributions: 1) we introduce a new kernel, lpNTK, which can take label information into account for ANNs to measure the interaction between sample during learning; 2) we provide a unified view to explain the learning difficulty of samples and forgetting events using the three types of relationships defined under lpNTK; 3) we show that generalisation performance in classification problems can be improved by carefully removing part of the samples that have similar lpNTK feature representations.
lpNTK: Interactions between Samples via First-Order Taylor Approximation
In this section, we first show the first-order Taylor approximation to the interactions between two samples in a typical supervised learning task, classification.
We then show how labels modify those interactions, which inspires our lpNTK.
At the end of this section, we formalise our lpNTK and mathematically prove that it asymptotically converges to eNTK (Jacot et al., 2018).
2.1 Derivation on Supervised Learning for Classification Problems
Suppose in a K -way classification problem, the dataset D consists of N labelled samples, i.e. D = {{ x 1 , y 1 } , . .
, { x N , y N }} .
Our neural network model is f ( x ; w ) ≜ q ( x ) = softmax( z ( x ; w )) where w ∈ R d is the vectorised parameters, z ∈ R K , and q ∈ ∆ K .
To update parameters, we use the cross-entropy loss function, i.e. L ( y, q ( x ))
=
− log q y ( x ) where q k represents the k -th element of the prediction vector q , and the back-propagation algorithm (Rumelhart et al., 1986).
At time t , suppose we take a step of stochastic gradient descent (SGD) with learning rate η on a sample x u , we show that this will modify the prediction on x o ̸ = x u as:
[Equation]
1) where A t ( x o ) ≜ ∇ z q t ( x o ) | z t , K t ( x o , x u ) ≜ ∇ w z t ( x o ) | w t · ( ∇ w z t ( x u ) | w t ) T , and p tar ( x u ) is a one-hot vector where only the y u -th element is 1 .
The full derivation is given in Appendix D.
K t ( x o , x u ) in Equation 1 is a dot product between the gradient matrix on x u ( ∇ w z t ( x u ) | w t ) and x o ( ∇ w z t ( x o )
| w t ).
Thus, an entry K tij > 0 means that the two gradient vectors, ∇ w ; i, · z t ( x u ) and ∇ w ; j, · z t ( x o ) , are pointing in similar directions, thus learning one leads to better prediction on the other; when K tij < 0 the two gradients are pointing in opposite directions, thus learning one leads to worse prediction on the other.
Regarding the case where K tij ≈ 0 , the two gradient vectors are approximately orthogonal, and learning one of them has almost no influence on the other.
Since
K t ( x o , x u ) is the only term that involves both x u and x o at the same time, we argue that the matrix itself or a transformation of it can be seen as a similarity measure between x u and x o , which naturally follows from the first-order Taylor approximation.
K t ( x o , x u ) is also known as the eNTK (Jacot et al., 2018) in vector output form.
Though K t ( x o , x u ) is a natural similarity measure between x o and x u , there are two problems.
First, it depends on both the samples and parameters w t , thus becomes variable over training since w t varies over t .
Therefore, to make this similarity metric more accurate, it is reasonable to prefer the parameters that perform the best on the validation set, i.e. parameters having the best generalisation performance.
Without loss of generality, we denote such parameters as w in the following, and the matrix as K .
The second problem is that we can compute K ( x , x ′ ) for every pair of samples ( x , x ′ ) , but need to convert them to scalars κ ( x , x ′ ) in order to use them as a conventional scalar-valued kernel.
This also makes it possible to directly compare two similarity matrices, e.g.
K and K ′ .
It is intuitive to use L p,q or Frobenius norm of matrices to convert K ( x , x ′ ) to κ ( x , x ′ ) .
However, through experiments, we found that the off-diagonal entries in K t ( x o , x u ) are usually non-zero and there are many negative entries, thus neither L p,q -norm nor F-norm is appropriate.
Instead, the findings from Mohamadi & Sutherland (2022) suggests that the sum of all elements in
K ( x , x ′ ) divided by K serves as a good candidate for this proxy scalar value, and they refer to this quantity as pNTK.
We follow their idea, but show that the signs of elements in K should be considered in order to have higher test accuracy in practice.
We provide empirical evidence that this is the case in Appendix B.
2.2 Information From Labels
Beyond the similarity matrix K , there are two other terms in Equation 1, the softmax derivative term A and the prediction error term p tar ( x u ) − q t ( x u ) .
As shown in Equation 6 from Appendix D, the signs in A are constant across all samples.
Thus, it is not necessary to take A into consideration when we measure the similarity between two specific samples.
Let’s now have a look at the prediction error term.
In common practice, p tar ( x ) is usually a one-hot vector in which only the y -th element is 1 and all others are 0 .
Since q = softmax( z ) ∈ ∆ K , we can do the following analysis of the signs of elements in the prediction error term:
[Equation]
The above analysis shows how learning x u would modify the predictions on x o , i.e. ∆ x u q ( x o ) .
Conversely, we can also approximate ∆ x o q ( x u ) in the same way.
In this case, it is easy to see that
K ( x u , x o ) =
K ( x o , x u ) T and all elements of p tar ( x o )
− q ( x o ) are negative except the y o -th digit, since K ( x u , x o ) · ( p tar ( x o ) − q ( x o ))
=
K ( x o , x u ) T · ( p tar ( x o )
− q ( x o )) .
Therefore, for a pair of samples x o and x u , their labels would change the signs of the rows and columns of the similarity matrix K ( x o , x u ) , respectively.
Note that we do not use the whole p tar ( x ) − q ( x ) but only the signs of the entries.
As illustrated in Section 3.1 below, if a large fraction of samples in the dataset share similar gradients, all their prediction errors would become tiny after a relatively short period of training.
In this case, taking the magnitude of p tar ( x ) − q ( x ) into consideration leads to a conclusion that those samples are less similar to each other because the model can accurately fit them.
Therefore, we argue that the magnitudes of prediction errors are misleading for measuring the similarity between samples.
2.3 Labelled Pseudo Neural Tangent Kernel (lpNTK)
Following the analysis in Section 2.2, we introduce the following lpNTK:
[Equation]
where ⊙ denotes that element-wise product between two matrices, and s ( · ) is defined in Equation 2.
The first line of Equation 3 is to emphasise the difference between our lpNTK and pNTK, i.e. we sum up the element-wise products between 1 K s ( y u ) · s ( y o ) T and K .
Equivalently, the second line shows that our kernel can also be thought of as a linear kernel on the feature space where a sample x along with its label y is represented as 1 √ K s ( y ) T ∇ w z ( x ) .
The second expression also shows the novelty of our lpNTK, i.e. the label information is taken into the feature representations of samples.
We first verify that the lpNTK kernel matrix approximates the corresponding eNTK.
Following Mohamadi & Sutherland (2022), we also study the asymptotic convergence of lpNTK in terms of the
Frobenius norm.
Theorem 1.
(Informal).
Suppose the last layer of our neural network z ( x ; w ) is a linear layer of width w .
Let κ (( x , y ) , ( x ′ , y ′ ))
be the corresponding lpNTK, and
˜ K
( x , x ′ ) be the corresponding eNTK.
When the final linear layer of z ( x ; w ) is properly initialised, with high probability, the following holds:
[Equation]
(4) where I K is a K -by K identity matrix.
A formal statement and proof of Theorem 1 are given in
Appendix E.
In the following sections, we first show the connection between our lpNTK and learning phenomena discussed by Toneva et al.
(2018) and Paul et al.
Then, we empirically show that our lpNTK improves the generalisation performance in a typical supervised learning task, image classification, compared to the pNTK.
Rethinking Easy/Hard Samples following lpNTK
Since each labelled sample ( x , y ) corresponds to a vector 1 √ K s ( y ) T ∇ w z ( x ) ∈ R d under lpNTK, and the angle between any two such vectors could be obtuse, right, or acute, it is straightforward to see the following three types of relationships between two labelled samples:
• interchangeable samples (where the angle between samples is acute) update the parameters of the model in similar directions, thus updates on one sample make the predictions on other interchangeable samples more accurate; • unrelated samples which update parameters in (almost) orthogonal directions, thus updates on one sample would not modify the predictions on other samples; • contradictory samples which update parameters in opposite directions, thus updates on one sample would make the predictions on the contradictory samples worse.
More details about the above three types of relationships can be found in Appendix F.
3.1 A Unified View for Easy/Hard Samples and Forgetting Events
Suppose at time-step t of training, we use a batch of data B t = { ( x ti , y ti ) } | B t | i =1 to update the parameters.
Let’s consider how this update would influence the predictions of samples in B t − 1 and B t +1 .
As shown by Equation 1, for a sample x ′ in either B t − 1 or B t +1 , the change of its prediction is approximately η · A t ( x ′ ) · (cid:80) x t ∈ B t K t ( x ′ , x t ) · ( p tar ( x t ) − q t ( x t )) .
Note that the sum on B t is still linear, as all elements in Equation 1 are linear.
Following the above analysis, it is straightforward to see that if a sample in B t − 1 is contradictory to most of the samples in B t , it would most likely be forgotten , as the updates from B t modify its prediction in the “wrong” direction.
Furthermore, if the probability of sampling data contradictory to ( x , y ) is high across the training, then ( x , y ) would become hard to learn as the updates from this sample are likely to be cancelled out by the updates from the contradictory samples until the prediction error on the contradictory samples are low.
On the other hand, if a sample in B t +1 is interchangeable with most of the samples in B t , it would be an easy sample, as the updates from B t already modified its prediction in the “correct” direction.
Moreover, if there is a large group of samples that are interchangeable with each other, they will be easier to learn since the updates from all of them modify their predictions in similar directions.
Therefore, we argue that the easy/hard samples as well as the forgetting events are closely related to the interactions between the lpNTK feature representations, i.e. 1 √ K s ( y ) T ∇ w z ( x ) , of the samples in a dataset.
The number of interchangeable/contradictory samples can be thought as the “learning weights” of their corresponding lpNTK feature representations.
An example of and further details about the above analysis are given in Appendix G.1.
3.2 Experiments to Verify the Unified View
The first experiment we run is to verify that the learning difficulty of samples is mainly due to the interactions between them, rather than inherent properties of individual samples.
The setup and results are further illustrated in Appendix A.1.
The second experiment we run is to check whether the forgetting events can be accurately predicted with lpNTK. The setup and results are demonstrated in Appendix A.2. As mentioned in Section 3.1, if there is a large group of samples that are interchangeable with each other, all of them should become easy to learn. To verify this point, we first group the labelled samples with the farthest point clustering (FPC) algorithm proposed by Gonzalez (1985).
We then rank the samples in MNIST and CIFAR-10 by their learning difficulty defined as above. Among the Top- 10% × N easiest samples in MNIST and CIFAR-10, 72% and 80% of them are in the largest cluster. Further details can be found in Appendix A.3.
3.3 Use Case: Supervised Learning for Image Classification
Inspired by the finding from Sorscher et al.
(2022) that selective data pruning can lead to substantially better performance than random selection 2 , we also explore if pruning samples following the clustering results under lpNTK could help to improve generalisation performance in a typical supervised learning task, image classification.
In the experiments below, we use the same models and benchmarks as in Appendix A.2. Following the clustering analysis from Appendix A.3, suppose we compose the lpNTK feature representations of samples from a cluster into a single vector.
The size of the cluster can then be thought as the weight for that vector during learning. Given the cluster size distribution is heavily long-tailed, the learning would be biased towards the samples in the head cluster.
However, we also know that the samples in the head cluster are more interchangeable with each other, i.e. they update parameters to similar directions. Therefore, we ask the following two questions:
do we really need all those interchangeable samples for good generalisation? can we improve the generalisation performance by removing the bias in the data towards the numerous interchangeable samples?
We answer the above two questions in Appendix B, and show that the lpNTK can help to improve the generalisation performance by removing the redundant (Appendix B.2) and poisoning (Appendix B.3) samples in the original training sets, following which we introduce some related works in Appendix C.  Conclusion
In this work, we start from approximating the interactions between labelled samples, and propose lpNTK to measure the similarity between samples. As illustrated in Section 3, it is then straightforward to see that samples in a dataset might be interchangeable, unrelated, or contradictory, following the lpNTK feature representation.
We show that the learning difficulty of samples as well as forgetting events can be well explained under a unified view in Appendix A, following these three types of relationships. At last, we show in Section 3.3 and Appendix B that the generalisation performance can be improved on both MNIST and CIFAR-10 through pruning out part of the interchangeable samples in the largest cluster obtained via farthest point clustering algorithm (Gonzalez, 1985) and lpNTK. Overall, we argue that our work provides a novel perspective to understand and analyse the learning of DL models through the lens of learning dynamics, label information, and sample relationships.
A.2 Experiment 2: Predict Forgetting Events with lpNTK
The second experiment we run is to check whether the forgetting events can be accurately predicted with lpNTK.
Following Toneva et al.
(2018), we say a sample is forgotten at epoch t
+ 1 if the model can correctly predict it at epoch t but then makes the wrong prediction at epoch t + 1 .
Given that the time for computing lpNTK (( x , y ) , ( x ′ , y ′ )) is quadratic in the number of parameters, we choose benchmarks that are not too large and models that are not too over-parameterised: we train a LeNet-5 model (LeCun et al., 1989) on the MNIST dataset (LeCun et al., 1998a) and a ResNetmodel (He et al., 2016) on the CIFAR-10 dataset (Krizhevsky et al., 2009), both using learning rate 5 × 10 − 4 ; we use a small learning rate to guarantee the accuracy of first-order Taylor approximation.
In order to sample forgetting events, we track the predictions on 10 batches of 128 samples overiterations in the middle of the training process, and observed more than 5 , 000 forgetting events.
The results of predicting these forgetting events by our lpNTK are given in Table 1.
As can be seen from Table 1, our lpNTK can accurately predict forgetting events during the training of models on both benchmark datasets.
The prediction metrics on CIFAR-10 are higher, perhaps because the number of parameters of ResNet-18 is greater than LeNet-5, thus the change of each [Table] parameter is smaller during a single iteration, which further leads to a more
A.3 Experiment 3: Farthest Point Clustering with lpNTK to Understand Sample Learning
As mentioned in Section 3.1, if there is a large group of samples that are interchangeable with each other, all of them should become easy to learn.
To verify this point, we first need to group the labelled samples.
Considering that the number of sample are usually tens of thousands and the time for computing a K ( x , x ′ ) is quadratic to the number of parameters, we choose the farthest point clustering (FPC) algorithm proposed by Gonzalez (1985), since: 1) in theory, it can be sped up to
O ( N log M ) where M is the number of centroids (Feder & Greene, 1988); 2) we cannot arbitrarily interpolate between two gradients 3 .
As for the number of centroids in FPC, we use 10% × N here, as our purpose is to show that the distributions over the sizes of clusters on both MNIST and CIFARare heavily long-tailed , i.e. there are a small number of large clusters and many small clusters.
The details on how we do FPC with lpNTK are given in Appendix G.2.
Through experiments, we find that most of the samples are actually in the head cluster, i.e. the largest cluster.
There are a further 5 − 20 clusters containing more than one sample, then the remaining clusters all contain just a single sample.
This indicates that the samples in the head cluster are more interchangeable with each other than with the samples in the tail clusters.
Therefore, the samples in the head clusters should be easier to learn than those in the tail clusters.
To verify the above analysis, we first rank the samples in MNIST and CIFAR-10 by their learning difficulty defined as above.
Among the Top- 10% × N easiest samples in MNIST and CIFAR-10, 72% and 80% of them are in the largest cluster.
Note that the learning difficulty is averaged across random seeds, whereas the clusters are obtained with the seed that leads to the optimal validation performance.
Our results suggest that samples in the largest clusters are indeed easier to learn than those in smaller clusters.
B Use Case: Supervised Learning for Image Classification
Inspired by the finding from Sorscher et al.
(2022) that selective data pruning can lead to substantially better performance than random selection 4 , we also explore if pruning samples following the clustering results under lpNTK could help to improve generalisation performance in a typical supervised learning task, image classification.
In the experiments below, we use the same models and benchmarks as in Section A.2.
B.1 Improving Generalisation with lpNTK
Following the clustering analysis from Section A.3, suppose we compose the lpNTK feature representations of samples from a cluster into a single vector.
The size of the cluster can then be thought as the weight for that vector during learning.
Given the cluster size distribution is heavily long-tailed, the learning would be biased towards the samples in the head cluster.
However, we also know that the samples in the head cluster are more interchangeable with each other, i.e. they update parameters to similar directions.
Therefore, we ask the following two questions:
do we really need all those interchangeable samples for good generalisation?
can we improve the generalisation performance by removing the bias in the data towards the numerous interchangeable samples?
B.2 Experimental Results on Removing Redundant Samples
To answer the first question, we define the redundant samples under lpNTK.
Formally, for a sample x , if there exists another sample x ′ ̸
= x such that lpNTK (( x , y ) , ( x ′ , y ′ )) > lpNTK
(( x , y ) , ( x ′ , y ′ )) , then x is considered as a redundant sample.
5 To verify that redundant samples identified in this way are indeed redundant and note required for accurate generalisation, we removed them from both MNIST and CIFAR-10, and compared the test accuracy of the models trained with or without those redundant samples.
We ran this experiment with 10 different random seeds.
Note that we did not fine-tune any hyperparameters, to make sure that the comparison between different instances of datasets are fair.
Thus, the average test accuracy we obtained is not as high as reported in some other works, e.g. Guo et al.
The test accuracy over training epochs on the whole training sets of MNIST and CIFAR-10 along with the de-redundant versions are shown in Figure 2.
As shown in both plots, removing the redundant samples under lpNTK leads to almost the same generalisation performance on both
MNIST and CIFAR-10 (converged test accuracy obtained with the whole training sets is not significantly higher than accuracy obtained using the de-redundant version using lpNTK: on MNIST, t (19) = 0 .
293 , p = 0 .
774 ; on CIFAR, t (19) = 1 . 562 , p = 0 . 153 ), whereas the de-redundant versions obtained by pNTK leads to significantly worse generalisation performance (on MNIST, t (19)
= 13 .
718 , p ≪ 0 .
01 ; on CIFAR, t (19) = 26 .
252 , p ≪ 0 . 01 ).
[Figure] Overall, our results suggest that it is not necessary to train on multiple redundant samples (as identified using lpNTK) in order to have a good generalisation performance; the fact that identifying redundant samples using pNTK does lead to a reduction in performance shows that taking the label information into account in evaluating the relationships between samples (as we do in lpNTK) indeed leads to better similarity measure than pNTK in practice.
B.3 Experimental Results on Debiasing Datasets
To answer our second question in Section B.1, we find several relevant clues from existing work.
Feldman (2020) demonstrated that memorising the noisy or anomalous labels of the long-tailed samples is necessary in order to achieve close-to-optimal generalisation performance.
Paul et al. (2021) and Sorscher et al.
(2022) pointed out that keeping the hardest samples can lead to better generalisation performance than keeping only the easy samples.
Particularly, the results in Figure 1 of Paul et al.
(2021) show that removing 10% or 20% of the easy samples can lead to better test accuracy than training on the full training set.
As discussed in Appendix A, both the long-tail clusters and hard samples can be connected to the three types of relationships under our lpNTK.
Specifically, samples in the head cluster are more interchangeable with each other thus easier to learn, while the samples in the tail clusters are more unrelated or even contradictory to samples in the head cluster and thus harder to learn.
Inspired by the connections between clustering results under lpNTK and the above works, we: 1) remove the redundant samples from MNIST and CIFAR-10 under lpNTK; 2) cluster the remaining samples with FPC and lpNTK; 3) randomly prune 10% of the samples in the head cluster; 4) compare the test accuracy of models trained with the original datasets and the pruned versions.
This sequence of steps involves removing roughly 20% of the total dataset.
The results are given in Figure 3.
[Figure]
As we can see in Figure 3, the pruned datasets actually lead to higher test accuracy than the original training sets of MNIST and CIFAR-10: a t -test comparing converged test accuracy after training on the full vs pruned data sets shows significantly higher performance in the pruned sets (MNIST: t (19) =
− 3 .
205 , p = 0 .
CIFAR: t (19) = − 3 .
996 , p = 0 . 003 ).
This suggest that it is possible to improve the generalisation performance on test sets by removing some interchangeable samples in the head cluster from FPC on lpNTK.
B.4 Discussion
Although the results illustrated above answer the two questions in Appendix B.1, our experiments involve decisions on several additional parameters that merit further discussion.
In Appendix B.3, we removed around 20% of the total dataset, and obtained better test accuracy than after training on the full training set.
However, the choice to prune this fraction is purely heuristic, and there is no theoretical guarantee for this choice.
More interestingly, since lpNTK can measure any pair of annotated samples, it is also possible to use lpNTK to measure the similarity between the training and test data.
So, it is intuitive to see that the interactions between training and test samples would also influence the fraction of pruned samples.
As this is related to wider research topics, e.g. transfer learning, we leave this point to future work.
We also needed to specify the number of clusters as a hyperparameter for FPC.
It is possible to use clustering algorithms like hierarchical clustering (HC) to get rid of this hyperparameter.
However, we found that on both MNIST and CIFAR-10, HC generates heavily imbalanced binary trees, i.e. the samples are organised almost as linked lists, which makes it hard to find the boundaries between the obtained clusters.
The problem of FPC, on the other hand, is that the optimal choice of M is also related to both the training and test datasets and needs to be further studied.
Since we focus on the effectiveness of lpNTK in this work, we leave the study about clustering on lpNTK to the future.
C Related Work
In this section, we discuss the connections between our work and other work from various disciplines. [Transmission Bottleneck in Iterated Learning]
This work originated as an exploration of the possible forms of transmission bottleneck in the iterated learning framework (Kirby & Hurford, 2002; Smith et al., 2003b; Kirby et al., 2014) on deep learning models.
In existing works (e.g.
Guo et al., 2020; Lu et al., 2020; Ren et al., 2020; Vani et al., 2021; Rajeswar et al., 2022), the transmission bottleneck is usually implemented as a limit on the number of pre-training epochs for the new generation of agents.
However, as shown in human experiments by Smith et al.
(2003a), the transmission bottleneck can also be a subset of the whole training set, which works well for human agents.
So, inspired by the original work on humans, we explore the possibility of using subsetsampling as a form of transmission bottleneck.
As shown in Appendix B.3, it is indeed possible to have higher generalisation performance through limiting the size of subsets.
Thus, we argue that this work sheds the light on a new form of transmission bottleneck of iterated learning for DL agents. [Neural Tangent Kernel]
The NTK was first proposed by Jacot et al.
and was derived on fully connected neural networks, or equivalently multi-layer perceptrons.
Lee et al.
Yang (2019, 2020); Yang & Littwin (2021) then extend NTK to most of the popular neural network architectures.
Beyond the theoretical insights, NTK has also been applied to various kinds of DL tasks, e.g. 1) Park et al.
and
Chen et al.
(2021) in neural architecture search; 2) Zhou et al. (2021) in meta learning; and 3) Holzm¨uller et al.
(2022) and Wang et al.
(2021) in active learning.
NTK has also been applied in dataset distillation (Nguyen et al., 2020, 2021), which is closely related to our work, thus we discuss it separately in the next subsection.
Mohamadi & Sutherland (2022) explore how to convert the matrix-valued eNTK for classification problems to scalar values.
They show that the sum of the eNTK matrix would asymptotically converge to the eNTK, which inspires lpNTK.
However, we emphasise the information from labels, and focus more on practical use cases of lpNTK.
We also connect lpNTK with the practical learning phenomena like learning difficulty of samples and forgetting events. [Coreset Selection and Dataset Distillation]
As discussed in Appendix B, we improve the generalisation performance through removing part of the samples in the training sets.
This technique is also a common practice in coreset selections (CS, Guo et al., 2022), although the aim of CS is usually to select a subset of training samples that can obtain generalisation performance similar to the whole set.
On the other hand, we aim to show that it is possible to improve the generalisation performance via removing samples.
In the meantime, as shown in Equation 3, our lpNTK is defined on the gradients of the outputs w.r.t the parameters.
Work on both coreset selection (Killamsetty et al., 2021) and dataset distillation (Zhao et al., 2021) have also explored the information from gradients for either selecting or synthesising samples.
However, these works aimed to match the gradients of the loss function w.r.t the parameter on the selected/synthesised samples with the whole dataset , whereas we focus on the gradients between samples in the training set.
D Full Derivation of lpNTK on Classification Problem in Supervised Learning
Following the problem formulated in Section 2, let’s begin with a Taylor expansion of ∆ x u q ( x o ) ,
[Equation]
To evaluate the leading term, we can plug in the definition of SGD and apply the chain rule repeatedly: [Equation] x u ) · [Equation]
p
[Equation]
x
[Equation]
For the higher-order terms, using as above that
[Equation]
and note that ∥ p tar ( x u ) − q t ( x u ) ∥ is bounded as p tar , q ∈ ∆ K , we have that
O (cid:0) ∥
w t +1 − w t ∥ 2 (cid:1) = O (cid:0) η 2 ∥ ( ∇ w z ( x u ) | w t ) T ∥ 2op ∥ p tar ( x u ) − q t ( x u ) ∥ 2 (cid:1) = O (cid:0) η 2 ∥∇ w z ( x u ) ∥ 2op (cid:1) .
The first term in the above decomposition is a symmetric positive semi-definite (PSD) matrix shown below, [Equation] 
[Equation]
The second term in the above decomposition, K t ( x o , x u ) , is the outer product of the Jacobian on x o and x u .
It is intuitive to see that the elements in this matrix would be large if the two Jacobians are both large and in similar direction.
Since K t ( x o , x u ) is a matrix rather than a scalar, so it is not a conventional scalar-valued kernel.
This matrix is also usually known as matrix-valued NTK.
The third term in the above decomposition is the prediction error on x u , p tar ( x u ) − q t ( x u ) .
During the training, for most of the samples, this term would gradually approach 0 , suppose the model can fit well on the dataset.
However, it is varying over the epochs, so it is not a constant like the sign vector s ( y u ) .
Thus, we introduce only the signs from this vector into our lpNTK, as illustrated in
Section 2.
E Further Results on the Approximation of lpNTK
In this section, we’ll provide the proof of Theorem 1 in Section 2.3.
This proof relies on subexponential random variables, and we recommend the Appendix B.1 of Mohamadi & Sutherland (2022) and notes written by Xiong (2022) for the necessary background knowledge.
Following the notation from Section 2.1, we assume our ANN model z ( · ) has L layers, and the vectorised parameters of a layer l ∈ { 1 , 2 , . . .
, L } is denoted as w l whose width is w l .
In the meantime, we assume the last layer of z ( · ) is a dense layer parameterised by a matrix W L , and the input to the last layer is g ( x ; w { 1 , ˙ ,L − 1 } )
∈ R w L , thus z ( x ) =
W L · g ( x ) .
Lastly, let’s assume the parameters of the last layer are initialised by the LeCun initialisation proposed by LeCun et al. (1998b), i.e. W Li,j ∼ N (0 , 1 w L ) .
Equivalently, W Li,j can be seen as W Li,j = 1 √ w L u where u ∼
N (0 , 1) .
It has been shown by Lee et al.
and Yang (2020) that the eNTK can be formulated as the sum of gradients of the outputs w.r.t the vectorised parameters w l .
That said, the eNTK of an ANN z can be denoted as
[Equation]
Since the last layer of z ( · ) is a dense layer, i.e. z ( x ) =
W L · g ( x ) , we know that ∇ w l z ( x ) = ∇ g z ( x ) · ∇ w l g ( x ) =
w L · ∇ w l g ( x ) .
Thus, the above Equation 7 can be re-written as
[Equation]
where I K is a K -by- K identity matrix.
Then, by expanding Equation 8, we can have that the elements in K z ( x , x ′ ) are
[Equation]
where 1 ( · ) is the indicator function.
From Equation 8, like the pNTK proposed by Mohamadi & Sutherland (2022), it can be seen that lpNTK simply calculates a weighted summation of the element in eNTK.
Thus, lpNTK can also be seen as appending another dense layer parameterised by 1 √ K s ( y ) to the ANN z ( · ) where s ( y ) is defined in Equation 2.
Similar to how we get Equation 9, we can get that the value of lpNTK between x and x ′ is
[Equation]
Now, let’s define the difference matrix between lpNTK and eNTK as D ( x , x ′ )
[Equation]
L K z ( x , x ′ ) , then the entries in D ( x , x ′ ) are
[Equation]
Since all parameters are i.i.d, we can further simplify the above equation as:
[Equation]
It is then straightforward to see that we can use inequalities for sub-exponential random variables to bound the above result, since: 1) u i ∼ N (0 , 1) ; 2)
u 2 i ∼ X 2 (1) and is sub-exponential [-withparameters ν 2 = 4 and β = 
-]with parameters
ν 2 = 4 and β = 4 ( SE (4 , 4) ); 3) u i u j ∈ SE (2 , √ 2) ; 4) linear combination of subexponential random variables is still sub-exponential; 5) for a random variable X ∈ SE ( ν 2 , β ) the following inequality holds with probability at least 1 − δ :
[Equation]
However, note that not all elements in Equation 12 are independent of each other in the case i = j .
Suppose that α = ||
K g ( x , x ′ )
|| ∞ , we can then rewrite the case as:
[Equation]
In the following, we will bound the above four D ii ( x , x ′ ) terms separately.
Let’s start with
D 1 ii ( x , x ′ ) .
Considering that a linear combination of a sequence of sub-exponential random variables { U i } where U i ∈ SE ( ν 2 i , β i ) is sub [Equation]
, i
[Equation] e. [Equation]
| [Equation]
| [Equation] we can have the following derivation:
[Equation]
With w L ≫ K > 2 and the property (5) from the paragraph below Equation 12, we can claim:
[Equation]
with probability at least 1 − δ 4 .
Regarding the remaining terms, the entries are products of two independent Gaussian r.v.
u · u ′ , thus the weighted sum of them are also sub-exponential.
Further, we can bound all of them with the property we used in Equation 15.
So, similar to the above analysis of D 1 ii ( x , x ′ ) , we can claim that:
[Equation]
with probability at least 1 − δ 4 .
Regarding the case i ̸
= j of Equation 12, the result is identical to | D 3 ii ( x , x ′ ) | shown in the above
Equation 17.
Therefore, considering Equation 15 to 18 all together, we can loosen the above bounds, and have the following inequalities:
[Equation]
with probability at least 1 − δ .
Therefore, to sum up from the above, since the Frobenius norm of D ( x , x ′ ) is (cid:113)(cid:80) i,j D i,j ( x , x ′ ) , we can get the following bound for the F-norm of the different matrix: [Equation] (cid:114)
[Equation]
log 8 Kδ , √ 2 log 8 Kδ (cid:33)(cid:33) ≥ 1 − δ.
F Further Details about the Sample Relationships under lpNTK
[Figure]
F.1 Interchangeable
In this case, the two gradient vectors construct a small acute angle, and projections on each other is longer than a fraction t of the other vector, formally the conditions can be written as
[Equation]
where t ∈ ( 12 , 1] .
This situation is visualised in Figure 4a.
F.2 Unrelated
In this case, the two gradient vectors are almost inter-orthogonal, formally the conditions can be written as
[Equation]
≈ 0 .
This situation is visualised in Figure 4b.
F.3 contradictory
In this case, the two gradient vectors are almost opposite.
Like the interchangeable case, the conditions can be formally written as:
[Equation]
where t ∈ [ − 1 , − 12 ) .
This situation is visualised in Figure 4c.
However, the possible causes of this case is more complicated than the above two.
• y u
= y
o , i.e. the two samples are from the same class.
Suppose x u is interchangeable with many other samples in the same class and x o is not, this may indicate that x o is an outlier of the class.
• y u ̸
= y
o , i.e. the samples are from different classes.
Since the annotated data in these two classes can not be learnt by the model at the same time, our guess is that the data is not completely compatible with the learning model, or the learning model may have a wrong bias for the task.
G Further Details about Relationships between lpNTK and Easy/Hard Samples G.1 An Extreme Example of Easy/Hard Samples Due to Interactions
In this section, we illustrate an extreme case of interchangeable sample and another extreme case of contradictory samples.
To simplify the task, we assume K = 2 , i.e. a binary classification problem, and there are only two annotated samples in the training set, ( x u , y u ) and ( x u , y o ) .
Note that the inputs are identical, and we discuss y o = y u and y o ̸
= y u separately in the following.
y o =
y u
In this case, the two annotated sample are actually identical, thus learning any one of them is equivalent to learning both.
It is then straightforward to see that these two (identical) samples are interchangeable.
Though they are identical, the one that appears later would be counted as an easy sample, since the updates on the other one automatically lead to good predictions for both.
y
o ̸
= y u
In this case, since there are only two categories, learning one of them would cancel the update due to the other one.
Suppose the model is being trained in an online learning fashion, and the two samples appear in turn.
Then, learning ( x u , y u ) drags the model’s prediction q ( x u ) towards y u .
On the other hand, learning ( x u , y o ) drags the model’s prediction q ( x u ) towards the opposite direction y o .
In this way, the two samples are completely contradictory to each other, and they both are difficult samples, since the learning is unlikely to converge
G.2 Farthest Point Clustering with lpNTK
In this section, we illustrate more details about how we cluster the training samples based on farthest point clustering (FPC) and lpNTK.
Suppose that there are N samples in the training set, { ( x i , y i ) } Ni =1 , and the lpNTK between a pair of samples is denoted as κ (( x , y ) , ( x ′ , y ′ )) .
The FPC algorithm proposed by Gonzalez (1985) running with our lpNTK is demonstrated in the following
Algorithm 1.
Input: Dataset { ( x i , y i ) } Ni =1 , The number of centroids M
Output: Cluter centroids { c 1 , . . . , c M } , LIst of samples indices in all clusters: { L 1 , . . .
, L M }
N c ← Number established clusters; c 1 ← arg max i κ (( x i , y i ) , ( x i , y i ))
Centroid first
L 1 ← { i ̸ = c 1 } Ni =
▷ the first clsuter; while N c <
M do d ← ∞ , c new ← −▷ 1.
Find the controid for the new cluster; for j ← 1 to N c do if min
j ′ ∈
L j κ (( x c j , y c J ) , ( x j ′ , y j ′ ))
<
d then d ← min j ′ ∈
L j κ (( x c j , y c J ) , ( x j ′ , y j ′ )) ;
c new ← arg min j ′ ∈
L j κ (( x c j , y c J ) , ( x j ′ , y j ′ )) ; end end
N c ← N c + 1 , c N c ← c new , L N c ← ∅ ; ▷ 2.
Move samples that are closer to the new centroid to the new cluster; for j ← 1 to N c − 1 do for i ∈
L j do if κ (( x i , y i ) , ( x c j , y c j )) <
κ (( x i , y i ) , ( x c Nc , y c Nc ))
then L j ← L j \ { i } ; L
N c ← L N c ∪ { i } ; end end end end
H Redundant Samples and Poisoning Samples H.1 Redundant Samples
Suppose there are two samples x 1 and x 2 from the same class, and their corresponding gradient vectors in the lpNTK feature space are drawn as two solid arrows in Figure 5.
The x 1 can be further decomposed into two vectors, x ′ 1 which has the same direction as x 2 and x ′′ 1 whose direction is perpendicular to x 2 , and both are drawn as dashed arrows in Figure 5.
Since x ′ 1 is larger than x 2 , this means that a component of x 1 is larger than x 2 on the direction of x 2 .
Further, this shows that learning x 1 is equivalent to learning x 2 (times some constant) and a hypothetical samples that is orthogonal to x 2 .
In this case, the sample x 2 is said to be redundant .
H.2 Poisoning Samples
Data poisoning refers to a class of training-time adversarial attacks Wang et al.
(2022): the adversary is given the ability to insert and remove a bounded number of training samples to manipulate the [Figure] behaviour (e.g. predictions for some target samples) of models trained using the resulted, poisoned data.
Therefore, we use the name of “poisoning samples” here rather than “outliers” or “anomalies”to emphasise that those samples lead to worse generalisation.
More formally, for a given set of samples T , if the averaged test accuracy of the models trained on X ⊂ T is statistically significantly greater than the models trained on T , the samples in T \ X are defined as poisoning samples.
To better illustrate our findings about poisoning samples, we first illustrate a hypothetical FPC results, as shown in Figure 6.
There are 5 resulted clusters, and the number of samples in each cluster is 7 , 2 , 1 , 1 , 1 .
Back to the three types of relationships defined in Section 3, samples in the same cluster (e.g. the blue dots) would be more likely to be interchangeable with each other, and samples from different clusters (e.g. the yellow, green, and red dots) would be more likely to be unrelated or contradictory.
Although those samples in the tail did not seem to contribute much to the major group, the interesting thing is that they are not always poisoning.
That is, the poisoning data are inconsistent over the varying sizes of the pruned dataset.
Let’s denote the original training set as T , and the pruned dataset as X .
When a large fraction of data can be kept in the pruned dataset, e.g. | X | / | T | > 85% , removing some samples that are more interchangeable with the others, or equivalently from the head cluster, would benefit the test accuracy, as demonstrated in Appendix B.3.
That is, in such case, the poisoning samples are the samples in the largest cluster, e.g. the blue dots in Figure 6.
However, where only a small fraction of data can be kept, e.g. 5% or 1% , we find that the poisoning samples are the samples from the tail clusters, e.g. the yellow/green/red dot in Figure 6.
This finding matches with the conclusions from Sorscher et al.
To verify it, we first randomly sample 5% from the MNIST training set, and put them all together in a training set T 1 .
In the meantime, we sample the same amount of data only from the largest cluster obtained on MNIST in Appendix A.3, and put them all together in another training set T 2 .
We then compare the generalisation performance of LeNet models trained with T 1 and T 2 , and we try two fractions, 1% and 5% .
The results are shown in the following Figure 7.
As shown in Figure 7, in both 1% and 5% cases, keeping only the samples from the largest cluster obtained via FPC with lpNTK lead to higher test accuracy than sampling from the whole MNIST training set.
This verifies our argument that the most interchangeable samples are more important for generalisation when only a few training samples can be used for learning.
Lastly, we want to give a language learning example to further illustrate the abover argument.
Let’s consider the plural nouns.
We just need to add -s to the end to make regular nouns plural, and we have a few other rules for the regular nouns ending with e.g. -s , -sh , or -ch .
Whereas, there is no specific rule for irregular nouns like child or mouse .
So, the irregular nouns are more likely [Figure] to be considered as samples in the tail clusters, as they are more dissimilar to each other.
For an
English learning beginner, the most important samples might be the regular nouns, as they can cover most cases of changing nouns to plurals.
However, as one continues to learn, it becomes necessary to remember those irregular patterns in order to advance further.
Therefore, suppose the irregular patterns are defined as in the tail clusters, they may not always be poisoning for generalisation performance.