Paper: lpNTK: Better Generalisation with Less Data via Sample Interaction During Learning (ICLR-2024)

Reviewer comment (Theory):
Should Eq (1) contain the sign operator wrapped around the last factor? In Section 2.2, it is stated that the paper does not use the whole error term but only the sign as shown in Eq (2). I am also not sure I was able to follow the explanation as to why the magnitudes should not matter. Could the magnitude be useful for ranking?

Author response:
We thank the reviewer for pointing out this. Actually, Eq (1) should not contain the sign operator, as it comes from the definition of gradient descent and Taylor expansion. For the magnitude part, it is true that this value is useful for ranking at the early stage of training, which is what most of the learning dynamic-based methods use, like forgetting score, C-score, etc. However, our method considers a different stage during training, i.e., a fully-trained model (but before overfitting). At this stage, the training accuracy on many samples is low, which means for most of the samples, the magnitudes of |p-q| are quite small. Thus, at this stage, the |p-q| are in the similar value for most of the samples. Hence at this stage, using the sign operator is a good approximation of the original p-q. In summary, we can consider Eq (1) in this way: $\Delta q = \eta\cdot A\cdot K\cdot sign(p-q)\cdot |p-q|$. We merge $K$ and $sign(p-q)$ to create an approximation.

Moreover, an extreme example could be two identical samples whose |p-q| is 0. If we take the magnitude of  |p-q| into consideration, then the similarity between the two identical samples is then 0, which is clearly a wrong conclusion.

Reviewer comment (Presentation):
Although Figure 2 and the displayed evolution of distinct FPC clusters is insightful, I was wondering whether additional visualizations of the lpNTK would have been possible?

Author response:
Thanks for pointing out the insights from the results. We were wondering if the reviewer has particular quantity to track during the FPC clustering? Since the computation complexity of FPC clustering is $\mathcal{O}(N log M)$, doing it every epoch is quite expensive, so we only did it after the convergence of the model. 
    
We are happy to provide further visualisation if there’s any interesting quantity in mind to track and visualise.

Reviewer comment (Evaluation):
It would have been great if the authors had provided a visualization of some of the pruned samples. Are there any patterns present in pruned examples in particular?

Author response:
We thank the reviewer for bringing up this interesting question. In fact, the samples in Figure 9 and Figure 10 are very likely to be pruned, as they are the most interchangeable ones to the centroid sample of the head cluster. Qualitatively speaking, one pattern we observed is that they are very typical and their feature is shared across each other. It will be very interesting to explore whether it is possible to quantify our observation.

Reviewer comment (Presentation):
It is evident that the authors tried really hard to fit the paper into the page limit. There are formatting artifacts in terms of very little vspace throughout the paper, especially towards the end of the paper (pages 7-9). I would strongly encourage the authors to reduce dangling words instead of resorting to these very evident formatting tricks.

Author response:
We thank the reviewer for the suggestion, we've removed dangling words, and made some space for the last few sections.

Reviewer comment (Theory):
Concerning Theorem 1, the authors assert that 'the gap between lpNTK and eNTK will not be significantly different.' However, this seems contradictory to subsequent analysis and empirical studies presented. The paper's central theme appears to be the integration of label information for a more nuanced understanding of sample relationships. If the lpNTK kernel closely resembles the original eNTK, could the authors clarify how this supports the stated claim? This warrants further explanation.

Author response:
We thank the reviewer for pointing out this inconsistency, which might harm the understanding of the whole story of the paper. We've updated the Theorem 1 to avoid the following misconception: readers might think when the width ($w$) increases, the RHS of equation 4 will converge to 0, and hence lpNTK converges to eNTK in terms of F-norm. However, the original equation 4 says |lpNTK - eNTK| / |eNTK| converges to $O(w^{-1/2})$. If we move the |eNTK| term, which is $O(w^{1/2})$, to RHS, the new Theorem 1 becomes |lpNTK - eNTK| → O(1). That is to say, even under the infinity width assumption, the gap between lpNTK and eNTK is relatively bounded (not diverging to an arbitrarily large value) and non-negligible (not converging to 0). We also added Appendix B.2 to interpret more about the Theorem 1.

Reviewer comment (Novelty):
The categorization of samples into three groups mirrors the 'data-IQ' framework in Seedat et al. (2022), which also segments samples into easy, ambiguous, and hard categories. Data-IQ assesses prediction error and confidence throughout the training process, a concept seemingly echoed in this paper. I recommend that the authors draw comparisons with this methodology to highlight distinct aspects and contributions of their work.

Author response:
We thank the reviewer for providing this good reference. We will add some discussion between our method and Data-IQ in the next version of the paper. In short, there are two major differences between these two works. First, they consider the difficulty of samples from different perspectives. Data-IQ focuses more on the properties of individual samples, while lpNTK focuses more on the interaction between different samples. Second, Data-IQ evaluates the difficulty by observing the learning curve, while lpNTK can explain why such curves might emerge. We believe the findings of these two works can support each other (e.g., contradictions in lpNTK might be a sign of the existence of a hard sample, which could be a high-confidence prediction with a wrong label.)

Reviewer comment (Experiments):
There lack baselines to validate the effectiveness of the proposed method for measuring sample similarity. For instance, the influence function is a known technique for understanding relationships between samples during training. A comparison with such established methods would provide a more robust validation of the proposed approach.

Author response:
We thank the reviewer for pointing out the influence function as a possible baseline. We will add influence-score as one of the baselines for the experiments in Section 4.3 in the next version. In the meantime, we also mention that, influence-score is usually not as good as EL2N or forgetting score on CIFAR datasets [1]. So, the validation in Section 4.3 is very likely to be unaffected. In fact, in our extension work of lpNTK, our lpNTK-inspired data pruning method outperforms [1] and other existing baselines, which can help to show the practical effectiveness of lpNTK.
    
Moreover, influence function focuses on how one sample might change the estimated parameters thus the predictions on other samples, whereas lpNTK focuses on the relationship between samples in the lpNTK representation space (a transformed gradient space).
    
[1] Yang, S., Xie, Z., Peng, H., Xu, M., Sun, M., & Li, P. (2022). Dataset pruning: Reducing training data by examining generalization influence. ICLR-2023.
    
[2] Koh, P. W., & Liang, P. (2017, July). Understanding black-box predictions via influence functions. In *International conference on machine learning* (pp. 1885-1894). PMLR.

Reviewer comment (Evaluation):
It would be beneficial if the authors could include a computational complexity analysis of the lpNTK. This information would be crucial for understanding the practicality and scalability of the proposed method in different settings.

Author response:
We thank the reviewer for pointing out this important problem. We've updated the discussion to cover the computational complexity of lpNTK. In short, the computational complexity of lpNTK is $\mathcal{O}(N^2d^2)$ where $N$ is the number of samples and $d$ is the number of parameters.
    
The bottleneck of calculating lpNTK is mainly the size of the dataset: larger $N$ means a larger matrix (the influence of the number of classes K is circumvented by using the pNTK trick). We are considering to reduce the cost of this part using proxy quantities. For example, the learning-curve-based method (e.g., C-score or Data-IQ), or quantities like losses. Once the easy sample is identified, it is then not necessary to calculate $K(x_1, _x2)$ if $x_1$ and $x_2$ are both easy samples and look similar to each other.

Furthermore, we can also use our method in some domains where the dataset size is not that big, e.g., alignment for LLM (roughly 10k samples needed) or few-shot learning. We would leave these to our future work.

Reviewer comment (Writing):
Maybe it's because I'm unfamiliar with the literature, but I feel that this paper can benefit from mathematically clearer definitions of some terms such as interaction and learning difficulty.

Author response:
We thank the reviewer for the suggestion. We fixed the minor problems pointed out in the minor issues above to make the definitions of $\mathbf{z}$ and $\mathbf{q}$ more rigorous.  Regarding the definition of learning difficulty, in this paper, we measure it by accumulated loss over training, and interpret it as a result of pair-wise influence between samples. We agree that mathematically clearer definitions can further improve the formulation of the problem. Regarding the interaction between samples, in short, given a threshold $e$, intuitively, we can have the following relationships:
    
- interchangeable: $lpNTK(x_1, x_2) \gg e$
    
- unrelated: $-e < lpNTK(x_1, x_2) < e$
    
- contradictory: $lpNTK(x_1, x_2) \ll  e$
    
However, as this relative difficulty is a qualitative measurement, which highly depends on the dataset and network, we find it hard to define it rigorously.

Reviewer comment (Experiments):
Since many methods for noisy label learning and selective prediction (classification with rejection) heuristically make use of the learning dynamics, it would be convincing to apply lpNTK to those applications. However, those can be future work directions.

Author response:
We appreciate the reviewer for pointing out this interesting direction. This is indeed very intriguing and promising. We will explore that in our future work.

Reviewer comment (Theory):
It is not completely clear to me why it is reasonable to fix the similarity matrix $\mathbf{K}$. Isn't it that a pair of training examples can be similar or dissimilar during different stages of training? How can we obtain the model that performs the best on the validation set in applications like data pruning?

Author response:
We thank a lot for the reviewer to point this out. In fact, we’re working on a data-pruning method based on lpNTK. The limitation of lpNTK in the dynamic case is its computational complexity. So, to make it possible to approximate the lpNTK between samples during the training, we find a relatively good enough and practical proxy, i.e. the loss of samples. Following our analysis in Section 4.3, easy samples contribute not much to the generalisation performance. So, in our data pruning work based on lpNTK, we can prune samples that have similar lpNTK representations, thus smaller losses, without significantly decreasing the generalisation performance. Our experiments on ImageNet-1K shows that this lpNTK-based pruning method indeed outperforms the existing SOTA data pruning methods.
    
Regarding the stability of $K$, it is true that the similarity matrix $K$ would behave quite differently during training. Specifically, at the beginning of pre-training, $K$ changes a lot and we cannot get too much useful information in this phase. On the other hand, at the very end of the training, $K$ jitters around a stable value, because most of the samples are already learned well, which is also not what we want. We speculate that is because the NN does feature learning at the beginning of training and then does the NTK-style learning after training for several epochs. In our experiments, we chose the model in this phase to calculate lpNTK, i.e., trained but not overfitted yet (we can observe the validation curve to select it, the phase change is quite obvious and the value of $K$ is quite stable in this phase). We added an outline for computing lpNTK in practice at the end of Section 2.3.