Paper: Spectrally Transformed Kernel Regression (ICLR-2024)

Reviewer comment (Novelty):
Claims: it is stated in the abstract and in the paper many times, that this contribution provides a unifyied theory for learning with unlabeled data and a base kernel. I think this message does not hold and is misleading for the reader. I consider that the paper provides a novel class of regression estimators able to exploit labeled and unlabeled data by considering that the right space to work in is the RKHS associated to a Spectrally transformed kernel or its approximation.

Author response:
Regarding the claims: We thank the reviewer for this excellent point. We agree with the reviewer that this paper is more about proposing a class of new regression estimators, than developing a unified theory. We have repositioned our paper and made this point very clear in the updated pdf, which we invite the reviewer to check out.

Reviewer comment (Theory):
I am surprised that the discussion in page 3 takes classic kernel ridge regression as a reference method while manifold learning with Laplacian regularization (in addition to $\ell_2$ norm) would be more interesting to discuss here: see Belkin et al. 2006 JMLR to be cited (other papers of Belkin are cited, though). Belkin, M., Niyogi, P., & Sindhwani, V. (2006). Manifold regularization: A geometric framework for learning from labeled and unlabeled examples. Journal of machine learning research, 7(11). In particular, the example discussed in figure 1 could have been solved with manifold learning. At this stage of the paper, I see the advantage of defining the right regularization term with the right kernel to empower the RKHS with the good smoothness properties. However I am a bit confused then by example 1 given in page 7.

Author response:
We have rewritten the analysis of the graph example entirely in Section 2.1 of the updated pdf and added a comparison with Belkin et al. (2006). Please see the third point of our general rebuttal. Here we use this simple graph example instead of a manifold learning example, because this paper does not contain lots of background of manifold learning, and we want to make sure that most people can understand the paper without too much background. We thank the reviewer for this great comment.

Reviewer comment (Theory):
In Example 1 page 7 (Inverse laplacian), I am interested in a discussion about the pros and cons of the proposed method where $K_s$ is defined as a combination of two kernels with a hyperamater eta that seems to play a similar role that the weight controlling the importance of $\ell_2$ norm and the Laplacian smoothness penalty. Here, I don't see how one can take into account the dependency of the excess risk on this "hidden" parameter.

Author response:
We added an ablation study on the effect of $\eta$ on the tasks we study in Appendix D. We also edited the definition of inverse Laplacian in Example 1. The observation of our ablation study is that: if $\eta$ is too close to 0, then the performance is very low; as long as it is bounded away from 0 (even as small as 0.001), the performance is good and fairly consistent as $\eta$ grows. There does have a slight performance increase as $\eta$ gets larger, but except for one dataset (CS), this increase is not significant. As for a theoretical analysis on the effect of $\eta$, it really depends on what the real unknown $s$ is, so a general result is hard to prove.

Reviewer comment (Novelty):
Originality: the paper resembles a bit to the recent work of Zhai et al. (2023) and it should be interested to highlight the differences; I will advise the authors not to go too far in the direction of data augmentation to strengthen the differences.

Author response:
We have added a comparison with Zhai et al. (2023) on page 9 of the updated pdf.

Reviewer comment (Writing):
The paper is very dense and clarity can be improved. Some suggestions: please provide a table (can be in the first page of the supplements) with a reminder of all the notations for the various kernels, $K_s$, $K^p$, $\tilde{f}$, $\hat{f}$, etc.

Author response:
A table of notation has been added at the beginning of the appendix in the updated pdf. Thank you for this great suggestion.

Reviewer comment (Presentation):
Clearly state when you introduce the intuition and then formally state the theoretical results, there is a mix of everything all along the paper which makes the paper sometimes difficult to read.

Author response:
We have largely rearranged Sections 3 and 4, adding clear separations between different parts and labeling them with arrow marks. Please kindly let us know if you find that the new version is easier to read, and if there is any further improvement to be made.

Reviewer comment (Presentation):
In experimental results (last page): complete the legend so that the reader can see what is the used criterion here (accuracy with a post-processing of the regression estimation outputs).

Author response:
We have edited all table and figure captions in the updated pdf.

Reviewer comment (Reproducibility):
There is no discussion about the base kernel $K$ upon which the spectrally-transformed kernel is built. How to choose it?

Author response:
How to choose a good $K$: This is a very broad and hard open question with quite a long history, which we do not intend to answer in this work, as we have mentioned in the limitations section.

Reviewer comment (Presentation):
To improve the clarity of the paper: please clearly state when you introduce the intuition and then formally state the theoretical results, there is a mix of everything all along the paper which makes the paper sometimes difficult to read.

Author response:
We have largely rearranged Sections 3 and 4, adding clear separations between different parts and labeling them with arrow marks. Please kindly let us know if you find that the new version is easier to read, and if there is any further improvement to be made.

Reviewer comment (Writing):
Express more clearly Theorem 1: how is defined $r_t$ prior to the conclusion: only a function from $L^2(P_X)$ to $\mathbb{R}$?

Author response:
Regarding $r_t$ in Theorem 1: The key assumption here is that the target smoothness $r_t$ has the same form as the multiscale smoothness $r_{K^p}$, with respect to a certain metric $d_t$. We have clarified this point in the updated pdf, and revised Sections 2.1 and 2.2.

Reviewer comment (Theory):
Explain the importance of centering the functions or alternatively using a "variance term" as a penalty.

Author response:
Regarding centering the kernel: We have added a paragraph about why the centeredness assumption is necessary on page 3 of the updated pdf. We have also added remarks on this point in Section 2.1. Thank you for bringing up this point.

Reviewer comment (Theory):
Give the analytic complexity in time of the algorithms.

Author response:
Analytic complexity: The time complexity is analyzed in Appendix C.

Reviewer comment (Experiments):
In the experiments provide the behaviour of Manifold learning (Belkin et al. 2006), especially the closed-form with ridge and Laplacian regularisation.

Author response:
Experiments of manifold learning: Thank you for this suggestion. Unfortunately, we don’t think we have any space left for the manifold learning experiment in this paper. We can put it in the appendix, but we feel that this experiment is important and should appear in the main body, which is currently over-stuffed. Thus, we intend to put it in a future paper, like what we also say in our response to other reviewers. This future paper will contain a much more extensive empirical study, including not only manifold learning but also modern self-supervised and semi-supervised learning.

Reviewer comment (Theory):
Proposition 1 assumes functions extended to \bar{X}. But what if X itself is a vector space? Then this extension basically means we are considering only linear functions over X. So proposition 1 would be restricted to linear functions over X. Am I missing something? Since Proposition 1 is used in Theorem 1, perhaps this is an issue?

Author response:
Regarding the definition of $\bar{X}$: In the updated pdf, we have added the clarification in Section 2.1 that the “addition” in $\bar{X}$ is an algebraic addition, not the Euclidean addition. Thus, $f(x)$ is a linear functional on $\bar{X}$, but it is not a “linear model”, because the “addition” is defined differently.

Reviewer comment (Writing):
On page 3, "p" is used as exponent as well as likelihood.

Author response:
"$p$ is used as exponent as well as likelihood": $dp(x)$ is a shorthand of $d P _X (x)$. It could be better to use a different symbol, but we think that the notation should be clear from context. We have also added a table of notations at the beginning of the Appendix in the updated pdf.

Reviewer comment (Presentation):
Though the flow of the paper is neat, the presentation may be improved which may help with reading such dense set of results. For instance, the definitions of the (target)-smoothness may be done clearer.

Author response:
We have largely improved the writing of our paper and we invite the reviewer to check out the updated pdf. Regarding the definition of the target smoothness, we have largely rewritten Section 2.1, and we believe that the motivation and intuition of how the target smoothness is defined should be much clearer now. If the reviewer has any other suggestions in terms of presentation, please kindly let us know.

Reviewer comment (Novelty):
The laplacian kernel and kernel PCA are known in the literature. In terms of final application, how exactly this work exactly differs in applying them for the problems? Are there other recommendations for choosing the transformation functions for practical use cases?

Author response:
"The laplacian kernel and kernel PCA are known in the literature": As we stated in our second contribution, the novelty of our methods is that they work for general transformation under the inductive setting. Most prior work only focuses on the transductive setting, and to our best knowledge, this work is the first to propose a scalable and general method for the inductive setting. Thus, previous methods such as label propagation need to know the entire graph and all test nodes during training, while our method only requires to know the training nodes and the edges between them, which we demonstrate in our experiments.

Reviewer comment (Theory):
Notation: p is an integer in eqn 7, while a real number in eqn 10. Are they consistent?

Author response:
$p$ is real-valued. It is only an integer in Section 3, where we have Assumption 1 that explicitly restricts $s(\lambda)$ to be a polynomial.

Reviewer comment (Experiments):
The experiments are rather limited to node classification in graphs, which, while interesting, raises the question of whether the proposed method, despite its generality, is useful on other types of problems.

Author response:
Experiments: We agree that a comprehensive empirical investigation would be an interesting study. Since our work focuses on developing theory and the paper in its current form is already very dense, we believe that it would be better to defer a more extensive empirical study on STKR to a future paper, especially within the context of modern self-supervised and semi-supervised learning. This is definitely something we plan to work on.

Reviewer comment (Experiments):
The results are only compared to one competitor method, label propagation, which limits the scope of empirical validation.

Author response:
STKR-Prop vs Label-Prop: We have pointed out in the “implementation” paragraph in Section 3 that STKR-Prop (with inverse Laplacian) has the same time complexity as Label-Prop on a graph under the transductive setting. The difference here is that STKR-Prop uses the inductive version of inverse Laplacian, while Label-Prop uses the transductive version, whose difference and connection are discussed in Appendix B.3.

Reviewer comment (Experiments):
It would be interesting to see how the proposed method compares to label propagation in terms of (training and) inference time.

Author response:
STKR-Prop vs Label-Prop: We have pointed out in the “implementation” paragraph in Section 3 that STKR-Prop (with inverse Laplacian) has the same time complexity as Label-Prop on a graph under the transductive setting. The difference here is that STKR-Prop uses the inductive version of inverse Laplacian, while Label-Prop uses the transductive version, whose difference and connection are discussed in Appendix B.3.

Reviewer comment (Experiments):
It would be interesting to see, if there is another known feasible inductive method, how this method compares to STKR.

Author response:
Other feasible inductive methods: To the best of our knowledge, this is the first work that proposes an efficient and scalable STKR implementation for the inductive setting. We have also pointed out that for the inductive setting, the inference of STKR-Prop is much faster than previous methods, such as Chapelle et al. (2002).

Reviewer comment (Experiments):
It would be interesting to see how the proposed method compares to other (non-kernel) methods, such as deep learning methods for these problems.

Author response:
Non-kernel methods: There are deep learning methods for these problems (we mentioned them in the second contribution in the introduction section), but these methods do not have rigorous guarantees. In contrast, the goal of this work is to propose an approach that is general, scalable, and comes with provable guarantees. We intend to do a more comprehensive empirical study of our method and deep learning methods in future work.

Reviewer comment (Experiments):
While the proposed method is already relevant through its theoretical analysis, a better experimental evaluation would help to better understand the practical relevance.

Author response:
Experiments: We agree that a comprehensive empirical investigation would be an interesting study. Since our work focuses on developing theory and the paper in its current form is already very dense, we believe that it would be better to defer a more extensive empirical study on STKR to a future paper, especially within the context of modern self-supervised and semi-supervised learning. This is definitely something we plan to work on.

Reviewer comment (Theory):
The assumptions used in the theoretical analysis appear to be relatively strong compared to what is known from the supervised learning literature, particularly in continuous input spaces rather than graphs.

Author response:
We assume $f^* \in H _{K _s}$, because $K _s$ is assumed to have captured the “target smoothness” that $f^*$ possesses. One possible way of relaxing this is to use the power spaces in Fischer & Steinwart (2020) with a smaller power, and our results shouldn’t be too hard to extend (similar to how we relaxed $H_t \subset H_K$).

Reviewer comment (Theory):
For example, if the base kernel was a Sobolev kernel, the associated RKHS would have to have a smoothness \( s > d/2 \), which might be unrealistic in high dimensions.

Author response:
In theory, we can use $H_{K^p}$ for $p < 1$, up to $p \ge 0$. The tricky thing here is that in this case, $H_{K^p}$ is not necessarily an RKHS, and consequently $H_t$ could no longer be an RKHS, so we cannot explicitly write out the kernel $K_s$ as in Theorem 1. But with the extra assumption that $H_{K^p} (p < 1)$ is still an RKHS, our results can be easily extended. We made a new remark on this point after Theorem 1 in the updated pdf.

Reviewer comment (Theory):
Many theoretical results, such as those in Fischer & Steinwart (2020), allow the target function to lie in an interpolation space larger than the RKHS. It is questioned whether such an assumption could be sensible in this paper, since having higher smoothness than the base kernel appears to be crucial.

Author response:
We assume $f^* \in H _{K _s}$, because $K _s$ is assumed to have captured the “target smoothness” that $f^*$ possesses. One possible way of relaxing this is to use the power spaces in Fischer & Steinwart (2020) with a smaller power, and our results shouldn’t be too hard to extend (similar to how we relaxed $H_t \subset H_K$).

Reviewer comment (Experiments):
The main weakness, in my view, is the lack of a comprehensive empirical investigation. While the paper does explore the effect of different transformations $s(\lambda)$ and provides a comparison to other methods such as label propagation and kernel PCA, it does not delve deeply into the conditions or characteristics of datasets that would lead to STKR's superior performance. It is probably very difficult to do this theoretically, but a more comprehensive empirical investigation could lead to insights. This could be seen as a limitation in fully understanding the potential and limitations of STKR.

Author response:
Empirical investigation: We agree that a comprehensive empirical investigation would be an interesting study. Since our work focuses on developing theory and the paper in its current form is already very dense, we believe that it would be better to defer a more extensive empirical study on STKR to a future paper, especially within the context of modern self-supervised and semi-supervised learning. This is definitely something we plan to work on.

Reviewer comment (Evaluation):
While the paper mentions the efficiency of the STKR-Prop, a detailed computational complexity analysis, especially in comparison with other methods, could add more depth to the evaluation of the methodology.

Author response:
Computational complexity and comparison with other methods: We have an analysis of the computational complexity of STKR in Appendix C. Regarding its comparison with Label-Prop, we have pointed out in the “implementation” paragraph in Section 3 that STKR-Prop (with inverse Laplacian) has the same time complexity as Label-Prop on a graph under the transductive setting. The difference here is that STKR-Prop uses the inductive version of inverse Laplacian, while Label-Prop uses the transductive version, whose difference and connection are discussed in Appendix B.3. As for the inductive setting, to our best knowledge this is the first work that proposes an efficient and scalable STKR implementation for the inductive setting, so there isn’t really a good baseline to compare to. We have pointed out that for the inductive setting, the inference of STKR-Prop is much faster than previous methods, such as Chapelle et al. (2002).

Reviewer comment (Reproducibility):
I don't see where the regularisation term $\eta$ in SP-Lap is specified within the paper.

Author response:
Values of $\eta$: In the updated pdf, we added the list of hyperparameters we used in our experiments in Appendix D, which includes $\eta$. Note that the best hyperparameters are always selected using a separate validation set.