Paper: Enhancing Kernel Flexibility via Learning Asymmetric Locally-Adaptive Kernels (ICLR-2024)

Reviewer comment (Experiments):
The most important caveat is that the paper has only been compared to vanilla KRR methods. It is not surprising that slightly better performance was obtained compared, for example, to FALKON. It is not at all convincing that this method is better than well-developed techniques such as: i. traditional Automatic Relevance Determination (ARD) known in the GP community, implemented with Gpytorch (see https://docs.gpytorch.ai/en/stable/kernels.html or section 5 of https://gaussianprocess.org/gpml/chapters/RW.pdf), ii. EigenPro3.0 (see https://arxiv.org/abs/2302.02605), and iii. Recursive Feature Machines (RFMs) (see https://arxiv.org/abs/2212.13881).

Author response:
## Accuracy Comparison in Regression with EigenPro3.0 and RFMs

In response to your concerns regarding accuracy improvement, as acknowledged by the first reviewer, our algorithms **showcase state-of-the-art accuracy in regression tasks.** Notably, our results surpass even the performance of ResNet, a benchmark for high-level regression methods in small and medium-sized datasets.

We sincerely appreciate the advanced kernel methods you recommended. To address your main concern, we have conducted an additional comparative experiment with EigenPro 3.0 and Recursive Feature Machines (RFMs).

**Table1: Performance of EigenPro3.0 and LAB RBF kernels on real datasets.**
|               Dataset               |   Airfoil (N=1503)  |  Parkinson (N=5875) | Electrical (N=10000)  | KC House (N=21623) | TomsHardware (N=28179) |
|:-----------------------------------:|:----------:|:----------:|:----------:|:--------:|:------------:|
| Eigenpro3.0-Laplace  ($R^2$) | 0.9201 | 0.9251 | 0.9593          | 0.8636 |    0.9436    |
|      Eigenpro3.0-#Centers       |     1202    |     4700    |     10000    |    20000   |      20000     |
| RFMs  ($R^2$) | 0.9394 | 0.9988 |  0.9582       | 0.9008 |   0.9115    |
|         LAB RBF kernels ($R^2$)        |   0.9608   |   0.9950   |   0.9642   |  0.8917  |    0.9809    |
|         LAB RBF kernels (#S.V.)        |     200    |     110    |     300    |    400   |      500     |


In our experimental setup, we train EigenPro3.0 for 50 iterations, while RFM for 5 iterations, aligning with the parameter setting outlined in their papers. As shown in the public code, RFMs use all training data points to compute the final decision function.  That is, they **use all training data as the support data.**

Both EigenPro3.0 and RFM exhibit excellent capabilities in managing large-size kernel models; however, their dependence on  symmetric kernel formulations poses a high requirement on large number of support data. While by utilizing locally adaptive bandwidth, **our algorithm can significantly reduce the requirement of support data—essentially, the model complexity—while maintaining a high level of accuracy.**

Moreover, we've observed that in RFMs, they learn a matrix $M$ in a generalization of the Laplace kernel $\mathcal{K}(x_i, x_j) = \exp(-\gamma\||x_i - x_j\||_M)$, where $\||x_i - x_j\||_M = (x_i - x_j)^\top M(x_i - x_j)$. That is, the matrix $M$ is designed to **capture relationships among features**.

In contrast, our proposed LAB RBF kernels $\mathcal{K}(x_i, x_j) = \exp(-\||\Theta_j\odot(x_i - x_j)\||_2^2)$  and corresponding learning algorithm aim at **learning locally-adaptive bandwidths for each support data**. Exploring a completely different mechanism, combining these two methods in the future could prove to be intriguing

Again, we appreciate your recommendation of these advanced methods,  and we are committed to incorporating this comparative experiment into the revised version of our work.

Reviewer comment (Experiments):
The authors claim that this method is scalable and provide Table 3 to justify this. However, those datasets are not at all large scale. The inverse problem can be done using direct calculation for those cases. The authors should try other datasets such as Taxi or CIFAR5m to justify consistency and scalability (both in data size and model size).

Author response:
## Scalability is not Our Contribution
 
First of all, we wish to emphasize our central focus, as indicated by the title. Our paper aims to explore **the application of asymmetric kernel learning in regression tasks** and assess its advantages over symmetric kernels, rather than concentrating on large datasets or large kernel machines.

Our main contributions lie in (i) introducing the first asymmetric KRR model and (ii) presenting a novel asymmetric kernel learning algorithm. These two techniques collectively offer **a fresh approach to learn flexible asymmetric kernels in regression tasks**. Current investigations into asymmetric kernels continue to face **a gap in terms of interpretable regression models and efficient kernel learning algorithms**, adding to the novelty of this work.

We  would like to clarify what we mean by scalability in the context of this paper. The introduction of locally-adaptive bandwidths largely enhance the flexibility of RBF kernels. This allows us to **significantly reduce the need for support data, and the decrease in support data consequently benefits the scalability of the model.** This is also what we want to emphasis in Table 3, with the reduction being more evident in the larger datasets. 

Consider that we can reduce the requirement for support data. When coupled with existing acceleration techniques, as you suggested, it becomes feasible for us to effectively manage millions of data points. The feasibility, however, hinges on the computing infrastructure and the number of support vectors, aspects beyond the scope of this paper.

It's noteworthy that a majority of current solutions for large kernel machines are founded on symmetric kernels. We believe extending their approach to accommodate asymmetric kernels would be particularly intriguing.

Nevertheless, we value your comments and will carefully revise the paper to eliminate any potential ambiguity.

Reviewer comment (Theory):
It is mentioned in Section 3 that the computation complexity is O(N_sv^3). This fundamentally shows that this method on its own is not scalable. Eventually, the required number of support vectors (or model size) needs to be scaled, as discussed in https://arxiv.org/abs/2302.02605. However, this method combined with other methods like FALKON or EigenPro3.0 can potentially be scalable.

Author response:
## Scalability is not Our Contribution
 
First of all, we wish to emphasize our central focus, as indicated by the title. Our paper aims to explore **the application of asymmetric kernel learning in regression tasks** and assess its advantages over symmetric kernels, rather than concentrating on large datasets or large kernel machines.

Our main contributions lie in (i) introducing the first asymmetric KRR model and (ii) presenting a novel asymmetric kernel learning algorithm. These two techniques collectively offer **a fresh approach to learn flexible asymmetric kernels in regression tasks**. Current investigations into asymmetric kernels continue to face **a gap in terms of interpretable regression models and efficient kernel learning algorithms**, adding to the novelty of this work.

We  would like to clarify what we mean by scalability in the context of this paper. The introduction of locally-adaptive bandwidths largely enhance the flexibility of RBF kernels. This allows us to **significantly reduce the need for support data, and the decrease in support data consequently benefits the scalability of the model.** This is also what we want to emphasis in Table 3, with the reduction being more evident in the larger datasets. 

Consider that we can reduce the requirement for support data. When coupled with existing acceleration techniques, as you suggested, it becomes feasible for us to effectively manage millions of data points. The feasibility, however, hinges on the computing infrastructure and the number of support vectors, aspects beyond the scope of this paper.

It's noteworthy that a majority of current solutions for large kernel machines are founded on symmetric kernels. We believe extending their approach to accommodate asymmetric kernels would be particularly intriguing.

Nevertheless, we value your comments and will carefully revise the paper to eliminate any potential ambiguity.

Reviewer comment (Reproducibility):
Clarification is needed on how line 4 of the algorithm is computed. Did you use FALKON, some other off-the-shelf algorithm, or a direct inverse?

Author response:
1. The computation at line 4 of the algorithm, specifically evaluating $$f_{\mathcal{Z},\Theta}(t) = K_{\Theta} (t,X_{sv})(K_\Theta(X_{sv},X_{sv})+\lambda I_N)^{-1}Y_{sv}$$
is not  time-consuming, especially when the number of support data is limited. In our experiments, we directly compute this inverse due to the manageable size of the support data. We agree that existing scalability techniques, while primarily designed for symmetric kernels, can be applied here to further speed-up the computation, opening new possibilities.

Reviewer comment (Theory):
More explanation is needed for asymmetric kernel methods. For example, why does the inverse even exist in equation 6? Or you claimed "this paper for the first time establishes an asymmetric KRR framework," but how is it different from He et al.’s paper? This is not clear.

Author response:
2. Distinguish from Prior Works: While previous works [[1](https://www.sciencedirect.com/science/article/pii/S1063520315001360)][[2](https://ieeexplore.ieee.org/document/10070836)] and our study use the same definition of asymmetric kernels, $\mathcal{K}(x,t) = \langle \phi(x), \psi(t)\rangle$, our paper introduces **the first formulation of asymmetric kernel ridge regression**.  In contrast to previous studies focusing on PCA and least square SVM, this results in distinct stationary solutions and applications.
Furthermore, prior works employ manually designed asymmetric kernels such as the Kullback-Leibler Kernel and SNE Kernels (refer to [2] for details). While these kernels exhibit good performance on asymmetric data (e.g., directed graphs), their advantage on general data is not apparent. To address this limitation, our paper proposes a novel asymmetric kernel learning algorithm, demonstrating **superior performance on general data**.

3. In Eq. (6), since $(K(X,X)+\lambda I)$ is a square matrix, its inverse exists as long as it is full-rank.

Reviewer comment (Presentation):
Please add what "M" means in the tables, as it would help with reading.

Author response:
-   The definitions of N and M have been incorporated. Please refer to Table 1 on page 7.

Reviewer comment (Theory):
Intuitively, the relation between the mapping function's distinctiveness and the loss function, which means the coefficient of the last term in the KRR optimization objective may vary with datasets.

Author response:
## Coefficient of the Error Terms in Asymmetric KRR Model
Thanks for your recognization on the novelty of our asymmetric kernel ridge regression model. We are glad that you are interested in this model. It is one of our major technical contributions to deal with asymmetric kernels on regression tasks and also provids theoretical support for subsequent asymmetric kernel learning algorithm.

To address you concern on the coefficient of the error terms, it would be better to start at the LS-SVM form of our model, i.e. the model in Eq. (7).

The objective function in Eq. (7) primarily incorporates a **regularization term** $w^\top v$ and a **error term** $\sum_{i=1}^N e_i r_i$. As most of machine learning models, these two terms are adjusted by **a trade-off hyper-parameter $\lambda$ varing with different datasets**.

By substituting constraints in Eq. (7) to the objective function, we obtian the model in Eq. (4),  whose corresponding objective function is represented as $$\lambda w^\top v + \frac{1}{2}(\phi^\top(X)w-Y)^\top(\psi^\top(X)v-Y). $$To enhance clarity, we delved deeper into the interpretation of $(\phi^\top(X)w-Y)^\top(\psi^\top(X)v-Y)$, outlining it as two loss terms and one mapping function’s distinctiveness term. That is,
$$2(\phi^\top(X)w-Y)^\top(\psi^\top(X)v-Y) = \||\psi^\top(X)v-Y\||_2^2 + \||\phi^\top(X)w-Y\||_2^2 - \||\psi^\top(X) v - \phi^\top(X) w\||_2^2.$$
Therefore, **the three terms** $\||\psi^\top(X) v - \phi^\top(X) w\||_2^2$, $\||\psi^\top(X)v-Y\||_2^2$, and $\||\phi^\top(X)w-Y\||_2^2$ **are not independent; instead, they are integrated as a whole** and thus do not require a extra hyper-parameter tuning.

Thank you for this insightful question. We will make it clearer in the revised version.

Reviewer comment (Experiments):
The initial data selection for support data in the kernel learning algorithm proposed in the article seems to be too random. Moreover, inappropriate data selection appears to have a significant impact on the model.

Author response:
## Impact of the Initial Support Data Selection 
Firstly, you are correct in pointing out the significant influence of support data selection on the algorithm. In light of this, **we have introduced a dynamic strategy aimed at mitigating the impact of initial support data selection.**

To address your concern, we explore three different approaches to initial data selection: two rational methods (Y-based and X-based) and one irrational method (Extreme Y).
 -   Y-based (utilized in the manuscript): data is sorted based on their labels, and support data is uniformly selected.
-   X-based: k-means is applied to the training data to identify cluster centers, followed by the selection of data points closest to these centers.
-   Extreme Y: data is sorted based on their labels, and those with the largest Y values are selected.

**Table1: Performance of Alg.1 with different selection of initial support data.**
|Dataset| Selection Approach |  Mean of $R^2$   |   Std of $R^2$   |
|:------------------:|:----------------:|:-------:|:------:|
Yacht|     Extreme Y    | 0.0012 | 0.4805 |
Yacht|     Y-based    | 0.9957 | 0.0025 |
Yacht|    X-based    | 0.9953 | 0.0032 |
Parkinson|     Extreme Y    |   0.8115 | 0.0126 |
Parkinson|     Y-based    |   0.9921 | 0.0015 |
Parkinson|    X-based    |   0.9928  |          0.0016|

The results indicate that the poor selection method does have a detrimental impact on our performance, particularly evident in the case of Yacht where we struggle to fit the data. In contrast, the other two sensible methods demonstrate good and comparable performance.

In order to further improve, we introduce a dynamic strategy at the end of Section 3. In this strategy, we dynamically incorporate hard samples into the support dataset. We then integrate these approaches with the proposed dynamic strategy to evaluate its effectiveness.

**Table 2: Performance of Alg.1 with dynamic strategy**
|Dataset| Selection Approach | Mean of $R^2$   |   Std of $R^2$   |
|:------------------:|:----------------:|:-------:|:-------:|
Yacht|   Extreme Y     | 0.9961 | 0.0126 |
Yacht|     Y-based    | 0.9982 | 0.0015 |
Yacht|    X-based    |  0.9981 | 0.0016 |
Parkinson|   Extreme Y     | 0.9712 | 0.0049  |
Parkinson|     Y-based    | 0.9972 | 0.0007  |
Parkinson|    X-based    |  0.9966 | 0.0013  |

Based on these results, it is evident that the proposed dynamic strategy has a significantly positive impact on performance. It not only enhances accuracy but also reduces variance, resulting in more stable solutions. Even with the bad selection selection, the final performance is improved to a satisfactory level.

Reviewer comment (Theory):
Is the final coefficient in the asymmetric KRR framework proposed in the article required to be 1/2? Can this be understood as simply for the convenience of computing stationary points?

Author response:
## Coefficient of the Error Terms in Asymmetric KRR Model
Thanks for your recognization on the novelty of our asymmetric kernel ridge regression model. We are glad that you are interested in this model. It is one of our major technical contributions to deal with asymmetric kernels on regression tasks and also provids theoretical support for subsequent asymmetric kernel learning algorithm.

To address you concern on the coefficient of the error terms, it would be better to start at the LS-SVM form of our model, i.e. the model in Eq. (7).

The objective function in Eq. (7) primarily incorporates a **regularization term** $w^\top v$ and a **error term** $\sum_{i=1}^N e_i r_i$. As most of machine learning models, these two terms are adjusted by **a trade-off hyper-parameter $\lambda$ varing with different datasets**.

By substituting constraints in Eq. (7) to the objective function, we obtian the model in Eq. (4),  whose corresponding objective function is represented as $$\lambda w^\top v + \frac{1}{2}(\phi^\top(X)w-Y)^\top(\psi^\top(X)v-Y). $$To enhance clarity, we delved deeper into the interpretation of $(\phi^\top(X)w-Y)^\top(\psi^\top(X)v-Y)$, outlining it as two loss terms and one mapping function’s distinctiveness term. That is,
$$2(\phi^\top(X)w-Y)^\top(\psi^\top(X)v-Y) = \||\psi^\top(X)v-Y\||_2^2 + \||\phi^\top(X)w-Y\||_2^2 - \||\psi^\top(X) v - \phi^\top(X) w\||_2^2.$$
Therefore, **the three terms** $\||\psi^\top(X) v - \phi^\top(X) w\||_2^2$, $\||\psi^\top(X)v-Y\||_2^2$, and $\||\phi^\top(X)w-Y\||_2^2$ **are not independent; instead, they are integrated as a whole** and thus do not require a extra hyper-parameter tuning.

Thank you for this insightful question. We will make it clearer in the revised version.

Reviewer comment (Evaluation):
Is the small number of support vectors in the experimental results of the proposed method due to the algorithm's termination condition?

Author response:
## Why is the Number of Support Data is so Small?

In fact, the limitation in support data is not attributed to the termination condition. Instead, the constrained support data stems from **the remarkable flexibility of LAB RBF kernels.** This flexibility empowers our algorithm to represent intricate data patterns with a reduced number of support data.

In determining the termination condition, we prefer achieving a modest number of support data, given its advantages in computational efficiency and memory conservation, all while upholding accuracy. 

Traditional symmetric kernels, lacking such flexibility, struggle to achieve the same efficiency as they apply a uniform kernel formula to all data points.
  
The example in Figure 1 in the manuscript precisely illustrates this point. Attempting to fit unevenly distributed data with a uniform bandwidth necessitates more support data to approximate local distributions, as depicted in (b) and (c). By leveraging the high flexibility of LAB RBF kernels, as demonstrated in (d), our algorithm perfectly tackles this challenge, resulting in advantages in both accuracy and model complexity.

Reviewer comment (Theory):
While the paper presents empirical evidence of the superior performance of the proposed algorithm, it may lack strong theoretical guarantees or formal analysis of its convergence properties. Further theoretical investigations may be needed to fully understand the behavior and limitations of LAB kernels.

Author response:
We sincerely appreciate your valuable comments and highly recognition of our work.

## About Convergence Properties
We agree with you regarding the significance of convergence properties in algorithmic discussions. Given that our optimization problem, as outlined in equation (9), is a non-convex unconstrained challenge, we have opted for the utilization of stochastic gradient descent (SGD) methods to efficiently address it. Consequently, the convergence analysis of SGD in the realm of general non-convex optimization is pertinent to our algorithm, e.g.[[1]](https://dl.acm.org/doi/abs/10.5555/3455716.3455852). 
For this reason, we chose not to delve into this topic in the manuscript. But we value your suggestions and intend to incorporate a discussion on it in the revised version.

[1] Fehrman, Benjamin, Benjamin Gess, and Arnulf Jentzen. "Convergence rates for the stochastic gradient descent method for non-convex objective functions." _The Journal of Machine Learning Research_ 21, no. 1 (2020): 5354-5401.

## Learning Behavior of LAB RBF Kenrels

The learning behavior of LAB RBF kernels, as you point out, is indeed both intriguing and notably distinct from existing kernels due to their inherently asymmetric nature. The loss of symmetry renders the analysis tools in traditional RKHS, or even in more general case e.g., reproducing kernel Kre&iuml;n spaces (RKKS) and reproducing kernel Banach spaces (RKBS), not applicable. Our ongoing work specifically addresses and explores these unique characteristics. 

Notably, LAB RBF kernels, featuring trainable bandwidths, share a resemblance with multiple kernel learning whose corresponding functional space comprises a direct sum space of Hilbert spaces. Inspired by this correlation, our analysis of LAB RBF kernels aims to construct a sparse model within the direct integral space of Hilbert spaces. This work is not only interesting but also presents a notable challenge due to the complex nature of the space being explored.

Reviewer comment (Experiments):
The performance of LAB kernels heavily relies on the accurate estimation of bandwidths. Selecting appropriate bandwidths for different data patterns can be a challenging task, and suboptimal choices may result in reduced performance or overfitting.

Author response:
## Effects of Inaccurate Estimation of Bandwidths

We appreciate your insights into the crucial role that bandwidths play in the performance of LAB RBF kernels. It is precisely for this purpose, we propose the kernel learning algorithm to determine proper bandwidths for different data patterns.

Indeed there might be some challenges to determine good bandwidth in practice. For example, within our kernel learning algorithm, various factors during training, such as the choice of initial points, stopping criteria, learning rate, and batch size, may result in distinct bandwidth estimations.

In our experiments, these hyper-parameters are determined through cross-validation, detailed in Appendix C. And the influence of initialization is thoroughly addressed in Appendix E.

Here, to address you concern, we present extra experimental results that demonstrate their impact on the performance of our algorithm. 

**Impact of Maximal Iteration Number**
1. Yacht dataset: 246 training data, batch size=32.

| Iterations | 0      | 100    | 200    | 300    | 400    | 500    | 1000   | 2000   | 3000   | 4000   | 5000   |
|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|
| Mean of $R^2$ | 0.5633 | 0.9396 | 0.9789 | 0.9872 | 0.9902 | 0.9913 | 0.9940  | 0.9956 | 0.9966 | 0.9966 | 0.997  |
| Std of $R^2$  | 0.0737 | 0.0316 | 0.0187 | 0.0105 | 0.0073 | 0.0065 | 0.0056 | 0.0033 | 0.0023 | 0.0024 | 0.0021 |

2. Pakinson dataset: 4700 training data, batch size=128.

| Iterations| 0      | 100    | 200    | 300    | 400    | 500    | 1000   | 2000   | 3000   | 4000   | 5000   |
|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|
| Mean of $R^2$ | 0.5458 | 0.9101 | 0.9445 | 0.9622 | 0.9752 | 0.9814 | 0.9889 | 0.9895 | 0.9925 | 0.9931 | 0.9944 |
| Std of $R^2$  | 0.0315 | 0.0065 | 0.0061 | 0.0052 | 0.0061 | 0.0055 | 0.0033 | 0.0023 | 0.0026 | 0.0019 | 0.0019 |

**Impact of Learning Rate**
1. Yacht dataset

|Learning Rate| 1.00E+00 | 1.00E-01 | 1.00E-02 | 1.00E-03 | 5.00E-04 | 1.00E-04 |
|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|
| Mean of $R^2$ |  0.9723 |  0.9968 |  0.9970 |  0.9911 |  0.9804 |  0.8089  |
|Std of $R^2$|  0.0164 |  0.0019 |  0.0017 |  0.0038 |  0.0104 |  0.0246  |

2. Pakinson dataset:

| Learning Rate | 5.00E-01 | 1.00E-01 | 5.00E-02 | 1.00E-02 | 5.00E-03 | 1.00E-03 |
|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|
|  Mean of $R^2$  | 0.98275  | 0.99411  | 0.9943   | 0.99066  | 0.9763   | 0.8947   |
| Std of $R^2$  | 0.0031   | 0.0018   | 0.0018   | 0.0022   | 0.0058   | 0.0052   |

**Impact of Batch Size**
1. Yacht dataset

|  Batch Size | 16       | 32       | 64        | 128      |
|:----------:|----------|----------|-----------|----------|
| Mean of $R^2$ | 0.9932 | 0.9944 | 0.9952 | 0.9965 |
|  Std of $R^2$ | 0.0065 | 0.0044 | 0.0047 | 0.0019 |

2. Pakinson Dataset:

| Batch Size | 16      | 32      | 64      | 128    |
|:----------:|----------|----------|-----------|----------|
| Mean of $R^2$  | 0.9863 | 0.9907 | 0.9931 | 0.9943 |
|  Std of $R^2$  | 0.0041 | 0.0027 | 0.0018 | 0.0018 |

These results underscore that the carefully selection of these hyperparameters enhances the final performance. Nevertheless, even under suboptimal hyperparameter settings, the performance remains commendable, albeit with varying bandwidth estimates. This highlights the robustness and insensitivity of our algorithm across a wide spectrum of hyperparameter choices.


---
We hope that our explanations, along with the additional experiments, effectively address your concerns regarding algorithm convergence and robustness. Furthermore, we welcome the opportunity for further discussions on the learning behavior of LAB RBF kernels if you are interested.