Paper: CLIP as Multi-Task Multi-Kernel Learning (ICLR-2024)

Reviewer comment (Novelty):
Claim that there were no prior theoretical work on multi-task multiple kernel learning. There are multiple work on the subject (this is an old topic) including theoretical analysis. See (Micchelli, C., & Pontil, M. (2004). Kernels for Multi--task Learning. Advances in neural information processing systems, 17.)

Author response:
We appreciate your concern and the reference provided. We apologize for the imprecise wording and the lack of citations in our initial submission. While there is indeed a substantial body of literature discussing kernel constructions, optimization, and computation aspects of MKL in the multi-task setting, we intended to highlight the scarcity of works that specifically analyze the optimal learning rate in multi-task multi-kernel learning. The statement "no theoretical analysis" refers to the limited research on the analysis of the statistical rate. In the revised version of our paper, we will include a discussion on the general theoretical research line in multi-task multi-kernel learning, along with relevant citations such as [1-6]. We will also ensure that the wording is accurate and precise.

[1] Micchelli, C., & Pontil, M. (2004). Kernels for Multi--task Learning. *Advances in neural information processing systems*, *17*.

[2] Dinuzzo, F., Ong, C. S., Pillonetto, G., & Gehler, P. V. (2011). Learning output kernels with block coordinate descent. In *Proceedings of the 28th International Conference on Machine Learning (ICML-11)* (pp. 49-56).

[3] Ciliberto, C., Mroueh, Y., Poggio, T., & Rosasco, L. (2015, June). Convex learning of multiple tasks and their structure. In *International Conference on Machine Learning* (pp. 1548-1557). PMLR.

[4] Ciliberto, C., Rosasco, L., & Villa, S. (2015). Learning multiple visual tasks while discovering their structure. In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition* (pp. 131-139).

[5] Jawanpuria, P. K., Lapin, M., Hein, M., & Schiele, B. (2015). Efficient output kernel learning for multiple tasks. *Advances in neural information processing systems*, *28*.

[6] Pentina, A., & Ben-David, S. (2015). Multi-task and lifelong learning of kernels. In *Algorithmic Learning Theory: 26th International Conference, ALT 2015, Banff, AB, Canada, October 4-6, 2015, Proceedings 26* (pp. 194-208). Springer International Publishing.

Reviewer comment (Novelty):
I have a big concern about the title of the paper. I do not see why the results in the paper are only specialized to CLIP models instead of general estimating functionals of high dimensional (features embeddings) inner products. In the experimental section, CLIP is barely used! Some embeddings are just extracted. I think if the paper is around CLIP as Multi-task multiple kernel, there should be a whole study on the architecture of CLIP, which to me seems to be a big task. Could the authors please elaborate more on this as this is really confusing to me about the real contribution of the paper?

Author response:
Thank you for the question. We appreciate the comment about the interpretation of our method beyond CLIP as general estimating functionals of high-dimensional inner products. Nonetheless, the main motivation for the proposed method is to provide an alternative method to solve the CLIP loss and some theoretical understanding of CLIP from the RKHS perspective instead of the neural network. We acknowledge the complexity of thoroughly analyzing the intricate neural architecture of CLIP, which is why we have focused on solving the loss function of CLIP and attempted to give an alternative interpretation of the essence of CLIP via the kernel method. Therefore, we believe our proposed MTMK method should be considered a novel methodological contribution to the CLIP community.

Reviewer comment (Novelty):
It is mentioned in the Related literature that no theoretical analysis of MKL has been conductied in the multi-task setting. Could the authors justify that as there is a big literature on the subject. For instance: See (Micchelli, C., & Pontil, M. (2004). Kernels for Multi--task Learning. Advances in neural information processing systems, 17.) which has not been cited/analysed. There are other related works. Could the authors elaborate on this?

Author response:
We appreciate your concern and the reference provided. We apologize for the imprecise wording and the lack of citations in our initial submission. While there is indeed a substantial body of literature discussing kernel constructions, optimization, and computation aspects of MKL in the multi-task setting, we intended to highlight the scarcity of works that specifically analyze the optimal learning rate in multi-task multi-kernel learning. The statement "no theoretical analysis" refers to the limited research on the analysis of the statistical rate. In the revised version of our paper, we will include a discussion on the general theoretical research line in multi-task multi-kernel learning, along with relevant citations such as [1-6]. We will also ensure that the wording is accurate and precise.

[1] Micchelli, C., & Pontil, M. (2004). Kernels for Multi--task Learning. *Advances in neural information processing systems*, *17*.

[2] Dinuzzo, F., Ong, C. S., Pillonetto, G., & Gehler, P. V. (2011). Learning output kernels with block coordinate descent. In *Proceedings of the 28th International Conference on Machine Learning (ICML-11)* (pp. 49-56).

[3] Ciliberto, C., Mroueh, Y., Poggio, T., & Rosasco, L. (2015, June). Convex learning of multiple tasks and their structure. In *International Conference on Machine Learning* (pp. 1548-1557). PMLR.

[4] Ciliberto, C., Rosasco, L., & Villa, S. (2015). Learning multiple visual tasks while discovering their structure. In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition* (pp. 131-139).

[5] Jawanpuria, P. K., Lapin, M., Hein, M., & Schiele, B. (2015). Efficient output kernel learning for multiple tasks. *Advances in neural information processing systems*, *28*.

[6] Pentina, A., & Ben-David, S. (2015). Multi-task and lifelong learning of kernels. In *Algorithmic Learning Theory: 26th International Conference, ALT 2015, Banff, AB, Canada, October 4-6, 2015, Proceedings 26* (pp. 194-208). Springer International Publishing.

Reviewer comment (Experiments):
In the experiments section (section 7.2), the authors do not specify which embeddings are used (are they CLIP embeddings?), justifying again my concerns of why CLIP as a motivation of the paper? If so, more detailed experiments should be performed using CLIP embeddings for large-scale image recognition problem (this is a big literature to be considered).

Author response:
Thank you for the question. We perceive $\phi(x)$ as the embedding of $x$ and derive the corresponding inner products between samples via the method described in Section 4.  Therefore, the embedding derived for images and phecodes in Section 7.2 corresponds to the estimated mapping of our MTMK method. As for the initial word embeddings for phecodes, we use the ones provided in the MIKGI dataset described in the third paragraph.

Reviewer comment (Theory):
The methods used in this paper are too heuristic. In the case of using neural networks, deep neural network models can be interpreted as NTK using some heuristics, while in order to train the contrastive loss using kernel learning, the paper performs some relaxations in the objective function.

Author response:
We appreciate the reviewer for bringing up this concern. We acknowledge that there is a certain degree of relaxation between the original loss and the objective we have derived. However, we argue that such relaxation is necessary to statistically characterize the complexity and learning rate of the model, as presented in Section 6.

Reviewer comment (Experiments):
The experimental section can be improved by adding more datasets and baselines.

Author response:
We appreciate your suggestion on improving the numerical evaluation. However, we would like to emphasize that this work mainly focuses on a theoretical interpretation of CLIP rather than proposing a new method. As CLIP has demonstrated excellent generalizability in broader domains, we believe our work can be extended to various tasks as CLIP. To further illustrate the performance of the neural network-based CLIP, we will conduct some new experiments to further showcase the empirical performance. We hope the additional results (to be uploaded) resolve your concerns.

Reviewer comment (Experiments):
It is not clear about the focus of the paper. While the primary motivation has been understanding CLIP, the latter part of the paper doesn't offer any discussions, neither there are any experimental comparisons to that of CLIP.

Author response:
Thank you for the question. The main focus of the paper is to provide an alternative method to solve the CLIP loss and thereby some theoretical understanding of CLIP from the RKHS perspective instead of the neural network. We acknowledge that we focus more on the theoretical analysis of MTMK in the latter parts while the discussions are mainly offered in Section 2. The reason why we organized in this fashion is that we believe once the equivalence between standard CLIP and our proposed method is built, the theoretical analysis (Section 6) and experiments (Section 7) can be applied to CLIP from a theoretical viewpoint. In the revised version, we will include a more comprehensive discussion in later sections.

Reviewer comment (Writing):
The notation itself is a bit confusing to understand. While CLIP uses pseudolabels, which have n different values for n samples, this paper has a notation T for the classes and n for the samples. It would be easier to have a clear writeup on the dataset construction before proceeding to the next equivalances.

Author response:
Thank you for the comment. The $T$ used in this paper refers to the classes that appeared in the whole dataset for ease of analysis of the learning rate. In the revised version, we will include further clarification with regard to the dataset construction.

Reviewer comment (Evaluation):
The experimental section is not in line with the initial claims of the paper.

Author response:
Thanks for the question. To further illustrate the performance of the neural network-based CLIP, we conducted some new experiments to showcase the empirical performance. We hope the additional results in the official comments resolve your concerns.

Reviewer comment (Theory):
What happens if we assume that the CLIP objective (2.1) has dot(\phi_I(x_i), \phi_t(x_j)) corresponding to the image and text embeddings. Are the results discussed in this paper hold without any loss of generality?

Author response:
The results hold without any loss of generality as we can directly write $\phi(x) = \mathbb{I}\\{x ~ \text{is an image}\\}\phi_I(x) + \mathbb{I}\\{x ~ \text{is a text}\\}\phi_t(x)$.

Reviewer comment (Writing):
In 2.7, is y_ti one hot encoding of the ith sample?

Author response:
We apologize for the confusion. In line with $y_i$, $y_{ti} \in \\{+1, -1\\}.$

Reviewer comment (Experiments):
In the synthetic experiments, more explanation might be needed on the ground truth function. And some explanation would be needed why the MTMK-L1 worked best in relation to the true model.

Author response:
Thank you the question. In the synthetic experiments, the detailed task configuration is provided in Appendix C, where each ground truth function can be represented via a sparse combination of the candidate kernels (e.g., in the task set 1, the truth is a combination of polynomials of order 1, 3 and 5). Thus, the candidate kernels consist of ones with a positive effect and ones with a negative effect and we need to correctly select the useful kernels. Our algorithm is specifically designed for this scenario and is capable of selecting the best kernels with the L1 penalty. Furthermore, we provide a theoretical guarantee to depict how our proposed method can correctly select the kernels.

Reviewer comment (Evaluation):
Since the model is motivated from CLIP, how do we compare against tasks which are used for evaluating CLIP?

Author response:
Thank you for the question. Our method can mainly be used on classification tasks aligned with CLIP. However, we would like to emphasize that this work mainly focuses on a theoretical interpretation of CLIP rather than proposing a new method.