Paper: Is Training Necessary for Representation Learning (ICLR-2024)

Reviewer comment (Experiments):
Another possible limitation relates to the fact that the method doesn't have any learnable parameters. Often there is a domain shift between the dataset for learning representations and the dataset of interest, or between the training set and test set. Standard NN-based approaches can work well in such cases (depending on the magnitude of the shift) or can be adjusted to these shifts by fine-tuning the feature extractor, for example. Yet, as the proposed method heavily relied on the algorithm for obtaining a multiscale mesh based on the training set, it is not clear how it will work in such cases.

Author response:
1. Our LagrangeEmbedding is a universal encoder that can be applied to multiple tasks without fine-tuning. In contrast with standard transfer learning, LagrangeEmbedding is train-free, ensuring lossless memory and avoiding catastrophic forgetting. Admittedly, due to its generality, LagrangeEmbedding-based networks may not outperform task-specific neural networks in terms of accuracy for a particular recognition task. However, our models adhere to the **error-bound formula**, meaning that we can reduce this performance gap by increasing the scale of LagrangeEmbedding.
2. Our algorithm for generating multiscale meshes is designed to ensure that LagrangeEmbedding-based networks satisfy the **error-bound formula**. This algorithm determines the size of LagrangeEmbedding and the shape of its Lagrange basis functions.

Reviewer comment (Novelty):
I expected to see a broader reference to kernel methods, but the paper seems to miss this related research direction entirely. Kernels also form a basis function and are also universal approximators, and perhaps there is some connection to the Lagrange basis function. More importantly, I find two studies particularly relevant (to address and compare to). First, the line of research on infinite-width NNs [1, 2] which also hinges on inner products in the input space. Second, the method presented in [3] also suggests to use simplices for approximating the full data kernel. When is the proposed approach preferred over these modeling choices?

Author response:
Thank you for your comments. We have added Appendix F to our paper to discuss the differences between LagrangeEmbedding and recommended papers [1][2][3]. Our LagrangeEmbedding-based network and kernel methods are both linear models. However, their underlying principles are different: kernel methods map low-dimensional linearly inseparable data to high-dimensional and linearly separable data, whereas our method only maps weak-correlated data to be linearly separable. Specifically, if two input data are very close, the inner product of their projections will be close to 1, but if their similarity exceeds the threshold (i.e., longer than one simplex), their projections will be orthogonal. From Figure 1 of our paper:

$$\mathrm{Projection}(\frac{1}{3} (p^{(1)} + p^{(2)} + p^{(3)})) = [0, \frac{1}{3}, \frac{1}{3}, \frac{1}{3}, 0, 0, 0, 0]$$

$$\mathrm{Projection}(\frac{1}{3} (p^{(2)} + p^{(3)} + p^{(4)})) = [0, 0, \frac{1}{3}, \frac{1}{3}, \frac{1}{3}, 0, 0, 0]$$

$$\mathrm{Projection}(\frac{1}{3} (p^{(0)} + p^{(5)} + p^{(6)})) = [\frac{1}{3}, 0, 0, 0, 0, \frac{1}{3}, \frac{1}{3}, 0]$$

The first point $\frac{1}{3} (p^{(1)} + p^{(2)} + p^{(3)}) \in \bigtriangleup p^{(1)}p^{(2)}p^{(3)}$ and the second point $\frac{1}{3} (p^{(2)} + p^{(3)} + p^{(4)}) \in \bigtriangleup p^{(2)}p^{(3)}p^{(4)}$ belong to adjacent triangles, so their LagrangeEmbedding projections are close, and the inner product is close to 1. However, the first point $\frac{1}{3} (p^{(1)} + p^{(2)} + p^{(3)}) \in \bigtriangleup p^{(1)}p^{(2)}p^{(3)}$ and the third point $\frac{1}{3} (p^{(0)} + p^{(5)} + p^{(6)}) \in \bigtriangleup p^{(0)}p^{(5)}p^{(6)}$ belong to distant triangles, so their projections are orthogonal.

LagrangeEmbedding has two characteristics:
1. The inner product of two data in the input space does not necessarily correlate with the inner product in the projection space. In the projection space, their inner product is solely determined by the multi-scale mesh, and the mesh structure is determined by learning the data distribution in the input space. Therefore, for LagrangeEmbedding, two data with inner products close to 1 in the input space may potentially be orthogonal in the projection space.
2. The LagrangeEmbedding-based network can have unlimited width if we continuously increase its degrees of freedom.

Finally, LagrangeEmbedding has a cost-efficient solution for extracting features from high-dimensional raw data (see W1, Part 1, **Low-Rank LagrangeEmbedding**). Furthermore, LagrangeEmbedding can be used independently or as a parameter-free module inserted into a neural network to enhance the performance (see RtZ7-W1 **Table R2-1**). Therefore, we believe our approach holds advantages over [1][2][3].

Reviewer comment (Theory):
A reference (or a proof) for the two properties of the Lagrange basis function in Section 2 is missing.

Author response:
1. We have now included relevant references in the revised paper, such as [6]. The property "Universal Approximation" is from approximation and linear interpolation theory, while the "Similarity Calculation" property is based on the theory of first-order linear finite elements.
2. The dimensionality reduction tool (the untrainable preprocessing layer) in Section 3.2.1 is not well-designed. Replacing it with random linear weighting for dimensionality reduction does not significantly impact the experimental results. Therefore, in Section 4, we stated that our future research direction focuses on constructing a better dimensionality reduction tool that compresses information without loss.
3. This is a conclusion from FEM. When $d=1,2$, the formula can be derived from the standard uniform mesh (see https://en.wikipedia.org/wiki/Triangular_tiling). In higher dimensions, it can be proven via mathematical induction.

Reviewer comment (Writing):
Intuition on the dimensionality reduction technique in Section 3.2.1 is missing.

Author response:
1. We have now included relevant references in the revised paper, such as [6]. The property "Universal Approximation" is from approximation and linear interpolation theory, while the "Similarity Calculation" property is based on the theory of first-order linear finite elements.
2. The dimensionality reduction tool (the untrainable preprocessing layer) in Section 3.2.1 is not well-designed. Replacing it with random linear weighting for dimensionality reduction does not significantly impact the experimental results. Therefore, in Section 4, we stated that our future research direction focuses on constructing a better dimensionality reduction tool that compresses information without loss.
3. This is a conclusion from FEM. When $d=1,2$, the formula can be derived from the standard uniform mesh (see https://en.wikipedia.org/wiki/Triangular_tiling). In higher dimensions, it can be proven via mathematical induction.

Reviewer comment (Theory):
An explanation is missing for why the following is true: $(n_t/d!)^{1/d} = \mathcal{O}(h^{-1})$.

Author response:
1. We have now included relevant references in the revised paper, such as [6]. The property "Universal Approximation" is from approximation and linear interpolation theory, while the "Similarity Calculation" property is based on the theory of first-order linear finite elements.
2. The dimensionality reduction tool (the untrainable preprocessing layer) in Section 3.2.1 is not well-designed. Replacing it with random linear weighting for dimensionality reduction does not significantly impact the experimental results. Therefore, in Section 4, we stated that our future research direction focuses on constructing a better dimensionality reduction tool that compresses information without loss.
3. This is a conclusion from FEM. When $d=1,2$, the formula can be derived from the standard uniform mesh (see https://en.wikipedia.org/wiki/Triangular_tiling). In higher dimensions, it can be proven via mathematical induction.

Reviewer comment (Writing):
At the beginning of Section 2.1, what is $m$? Did you mean $N-1$?

Author response:
Thank you for correcting me. A clear statement would be: ``For any given simplex, select a subset $\{ (x^{(k_0)}, y^{(k_0)}), \cdots, (x^{(k_{m'-1})}, y^{(k_{m'-1})}) \}$ from the training set $\{ (x^{(0)}, y^{(0)}), \cdots, (x^{(m-1)}, y^{(m-1)}) \}$ where $m'$ is the cardinality of subset, $m$ is the cardinality of subset, and all subset elements reside within the given simplex."

Reviewer comment (Theory):
Perhaps I didn't understand something, but to me the definition of the Lagrange function $\mathcal{L}_i$ seems to be the indicator function and not a linear function peaking at $p^{(i)}$.

Author response:
The Lagrange basis function is globally continuous and piecewise linear, not an indicator function. Please refer to Figure 6 (Middle) in Appendix B, It's a function image demo for 2D Lagrange function $L_7$ and $L_{20}$. Note: unlike some kernel methods, the support of Lagrange bais are not isolated from each other.

Reviewer comment (Experiments):
The paper over-promises and under-delivers. Among other broad claims - e.g. first paragraph of the conclusion, “unparalleled level of explainability” - the title itself “Is training necessary for representation learning” suggests that the proposed method can be comparable to training-based approaches such as neural networks. Yet, there exists few, if any, quantitative comparisons between the proposed method and neural network approaches, especially for the (toy) computer vision and NLP experiments.

Author response:
1. Thank you for your feedback. We place a strong emphasis on the explainability of our LagrangeEmbedding. As we demonstrated in Section 2 & 3.1.3, LagrangeEmbedding-based models strictly follow the complete FEM theory due to adhering to the **error-bound formula**:
$$\frac{1}{m} \sum_{i=0}^{m-1} |f(x^{(i)};\theta)-y^{(i)}|^2 = O(h) = O(n_t^{-1/d}) = O(n^{-1/d})$$
For $d$-dimensional inputs and MSE loss function, this formula shows that when we double the parameters counts of the last linear layer, the training error $MSE$ of LagrangeEmbedding-based models will get reduced to $2^{-1/d}*MSE$. **Therefore, we can continuously improve the model performance by increasing the number of parameters in the linear layer.** 

2. In Sections 3 & 5, when we mentioned that "LagrangeEmbedding can be comparable to neural network-based encoders," we meant that "the LagrangeEmbedding-based network performs equally well or even better than models of similar scale." To avoid any misunderstanding, we have included additional experiments for image classification tasks:

**Table R2 - 1**: Ablation studies. We present the experimental results for image classification tasks conducted on MNIST, FashionMNIST, CIFAR-10, and CIFAR-100 datasets. The model instances used in these experiments are obtained from the PyTorch tutorial page (https://github.com/pytorch/examples/tree/main/mnist). When the "LagrangeEmbed" column is marked as "False," it indicates that we trained the entire CNN. When "LagrangeEmbed" is marked as "True," we freeze the CNN (excluding the last linear layer) as a preprocessing layer and insert our parameter-free LagrangeEmbedding before the final linear layer, then only train the final linear layer. The "data_aug=True" signifies using random horizontal flip as a data augmentation technique.
The "init time" column reports the wall-clock time taken for initializing our LagrangeEmbedding on an RTX 2080ti GPU. All training configurations are taken from the source PyTorch tutorial page, and all experiments can be reproduced by running `bash run.sh` in our revised supplementary material.
| dataset  | LagrangeEmbed | data aug | test acc | init time |
| -------- | ------------- | -------- | -------- | --------- |
| minist       | False     | False    | 99.14%   | N/A       |
| minist       | True      | False    | 99.17%   | 1.55s     |
| fashionmnist | False     | False    | 92.47%   | N/A       |
| fashionmnist | True      | False    | 92.56%   | 1.55s     |
| fashionmnist | False     | True     | 92.14%   | N/A       |
| fashionmnist | True      | True     | 92.14%   | 2.08s     |
| cifar10      | False     | False    | 71.43%   | N/A       |
| cifar10      | True      | False    | 71.54%   | 1.83s     |
| cifar10      | False     | True     | 71.93%   | N/A       |
| cifar10      | True      | True     | 72.09%   | 2.29s     |
| cifar100     | False     | False    | 36.50%   | N/A       |
| cifar100     | True      | False    | 36.78%   | 1.84s     |
| cifar100     | False     | True     | 36.58%   | N/A       |
| cifar100     | True      | True     | 36.97%   | 2.30s     |

**Table R2-1** shows that LagrangeEmbedding can be inserted as a parameter-free module into a frozen model, where the improved model size remains almost unchanged, yet achieves better model performance. For text classification tasks, our LagrangeEmbedding-based network in Section 3.3 contains only 256 parameters but performs on par with the 6.13 million-parameter word2vec model.

Reviewer comment (Experiments):
The basic 2-layer convolutional network, for instance taken from the PyTorch tutorial page (https://github.com/pytorch/examples/tree/main/mnist), already achieves 98% accuracy on MNIST in the first epoch (outperforming the proposed approach), which completes in under a minute on a CPU and presumably orders of magnitude faster on GPU.

Author response:
Thank you to the reviewer for providing the model instance (https://github.com/pytorch/examples/tree/main/mnist). We checked it and found it has 11 times more parameters than our model used in Section 3.2. To evaluate the Lagrange-based network fairly, we revised the implementation of our multi-scale domain decomposition method to run on GPU and conducted the experiments as shown in **Table R2-1**.

In these experiments, the PyTorch model instance has the same size as the LagrangeEmbedding-based network. However, while the former trains the entire model, the latter only trains the final linear layer. Notably, the latter performs equal time efficiency during inference, and outperforms in evaluation results. Although the LagrangeEmbedding-based network requires initialization before use, its actual initialization time is minimal.

Reviewer comment (Reproducibility):
Section 3.1.2 compares against neural networks when fitting distributions drawn from 2-dimensional distributions, but it is not stated what network parameters nor training parameters are used other than the fact that it is a MLP.

Author response:
Thank you for your feedback. The neural network we described in Figure 3 is a 5-layer MLP using the ReLU activation function. No matter how we increase the number of parameters of this MLP or tune the training parameter configuration, the MLP cannot fit the high-frequency area in dataset $\mathbb{C}$ well. This experimental phenomenon is consistent with the description in the paper [1].

Reviewer comment (Experiments):
In Table 1, Random Forest is already highly effective at achieving almost perfect R^2 scores, and performance on most of the distributions considered appears to have already saturated.

Author response:
Yes. For data fitting tasks, random forests exhibit similar performance to our LagrangeEmbedding-based network and overcome the challenge of fitting multi-frequency data. However, unlike our LagrangeEmbedding-based network, in most cases, random forests are not suitable for image and text-related recognition tasks.

Reviewer comment (Theory):
How did the projection layer in Section 3.2.1 arise? There is no explanation for why this specific projection equation was introduced, and while it claims to contain “no trainable model parameters”, it appears to require careful hand-crafting as well.

Author response:
We appreciate the feedback from the reviewer. The architecture of our LagrangeEmbedding-based network consists of a "preprocessing layer -> LagrangeEmbedding -> linear layer," where the static preprocessing layer serves as a dimensionality reduction tool, and the train-free LagrangeEmbedding acts as the projection layer.

As we explained in the **Remark** of Section 3.2.1, the dimensionality reduction tool (the untrainable preprocessing layer) in Section 3.2.1 wasn't well-designed. Replacing it with random linear weighting for dimensionality reduction had minimal impact on the experimental results. To prevent any misunderstanding and highlight the advantages of LagrangeEmbedding, please refer to the preceding **Table R2-1**.

Reviewer comment (Experiments):
Speed is touted as an advantage of the method, but there exists no wall-clock timing comparisons for computing the proposed embedding.

Author response:
We have updated our supplementary materials, and the multi-scale domain decomposition method can now be executed on GPU. In terms of image classification tasks, our model demonstrates similar time efficiency compared to other networks (refer to **Table R2-1** for details. To reproduce these results, please run `bash run.sh`).

For text classification tasks, our model achieves faster performance, with an epoch taking 74.99 seconds (online training on RTX 2080Ti), compared to the word2vec model (https://pytorch.org/tutorials/beginner/text_sentiment_ngrams_tutorial.html), which requires 91.24 seconds per epoch (online training on RTX 2080Ti). To reproduce these results, please run `textv2.py` and `textv2_comparion.py`.

Reviewer comment (Writing):
Equation (2) $y^{(j)}$ should be $y^{(i)}$ instead.

Author response:
Thank you for reminding us of the typo!

Reviewer comment (Experiments):
In Section 3.3, does “the neural network” refer to the proposed method (i.e. typo)? If not, are there quantitative results and comparisons for the proposed method?

Author response:
It is not a typo. Here, "the neural network" refers to our LagrangeEmbedding-based network. **Our LagrangeEmbedding has only one hyperparameter: the degrees of freedom (dof).** When we set the dof to 64, the test accuracy on AG news dataset exceeds 90%. For lower dof, the test accuracy will decrease.

Reviewer comment (Writing):
In Appendix D.2., the phrase “Remarkably, after just 4 epochs of training, the neural network outputs close approximate the target values” is assumed to be a typo.

Author response:
This is not a typo. The meaning of this sentence is that the LagrangeEmbedding-based network converged after the first epoch training and did not obtain more test accuracy in the following four epochs.

Reviewer comment (Reproducibility):
In Section 3.3 - can you elaborate on how the pre-processing layer is implemented?

Author response:
Yes, from our supplementary materials, the pre-processing layer outputs the TF [2] value where
$$\\mathrm{TF} = \\frac{\\mathrm{number\\ of\\ times\\ the\\ term\\ appears\\ in\\ the\\ document}}{\\mathrm{total\\ number\\ of\\ terms\\ appears\\ in\\ the\\document}}$$

Reviewer comment (Experiments):
The proposed method seems to only work on some toy tasks.

Author response:
We sincerely appreciate the reviewer's feedback. We have now updated the GPU implementation of the multi-scale domain decomposition method in the supplementary materials. Additionally, for other datasets or more complex tasks, we have proposed several approaches and conducted further experiments to analyze their impact. The experimental results demonstrate that our LagrangeEmbedding is also fast and effective for other datasets (Please refer to our first response to YmUG-W1 and RtZ7-W1 for details). 

Due to the limited time of the rebuttal phase, we have not yet tuned the training hyper-parameters on ImageNet and COCO-related experiments. Once we obtain SOTA results, we will promptly provide the corresponding code in our open-source project and publish it as a new paper. We again express our gratitude for your valuable feedback and look forward to your further review.

Reviewer comment (Novelty):
The related work sections or some closely related works are not provided.

Author response:
Thank you for the reviewer's reminder. Our LagrangeEmbedding encoder is original. Unlike other other FEM-related models, it is an explicit implementation of FEM rather than an imitation, and it is the first model that satisfies the FEM **error-bound formula**. Then, our approach differs from kernel methods, it maps two raw data to orthogonal vectors only when the two raw data have significant differences. We have included Appendix F in the revised paper to discuss the distinctions between LagrangeEmbedding-based networks and kernel methods.

Reviewer comment (Experiments):
The proposed method is only validated on simple image and text datasets. The comparison results with SOTA methods are not given. Even the proposed method achieves lower accuracy than SOTA methods. The comparison experiment with SOTA methods can assist the reader in finding the gap between the proposed and SOTA methods.

Author response:
Thank you for your feedback. Admittedly, due to the train-free property and generality of LagrangeEmbedding, LagrangeEmbedding-based networks may not outperform task-specific neural networks in terms of accuracy for a particular recognition task. However, our LagrangeEmbedding, can be inserted into any frozen neural network as a plug-and-play parameter-free module (Please refer to **Table R2-1** in our response to RtZ7-W1). It barely changes the model scale, only requires training the final linear layer, but enhances the model's performance. These experiments indicate that LagrangeEmbedding has the potential to help models achieve SOTA results.

Reviewer comment (Experiments):
The proposed method only runs in a non-parallel manner, as mentioned in the future directions section.

Author response:
We have recently upgraded the GPU implementation of the multi-scale domain decomposition method in the revised supplementary materials. It is available to run the whole data pipeline of LagrangeEmbedding-based networks in GPU parallel mode.

Reviewer comment (Experiments):
How can we extend the proposed method for complex tasks in actual situations? The author is suggested to add some discussion.

Author response:
If users have sufficient computing resources, they can address this issue by increasing the scale of LagrangeEmbedding. As LagrangeEmbedding-based networks adhere to the error-bound formula, **we can continuously improve the model performance by increasing the number of parameters in the linear layer**. For users with limited computational resources, we can consider the solution described in the previous question by partitioning the high-dimensional input space to reduce computational costs.

Reviewer comment (Novelty):
I have not seen the author mention some closely related works. Is the proposed method totally original? If not, please provide the detailed related works and the difference between the proposed method and the related works.

Author response:
Yes, our multi-scale domain decomposition method (DDM) and the construction method of the Lagrange basis are totally original. Specifically, we construct the LagrangeEmbedding architecture by using Eqn (3) and initialize LagrangeEmbedding size via the multi-scale domain decomposition method. Unlike existing FEM-related neural networks [1][2], we have implemented FEM explicitly, rather than imitating it. The evidence lies in the fact that our model adheres to the **error-bound formula**.

Reviewer comment (Writing):
In section 2.1, the definition of m in x^{(m)} is not given. What's the difference bettween the x^{(N-1)} and x^{(m)}?

Author response:
Thanks to the reviewer's reminder. Here, $m$ is the cardinality of the training set, and $N$ is the cardinality of the subset. A clear statement would be: ``For any given simplex, select a subset $\{ (x^{(k_0)}, y^{(k_0)}), \cdots, (x^{(k_{m'-1})}, y^{(k_{m'-1})}) \}$ from the training set $\{ (x^{(0)}, y^{(0)}), \cdots, (x^{(m-1)}, y^{(m-1)}) \}$ where $m'$ is the cardinality of subset, $m$ is the cardinality of subset, and all subset elements reside within the given simplex."

Reviewer comment (Writing):
In Eqn(2), the ‘i’ is suggested to be replaced with 'n'.

Author response:
We want to express our gratitude for the reviewer's valuable suggestions. We have incorporated these recommendations into the content of the newly revised paper.

Reviewer comment (Writing):
”given function F (x) to be fitted”  -> ``given function F (x) to be fitted”.

Author response:
We want to express our gratitude for the reviewer's valuable suggestions. We have incorporated these recommendations into the content of the newly revised paper.