Paper: TABLEYE: SEEING SMALL TABLES THROUGH THE LENS OF IMAGES (ICLR-2024)

Reviewer comment (Novelty):
Only two papers are discussed in the related work section, which makes reader difficult to place the paper in an appropriate context.

Author response:
- It seems we misunderstood your question earlier. We have revised the content of the related work section to reflect your opinion. The following content has been added to the related work section, providing a general explanation of tabular learning, few-shot learning, and few-shot tabular learning. Additionally, we have adjusted the descriptions of STUNT and TabLLM in the related work.

*→  Tabular learning refers to the process of learning the mapping between input and output data using tabular data[1].
Tabular data is often also called structured data[2] and is a subset of heterogeneous data presented in a table format with rows and columns. Each feature in this data is composed of either categorical or numerical features.
Currently, methods based on decision trees and those based on Multi-Layer Perceptrons (MLP) are showing almost equal performance.
Tabular learning still requires a large amount of labeled data.
In the image domain, few-shot learning can easily acquire prior knowledge using many related images. For example, ProtoNet (Prototypical Network)\citep{snell2017prototypical} learns using similarities between images, and MAML (Model-Agnostic Meta-Learning)\citep{finn2017model} quickly adjusts the model across various tasks, enabling rapid learning with limited data. 
However, in the tabular domain, there are no equivalent sets of related tabular data. Therefore, few-shot tabular learning faces significant challenges in forming prior knowledge.
Therefore, the current state-of-the-art (SOTA) methods for few-shot tabular learning utilize semi-few-shot learning approaches using unlabeled data samples or transfer tabular data to the text domain and employ Large Language Models.*

[1] Borisov, V., Leemann, T., Seßler, K., Haug, J., Pawelczyk, M., & Kasneci, G. (2022). Deep neural networks and tabular data: A survey. IEEE Transactions on Neural Networks and Learning Systems.

[2] Ryan, M. (2020). Deep learning with structured data. Simon and Schuster.

Reviewer comment (Experiments):
The relationship between the quality of the visual encoder and the few-shot tabular performance is not shown.

Author response:
- We believe there may have been a mistake in the experimental setup of our experiment using CLIP. We agree with your opinion that using a powerful visual encoder, including CLIP, can positively influence the performance of TablEye. However, we lack the time to conduct further experiments from multiple perspectives at this moment. We will proceed with additional experiments in the future and report the results.

Reviewer comment (Novelty):
Can you discuss more related works in the paper? For example, a brief introduction to the tabular learning literature.

Author response:
- Thank you for your question. I realize that I did not consider that readers might be unfamiliar with the concept of few-shot learning. For those who are completely new to few-shot learning, We have added a background section in Appendix A on page 12 of the paper to provide essential context and understanding.

Reviewer comment (Experiments):
Can you give results using a more powerful visual encoder? In Luo et. al [1], it has been shown that better visual encoders can lead to better few-shot learning performance. Perhaps, you can try to use pretrained CLIP [2] or DINO-v2 [3] and report the results.

Author response:
- We conducted experiments using CLIP's ViT-B/32 encoder as the backbone and the Proto-layer as the classifier. Although we anticipated the excellent few-shot performance of CLIP, the experimental results showed that CLIP did not learn effectively on any of the four datasets (CMC, Diabetes, Karhunen, Optdigits). We believe this is because CLIP's pre-trained visual encoder is designed to reduce the distance between the image and text domains, which is a different concept compared to backbones like conv2, conv3, which were trained specifically for natural image few-shot classification.

- Therefore, we also conducted experiments using the ViT as the backbone and the Proto-layer as the classifier. The results showed that ViT performed reasonably well on all four datasets but did not surpass the performance of other methods used in TablEye. We attribute this to the fact that while the learning parameters of the current models have been optimized through numerous experiments, ViT has not undergone such optimization. However, this also indicates the potential applicability of the ViT structure to tabular images. We have added these experimental results to Appendix H on page 16 of the paper.

Reviewer comment (Experiments):
Another straight way of transforming the tabular data into images is to directly visualize the table on an image in its original form. This should be a baseline to illustrate the advantage of your proposed tabular data transformation.

Author response:
- Thank you for mentioning an important baseline. We conducted additional experiments with two methods, including the baseline you mentioned, and compared them with domain transformation. The graph below experimentally demonstrates that domain transformation more effectively converts tabular data into a form suitable for CNNs compared to simply changing the order and size of tabular data to form a 2D image and representing it graphically. We have added a section in Appendix G on page 13, 15 of the paper to provide this experiment. Thank you for providing important comments that helped improve the paper.

Reviewer comment (Theory):
Creating the (N,N) feature matrix R via Euclidean distance between N features lacks clear intuition. Given that Euclidean distance is symmetric, the (N,N) matrix should also be symmetric (if unranked) or have double-entries (if sorted/ranked), which is not addressed.

Author response:
First, we have incorporated comments by adding detailed explanations to Section 3.2, "Domain Transformation," on page 4 and to "Algorithm 1" on page 14 of the paper. As described in the paper, we aimed to create an (n, n) feature matrix to intuitively represent the similarities among features of tabular data, similar to an adjacency matrix. Therefore, as you have understood, the (n, n) matrix is a symmetric matrix.

Reviewer comment (Writing):
The process of generating an (N,N) pixel matrix Q by measuring distance and rank between N elements is unclear, including what exactly is being compared/contrasted and what ‘pixels’ are used in this context.

Author response:
Thank you for providing a detailed question. I recognize that the wording I used in that section did not clearly convey my intent, and we have revised section 3.2 on page 4 of the paper accordingly. As explained in the revised paper, when tabular data with n features is directly converted into a 2-dimensional image form, each pixel represents one of the n features. Therefore, the feature matrix calculates the distances between the coordinates of these n pixels. For instance, if there is tabular data where n=6=2*3, converting this data into a 2-dimensional image form would assign the 6 features coordinates like (0,0), (0,1), (0,2), (1,0), (1,1), (1,2). The feature matrix then calculates the distances between these coordinates, as illustrated in the figure.

|   |   |   |   |   |   |
|---|---|---|---|---|---|
| 0 | 1 | 2 | 1 | √2 | √5 |
| 1 | 0 | 1 | √2 | 1  | √2 |
| 2 | 1 | 0 | √5 | √2 | 1  |
| 1 | √2 | √5 | 0 | 1  | 2  |
| √2 | 1 | √2 | 1 | 0  | 1  |
| √5 | √2 | 1 | 2 | 1  | 0  |

Reviewer comment (Theory):
The subsequent step of computing another Euclidean distance between R and Q is missing clear intuition or justification.

Author response:
The feature matrix R represents the similarity between features when tabular data is simply reshaped into two dimensions. Q, on the other hand, signifies the distances between coordinates in an image of the same shape. Therefore, Q is a matrix of an image with the natural characteristic (spatial relation) where similar values are located near each other. Hence, we hypothesized that by calculating and reducing the distance between R and Q, we could impart spatial relations onto R. Indeed, our experiments confirmed that altering the order of features in R to minimize its distance from Q contributed to performance improvements in few-shot learning. The experimental results show that the method of directly converting tabular data into tabular images, termed 'direct', does not involve the process of reducing the distance between R and Q. It becomes evident that adding spatial relations indeed aids in enhancing performance.

If you look at the additional experiments added to Appendix G on page 16 of the paper, you will find a comparison of the performance of three methods: a direct method that manipulates shapes in 2D without considering spatial relations, a Graph method that represents the shapes as graphs, and our domain transformation method. The experimental results demonstrate that the presence or absence of spatial relations significantly impacts the performance of few-shot learning.

Reviewer comment (Theory):
The claim that the process results in “a 2-dimensional image of size (Nr x Nc)” is unclear, specifically how this image is obtained from computing the Euclidean distance between two (N,N) matrices.

Author response:
To provide a simple example, let's assume feature matrix R initially represents the similarity between (f1, f2, f3, f4) and (f1, f2, f3, f4), where f1, f2, f3, f4 are arbitrary features. If Nr=2 and Nc=2, the image formed would be [[f1, f2], [f3, f4]]. However, if the order of the features is changed, it can be seen as altering the similarity to that between (f3, f1, f4, f2) and (f3, f1, f4, f2). In this case, the image would become [[f3, f1], [f4, f2]]. This process is detailed in Algorithm 1 on page 14 of the paper.

Reviewer comment (Reproducibility):
The arrangement of ranked features to form a 2D ‘image’ is not described in sufficient detail, despite its potential significant effect on ConvNet performance.

Author response:
The method of forming a 2D image using ranked features is explained in conjunction with Q4. Additionally, more detailed information can be found in section 3.2 on page 4 of the paper and in Algorithm 1 on page 14.

Reviewer comment (Theory):
The rationale behind why a ranking of distances between features and pixels, followed by rearrangement, would resemble information in natural images is not sufficiently explained. The relationship between local pixel arrangements in natural images and the artificially created tabular images is unclear, and the approach to overcome the potential domain gap between them is not addressed satisfactorily. While Figure 4 provides some insight, the use of a 2D circle in t-SNE may not be representative due to projection hyperparameters.

Author response:
As you mentioned, pixels in natural images that are locally related tend to form objects of varying sizes and often share similar values. We believed that our domain transformation method could bridge this gap. For example, if you flatten a natural image to create n features and calculate the similarity among these n features, you can form an (n, n) matrix M. (M_ij represents the similarity between the i-th and j-th features, and M is symmetric.) Visualizing this normalized matrix M would likely show that areas in close proximity have high similarity, resulting in a natural appearance. This is similar to the pixel matrix Q generated from coordinates in domain transformation.

Therefore, by altering R to closely resemble Q through domain transformation, we were able to transform tabular data into a format that shares some characteristics with natural images. While still distinguishable to the human eye, from the perspective of a CNN, tabular images and natural images exhibit similar patterns. This finding underscores the adaptability of CNNs in recognizing patterns across different data modalities, and the potential of domain transformation in bridging the gap between disparate data domains.

Reviewer comment (Experiments):
Since common CNNs take in RGB images but the authors create only single-channel images, they repeat the same image three times for each channel. This appears to be unnecessary overhead engineered to fit existing input layers. If created images are grayscale, it might be more reasonable to pretrain the backbone on grayscale images.

Author response:
Thank you for the insightful question. Indeed, stacking 1 channel to create a 3-channel image does introduce computational overhead for image processing. However, the reason we experimented with 3-channel images is that we wanted to leverage as much information as possible in the process of learning prior knowledge from large image datasets. The table below shows the experimental results of learning prior knowledge from 1-channel grayscale images and performing prediction tasks on 1-channel tabular images. '1-ch' refers to 1-channel, and '3-ch' refers to 3-channel. The results indicate that 3-channel generally performed better in most cases. While there are a few instances where 1-channel closely matches or slightly outperforms 3-channel, there is a significant variation in performance in general scenarios. The reason for the superior performance of 3-channel images is not yet clear. We plan to explore and share more detailed reasons in future research. Additional experiments have been included on pages 14 and 15 of the paper.

Reviewer comment (Reproducibility):
The introduction mentions that features in tabular data have independent distributions and ranges, and that missing values may be present. Neglecting missing values, it is unclear how the challenge of differing ranges is addressed, especially since Euclidean distances between features can vary greatly if ranges differ. The method of converting these values into image pixels in the fixed [0, 255] per channel range is not explained.

Author response:
We performed min-max normalization on all tabular data to scale the values between 0 and 1. Then, we applied the domain transformation algorithm and finally multiplied all values by 255 to achieve an integer value in range between 0 and 255.

Reviewer comment (Experiments):
For Table 1, which aims to demonstrate the benefit of “Prior Knowledge Learnt from the image domain,” the exact experimental setting is unclear. It is not specified whether experiments without image-pretraining are trained solely on the tabular images, whether a randomly initialized backbone is used, or whether image-pretrained methods are further fine-tuned on some tabular image data. This information is necessary for judging the extent of information transfer and the risk of overfitting.

Author response:
Thank you for pointing out the unclear part of the experimental setup. In the experiment of Table 1, the "no img" condition refers to using a randomly initialized backbone to train only on the tabular images provided in the support set. Additionally, fine-tuning methods were not applied in any of the experiments. This information has been added to the description of Table 1 on page 7 of the paper.

*Before*: ‘No Img’ represents the condition where no prior knowledge learning has occurred in the image domain, whereas ‘Img’ denotes cases where prior knowledge has been acquired using mini-ImageNet.

*After*: ‘No Img’ represents the condition where no prior knowledge learning has occurred in the image domain. Randomly initialized backbone is applied and trained on a tabular image. ‘Img’ denotes cases where prior knowledge has been acquired using mini-ImageNet

Reviewer comment (Evaluation):
While the paper presents positive results, it lacks detailed discussions regarding the differences in dataset performances. A more in-depth analysis of why certain structures perform better on specific datasets would provide a deeper understanding of TablEye's strengths. It seems that the improvement in performance is partly based on the variety of structures since none of them perform well in most of the experiments.

Author response:
**Q1. Could you please provide more details on the differences in performances across different datasets, particularly explaining why some structures perform better on specific datasets? This would help in understanding TablEye's strengths and limitations better. Could you explain why STUNT performs better in the dataset Karhunen? Similarly, why in some cases Conv2 is better than Conv4, for example, the datasets Optdigits and Karhunen?**

- Generally, TablEye demonstrates consistent high performance across numeric, categorical, and mixed values. TabLLM excels particularly with categorical values, while STUNT tends to perform exceptionally well with numeric values. In the case of STUNT, being a semi-supervised learning approach, the performance improvement was significantly noticeable over the baseline (XGB) when there was an abundance of unlabeled data and a higher number of features (collected information for prediction).

- A trend was observed where performance tended to reverse with an increasing number of features. This was not only evident in datasets like optdigits (64 features) and karhunen (64 features) but also in Cancer (32 features), Heart (12 features), and Lung (16 features). The authors speculate that this might be due to the training epochs being uniformly limited to 100. In simpler structures (small images), convergence up to conv4 leads to superior performance, whereas in more complex structures (large images), Conv3 or Conv2 tends to perform better. Taking into consideration the reviewer's comments, the authors agree on the necessity to experiment by increasing the number of training epochs and comparing accuracy, and plan to conduct and report this in the future.

Reviewer comment (Experiments):
The experiments of T-M-C2, T-M-C3, T-M-C4 on comparison with TabLLM and in the context of medical results are lacking.

Author response:
**Q2. Moreover, Could you please provide experiments on T-M-C2, T-M-C3, T-M-C4 on comparison with TabLLM and in the context of medical results?**

- We conducted experiments on T-M-C2, T-M-C3, and T-M-C4 with TabLLM and in the context of medical results These findings are reflected in Table 2 on page 7 and Table 7 on page 15 of the paper. The results showed that methods using the MAML-layer achieved higher numbers in some cases compared to previous figures.

Reviewer comment (Reproducibility):
Lack of Detailed Implementation: The paper offers an overview of the framework but lacks detailed implementation specifics. It would be better if you could present the structure of classifiers and also some equations or pseudo-code for all the parts of the model, especially the domain transformation.

Author response:
**Q4. Could you offer more detailed implementation specifics? The figures, equations, or pseudo-code can help understand each part of the model better, especially the domain transformation, which was kind of hard to understand at first. For the part about repeating the matrix, have you tried other methods like resizing or padding besides tilling when dealing with the matrix? Is tilling the best solution?**

- For providing detailed implementation insights, we have published pseudocode covering domain transformation. This content has been added to the Appendix D Algorithm section on pages 14 and 15 of the paper. We also enhanced the section 3.2 on domain transformation to facilitate a deeper understanding of the details among readers. Regarding the repetition of elements in the matrix, what we actually repeat are the elements contained within the matrix. Visually, this has an effect akin to magnifying a small image, which I believe is what you referred to as resizing. We initially experimented with filling a larger matrix with multiple smaller matrices. However, this approach resulted in very small edges and ambiguous boundaries, making it challenging for CNNs to extract meaningful features. Similarly, padding around the matrix didn't yield good results, especially when the base matrix size was very small, like (8,8). Among the intuitively conceivable methods, we believe the one we employed demonstrated the best performance.

Reviewer comment (Writing):
Additional Context for Few-Shot Learning: Providing a brief introduction to the general few-shot learning problem, its significance, and existing challenges would be beneficial for readers unfamiliar with the field.

Author response:
- Also, I agree with your opinion that some readers might not be familiar with few-shot learning in the process of understanding the classifier's algorithm. Therefore, we have added background on few-shot learning for readers who are not familiar with it in Appendix A on page 12 of the paper.

Reviewer comment (Evaluation):
Could you please provide more details on the differences in performances across different datasets, particularly explaining why some structures perform better on specific datasets? This would help in understanding TablEye's strengths and limitations better.

Author response:
**Q1. Could you please provide more details on the differences in performances across different datasets, particularly explaining why some structures perform better on specific datasets? This would help in understanding TablEye's strengths and limitations better. Could you explain why STUNT performs better in the dataset Karhunen? Similarly, why in some cases Conv2 is better than Conv4, for example, the datasets Optdigits and Karhunen?**

- Generally, TablEye demonstrates consistent high performance across numeric, categorical, and mixed values. TabLLM excels particularly with categorical values, while STUNT tends to perform exceptionally well with numeric values. In the case of STUNT, being a semi-supervised learning approach, the performance improvement was significantly noticeable over the baseline (XGB) when there was an abundance of unlabeled data and a higher number of features (collected information for prediction).

- A trend was observed where performance tended to reverse with an increasing number of features. This was not only evident in datasets like optdigits (64 features) and karhunen (64 features) but also in Cancer (32 features), Heart (12 features), and Lung (16 features). The authors speculate that this might be due to the training epochs being uniformly limited to 100. In simpler structures (small images), convergence up to conv4 leads to superior performance, whereas in more complex structures (large images), Conv3 or Conv2 tends to perform better. Taking into consideration the reviewer's comments, the authors agree on the necessity to experiment by increasing the number of training epochs and comparing accuracy, and plan to conduct and report this in the future.

Reviewer comment (Experiments):
Could you please provide experiments on T-M-C2, T-M-C3, T-M-C4 on comparison with TabLLM and in the context of medical results?

Author response:
**Q2. Moreover, Could you please provide experiments on T-M-C2, T-M-C3, T-M-C4 on comparison with TabLLM and in the context of medical results?**

- We conducted experiments on T-M-C2, T-M-C3, and T-M-C4 with TabLLM and in the context of medical results These findings are reflected in Table 2 on page 7 and Table 7 on page 15 of the paper. The results showed that methods using the MAML-layer achieved higher numbers in some cases compared to previous figures.

Reviewer comment (Experiments):
Are there any recommendations or guidelines for selecting the most appropriate structure when using datasets?

Author response:
**Q3. Are there any recommendations or guidelines for selecting the most appropriate structure when using datasets?**

The type of recommendation or guidance needed varies depending on the data and learning environment. If there are a few features, a complex structure (like conv4) might converge with a small amount of training, showing a high possibility of achieving high accuracy. However, if there are many features and training is minimal, a complex structure may not learn effectively, and a simpler structure could be more beneficial. If there are many features but sufficient resources for adequate training, we recommend without hesitation using a complex structure.

| Epoch Count / Number of Features | Few      | Many                                      |
|----------------------------------|----------|-------------------------------------------|
| Few                              | Complex  | Simple                                    |
| Many                             | Complex  | Unknown, but complex structure recommended |

Reviewer comment (Reproducibility):
Could you offer more detailed implementation specifics? The figures, equations, or pseudo-code can help understand each part of the model better, especially the domain transformation, which was kind of hard to understand at first.

Author response:
**Q4. Could you offer more detailed implementation specifics? The figures, equations, or pseudo-code can help understand each part of the model better, especially the domain transformation, which was kind of hard to understand at first. For the part about repeating the matrix, have you tried other methods like resizing or padding besides tilling when dealing with the matrix? Is tilling the best solution?**

- For providing detailed implementation insights, we have published pseudocode covering domain transformation. This content has been added to the Appendix D Algorithm section on pages 14 and 15 of the paper. We also enhanced the section 3.2 on domain transformation to facilitate a deeper understanding of the details among readers. Regarding the repetition of elements in the matrix, what we actually repeat are the elements contained within the matrix. Visually, this has an effect akin to magnifying a small image, which I believe is what you referred to as resizing. We initially experimented with filling a larger matrix with multiple smaller matrices. However, this approach resulted in very small edges and ambiguous boundaries, making it challenging for CNNs to extract meaningful features. Similarly, padding around the matrix didn't yield good results, especially when the base matrix size was very small, like (8,8). Among the intuitively conceivable methods, we believe the one we employed demonstrated the best performance.

Reviewer comment (Experiments):
For the part about repeating the matrix, have you tried other methods like resizing or padding besides tilling when dealing with the matrix? Is tilling the best solution?

Author response:
**Q4. Could you offer more detailed implementation specifics? The figures, equations, or pseudo-code can help understand each part of the model better, especially the domain transformation, which was kind of hard to understand at first. For the part about repeating the matrix, have you tried other methods like resizing or padding besides tilling when dealing with the matrix? Is tilling the best solution?**

- For providing detailed implementation insights, we have published pseudocode covering domain transformation. This content has been added to the Appendix D Algorithm section on pages 14 and 15 of the paper. We also enhanced the section 3.2 on domain transformation to facilitate a deeper understanding of the details among readers. Regarding the repetition of elements in the matrix, what we actually repeat are the elements contained within the matrix. Visually, this has an effect akin to magnifying a small image, which I believe is what you referred to as resizing. We initially experimented with filling a larger matrix with multiple smaller matrices. However, this approach resulted in very small edges and ambiguous boundaries, making it challenging for CNNs to extract meaningful features. Similarly, padding around the matrix didn't yield good results, especially when the base matrix size was very small, like (8,8). Among the intuitively conceivable methods, we believe the one we employed demonstrated the best performance.