Paper: On Transferring Expert Knowledge from Tabular Data to Images (ICLR-2024)

Reviewer comment (Writing):
The paper has a lot of mathematical notations that are not defined or ambiguous. For example, according to Section 4.2, $\mathbf{x}^T_{num}$ contains numerical data in the table, which is transformed to $\psi(\mathbf{x}^T) \in \mathbb{R}^{D \times E}$ together with the categorical data. I think it’s not straightforward to transform numerical values into an embedding. A more specific definition is necessary. Also, I think $S_{I_j}$, which I guess is the $j$-th channel of $S_I$ and is not explicitly mentioned in the paper, should be a matrix but implicitly used as a vector in Eq. (5). The second part of Eq. (5) if $<\mathbf{C}, \mathbf{T}>_F$ stands for the Frobenius norm, which is not explained in the paper, the summations over $i$ and $j$ should not be necessary. At least, min in this equation should be argmin.

Author response:
Thank you for your suggestions. Specifically, the transform numerical and categorical values into an embedding are as follows: 
$T_j = b_j + f_j(x_j) \in \mathbb{R}^d$, where $b$ is the feature bias, $f$ is implemented as the element-wise multiplication with the vector $\boldsymbol{W}$. Overall $\boldsymbol{x_{num}^T} = b_{num} + \boldsymbol{x_{num}} \boldsymbol{W_{num}}$  is for numerical attributes and $\boldsymbol{x_{cat}^T}= b_{cat} + \boldsymbol{x_{cat}} \boldsymbol{W_{cat}}$ for categorical. By denoting different attributes distinctly, we can effectively distinguish between numerical and categorical variables.

We have also corrected some remaining errors, including the representation of vectors and matrices, the modification of argmin, and the specification of the Frobenius norm. Thank you for your generosity in pointing out our problems.

Reviewer comment (Experiments):
According to Eq. (6), what this method actually does is multi-task training, where in addition to the target classification task, it also leans to classify/regress the attributes in the table (I guess $\phi(\mathbf{x}^T)$ in the second term of $\mathcal{L}_{i2t}$ is a typo of $\phi(\mathbf{x}^I)$ as well as $\mathcal{A}_p$ and $\mathcal{A}_q$ are $A$ corresponding to the numerical and categorical values in the table, respectively). If this is the case, this approach is suboptimal as the computation of $A$ is outside of the optimization for the target task and the attribute prediction tasks. In this sense, a comparison with the case where, instead of the tabular alignment in the method, a fully connected layer or an MLP dedicated to each attribute prediction task is used is necessary.

Author response:
Thank you for pointing out our typo. Thank you also for pointing out the shortcomings of our comparison methods. In fact, **we have taken this situation into account in our Appendix B1**. We use the self-attention method to weigh the image channels differently and learn each tabular attribute separately, which can be seen as an enhanced MLP. The self-attention method constructs a learnable mask for each attribute and learns each attribute separately based on the extracted feature.

The result shows that the self-attention method did not perform as well as our method on all datasets. Specifically, on the DVM dataset, which involves a complex downstream task of 129 classification tasks, the attention method constructed different attentions for different attributes, which confused the backbone network and led to a decrease in overall task performance. This finding highlights the impracticality of using the attention mechanism alone to integrate the abundant information in tabular data into the image model. In other words, if we **directly let the image predict every attribute of the table**, this does not transfer the knowledge from the table to the image, but instead clutters the image model and causes it to **learn without focus**.

However, in our approach, we automatically find out the tabular attributes associated with the image and pay more attention to the learning of the relevant parts, which serves to guide the image for more detailed learning, making our approach perform better.

Reviewer comment (Theory):
If my assumption about $\phi(\mathbf{x}^T)$ in Eq. (6) is correct, I think the second term of $\mathcal{L}$ in Eq. (6) does nothing to the performance.

Author response:
The second term in Eq. 6 is to update the tabular model. The tabular model is updated in order to **get a more accurate representation of each tabular attribute**. As shown in Eq. 5, we need to use the feature similarity matrix of each attribute of the table when calculating the cost matrix of OT. Therefore the tabular model is updated to get a more accurate representation of each attribute of the table, so as to **compute a more accurate transfer matrix of the OT**, which lays the foundation for a specific channel to learn a certain attribute of the table.

As we stated in Appendix A2, we choose to update the cost matrix every 5 epochs, striking a balance between updating them without stable knowledge and minimizing the computational burden. An update of the cost matrix is essential. As the model is updated, the representation of each attribute and channel is more and more accurate. Therefore, the second term of loss in Eq. 6 is essential. 

To prevent any confusion regarding the formula, we acknowledge that there might have been some oversight on our part. Initially, we mistakenly assumed that this method was universally understood by everyone. We would like to rephrase our statement in the hope of providing clearer comprehension. Thanks again for pointing out our typo. We will also make this clear in the final version.

Reviewer comment (Writing):
Phi has been first used in page 4, equation 3 but hasn’t been explained until page 6. The mathematical definition below equation 1 to introduce phi is insufficient given the context of introducing the preliminaries.

Author response:
**A1**: Thank you very much for pointing out our problem. Phi is the feature extractor to extract the embedding of the images. We will explain phi in more detail under equation 1 in a future version.

Reviewer comment (Evaluation):
It would strengthen the paper to see some human evaluation to show the model is useful for an expert using the system.

Author response:
**A4**: Thank you for your advice. We are **working with hospitals** to try to apply our method to the task of infection prediction. However, it is difficult to give a quantitative result to demonstrate the performance of our method due to the **privacy** of the users and the special requirements of medical treatment. If there is further result from expert evaluations, we will update it in time for later versions. Thanks again for your suggestion.

Reviewer comment (Writing):
A clear explanation on the different types of tasks and their usage for both the types of tasks would help understand the impact of the proposed model better.

Author response:
**A5**:   Thanks for your suggestion. Due to the limitations of the paper's length, we will explain the use of our model in the different tasks in further detail in Appendix.

CHARMS is a general method for both classification and regression tasks, in detail, we use cross entropy loss for classification task and mean square error loss for regression task. We achieved an improved image representation by employing the CHARMS method, which leverages the guidance of tabular data on the image data. Specifically, for the classification task, our approach facilitated the representation with a more discerning distribution over the target categories. On the other hand, the regression task enabled us to learn an image representation that better approximated the target values during prediction. The fact that our method performs well on both tasks underscores its generalizability and effectiveness.

In summary, our approach demonstrates its versatility by excelling in both classification and regression tasks, showcasing its ability to enhance image representations using guidance from tabular data.

Reviewer comment (Writing):
What is “attention image” referred to above equation 6?

Author response:
**A6**:  We use A to weigh the features of the image after extracting them, which can be seen as an attentional approach and pays more attention to the channels corresponding to the tabular attribute. More information about attention is in Appendix B1. We will make it clear in the final version.

Reviewer comment (Evaluation):
According to Table 1, the proposed model underperforms the LGB and RTDL on the DVM task by a large margin. Why is this so? A one-line explanation in the paper would help the reader understand the proposed method better.

Author response:
**A7**: In some datasets, the performance after cross-modal transfer is not as good as the performance of the tabular model may be due to the fact that the attributes in the tables that can not be transferred are critical for the task. In these datasets, the tabular data is **strong modality** therefore the tabular data performs better. 

On DVM dataset, the tabular model demonstrated promising results, primarily because a significant portion of the knowledge contained in the tabular data, such as engine parameters and prices. These attributes play a crucial role in the classification task. 

In summary, it is possible that our approach can outperform the tabular modality even if it is a strong modality. We've already stated this in result in Sec 5.1, and we'll make it clearer later.

Reviewer comment (Reproducibility):
Is the code for this method openly available for checking the reproducibility of this study?

Author response:
**A8**: Yes, our code is being organized and will be open-sourced later for reproducibility.

Reviewer comment (Writing):
Even though I appreciate injecting an empirical argument against using mutual information methods (i.e. section 4.1) before going forward with the presentation of the proposed method, the section itself (i.e. section 4.1) was entirely confusing to me. First, I wasn't sure if the authors intended to refer to Figure 8 (placed in the appendix) or Figure 2 (placed right inside section 4.1). Second, the flow of the explanation was quite difficult to follow and I cannot say I understood it. I understand the conclusion that the authors want to draw (i.e. mutual information is not great), but I could not understand the reasoning. Finally, I didn't understand the elements of the figure itself which made it impossible for me to see what the intended conclusion is.

Author response:
Thank you for question. Firstly, here we would like to point to Figure 2. We have used this Figure 2 again in the appendix to make it clearer for the reader to read and analyse in conjunction with Figure 9.

Secondly, we calculated the mutual information with tabular data versus the image data to construct Fig. 2. We want to illustrate that the mutual information is not the same for different methods and different modalities. Then, we found that the mutual information of and image can be improved by distillation of **morphological data**. Finally, we conclude that different attributes in tabular data have different effects on the image and we need to explore the useful knowledge in tabular data and transfer it to the image. 

Finally, we will also describe the figure  in more detail in the article so that readers can capture the information in the diagram more clearly.

Reviewer comment (Reproducibility):
There is some information that I wish was present in the experimental results section. Specifically, I would like to see a description of the experimental protocol explaining how the numbers in Table 1 were derived (I can make an educated guess but I would prefer to see it explicitly).

Author response:
Thanks for your reminder. We describe our experimental setup in **Sec 5 and Appendix A**. Specifically, there are experimental hyperparameter settings, as well as experimental details. We will select some important information to put in the main paper if space permits in the final version.

In Table 1 we show the performance of them is to tell the reader roughly where each modal's performance would be. This allows us to clearly see how performance changes after modal knowledge transfer, e.g. if there is a negative transfer. We think it is necessary to show the performance of the tabular method here because we can see the difference in the amount of information in the tabular data and the image data. The **amount of change** in performance after transfer shows how much knowledge from the tabular data is actually transferred.

Thank you for pointing out that we should explain the fact that our method does not perform as well as the tabular model in some datasets. In our setting we expect to transfer knowledge from the tabular data to the images, but not all tabular knowledge is transferable. In some datasets, the performance is not so good may be due to the fact that the attributes in the tables that can not be transferred are critical for the task. In these datasets, the tabular data is **strong modality** therefore the tabular data performs better. For example, in the Pet dataset, the pet's vaccine information is important for pet adoption, but this information is not displayed in the image.

In summary, we expect to transfer useful, image-related knowledge from tables to images to improve the performance of image models.

Reviewer comment (Evaluation):
In the experimental results, some of these models seem to outperform knowledge transfer methods. Even though this doesn't necessarily kill the main thesis of the paper, I still feel like it should be discussed explicitly as opposed to merely ignoring it.

Author response:
Thanks for your reminder. We describe our experimental setup in **Sec 5 and Appendix A**. Specifically, there are experimental hyperparameter settings, as well as experimental details. We will select some important information to put in the main paper if space permits in the final version.

In Table 1 we show the performance of them is to tell the reader roughly where each modal's performance would be. This allows us to clearly see how performance changes after modal knowledge transfer, e.g. if there is a negative transfer. We think it is necessary to show the performance of the tabular method here because we can see the difference in the amount of information in the tabular data and the image data. The **amount of change** in performance after transfer shows how much knowledge from the tabular data is actually transferred.

Thank you for pointing out that we should explain the fact that our method does not perform as well as the tabular model in some datasets. In our setting we expect to transfer knowledge from the tabular data to the images, but not all tabular knowledge is transferable. In some datasets, the performance is not so good may be due to the fact that the attributes in the tables that can not be transferred are critical for the task. In these datasets, the tabular data is **strong modality** therefore the tabular data performs better. For example, in the Pet dataset, the pet's vaccine information is important for pet adoption, but this information is not displayed in the image.

In summary, we expect to transfer useful, image-related knowledge from tables to images to improve the performance of image models.

Reviewer comment (Writing):
What is the reasoning behind using the term "channel" to refer to high-level image features? I find it confusing since the word channel in images typically refers to e.g. the color channels. Especially since in the paper, you use the term in both the high-level feature sense and the color channel sense.

Author response:
In the AlexNet's paper[1], the authors indeed mentioned that the image consists of **RGB channels**. Similarly, in the case of ResNet[2], the authors described the image processing as being conducted **channel by channel**. We think it is common to use channel to represent high-level or low-level expressions of image information.

Reviewer comment (Writing):
In section 5.2, you measure the mutual information during training. What are the quantities between which you are measuring the mutual information? This is not clear to me and makes the entire section (including the figure) quite hard to understand.

Author response:
In **Appendix A3** of the paper, we explain in detail the exact meaning of this figure and the calculation of mutual information. Specifically, we calculate the mutual information between the representation obtained from our model and the representations of both the optimal table model and image model with MINE method. This allows us to **quantify the amount of shared information** or knowledge between these representations. By measuring mutual information, we can assess the extent to which our model captures and transfers knowledge from both the table and image domains. We may have misjudged the audience for this paper and put the relevant text in the appendix, this is a problem and we will try to **reorganize the writing**.

Reviewer comment (Experiments):
Limited exploration of extremely large tabular datasets with hundreds of attributes.

Author response:
Thank you for bringing this to our attention. Our objective is to transfer knowledge from tabular data to images. Through the use of POT, we are able to automatically match the **corresponding tabular attributes** to the images. Consequently, even with a substantial number of tabular attributes, our method performs admirably. For instance, in our SUN dataset, which contains 101 tabular attributes, our method outperforms all the compared methods. We have made significant efforts to locate a suitable tabular-image dataset, and it has proven to be **quite challenging**. We will continue our endeavors to conduct experiments on larger tabular datasets.

Reviewer comment (Experiments):
It is widely witnessed that there are many uninformative features in the tabular data, I am not sure if the model can consistently perform well in such cases. The authors can test on tables that are manually added some Gaussian noisy columns/features (e.g., 10 columns, 20 columns) and observing the performance changes.

Author response:
Thank you for your valuable advice. In our approach, the inclusion of POT enables **automatic selection** of image-related features for knowledge transfer. As you rightly mentioned, in the dataset we are utilizing, certain attributes such as the pet's health or vaccine information are not visually depicted in the image. Therefore, these attributes can indeed be considered as **uninformative features**, just as you suggested. For instance, in Pet dataset, we can treat these attributes as uninformative features that do not significantly contribute to image representation and so as other datasets. Your suggestion of adding noise features to verify the performance changes is very reasonable. We conducted experiments based on your suggestions, but the results were **basically unchanged**.

Reviewer comment (Experiments):
Can this model work on complex vision tasks like segmentation/detection that may require localization?

Author response:
In complex tasks like segmentation and detection, the objective is to accurately construct bounding boxes or differentiate between objects with distinct semantics. When dealing with tables that contain semantically specific attributes, we hypothesize that our approach can do this. However, due to the inconsistency of the task goals, when using our method to the detection task, we may need to do **extra processing** on the bounding box to make it adapt our method. We appreciate your suggestion, and it indeed aligns with potential future work. Thank you for your valuable question.

Reviewer comment (Novelty):
Some related work about tabular learning are missing, e.g., T2G-Former, TabPFN, TabNet, TANGOS, TapCaps.

Author response:
Thank you for your suggestion. We conducted a comparison between the traditional tree-based approach and the FT-Transformer approach to gain a preliminary understanding of tabular data performance. While the methods you mentioned can achieve comparable results, they cannot be directly applied in our specific setting. When referencing tabular models, a comprehensive understanding of the domain will enhance the completeness of the paper. We will update the related work.