Paper: MCM: Masked Cell Modeling for Anomaly Detection in Tabular Data (ICLR-2024)

Reviewer comment (Novelty):
Why is learnable masking so important for AD, when random masks are fine for other tasks?

Author response:
This is mainly due to the difference in **data type** and  **the purpose of masking**. In CV domain, a simple random masking strategy with a high masking ratio works quite well [1]. Because all the features of image data are pixels, which are **homogeneous and highly relevant**, no matter where to mask, predicting a large proportion of masked patches with just a small proportion of unmasked patches is meaningful for the model to **understand an image**, such that useful representations for downstream tasks can be learnt. On the contrary, the features of tabular data are **heterogeneous**, each feature has a different meaning, some of them are totally **irrelevant**. Our purpose is to **capture intrinstic correlations** in normal data by finding using which unmasked features can reconstruct the left masked features well. Random masking strategy is easy to produce meaningless masks for this purpose. But by learnable masking strategy, the mask generator can generate optimal masks for our purpose. Thus, correlations that can characterize normal data will be captured and anomalies can by judged by whether deviating from such correlations.  

[1] Masked autoencoders are scalable vision learners. CVPR 2022

Reviewer comment (Experiments):
The batch size is very large, and is an important parameter as it affects the dimensionality of M. How does the methodology perform with a smaller batch size? Also, can the methodology still work if the batch size changes between training and test sets?

Author response:
In our method, there are only two matrix operations involving M, the first is element-wise product between M and the original data to produce masked data, the second is the calculation of diversity loss. Both of them are calculated independently for each sample and do not involve calculations between different data. Therefore, although batch size affects the dimensionality of M, it does not affect the overall time and complexity of matrix operations. So a large batch size is quite acceptable, especially considering that the dimensions of tabular data are mostly relatively low. However, we still supplement the parameter sensitivity experiments for batch size in **Table 12** and **Table 13** in the latest version. As is shown, the average AUC-PR on the 20 benchmark datasets with batch size of **{64, 128, 256, 512}** are **{0.7052, 0.7108, 0.7155, 0.7486}**. The performance does decrease as the batch size decreases, but even with a batch size of 64, our method can still surpass all  the compared methods and achieve SOTA performance. Moreover, as we illustrated above, our method is calculated for each data separately, so the batch size of the test set can be any value. Of course, it can also be different from the batch size of the training set, and it will not affect the final results.

Reviewer comment (Theory):
Diversity loss (Eq. 5) has log-sum-exp form. The motivation for this design choice is not clear since other formulations may also be effective.

Author response:
**First**, the motivation of our log-sum-exp formulation is derived from the InfoNCE loss of contrastive learning [1]. In the expression of InfoNCE loss, the numerator aims to increase the similarity between positive pairs, while the denominator aims to decrease the similarity between negative pairs. As we desire all masks of a sample to be dissimilar, all pairs of masks can be considered as negative pairs. Thus, we adopted the form of the denominator in InfoNCE, which is in log-sum-exp form. Experiments show the design leads to satisfying performance. 

**Second**, to investigate different designs of diversity loss, we conduct ablation experiments on 20 benchmark datasets. Because diversity loss is used to decrease the similarity of different masks of a sample, we set the following formulations to calculate the similarity of different masks  : *1. Euclidean distance 2. Cosine similarity 3. Dot product 4. Our proposed form without log operation 5. Ours*.  As shown in **Table 14** and **Table 15** in Appendix of the latest version, the average AUC-ROC are **0.8683, 0.8834, 0.8465, 0.8609, 0.9044**, while the average AUC-PR are **0.6883, 0.6951, 0.6463, 0.6701, 0.7486,** respectively, demonstrating the effectiveness of our design.

[1] Momentum Contrast for Unsupervised Visual Representation Learning. CVPR 2020

Reviewer comment (Experiments):
Ablating different designs of diversity loss are missing.

Author response:
**First**, the motivation of our log-sum-exp formulation is derived from the InfoNCE loss of contrastive learning [1]. In the expression of InfoNCE loss, the numerator aims to increase the similarity between positive pairs, while the denominator aims to decrease the similarity between negative pairs. As we desire all masks of a sample to be dissimilar, all pairs of masks can be considered as negative pairs. Thus, we adopted the form of the denominator in InfoNCE, which is in log-sum-exp form. Experiments show the design leads to satisfying performance. 

**Second**, to investigate different designs of diversity loss, we conduct ablation experiments on 20 benchmark datasets. Because diversity loss is used to decrease the similarity of different masks of a sample, we set the following formulations to calculate the similarity of different masks  : *1. Euclidean distance 2. Cosine similarity 3. Dot product 4. Our proposed form without log operation 5. Ours*.  As shown in **Table 14** and **Table 15** in Appendix of the latest version, the average AUC-ROC are **0.8683, 0.8834, 0.8465, 0.8609, 0.9044**, while the average AUC-PR are **0.6883, 0.6951, 0.6463, 0.6701, 0.7486,** respectively, demonstrating the effectiveness of our design.

[1] Momentum Contrast for Unsupervised Visual Representation Learning. CVPR 2020

Reviewer comment (Evaluation):
Are there any limitations of the proposed method?

Author response:
**A1**: In the current field of deep learning, most attention is paid to general large models that can be trained on various datasets and applied to downstream tasks. Although our network architecture and parameters almost do not need to change for different datasets, our method still requires training a model from scratch for each dataset, which can not combine knowledge from different datasets. In the future, we will try to change our network to a Transformer-like architecture, train one general model to learn knowledge from different datasets, and address various tabular AD tasks of different dimensions and areas by just using one model.