Paper: Polynomial-based Self-Attention for Table Representation learning (ICLR-2024)

Reviewer comment (Theory):
Lack of detailed analysis of computational complexity.

Author response:
Starting with the self-attention mechanism’s time complexity, the original attention mechanism operates at $\mathcal{O}(n^2d)$, where $n$ represents the number of tokens and $d$ is the dimension of each token. Our method introduces additional complexity to compute $A^k$ with $k-1$ matrix multiplications, resulting in a time complexity of $\mathcal{O}(n^2d + (k-1)n^{2.371552})$, where we assume that we use algorithm in [1]. Practically, if $d > (k-1)n^{2.371552}$, the time complexity of CheAtt becomes $\mathcal{O}(n^2d)$, a condition observed to be met across almost all 10 datasets.

Also, we have updated our manuscript based on your suggestion in red.

Reviewer comment (Experiments):
Inadequate comparison with other recent approaches.

Author response:
To the best of our knowledge, this study is the first to investigate over-smoothing in transformers applied to tabular data. Unfortunately, this uniqueness made it impossible to compare our work with other methods addressing over-smoothing in Transformer-based tabular data approaches.

Reviewer comment (Presentation):
Presentation of results could be improved.

Author response:
Thank you for the suggestion; we have added arrows to the tables.

Reviewer comment (Experiments):
Unclear scalability to larger datasets or more complex models.

Author response:
We emphasize that our proposed layer is applicable to any transformer-based model, regardless of its complexity. Scaling up the layer for larger or more complex datasets can be achieved through conventional methods, such as increasing the number of transformer blocks or augmenting the hidden vector size. This scaling process enables the attention matrix to expand, and the Chebyshev polynomial of the attention matrix can express a more complex data distribution. Additionally, we are conducting further experiments on larger datasets and will promptly report the results upon completion.

Reviewer comment (Theory):
The paper does not provide a detailed analysis of the computational complexity of the proposed matrix polynomial-based self-attention layer.

Author response:
Starting with the self-attention mechanism’s time complexity, the original attention mechanism operates at $\mathcal{O}(n^2d)$, where $n$ represents the number of tokens and $d$ is the dimension of each token. Our method introduces additional complexity to compute $A^k$ with $k-1$ matrix multiplications, resulting in a time complexity of $\mathcal{O}(n^2d + (k-1)n^{2.371552})$, where we assume that we use algorithm in [1]. Practically, if $d > (k-1)n^{2.371552}$, the time complexity of CheAtt becomes $\mathcal{O}(n^2d)$, a condition observed to be met across almost all 10 datasets.

Also, we have updated our manuscript based on your suggestion in red.

Reviewer comment (Experiments):
The paper does not provide a thorough comparison of the proposed approach to other methods for addressing the over-smoothing issue in Transformer-based methods for tabular data. While the experiments show that the proposed layer outperforms state-of-the-art methods, it is unclear how the proposed approach compares to other recent approaches in the literature.

Author response:
To the best of our knowledge, this study is the first to investigate over-smoothing in transformers applied to tabular data. Unfortunately, this uniqueness made it impossible to compare our work with other methods addressing over-smoothing in Transformer-based tabular data approaches.

Reviewer comment (Presentation):
It is recommended to add up or down arrows to the indicators in the chart, such as Table 2, Table 3, Table 4, and Table 5.

Author response:
Thank you for the suggestion; we have added arrows to the tables.

Reviewer comment (Experiments):
The experiments show that the layer is effective. While the dataset used in the experiment is small, it is unclear how the layer would scale to larger datasets or more complex models. It is recommended to increase the results of testing on large datasets.

Author response:
We emphasize that our proposed layer is applicable to any transformer-based model, regardless of its complexity. Scaling up the layer for larger or more complex datasets can be achieved through conventional methods, such as increasing the number of transformer blocks or augmenting the hidden vector size. This scaling process enables the attention matrix to expand, and the Chebyshev polynomial of the attention matrix can express a more complex data distribution. Additionally, we are conducting further experiments on larger datasets and will promptly report the results upon completion.

Reviewer comment (Experiments):
It is better to present more details about the task of tabular data understanding.

Author response:
Thank you for the comment. In response, we have included additional details about the task in our manuscript.

Reviewer comment (Novelty):
As I have limited experience in dealing with the tabular data, could the authors provide if any existing method tackling the issue of over-smoothing for this task?

Author response:
To the best of our knowledge, our study is the first to address the issue of over-smoothing specifically for tabular data. The mitigation of over-smoothing for GCN has been a long-standing research area [1, 2]. In specific, GREAD [2] is our special case, where Equation (7) in our main paper with $k=2$, $w_0=1$, and $w_1=-1$. Moreover, research to alleviate over-smoothing and feature collapse in ViT (Vision Transformer) has recently become active [3].

Reviewer comment (Theory):
The motivation for the paper is not adequately supported by theory. The paper mentions that better flexibility and customization can be achieved by considering the self-attention mechanism as a graph filter in graph signal processing, but does not cite enough papers or theorems to fully convince this point. In addition, when proving that the self-attention matrix conforms to the three properties of the transition matrix required by the convergence of pagerank algorithm, the authors only make a rough qualitative analysis but do not carry out a more sufficient and detailed analysis and relevant theoretical or experimental proof. Since the self-attention matrix in the transformer is unpredictable, once it does not meet the corresponding requirements, the self-attention based on matrix polynomials will not be able to approximate the graph filter, and thus will not be able to realize the expected results. Therefore, I suggest the authors to provide more details in this regard.

Author response:
The self-attention matrix and the PageRank matrix share one key common characteristic that their adjacency matrices are fully connected with many small values. PageRank uses a matrix of $(1-\alpha) \mathbf{A} + \alpha\mathbf{\frac{1}{N}}$, where $\mathbf{A}$ is an adjacency matrix, $N$ is the number of nodes, and $\alpha$ is a damping factor, and therefore, all elements have non-zero (small) values. (Please refer to Section 3.2 PageRank and Equation (5) for details.) In the self-attention matrix, this is also the case since Transformers use the softmax function. Because of this characteristic, in addition, PageRank converges and so does our method.

As you said, the self-attention matrix is unpredictable. As long as the self-attention matrix is fully connected, however, our method is correct (since we rely on the PageRank theory). It is practically unbelievable that the softmax produces zeros although some values are small. Note that in PageRank, small values are also used when $N$ is very large, i.e., a web-scale graph of billions of nodes.

The first and second conditions shown in the paper are obvious. For the last condition, we refer to [1]. An irreducible chain, as defined in [1], has a period that is common to all states. The irreducible chain will be called $\textit{\textbf{aperiodic}}$ if all states have a period of 1. This implies that a Markov chain is aperiodic if there is at least one self-loop. As discussed earlier, the self-attention matrix has non-negative values for all elements, including diagonal elements, making the self-attention matrix aperiodic.


We added a section in Appendix in red regarding this point.

Reviewer comment (Evaluation):
Some experimental data with excessive errors will interfere with the experimental results. In the experimental part, the error range of individual experimental results is too large relative to other data. For example, 70.9±13.90 in Table 2 and 58.1±24.32 in Table 3. When these margins of error are taken into account, it becomes a question which method yields the best experimental results. This may potentially interfere with the fairness of comparisons between different methods, thereby affecting the correctness of experimental results.

Author response:
For reproducibility, we repeated all experiments 5 times and reported the mean and standard deviation. We used the same seed numbers for all experiments for fair comparison. The two scores you mentioned have one or two outliers in the set. In Table 2, 70.9±13.90 is the mean and standard deviation of 73.58, 78.07, 79.44, 76.87, **46.30**. In Table 3, 58.1±24.32 is the mean and standard deviation of 77.94, **25.98**, 78.17, 77.56, **30.72**. Among those 5 repetitions, one or two scores are unusually low, attributed to the unstable training of AE.

Reviewer comment (Evaluation):
The applicability of CheAtt should be further discussed. The paper mentioned that the effect of CheAtt is very dependent on the quality of the base model. As can be seen from Table 1 and Table 4, in TabTransformer and MET, the effect of CheAtt is outstanding, but there is almost no improvement in SAINT. They are all table representation methods based on transformer, and the original performance of SAINT is the best among the three. So what exactly does the "quality of the base model" mentioned in the paper refer to? According to the author's analysis, self-attention based on Chebyshev polynomials can improve the flexibility of the self-attention mechanism. This improvement should not be strongly related to the base model, so do the experimental results mean that CheAtt is not applicable in many situations? I suggest that the authors conduct further analysis in this area.

Author response:
We are now testing more hyperparameters for SAINT+CheAtt since as can be seen in Appendix C.3 Table 8, we have searched only 3 hyperparameters recommended in its original paper even though SAINT has a number of hyperparameters to search. We will provide the results as soon as the ongoing experiments are completed. We will also update our paper one more time afterward.

Reviewer comment (Evaluation):
The complexity of CheAtt still needs further discussion. First, the data in Table 5 are all in the range of a few milliseconds, does it refer to the time to generate output after the model training is completed? If so, this does not take into account the large number of matrix multiplication operations required during model training. In addition, it is meaningless to only compare the absolute time difference, and it is more convincing to compare the relative time consumption. It can be seen that in Phishing dataset, the additional time spent can exceed up to 40% of the original, which is a huge and unacceptable increase. Another question is why in the MET+CheAtt method and Phishing dataset, the time after using CheAtt is reduced (from 2.7538 to 2.4625). Is this a clerical error or real experimental data? I recommend the authors to perform a more comprehensive analysis and more realistic experiments in terms of computational complexity.

Author response:
We have incorporated your suggestions into our manuscript, and we kindly ask you to refer to the updated version.
To begin, we have discussed computational complexity in detail in our revised manuscript, marked in red. Originally, the attention mechanism operated at $\mathcal{O}(n^2d)$, where $n$ represents the number of tokens, and $d$ is the dimension of each token. Our method introduces additional complexity to compute $A^k$ with $k-1$ matrix multiplications, resulting in a time complexity of $\mathcal{O}(n^2d + (k-1)n^{2.371552})$, assuming the use of the algorithm in [2]. In practical terms, if $d > (k-1)n^{2.371552}$, the time complexity of CheAtt becomes $\mathcal{O}(n^2d)$, a condition observed across almost all 10 datasets.


Secondly, we originally reported the absolute wall clock time to emphasize efficiency. However, we also acknowledge the importance of relative time. Therefore, in Table 5, we now present both absolute and relative time consumption. Additionally, we have updated the increase in inference time to the time taken to infer 1,000 samples. We have also included a summary of training time in Table 5, representing the average time for 5 epochs of training, along with reporting the relative time. For training time, we train the model for 5 epochs and average the time. The updated table is presented below.


Lastly, we apologize for any confusion, but the reported reduction in time after using CheAtt on Phishing was a typo. This has been corrected in our updated paper. Thank you for bringing it to our attention. 

Below is the updated Table 5:

|                  | Training time (per epoch)           | Inference time (for 1,000 samples)           |
|------------------|--------------------------|---------------------------|
| TabTransformer   | 3.07s                    | 5.42ms                    |
| TabTrans.+CheAtt | 3.57s ($\uparrow$20.32%) | 6.26ms ($\uparrow$17.27%) |
| SAINT            | 4.34s                    | 5.25ms                    |
| SAINT+CheAtt     | 5.27s ($\uparrow$18.91%) | 6.68ms ($\uparrow$28.95%) |
| MET              | 2.68s                    | 2.67ms                    |
| MET+CheAtt       | 3.34s ($\uparrow$23.56%) | 3.47ms ($\uparrow$31.09%) |