Paper: PTaRL: Prototype-based Tabular Representation Learning via Space Calibration (ICLR-2024)

Reviewer comment (Experiments):
It would be great if the authors apply the proposed method to recent deep models for tabular representation, such as SAINT [1].

Author response:
Thanks for your valuable suggestion. We have applied PTaRL to recent deep tabular model SAINT [1], making it one of our baseline models.
We provide additional experiment results in Section 6 Table 1, Appendix Table 5 and Table 6.
The main results as follows:  


|                        | AD $\uparrow$  | HI $\uparrow$  | HE $\uparrow$  | JA $\uparrow$  | AL $\uparrow$  | CA $\downarrow$ | Win |
|------------------------|----------------|----------------|----------------|----------------|----------------|-----------------|-----|
| SAINT                  | 0.826          | 0.689          | 0.363          | 0.675          | 0.913          | 0.492           | 0   |
| SAINT + PTaRL          | **0.861**      | **0.728**      | **0.401**      | **0.728**      | **0.950**      | **0.471**       | **6**|

$\uparrow$ represents higher evaluation metric is better for classification and $\downarrow$ represents lower evaluation metric is better for regression. The results show that PTaRL has the ability to generally improve deep tabular models' performance, including recently proposed models such as SAINT. 

[1] SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training (Gowthami Somepalli and Avi Schwarzschild and Micah Goldblum and C. Bayan Bruss and Tom Goldstein) [NeurIPS 2022 Workshop]

Reviewer comment (Experiments):
I wonder if the authors believe that the proposed method can be applied to generative models.

Author response:
Thanks for your constructive suggestion.
We hope that our proposed method could offer valuable insights to enhance generative model representations and it will be an important direction for our future work.
Take the research line of deep generative models for representation learning as an example.
Deep generative models excel in discerning the true data distribution by learning the distribution of input data within the latent space. 
In this context, the representation of data in the latent space serves as a comprehensive data representation. 
It appears that PTaRL offers a potential solution to incorporate global data information through prototypes.
Leveraging such global data information to enhance the representation of latent structures aids in better modeling the true data distribution, consequently improving the generative model's performance.

Reviewer comment (Presentation):
The illustration (Fig. 2) does not clearly depict the overall pipeline; it still remains unclear.

Author response:
Thanks very much for pointing this confusion. We have revised our Fig. 2 in the original paper. 

We have clarified the hidden representation in Phase 1 and the projection representation in Phase 2. 
In phase 1, given any deep tabular model $F(\cdot;\theta) = G_h\left(G_f(\cdot;\theta_f); \theta_h\right)$, 
we initially train it with Eq. 2. Subsequently, K-Means clustering is applied to the output of the trained backbone $G_f(\cdot;\theta_f)$ to obtain global prototypes $B$ for constructing P-Space.
Moving on to Phase 2, $G_f(\cdot;\theta_f)$ and $G_h(\cdot;\theta_h)$ are re-initialized as $G_f(\cdot;\theta_f^*)$ and $G_h(\cdot;\theta_h^*)$.
The representation generated by $G_f(\cdot;\theta_f^*)$ is then projected into P-Space to learn the projection representation, denoted as ''Space Calibration'' in Fig. 2.
In this process, a shared coordinates estimator calculates coordinates for the projection in P-Space, and then the projection representation is obtained by $\sum_{k=1}^{K} r_{i}^{k} {\beta_{k}}$. 
The projection representation in P-Space would be finally processed by $G_h(\cdot;\theta_h^*)$ to give predictions.  

Moreover, we give further explanation about the three sentences on the right side of Fig. 2.
''Projecting Via Optimal Transport'' indicates that the representation by $G_f(\cdot;\theta_f^*)$ is pushed forward to projection representation by minimizing the Optimal Transport Distance between the two representations.
The sentences ''coordinates diversifying'' and ''prototypes orthogonalization'' correspond to two constraints for representation disentanglement, i.e. P-Space Coordinates Diversifying Constraint and Global Prototype Matrix Orthogonalization Constraint.

Reviewer comment (Reproducibility):
More details of the optimization process could be provided.

Author response:
We are sorry for the confusion and we would like to give a more clear description. 

The Optimal Transport (OT) problem is usually to find the most cost-effective way to transform a given distribution to another distribution, which is typically achieved by calculating the specified transportation plan that minimizes the total transportation cost, while the minimized cost is usually called OT distance.
In our paper, we minimize the distance between original representation distribution over each sample $P_i$ by deep tabular models and the corresponding projection representation distribution $Q_i$ in P-Space with global prototypes, in order to preserve original data information (of $P_i$) in $Q_i$.
We follow the typical setting of OT problem to first estimate the transport plan to obtain the OT distance between $P_i$ and $Q_i$.
Then, the obtained OT distance is further used as loss function to jointly learn the two representations.

Specifically, after initializing the global prototypes $B$ of P-Space, we project the original data samples into P-Space to learn the representations with global data structure information. 
To better illustrate the optimization process, we revise the Eq. 4 in the original paper to make it more readable. 
In Eq. 4, the $i$-th sample representation by deep tabular model is denoted as $G_f(x_i;\theta_f)$, the empirical distribution over this sample representation is $P_i = \delta_{G_f(x_i;\theta_f)}$, the projection representation distribution is denoted as: $Q_i = \sum_{k=1}^{K} r_{i}^{k} \delta_{\beta_{k}}$, where $r_i$ is coordinates. 
To capture the shared global data structure information, we formulate the representation projecting as the process of extracting instance-wise data information by $G_f(x_i;\theta_f)$ to $P_i$, and then pushing $P_i$ towards $Q_i$ to encourage each prototype $\beta_k$ to capture the shared global data structure information, a process achieved by minimizing the OT distance between $P_i$ and $Q_i$. 
The OT distance between $P_i$ and $Q_i$ could first be calculated by: $\textbf{OT}(P_i, Q_i) = \min_{\textbf{T}_i\in \Pi (P_i, Q_i)} \langle \textbf{T}_i, \textbf{C}_i \rangle$, 

where $C_{ik} = 1-cos(G_f(X_i;\theta_f), \beta_k)$, the average OT distance between $P_i$ and $Q_i$ over train sets could be viewed as loss function $L_{projecting}(X, B)$ to be further optimized: 

$\min\frac{1}{n}\sum_{i=1}^{n} \textbf{OT}(P_i, Q_i) = \min_{\theta_{f}, \gamma, B}  \frac{1}{n}\sum_{i=1}^{n} \min_{\textbf{T}_i\in \Pi (P_i, Q_i)} \langle \textbf{T}_i, \textbf{C}_i \rangle$, we use gradient descent to update $\theta_f, \gamma, B$. We also provide the detailed description in Appendix A.7 in the original paper.

The idea of applying OT distances as loss functions has been employed in multiple applications. 
For example, [1] proposes using this idea for semantic segmentation, and [2] applies this idea to outlier detection in data.

[1] Importance-aware semantic segmentation in self-driving with discrete wasserstein training (Liu, Xiaofeng and Han, Yuzhuo and Bai, Song and Ge, Yi and Wang, Tianxing and Han, Xu and Li, Site and You, Jane and Lu, Jun) [AAAI 2020]

[2] Outlier-robust optimal transport (Mukherjee, Debarghya and Guha, Aritra and Solomon, Justin M and Sun, Yuekai and Yurochkin, Mikhail) [ICML 2021]

Reviewer comment (Reproducibility):
How is equation (4) optimized? Compared to the traditional OT problem, it includes \theta_f as a variable to be optimized.

Author response:
We are sorry for the confusion and we would like to give a more clear description. 

The Optimal Transport (OT) problem is usually to find the most cost-effective way to transform a given distribution to another distribution, which is typically achieved by calculating the specified transportation plan that minimizes the total transportation cost, while the minimized cost is usually called OT distance.
In our paper, we minimize the distance between original representation distribution over each sample $P_i$ by deep tabular models and the corresponding projection representation distribution $Q_i$ in P-Space with global prototypes, in order to preserve original data information (of $P_i$) in $Q_i$.
We follow the typical setting of OT problem to first estimate the transport plan to obtain the OT distance between $P_i$ and $Q_i$.
Then, the obtained OT distance is further used as loss function to jointly learn the two representations.

Specifically, after initializing the global prototypes $B$ of P-Space, we project the original data samples into P-Space to learn the representations with global data structure information. 
To better illustrate the optimization process, we revise the Eq. 4 in the original paper to make it more readable. 
In Eq. 4, the $i$-th sample representation by deep tabular model is denoted as $G_f(x_i;\theta_f)$, the empirical distribution over this sample representation is $P_i = \delta_{G_f(x_i;\theta_f)}$, the projection representation distribution is denoted as: $Q_i = \sum_{k=1}^{K} r_{i}^{k} \delta_{\beta_{k}}$, where $r_i$ is coordinates. 
To capture the shared global data structure information, we formulate the representation projecting as the process of extracting instance-wise data information by $G_f(x_i;\theta_f)$ to $P_i$, and then pushing $P_i$ towards $Q_i$ to encourage each prototype $\beta_k$ to capture the shared global data structure information, a process achieved by minimizing the OT distance between $P_i$ and $Q_i$. 
The OT distance between $P_i$ and $Q_i$ could first be calculated by: $\textbf{OT}(P_i, Q_i) = \min_{\textbf{T}_i\in \Pi (P_i, Q_i)} \langle \textbf{T}_i, \textbf{C}_i \rangle$, 

where $C_{ik} = 1-cos(G_f(X_i;\theta_f), \beta_k)$, the average OT distance between $P_i$ and $Q_i$ over train sets could be viewed as loss function $L_{projecting}(X, B)$ to be further optimized: 

$\min\frac{1}{n}\sum_{i=1}^{n} \textbf{OT}(P_i, Q_i) = \min_{\theta_{f}, \gamma, B}  \frac{1}{n}\sum_{i=1}^{n} \min_{\textbf{T}_i\in \Pi (P_i, Q_i)} \langle \textbf{T}_i, \textbf{C}_i \rangle$, we use gradient descent to update $\theta_f, \gamma, B$. We also provide the detailed description in Appendix A.7 in the original paper.

The idea of applying OT distances as loss functions has been employed in multiple applications. 
For example, [1] proposes using this idea for semantic segmentation, and [2] applies this idea to outlier detection in data.

[1] Importance-aware semantic segmentation in self-driving with discrete wasserstein training (Liu, Xiaofeng and Han, Yuzhuo and Bai, Song and Ge, Yi and Wang, Tianxing and Han, Xu and Li, Site and You, Jane and Lu, Jun) [AAAI 2020]

[2] Outlier-robust optimal transport (Mukherjee, Debarghya and Guha, Aritra and Solomon, Justin M and Sun, Yuekai and Yurochkin, Mikhail) [ICML 2021]

Reviewer comment (Reproducibility):
Could you provide more technique details about the workflow of the PTARL algorithm (Algorithm 1)?

Author response:
We have revised the workflow of PTaRL in Algorithm 1 in the original paper's Appendix to provide more technique details. 

As shown in the algorithm, Line 1 to 3 represents the global prototype initialization process. 
Given any deep tabular model $F(\cdot;\theta) = G_h\left(G_f(\cdot;\theta_f); \theta_h\right)$, we first train it by Eq. 2 and then apply K-Means clustering to the output of the trained backbone $G_f(\cdot;\theta_f)$ to get global prototypes $B$.

From Line 4 to 19, the second stage of representation projecting is illustrated. 
The parameters of $F(\cdot;\theta)$ need to be re-initialized. 

Line 8 to 14 details the process of calculating $L_{projecting}$.
Within a minibatch, for each instance, we calculate the OT distance between its representation by the deep tabular model and the projection representation in P-Space.
We then average the OT distance of each instance within the minibatch as the $L_{projecting}$.
In Line 12, $C_i$ is the transport cost matrix calculated by cosine distance: $C_{ik} = 1-cos(G_f(X_i;\theta_f), \beta_k)$.
The $L_{projecting}$ would be further optimized to push $P_i$ towards $Q_i$ to encourage each prototype to capture the shared global data structure information.

We introduce $L_{task}$ in Line 15 to incorporate task-specific label supervision.
$L_{diversifying}$ in Line 16 is introduced as the P-Space Coordinates Diversifying Constraint to alleviate the collapse of coordinates generating.
Additionally, $L_{orthogonalization}$ in Line 17 serves as the Global Prototype Matrix Orthogonalization Constraint, ensuring the independence of global prototypes from each other.
Here, $M \in [0, 1]^{K\times K}$ and $M_{ij} = \\|\cos(\beta_i, \beta_j)\\|_1$.

Reviewer comment (Presentation):
The illustration (Fig. 2) could benefit from improvement as it currently lacks clarity in depicting the overall pipeline. For instance, there are two blocks labeled "Hidden Representation"; could you clarify the distinction between them? Additionally, the three sentences on the right side of the figure require further explanation for better understanding.

Author response:
Thanks very much for pointing this confusion. We have revised our Fig. 2 in the original paper. 

We have clarified the hidden representation in Phase 1 and the projection representation in Phase 2. 
In phase 1, given any deep tabular model $F(\cdot;\theta) = G_h\left(G_f(\cdot;\theta_f); \theta_h\right)$, 
we initially train it with Eq. 2. Subsequently, K-Means clustering is applied to the output of the trained backbone $G_f(\cdot;\theta_f)$ to obtain global prototypes $B$ for constructing P-Space.
Moving on to Phase 2, $G_f(\cdot;\theta_f)$ and $G_h(\cdot;\theta_h)$ are re-initialized as $G_f(\cdot;\theta_f^*)$ and $G_h(\cdot;\theta_h^*)$.
The representation generated by $G_f(\cdot;\theta_f^*)$ is then projected into P-Space to learn the projection representation, denoted as ''Space Calibration'' in Fig. 2.
In this process, a shared coordinates estimator calculates coordinates for the projection in P-Space, and then the projection representation is obtained by $\sum_{k=1}^{K} r_{i}^{k} {\beta_{k}}$. 
The projection representation in P-Space would be finally processed by $G_h(\cdot;\theta_h^*)$ to give predictions.  

Moreover, we give further explanation about the three sentences on the right side of Fig. 2.
''Projecting Via Optimal Transport'' indicates that the representation by $G_f(\cdot;\theta_f^*)$ is pushed forward to projection representation by minimizing the Optimal Transport Distance between the two representations.
The sentences ''coordinates diversifying'' and ''prototypes orthogonalization'' correspond to two constraints for representation disentanglement, i.e. P-Space Coordinates Diversifying Constraint and Global Prototype Matrix Orthogonalization Constraint.