Paper: Knowledge Graph Completion by Intermediate Variables Regularization (ICLR-2024)

Reviewer comment (Presentation):
The paper is not organized clearly, which is not friendly for understanding. For instance, there is a lack of an intuitive explanation for how this method can mitigate the overfitting issue.

Author response:
**Q1: The paper is not organized clearly, which is not friendly for understanding. For instance, there are lack of an intuitive explanation for how this method can mitigate the overfitting issue.**

A1: We have explained the intuition of our regularization from two perspectives. Previous works only minimize the norms of the embeddings to prevent overfitting, while our proposed regularization IVR minimizes the norms of the intermediate variables involved in the processes of computing the predicted tensor, which can further reduce the complexity of the model to prevent overfitting.

From the perspective of theoretical analysis, we further propose a theoretical framework to explain how the model mitigate the overfitting issue. We prove that our regularization IVR can minimize the Frobenius norms of the intermediate variables to minimize the overlapped trace norm of the predicted tensor as shown in Section 3.3. The overlapped trace norm can be a measure of the complexity of the predicted tensor, and minimizing it can regularize the model and prevent overfitting. We first prove Lemma 1 in Appendix B, which relates the Frobenius norm and the trace norm of a matrix. Lemma 1 shows that the trace norm of a matrix is an upper bound of a function of several Frobenius norms of intermediate variables. Based on Lemma 1, we then prove that IVR serves as an upper bound for the overlapped trace norm of the predicted tensor, which promotes the low nuclear norm of the predicted tensor to regularize the model, which is known to reduce overfitting.

We will make this statement clearer in the revision. And we want to highlight that the other three reviewers appreciate the clarity of our paper. We welcome any suggestions to improve the organization of our paper further.

Reviewer comment (Reproducibility):
The regularization approach as shown in Eq.(4) is too complex to use.

Author response:
**Q2: The regularization approach as shown in Eq.(4) is too complex to use.**

A2: Our regularization IVR is not complex to use. First, IVR is composed of several Frobenius norms of variables, which are simple to compute. Second, we provide a pseudocode for computing IVR in Appendix C. The pseudocode shows that IVR can be computed incidentally during the process of computing the predicted tensor $X$. Third, we also provide a source code for IVR in the Supplementary Material. The model.py file shows that the implementation of IVR is concise and straightforward. In summary, IVR is not complex to use and implement.

Reviewer comment (Experiments):
This paper misses some strong baselines such as [1][2]  
[1] Low-Dimensional Hyperbolic Knowledge Graph Embeddings  
[2] ER: equivariance regularizer for knowledge graph completion

Author response:
**Q3: This paper misses some strong baselines such as [1][2]. [1] Low-Dimensional Hyperbolic Knowledge Graph Embeddings. [2] ER: equivariance regularizer for knowledge graph completion.**

A3: In this paper, we mainly focus on tensor decomposition-based (TDB) models, which have wide applicability and great performance in knowledge graph completion. Therefore, we mainly include the TDB models as baselines for fair comparison. We will make the baselines more complete in the revision.

Reviewer comment (Novelty):
The main focus should be the regularization term they proposed instead of the general form. The weakness is novelty and significance, which is limited by my point of view.

Author response:
**Q1: The main focus should be the regularization term they proposed instead of the general form.**

A1: We agree that the main contribution of our paper is the regularization term, but we also argue that the general form is necessary, as it serves as a foundation for our further analysis. Without the general form, we cannot guarantee the generality of our regularization term and the theoretical analysis, as they depend on the unified view of the TDB models provided by the general form.

Reviewer comment (Novelty):
For the theoretical side, the nuclear p-norms and squared F-norm methods were well-studied. It's not a surprise that Equation 4 can generalize these, and this is not new. As the author claimed, they incorporated the N3 norm and DURA, where some of the analysis was established. Their weight IVR norm is not theoretically novel and significant. Could the author clarify what is the unique hardness and novelty of this proposed IVR norm?

Author response:
**Q2: Their weight IVR norm is not theoretically novel and significant. Could the author clarify what is the unique hardness and novelty of this proposed IVR norm?**

A2: Our proposed regularization is notable for its generality and effectiveness, as it can be applied to most TDB models and can improve their performance significantly. Although some of the theoretical analysis was established for F2, N3 and DURA, it is still challenging to design a regularization method that has both the properties of generality and effectiveness. As previous regularization methods either have poor performance (such as the F2 norm method, which achieves suboptimal results) or are not general enough (such as the N3 norm method and the DURA method, which are only suitable for CP and ComplEx models), while our method overcomes both limitations. Table 1 shows the experimental results that demonstrate the effectiveness and generality of our method. Our method can be applied to various TDB models, such as CP, ComplEx, SimplE, ANALOGY, QuatE, TuckER, etc., and achieve SOTA results. For example, TuckER with our method achieves new SOTA results on the WN18RR dataset.

Meanwhile, we provide a general theoretical framework for most TDB models in Section 3.3, which proves that our regularization method can promote low trace norm of the predicted tensor, which is known to reduce overfitting. Previous works only provide theoretical analysis for CP and ComplEx models, which limits their applicability. We discover the relationship between the overlapped trace norm and our proposed IVR regularization, and show that IVR serves as an upper bound for the overlapped trace norm. We also verify the validity of our theoretical analysis experimentally in Section 4.4. You can also refer to A3 of the rebuttal for Reviewer TVtg.

Reviewer comment (Experiments):
For the experiment side, as the author claimed in the paper, their method promotes low rank. Do you actually see the zero columns in the factors, compared to the existing method?

Author response:
**Q3: For the experiment side, as the author claimed in the paper, their method promotes low rank. Do you see the zero columns in the factors, compared to the existing methods?**

A3: We argue that a low rank matrix does not necessarily imply sparse embeddings, as a low rank matrix can have non-zero columns. For example, consider a matrix $X$ whose entries are all 1. The rank of $X$ is 1, but $X$ has no zero columns. Therefore, a better metric to measure the rank of a matrix is the overlapped trace norm, which is a convex surrogate of the rank. We experimentally verify that minimizing our proposed regularization can effectively minimize the overlapped trace norm $L(X)$ in Section 4.4. To make the experiments more detailed, we further show the results of different regularizations in the following Table 1, which demonstrate that our method is better at promoting low rank than other methods.

Table 1: The results on Kinship dataset with different regularizations. Larger MRR and lower L(X) are better.
| | MRR| L(X)|
|  ----  | ----  | ----  |
| CP | 0.894 | 26439 |
| CP-F2 | 0.900 | 25934 |
| CP-N3 | 0.900 |21801 |
| CP-DURA |0.901| 18641 |
| CP-IVR | 0.907 | 15366 |

We further analyze the sparsity of embeddings induced by different regularizations as below. Generally, there are few entries of embeddings that are exactly equal to 0 after training, which means that it is hard to obtain sparse embeddings directly. To get sparse embeddings, we can set a proper threshold and set the absolute value of the embedding entries less than the threshold to be exactly 0. Denote the ratio of zero entries as $\alpha$, then we can compare the performance of different models under the same $\alpha$. The following Table 2 shows that our regularization CP-IVR can achieve better performance than other regularizations under the same sparsity circumstance.

Table 2: The MRR results of models with different sparsity. Larger MRR is better.
| $\alpha$ | 0%| 20%| 40%| 60%| 80%|
| ---- | ---- | ---- | ----  | ---- | ---- |
| CP | 0.438 | 0.438 | 0.431 | 0.420 | 0.388 |
| CP-F2 | 0.449 | 0.449 | 0.447 | 0.445 | 0.424 |
| CP-N3 | 0.469 | 0.468 | 0.465 | 0.457 | 0.426 |
| CP-DURA | 0.471 | 0.471 | 0.469 | 0.465 |0.440 |
| CP-IVR | 0.478 | 0.478 | 0.475 |0.471 | 0.445 |

Reviewer comment (Experiments):
First, the experimental results look marginal. It might indicate that the new regularization is useful to prevent overfitting since it is a upper bound. However, what if we increase the coefficients of regularization in the baseline models? The experimental results does not show the effectiveness of the proposed new regularization.

Author response:
A2: We have shown the results of experiments with respect to the hyper-parameters in Appendix C. We analyze the impact of the hyper-parameters, the power of the Frobenius norm $\alpha$ and the regularization coefficient $\lambda_i$ in Appendix C (Table 6 to Table 9). The results show that the performance generally increases as $\alpha$/$\lambda_i$ increases and then decreases as $\alpha$/$\lambda_i$ increases. The results indicate the effectiveness of the hyper-parameters.

Reviewer comment (Theory):
Second, the proposed new regularization lacks motivation. The paper proves that the proposed new regularization is an upper bound of previous overlapped trace norm. However, the motivation of this new proposal is not well justified. Why an upper bound of the regularization is better in the optimization? It does not make sense to me.

Author response:
A3: The motivation of our regularization IVR is that we can minimize the Frobenius norms of the intermediate variables (involved in the processes of computing the predicted tensor) to minimize the overlapped trace norm of the predicted tensor. The overlapped trace norm can be a measure of the complexity of the predicted tensor, and minimizing it can regularize the model and prevent overfitting. To establish a theoretical framework for IVR, we prove Lemma 1 in Appendix B, which relates the Frobenius norm and the trace norm of a matrix. Lemma 1 shows that the trace norm of a matrix is an upper bound of a function of several Frobenius norms of intermediate variables. Based on Lemma 1, we prove that IVR serves as an upper bound for the overlapped trace norm of the predicted tensor.

From another view, since computing the trace norm has a high computational complexity of $\mathcal O(n^3)$, which is impractical for large knowledge graphs, our regularization IVR is composed of several Frobenius norms, which are computationally efficient. We prove that minimizing IVR can effectively minimize the overlapped trace norm, and we also verify this experimentally in Section 4.4. Thus, IVR serves as a good surrogate of the overlapped trace norm.

Reviewer comment (Writing):
The loss function in Section 3.2 is not well explained (Sorry, I do not see the parameters in the loss function). What are the parameters? W, H, T? I will assume $X$ indicates both the ground truth tensor value and the parameterized tensor value from W, H, T? Also, in the KGC applications, the loss function should have tensor mask to indicate the observed entries.

Author response:
A4: We can substitute Eq.(1) into the loss function to see the parameters. The parameters come from two parts, the core tensor $W$ and the embedding matrices $H$, $R$ and $T$. Please refer to Section 3.1 Paragraph “The Number of Parameters and Computational Complexity” for more details.

A5: We do not need a mask tensor to distinguish the observed and unobserved entries. We can treat every observed entry as a positive sample, and every unobserved entry as a negative sample. Then the loss function maximizes the score of the observed entries (i.e., $X_{ijk}$) and minimizes the score of the unobserved entries (i.e., $X_{ijk'}$, where $k'\neq k$). In other words, the loss function implicitly incorporates the mask tensor. Therefore, we do not need to explicitly use a mask tensor.