Paper: Unified Interpretation of Smoothing Methods for Negative Sampling Loss Functions in Knowledge Graph Embedding (ICLR-2024)

Reviewer comment (Theory):
There are too many uncertainties in the contribution. Without confirmative mathematical approval, it's less convincing to argue that the conventional usages of SANS and subsampling can be covered by T-SANS, and the empirical study results also weaken this argument.

Author response:
- Our motivation is to solve the problem in KGC caused by the sparsenes of KGs with our propsed theoretically appropriate smoothing method, T-SANS.
- As stated in Section $\S1$, for doing that, we mainly contributed in:
  - By focusing on the smoothing targets, we theoretically reveal the difference between SANS and subsampling and induce a new NS loss, Triplet-based SANS (T-SANS), that can cover the smoothing target of both SANS and subsampling. 
  - We theoretically and empirically verify how our proposed method can potentially cover the conventional methods and improve KGC performance on sparse KGs.
- To suport the explanation of the above motivation, we added the explanation to $\S3.1$ in the updated version of our paper. Besides, the difference of T-SANS and conventional subsampling methods are discussed in $\S3.3$

Reviewer comment (Novelty):
The contribution is over claimed.

Author response:
- Our motivation is to solve the problem in KGC caused by the sparsenes of KGs with our propsed theoretically appropriate smoothing method, T-SANS.
- As stated in Section $\S1$, for doing that, we mainly contributed in:
  - By focusing on the smoothing targets, we theoretically reveal the difference between SANS and subsampling and induce a new NS loss, Triplet-based SANS (T-SANS), that can cover the smoothing target of both SANS and subsampling. 
  - We theoretically and empirically verify how our proposed method can potentially cover the conventional methods and improve KGC performance on sparse KGs.
- To suport the explanation of the above motivation, we added the explanation to $\S3.1$ in the updated version of our paper. Besides, the difference of T-SANS and conventional subsampling methods are discussed in $\S3.3$

Reviewer comment (Writing):
Is the focus of this study to improve interpretation or model performance of KGE?

Author response:
- The main focus is to provide a systematic interpretation, by doing this, we propose the T-SANS and the unified loss function from the viewpoint of smoothing methods that formulate our findings and even help improve the performance of KGC.

Reviewer comment (Novelty):
My main concern of the paper is limited novelty contribution. T-SANS adds the subsampling based on SANS, referring to \( p_{\theta}(x;\gamma) \) in Equation (12), which is the key difference between T-SANS and SANS. While as mentioned by the author in the footnote, Sun et al. (2019); Zhang et al. (2020b) use subsampling in their released implementation without referring to it in their paper. Thus, if I understood correctly, I would like to say the actual implementation of methods of Sun et al. (2019); Zhang et al. (2020b) is very similar to T-SANS. Thus the novelty of this paper is limited.

Author response:
- T-SANS does not applied the subsampling implemented by previous work and is completely different from SANS. Please allow me to explain the differences as below:
    - As discussed in $\S2.2$ and $\S2.3.1$, respectively, Eq. (2) denotes NS loss: \begin{equation} \ell_{\text{NS}}(\mathbf{\theta}) = -\frac{1}{|D|}\sum_{(x,y) \in D} \Bigl[\log(\sigma(s_{\mathbf{\theta}}(x,y)+\tau)) + \frac{1}{\nu}\sum_{y_{i}\sim U}^{\nu}\log(\sigma(-s_{\theta}(x,y_i)-\tau))\Bigr], \end{equation} Eq. (3) denotes conventional subsampling methods: \begin{align} &\ell_{\text{SUB}}(\mathbf{\theta}) \\ =&-\frac{1}{|D|}\!\!\sum_{(x,y) \in D} \!\!\Bigl[A(x,y;\alpha)\log(\sigma(s_{\theta}(x,y)\!+\!\tau)) \!+\!\frac{1}{\nu}\!\sum_{y_{i}\sim U}^{\nu}B(x,y;\alpha)\log(\sigma(-s_{\theta}(x,y_i)\!-\!\tau))\Bigr], \end{align} where Eq. (4) Eq. (6) Eq. (7) denotes different conventional subsampling strategies based on count in Eq. (5). From above, we can understand conventional subsampling methods mean applying term $A$ and $B$ on NS loss using counted triplets and queries frequencies. While T-SANS does not apply these couting strategies. 
    - As discussed in $\S2.3.2$ and $\S3.2$, respectively, Eq. (8) denotes SANS loss: \begin{align} &\ell_{\text{SANS}}(\mathbf{\theta}) =-\frac{1}{|D|}\sum_{(x,y) \in D} \Bigl[\log(\sigma(s_{\theta}(x,y)+\tau)) +\!\! \sum_{y_{i}\sim U}^{\nu}p_{\theta}(y_i|x;\beta)\log(\sigma(-s_{\theta}(x,y_i)-\tau))\Bigr], \end{align} Eq. (12) denotes T-SANS loss, and Eq. (13) denotes the detailed calculation for $p_{\mathbf{\theta}}(x;\gamma)$ which is innovated by us:
\begin{align}
&\ell_{\text{T-SANS}}(\mathbf{\theta}) \nonumber\\
=& -\frac{1}{|D|}\sum_{(x,y) \in D} \!\!\!p_{\theta}(x;\gamma)\Bigl[\log(\sigma(s_{\theta}(x,y)+\tau)) +\!\! \sum_{y_{i}\sim U}^{\nu}p_{\theta}(y_i|x;\beta)\log(\sigma(-s_{\theta}(x,y_i)-\tau))\Bigr],\\
&p_{\mathbf{\theta}}(x;\gamma) = \sum_{y_{i}\in D} p_{\mathbf{\theta}}(x,y_{i};\gamma),\:\:\:\:\:p_{\mathbf{\theta}}(x,y_i;\gamma) = \frac{\exp{(\gamma s_{\theta}(x,y_i))}}{\sum_{(x',y')\in D}\exp{(\gamma s_{\theta}(x',y'))}}, 
\end{align} From above, we can understand that T-SANS and SANS are completely different, with SANS only taking into account the conditional probability of negative samples and T-SANS being a loss function that considers the joint probability of the pair of queries and their answers. 
    - Thus, our proposal is novel, with the key innovation in weighting through $p_{\mathbf{\theta}}(x;\gamma)$.
- We also conducted additional experiments and illustrated the performance of T-SANS compared with conventional subsampling methods and SANS in Figure 3(a) to show their difference in essence. 
- Since the first version of our paper did not emphasize the difference between T-SANS and SANS, we added the explanation to $\S3.1$. Besides, the difference of T-SANS and conventional subsampling methods is discussed in $\S3.3$.

Reviewer comment (Theory):
The work is motivated by that conventional works use SANS and subsampling with no theoretical background, and authors believed there is room for further performance improvement. It is unclear why the lack of the theoretical background would lead to potential performance improvement.

Author response:
- Since the previous smoothing methods are not comprehensive, the lack of theoretical background will lead to mistakes in applying a proper conventional smoothing strategy. If the smoothing strategy is not proper, the performance is not optimized. Besides, the SANS does not consider the query probability thus is weak to sparse KGs. On the contary, our proposed method are built upon a comprehensive theoretical foundation that combines both subsampling and enhanced SANS, thus can help improve the KGC performance. The experimental results showing the improvement can be seen in Figure 3,4.

Reviewer comment (Writing):
Some parts of the paper are not clearly explained or are inaccurate and need further improvement, such as the \( {}^{-\alpha} \) in Equation (4) being unexplained.

Author response:
- As explained in the first version of our paper, $\alpha$ is a temperature term to adjust the frequecy of triplets and queries. Note that we incorporate $\alpha$ into Eq. (3) to consider various loss functions even though Kamigaito & Hayashi (2022a;b) do not consider $\alpha$. Eq. (4-7) shows several assumptions for deciding Eq. (3), thus the annotation of $\alpha$ holds the same meaning as explained by us when introducing Eq. (3).
- In Eq. (11), we mean directly calculating $p_\theta(x,y_i)$, because the score function in KGE model aims to approximate the plausibility of an example, so the most direct way to calculate plausibility in KGE models is using $p_\theta(x,y_i) = s_{\theta}(x,y_i)$, where softmax function is not applied. Instead, as we stated below Eq. (11), we choose to calculate this probability using the decomposition $p_{\mathbf{\theta}}(x,y)=p_{\mathbf{\theta}}(y|x)p_{\mathbf{\theta}}(x)$ so that the sum of $p_{\mathbf{\theta}}(y|x)$ of all negative samples is always 1 but the sum of $p_{\mathbf{\theta}}(x,y)$ is not always 1 by $p_{\mathbf{\theta}}(x)$. 
- Following your reminding, we have modified the captions for Figure 3 and 4.
- Considering your first review is not by the accurate understanding in our paper, would it be possible to increase the score based on the updated information? Also, we hope to have the opportunity to hear more of your feedback.

Reviewer comment (Presentation):
The caption of Figure 4 is the same as Figure 3.

Author response:
- As explained in the first version of our paper, $\alpha$ is a temperature term to adjust the frequecy of triplets and queries. Note that we incorporate $\alpha$ into Eq. (3) to consider various loss functions even though Kamigaito & Hayashi (2022a;b) do not consider $\alpha$. Eq. (4-7) shows several assumptions for deciding Eq. (3), thus the annotation of $\alpha$ holds the same meaning as explained by us when introducing Eq. (3).
- In Eq. (11), we mean directly calculating $p_\theta(x,y_i)$, because the score function in KGE model aims to approximate the plausibility of an example, so the most direct way to calculate plausibility in KGE models is using $p_\theta(x,y_i) = s_{\theta}(x,y_i)$, where softmax function is not applied. Instead, as we stated below Eq. (11), we choose to calculate this probability using the decomposition $p_{\mathbf{\theta}}(x,y)=p_{\mathbf{\theta}}(y|x)p_{\mathbf{\theta}}(x)$ so that the sum of $p_{\mathbf{\theta}}(y|x)$ of all negative samples is always 1 but the sum of $p_{\mathbf{\theta}}(x,y)$ is not always 1 by $p_{\mathbf{\theta}}(x)$. 
- Following your reminding, we have modified the captions for Figure 3 and 4.
- Considering your first review is not by the accurate understanding in our paper, would it be possible to increase the score based on the updated information? Also, we hope to have the opportunity to hear more of your feedback.

Reviewer comment (Novelty):
What is the key/significant difference between T-SANS and the actual implementation of Sun et al. (2019); Zhang et al. (2020b) methods, i.e., SANS with subsampling?

Author response:
- T-SANS does not applied the subsampling implemented by previous work and is completely different from SANS. Please allow me to explain the differences as below:
    - As discussed in $\S2.2$ and $\S2.3.1$, respectively, Eq. (2) denotes NS loss: \begin{equation} \ell_{\text{NS}}(\mathbf{\theta}) = -\frac{1}{|D|}\sum_{(x,y) \in D} \Bigl[\log(\sigma(s_{\mathbf{\theta}}(x,y)+\tau)) + \frac{1}{\nu}\sum_{y_{i}\sim U}^{\nu}\log(\sigma(-s_{\theta}(x,y_i)-\tau))\Bigr], \end{equation} Eq. (3) denotes conventional subsampling methods: \begin{align} &\ell_{\text{SUB}}(\mathbf{\theta}) \\ =&-\frac{1}{|D|}\!\!\sum_{(x,y) \in D} \!\!\Bigl[A(x,y;\alpha)\log(\sigma(s_{\theta}(x,y)\!+\!\tau)) \!+\!\frac{1}{\nu}\!\sum_{y_{i}\sim U}^{\nu}B(x,y;\alpha)\log(\sigma(-s_{\theta}(x,y_i)\!-\!\tau))\Bigr], \end{align} where Eq. (4) Eq. (6) Eq. (7) denotes different conventional subsampling strategies based on count in Eq. (5). From above, we can understand conventional subsampling methods mean applying term $A$ and $B$ on NS loss using counted triplets and queries frequencies. While T-SANS does not apply these couting strategies. 
    - As discussed in $\S2.3.2$ and $\S3.2$, respectively, Eq. (8) denotes SANS loss: \begin{align} &\ell_{\text{SANS}}(\mathbf{\theta}) =-\frac{1}{|D|}\sum_{(x,y) \in D} \Bigl[\log(\sigma(s_{\theta}(x,y)+\tau)) +\!\! \sum_{y_{i}\sim U}^{\nu}p_{\theta}(y_i|x;\beta)\log(\sigma(-s_{\theta}(x,y_i)-\tau))\Bigr], \end{align} Eq. (12) denotes T-SANS loss, and Eq. (13) denotes the detailed calculation for $p_{\mathbf{\theta}}(x;\gamma)$ which is innovated by us:
\begin{align}
&\ell_{\text{T-SANS}}(\mathbf{\theta}) \nonumber\\
=& -\frac{1}{|D|}\sum_{(x,y) \in D} \!\!\!p_{\theta}(x;\gamma)\Bigl[\log(\sigma(s_{\theta}(x,y)+\tau)) +\!\! \sum_{y_{i}\sim U}^{\nu}p_{\theta}(y_i|x;\beta)\log(\sigma(-s_{\theta}(x,y_i)-\tau))\Bigr],\\
&p_{\mathbf{\theta}}(x;\gamma) = \sum_{y_{i}\in D} p_{\mathbf{\theta}}(x,y_{i};\gamma),\:\:\:\:\:p_{\mathbf{\theta}}(x,y_i;\gamma) = \frac{\exp{(\gamma s_{\theta}(x,y_i))}}{\sum_{(x',y')\in D}\exp{(\gamma s_{\theta}(x',y'))}}, 
\end{align} From above, we can understand that T-SANS and SANS are completely different, with SANS only taking into account the conditional probability of negative samples and T-SANS being a loss function that considers the joint probability of the pair of queries and their answers. 
    - Thus, our proposal is novel, with the key innovation in weighting through $p_{\mathbf{\theta}}(x;\gamma)$.
- We also conducted additional experiments and illustrated the performance of T-SANS compared with conventional subsampling methods and SANS in Figure 3(a) to show their difference in essence. 
- Since the first version of our paper did not emphasize the difference between T-SANS and SANS, we added the explanation to $\S3.1$. Besides, the difference of T-SANS and conventional subsampling methods is discussed in $\S3.3$.

Reviewer comment (Presentation):
Should the caption of Figure 4 be updated?

Author response:
- Following your reminding, we have modified the captions for Figure 3 and 4.

Reviewer comment (Writing):
While the paper provides a comprehensive theoretical understanding of smoothing methods for NS loss in KGE, it might be too dense for a broader audience. The depth of the theoretical content might make it less accessible to practitioners or researchers from adjacent fields.

Author response:
- Please allow me to cite an example to show the difference between our T-SANS and conventional smoothing mothods: Our T-SANS can work even when the entity or relations included in the target triplet appear more than once, which is theoretically different from conventional approaches.  
- Following your suggestions, we acknowledged the importance of accessibility and added this example in $\S3.3$ to make it easier for a wider audience to comprehend and apply our methods.

Reviewer comment (Experiments):
How does T-SANS handle extremely sparse datasets compared to other methods? Is there a threshold of sparsity beyond which T-SANS might not be as effective?

Author response:
- T-SANS can effectively leverage model-predicted probabilities, enabling it to handle both normally distributed and extremely sparse datasets. In contrast, baseline methods struggle to adequately consider the frequency of sparse triples, resulting in sub-optimal performance on such challenging datasets.  
- To thoroughly evaluate T-SANS, we conducted experiments on three extremely sparse datasets derived from FB15k-237, WN18RR, and YAGO3-10. As outlined in $\S6$ and illustrated in Figure 4, T-SANS consistently demonstrates improved Knowledge Graph Completion (KGC) performance in scenarios where KGs are extremely sparse. Further, by comparing with the results showcased in Figure 3, we can understand the trend that the greater the dataset sparsity, the more T-SANS outperforms conventional methods.  
- Following your viewpoint, we have added a concise comparision when analyzing in $\S6$ and moved the picture positions for better accessibility.

Reviewer comment (Novelty):
In my view, the proposed method is incremental work based on previous studies. It is an extension of SANS.

Author response:
- Although the name T-SANS gives the impression that it is just an extension of SANS, this name was given out of respect for SANS. In fact, they are completely different, with SANS only taking into account the conditional probability of negative samples and T-SANS being a loss function that considers the joint probability of the pair of queries and their answers. This difference is large from the viewpoint of loss functions [1].
    - [1]: Zhuang Ma, Michael Collins. "Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency", https://aclanthology.org/D18-1405/
- We conducted experiments and illustrated the performance differences of T-SANS compared with SANS in Figure 3, which verified their difference in the actual data. 
- However, Reviewer @4kKk also misunderstood this point, so we added a concise explanation emphasizing the difference in $\S3.1$.

Reviewer comment (Experiments):
Another weakness is that the selected KGE models in the experiments are old. Some popular or recent models, such as TuckER [1] and HousE [2], are not included, which, in my view, may weaken the soundness the work.

Author response:
- Following your suggestion, we are trying to implement TuckER and HouseE now. Would it be possible to increase the score based on the results of the additional experiments?

Reviewer comment (Experiments):
Why are some results on YGAO3-10 missing? I think it would be better to produce the results using open-source implementations.

Author response:
- Since YAGO3-10 is larger than FB15k-237 and WN18RR, it requires a lot of computational resources. To deal with this problem, we made the decision not to train the YAGO3-10 dataset on all models but only on two best-performing models whose efficient hyperparameters on YAGO3-10 are provided by previous work.

Reviewer comment (Experiments):
Is it possible to provide any analysis or experimental results to assess the effect of negative sampling on the convergence rate?

Author response:
- Following your valuable suggestion, we believe that this analysis will further provide valuable insights and enhancing the credibility of our work, thus we are wholeheartedly committed to conducting this analysis. Would it be possible to increase the score based on the results of the additional analysis?