Unified Interpretation of Softmax Cross-Entropy and Negative Sampling: With Case Study for Knowledge Graph Embedding (2021)

Abstract: In knowledge graph embedding, the theoretical relationship between the softmax cross-entropy and negative sampling loss functions has not been investigated. This makes it difficult to fairly compare the results of the two different loss functions. We attempted to solve this problem by using the Bregman divergence to provide a unified interpretation of the softmax cross-entropy and negative sampling loss functions. Under this interpretation, we can derive theoretical findings for fair comparison. Experimental results on the FB15k-237 and WN18RR datasets show that the theoretical findings are valid in practical settings.

## introduction negative sampling (ns) @xcite is an approximation of softmax cross-entropy (sce). due to its efficiency in computation cost, ns is now a fundamental loss function for various natural language processing (nlp) tasks such as used in word embedding @xcite , language modeling @xcite , contextualized embedding @xcite , and knowledge graph embedding (kge) @xcite . specifically, recent kge models commonly use ns for training. considering the current usages of ns, we investigated the characteristics of ns by mainly focusing on kge from theoretical and empirical aspects. first, we introduce the task description of kge. a knowledge graph is a graph that describes the relationships between entities. it is an indispensable resource for knowledge-intensive nlp applications such as dialogue @xcite and questionanswering @xcite systems. however, to create a knowledge graph, it is necessary to consider a large number of entity combinations and their relationships, making it difficult to construct a complete graph manually. therefore, the prediction of links between entities is an important task. currently, missing relational links between entities are predicted using a scoring method based on kge @xcite . with this method, a score for each link is computed on vector space representations of embedded entities and relations. we can train these representations through various loss functions. the sce @xcite and ns @xcite loss functions are commonly used for this purpose. several studies @xcite have shown that link-prediction performance can be significantly improved by choosing the appropriate combination of loss functions and scoring methods. however, the relationship between the sce and ns loss functions has not been investigated in kge. without a basis for understanding the relationships among different loss functions, it is difficult to make a fair comparison between the sce and ns results. we attempted to solve this problem by using the bregman divergence @xcite to provide a unified interpretation of the sce and ns loss functions. under this interpretation, we can understand the relationships between sce and ns in terms of the model's predicted distribution at the optimal solution, which we called the objective distribution. by deriving the objective distribution for a loss function, we can analyze different loss functions, the objective distributions of which are identical under certain conditions, from a unified viewpoint. we summarize our theoretical findings not restricted to kge as follows: to check the validity of the theoretical findings in practical settings, we conducted experiments on the fb15k-237 @xcite and wn18rr @xcite datasets. the experimental results indicate that ## understanding loss functions for fair comparisons we summarize the theoretical findings from sections 2, 3, and 4 in ## experiments and discussion we conducted experiments to investigate the validity of what we explained in section 5 through a comparison of the ns and sce losses. ## conclusion we revealed the relationships between sce and ns loss functions in kge. through theoretical analysis, we showed that sce and ns w/ uni are equivalent in objective distribution, which is the predicted distribution of a model at an optimal solution, and that sce w/ ls and sans have similar objective distributions. we also showed that sce more strongly fits a model to the training data than ns due to the divergence and convexity of sce. the experimental results indicate that the differences in the divergence of the two losses were not large enough to affect dataset differences. the results also indicate that sce works well with highly flexible scoring methods, which do not have any bound of the scores, while ns works well with rotate, which cannot express positive values due to its bounded scoring. moreover, they indicate that sce and sans work better in pre-training than ns w/ uni, commonly used for learning word embeddings. for future work, we will investigate the properties of loss functions in out-of-domain data.