Unified Interpretation of Smoothing Methods for Negative Sampling Loss Functions in Knowledge Graph Embedding (2024)

Abstract: Knowledge Graphs (KGs) are fundamental resources in knowledge-intensive tasks in NLP. Due to the limitation of manually creating KGs, KG Completion (KGC) has an important role in automatically completing KGs by scoring their links with KG Embedding (KGE). To handle many entities in training, KGE relies on Negative Sampling (NS) loss that can reduce the computational cost by sampling. Since the appearance frequencies for each link are at most one in KGs, sparsity is an essential and inevitable problem. The NS loss is no exception. As a solution, the NS loss in KGE relies on smoothing methods like Self-Adversarial Negative Sampling (SANS) and subsampling. However, it is uncertain what kind of smoothing method is suitable for this purpose due to the lack of theoretical understanding. This paper provides theoretical interpretations of the smoothing methods for the NS loss in KGE and induces a new NS loss, Triplet Adaptive Negative Sampling (TANS), that can cover the characteristics of the conventional smoothing methods. Experimental results of TransE, DistMult, ComplEx, RotatE, HAKE, and HousE on FB15k-237, WN18RR, and YAGO3-10 datasets and their sparser subsets show the soundness of our interpretation and performance improvement by our TANS.

## introduction knowledge graphs (kgs) represent human knowledge using various entities and their relationships as graph structures. kgs are fundamental resources for knowledge-intensive tasks like dialog @xcite , question answering @xcite , named entity recognition @xcite , open-domain questions @xcite , and recommendation systems @xcite , etc. however, to create complete kgs, we need to consider a large number of entities and all their possible relationships. taking into account the explosively large number of combinations between entities, only relying on manual approaches is unrealistic to make complete kgs. knowledge graph completion (kgc) is a task to deal with this problem. kgc involves automatically completing missing links corresponding to relationships between entities in kgs. to complete the kgs, we need to score each link between entities. for this purpose, current kgc commonly relies on knowledge graph embedding (kge) @xcite . kge models predict the missing relations, named link prediction, by learning structural representations. in the current kge, models need to complete a link (triplet) (e i , r k , e j ) of entities e i and e j , and their relationship r k by answering e i or e j from a given query (?, r k , e j ) or (e i , r k , ?), respectively. hence, kge needs to handle a large number of entities and their relationships during its training. to handle a large number of entities and relationships in kgs, negative sampling (ns) loss @xcite is frequently used for training kge models. the original ns loss is proposed to approximate softmax cross-entropy loss to reduce computational costs by sampling false labels from its noise distribution in training. @xcite import the ns loss from word embedding to kge with utilizing uniform distribution as its noise distribution. @xcite extend the ns loss to self-adversarial negative sampling (sans) loss for efficient training of kge. unlike the ns loss with uniform distribution, the sans loss utilizes the training model's prediction as the noise distribution. since the negative samples in the sans loss become more difficult to discriminate for models in training, the sans can extract models' potential compared with the ns loss with uniform distribution. from the long-tail distribution of this figure, we can understand that both queries and answers necessary for training kge models may suffer from the sparsity problem. as a solution, several smoothing methods are used in kge. @xcite import subsampling from word2vec @xcite to kge. subsampling can smooth the appearance frequency of triplets and queries in kgs. @xcite show a general formulation that covers the basic subsampling of @xcite (base), their frequency-based subsampling (freq) and uniquebased subsampling (uniq) for kge. @xcite indicate that sans has a similar effect of using label-smoothing @xcite and thus sans can smooth the frequencies of answers in training. figure 2 shows the effectiveness of sans and subsampling in kgc performance. from the figure, since fb15k-237 is more sparse (imbalanced) than wn18rr and yago3-10 based on figure 1 , we can understand that strategy in choosing smoothing methods have more considerable influences than models when data is sparse. while sans and subsampling can improve model performance by smoothing the appearance frequencies of triplets, queries, and answers, their theoretical relationship is not clear, leaving their capabilities and deficiencies a question. for example, conventional works @xcite ; kamigaito and hayashi, 2022a) foot_0 jointly use sans and subsampling with no theoretical background. thus, there is a call for further interpretability and performance improvement. to solve the above problem, we theoretically and empirically study the differences of sans and subsampling on three common datasets and their sparser subsets with six popular kge models foot_1 . our contributions are as follows: ## background in this section, we describe the problem formulation for solving kgc by kge and explain the conventional ns loss functions in kge. ## triplet adaptive negative sampling in this section, we explain our proposed triplet adaptive negative sampling (tans) in detail. we first show the overview of our tans through the comparison with the conventional smoothing methods of the ns loss for kge (see §2.3) in §3.1 and after that we explain the details of tans through its mathematical formulations in §3.2 and §3.3. ## unified interpretation of sans and subsampling in the previous section, we understand that our tans can smooth triplets, queries, and answers partially covered by sans and subsampling methods. on the other hand, tans only relies on modelpredicted frequencies to smooth the frequencies. temperature induced ns loss α β γ = 0 = 0 = 0 equivalent to ℓns(θ), the basic ns loss in kge (eq. ( 2 )) = 0 = 0 ̸ = 0 currently does not @xmath0, the sans loss (eq. ( 9 )) = 0 ̸ = 0 ̸ = 0 equivalent to our ℓtans(θ), the tans loss (eq. ( 12 )) ̸ = 0 = 0 = 0 proportional to ℓns(θ), the basic ns loss in kge (eq. ( 2 )) with subsampling in §2.3 ̸ = 0 = 0 ̸ = 0 currently does not exist ̸ = 0 ̸ = 0 = 0 proportional to ℓsans(θ), the sans loss (eq. ( 9 )) with subsampling in §2.3 ̸ = 0 ̸ = 0 ̸ = 0 equivalent to our ℓuni(θ), the unified ns loss in kge (eq. ( 16 )) and also equivalent to our ℓtans(θ), the tans loss (eq. ( 12 )) with subsampling in §2. 3 neubig and dyer (2016) point out the benefits of combining count-based and model-predicted frequencies in language modeling. this section integrates smoothing methods for the ns loss in kge from a unified interpretation. ## experiments in this section, we investigate our theoretical interpretation in §3.3 and §4.2 through experiments. ## analysis we analyze how tans mitigates the sparsity problem in imbalanced kgs commonly caused by low frequent triplets in kgc. by considering that all triplets in kgs appear at most once, we focus on queries. we extracted 0.5% triplets with the highest or lowest frequent queries in training, validation, and test splits as the sparser subsets fb15k-237-hl, wn18rr-hl, and yago3-10-hl, respectively 8 from original data, for the investigation. figure 4 shows mrrs for each model on each sparser dataset. from the result, we can understand that tans can perform even much better in kgc when kgs get more imbalanced. you can see further detailed results in ## related work knowledge graph knowledge graphs have important roles in various knowledge-intensive nlp tasks like dialog @xcite , question answering @xcite , named entity recognition @xcite , open-domain questions @xcite , recommendation systems @xcite , and commonsense reasoning @xcite , etc. in addition to these text-only tasks, knowledge-intensive vision and language (v&l) tasks such as visual question answering (vqa) @xcite , image generation (kamigaito et al., 2023), explanation generation @xcite , and image review generation @xcite also require external knowledge. visual kgs @xcite have the potential to contribute to solving these tasks. therefore, kgs are important materials in various different fields. knowlege graph completion even though kgs are useful, their sparsity is a fundamental prob-8 note that we show their appearance frequencies of queries and answers in the training data in figure 5 and detailed statistics in ## conclusion we reveal the relationships between sans loss and subsampling for the kg completion task through theoretical analysis. we explain that sans loss and subsampling under three assumptions, base, freq, and uniq have similar roles to mitigate the sparseness problem of queries and answers of kgs by smoothing the frequencies of queries and answers. furthermore, based on our interpretation, we induce a new loss function, triplet adaptive negative sampling (tans), by integrating sans loss and subsampling. we also introduce a theoretical interpretation that tans with subsampling can cover all conventional combinations of sans loss and subsampling. we verified our interpretation by empirical experiments in three common datasets, fb15k-237, wn18rr, and yago3-10, and six popular kge models, transe, distmult, complex, ro-tate, hake, and house. the experimental results show that our tans loss can outperform subsampling and sans loss with many models in terms of mrr as expected by our theoretical interpretation. furthermore, the combinatorial use of tans and subsampling achieved comparable or better performance than other combinations and showed the validity of our theoretical interpretation that tans with subsampling can cover all conventional combinations of sans loss and subsampling in kge.