Improving Knowledge Graph Completion with Generative Hard Negative Mining (2023)

Abstract: Contrastive learning has recently shown great potential to improve text-based knowledge graph completion (KGC). In this paper, we propose to learn a more semantically structured entity representation space in text-based KGC via hard negatives mining. Specifically, we novelly leverage a sequence-to-sequence architecture to generate high-quality hard negatives. These negatives are sampled from the same decoding distributions as the anchor (or correct entity), inherently being semantically close to the anchor and thus enjoying good hardness. A self-information-enhanced contrasting strategy is further incorporated into the Seq2Seq generator to systematically diversify the produced negatives. Extensive experiments on three KGC benchmarks demonstrate the sound hardness and diversity of our generated negatives and the resulting performance superiority on KGC.

## introduction knowledge graph (kg) is an efficient method of representing global knowledge @xcite , playing a fundamental role in many natural language processing (nlp) tasks like question answering @xcite , recommender systems @xcite , and web search @xcite , etc. the knowledge graph is composed of triples (h, r, t), where h, t, and r denote a head entity, a tail entity, and their relation, respectively. modern public kgs such as freebase, yago, and wikidata, although covering massive knowledge with a large number of entities, are inevitably incomplete. therefore, knowledge graph completion (kgc) has become a popular area of research in recent years @xcite . generally, two types of methods are applied for this task. embedding-based methods assign each entity/relation with a dense vector trained † corresponding authors. ## methodology the link prediction task of kgc is to infer the missing triples given an incomplete g. in this section, we mathematically describe the proposed generative hard negative mining (ghn) method in detail. the core idea of ghn is to obtain better negatives in terms of vicinity and uniformity. we first introduce the overall process of ghn in 2.1 which includes how to generate and use these negative samples to enhance kgc model training. then, to further improve the vicinity and uniformity of generated negatives, we propose a self-informationenhanced token-level contrastive learning method which is discussed in 2.2. finally, we introduce how to facilitate inference on kgc tasks in 2.3. ## related work text-based kgc methods aims to leverage text information in kgs to assist kgc. dkrl @xcite ) obtained text representations by cnn. kepler @xcite leveraged the typical kge training objective into the masked language model training. mtl-kgc @xcite proposed to learn more relational properties in kgs with multi-task learning method. kg-bert @xcite and star @xcite leveraged pre-trained language models to produce entity embeddings in the cross-encoder style and biencoder style, respectively. simkgc @xcite increase the number of negatives by incorporating three types of negatives and achieved notable performance improvements. negative mining aims to find property negatives to assist contrastive representation learning @xcite . the most effective way is to use the samples within the same mini-batch for negative candidates @xcite . another effective and widely used method is to store negative samples with an asynchronous update mechanism @xcite , which allows more negative candidates to be involved during training. to make contrastive pairs difficult to discriminate, @xcite ; chen et al. (2017); xiong et al. (2020) compute propensity score for @xmath0, sample candi-@xmath1. ying et al. (2018) further utilizes page-rank score to calculate weights for negative candidates. motivated by the generative adversarial networks (goodfellow et al., 2014), wang et al. (2020) proposed a sampling strategy by adaptively receiving knowledge-aware rewards. and hu et al. (2020) proposed to adversarially generate the hard negative samples together with the representation network. ## conclusion we have presented our method for kgc tasks, which incorporate generative methods with a novel self-information-enhanced training strategy to produce high-quality negatives. and we further reveal that the proposed method systematically balances uniformity and vicinity, two essential properties for negative sample selection. empirical results on three widely-used datasets (wn18rr, fb15k-237, wikidata5m) have verified the superiority of our method.