Alleviating Sparsity of Open Knowledge Graphs with Ternary Contrastive Learning (2022)

Abstract: Sparsity of formal knowledge and roughness of non-ontological construction make sparsity problem particularly prominent in Open Knowledge Graphs (OpenKGs). Due to sparse links, learning effective representation for few-shot entities becomes difficult. We hypothesize that by introducing negative samples, a contrastive learning (CL) formulation could be beneficial in such scenarios. However, existing CL methods model KG triplets as binary objects of entities ignoring the relation-guided ternary propagation patterns and they are too generic, i.e., they ignore zero-shot, few-shot and synonymity problems that appear in OpenKGs. To address this, we propose TernaryCL, a CL framework based on ternary propagation patterns among head, relation and tail. TernaryCL designs Contrastive Entity and Contrastive Relation to mine ternary discriminative features with both negative entities and relations, introduces Contrastive Self to help zero- and few-shot entities learn discriminative features, Contrastive Synonym to model synonymous entities, and Contrastive Fusion to aggregate graph features from multiple paths. Extensive experiments on benchmarks demonstrate the superiority of TernaryCL over state-of-the-art models.

## introduction open knowledge graphs (openkgs) structure textual facts in the form of (subject entity, relation, object entity) without depending on an ontology schema @xcite . they benefit knowledge-intensive tasks, such as question answering @xcite and dialogue systems @xcite . representation learning of openkgs aims to learn implicit embeddings of entities and relations @xcite , and has become an indispensable step in the applications of openkgs. due to the rigidness of grammatical patterns and roughness of non-ontological construction, a common challenge in representation learning of openkgs is the sparsity problem, where a large portion of entities have few-or zero-shot links. in the two standard openkgs: reverb20k and re-verb45k @xcite , the degree of 55% and 89% entities is less than 3. as such, fewand zero-shot entities in these openkgs do not get enough training, resulting in poor generalization. we hypothesize that negative samples could help existing sparse links to learn discriminative features in the form of a negative feedback. being popular in self-supervised learning, contrastive learning (cl) aims to learn representations by contrasting negative samples with the positive ones @xcite @xcite . although existing cl in graphs @xcite and curatedkgs @xcite have shown promising results, they could not effectively tackle the sparsity and synonymity problems of openkgs. first, they model kg triplets as binary objects of entities, ignoring the relationguided ternary propagation patterns where entities propagate to multiple neighbor entities through multiple relations. second, they do not specifically tackle the sparsity issues of zero-and few-shot entities. finally, they can not address the synonymity problem in openkgs where multiple entities with different surface forms have the same meaning. to alleviate the above problems, we propose ternarycl, a contrastive learning framework based on ternary relational patterns among head, relation and tail. ternarycl uses the following key ideas: (1) contrastive entity learns ternary discriminative features of different entities under the same (head entity, relation)-pair, which alleviates the sparsity issue by negative entities (fig. 1b ); (2) contrastive relation learns ternary discriminative features of different relations under the same (head entity, tail entity)-pair, which alleviates the sparsity by negative relations (fig. 1c ); (3) contrastive self designs self positive samples to give zero-and in summary, our key contributions are: ## related work openkgs represent factual knowledge in structured forms, as triples of head-relation-tail or (h, r, t). they are extracted with openie tools @xcite , and generally do not rely on specification of ontology. although openkgs have the advantage that they can be easily bootstrapped to new domains, because of the sparsity of formal grammatical patterns and non-ontological construction, many relevant facts are often missing from such openkgs. this makes them difficult to be used effectively in downstream tasks @xcite . representation learning of kgs devotes to learning informative features of entities and relations, which can be used for other kg-related downstream tasks. general representation learning models over kgs focus on inducing structural features with linear @xcite , bilinear @xcite , complex @xcite or convolutional @xcite operations, while openkg-specific models enhance the embeddings with side information @xcite and pretrained language models (chandrahas and talukdar, 2021). however, these methods are limited in alleviating the sparsity issue. we propose a contrastive learning method that is more effective and does not rely on external resources. contrastive learning aims to learn effective representation by pulling close neighbors and pushing apart non-neighbors @xcite , which has achieved great success in vision @xcite ), text @xcite and graph @xcite . contrastive learning methods in graphs attempt to leverage a contrastive loss at node @xcite , graph @xcite and multi-view levels @xcite . @xcite al. (2020) and wang et al. (2022) introduce contrastive learning into curatedkgs by designing negative sampling strategies. however, existing contrastive learning on graphs and curatedkgs only model binary objects. in contrast, we propose contrastive learning to study ternary patterns in openkgs. ## proposed ternarycl model we propose ternarycl to alleviate the sparsity of openkgs in representation learning (fig. 1 ). in the following, we first introduce a simple ternary similarity function to compute a similarity score for each (h, r, t) triple in an openkg by considering both textual and structural information ( §4.1). we present contrastive entity in §4.2 to learn embeddings of different entities with the same (h, r)-pair, followed by contrastive relation in §4.3 to learn embeddings of different relations with the same (h, t)-pair. we describe contrastive self in §4.4 that constructs a positive sample (h, r sf , h + ) and contrasts it with negative samples to give zero-and few-shot entities chances to learn informative features. we present contrastive synonym in §4.5 to aggregate signals from multiple synonymous entities. contrastive fusion in §4.6 futher extends propagation patterns from the above 1-1 to 1-n. finally, the training procedure is described in §4.7. ## conclusion in this work, we have provided empirical insights about the sparsity problem of openkgs, and proposed ternarycl, a contrastive learning framework to alleviate the sparsity by introducing contrastive entity, relation, self, synonym and fusion methods. through extensive experiments and comprehensive analysis, we have shown that ternarycl outperforms state-of-the-art baselines by a good margin.