Multi-view Contrastive Learning for Entity Typing over Knowledge Graphs (2023)

Abstract: Knowledge graph entity typing (KGET) aims at inferring plausible types of entities in knowledge graphs. Existing approaches to KGET focus on how to better encode the knowledge provided by the neighbors and types of an entity into its representation. However, they ignore the semantic knowledge provided by the way in which types can be clustered together. In this paper, we propose a novel method called Multi-view Contrastive Learning for knowledge graph Entity Typing MCLET, which effectively encodes the coarse-grained knowledge provided by clusters into entity and type embeddings. MCLET is composed of three modules: i) Multi-view Generation and Encoder module, which encodes structured information from entity-type, entity-cluster and cluster-type views; ii) Cross-view Contrastive Learning module, which encourages different views to collaboratively improve view-specific representations of entities and types; iii) Entity Typing Prediction module, which integrates multi-head attention and a Mixture-of-Experts strategy to infer missing entity types. Extensive experiments show the strong performance of MCLET compared to the state-of-the-art

## introduction knowledge graphs (kgs) @xcite store graph-like knowledge using triples of the form (s, r, o), indicating that entities s and o are related to each other through a relation type r. kgs also contain entity type knowledge described as (e, has_type, t), denoting that entity e has type t; e.g., we can express that joe biden has type amer-ican_politician, cf. figure 1 . entity type knowledge plays a key role in various natural language processing related tasks, such as entity and relation linking @xcite , knowledge graph completion @xcite @xcite , question * contact authors e1 joe biden e2 barack obama american politician male lawyer american_politician male_politician male_lawyer american_lawyer entity-cluster view entity-type view cluster-type view homogeneous graph heterogeneous graph entity cluster type fine-grained coarse-grained entity2type link entity2cluster link type2cluster link c1 c2 c3 c4 t1 t2 t3 t3 c1 c2 c3 c4 e1 e2 t4 t1 e1 e2 t1 t2 t3 t4 c3 c4 c2 c1 figure 1 : heterogeneous graph with entities, coarsegrained clusters, and fine-grained types, covering three homogeneous views entity-type, entity-cluster, and cluster-type. a wide variety of approaches to kget have been already proposed, including embedding- @xcite , transformer- @xcite and graph neural network (gnns) based methods @xcite . each of these approaches have their own disadvantages: embedding-based methods ignore the existing neighbor information, transformer-based are computationally costly due to the use of multiple transformers to encode different neighbors and gnns-based ignore important higher-level semantic content beyond what is readily available in the graph-structure. indeed, in addition to relational and type neighbors, entity types can often be clustered together to provide coarse-grained information. importantly, this type of coarse-grain information is available in many kgs, such as yago @xcite , which provides an alignment between types and @xcite concepts. the example in figure 1 shows that between the entity joe biden and the type american_politician, there is a layer with cluster-level information american and politician. on the one hand, compared with the type content, the cluster information has a coarser granularity, which can roughly give the possible attributes of an entity and reduce the decision-making space in the entity type prediction task. on the other hand, the introduction of cluster information can enhance the semantic richness of the input kg. for example, from the type assertion (joe biden, has_type, american_politician) and the clusters american and politician corresponding to type american_politician, we can obtain new semantic connection edges between entities and clusters, i.e., (joe biden, has_cluster, american) and (joe biden, has_cluster, politician). note that as a type might belong to multiple clusters and a cluster may contain multiple types, a similar phenomenon occurs at the entity-type and entity-cluster levels, e.g, an entity might contain many clusters. through the interconnection among entities, coarse-grained clusters and fine-grained types, a dense entity-cluster-type heterogeneous graph with multi-level semantic relations can be formed. to effectively leverage the flow of knowledge between entities, clusters and types, we propose a novel method mclet, a multi-view contrastive learning model for knowledge graph entity typing, including three modules: a multiview generation and encoder module, a crossview contrastive learning module and a entity typing prediction module. the multi-view generation and encoder module aims to convert a heterogeneous graph into three homogeneous graphs entity-type, entity-cluster and cluster-type to encode structured knowledge at different levels of granularity (cf. right side of figure 1 ). to collaboratively supervise the three graph views, the crossview contrastive learning module captures the interaction by cross-view contrastive learning mechanism and mutually enhances the view-specific representations. after obtaining the embedding representations of entities and types, the entity typing prediction module makes full use of the relational and known type neighbor information for entity type prediction. we also introduce a multi-head attention with mixture-of-experts (moe) mechanism to obtain the final prediction score. our main contributions are the following: data, code, and an extended version with appendix are available at https://github.com/ zhiweihu1103/et-mclet . ## related work embedding-based methods. these methods have been introduced based on the observation that the kget task can be seen as a sub-task of the completion task (kgc). ete @xcite unifies the kget and kgc tasks by treating the entitytype pair (entity, type) as a triple of the form (entity, has_type, type). connecte @xcite builds two distinct type inference mechanisms with local typing information and triple knowledge. ## method in this section, we introduce our proposed method mclet, which consists of three components: (1) multi-view generation and encoder ( §4.1); (2) cross-view contrastive learning ( §4.2); and (3) entity typing prediction ( §4.3). ## conclusions we propose mclet, a multi-view contrastive learning (cl) framework for kget. we design three different views with different granularities, and use a cl strategy to achieve cross-view cooperatively interaction. by introducing multi-head attention with a mixture-of-experts mechanism, we can combine different neighbor prediction scores. for future work, we plan to investigate inductive scenarios, dealing with unseen entities and types.