Open Knowledge Graphs Canonicalization using Variational Autoencoders (2021)

Abstract: Noun phrases and Relation phrases in open knowledge graphs are not canonicalized, leading to an explosion of redundant and ambiguous subject-relation-object triples. Existing approaches to solve this problem take a two-step approach. First, they generate embedding representations for both noun and relation phrases, then a clustering algorithm is used to group them using the embeddings as features. In this work, we propose Canonicalizing Using Variational AutoEncoders and Side Information (CUVA), a joint model to learn both embeddings and cluster assignments in an end-to-end approach, which leads to a better vector representation for the noun and relation phrases. Our evaluation over multiple benchmarks shows that CUVA outperforms the existing state-of-the-art approaches. Moreover, we introduce CanonicNell, a novel dataset to evaluate entity canonicalization systems.

## introduction open information extraction (openie) methods @xcite can be used to extract triples in the form (noun phrase, relation phrase, noun phrase) from given text corpora in an unsupervised way without requiring a pre-defined ontology schema. this makes them suitable to build large open knowledge graphs (openkgs) from huge collections of unstructured text documents, thereby making the usage of ope-nie methods highly adaptable to new domains. although openie methods are highly adaptable, one major shortcoming of openkgs is that noun phrases (nps) and relation phrases (rps) are not canonicalized. this means that two nps (or rps) having different surface forms, but referring to the same entity (or relation) in a canonical kb, are treated differently. consider the following triples as an example: (nbc-tv, has headquarters in, nyc), (nbc television, is in, new york city) and (nbc-tv, has main office in, nyc). looking at the previous example, both openie methods and associated open kgs would not have any knowledge that nyc and new york city refer to the same entity, or has headquarters in and has main office in are similar relations. moreover, while it is true that similar relations will have same argument types (see the previous example), the converse need not hold true. for example, given the following two triples (x, is born in, y) and (x, has died in, y) in an open kg, where x is of type person and y is of type location, does not imply is born in and has died in are similar relations. thus, the task of canonicalizing nps and rps within an open kg is significant. otherwise, open kgs will have an explosion of redundant facts, which is highly undesirable, for the following reasons. firstly, redundant facts use a higher memory footprint. secondly, querying an open kg is likely to yield sub-optimal results, for e.g. it will not return all facts associated with nyc when using new york city as the query. finally, allowing downstream applications such as link prediction @xcite to know that nyc and new york city refers to the same entity, will improve their performance while operating on large open kgs. hence, it is imperative to canonicalize nps and rps within an open kg. in this paper, we introduce canonicalizing using variational autoencoders (cuva), a neural network architecture that learns unique embeddings for nps and rps as well as cluster assignments in a joint fashion. cuva combines a) the variational deep embedding (vade) framework @xcite , a generative approach to clustering, and b) a kg embedding model that aims to utilize the structural knowledge present within the open kg. in addition, cuva uses additional contextual information obtained from the documents used to build the open kg. the input to cuva is a) an open kg expressed as a list of triples and b) contextual information obtained from the documents. the output is a set of np and rp clusters grouping all items together that refer to the same entity (or relation). ## related work extracting triples from sentences is the first step to build open kgs. the openie technique has been originally introduced in @xcite . thereafter, several approaches have been proposed to improve the quality of the extracted triples. rulebased approaches, such as reverb @xcite and @xcite , use patterns on top of syntactic features to extract relation phrases and their arguments from text. learningbased methods, such as ollie @xcite and rnnoie @xcite , train a self-supervised system using bootstrapping techniques. clause-based approaches @xcite navigate through the dependency trees to split the sentences into simpler and independent segments. there have been several previous works to group nps and rps into coherent clusters. a traditional approach to canonicalize nps is to map them to an existing kb such as wikidata, also referred to as the entity linking (el) task @xcite . a major problem with these el approaches is that many nps may refer to entities that are not present in the kb, in which case they are not clustered. the resolver system @xcite uses string similarity features to cluster phrases in textrunner @xcite triples. @xcite uses manually defined features for np canonicalization, and subsequently performs relation phrase clustering by using amie algorithm @xcite . @xcite propose a modification to the previous approach by using pruning and bounding techniques. concept resolver @xcite , which makes "one sense per category" assumption, is used for clustering np mentions in nell. this approach requires additional information in the form of a schema of relation types. kb-unify @xcite addresses the problem of unifying multiple canonical and open kgs into one kg, but requires additional sense inventory, which may not be available. the cesi architecture @xcite ) models the canonicalization task in a two-step pipeline approach, i.e., in the first step, it uses a hole algorithm @xcite to learn embeddings for nps (and rps), and then in an independent second step, it "plugs" these learned embeddings into a hierarchical agglomerative clustering (hac) algorithm to generate clusters. currently, cesi is the state of the art on this task. unlike cesi, our proposed model cuva learns the embedding representations and the cluster assignments of both nps and rps in an end-to-end manner, using a single model. ## open kgs canonicalization using vae formally, the canonicalization task is defined as follows: given a list of triples @xmath0, r, t) from an openie system o on a document collection c, where h, t are noun phrases (nps) and r is a relation phrase (rp), the objective is to cluster nps (and rps), so that items referring to the same entity (or relation) are in the same cluster. we assume that each cluster corresponds to either a latent entity or a latent relation; the label of such a latent entity/relation is unknown to the learner. cuva uses two variational autoencoders i. fig. 1 illustrates an instantiation for cuva. a description of each of the components of cuva follows below. ## evaluation the canonicalization task is inherently unsupervised, i.e. we are not given any manually annotated data for training. with this in mind, we train the cuva model according to the procedure described in section b of the appendix and then evaluate our approach on the entity canonicalization task only. we do not include quantitative evaluations on the relation canonicalization task, as none of the benchmarks described below have ground-truth annotations for canonicalizing relations, leaving the creation of a dataset for relation clustering as an interesting future work. ## qualitative analysis the bottom four rows in ## further analysis in this section, we analyze cuva under three different configurations. section 6.1 compares how cuva performs against pretrained language models. section 6.2 analyzes the effect of ablating components from our proposed network architecture. finally, section 6.3 demonstrates the effectiveness of a joint learning approach over a pipeline-based strategy using the same network architecture. ## conclusion in this paper, we introduced cuva, a novel neural architecture to canonicalize noun phrases and relation phrases within an open kg. we argued that cuva learns unique mention embeddings and cluster assignments in a joint fashion, compared to a pipeline strategy followed by the current state of the art methods. moreover, we also introduced canonicnell, a new dataset for entity canonicalization. an evaluation over four benchmarks demonstrates the effectiveness of cuva over state of the art baselines.