COMBO : A Complete Benchmark for Open KG Canonicalization (2023)

Abstract: Open knowledge graph (KG) consists of (subject, relation, object) triples extracted from millions of raw text. The subject and object noun phrases and the relation in open KG have severe redundancy and ambiguity and need to be canonicalized. Existing datasets for open KG canonicalization only provide gold entity-level canonicalization for noun phrases. In this paper, we present COMBO, a Complete Benchmark for Open KG canonicalization. Compared with existing datasets, we additionally provide gold canonicalization for relation phrases, gold ontology-level canonicalization for noun phrases, as well as source sentences from which triples are extracted. We also propose metrics for evaluating each type of canonicalization. On the COMBO dataset, we empirically compare previously proposed canonicalization methods as well as a few simple baseline methods based on pretrained language models. We find that properly encoding the phrases in a triple using pretrained language models results in better relation canonicalization and ontology-level canonicalization of the noun phrase. We release our dataset, baselines, and evaluation scripts at path/to/url.

## introduction large ontological knowledge graphs (kg) such as wikidata @xcite , dbpedia @xcite , freebase @xcite use a complex ontology to formalize and organize all the entities and relations. figure 1(a) shows an example ontological knowledge graph (wikidata): "joe biden (q6279)" is categorized as "human (q5)" in wikidata and linked to "scranton (q271395)" with relation "birth place (p19)", where prefix q and p denote unique identities for entity and relation respectively in wikidata foot_1 . as ontological kgs are well organized and canonicalized, one can efficiently query information and extract knowledge from them to assist nlp models in various tasks @xcite @xcite @xcite @xcite . however, building and maintaining an accurate ontological kg requires large human effort @xcite . in contrast, open knowledge graphs such as re-verb @xcite and ollie @xcite are built using (subject, relation, object) triples automatically extracted from millions of raw text by openie systems @xcite @xcite . they are frequently used to assist in building ontological kgs @xcite and slot filling @xcite . as openie systems do not rely on pre-defined ontologies or human supervision, the extracted triples contain noun phrases (nps) and relation phrases (rps) that are not canonicalized. take the open kg triples shown in figure 1 (b) as an example. the np "joseph biden" and "biden" both refer to the us president joe biden, but the open kg regards them as two different nodes because of their different surface forms. on the other hand, "was born in" in the first and second triple means "birth place of" and "birth time of" respectively, but the open kg cannot disambiguate them. these examples reveal the redundancy and ambiguity of uncanonicalized open kg @xcite , which makes querying open kg inaccurate and inefficient. to this end, open kg canonicalization aims to improve the quality of open kgs to the level of ontological kgs. it is therefore different from tasks such as entity linking @xcite and kb aligning @xcite that align entity mentions or sentences to an existing ontological kg. existing open kg canonicalization datasets such as reverb-base, reverb-ambiguous @xcite , @xcite and @xcite ) mainly focus on entity-level canonicalization of nps, providing the gold entity-level np canonicalization (npc-e). the blue tags and dashed boxes in figure 1(b) show examples of npc-e, e.g., "biden" and "joseph biden" should be canonicalized as the same entity q6279. however, these datasets do not provide the gold rp canonicalization (rpc), and do not consider the ontology-level canonicalization of np (npc-o). rpc is to canonicalize rps that mean same relation together, for example, the second and the third "was born in" in figure 1 (b) should be canonicalized into the same cluster of birth place (p19), different from the first one which means "birth time (p569)". similarly, npc-o is to canonicalize nps that have same type together, for example, the "scranton" should be canonicalized into class "countyseat" and into class "local government" together with "atlantic county", it can be viewed as canonicalizing special ontological relations such as "instance of ", "subclass of " represented by dotted arrows in fig. 1(a ). we formally define these tasks in sec. 3. rpc and npc-e are important as parts of a canonicalization benchmark (1) relations and ontology are necessary for an expressive kg @xcite (2) most kg queries involve relations and ontology (e.g., the query "actress that was born in california", involve the relational constraint "x, birth place, california" and the ontological constraint "x, instance of, actress"). in this paper, we present combo, a complete benchmark for open kg canonicalization consisting of three subtasks: besides npc-e which has been adequately studied in previous work, we additionally provide gold rpc and npc-o along with their evaluation metrics. gold npc-o is obtained by querying the wikidata using sparql, and rpc is obtained by performing stanford ope-nie on sentences from wiki20 (a distantly labeled relation extraction dataset), and a per-instance human revision process to ensure the quality of extracted rps. we introduce the data construction process detailedly in sec. 4. our new benchmark makes it possible for the first time to quantitatively evaluate the full range of open kg canonicalization. we conduct comprehensive experiments to compare existing canonicalization methods as well as a few simple baseline methods proposed by us. somewhat surprisingly, none of the existing methods utilizes pretrained contextualized word embedding, probably because previous work only focuses on npc-e and nps are often not very ambiguous, making contextualization not so helpful. for example, the "joe biden" and "joseph biden". however, contexts are more helpful in rpc and npc-o. for rpc, relations are more ambiguous and diverse in surface forms (e.g., "was born in" in figure 1 ) and contexts are needed for disambiguation. for npc-o, the rp and the other np in the triple will help understand the type of an np. therefore, our proposed baseline methods are based on pretrained language models (plm) @xcite @xcite which produce contextualized embedding and have been shown to contain a certain amount of factual knowledge @xcite . we found that, after properly encoding triples and contexts, our baseline methods outperform well on all three subtasks compared with previous state-of-the-art methods, especially on rpc and npc-o. we also propose a triple-based pretraining method and find that it further boosts the performance on all subtasks. therefore, our work provides strong baselines for future research on open kg canonicalization. in summary, our contributions are threefold. first, we propose a complete definition of the open kg canonicalization problem along with the metrics. second, we construct the complete benchmark for open kg canonicalization consisting of entitylevel and ontology-level np canonicalization and rp canonicalization. third, we propose a stronger baseline based on autoencoding plms and conduct a comprehensive empirical comparison of canonicalization methods on our benchmark. ## open kg canonicalization datasets we introduce existing open kg canonicalization datasets and combo. the statistics of datasets are shown in ## task definition and evaluation metrics task evaluation most clustering algorithms such as k-means @xcite and hierarchical agglomerative clustering (hac) @xcite produce non-overlapping cluster assignments, and several algorithms (e.g., hac) can also produce hierarchical and overlapping cluster assignment. for the npc-e subtask, we adopt the classic macro, micro and pairwise metrics to compare the gold and predicted npc-e cluster assignments (please refer to app. c for details). for rpc, the macro metrics that calculate the fractions of pure clusters are too strict because gold rp clusters are large and hence are unlikely to be pure. therefore we only use the micro and pairwise metrics to evaluate rpc. for npc-o, the gold cluster assignments are overlapping. if the predicted clusters are nonoverlapping, we can apply the macro and pairwise metrics and a modified micro metric (appendix d). if the predicted clusters are overlapping, say @xmath0. . . c p m }, we propose evaluation metrics j @xmath1. j @xmath2.1) calculates the average jaccard index of a gold cluster and its best matched predicted cluster. j @xmath3. we illustrate the construction process of combo in figure 2 . we rely on the wiki20 dataset @xcite to obtain the source sentence and the gold npc-e. wiki20 is a large multi-domain relation extraction dataset constructed by aligning the wikipedia corpus with wikidata using distant supervision. as shown in the bottom of figure 2 , each sample of wiki20 contains a sentence with the object and subject np spans labeled and linked to entities in wikidata and the relation between them is also labeled. to ensure data quality, we use the recently revised version of wiki20 @xcite , which aligns the wiki20 relation labels with the supervisedly constructed wiki80 dataset @xcite and provides 56k human-annotated data samples. the object and subject np spans and its entity linking information (e.g., q6275) are from wikipedia and have high precision, so we directly use it for task npc-e. ## comprehensive evaluation of methods our benchmark makes it possible to conduct a comprehensive empirical comparison of different methods on the full range of open kg canonicalization. below we first give an overview of existing methods and propose a few new baseline methods. then we present our experimental settings and results. ## conclusion we present combo, a complete benchmark for open kg canonicalization. combo consists of three subtasks, entity-level and ontology-level np canonicalization, and rp canonicalization. we construct the data and propose the evaluation metrics for the rpc and npc-o that are not been adequately studied before. we also propose a stronger canonicalization method based on autoencoding plms and conduct a comprehensive comparison of different canonicalization methods on our dataset. for future study, npc-o and rpc still have a lot of room for improvement and the efficiency of canonicalization methods is also worth studying. we also note that combo can be additionally used as a probing benchmark for plms and as a phrase-level relation classification dataset.