CONTOR : Benchmarking Strategies for Completing Ontologies with Plausible Missing Rules (2024)

Abstract: We consider the problem of finding plausible rules that are missing from a given ontology. A number of strategies for this problem have already been considered in the literature. Little is known about the relative performance of these strategies, however, as they have thus far been evaluated on different ontologies. Moreover, existing evaluations have focused on distinguishing held-out ontology rules from randomly corrupted ones, which often makes the task unrealistically easy and leads to the presence of incorrectly labelled negative examples. To address these concerns, we introduce a benchmark with manually annotated hard negatives and use this benchmark to evaluate ontology completion models. In addition to previously proposed models, we test the effectiveness of several approaches that have not yet been considered for this task, including LLMs and simple but effective hybrid strategies.

## introduction ontologies, in the context of artificial intelligence (ai), are essentially sets of rules which describe how the concepts from a given domain are related @xcite . they generalise taxonomies by expressing these relationships using logical connectives, which makes it possible to describe conceptual relationships in a more precise way. throughout this paper, we will use the common description logic syntax for encoding ontology rules @xcite . for instance, an ontology might contain the following rules: in this syntax, rules are formulated as concept inclusions x ⊑ y , which encode that every instance of the concept x is also an instance of the concept y . the connective ⊓ corresponds to intersection and ∃r.c is the set of concepts that are related through relation r to some concept from c. for instance, (1) expresses the knowledge that every biologist who is located somewhere in the uk is called a "uk scientist". ontologies are most commonly used for defining specialised terminologies, which is crucial for ai systems in many technical domains, including healthcare @xcite , building information modelling @xcite and geographic information systems @xcite , just to name a few. they can also be used to enrich knowledge graphs (kgs) @xcite and to improve classifiers by modelling label dependencies @xcite . like other types of knowledge bases, ontologies are often incomplete @xcite . with this in mind, we consider the problem of predicting missing rules. we are specifically interested in the problem of predicting rules between known concepts, rather than in enriching ontologies with new concepts. previous work has shown the feasibility of ontology completion, but the strengths and weaknesses of existing methods are poorly understood, and suitable evaluation frameworks for studying ontology completion are currently lacking. to address this, we make the following contributions: ## preliminaries ontologies we focus on description logic ontologies, which express knowledge in terms of concept inclusions. let c be a set of atomic concepts and r a set of relations (often called roles in this context). a concept is (i) an expression of the form c with @xmath0, (iii) an expression of the form ∃r.x with @xmath1, or (iv) one of the trivial concepts ⊥ or ⊤. an ontology is a set of rules of the form x ⊑ y (also known as concept inclusions), where x and y are concepts, called the body and the head of the rule respectively. equations ( 1 )-( 3 ) provide examples of such rules. semantically, each concept corresponds to a set of individuals and x ⊑ y expresses that every individual which belongs to the concept x also belongs to the concept y . the concept x ⊓ y contains all individuals that belong to both x and y . the concepts ⊤ and ⊥ correspond to the universe of all individuals and the empty set respectively. finally, an individual x belongs to ∃r.x if there exists some individual y such that the relation r(x, y) holds and y belongs to x. a detailed understanding of description logics is not needed for this paper. for more details, we refer to @xcite . task formulation let k be an ontology and let x ⊑ y be a rule which is not in k (nor logically entailed by this ontology). we consider the problem of predicting whether x ⊑ y is a valid rule, which we treat as a binary classification problem. note that x and y may be complex expressions, built from a set of atomic concepts and relations. while x and y themselves might not appear in k, we assume that these atomic concepts and relations do, i.e. we focus on predicting missing knowledge about a given set of atomic concepts. note that in this setting we do not have access to examples of instances of the various concepts. this means that traditional approaches to rule learning cannot be used. instead, two types of information are available for predicting missing rules. first, we may be able to exploit the structure of the ontology. for instance, from the rules (1)-(3) we may be able to infer that biologist, geologist and chemist share something in common, as these rules follow a similar pattern. whenever we encounter some knowledge that is true for biologists and geologists, we may then assume that this knowledge also plausibly holds for chemists. second, we can exploit the fact that ontology concepts generally correspond to natural language terms, which means that language models @xcite or concept embeddings @xcite can be used to inject relevant knowledge. differences with related tasks various forms of knowledge base completion have been studied in the literature, which differ from ontology completion in fundamental ways. for instance, knowledge graph completion is about predicting missing factual assertions (e.g. paris is the capital of france) rather than about predicting rules. rule learning has been studied in many settings, but usually the aim is to learn rules that capture statistical dependencies, based on examples. in the case of ontology completion, the purpose is to learn rules that are universally valid, expressing knowledge that can be used across a wide range of applications (within a given technical domain). the problem of taxonomy expansion is also related, but focuses on relations between individual terms, rather than between more complex expressions. moreover, the primary focus in that context has been on adding missing terms to existing taxonomies, rather than on predicting missing concept inclusions. benchmarking progress on these various knowledge base completion tasks has been crucially enabled by the emergence of standard benchmarks. in the case of ontology completion, the evaluation methodologies that were used in previous work have serious shortcomings. moreover, the evaluation protocols that were used are not comparable, meaning that little is known about the relative effectiveness of different kinds of strategies. the main aim of this paper is to address this issue, and thus lay the foundations for future work on this topic. ## the contor benchmark existing evaluations of ontology completion have important limitations. for instance, @xcite and @xcite use different formulations of the problem, which makes their evaluations not directly comparable. more fundamentally, these methods were evaluated by using randomly corrupted positive examples as negatives. however, such random corruptions are often relatively easy to identify, especially if they were obtained by swapping concepts for semantically distinct alternatives. for instance, it should be easy enough for a baseline system to predict that biologist ⊑ uk is not a plausible rule. furthermore, randomly corrupted rules might sometimes correspond to semantically valid rules. to address these limitations, we propose contor (completing ontology rules), a benchmark for ontology completion with manually validated hard negative test examples. in section 5 we will then use this benchmark to compare, for the first time, the different approaches that have been proposed for ontology completion. ontologies contor is based on seven wellknown ontologies: the five ontologies that were used by @xcite , namely wine, economy, olympics, transport and sumo, and two ontologies that were used by @xcite , namely foodon and the gene ontology (go). wine, economy, olympics and transport are small domainspecific ontologies. sumo is used as a representative example of a larger general-domain ontology. finally, foodon and go are considerably larger than all the others, while being focused on specialised domains. for wine, economy, olympics, transport and sumo, we keep 20% of the rules for testing. for foodon and go, we keep 5% for testing. the remaining rules are split into training and development sets. some basic statistics of the considered ontologies are summarised in negative training examples for the training set, using manually annotated negative examples is not feasible, due to the large number of rules which would have to be checked. therefore, we still use randomly corrupted negative examples for the training set, following the strategies that were proposed in previous work @xcite (see appendix b for details). finally, note that we only add these randomly corrupted rules as negative examples if they cannot be entailed by the positive examples (i.e. the training split of the given ontology). negative test examples to obtain negative examples for the test set, we rely on human annotators to ensure that the corrupted rules are indeed negative examples. moreover, we aim to select hard negative examples, given that random corruption often leads to nonsensical rules which are too easy to identify. specifically, for each positive rule α ⊑ β, we randomly select one of the concepts c appearing in that rule and replace it with another concept d. rather than selecting this concept arbitrarily, we choose d to be among the 5 most similar concepts to c, in terms of the cosine similarity between the fasttext embeddings of the corresponding names foot_0 . note that selecting similar concepts increases the chances that the corrupted rule is actually a valid rule, which means that human annotation is critical in this case. each corrupted rule was checked by two annotators, who were trained in formal knowledge representation and were fluent in english. the agreement between the annotators is reported in ## ontology completion strategies our main aim in this paper is to empirically compare the success of different ontology completion strategies. first, we focus on two strategies from the literature: an nli based strategy @xcite and a model based on gnns and pre-trained concept embeddings @xcite . in addition to providing the first comparison of these two strategies, we will also explore a range of variants of the original models. as we will see, both strategies are in fact highly complementary. inspired by this finding, we also explore some hybrid strategies. ## experiments we now present our experimental analysis. 4 models we experiment with the nli based models from the deeponto library, which correspond to fine-tuned roberta-base and roberta-large encoders. regarding the fine-tuned llms, we experiment with mistral-7b, llama3-8b, phi3medium and gemma-7b. where available, we use both the base models and their instruction finetuned variants. the latter will be indicated by it in the results tables. 2018) and gatv2 @xcite . we furthermore compare with the model from @xcite , which uses a graph with different edge types, corresponding to the binary templates, and relies on an r-gcn @xcite to take these edge types into account. as a baseline, to show the benefit of using gnns, we also compare with a variant which feeds the concept embeddings to a feedforward network instead of a gnn (shown as ffn). unless specified otherwise, we use the concn concept embeddings @xcite as input features for the gnns. full experimental details can be found in appendix b. ## conclusions we have considered the problem of finding plausible rules which are missing from a given ontology. specifically, we have introduced a new benchmark with hard negatives, which were manually verified by human annotators. we then compared, for the first time, the two main families of ontology completion methods: nli based methods and gnn based methods. beyond existing nli based methods, we also presented the first analysis of llms for ontology completion. finally, we found hybrid strategies to achieve surprisingly strong results, clearly outperforming the current state-of-the-art. acknowledgments this work was supported by epsrc grant ep/v025961/1, leverhulme trust grant rpg-2021-140, anr-22-ce23-0002 eriana, anr-20-chia-0028 and hpc resources from genci-idris (grant 2024-[ad011013338r2]).