Language Model Analysis for Ontology Subsumption Inference (2023)

Abstract: Investigating whether pre-trained language models (LMs) can function as knowledge bases (KBs) has raised wide research interests recently. However, existing works focus on simple, triple-based, relational KBs, but omit more sophisticated, logic-based, conceptualised KBs such as OWL ontologies. To investigate an LM’s knowledge of ontologies, we propose OntoLAMA, a set of inference-based probing tasks and datasets from ontology subsumption axioms involving both atomic and complex concepts. We conduct extensive experiments on ontologies of different domains and scales, and our results demonstrate that LMs encode relatively less background knowledge of Subsumption Inference (SI) than traditional Natural Language Inference (NLI) but can improve on SI significantly when a small number of samples are given. We will open-source our code and datasets.

## introduction the advancements of large pre-trained language models (lms) have sparked research interests in investigating how much explicit semantics lms can learn or infer from knowledge bases (kbs) @xcite . the lama (language model analysis) probe @xcite is among the first works that adopt prompt-based methods to simulate the process of querying relational knowledge from various kbs such as con-ceptnet @xcite and googlere 3 . some subsequent studies focus on probing specific types of knowledge from sources like commonsense kbs @xcite , biomedical kbs (sung 1 an ontology concept is also known as a class. to avoid confusion with class in machine learning classification, we stick to use the term concept. however, existing "lms-as-kbs" works focus on simple, triple-based, relational kbs, but neglect more formalised, logic-based, conceptualised kbs. for example, a statement like "london is the capital of the uk" can be expressed in the triple (london, capitalof, uk); but a sentence like "arthritis is a kind of arthropathy with an inflammatory morphology", which describes the concept "arthritis", cannot be easily expressed using just triples. conceptual knowledge like this requires a formal and expressive representation to be defined precisely. a well-known model for conceptual knowledge is the owl foot_0 ontology @xcite , which can be seen as a description logic (dl) kb with rich built-in vocabularies for knowledge representation and various reasoning tools supported. taking the example of "arthritis", in dl the concept can be described as arthritis arthropathy ∃hasm orphology.inf lammatory. in this work, we take a further step along the "lms-as-kbs" research line towards more formalised semantics by targeting dl kbs and in particular the owl ontologies. current works on lms concerning ontologies are mostly driven by a target application. @xcite , @xcite , and @xcite apply language model finetuning to address ontology curation tasks such as concept insertion and matching, while @xcite transform ontologies into graphs for data augmentation in few-shot learning. in contrast to these application-driven approaches, we investigate a more fundamental question: to what extent can lms infer conceptual knowledge modelled by an ontology? particularly, we focus on the subsumption relationships between ontology concepts. as shown in figure 1 , we first extract concept pairs (c, d) that are deemed as positive (c and d are in a subsumption relationship) and negative (c and d are assumed to be disjoint) samples from an ontology. note that the sampling procedure is fully automatic with the syntax and semantics of owl ontology carefully considered. to translate the concepts and especially the ones with complex logical expressions into natural language texts, we develop a recursive concept verbaliser. we formulate the subsumption inference (si) task similarly to the natural language inference (nli) task and treat the concept pairs as premise-hypothesis pairs @xcite , which will then be wrapped into a template for generating inputs of lms. we have created si datasets from ontologies of various domains and scales, and conducted extensive experiments. our results demonstrate that lms perform better on a typical nli task than the constructed si tasks under the zero-shot setting, indicating that lms encode relatively less background knowledge of ontology subsumptions. however, by providing a small number of samples (k-shot settings), the performance on si is significantly improved. this observation is consistent with the three lms that are studied in this work. ## datasets in this work, we consider ontologies of different domains and scales including: we construct an atomic si dataset for each ontology, but complex si datasets are created for foodon and go only, due to their abundance of equivalence axioms. to avoid too many repetitive concept expressions brought by a particular equivalence axiom, we sample at most 4 positive and 4 negative samples for each equivalence axiom in the complex si setting. to attain class balance, we purposely keep the number of negative samples the same as the positive samples in each data split. for most of the resulting datasets, we divide each into 8 : 1 : 1 for training, development, and testing; for the schema.org's atomic si and the foodon's complex si datasets, which are relatively smaller, we apply a 2 : 1 : 7 division instead. note that we mainly focus on k-shot settings in the probing study, thus the required training and development sample sets are small. to compare with how lms perform on traditional nli, we additionally create bimnli, a subset of the multi-genre natural language inference (mnli) corpus @xcite where (i) the neutral class and its samples are removed, (ii) the matched and mismatched testing sets are merged into one testing set, (iii) 10% of the training data is used as the development set, and (iv) the entailmentcontradiction ratio is set to 1 : 1 (by discarding extra samples from the dominant class) for a balanced prior. the numbers of named concepts and equivalence axioms in ontologies, and the numbers of samples in (each split of) si datasets and the bimnli dataset are reported in ## conclusion and discussion as a work that introduces ontologies to the "lmsas-kbs" collection, this paper emphasises on how to establish a meaningful adaptation from logical expressions to natural language expressions, following their formal semantics. to this end, we leverage the natural language inference (nli) setting to define the subsumption inference (si) task with careful considerations to address the differences between textual entailment and logical entailment. we also develop the recursive concept verbaliser for owl ontologies as an auxiliary tool. our results demonstrate that with our si set-ups, lms can successfully learn to infer both atomic and complex subsumptions when a small number of annotated samples are provided. this paves the way for investigating more complex reasoning tasks with lms or guiding lms using ontology semantics with limited training. in fact, the current si setting is not the only way for probing subsumption knowledge of an ontology; for example we can directly verbalise c d as "v(c) is a kind of v(d)" and formulate the probing task similar to fact-checking or equivalently, an inference task with empty premises. however, our pilot experiments demonstrate that such setting is not as effective as the current si setting. the presented work brings opportunities for future work as (i) the proposed ontology verbalisation method has not covered all possible patterns of complex concepts (e.g., with cardinality restrictions and nominals); (ii) we have not fully considered textual information such as synonyms, definitions, and comments, that are potentially available in an ontology; (iii) we have considered only tbox (terminological) axioms, but abox (assertional) axioms can be involved in, e.g., the membership prediction task, where the objective is to classify which concept an individual belongs to. therefore, developing a robust tool for verbalising logical expressions and extending the ontology inference settings are potential next tasks. another interesting line for the near future is to train an lm using ontologies with their logical semantics considered. the resulting lm is expected to be applicable to different downstream ontology curation tasks such as ontology matching and entity linking, with fewer samples necessary for fine-tuning.