Can Large Language Models Understand DL -Lite Ontologies? An Empirical Study (2024)

Abstract: Large language models (LLMs) have shown significant achievements in solving a wide range of tasks. Recently, LLMs’ capability to store, retrieve and infer with symbolic knowledge has drawn a great deal of attention, showing their potential to understand structured information. However, it is not yet known whether LLMs can understand Description Logic (DL) ontologies. In this work, we empirically analyze the LLMs’ capability of understanding DL-Lite ontologies covering 6 representative tasks from syntactic and semantic aspects. With extensive experiments, we demonstrate both the effectiveness and limitations of LLMs in understanding DL-Lite ontologies. We find that LLMs can understand formal syntax and model-theoretic semantics of concepts and roles. However, LLMs struggle with understanding TBox NI transitivity and handling ontologies with large ABoxes. We hope that our experiments and analyses provide more insights into LLMs and inspire to build more faithful knowledge engineering solutions.

## introduction large language models (llms) @xcite @xcite have showcased remarkable proficiency in understanding textual data and revolutionized the field of natural language processing. recent studies suggest that llms possess adaptability to store, retrieve and infer with symbolic knowledge such as knowledge graphs (kgs) @xcite , sparking interest in their potential for understanding structured information. however, llms' capacity in understanding more complex symbolic knowledge, description logic (dl) ontologies, remains unexplored. compared with kgs, dl ontologies have more fined-grained knowledge representation with for- † corresponding author. mal syntax and model-theoretic semantics. for syntax, while most kgs generally only support atomic entities like p hdstudent, dl ontologies can support various constructors and compound concepts such as ¬p hdstudent ⊓ ∃hasstudentid. for semantics, dl ontologies have model-theoretic semantics. for example, the above complex concept can be interpreted as the set of individuals who are not phd students but do have a student id. further, dl ontologies efficiently support logical reasoning such as c 1 ⊑ c 2 , c 3 ⊑ ¬c 2 → c 1 ⊑ ¬c 3 . this denotes if c 1 is a subclass of c 2 and c 3 is disjoint with c 2 , then it logically follows that c 1 must also be disjoint from c 3 (e.g., dog ⊑ m ammal, bird ⊑ ¬m @xmath0. understanding a dl ontology goes beyond just the capabilities of storage, retrieval, and inference, but involves a deeper comprehension of its formal syntax and semantic interpretations. while the necessity for more detailed investigations for llms' capacity in understanding dl ontologies is clear, a comprehensively evaluation presents a challenge. most related works focus on llms' capacity to capturing patterns in kgs @xcite , far away from indicating that llms possess the ability to understand dl ontologies. even though many endeavors study whether llms can do logical reasoning @xcite @xcite , few of them explore llms' capacity with dl services. dl is primarily focused on representing and reasoning about the hierarchical relationships and properties of concepts within a domain, distinguishing it from other logics by its emphasis on structured, formal ontology. this research gap highlights the significance and challenges in comprehensively evaluating whether llms can understand dl ontologies. in this study, we investigate how effectively llms can understand dl-lite ontologies, a member of the dl ontology family known for simplicity and efficient reasoning. we present an evaluation framework that comprehensively assesses llms' capability to understand dl-lite ontologies in two aspects, respectively, whether llms can grasp the formal representations (the syntactic aspect) and whether llms can understand the semantic interpretations of ontologies and effectively utilize them (the semantic aspect). for the syntactic aspect, we investigate whether llms can comprehend the structural rules, valid statements, and expressions of dl-lite through syntax checking. for the semantic aspect, we first investigate whether llms can understand the semantics of concepts and roles from two aspects, intension and extension, by subsumption of concepts or roles and instance checking respectively. additionally, we probe property characteristics in dl-lite ontologies, such as inverse roles and functional roles. further, we conduct query answering and ontology satisfiability checking to evaluate whether llms can understand the semantics of the whole ontologies. figure 1 gives an illustration of these tasks. through extensive experiments, we find that: to the best of our knowledge, this is the first study to conduct comprehensive evaluations about whether llms can understand dl-lite ontologies. overall, our work contributes to a better understanding of llms' behaviors and inspires to build more faithful knowledge engineering solutions. ## preliminaries in this section, we briefly recall some basic notions about dl-lite ontology @xcite . particularly, we focus on dl-lite core , dl-lite f and dl-lite r , three members in dl-lite family, while our evaluation framework can be applied to any other description logics (dls) such as dl-lite a , alc and el. ## conclusion we have empirically investigated whether llms can understand dl-lite ontologies. extensive experimental results demonstrated the effectiveness and limitations of llms in understanding the syntax and semantics of dl-lite ontologies. for instance, llms possess the ability to understand formal syntax and semantics of concepts, roles and property characteristics. however, llms still struggle with understanding tbox ni transitivity rules and handling ontologies with large scale aboxes. as future works, we will consider exploring the ability of llms to understand ontologies in other lightweight ontology languages, such as el, and to understand ontologies in intractable ontology languages, such as alc and shoiq.