Do PLM s Know and Understand Ontological Knowledge? (2023)

Abstract: Ontological knowledge, which comprises classes and properties and their relationships, is integral to world knowledge. It is significant to explore whether Pretrained Language Models (PLMs) know and understand such knowledge. However, existing PLM-probing studies focus mainly on factual knowledge, lacking a system- atic probing of ontological knowledge. In this paper, we focus on probing whether PLMs store ontological knowledge and have a semantic un- derstanding of the knowledge rather than rote memorization of the surface form. To probe whether PLMs know ontological knowledge, we investigate how well PLMs memorize: (1) types of entities; (2) hierarchical relationships among classes and properties, e.g., Person is a subclass of Animal and Member of Sports Team is a subproperty of Member of ; (3) domain and range constraints of properties, e.g., the subject of Member of Sports Team should be a Person and the object should be a Sports Team. To further probe whether PLMs truly understand ontological knowledge beyond memorization, we comprehensively study whether they can reliably perform logical reasoning with given knowledge according to ontological entailment rules. Our probing results show that PLMs can memorize certain ontological knowledge and utilize implicit knowledge in reasoning. How- ever, both the memorizing and reasoning per- formances are less than perfect, indicating in- complete knowledge and understanding.

## introduction pretrained language models (plms) have orchestrated impressive progress in nlp across a wide variety of downstream tasks, including knowledge-intensive tasks. previous works propose that plms are capable of encoding a significant amount of knowledge from the pretraining corpora @xcite , and determine to explore the kinds of knowledge within plms. i. explicitly given ii. implicitly given iii. not given (b) memorizing person person organisation organisation premise 1/2 one has to be a particular @xcite to be a player at a sports team. … … z member of sports team @xcite . soft: domain pseudoword (p1) one has to be a person to be a player at a sports team. ## benchmark construction in this section, we present our methodology for ontology construction and the process of generating memorizing and reasoning tasks based on the ontology for our probing analysis. ## probing methods we investigate encoder-based plms (bert @xcite and roberta @xcite ) that can be utilized as input encoders for various nlp tasks. prompt is an intuitive method of our probing task as it matches the mask-filling nature rule premises conclusion candidate remark rdfs2 [p 1 ] aaa domain xxx. [p 2 ] uuu aaa vvv. uuu type xxx. class type inference through domain constraint. rdfs3 [p 1 ] aaa range xxx. [p 2 ] uuu aaa vvv. vvv type xxx. class type inference through range constraint. rdfs5 [p 1 ] bbb subproperty of ccc. [p 2 ] aaa subproperty of bbb. aaa subproperty of ccc. property transitivity of subproperty. rdfs7 [p 1 ] aaa subproperty of bbb. [p 2 ] uuu aaa vvv. uuu bbb vvv. property pattern property inheritance through subproperty. rdfs9 [p 1 ] xxx subclass of yyy. [p 2 ] uuu type xxx. uuu type yyy. class type inheritance through subclass. rdfs11 [p 1 ] yyy subclass of zzz. [p 2 ] xxx subclass of yyy. xxx subclass of zzz. class transitivity of subclass. ## results and findings in this section, we introduce the performance of plms foot_0 on the test sets of memorizing and reasoning tasks, and analyze the results to posit a series of findings. we then analyze the effectiveness of different prompts. detailed experimental results can be found in appendix c. ## preliminary evaluation of chatgpt after we finished the majority of our probing experiments, chatgpt, a decoder-only model, was publicly released and demonstrated remarkable capabilities in commonsense knowledge and reasoning. therefore, we additionally perform a preliminary probe of the ability of chatgpt to memorize and ## conclusion in this work, we systematically probe whether plms encode ontological knowledge and understand its semantics beyond the surface form. experiments show that plms can memorize some ontological knowledge and make inferences based on implicit knowledge following ontological entailment rules, suggesting that plms possess a certain level of awareness and understanding of ontological knowledge. however, it is important to note that both the accuracy of memorizing and reasoning is less than perfect, and the difficulty encountered by plms when processing paraphrased knowledge is confirmed. these observations indicate that their knowledge and understanding of ontology are limited. therefore, enhancing the knowledge and understanding of ontology would be a worthy future research goal for language models. our exploration into chatgpt shows an improved performance in both memorizing and reasoning tasks, signifying the potential for further advancements.