Commonsense Reasoning Using W ord N et and SUMO : a Detailed Analysis (2019)

Abstract: We describe a detailed analysis of a sample of large benchmark of commonsense reasoning problems that has been automatically obtained from WordNet, SUMO and their mapping. The objective is to provide a better assessment of the quality of both the benchmark and the involved knowledge resources for advanced commonsense reasoning tasks. By means of this analysis, we are able to detect some knowledge misalignments, mapping errors and lack of knowledge and resources. Our final objective is the extraction of some guidelines towards a better exploitation of this commonsense knowledge framework by the improvement of the included resources.

## introduction any ontology tries to provide an explicit formal semantic specification of the concepts and relations in a domain @xcite @xcite @xcite . as with other software artefacts, ontologies typically have to fulfil some previously specified requirements. usually both the creation of ontologies and the verification of its requirements are manual tasks that require a significant amount of human effort. in the literature, some methodologies collect the experience in ontology development @xcite and in ontology verification @xcite . in order to evaluate the competency of sumobased ontologies in the sense proposed by @xcite , @xcite propose a method for the semi-automatic creation of competency questions (cqs). concretely, they the problem that results from figure 1 consists of the above conjecture, which is considered to be true according to our commonsense knowledge, and its negation, which is therefore assumed to be false. state-of-the-art atps for first-order logic (fol) such as vampire @xcite or e @xcite have been proved to provide advanced reasoning support to large fol conversions of expressive ontologies @xcite @xcite . however, the semi-decidability of fol and the poor scalability of the known decision procedures have been usually identified as the main drawbacks for the practical use of fol ontologies. in particular, given an unsolved problem (i.e. a problem such that atps do not find any proof for its pair of conjectures) it is not easy to know if (a) the conjectures are not entailed by the ontology or (b) although some of the conjecture is entailed, atps have not been able to find the proof within the provided execution-time and memory limits. on the contrary, given a solved problem, it is hard to know whether the solution is obtained for a good reason, because an expected result does not always indicate a correct ontological modelling. in this paper, we provide a detailed analysis of the large commonsense reasoning benchmarks created semi-automatically by @xcite . the aim of this analysis is to shed light on the commonsense reasoning capabilities of both the benchmark and the involved knowledge resources. to that end, we have randomly selected a sample of 169 problems (1% of the total) following a uniform distribution and manually inspected their source knowledge and results. by means of this detailed analysis, we are able to evaluate the quality of automatically created benchmarks of problems and to detect hidden problems and misalignments between the knowledge of wordnet, sumo and their mapping. outline of the paper. in order to make the paper self-contained, we first introduce the ontology, the mapping to wordnet and the evaluation framework in section 2. next, we provide a full summary and the main conclusions obtained from our manual analysis in section 3. then, we individually examine some of the problems in section 4. finally, we provide some conclusions and discuss future work in section 5. ## commonsense reasoning framework in this section we describe briefly the whole commonsense reasoning reasoning framework. first, we present the knowledge resources needed and then the reasoning framework. ## detailed analysis of the experimental results in this section, we report on a detailed and manual analysis of the experimental results obtained from a small number of the cqs described in section 2. from this experimentation, we have randomly selected a sample of 169 problems (1% of the total) following a uniform distribution and analysed the results obtained for those problems by focusing on two questions: 1) we analyse the quality of mapping of the involved synsets and 2) we analyse the knowledge required for solving the problems. regarding the quality of the mapping (first question), we classify the mapping of synsets as either correct or incorrect according to the following criteria: a mapping is classified as correct if the semantics associated with the sumo concept and with the synset are compatible, and it is classified as incorrect otherwise. for example, both the verb synset machine 1 v and the adjective synset homemade 1 a are connected to making c +, regarding the required knowledge (second questions), we distinguish three cases: it is worth noting that, in the case of unsolved problems such that the required knowledge is classified as existing, atps cannot find a proof for its in in the fourth part (unsolved, three columns), we provide the result of our analysis for the unsolved problems: finally, in the last part (total, five columns) we summarise the result of our analysis: in total, the synsets in 111 problems (66 %) are decided to be correctly connected to sumo and, among them, the synsets in 24 problems (14 %) are decided to be precisely connected. thus, some of the synsets are not correctly connected to sumo in 58 problems (34 %). further, 82 problems (49 %) are solved and the knowledge of the ontology that is used in the proofs reported by atps is decided to be correct (100 %) according to our world knowledge. among solved problems, 65 problems (79 %) are classified as entailed and 17 problems (21 %) are classified as incompatible. by manually analysing incompatible problems, we have discovered that the knowledge of wordnet and sumo related to all the problems with a correct mapping is not wellaligned. thus, we can conclude that this reasoning framework also enables the correction of the alignment between wordnet and sumo. for example, cloud 1 n ("any collection of particles (e.g., smoke or dust) or gases that is visible") is hyponym of physical phenomenon 1 n ("a natural phe-nomenon involving the physical properties of matter and energy") in wordnet. however, cloud 1 n and physical phenomenon 1 n are respectively connected to cloud @xmath0, which are inferred to be disjoint classes in fol-sumo. further, the mapping of the involved synsets is classified as correct in 51 of 65 entailed problems (78 %), while only 14 problems (22 %) are classified as entailed with an incorrect mapping. by contrast, the percentage of problems with an incorrect mapping is much higher among incompatible and unsolved problems: 42 % (8 of 17 entailed problems) and 41 % (36 of 87 unsolved problems) respectively. this is especially the case of the problems from the antonym categories: 26 of 40 antonym problems (65 %) have an incorrect mapping. this fact reveals the poor quality of the mapping of sumo to wordnet adjectives. finally, we have manually checked that 45 of the 51 unsolved problems with a correct mapping (88 %) cannot be entailed by the knowledge in sumo, which sets an upper bound on the number of problems that can be classified as solved although augmenting the knowledge of the ontology and correcting the mapping and the alignment between wordnet and sumo. next, we summarise the main conclusions drawn from our detailed analysis: ## exhaustive analysis of some problems in this section, we present a detailed analysis of some of the examples that have been reported in ## conclusion and future works in this paper we have presented a detailed analysis of a sample of large benchmark of commonsense reasoning problems that has been automatically obtained from wordnet, sumo and their mapping. based on this analysis, we can detect that although the framework enables the resolution of around 49 % of the total problems, only 36 % of the total are resolved for the good reasons: 60 problems resolved with a correct mapping. we have also detected that the mapping requires a general revision and correction: in particular, in the case of adjectives. on the contrary, the knowledge in sumo involved in the revised proofs seems to be correct according to our commonsense knowledge. further, the problems classified as incompatible enable the detection of misalignments between wordnet and sumo, while the problems classified as unknown can be taken as a source of knowledge for the augmentation of sumo. actually, we are planning to develop an automatic procedure for the augmentation of sumo on the basis of the knowledge in wordnet. finally, we have detected some problems that can be solved on the basis of the knowledge of sumo but that are not solved due to the limitation of resources of atps.