OWLSIZ : An isi Z ulu CNL for structured knowledge validation (2020)

Abstract: In iterative knowledge elicitation, engineers are expected to be directly involved in validating the already captured knowledge and obtaining new knowledge increments, thus making the process time consuming. Languages such as English have controlled natural languages than can be repurposed to generate natural language questions from an ontology in order to allow a domain expert to independently validate the contents of an ontology without understanding a ontology authoring language such as OWL. IsiZulu, South Africa’s main L1 language by number speakers, does not have such a resource, hence, it is not possible to build a verbaliser to generate such questions. Therefore, we propose an isiZulu controlled natural language, called OWL Simplified isiZulu (OWLSIZ), for producing grammatical and fluent questions from an ontology. Human evaluation of the generated questions showed that participants’ judgements agree that most (83%) questions are positive for grammaticality or understandability.

## introduction ontology developers often rely on domain experts when building models or ontologies. this knowledge elicitation is often done in an iterative manner to ensure a high quality artefact. that is, the first interaction between knowledge engineers and domain experts generally focuses on only obtaining the first usable knowledge increment. further interactions focus on obtaining further increments and also validating the previous iteration's codified knowledge. this requires the engineer to have a method of presenting the codified knowledge in an accessible manner. one method to resolve this, is to render the ontology with a controlled natural language (cnl) and provide supporting tooling for that; see @xcite for a recent overview. the manner in which these tools are to be used throughout the knowledge acquisition process is left to the discretion of the ontology developer, as they have not been incorporated into existing structured methods of knowledge validation. this may lead to inefficiencies, especially for inexperienced ontologists, as it requires one to be directly involved in obtaining the additional knowledge increment and also validating the already captured knowledge. for a language like english, one could repurpose a cnl such as owl simplified english @xcite to generate questions that domain experts could answer without involving an ontologist. this is more challenging for other languages, since fewer resources exist for them. for our context in south africa, isizulu is relevant, since it is the main language by first language speakers and serves as one of the main communication languages alongside english, afrikaans, and setswana (out of 11 official languages and sign language). it is not possible to generate questions from ontologies in isizulu, since such a cnl does not exist and only some statements can be generated with the cnl of (keet and @xcite . we propose a solution in the form of owl simplified isizulu (owlsiz), an isizulu cnl for authoring questions to be used in structural questionnaires and/or likert items to reduce the novice ontology engineer's efforts in validating a model or ontology's contents. to support its use, we developed an owl verbaliser (in java) that uses the cnl to generate questions. evaluation of the quality of the generated questions shows that participants agree that most (83%) of the texts are positive since they have at most one participant who considers them to be ungrammatical and unacceptable. the rest of the paper is structured as follows. section 2 outlines various methods used for knowledge elicitation, section 3 discusses existing work on generating controlled natural languages from ontologies, section 4 presents our cnl, section 5 presents the design of the verbaliser, section 6 presents the quality of our cnl, section 7 the discussion of results, and section 8 concludes. ## knowledge elicitation and validation ontology engineering methodologies (oems) can be categorised into three groups: collaborative, non-collaborative, and custom @xcite . of interest to this work are collaborative oems, which have well-defined phases and involve numerous stakeholders across the different phases. the methodologies do not restrict one to specific kinds of elicitation methods as evidenced by the nine collaborative oems considered by @xcite . of course, many semi-automatic techniques have been proposed over the years, notably nlp-based ontology learning techniques, but the focus here is the stage where there is a domain expert in the loop, who is typically not well-versed in logic. some 25 years ago already, strategies like observations, interviews, or task analysis based methods were already proposed @xcite . the chosen method impacts the overall time it takes to acquire the necessary domain knowledge, especially since most oems (7 out of 9) are fully iterative and the rest are either partially iterative or agile. structured interview techniques such as the use of questionnaires, twenty-questions, and likert scale items @xcite are less time consuming because they are guided. they can still be improved through the automation of question or likert item creation. to the best of our knowledge, while there are verbalisers that can be used to create likert items, there are currently no methods for creating questions to be used within structured techniques. for instance, if an axiom human ∃hasp art.heart had been added in an ontology authoring iteration, then in its validation stage, a structured questionnaire could link that to a cnl containing the template "does each {c1} {op1} some {c2}?" to generate the yes/no validation question 'does each human have as part some heart?'. the domain expert would be validating the knowledge added to the ontology by answering 'yes' and indicating a mistake by answering 'no'. ## text generation from ontologies there are numerous tools that take ontologies or similar models as input and produce text. for the present purpose, we will categorise them as either educational question generators and model/ontology verbalisers. tools for the former take an ontology and generate questions to be presented to people to test their knowledge. tools for the latter take an ontology's axioms or a model's contents and convert them into natural language text to make the input accessible to stakeholders who are not familiar with modelling languages. the existing educational question generators are built for english only @xcite @xcite @xcite . they either use bare templates or simplenlg @xcite for realisation, where 'bare' templates are a sequence of fixed words and slot, such as, model-t's template @xcite . none of the two realisation components are usable for isizulu, because of the language's grammatical complexity or its lack of reusable and comprehensive computational grammar rules. contrarily, the verbalisers offer components that can be used to build an isizulu cnl for question generation even though none of the existing verbalisers were designed for such a purpose. this is because there is a lot of variability concerning the realisation methods and languages supported by ontology verbalisers, as can be seen in while early verbalisers (e.g., @xcite ) supported only english, a number of systems have since been built to support other languages; see w ncx,pl sc ncx,pl -conj-p ncy rc ncy -qc ncy -dwa for instance, the "qcall ncx,pl " is for the universal quantification (∀, 'for all'), which is generated al. (2012) owl grammatical framework × english, catalan, dutch, finnish, french, german, italian, spanish, swedish, and urdu @xcite owl canned text × arabic, english, french, hindi, japanese, mandarin chinese, russian, ukrainian and tibetan @xcite orm template × english, dutch, german, italian, spanish, catalan, french, lithuanian, russian, and arabic lim and halpin (2016) orm "pattern" p english, malay, and mandarin dannélls et al. (2013a ) owl grammatical framework c english, french, italian, finnish and swedish dannélls (2012) owl grammatical framework c english, swedish, and hebrew androutsopoulos et al. (2013) owl "sentence plan" ce english and greek davis et al. (2012) ? grammatical framework c english and dutch bouayad-agha et al. (2012) owl meaning-text theory × english and finnish liang et al. (2011) owl template × english and mandarin gruūz ītis (2011) owl grammatical framework ce english and latvian dannélls (2010) owl grammatical framework c english, swedish, and hebrew dannélls (2008) owl grammatical framework ce english and swedish demey and heath (2014) fbm "pattern" e english and chinese keet and khumalo (2017) owl "pattern" ep isizulu sanby et al. (2016) owl template × afrikaans byamugisha et al. (2016) owl "pattern" ep runyankore aguado et al. (1998) -kpml × spanish halpin and curland (2006) orm template × english bouayad-agha et al. (2011) owl template × spanish based on what the plural of the noun class is of the noun that denotes the entity that plays the whole in the has-part relation (the "w ncx,pl "), combining the appropriate quantitative concord (e.g., ba-for noun class 2) with -onke to generate the appropriate surface realisation (e.g., bonke, for nouns in noun class 2). an axiom with ∀isibhedlela ('hospital', in noun class 7) then verbalises as zonke izibhedlela ('all hospitals') after applying the appropriate algorithms associated with the pattern. compare this with straightforward templates for english for the same scenario: each [whole] has as part at least one [part]. all [whole] pl have as part at least one [part]. demey and heath (2014)'s patterns are syntax templates, which are not suitable for isizulu due to lack of existing computational syntax rules. the other patterns are potentially suitable because they are created to introduce rules for noun-related dependencies, which are essential to all the languages in the niger-congo b family. for instance, lim and halpin (2016) use them to encode malay and mandarin noun classifiers and keet and @xcite ; @xcite use theirs to capture agreement between nouns and a number of parts-of-speech in isizulu and runyankore. these two kinds of noun-centered patterns were not used to generate questions from owl in isizulu, but statements, rather. moreover, they are either limited capturing a single pos that is controlled by noun (lim and @xcite or tightly couple the linearization algorithm with the tem-plate (keet and @xcite . in this work, in contrast, we design a cnl that is able to generate questions, whose grammarinfused templates are not tightly coupled with the linearisation algorithm, and are able to capture dependencies between the noun and numerous other parts-of-speech. ## owl simplified isizulu the approach used to design the cnl to generate validation questions in isizulu from ontologies, is incremental and bottom-up. we begin by selecting the first set of owl constructors to support by selecting ones already supported by an existing english cnl and use its templates as inspiration for our isizulu templates. the chosen cnl for that is owl simplified english @xcite and the resulting owlsiz templates are provided in the isizulu templates use several concords 1 , copulas, and locative affixes. they are polymorphic affixes whose values change depending on the noun in which they are found in the case of the locative prefix and copula or due to the noun class of another noun in the sentence in the case of concords. in to illustrate the use of the isizulu templates, let us consider how to verbalise as a question an axiom of the type a b, e.g., ihebhu umuthi and in owl functional syntax style, subclassof(ihebhu umuthi). template 1 would be chosen, since the b in the position of c2 is a named class. one would then obtain the following text (i) ingabe lonke ihebhu lingumuthi? ## verbaliser implementation we designed the verbaliser for owlsiz in the way shown with the architecture diagram in figure 1 . it was implemented in java and it uses the owl api foot_0 to parse the ontology. unlike the predominant realisation method as listed in we will discuss the planning and realisation components in the remainder of this section. there are four different templates for the ob-jectsomevaluesfrom class expressions (number 6). axiom type (in owl functional syntax style) approximate english template translation isizulu template 1. subclassof(c1 c2) is every {c1} a(n) {c2}? ingabe {sc} onke {c1} {sc} {cop} {c2} ? 1.1. subclassof(c1 ce) does/is every {c1} {ce}? ingabe {sc} onke {c1} {sc} {ce} ? 2. classassertion(c i) is {i} a(n) {c}? ingabe {i} {sc} {cop} {c} ? 3. objectpropertyassertion(op i1 i2) {i1} {op} {i2}? {i1} {sc} {op} {i2} ? 4. equivalentclasses(c1 c2) is every {c2} a(n) {c1}? ingabe noma yi {oc} phi {c2} {sc} {cop} {c1} ? 5. disjointclasses(c1 c2) is there no {c1} that is a(n) {c2}? a {sc} kho yini {c1} {relc} {cop} {c2} ? 6. objectsomevaluesfrom(op c) {op} a(n) {c}? {op} {c}? 6.1. objectsomevaluesfrom(op ce) {op} a {ce}? {op} {ce} ? 6.2. objectsomevaluesfrom(opnoun c) {opnoun}s a(n) {c}? yi {op} {locpre} {c} ini ? 6.3. objectsomevaluesfrom -{op} {locpre} {c} ini ? 6.4. objectsomevaluesfrom(opnoun c) {opnoun}s a(n) {c}? yi {op} {locpre} {c} ? 7. objecthasvalue(op i) {op} {i}? {op} {i}? 8. datapropertyassertion(dp i l) does/will/did/is {i} {dp} {l}? ingabe {i} {dp} {l}? 9. datahasvalue(dp l) {dp} {l}? {dp} {l}? 10. objectallvaluesfrom(op c) {op} only {c}? {op} {c} {sc} odwa 11. objectexactcardinality(n op c) {op} exactly {n} {c}? {op} {c} {relc} yi-{n} ncamashi/ngqo ? 12. objectmincardinality(n op c) {op} at least {n} {c}? {op} {c} {relc} ngaphezulu kuka-{n} ? 13. objectmaxcardinality(n op c) {op} at most {n} {c}? {op} {c} {relc} mbalwa ku-{n} ? template 6.4 is applicable when the object property is a noun and the owl class belongs to noun classes 1, 1a, 2, or 2a, whereas template 6.2 is used when that owl class does not belong to classes 1a, 2, or 2a. template 6.3 is applicable when the object property is the containment part-whole relation (keet and @xcite . template 6 and 6.1 are applicable when the range is an owl class and class expression respectively. the verbaliser has a planning rule that chooses an appropriate template when given a class expression. the nncpairs of figure 1 is a file nncpairs.txt that is used for determining whether an object property is or contains a noun and to retrieve the noun class of the noun, which is reused from @xcite . we extended the file by adding 8 words to add more variation (umakhalekhukhwini, uzola, inokia 3310, ifoni, umfundi, ukubhukuda, ihebhu, and umdlalo) and changed the noun class annotations of 6 nouns (ufulawa 3a, amanzi 6, irhaba 9, ivazi-9, ithiyetha yokuhlinzela 9, indoda 5) due to two typos, mass noun marker deletion, and differences of opinion on whether noun class 9a is indeed separate from noun class 9. the realisation module takes a template and slot fillers, inserts the fillers into their respective slots, resolves the values of concords, locatives, and copulas in polymorphic words, and then forms words by appending the affixes together while relying on isizulu phonological conditioning rules taken from @xcite @xcite @xcite . ## evaluation in order to evaluate the quality of questions generated by the verbaliser, we make use of the test ontology presented in @xcite . we extended the ontology's 82 logical axioms with 12 axioms to ensure coverage for owlsiz: classassertion (2), objectproperty (2), objectpropertyassertion (1), equivalentclasses (1), datapropertyassertion (1), dataproperty (1), objectallvaluesfrom (1), objectexactcardinality (1), objectmincardinality (1), and objectmaxcardinality (1). this update resulted in an ontology with 91 axioms (it is not 94 because 3 axioms were used to constrain the existing axioms). for internal evaluation, the ontology was verbalised and we categorized the resulting texts into verbalisable and unverbalisable. these two classes capture our verbaliser's (un)supported axioms. for each of the verbalisable axioms, we analysed their corresponding text to determine whether there are phonological conditioning errors, morphological agreement errors, and any other grammatical error. for external evaluation, the ontology was verbalised and we packaged the resulting questions into a survey. participants were recruited via snowball sampling using the first author's twitter and whatsapp accounts. the participants were asked to judge the quality of each question by choosing either "grammatical and acceptable", "grammatical and ambiguous", "ungrammatical and understandable", or "ungrammatical and unacceptable". in order to obtain high quality judgements, we ensured that each participants did not judge more than 40 sentences by randomly diving the 76 texts into two surveys each containing 38 texts (henceforth, survey a and b). survey a includes texts generated by templates 3, 10, 12, and 13 while survey b does not and survey b includes texts generated by templates 4, 8, and 11 while survey a does not. participants were randomly assigned to a survey. the verbaliser's input, chosen templates, and numbered output texts are given as supplementary material at https://github.com/adeebnqo/ grammarinfusedtemplates . ## results and discussion out of the total 91 axioms, 76 were verbalisable axioms and of their corresponding texts, 74 texts are free of morphological agreement errors, phonological conditioning errors, or any other grammatical errors as assessed by the first author, an l2 isizulu speaker with experience with various aspects of isizulu grammar (e.g., @xcite ). template 5 was selected when verbalising disjoint-classes(isidlanyama isidlazitshalo), and the system generated the following output: one out of the two questions determined to have errors had an morphological agreement error (question 42 in the supplementary material). the second was generated by template 8 and, when verbalising datapropertyassertion(neminyaka uzola 50), the system generated the following output: the 15 that were classified as unverbalisable had several causes. 10 had nouns whose classes were unresolved; e.g., one where the compound noun's noun class was unresolved (isampula igazi ought to have been isampula egazi 'blood sample' in noun class 5). the other five included unsupported axiom types, such as objectcomplementof. six participants filled in one of the surveys, of 2 . observe that there are only two questions (labelled 25 and 42 in the figure) for which participants agree that they are ungrammatical and unacceptable, which are: 25 : inokia 3310 lifundisa uzola? 'the nokia 3310 teaches zola' 42 : ingabe noma yiyiphi indlu eyinyama? figure 2 also shows that 83% of the texts are judged positively as they have at most one participant who considers them to be ungrammatical and unacceptable. furthermore, since survey a only had two participants, even if we adjust how we determine the number texts judged positively and define them as the texts where is no participant who considers the text to be ungrammatical and unacceptable, we still find that most (71%) of survey a's questions were judged positively by the participants. participants agreed in their judgements (i.e., they chose the same options out of the ones listed in section 6) for 25/76 texts (more specifically, 16/38 for survey a and 9/38 for survey b). furthermore, out of the 25, they agreed positively to 24 of the texts and agreed that remaining question is ungrammatical and unacceptable. while in (keet and @xcite it seemed that there was more disagreement in the human evaluation the longer the sentences were, here, the participants' disagreement is not due to differences in text length. this since the texts for which the participants agree and disagree are of similar length (@xmath14, save for a single outlier with 7 words in the texts for which they disagree. the disagreement may be due to a misunderstanding that the participants have regarding how to evaluate the texts. for instance, when given text that reflects an unacceptable conceptualisation of the real world, then participants may be selecting "ungrammatical and unacceptable" to reflect the unacceptability of the conceptualisation, as opposed to evaluating the quality of the text. this is suspected in the evaluation of the question ingabe lonke ibhotela lenza ifoni eliyi-1 ncamashi? ('does every butter make exactly 1 phone?') where three participants selected "ungrammatical and unacceptable" and one selected "grammatical and ambiguous", even though the text is grammatically correct as judged by the first author. furthermore, regional differences in the dialects spoken by the participants may result in one participant judging a particular text acceptable while others judge it ambiguous. we also determined the number of texts that all participants judge as being grammatically correct irrespective of whether they are acceptable or ambiguous. this was calculated by counting the number of texts where participants judged them as any combination of 'grammatical and ambiguous' and 'grammatical and acceptable'. we found that only 51% were judged as being grammatically correct hence 37/76 are ungrammatical. moreover, the participants all agree that a majority (20/37) of the ungrammatical texts are still acceptable. the evaluation shows that the templates produce texts, whose majority is found to be understandable and grammatically correct by isizulu speakers. moreover, even when generated texts are ungrammatical, most of them are still acceptable. most of the texts generated by the templates are free of morphological agreement errors. in the only case where there was such an error (question 42), it was due to human error when creating the serialised template and not a problem with the templates listed in ## conclusions and future work we have created the first isizulu cnl and verbaliser capable to generating questions from an ontology for the purpose of knowledge elicitation and validation. evaluation of the quality of text generated by the implemented verbaliser shows that most (81%) of the participants' judgements, overall, are positive. moreover, when we analysed judgements aggregated for each question, we find that most of the texts (83%) are judged positively because they have at most one participant who considers them to be ungrammatical and unacceptable. adjusting how we determine the number of texts judged positively for survey a since there were only two participants, we found that most (71%) of survey a's questions were also judged positively because they had no participant who considers them to be ungrammatical and unacceptable. overall, while there is a sizeable number (37/76) of questions for which the participants agree that they are ungrammatical, a majority of them (20/37) are still judged as being acceptable. future work includes determining which templates are generating unacceptable texts, soliciting an isizulu grammarian's feedback regarding its grammaticality, and correcting the identified problems.