On the Feasibility of LLM -based Automated Generation and Filtering of Competency Questions for Ontologies (2025)

Abstract: Competency questions for ontologies are used in a number of ontology development tasks. The questions’ sentences structure have been analysed to inform ontology authoring and validation. One of the problems to make this a seamless process is the hurdle of writing good CQs manually or offering automated assistance in writing CQs. In this paper, we propose an enhanced and automated pipeline where one can trace meticulously through each step, using a mini-corpus, T5, and the SQuAD dataset to generate questions, and the CLaRO controlled language, semantic similarity, and other steps for filtering. This was evaluated with two corpora of different genre in the same broad domain and evaluated with domain experts. The final output questions across the experiments were around 25% for scope and relevance and 45% of unproblematic quality. Technically, it provided ample insight into trade-offs in generation and filtering, where relaxing filtering increased sentence structure diversity but also led to more spurious sentences that required additional processing

## introduction the use of competency questions (cqs) for ontology scoping, development, and validation is wellestablished since its introduction in 1996 @xcite , as illustrated in, e.g., @xcite @xcite @xcite . authoring cqs is not trivial and a question's wording may be problematic for a number of reasons @xcite . therefore, effort has gone into cq authoring assistance. early efforts went into creating a controlled natural language to assist writing, called claro @xcite , but with the advances and popularization of large language models (llms), the allure of llm-assisted authoring has gained traction @xcite . variants include retrofitting cqs onto an existing ontology using, e.g., a prompting-based approach @xcite , or generating cqs for a prospective ontology yet to be developed, which can be done with training or fine-tuning @xcite or prompting @xcite . while retroactively generating cqs for an existing ontology has usage scenarios relevant for the ontology development lifecycle, we are interested in the scenario where the ontology is yet to be developed, irrespective of, though possibly including, ontology reuse, such as for scoping the subject domain and therewith formulating the requirements. structured cqs can then feed into semi-automated ontology authoring @xcite and querying @xcite . the broad question it raises is how to automate and obtain relevant cqs and to do this in such a manner that the cqs can be traced to the source?. agocqs @xcite aims to cater for this scenario, using the t5 llm, the squad dataset for fine-tuning, filtering with the claro v2 cnl for cqs, and a semantic filtering step. however, it was evaluated with only one use case, a very small corpus of 7 scientific articles, and the effects of the different steps in the pipeline are unclear as only the final output was evaluated. our aims are to focus on fully automating all aspects of that pipeline from text extraction, generation, and filtering, in a traceable manner, possibly enhance it where promising, and test it on another subject domain. specific questions we seek to answer are: to answer this, we refactored the jupyter notebook from @xcite and ran preliminary tests to answer rq-3. in the first experiment, we ran the pipeline with two mini-corpora, one consisting of guidelines and another with scientific documents, and evaluated the generated questions with two domain experts and an ontologist, to answer rq-2 and rq-1, and aimed to answer rq-4 and rq-5. in experiment 2 we modified the pipeline in a number of ways to obtain more fine-grained insights and answers to rq-4, rq-5, and rq-1. the questions outputted by the pipeline for both experiments were around 25% for scope and relevance out of the total evaluated, and when within scope, then they were for 69-75% relevant, with quality from an ontological viewpoint varying between 53% and 40% as acceptable or good cq for ontologies. this was obtained with full automation, cf. the original agocqs that required manual curation. the tracing in the automation provided ample insight into trade-offs. important steps affecting the process are the squad training data set and the filtering step with the claro cnl, and various minor gains were obtain with grammar checking, english checking, and an additional conceptual filter that removed cqs appropriate for conceptual data models and the abox rather than ontologies. in the remainder of the paper, we describe the materials and methods in section 2, present the results in section 3, and discuss and conclude in sections 4 and 5. ## methodology for purposes of being self-contained, this section will first summarise agocqs, and subsequently the materials and methods for the two experiments. ## results the results are presented in order of the experiments. ## discussion the data showed that agocqs+ with additional conceptual filtering and grammar correction, i.e., agocqs++, yielded the best results. ## conclusions we have demonstrated that agocqs++, now a fully automated pipeline, can generate competency questions that have the highest reported rate of being in scope and relevant, as judged by domain experts, where about half of the questions were deemed acceptable as competency questions for ontologies. the pipeline can generate questions for different genres of corpora, being at least scientific articles and guidelines and standards, with no significant difference in quality with respect to scope and relevance. future work will focus on creating a dataset of contexts and competency questions to alleviate the issues that arise due to the usage of squad in the pipeline. further research into metrics for measuring competency question quality will also be of value.