Ontology of Visual Objects (2022)

Abstract: The focus of the paper is the Ontology of Visual Objects based on WordNet noun hierarchies. In particular, we present a methodology for bidirectional ontology engineering, which integrates the pre-existing knowledge resources and the selection of visual objects within the images representing particular thematic domains. The Ontology of Visual Objects organizes concepts labeled by corresponding classes (dominant classes, classes that are attributes to dominant classes, and classes that serve only as parents to dominant classes), relations between concepts and axioms defining the properties of the relations. The Ontology contains 851 classes (706 dominant and attribute classes), 15 relations and a number of axioms built upon them. The definition of relations between dominant and attribute classes and formulations of axioms based on the properties of the relations offers a reliable means for automatic object or image classification and description.

## introduction the recent trends in computer vision are directed towards the robust combination of deep learning techniques and image processing methods to solve problems, such as image and video understanding, robot vision and processing of multimodal and multilingual content. despite this, much effort is still directed to specific domain knowledge or even to specific object instance recognition, and the significant progress in the field as a whole does not mean that particular tasks have been solved satisfactorily. the concept of cognitive vision @xcite ) was introduced quite a long time ago @xcite : "a cognitive vision system can achieve the four levels of generic computer vision functionality of detection, localization, recognition, and understanding. it can engage in purposive goal-directed behaviour, adapting to unforeseen changes of the visual environment, and it can anticipate the occurrence of objects or events" @xcite . such understanding of cognitive vision systems involves the application of ontology-based representations in modern computer vision systems in order to add real world relations between static objects and video feed @xcite . ontology-based applications might be powerful tools for diverse computer vision tasks: application of semantics according to the function of an object @xcite , ontology-based object recognition in robotics @xcite , and so on. the focus of the paper is the ontology of visual objects. 1 in particular, we present a methodology for bidirectional ontology engineering, which integrates the pre-existing knowledge resource (word-net) and the selection of visual objects within the images representing particular thematic domains. we show how the presented ontology benefits from wordnet @xcite : providing ontological representation of visual objects based on wordnet noun hierarchies, and building interconnectivity of classes by means of the wordnet. on the other hand, we present how the ontology of visual objects builds on the wordnet by adding new concepts corresponding to concrete objects, and formulating new relations that express the objects' function, purpose, location, etc. we begin with a brief overview of the current state in the art in section 2. in section 3 we present the principles of ontology-based image annotation. section 4 is dedicated to the main components of an ontology and the description of the ontology of visual objects. finally, evaluation (section 5), conclusions and future directions of our work (section 6) are presented. ## related work in this section, we briefly present some of the most prominent knowledge representation resources, the image datasets, which involve (in different ways) ontologies in the process of their building, and the few existing examples of ontologies specially dedicated to image descriptions. ## ontology-based image annotation the ontology of visual objects was developed to serve for the annotation of the image objects in the multilingual image corpus, 9 which provides pixel-level annotations, thus offering data to train models specialised in object detection, segmentation and classification in these domains @xcite . different ways of incorporating semantics to describe an image are discussed @xcite . one possible level incorporates the relations between concrete and abstract objects, for example, a crying person vs. the notion of pain, which might be a subjective conclusion based on the knowledge of the semantic context. the other level describes generic vs. specific objects (individual instances), i.e., a bridge vs. golden gate bridge. in our approach, we concentrate on visual (concrete) objects; however, specific instances of an object can be further related with it, and further inferences to abstract notions might be drawn as well. we defined the following criteria for the development of the ontology of visual objects: the proposed ontology of visual objects includes concepts that are characteristic for the thematic domains of sport, transport, arts, and security. the multilingual image corpus contains 130 smaller datasets pertaining to different subdomains, each of which can be classified to one of the four main ones, for example, chess and pole vaulting are subdomains of sport, while sedan and double-decker -to transport, and so on. the choice of thematic domains and subdomains is motivated by two main factors: ontologies are classified into three basic types: top ontologies, which contain a restricted set of general classes and are not related to a particular thematic domain; top-domain ontologies, which include essential classes that represent a particular thematic domain; and domain ontologies, which contain classes that comprehensively describe a particular thematic domain @xcite . from the point of view of this classification, the proposed ontology can be classified as a set of several domain ontologies. the ontology of visual objects provides options for extracting relationships between annotated objects, between diverse datasets with different levels of granularity of object classes, or between appropriate sets of images illustrating different thematic domains. last but not least, the use of the ontology of visual objects allows the expansion of the dataset depending on the specific needs of scientific or commercial projects. the annotators' tasks were to create new polygons or approve or modify the automatic segmentation for objects in the images, and then classify the objects according to the specified ontology's classes. the annotation adheres to the following conventions: the following are some advantages of utilizing an ontology for object classification: ## ontology of visual objects it was pointed out that different knowledge representations share the following minimal set of components @xcite : concepts, which represent sets or classes of entities in a thematic domain; relations between concepts; instances, which represent the actual entities (individuals); and axioms, which represent facts that are always true in the topic area of the ontology. we accepted the following definition @xcite : an ontology is a structure the ontology of visual objects organizes concepts (represented by dominant classes, classes that are attributes to dominant classes and classes that serve only as parents to dominant classes), relations between concepts and axioms. ## evaluation a number of studies aimed at ontologies' evaluation are known @xcite @xcite @xcite . on their basis several criteria for the evaluation of ontologies can be defined directed to confirm the ontology quality and correctness: we can define our approach for the evaluation as a corpus-based approach @xcite . instead of comparing an ontology with the content of a text corpus that covers significantly a given domain, we use the image annotation process to evaluate the ontology of visual objects. at the beginning, we have identified 1,037 classes grouped in ten thematic domains: sport, medicine, arts, education, food, transport, clothing, security, indoors, and nature. for four of them (sport, transport, arts and security) an evaluation of the ontology classes is performed during the annotation: whether a class is a visual object or not; whether all depicted objects in selected images can be described with the ontology classes; and whether new classes can be added if necessary. for the definition of classes we rely on the definition of concepts in wordnet; the definition of new classes is provided by means of finding their correct place within the wordnet taxonomy by linking them with already defined concepts. finally, we made some evaluation tests for all selected classes with other sources providing lists with concrete objects, such as concreteness ratings @xcite , word acquisition ratings (in our case of nouns) @xcite and picture dictionaries @xcite . ## conclusion and future work to improve object annotation and classification, several approaches based on ontologies have been proposed. however, image classification and annotation remain a challenging problem and one of the reasons is possible overlapping of selected classes. the use of a specially designed ontology improves the speed of object annotation as well as the accuracy of object classification. our contributions consist of the following: using the ontology of visual objects ensures the selection of mutually exclusive classes, builtin interconnectivity of classes via formal relations, and the ability to easily extend the proposed ontology with more concepts corresponding to visual objects. applying semantics can improve not only the performance of object recognition but also the performance and quality of individual tasks required for object recognition, such as image segmentation. furthermore, the ontology can be used to reduce the gap between human image comprehension and machine image interpretation, allowing for better automation in training neural networks @xcite . a possible application of the ontology of visual objects includes further use of the relations to compile bigger training datasets (for example, utilizing higher level concepts) or to construct contexts in which a particular object may or may not appear. the ontology of visual objects provides options for extracting: relationships between annotated objects, diverse datasets with different levels of granularity of object classes and appropriate sets of images illustrating different thematic domains. the ontological organization of object classes provides data for learning associations between objects in images, for identifying relations between objects and for aligning objects and relations with text fragments. last but not least, using the ontology of visual objects enables the dataset to be expanded based on the particular needs.