Multilingual Image Corpus: Annotation Protocol (2021)

Abstract: In this paper, we present work in progress aimed at the development of a new image dataset with annotated objects. The Multilingual Image Corpus consists of an ontology of visual objects (based on WordNet) and a collection of thematically related images annotated with segmentation masks and object classes. We identified 277 dominant classes and 1,037 parent and attribute classes, and grouped them into 10 thematic domains such as sport, medicine, education, food, security, etc. For the selected classes a large-scale web image search is being conducted in order to compile a substantial collection of high-quality copyright free images. The focus of the paper is the annotation protocol which we established to facilitate the annotation process: the Ontology of visual objects and the conventions for image selection and for object segmentation. The dataset is designed both for image classification and object detection and for semantic segmentation. In addition, the object annotations will be supplied with multilingual descriptions by using freely available wordnets.

## introduction we are surrounded by information represented by text, images and video data in multimodal streams. one of the processing tasks for large multimodal data streams is automatic image description (image classification, object segmentation and classification), which is directly connected with the task of image search, as well as with the expansion of the scope for automatic question answering regarding images. the goal of our project multilingual image corpus (mic 21) foot_0 is to provide a large image dataset with annotated objects and object descriptions in (at least) 20 european languages. the multilingual image corpus consists of an ontology of visual objects (based on wordnet) and a collection of thematically related images whose objects are annotated with segmentation masks and labels describing the ontology classes. the dataset is designed both for image classification and object detection and for semantic segmentation. the main contributions of our work are: a) the provision of large collection of high-quality copyright free images; b) the formulation of the ontology of visual objects based on wordnet noun hierarchies; c) the precise manual correction of automatic object segmentation within the images and the annotation of object classes; and d) the association of objects and images with extended multilingual descriptions based on wordnet innerand interlingual relations. we have divided the annotation process into four main stages: a) definition of an ontology of visual objects; b) collection of appropriate images; c) automatic object segmentation; and d) manual correction of object segmentation and manual classification of objects. the annotation protocol includes the ontology of visual objects and the conventions for image selection and for object segmentation. the focus of the paper is the annotation protocol which is established to facilitate the manual annotation. we begin with a brief overview of the current state in the field in section 2. section 3 is dedicated to the description of the ontology of visual objects. dataset collection is described briefly in section 4. section 5 provides an outline of the annotation protocol. finally, conclusions and future directions of our work are presented. we will show how the presented image dataset benefits from wordnet: providing ontological representation of visual objects based on wordnet noun hierarchies; building interconnectivity of classes by means of the wordnet relations; and ensuring multilinguality by using freely available wordnets. ## related work there is a tradition already established in the image dataset collection and annotation; the available datasets show an increase both in the number of training images and in the number of object classes. caltech-256 dataset consists of 30,607 images and covers 256 object categories foot_1 (classes). the annotation includes bounding boxes, in which the objects are located, and object outlines provided by humans @xcite . the categories are organised in a taxonomy grouping categories into animate and inanimate and other finer distinctions; for example, the category electronics is divided further into entertainment, computing, home, office and others. the caltech 101 silhouettes foot_2 dataset consists of 4,100 training samples, 2,264 validation samples and 2,307 test samples. the dataset is based on caltech 101 image annotations. each image in the caltech 101 silhouettes dataset includes a highquality polygon outline of the primary object in the scene @xcite . the tinyimages dataset @xcite ) is a large dataset containing 80 million small images (32 x 32 pixels) automatically collected from the internet using 53,464 nouns from wordnet as queries. the dataset is not available online since it has not been manually evaluated 4 . the scene understanding (sun) foot_4 collection contains 899 categories and 130,519 images. sun annotates scene types and the objects that commonly occur in them. there are 397 categories designed to evaluate numerous state-of-the-art algorithms for scene recognition @xcite . the sun attribute foot_5 dataset consists of 14,340 images from 717 scene categories, and each category is annotated with a taxonomy of 102 attributes @xcite . modanet foot_6 is a dataset consisting of annotations of street fashion images. modanet provides multiple polygon annotations for each image. each polygon is associated with a label from 13 meta fashion categories (bag, belt, footwear, outer, dress, etc.), where each meta category groups highly related categories to reduce the ambiguity in the annotation process @xcite . there are several datasets which have been widely used as a benchmark for object detection, semantic segmentation and classification tasks. the pascal visual object classes (voc) 2012 foot_7 dataset contains 20 object categories including vehicles, household, animals, and others: aeroplane, bicycle, boat, etc. each image has pixellevel segmentation annotations, bounding box annotations, and object class annotations @xcite . labelme is a dynamically developing dataset foot_8 which contains hundreds of thousands of polygon annotations, thousands of static images and sequence frames with at least one labelled object @xcite . a particular feature of this collection is that it is being developed by users who can add images and categories and can annotated uploaded images. this option however may result in some level of inconsistency based on the decisions of the different users about the annotation protocol. the wordnet noun synonymous sets (synsets) are used to extend the categories, to avoid the inconsistency by means of manual editing and to unify the descriptions provided by different users. one of the collections that set standards in the increase of datasets sizes is imagenet foot_9 . the aim is for a the dataset with about 50 million cleanly labelled full resolution images @xcite . another important feature of this dataset is that it uses wordnet noun hierarchies for image collection and labelling. imagenet uses 21,841 synsets and contains 14,197,122 annotated images organised by the semantic hierarchy of wordnet (as of august 2014) @xcite . the taxonomic organisation of nouns in word-net allows for using more abstract and fine-grained categories when describing objects. wordnet is a semantic network whose nodes host synonyms denoting different concepts and whose arcs, connecting the nodes, encode different types of relations (semantic: genus-kind, part-whole, etc.; extralinguistic: membership in a thematic domain; inter-language: translation equivalents). the idea for organising the lexicon of a given language into a (lexico-)semantic network was first executed in the princeton wordnet @xcite . some of the fundamental ideas on which the wordnet is based encompass: a) the use of a semantic network which embraces taxonomies, meronomies and non-hierarchical relations with clearly defined properties which allow for quick and easy automatic processing; b) a different organisation of the lexicon in comparison with the traditional dictionaries where words are ordered alphabetically and the links among semantically related words (such as between sister hyponyms, between a whole and its parts, etc.) are not explicitly presented @xcite . the coco (microsoft common objects in context) dataset @xcite ) contains more than 328,000 images with manually annotated object instances (2.5 million) 11 . the dataset has had several releases since 2014 and it addresses object detection, segmentation, keypoint detection and captioning. the different parts of the dataset are annotated with bounding boxes (for object detection) and per instance segmentation masks with 80 object categories; natural language descriptions of the images; keypoints (17 possible keypoints, such as left eye, nose); per pixel segmentation masks with 91 stuff categories, such as grass, wall; full scene segmentation, with 80 thing categories (such as person, bicycle, elephant); dense pose -each labelled person is annotated with a mapping between image pixels and a template 3d model. the image processing is generally classified as model based (using manually-labelled training data) and search based (using automatically collected training data). the search based approaches might include: effective learning mechanisms for matching a given query @xcite ; methods for automatic removing of noisy images @xcite ; frameworks combining discovering of multiple textual queries, filtering of noisy textual queries and noisy images @xcite . in the largest collection of datasets available on the internet 1,455 image datasets are listed foot_11 (as of august 2021) provided with descriptions and links to the sources and related papers. among them 134 datasets are designed for semantic segmentation; 104 -for image classification and 102for object detection. ten datasets provide polygon annotations. to summarise, the tendency in image annotation is from small training datasets to large-scale col-lections which require crowdsourcing in order to engage a large amount of human effort. although the number and the diversity of image datasets is constantly expanding still there is a huge demand for more datasets in terms of variety of domains and object classes covered. ## ontology of visual objects in current practice, wordnet is usually used in generating text queries for creation of search based image collections. a visual concept ontology is proposed which organises visual concepts (objects or abstract notions that are typically depicted in photos) @xcite . for the construction of visual concept ontology over 400 "significant" noun synsets (that have at least 300 hyponyms) are extracted from wordnet, then synsets with very "general" meaning such as entity or thing were removed. this results in 14 top-level ontology classes, which are divided further into 90 more specific classes. semantically similar synsets are merged into a common class and additional links are established between semantically related synsets such as roof and house. we identified 10 thematic domains: sport, medicine, arts, education, food, transport, clothing, security, indoors, nature. the proposed ontology includes concepts which are particular for these domains. following the strategy for category selection of the imagenet we applied the rule for no overlapping between the classes: "for any synsets i and j, i is not an ancestor of j" @xcite . mutually exclusive classes are also defined for other well-known datasets, for example for the coco thing and stuff classes @xcite . as it was pointed out, the mutual inclusion might lead to some inconsistencies. an example was given with the pascal context @xcite classes bridge and footbridge, which are in a parentchild relation @xcite . the parent term can replace the child term in some context, but not vice versa, thus: if two images are annotated as bridge and footbridge respectively, it will not be known whether the parent concept can refer also to the child concept or not. the ontology of visual objects has the following components: classes which can be represented by visual objects and correspond to the respective wordnet concepts. among the classes we made a differ-entiation between dominant classes and attribute (contextual) classes. each thematic domain is represented by several dominant classes, which show the main "players" within this domain differentiated by their type or their function. for example, the dominant classes for the domain sport are: skier, cricket player, hockey player, volleyball player, swimmer, oarsman, etc., altogether 31 dominant classes. for the definition of the dominant classes, we use the word-net sister hyponyms at a certain level (the lowest level allowing classification without specific knowledge for the domain). so far, the selected dominant classes for all thematic domains in focus are 277. for each dominant class a parent class is selected from the wordnet noun hierarchies and this procedure is repeated consecutively up to the final class which represents a visual object. for example, classes like basketball player, acrobat, football player, etc. are hyponyms of athlete 'a person trained to compete in sports'. athlete in its turn is a hyponym of contestant 'a person who participates in competitions' which is a hyponym of person. however, the hypernym of person is organism, an abstract notion, which is not included in the ontology. as a result of this approach, thousands of annotations will be assigned to objects representing small number of classes, while the annotations with more general classes will be inherited automatically. attributes in the ontology are classes related with the dominant ones. the type of the dominant class and the type of attribute class determine the type of the relation between them which expresses the specificity of property attribution: has instrument, wears, uses, has part, etc. for example, the attribute classes for cricketer are cricket bat, cricket ball, cricket helmet, wicket and referee, while for climberclimbing helmet, chalk bag, claiming backpack, and so on. relations between dominant and attribute classes are not hierarchical. for the definition of attribute classes, we use wordnet relations such as meronymy and morpho-semantic relations between nouns. in many cases, such relations are not overtly established in wordnet and they were additionally inserted in the ontology. finally, we made some evaluation tests for all selected classes with other sources providing lists with concrete objects, such as concreteness ratings @xcite and acquisition ratings of words (in our case of nouns) @xcite . so far, we have identified 1,037 classes grouped in ten thematic domains: sport, medicine, arts, education, food, transport, clothing, security, indoors, nature. the relations used in the ontology are relations between classes. part of the relations and their properties are inherited form wordnet. additional relations are included in the ontology in case they are not explicit in the wordnet structure. each class in the ontology is represented by a unique label, which in most cases is one of the synonyms in the corresponding wordnet synset (in case of ambiguity, a descriptive label is constructed). benefits of using an ontology for image labelling can be outlined as follows: what it more, since wordnets for many languages are linked to princeton wordnet @xcite , we will provide multilingual descriptions of the images. freely available wordnets foot_12 with various lexical coverage for 17 official eu languages (bulgarian, croatian, danish, dutch, english, finnish, french, greek, italian, lithuanian, polish, portuguese, romanian, slovak, slovene, spanish, swedish) and for albanian, icelandic, hebrew and serbian will be used for a multilingual representation of the selected classes. ## image collection there are many repositories that can be used for searching and downloading images. some of the images are assigned with multiple labels or short descriptions, which is used to facilitate the automatic collection of appropriate images. for the selected classes a focused web image search is being conducted to compile a database with images -candidates for annotation. so far, more than 450,000 images were collected from different image providers, which are selected on the basis of the following criteria: the repositories should offer an api and images should be licensed with one of the following standards: universal public domain dedication (cc0 1.0); attribution 4.0 international (cc by 4.0) and attribution-sharealike 4.0 international (cc by-sa 4.0) foot_13 . thus, we are avoiding copyrighted material, which might limit the use of our dataset only for academic purposes. after the collection of images, we perform additional manual selection to ensure the quality of the dataset. the following criteria for selection are observed: the final selection of images is triple checked independently by different experts: after the automatic collection, after the automatic generation of segmentation masks and after the manual annotation: correction of the segmentation masks, new polygon outlines and selection of appropriate ## annotation conventions our aim is to provide a dataset that will support image classification, instance segmentation and object detection formats. several open source tools for image annotation (makes sense foot_14 , coco annotator 16 , vgg image annotator foot_16 , labelme foot_17 , labelimg foot_18 , etc.) have been evaluated in order to choose the most appropriate one for our purposes. each annotation tool is usually designed for a specific application and for a specific annotation process. for example, we experimented with the web-based image annotation tool labelme to create polygon annotations; with the desktop annotation tool labelimg to create bounding boxes, etc. to avoid converting annotations for frameworks such as yolact foot_19 , detectron foot_20 , etc., which provide segmentation masks and require coco formatted annotations, we decided to work with the coco annotator foot_21 . the coco annotator can be containerised, allows for simultaneous work on a project, and offers useful functions that facilitate image annotation: tracking object instances, labelling objects with disconnected visible parts, etc. it is a known fact that semi-automatic annotation approaches can significantly speed up the annotation process by automatic generation of annotation proposals to support the annotators. the main idea is to reduce the human interaction with the annotation tool and to save time, while maintaining the quality of the annotations. we experimented with mask r-cnn and yolact, which provides instance segmentation on datasets like coco. mask r-cnn @xcite is an implementation based on python 3, keras and tensorflow. the model generates bounding boxes and segmentation masks for each instance of an object in the image. yolact @xcite ) is a framework, which breaks up instance segmentation into two parallel tasks: a) generating a dictionary of nonlocal prototype masks over the entire image, and b) predicting a set of linear combination coefficients per instance. since the number of prototype masks is independent of the number of classes (e.g., there can be more classes than prototypes), yolact learns a distributed representation in which each instance is segmented with a combination of prototypes that are shared across classes. the task for the annotators is to outline polygons for individual objects in the image (either by approving or correcting the automatic segmentation or by creating new polygons) and to classify the objects against the classes from the predefined ontology. the annotation follows the following conventions (only the more significant ones are listed here): the quality control is provided by a second an-notator who validates the implementation of the conventions and discusses the quality with the annotation group weekly. if necessary, some of the images are re-annotated. ## conclusion and future work the multilingual image corpus will provide pixel-level annotations for the selected dominant classes and their parent and attribute classes in ten thematic domains, thus offering more data to train models specialised in object detection, segmentation and classification in these domains. the selected classes for annotation are organised in an ontology of visual objects that provides options to organise annotated images in different datasets regarding the envisaged tasks. the multilingual image corpus will be released in autumn of 2021 and will provide: a) a large number of copyright-free images, b) a large number of object classes organised in an ontology, c) a large number of pixel-level annotations; and d) extended image descriptions in (at least) 20 languages based on wordnet. an important result with great significance for the development of different applications for image processing will be the open distribution of the collection. we are currently planning some experiments with a set of state-of-the-art algorithms on each of the tasks of object detection and segmentation, in order to establish a common baseline for future work.