Image Models for large-scale Object Detection and Classification (2022)

Abstract: Recent developments in computer vision applications that are based on machine learning models allow real-time object detection, segmentation and captioning in image or video streams. The paper presents the development of an extension of the 80 COCO categories into a novel ontology with more than 700 classes covering 130 thematic subdomains related to Sport, Transport, Arts and Security. The development of an image dataset of object segmentation was accelerated by machine learning for automatic generation of objects’ boundaries and classes. The Multilingual image dataset contains over 20,000 images and 200,000 annotations. It was used to pre-train 130 models for object detection and classification. We show the established approach for the development of the new models and their integration into an application and evaluation framework.

## introduction the shift of traditional data fusion methods challenged by multimodal big data motivates the creation of a new image corpus, the multilingual image corpus, which is characterised by carefully selected images that illustrate thematically related domains and precise manual annotation for the segmentation and classification of objects in the images. recent developments in computer vision applications that are based on machine learning models allow real-time object detection, segmentation and captioning in image or video streams @xcite . we developed an image processing pipeline for object detection and object segmentation using pre-trained models. we also delivered a reliable service for automatic annotation of objects in images using advanced deep learning techniques @xcite and some existing tools and machine learning frameworks. the paper presents the development of an extension of the 80 coco categories into a novel ontology with more than 700 classes covering 130 thematic subdomains related to sport, transport, arts and security (presented in section 2). the development of an image dataset for object segmentation and classification (the multilingual image corpus, mic21) was accelerated by machine learning for automatic generation of objects' boundaries and classes (presented in section 3). the mic21 image dataset containing more than 20,000 images and 200,000 annotations was used to pre-train 130 models for object detection and classification. we show the accepted approach for the development of machine learning models and their integration into a framework for the evaluation and running of models (in section 4). in other words, we will demonstrate the application of models for the prediction of object outlines and classes in images as part of the development of the multilingual image corpus, and then we will show how the new dataset, in turn, can be used to pre-train existing models so that they predict large number of object classes. ## multilingual image corpus in brief the multilingual image corpus offers data to train models specialized in object identification, segmentation and classification by providing fully annotated objects within images with segmentation masks categorised according to an ontology of visual objects. the multilingual image corpus is distinguished by the following key features: a) large image collection containing thousands of images and annotations; b) an ontology of visual objects specifically created for object classification; c) preparatory automatic object segmentation and classification evaluated by experts; d) translation of object classes and attaching definitions of concepts in 25 languages. the dataset contains images from 4 thematic domains (sport, transport, arts and security), which represent highly related objects such as tennis player and soccer player, limousine and taxi, singer and violinist, fire engine and police boat grouped in 130 subsets of images. the images in the dataset are collected from a range of repositories offering api: wikimedia, pexels, flickr, pixabay, creative commons search. each image is equipped with a metadata description in json format. the metadata include fields such as: the name of the sub-dataset, sub-dataset id, image author, author's web address, image original size, file name, image license, image source, last access to the source, source's web address, mic21 project url, etc. @xcite the selected classes for annotation are organized into an ontology of visual objects @xcite . the ontology consist of 706 classes that describe visual objects, 147 classes that represent their hypernyms, 14 relations between concepts and axioms that make explicit claims about the relations between concept classes. the ontology classes correspond (but are not limited) to wordnet concepts @xcite which can be represented by visual objects (almost half of the ontology classes are not contained in the wordnet). two of the relations and their properties are also inherited from wordnet. for example, the dominant class accordionist is represented in wordnet, while the dominant class handball player -not. for new dominant classes the appropriate hypernym in the wordnet structure is determined, in this case -athlete. the attribute classes for handball player are: handball referee, handball court, handball, handball goal, handball jersey, handball pants, handball shorts, handball shoe, handball sock, race number, knee pad, from which only handball and knee pad are part of the wordnet. ontology relations between the dominant class and its attribute classes link depicted relations between visual objects, for example: handball player is next to handball referee, handball player plays at handball court, handball player plays with handball and so on. the use of the ontology of visual objects ensures the selection of mutually exclusive classes, the interconnectivity of classes by means of formal relations and an easy extension of the ontology with more concepts corresponding to visual objects. all ontology classes have been translated into 25 languages using publicly available wordnets and babelnet: english, albanian, bulgarian, basque, catalan, croatian, danish, dutch, galician, german, greek, finnish, french, icelandic, italian, lithuanian, polish, portuguese, romanian, russian, serbian, slovak, slovenian, spanish, and swedish @xcite . an image processing pipeline for object detection and object segmentation was developed. two software packages -yolact @xcite and detectron2 @xcite , and fast r-cnn (girshick, 2015) models trained on the coco dataset were used for the generation of annotation proposals. the coco format is a commonly used format for the instance segmentation representation @xcite @xcite . the task for the annotators was to correct, reject or create new polygons for individual objects in the image and to classify the objects against the classes from the predefined ontology. the metadata of images, ontology, object annotations and multilingual descriptions of ontology classes are available to be downloaded, copied, modified, distributed, displayed and used in accordance with the creative commons attribution-sharealike 4.0 international license. 123 the multilingual image dataset can be implemented in: automatic identification and annotation of objects in images (a prerequisite for effective search of images and (within) video content), automatic annotation of images with short descriptions in european languages. we developed a framework based on fiftyone, yolact and detectron2, and implemented it over mask r-cnn on python3, keras and tensorflow. we pre-trained fast r-cnn models using the de-tectron2 framework with ground truth annotations, which resulted in 130 models that generate bounding boxes and segmentation masks for each instance of a particular object within an image. the framework maintains an api functionality for processing new images with any of the three models: yolact, detectron2 and mic21. the mic21 framework allows for evaluation and comparison of the grand truth and mic21 models annotations as well as for running the models on new image datasets. 4 we will present in more detail the integration of existing models in our image processing pipeline in order to automatically predict the objects' boundaries in images and their classes, as well as the development of 130 models based on the multilingual image corpus, which can be used for object recognition and classification and for future experiments. ## image processing pipeline the image processing pipeline contains a number of modules that support the work of annotators in several ways: by predicting object outlines and by managing images and annotations. the multilingual image corpus is organized into manageable in size datasets containing at least 100 images (in rare cases) and in the most common case -about 150 images. the initial datasets roughly correspond to the final thematic subdomains that will be formed. however, in the preliminary stage of the work, it was acceptable to have several datasets representing a single thematic subdomain, images classified in inappropriate datasets, etc. therefore, the initial organization of the images into datasets is mostly with respect to the collection methodology and the decision on the size of the data. after the initial processing of the images and the manual annotation, some images are reorganized, if necessary, which reflects the final content of the thematic subdomains. all input images are represented in raster image compression formats such as: portable network graphic (png), joint photographic experts group (jpeg) or tag image file format (tiff). the size of images varies considerably between 2 to 11 megapixels. the small image dimensions may affect the quality of the annotated regions. on the other hand, when the image dimensions are too large, the amount of allocated memory, as well as the processing time, increases exponentially. another requirement for the input image is its colour space format to be in red, green and blue (rgb) channels without additional or missing channels. for an effective processing of images with a convolution model they have to be in proper dimensions and in the rgb color space. hence, the first module of the pipeline examines each image and performs the necessary transformations (resizing and/or color space mapping); in case the transformations are not possible, the image is excluded from the dataset. as a result, the images are described by their attributes as json objects. the pre-processing step is automated by a python script, which calls some of the opencv foot_4 routines for performing the operations over the images. the open source yolact @xcite provides several convolution neural network models for object detection and segmentation within the coco domain @xcite . the model we have employed for the automatic annotation is resnet50-fpn. the notation indicates that each convolution layer from the backbone component of the model includes a feed-through connection from the input to the output of the layer. such structure is appropriate for training highly stacked models (as those used for computer vision tasks) in order to improve the numerical stability of the optimization procedure and to prevent over-fitting. the intuition behind this is that each convolution layer from the stack is approximating only the residual error between the target and the output of the previous layer. first the data is processed through a stack of input convolutional layers with decreasing resolution. consequently, a stack of output convolutional layers with increasing resolution is applied. the dimension of each layer from an input stack is matched by a layer from the output stack. in addition, 1x1 convolutional connections are established between the corresponding layers from the input and the output stacks. the result from dataset processing through yolact software is the instance segmentation and classification within the coco domain. the results are stored in ms coco json format. the format provides two options for recording the bounding contour of each detected object -run-length encoding (rle) and point coordinates. the rle is a compression format over the point coordinates and allows for more compact representation; however, not all systems are able to work with it directly. a useful tool for converting between formats is the python library pycocotools. in some cases, more processing is required because the raw segmentation mask resulting from the convolution model is a binary mask. for the conversion of a binary mask to an object contour a useful routine from opencv library, findcontours, is used. as noted, the object detection model produces annotation data in the domain of the 80 coco categories. the automatically obtained annotation data are imported in an open-source annotation software, the coco-annotator @xcite . the process is automated through a bash script, which connects to the database docker of the annotator, creates a new dataset, copies the images and imports the annotation data. the coco-annotator features multi-user environment composed of a mongodb database, flask backend and vue frontend employing a worker processing model. in the coco-annotator, software images are organized into datasets and the front-end provides a tool for performing manual editing of annotation contours creating/deleting annotations or changing/assigning object labels. to further accelerate manual annotation an automatic relabelling of the imported annotations in the coco-annotator database is implemented. it takes as an input a dictionary that states the relabelling rules specific for a sub-dataset. for example, the category 'person' in the sub-dataset 'basketball' is replaced with the class 'basketball player' and the identifier of the new class replaces the identifier of all annotations 'person' within the sub-dataset 'basketball'. certain manipulations during the manual annotation were performed to provide functionalities that are not implemented in the coco-annotator software. we have developed a dedicated python script performing such operations by connecting to the mongodb engine of the annotation software using pymongo python library: python scripts foot_5 execute each of the described operations. after manual annotation and database postprocessing, the images from the resulting 130 thematic subdomains are exported together with their ground truth annotations represented in ms coco format. the structure of the mic21 dataset is as follows: the data sub-directory for the respective thematic subdomain contains the images in jpg, jpeg or png format. the * gt.json field is a coco format json file describing the polygonal segmentation of objects in images. ## mic21 models we trained domain specific models, which are able to detect and label objects in an image with classes from the mic21 ontology. the benefits of such models are twofold -first, they allow further acceleration of the manual annotation reflecting the mic ontology of visual objects; second, they represent an extension of the standard coco classes to 130 thematic domains. for the training of the domain specific models we use detectron2 framework of the facebook research group 7 , which is an opensource python software based on pytorch library. the general structure of the object detection model is presented in figure 1 . it is composed of a convolution neural network (cnn) backbone component, a proposal generator and a region of interest (roi) head @xcite . detectron2 supports 3 backend structures -resnet, regnet and fpn. the backbone is represented as stacked convolution layers with different interconnections depending on the structure type. in the resnet structure, the residual building block has an option for a direct shortcut connection from the input of the layer to its output, i.e. projecting the input features into the output, and the actual network is keeping the difference between the input and the target. the output of the backend component comprises selected feature maps from the stack of layers depending on the network design. usually, in addition to the output of the last layer, 3 to 4 of the output feature maps from the 7 https://github.com/facebookresearch/ detectron2 deeper layers are selected. region proposals in the modern object detection networks are generated through a region proposal convolution network (rpn), composed of a 3x3 convolution layer followed by 1x1 convolution layers for the generation of object box deltas and an objectiveness score for a box. as a basis for region proposals, a set of anchors is generated within the image, for example, by dividing the image into a grid of large boxes and putting an anchor point at the centre of each box. then the anchor boxes are refined during the training of the network by fitting them to the ground truth. the result of the proposal generator layer is a list of box coordinates and an objectiveness score indicating whether the respective box contains an object or not. the third stage in the object detection framework is a roi head network, which iterates over each of the generated proposal boxes and performs per region classification and binary mask extraction. the prediction is based on the features from the backbone layer constrained to the current examined box. each of the trained models is characterised by input data, output data and parameters. the parameters represent the internal weights of the model obtained during training, which are specific for each thematic subdomain and have to be loaded through a detectioncheckpointer class. the input for the model is a python list structure list[dict], where each element of the list is a dictionary field image, representing a 3-dimensional array with colours for each pixel from the image in rgb colour space, and also width and height attributes for the image in pixels. if the model weight is updated during the training, the input dictionary for each image has to include a field instances describing the coordinates of the bounding boxes for the ground truth objects in the image, as well as a class label for each object in the range [0,num_categories] and a ground truth binary mask for each object. the training of the models is performed with the detectron2 framework by inheriting the defaulttrainer class. the training loop and each of the network layers are aligned with the pytorch requirements for building neural network models. each layer has to provide a loss function, which calculates the residual error given the training targets and network outputs, and additionally to provide a forward function, which calculates the layer outputs from inputs. to compute the gradients during the backward network pass the pytorch features an autograd engine, which (when enabled) is able to track each arithmetic operation during the forward pass and to obtain the gradient of the residual error of the layer with respect to the parameters. during the training, the model is evaluated periodically when a certain iteration count is passed by tracking the intersection over union (iou) metric by category. the library pycocotools contains useful routines for comparing results from computer vision models either by using bounding boxes or binary masks. the iou metric is defined as: with respect to the classification outcome for a given iou threshold, three additional metrics for the model are commonly examined, which are model precision ## framework for running and evaluation the mic21 models for the purposes of presentation, comparison and evaluation the dataset is organized into a system of components, called 'mic21 framework' (figure 2 ). the framework is composed of a backend (processing service) and frontend (visualization service) component. the processing service is implemented as a flask server implemented in python, which is able to run the yolact, detectron2 and mic21 pre-trained domain specific models (figure 2 ). the processing service offers a set of web apis implemented over an http, with the following functions: the repository for the framework is freely accessible at github and includes: source code of mask r-cnn built on fpn and resnet101, training code for ms coco, pre-trained weights for ms coco and mic21 classes, jupyter notebooks for visualization the detection pipeline and evaluation routines for ms coco metrics integrated in the fiftyone. the original fiftyone code is extended with a function, which allows a user to upload a new image into a selected mic21 sub-dataset. when uploaded, the image is automatically processed by the backend service and it is annotated independently by 3 different models (yolact, detectron2 and mic21). the results can be compared in the fiftyone. ## results and evaluation the mc21 object detection models produce the output in four components: the fiftyone framework integrates functionalities to compare different object detection models (in our case mic21 model outcomes and the ground trough annotations). as each image can represent many objects from different classes, the comparison shows the correct classification for a given object within an image. overall, the results can be summaries as follows: ## conclusion and future work the multilingual image corpus offers data to train models specialized in object detection, segmentation, and classification by providing fully annotated objects within images with segmentation masks, categorised according to an ontology of visual objects. the ontology of visual objects allows easy integration of annotated images in different datasets as well as learning the associations between objects in images. models trained on the coco dataset were used for the generation of annotation proposals. we developed a framework based on fiftyone, yolact and detectron2, and implemented it over mask r-cnn on python3, keras and tensorflow. we pre-trained fast r-cnn models with the mic 21 dataset, which resulted in 130 models that generate bounding boxes, segmentation masks and object classes. the mic 21 framework supports web-based visualization, evaluation and comparison of different models together with the ground truth annotations. we can provide a number of alternatives for completing multimodal tasks using the created datasets, including automatic image caption generation aligning sentences with images in various multimodal documents and visual question answering. interpreting an image and the brief text that goes with it, such as a caption, a question or a description of the objects in the image, can be a supporting task. prospective developments also include: automatic extension of the dataset using the pre-trained models, which will considerably accelerate manual annotation in the target thematic domains; training models for automatic image captioning @xcite or question answering @xcite ; adaptation of the pre-trained models for video processing @xcite ; identification (automatic generation) of images representing particular objects or particular textual descriptions; application in motion analysis systems.