Fatality Killed the Cat or: B abel P ic, a Multimodal Dataset for Non-Concrete Concepts (2020)

Abstract: Thanks to the wealth of high-quality annotated images available in popular repositories such as ImageNet, multimodal language-vision research is in full bloom. However, events, feelings and many other kinds of concepts which can be visually grounded are not well represented in current datasets. Nevertheless, we would expect a wide-coverage language understanding system to be able to classify images depicting recess and remorse, not just cats, dogs and bridges. We fill this gap by presenting BabelPic, a hand-labeled dataset built by cleaning the image-synset association found within the BabelNet Lexical Knowledge Base (LKB). BabelPic explicitly targets non-concrete concepts, thus providing refreshing new data for the community. We also show that pre-trained language-vision systems can be used to further expand the resource by exploiting natural language knowledge available in the LKB. BabelPic is available for download at http://babelpic.org .

## introduction there is growing research interest in developing effective systems capable of achieving some understanding of the content of an image. as in most fields of applied ai, this requires annotated data to train a supervised system on. while imagenet 1 @xcite , one of the most influential projects in computer vision, was undeniably an important milestone towards image understanding, there is still a lot of ground to be covered. ima-genet's initial aim was to collect pictures for most wordnet synsets @xcite . yet, at the time of writing, only some 21,841 nominal synsets are covered according to imagenet's official website. one issue with imagenet and most other image repositories like coco @xcite ) and 1 http://www.image-net.org flickr30kentities @xcite is their focus on concepts denoting concrete, tangible things, such as cat, traffic light and so on. concepts whose denotation is not clearly identifiable with a set of objects having distinct boundaries, such as events (e.g., fatality, competition), emotions (e.g., sadness) and psychological features (e.g., sharpness), have enjoyed less attention. for lack of a better term, we will henceforth refer to them as non-concrete (nc) concepts. on one hand, the inclusion of nc concepts would be an important step towards wide-coverage image semantic understanding. on the other hand, it also goes in the same direction as recent multimodal language-vision approaches, e.g., monoand cross-lingual visual sense disambiguation @xcite @xcite @xcite . taking into account nc concepts could also be of crucial importance for fascinating languagefocused applications, such as multimodal machine translation. last but not least, nc concepts would represent a significative benchmark for real-world multimodal applications. in fact, traditional computer vision approaches rely on the detection of objects within the image, but many nc concepts are not well described by a bag of objects. consider, for instance, figure 1 . the two images illustrate different nc concepts (i.e., high jump and pole vault) which are different configurations of the same elementary objects (i.e., person, rod, bleachers). thus, nc concepts require complex image understanding, integrating a fair amount of common sense knowledge. as a contribution towards this goal of expanding the scope of research, we introduce babelpic, the first dataset for multimodal language-vision tasks with a focus on nc concepts and that is also linked to wordnet. babelpic has been built by manually validating synset-image associations available in babelnet @xcite , a large multilingual resource linking wordnet to wikipedia and other resources. furthermore, we provide a methodology to extend the babelpic coverage to all the babelnet synsets. to this end, we adapt the recently introduced vision-language pre-training (vlp) model @xcite . we define the verification of synset-image associations as a visual question answering (vqa) task with two possible answers. the evaluation demonstrates that our methodology achieves high performances on zero-shot classification as well, thus enabling verification across the inventory. thanks to the automatic production of a silver dataset, babelpic constitutes a significant extension of imagenet. a few examples from ba-belpic (both gold and silver) are shown in figure 2 . ## related work to the best of our knowledge, no dataset of annotated images exists which has a focus on nc nominal and verbal concepts and is also linked to lexical knowledge bases (lkb) such as wordnet and ba-belnet. for example, the very popular imagenet dataset, which includes images belonging to around 21,800 categories organized according to the word-net nominal hierarchy, offers only sparse coverage of nc concepts. jft @xcite @xcite is an internal dataset at google containing 300m images annotated with over 19,000 classes including objects, scenes (e.g., sunset), events (e.g., birthday) and attributes (e.g., red). jft differs from our work in not being linked to an lkb and in not being publicly released. the open images dataset @xcite ) contains 9m images annotated with 19,794 classes taken from jft. while open images does contain nc labels, the classes are not linked to an lkb, thus limiting their usefulness. the tencent ml-images dataset @xcite was created starting from a subset of imagenet and open images and includes images annotated with 11,166 categories, which are then linked to word-net synsets. the dataset differs from our work since any nc label has been explicitly discarded. our work is in some sense similar to multisense @xcite and verse @xcite , two datasets including images annotated with verbal senses. however, multisense is not directly linked to an lkb and neither of these two datasets deals with nominal synsets. finally, we note that datasets including images annotated with objectlevel categories @xcite or videos @xcite @xcite @xcite are outside the scope of this work, since we are only interested in the main nc concepts depicted within images. ## gold dataset babelpic is built by exploiting the link between wordnet @xcite and wikipedia within ba-belnet foot_0 @xcite . our approach is organised in a three-step process. first, we select a set of nc synsets from wordnet, on the basis of both their paradigmatic nature and relations in the knowledge base. second, we gather all the corresponding images in babelnet, which are themselves mostly taken from wikipedia pages. third, we manually validate the synset-images mapping. note that, having defined the task as a validation of concept-image associations, we do allow images to be mapped to more than one concept and vice versa. for instance, both images in figure 1 could be mapped to the concept competition as well. the result is a gold dataset containing 2,733 synsets and 14,931 images. ## model since manual validation is time consuming, we are interested in developing a methodology for the automatic verification of synset-image associations. in the recent past there has been a great research effort to develop models for vision-language pretraining. many such models (e.g., vlp @xcite , @xcite , vil-bert @xcite , lxmert @xcite ) are built upon bert @xcite , a popular system for contextualized embeddings. bert-based models achieve state-of-the-art scores on many language-vision tasks, hence they represent a promising resource for our task. the system that we use to perform classification is the fine-tuned vlp model. despite the fact that lxmert @xcite achieves a slightly higher score on yes/no questions on the vqa 2.0 dataset @xcite , our preference goes for the vlp system since it is pre-trained on a wider and more general dataset. more specifically, the vlp model is pre-trained on conceptual captions (cc) @xcite , a dataset including more than 3m image-caption pairs, using two unsupervised vision-language tasks: bidirectional and sequence-to-sequence masked language prediction. the input images are preprocessed using faster r-cnn @xcite pre-trained on visual genome @xcite , hence obtaining 100 object regions per image. the model input consists of both class-aware region embeddings and word embeddings, the former obtained by combining the corresponding region features with the probability of each object label and region geometric information. furthermore, a multi-layer perceptron (mlp) is trained during the fine-tuning phase in order to select the chosen answer starting from the hidden state of the encoder. in order to adapt the vlp model to extend the babelpic coverage to all the babelnet synsets, we define the verification of synset-image associations as a vqa task with two possible answers. more specifically, we define a question template as in the following: ## experiments to test the reliability of our approach for the automatic verification of concept-image associations we experiment in a zero-shot setting (see section 5.3). as a first step toward this goal, we need to augment our dataset with negative instances (see section 5.1) and select the most suitable vlp version (see section 5.2). a deeper analysis of how the sampling of negative instances affects the performances of the system is described in section 5.4. ## conclusions in this work we introduced babelpic, a new resource for language-vision tasks, built by validating the existing image-to-synset associations in the babelnet resource. babelpic is innovative in being the first dataset with a focus on nominal and verbal non-concrete concepts linked to the word-net and babelnet lexical knowledge bases. furthermore, we presented a methodology to extend the resource by fine-tuning vlp, a state-of-the-art pre-trained language-vision architecture. in our approach, we automatically verify the synset-image associations by exploiting the natural language definitions in wordnet, showing strong results on zero-shot classification as well. we exploited our method for the automatic generation of a widecoverage silver dataset containing around 10,013 synsets. we make babelpic (both gold and silver data) available to the community for download at