V isual S em: a high-quality knowledge graph for vision and language (2021)

Abstract: An exciting frontier in natural language understanding (NLU) and generation (NLG) calls for (vision-and-) language models that can efficiently access external structured knowledge repositories. However, many existing knowledge bases only cover limited domains, or suffer from noisy data, and most of all are typically hard to integrate into neural language pipelines. To fill this gap, we release VisualSem: a high-quality knowledge graph (KG) which includes nodes with multilingual glosses, multiple illustrative images, and visually relevant relations. We also release a neural multi-modal retrieval model that can use images or sentences as inputs and retrieves entities in the KG. This multi-modal retrieval model can be integrated into any (neural network) model pipeline. We encourage the research community to use VisualSem for data augmentation and/or as a source of grounding, among other possible uses. VisualSem as well as the multi-modal retrieval models are publicly available and can be downloaded in this URL: https://github.com/iacercalixto/visualsem .

## introduction publicly-available multilingual resources such as wikipedia are crucial when used as knowledge bases in recent neural language models (lms) that retrieve documents in order to solve complex tasks such as open-ended question answering @xcite . however, wikipedia's wealth of visual information is typically hard to use, 1 which prevents models from including rich multi-modal data. although a small number of structured knowledge graphs (kgs) with images exist and are publicly available, they either cover lim- * work initiated while in the university of amsterdam during her msc. research. in this paper, we introduce visualsem, a multilingual and multimodal kg with ∼ 90k nodes, ∼ 1.3m glosses and ∼ 938k images. visualsem's nodes denote concepts and named entities, include multiple curated illustrative images, as well as glosses in up to 14 diverse languages. typed semantic relations connect nodes in the graph, and nodes in visualsem are linked to wikipedia articles, wordnet synsets, and (when available) highquality images from imagenet @xcite . visualsem integrates seamlessly with existing resources. compared to existing multimodal kgs, visualsem includes data from different sources and thus it is also more diverse in terms of the domains. we source the images in visualsem using babelnet @xcite , a large multilingual and multimodal resource that semiautomatically aggregates information from many different sources. we address the known issue of noisy images in babelnet @xcite by applying multiple filtering steps to ensure that we remove noise while maintaining breadth of coverage and image diversity. we also release pre-trained models to retrieve entities from visualsem using either images or sentences as queries in a k-nearest neighbor search. this effectively allows researchers to integrate entities and facts in visualsem into their (neural) model pipelines. code to generate and download visualsem, as well as retrieval models, is publicly available. figure 1 : example nodes in visualsem, some of their glosses and images, and how they relate to other nodes. we also show examples of images we collected for the nodes that were filtered out and that were kept in following our data collection pipeline (section 2.1). our main contributions are: this paper is organised as follows. in section 2 we explain how we build visualsem and provide details on its data collection pipeline. in section 3 we include dataset statistics and also analyse visu-alsem's content and structure qualitatively. in section 4, we describe the multi-modal retrieval models that use sentences and images to retrieve entities from the kg. in section 5 we discuss relevant related work, including existing multi-modal knowledge bases and how they compare to our work. finally, in section 6 we discuss our main findings as well as provide avenues for future work. ## approach visualsem is a multilingual and multimodal knowledge graph consisting of 89, 896 unique nodes and 1, 481, 007 facts, where each fact is a @xmath0, r j , n @xmath1. ## data statistics and analysis in this section, we explore the structure and content of visualsem. in section 3.1, we report relevant statistics, as well as show some examples of nodes and relations in it. in section 3.2, we provide a more qualitative analysis using unsupervised topic models to induce topic distributions for visualsem. ## entity retrieval in this section, we describe retrieval models that retrieve entities in the kb given a sentence or an image. our use-case includes tasks with possibly multi-modal inputs-i.e., either a sentence, an image, or both-where we use these inputs to retrieve relevant entities from visualsem. the idea is to ground the task inputs or to augment the available data to train the task model. this task could be textonly, such as named entity recognition or machine translation, or multi-modal, such as visual question answering or image captioning. we do not include experiments on specific tasks but instead leave that investigation for future work. we use a standard retrieval as ranking framework where the model is trained to rank nodes in visualsem given an input, and we use the top-k ranked nodes as the results. these retrieval modules allow for an easy integration of visualsem's entities, glosses, and images into neural pipelines. we frame entity retrieval using sentences (henceforth sentence retrieval, section 4.1) as a sentenceto-gloss ranking problem, and use glosses available in all languages in the model. given an input sentence, we transform the resulting ranking over glosses into a ranking over nodes (i.e., by retrieving the nodes these glosses are linked to from their 1to-1 mappings). we similarly frame retrieval using 8 we use the following hyperparameters with t-sne: perplexity is set to 70, and number of iterations to 5000. ## related work knowledge bases (kbs) and kgs have a long and rich history, and many publicly available kbs exist and have been built for different purposes. foot_5 a seminal example is cyc @xcite , an early effort towards building a general-purpose knowledge base to store common sense facts and rules about how the world works. more recent examples of knowledge bases built with different purposes include wordnet @xcite , dbpedia @xcite , wikidata @xcite , freebase @xcite , yago @xcite , conceptnet @xcite , atomic @xcite , among many others (see @xcite for a detailed list of knowledge bases and algorithms). our main goal is to design a kg to support research in vision & language, which none of the abovementioned kbs are designed to do. ## conclusions and future work we present the visualsem knowledge graph with ∼ 90k nodes, over 1.3m glosses in up to 14 diverse languages, and around 938k curated and cleaned images illustrating nodes' concepts. visualsem bridges a gap in the available resources to train grounded models of language, and is designed to be useful in vision and language research. we release neural entity retrieval models that accept text and image inputs. sentences in any of the ∼ 50 languages supported by multilingual sentence bert can be used to retrieve entities from the kg, and we evaluate retrieval using all 14 languages in visualsem in future work we will use visualsem sentence/image retrieval mechanisms and gloss and image features for data augmentation in nlp tasks, e.g. word sense disambiguation and named entity recognition, and on vision and language tasks, e.g. image captioning and visual question answering. the reason for these tasks is that they all could intuitively benefit from the added knowledge, be it visual, textual, or multi-modal. we will also investigate how to improve the quality of the image and sentence retrieval modules, both central to the kg impact on current neural-based models. finally, we plan to improve and grow the kg, which is under active development. we will particularly gauge whether there is interest from the research community in increasing its coverage to include other parts of, or the entirety of, wikipedia, for example.