VISTA : Visual-Textual Knowledge Graph Representation Learning (2023)

Abstract: Knowledge graphs represent human knowledge using triplets composed of entities and relations. While most existing knowledge graph embedding methods only consider the structure of a knowledge graph, a few recently proposed multimodal methods utilize images or text descriptions of entities in a knowledge graph. In this paper, we propose visual-textual knowledge graphs (VTKGs), where not only entities but also triplets can be explained using images, and both entities and relations can accompany text descriptions. By compiling visually expressible commonsense knowledge, we construct new benchmark datasets where triplets themselves are explained by images, and the meanings of entities and relations are described using text. We propose VISTA, a knowledge graph representation learning method for VTKGs, which incorporates the visual and textual representations of entities and relations using entity encoding, relation encoding, and triplet decoding transformers. Experiments show that VISTA outperforms state-of-the-art knowledge graph completion methods in real-world VTKGs.

## introduction knowledge graphs provide diverse human knowledge in a structured form, representing each fact as a triplet consisting of entities and a relation. knowledge graph representation learning methods @xcite aim to convert the entities and relations into a set of representation vectors which can be utilized in predicting missing triplets or in other applications such as commonsense reasoning @xcite and question answering models @xcite . while most existing knowledge graph embedding methods focus solely on the structure of a knowledge graph to learn the representations @xcite , using additional images or text descriptions can result in much better representations. a few recently proposed multimodal knowledge graph completion methods @xcite consider the case where entities can have their images and text descriptions; figure 1 (a ) is a form of knowledge graph existing multimodal knowledge graph completion methods consider. we focus on the fact that some triplets can be more intuitively expressed by their images. for example, in figure 1 (b) , ⟨person, ride, horse⟩ and ⟨person, pull, cart⟩ have images, where the triplets themselves are represented by images, providing visual insights. on the other hand, existing knowledge graphs do not provide images for triplets and may lack visually expressible triplets because they are constructed by extracting information mainly from texts. therefore, we propose forming a knowledge graph by extracting information from images and utilizing those images to represent visual commonsense knowledge. to this end, we compile visual phrases in triplets and their images from diverse computer vision tasks, including visual relationship detection @xcite , human-object interaction detection @xcite , and visual knowledge extraction @xcite . we propose visual-textual knowledge graphs (vtkgs) where the entities and triplets can be represented by images, and the entities and relations have their text descriptions as shown in figure 1 @xcite . we build new vtkg benchmark datasets containing visually expressible commonsense knowledge, the entities' and triplets' images, and detailed descriptions of the entities and relations. to learn representations of entities and relations in vtkgs, we propose the visual-textual (vista) knowledge graph representation learning method that utilizes not only the structure of a visual-textual knowledge graph but also the visual and textual features extracted from images and text descriptions. in particular, when a triplet is provided with an image, vista learns the visual representation of the relation in the given triplet. the resulting visual representation of the relation is also utilized when the relation appears in other triplets, which enhances the entire representation learning process. to the best of our knowledge, vista is the first knowledge graph representation learning method that learns the visual representations for visually expressible relations. we design vista by proposing three transformers: entity encoding, relation encoding, and triplet decoding transformers. the entity and relation encoding transformers represent entities and relations using their visual and textual feature vectors, whereas the triplet decoding transformer predicts a missing entity in a triplet using a masking scheme. experimental results on four real-world datasets demonstrate that vista outperforms 10 different state-of-the-art knowledge graph completion methods. our datasets and codes are available at https://github.com/bdi-lab/vista . ## related work visual commonsense reasoning there have been some attempts to extract visual knowledge from images by using visually verifiable relations @xcite . however, they were not studied in the context of multimodal knowledge graph representation learning, and some datasets are not accessible now. more recently, the visual genome dataset @xcite has been released, where diverse computer vision datasets are merged. however, this dataset contains heterogeneous information which is not in the form of triplets or is hard to be considered as commonsense knowledge. on the other hand, our vtkg datasets provide visual commonsense knowledge in the form of triplets with images and can seamlessly enlarge existing knowledge bases. we believe our work can be utilized in visual commonsense reasoning @xcite and visual question answering (vqa) @xcite . knowledge integration ilievski et al. have attempted to examine the characteristics of information from different sources by manually categorizing the kind of knowledge @xcite . from a multimodal learning point of view, adding different modalities to existing knowledge bases has been considered @xcite , e.g., adding images or texts to entities in a knowledge graph. different from these approaches, our vtkgs are proposed to represent visually expressible knowledge, and our benchmark datasets are created by a fine-level alignment of entities and relations from different sources using wordnet synsets @xcite . details are described in section 3.2. ## visual-textual knowledge graphs we describe how we collect visual commonsense knowledge and create vtkg benchmark datasets. ## vista: learning representations in visual-textual knowledge graphs we propose vista shown in figure 3 . we extract visual and textual features from images and text de-scriptions using pretrained vit @xcite and bert @xcite . by utilizing the extracted features, vista learns representations of entities and relations, which are computed by the entity and relation encoding transformers, respectively. based on the resulting entity and relation representations, vista predicts a missing entity in a triplet using the triplet decoding transformer. ## experiments we evaluate the performance of vista using four real-world datasets by comparing it to 10 different knowledge graph completion methods. we implement vista using pytorch @xcite with the adam optimizer @xcite . we apply dropout @xcite and use the cosine annealing scheduler @xcite . ## conclusion & future work we propose vtkgs, where visually expressible triplets are augmented by images, and both entities and relations have textual descriptions. by appropriately utilizing all this rich information, vista substantially outperforms 10 different state-of-theart knowledge graph completion methods in realworld vtkg datasets. our vtkg datasets and vista model can be utilized in diverse applications and scenarios @xcite @xcite , including those requiring visual commonsense knowledge such as vqa @xcite or scene graph generation @xcite and commonsense reasoning @xcite . we will extend our work to hyper-relational knowledge graphs @xcite or bi-level knowledge graphs @xcite where more information is added to each triplet using qualifiers, or higher-level relationships are considered to enrich information between triplets. by structuring an image or a description as auxiliary information or qualifiers, vista can be easily extended to hyper-relational knowledge graphs with images and texts.