Toward General Scene Graph: Integration of Visual Semantic Knowledge with Entity Synset Alignment (2020)

Abstract: Scene graph is a graph representation that explicitly represents high-level semantic knowledge of an image such as objects, attributes of objects and relationships between objects. Various tasks have been proposed for the scene graph, but the problem is that they have a limited vocabulary and biased information due to their own hypothesis. Therefore, results of each task are not generalizable and difficult to be applied to other down-stream tasks. In this paper, we propose Entity Synset Alignment(ESA), which is a method to create a general scene graph by aligning various semantic knowledge efficiently to solve this bias problem. The ESA uses a large-scale lexical database, WordNet and Intersection of Union (IoU) to align the object labels in multiple scene graphs/semantic knowledge. In experiment, the integrated scene graph is applied to the image-caption retrieval task as a down-stream task. We confirm that integrating multiple scene graphs helps to get better representations of images.

## introduction beyond detecting and recognizing individual objects, research for understanding visual scenes is moving toward extracting semantic knowledge to create scene graph from natural images. starting with @xcite , various studies have been proposed to generate this semantic knowledge from images @xcite @xcite . however, each study extracts only highly biased information from an image due to the limited vocabulary depending on their own hypothesis and the statistical bias of the dataset. for example, in @xcite , the author conducted a study on extracting information of both object and attribute for each entity using 1,600 object and 400 attribute class labels. in addition, @xcite generate a relationship between objects in a form of triplet (head entitypredicate -tail entity) in an image by using 150 object and 50 predicate class labels. in @xcite , the author constructed a visually-relevant relationships(vrr-vg) based on @xcite to mine more valuable relationships with 1600 objects and 117 predicate class labels. as such, each task defines and uses its own vocabulary, but the problem is that the vocabulary is limited. as shown in figure 1 ,if some of objects in an image do not belong to the dataset-specific vocabulary, objects as well as relations are omitted frequently even though they are in an image. in addition, there are cases where the same object is defined with different vocabulary in a common image (e.g. man, person). in this paper, we propose entity synset alignment (esa) to perform scene graph integration. with a large-scale lexical database wordnet and iou, the esa aligns the entity labels in scene graphs generated from each dataset. the contributions of the method proposed in this paper are as follows: 1) scene graphs can be generated from raw image inputs, 2) integrating multiple scene graphs inferred from each dataset into one via esa, 3) the qualitative results show that an integrated scene graph can extract richer semantic information in an image, 4) quantitative results show the significance of integrated scene graph by applying integrated scene graph to image-caption retrieval task. ## method as shown in figure 2 , we employ bottom-up attention @xcite model to generate only nodes containing information of both object and attribute, and comptransr model to generate scene graphs from raw images. entity synset alignment(esa) integrates scene graphs generated from each dataset. we introduce a simple model, comptransr, for scene graph generation in section 3.1 and a scene graph integration technique, entity synset alignment(esa) in section 3.2. ## conclusion in this paper, we present a simple and efficient method to integrate multiple visual semantic knowledge into general scene graph. with a large-scale lexical database wordnet and iou, the esa aligns the entity labels in scene graphs generated from each dataset. the integrated scene graph has richer information and is less biased. to evaluate our proposal, we conduct the image-caption retrieval task as a down-stream task and show better performance than each scene graph. for future work, we plan to integrate more diverse visual semantic knowledge such as human-object interaction @xcite . image captioning and visual question answering. in cvpr. fartash faghri, david j fleet, jamie ryan kiros, and sanja fidler. 2017. vse++: improving visualsemantic embeddings with hard negatives. arxiv preprint arxiv:1707.05612. georgia gkioxari, ross girshick, piotr dollár, and kaiming he. 2018. detecting and recognizing human-object interactions. in proceedings of the ieee conference on computer vision and pattern recognition, pages 8359-8367. thomas n kipf and max welling. 2016. semisupervised classification with graph convolutional networks. arxiv preprint arxiv:1609.02907. ryan kiros, ruslan salakhutdinov, and richard s zemel. 2014. unifying visual-semantic embeddings with multimodal neural language models. arxiv preprint arxiv:1411.2539.