Scene Graph Parsing via A bstract M eaning R epresentation in Pre-trained Language Models (2022)

Abstract: In this work, we propose the application of abstract meaning representation (AMR) based semantic parsing models to parse textual descriptions of a visual scene into scene graphs, which is the first work to the best of our knowledge. Previous works examined scene graph parsing from textual descriptions using dependency parsing and left the AMR parsing approach as future work since sophisticated methods are required to apply AMR. Hence, we use pre-trained AMR parsing models to parse the region descriptions of visual scenes (i.e. images) into AMR graphs and pre-trained language models (PLM), BART and T5, to parse AMR graphs into scene graphs. The experimental results show that our approach explicitly captures high-level semantics from textual descriptions of visual scenes, such as objects, attributes of objects, and relationships between objects. Our textual scene graph parsing approach outperforms the previous state-of-the-art results by 9.3% in the SPICE metric score.

## introduction understanding and representing a scene is straightforward for humans, but an ai system requires various techniques to implement it. one such technique is scene graph proposed by @xcite . scene graph is a graph-structured representation that captures high-level semantics of visual scenes (i.e. images) by explicitly modeling objects along with their attributes and relationships with other objects. scene graph is demonstrated effective in various tasks including semantic image retrieval @xcite , image captioning @xcite , and visual question answering @xcite . approaches for scene graph generation are classified into two categories: 1) scene graph generation based on image as input and 2) scene graph generation based on text (i.e. image caption) as input. various approaches @xcite @xcite are proposed for the former category. on the other hand, only a fewer approaches @xcite @xcite are proposed for the latter. in this paper, we focus on the latter category, which is also called textual scene graph parsing. textual scene graph parsing has the advantage of being able to capture the high-level meaning of the image scene from the text. most of previous works @xcite @xcite for scene graph parsing generated scene graphs using dependency parsing to acquire the dependency relationships for all words in a text, as shown in figure 1 (a). apart from dependency parsing, there is also another approach for parsing semantic graphs from textual descriptions, which is called abstract mean-ing representation (amr) proposed by @xcite . amr abstracts semantic concepts from words, and we therefore consider amr is more suitable for scene graph parsing. however, the use of dependency parsing appeared to be a common theme in the literature rather than amr, hence scene graph parsing with amr has been left as future work in @xcite . to this end, we investigate the use of amr with pre-trained language models (plm), such as bart @xcite and t5 @xcite , for parsing scene graphs from textual descriptions of visual scenes. we first parse sentences to amr graphs using a pre-trained amr parsing model, and then we generate scene graphs from amr graphs using the plm. our contributions are the following: i) to the best of our knowledge, ours is the first work for parsing scene graphs from texts using abstract meaning representation (amr) contrary to the previous works @xcite @xcite . ii) we extend pre-trained language models such as bart and t5 to generate scene graphs from texts and amr graphs. iii) our approach outperforms the previous state-of-the-art result by 9.3% on spice metric for scene graph parsing task on intersection of visual genome and ms coco datasets. ## methodology in this section, we use pre-trained language models (plm), bart @xcite and t5 @xcite , as baselines to parse scene graph (sg) from text directly (text-to-sg). we then describe how to generate scene graphs from amr graphs (amr-to-sg) using plm models. ## conclusion in this work, we investigate the application of abstract meaning representation (amr) for parsing scene graph by using pre-trained language models (plm), bart and t5, with amr parsing model of spring. we conducted two sets of experiments: 1) scene graph parsing using plm models, directly from region descriptions, and 2) scene graph parsing using plm models from amr graphs parsed from region descriptions via amr parsing pretrained models. our results show amr graphs capture high-level abstract semantics of region descriptions. we evaluate our approach using the spice metric score. the results of text-to amr are comparable and of amr-to-text outperform the existing state-of-the-art models by 9.3%. in our future work, we will investigate an adapter-based method @xcite to encode graph structures into plm models to improve the performance of textual scene graph parsing. furthermore, we will examine our approach based on the lately published, pre-trained amr parsing model, amrbart foot_2 @xcite . as our scene graph parser performance improves further, we expect to be able to use it to automatically generate either an image scene graph or video scene graph datasets with less biased and more diverse labels.