Visual Semantic Parsing: From Images to A bstract M eaning R epresentation (2022)

Abstract: The success of scene graphs for visual scene understanding has brought attention to the benefits of abstracting a visual input (e.g., image) into a structured representation, where entities (people and objects) are nodes connected by edges specifying their relations. Building these representations, however, requires expensive manual annotation in the form of images paired with their scene graphs or frames. These formalisms remain limited in the nature of entities and relations they can capture. In this paper, we propose to leverage a widely-used meaning representation in the field of natural language processing, the Abstract Meaning Representation (AMR), to address these shortcomings. Compared to scene graphs, which largely emphasize spatial relationships, our visual AMR graphs are more linguistically informed, with a focus on higher-level semantic concepts extrapolated from visual input. Moreover, they allow us to generate meta-AMR graphs to unify information contained in multiple image descriptions under one representation. Through extensive experimentation and analysis, we demonstrate that we can re-purpose an existing text-to-AMR parser to parse images into AMRs. Our findings point to important future research directions for improved scene understanding.

## introduction the ability to understand and describe a scene is fundamental for the development of truly intelligent systems, including autonomous vehicles, robots navigating an environment, or even simpler applications such as language-based image retrieval. much work in computer vision has focused on two key aspects of scene understanding, namely, recognizing entities, including object detection @xcite @xcite *work done during an internship at samsung ai centre -toronto †work done while at samsung ai centre -toronto et al., 2020; @xcite and activity recognition @xcite @xcite , as well as understanding how entities are related to each other, e.g., human-object interaction @xcite and relation detection @xcite @xcite . a natural way of representing scene entities and their relations is in graph form, so it is perhaps unsurprising that a lot of work has focused on graphbased scene representations and especially on scene graphs @xcite . scene graphs encode the salient regions in an image (mainly, objects) as nodes, and the relations among these (mostly spatial in nature) as edges, both labelled via natural language tags; see fig. 1( @xcite for an example scene graph. along the same lines, @xcite propose to represent a scene as a semantic role labelled frame, drawn from framenet @xcite -a linguistically-motivated approach that draws on semantic role labelling literature. scene graphs and situation frames can capture important aspects of an image, yet they are limited in important ways. they both require expensive manual annotation in the form of images paired with their corresponding scene graphs or frames. scene graphs in particular also suffer from being limited in the nature of entities and relations that they capture (see section 2 for a detailed analysis). ideally, we would like to capture event-level semantics (same as in situation recognition) but as a structured graph that captures a diverse set of relations and goes beyond low-level visual semantics. inspired by the linguistically-motivated image understanding research, we propose to represent images using a well-known graph formalism for language understanding, i.e., abstract meaning representations (amrs @xcite . similarly to ( visual when, and how?" @xcite , but in a more structured way via transforming an image into a graph representation. amrs not only encode the main events, their participants and arguments, as well as their relations (as in semantic role labelling/situation recognition), but also relations among various other participants and arguments; see fig. 1(a ). importantly, amr is a broadly-adopted and dynamically evolving formalism (e.g., @xcite @xcite , and amr parsing is an active and successful area of research (e.g., @xcite @xcite . finally, given the high quality of existing amr parsers (for language), we do not need manual amr annotations for images, and can rely on existing image-caption datasets to create high quality silver data for image-to-amr parsing. in summary, we make the following contributions: our analyses suggest that amrs encode aspects of an image content that are not captured by the commonly-used scene graphs. our initial results on re-purposing a text-to-amr parser for image-to-amr parsing, as well as on creating image-level meta-amrs, point to exciting future research directions for improved scene understanding. ## motivation: amrs vs. scene graphs scene graphs (sgs) are a widely-adopted graph formalism for representing the semantic content of an image. scene graphs have been shown useful for various downstream tasks, such as image captioning @xcite @xcite , visual question answering @xcite @xcite , and image retrieval @xcite @xcite . however, learning to automatically generate sgs requires expensive manual annotations (object bounding boxes and their relations). sgs were also shown to be highly biased in the entity and relation types that they capture. for example, an analysis by @xcite reveals that clothing (e.g., dress) and object/body parts (e.g., eyes, wheel) make up over one-third of entity instances in the sgs corresponding to the visual genome images @xcite , and that more than 90% of all relation instances belong to the two categories of geometric (e.g., behind) and possessive (e.g., have). one advantage of amr graphs is that we can draw on supervision through captions associated with images. nonetheless, the question remains as to what types of entities and relations are encoded by amr graphs, and how these differ from sgs. to answer this question, we follow an approach similar to @xcite , and categorize entities and relations in sg and amr graphs corresponding to a sample of 50k images. we use the same categories as zellers et al., but add a few new ones to capture relation types specific to amrs, namely, attribute (small), quantifier (few), event (soccer), and amr specific (date-entity). details of our categorization process are provided in appendix a. figure 2 shows the distribution of instances for each entity and relation category, compared across sg and amr graphs. amrs tend to encode a more diverse set of relations, and in particular capture more of the abstract semantic relations that are missing from sgs. this is expected because our caption-generated amrs by design capture the essential meaning of the image descriptions and, as such, encode how people perceive and describe scenes. in contrast, sgs are designed to capture the content of an image, including regions representing objects and (mainly spatial/geometric) visually-observable relations; see fig. 1 for sg and amr graphs corresponding to an image. in the context of entities, and a major departure from sgs, (object/body) parts are less frequently encoded in amrs, pointing to the well-known whole-object bias in how people perceive and describe scenes @xcite . in contrast, location is more frequent in amrs. the focus of amrs on abstract content suggests that they have the potential for improving downstream tasks, especially when the task requires an understanding of the higher level semantics of an image. interestingly, a recent study showed that using amrs as an intermediate representation for textual sg parsing helps improve the quality of the parsed sgs @xcite , even though amrs and sgs encode qualitatively different information. since amrs tend to capture higher level semantics, we propose to use them as the final image representation. the question remains as to how difficult it is to directly learn such representations from images. the rest of the paper focuses on answering this question. ## discussion and outlook in this paper, we proposed to use a well-known linguistic semantic formalism, i.e., abstract meaning representation (amr) for scene understanding. we showed through extensive analysis the advantages of amr vs. the commonly-used visual scene graphs, and proposed to re-purpose existing text-to-amr parsers for image-to-amr parsing. additionally we proposed a graph transformation algorithm that merges several caption-level amr graphs into a more descriptive meta-amr graph. our quantitative (intrinsic and extrinsic) and qualitative evaluations demonstrate the usefulness of (meta-)amrs as a scene representation formalism. our findings point to a few exciting future research directions. our image-to-amr parsers can be improved by incorporating richer visual features, a better understanding of the entity and relation categories that are particularly hard to predict for our current models, as well as drawing on methods used for scene graph generation (e.g., @xcite . our meta-amr generation algorithm can be further tuned to capture visuallysalient information (e.g., quantifiers are too hard to learn from images, and perhaps can be dropped from a visual amr formalism). our qualitative samples of captions generated from meta-amrs show their potential for generating descriptive and/or controlled captions. controllable image captioning has received a great deal of attention lately (e.g., @xcite @xcite . it focuses on the use of subjective control, including personalization and stylefocused caption generation, as well as objective control on content (controlling what the caption is about, e.g., focused on a set of regions), or on the structure of the output sentence (e.g., controlling sentence length). we believe that by using amrs as intermediate scene representations, we can bring together the work on these various types of control, as well as draw on the literature on controllable natural language generation @xcite for advancing research on rich caption generation. (c) a large long passenger train going across a wooden beach plate, traveling and passing by water. (d) a woman sitting at a table eating a sandwich and holding a hot dog in a building smiling while eating. (e) a white area filled with lots of different kinds of donuts with various toppings sitting on them. (f) a group of people sitting around at a dining table with water posing for a picture. (g) a person in a red jacket cross country skiing down a snow covered ski slope with a couple of people riding skis and walking on the side of the snowy mountain. (h) a person in black shirt sitting at a table in a building with a plate of food with and smiling while having meal. interaction detection with hoi transformer. in proceedings of the ieee/cvf conference on computer vision and pattern recognition.