Learning to Represent Image and Text with Denotation Graph (2020)

Abstract: Learning to fuse vision and language information and representing them is an important research problem with many applications. Recent progresses have leveraged the ideas of pre-training (from language modeling) and attention layers in Transformers to learn representation from datasets containing images aligned with linguistic expressions that describe the images. In this paper, we propose learning representations from a set of implied, visually grounded expressions between image and text, automatically mined from those datasets. In particular, we use denotation graphs to represent how specific concepts (such as sentences describing images) can be linked to abstract and generic concepts (such as short phrases) that are also visually grounded. This type of generic-to-specific relations can be discovered using linguistic analysis tools. We propose methods to incorporate such relations into learning representation. We show that state-of-the-art multimodal learning models can be further improved by leveraging automatically harvested structural relations. The representations lead to stronger empirical results on downstream tasks of cross-modal image retrieval, referring expression, and compositional attribute-object recognition. Both our codes and the extracted denotation graphs on the Flickr30K and the COCO datasets are publically available on https://sha-lab.github.io/DG .

## introduction there has been an abundant amount of aligned visual and language data such as text passages * work done while at google † authors contributed equally ‡ on leave from usc (feisha@usc.edu) describing images, narrated videos, subtitles in movies, etc. thus, learning how to represent visual and language information when they are semantically related has been a very actively studied topic. there are many vision + language applications: image retrieval with descriptive sentences or captions @xcite @xcite , image captioning @xcite , visual question answering @xcite , visual navigation with language instructions @xcite , visual objects localization via short text phrases @xcite , and others. a recurring theme is to learn the representation of these two streams of information so that they correspond to each other, highlighting the notion that many language expressions are visually grounded. a standard approach is to embed the visual and the language information as points in a (joint) visualsemantic embedding space @xcite @xcite . one can then infer whether the visual information is aligned with the text information by checking how these points are distributed. how do we embed visual and text information? earlier approaches focus on embedding each stream of information independently, using models that are tailored to each modality. for example, for image, the embedding could be the features at the last fully-connected layer from a deep neural network trained for classifying the dominant objects in the image. for text, the embedding could be the last hidden outputs from a recurrent neural network. recent approaches, however, have introduced several innovations @xcite @xcite . the first is to contextualize the embeddings of one modality using information from the other one. this is achieved by using co-attention or cross-attention (in addition to selfattention) in transformer layers. the second is to leverage the power of pre-training @xcite : given a large number of parallel corpora of images and their descriptions, it is beneficial to identify pre-trained embeddings on these data such that they are useful for downstream vision + language tasks. despite such progress, there is a missed opportunity of learning stronger representations from those parallel corpora. as a motivating example, suppose we have two paired examples: one is an image x 1 corresponding to the text y 1 of two dogs sat in front of porch and the other is an image x 2 corresponding to the text y 2 of two dogs running on the grass. existing approaches treat the two pairs independently and compute the embeddings for each pair without acknowledging that both texts share the common phrase y 1 ∩ y 2 = two dogs and the images have the same visual categories of two dogs. we hypothesize that learning the correspondence between the common phrase y 1 ∩ y 2 and the set of images {x 1 , x 2 }, though not explicitly annotated in the training data, is beneficial. enforcing the alignment due to this additionally constructed pair introduces a form of structural constraint: the embeddings of x 1 and x 2 have to convey similar visual information that is congruent to the similar text information in the embeddings of y 1 and y 2 . in this paper, we validate this hypothesis and show that extracting additional and implied correspondences between the texts and the visual information, then using them for learning leads to better representation, which results in a stronger performance in downstream tasks. the additional alignment information forms a graph where the edges indicate how visually grounded concepts can be instantiated at both abstract levels (such as two dogs) and specific levels (such as two dogs sat in front of the porch). these edges and the nodes that represent the concepts at different abstraction levels form a graph, known as denotation graph, previously studied in the nlp community @xcite @xcite for grounding language expressions visually. our contributions are to propose creating visuallygrounded denotation graphs to facilitate representation learning. concretely, we apply the technique originally developed for the flickr30k dataset @xcite also to coco dataset @xcite to obtain denotation graphs that are grounded in each domain respectively ( § 3). we then show how the denotation graphs can be used to augment training samples for aligning text and image ( § 4). finally, we show empirically that the representation learned with denotation graphs leads to stronger performance in downstream tasks ( § 5). ## denotation graph (dg) visually grounded text expressions denote the images (or videos) they describe. when examined together, these expressions reveal structural relations that do not exhibit when each expression is studied in isolation. in particular, through linguistic analysis, these expressions can be grouped and partially ordered and thus form a relation graph, representing how (visually grounded) concepts are shared among different expressions and how different concepts are related. this insight was explored by @xcite and the resulting graph is referred to as a denotation graph, schematically shown in the top part of fig. 1 . in this work, we focus on constructing denotation graphs from the flickr30k and the coco datasets, where the text expressions are sentences describing images. formally, a denotation graph g is a polytree where a node v i in the graph corresponds to a pair of a linguistic expression y i and a set of images a directed edge e ij from a node v i to its child v j represents a subsumption relation between y i and y j . semantically, y i is more abstract (generic) than y j , and the tokens in y i can be a subset of y j 's. for example, two dogs describes all the images which two dogs are running describes, though less specifically. note that the subsumption relation is defined on the semantics of these expressions. thus, the tokens do not have to be exactly matched on their surface forms. for instance, in front of person or in front of crowd are also generic concepts that more formally, the set of images that correspond to v i is the union of all the images corresponding we also use pa(v j ) to denote the set of v j 's parents. denotation graphs (dg) can be seen as a hierarchical organization of semantic knowledge among concepts and their visual groundings. in this sense, they generalize the tree-structured object hierarchies that have been often used in computer vision. the nodes in the dg are composite phrases that are semantically richer than object names and the relationship among them is also richer. constructing dg we used the publicly available tool foot_0 , following young et al. @xcite . for details, please refer to the appendix and the reference therein. once the graph is constructed, we attach the images to the proper nodes by setunion images of each node's children, starting from the sentence-level node. ## learning with denotation graphs the denotation graphs, as described in the previous section, provide rich structures for learning representations of text and image. in what follows, we describe three learning objectives, starting from the most obvious one that matches images and their descriptions ( § 4.1), followed by learning to discriminate between general and specialized concepts ( § 4.2) and learning to predict concept relatedness ( § 4.3). we perform ablation studies of those objectives in § 5.4. ## experiments we examine the effectiveness of using denotation graphs to learn image and text representations. we first describe the experimental setup and key implementation details ( § 5.1). we then describe key image-text matching results in § 5.2, followed by studies about the transfer capability of our learned representation ( § 5.3). next, we present ablation studies over different components of our model ( § 5.4). finally, we validate how well abstract concepts can be used to retrieve images, using our model ( § 5.5). ## conclusion image and text aligned data is rich in semantic correspondence. besides treating text annotations as "categorical" labels, in this paper, we show that we can make full use of those labels. concretely, denotation graphs (dgs) encode structural relations that can be automatically extracted from those texts with linguistic analysis tools. we proposed several ways to incorporate dgs into learning representation and validated the proposed approach on several tasks. we plan to investigate other automatic tools in curating more accurate denotation graphs with a complex composition of fine-grained concepts for future directions. and 7 maximum levels, respectively. the training hyper-parameters remain the same as vilbert + dg-flickr30k with 3 maximum layers. the aim is to check how much gain we could get from the additional annotations. we report the results in once the graph is constructed, we attach the images to the proper nodes by set-union images of each node's children, starting from the sentence-level node.