Are Scene Graphs Good Enough to Improve Image Captioning? (2020)

Abstract: Many top-performing image captioning models rely solely on object features computed with an object detection model to generate image descriptions. However, recent studies propose to directly use scene graphs to introduce information about object relations into captioning, hoping to better describe interactions between objects. In this work, we thoroughly investigate the use of scene graphs in image captioning. We empirically study whether using additional scene graph encoders can lead to better image descriptions and propose a conditional graph attention network (C-GAT), where the image captioning decoder state is used to condition the graph updates. Finally, we determine to what extent noise in the predicted scene graphs influence caption quality. Overall, we find no significant difference between models that use scene graph features and models that only use object detection features across different captioning metrics, which suggests that existing scene graph generation models are still too noisy to be useful in image captioning. Moreover, although the quality of predicted scene graphs is very low in general, when using high quality scene graphs we obtain gains of up to 3.3 CIDEr compared to a strong Bottom-Up Top-Down baseline.

## introduction scene understanding is a complex and intricate activity which humans perform effortlessly but that computational models still struggle with. an important backbone of scene understanding is being able to detect objects and relations between objects in an image, and scene graphs @xcite are a closely related data structure that explicitly annotates an image with its objects and relations in context. scene graphs can be used to improve important visual tasks that require scene understanding, e.g. image indexing and search @xcite or scene construction and generation @xcite , and there is evidence that they can also be used to improve image captioning @xcite . however, the de facto standard in top-performing image captioning models to date use strong object features only, e.g. obtained with a pretrained faster r-cnn @xcite , and no explicit relation information @xcite @xcite . one possible explanation to this observation is that by using detected objects we already capture the more important information that characterises a scene, and that relation information is already implicitly learned in such models. another explanation is that relations are simply not as important as we hypothesise and that we gain no valuable extra information by adding them. in this work, we investigate these empirical observations in more detail and strive to answer the following research questions: (i) can we improve image captioning by explicitly supervising a model with information about object relations? (ii) how does the content of the captions improve when utilising scene graphs? (iii) how does scene graph quality impact the quality of the captions? the most recent best-performing image captioning models make use of the transformer architecture @xcite @xcite . however, in this paper we build upon the influential bottom-up top-down architecture @xcite which uses lstms, and since we want to measure to what extent scene graphs are helpful or not, we remove any "extras" to make model comparison easier, e.g. reinforcement learning step after cross-entropy training, ensembling at inference time, etc. scene graph generation (sgg) is the task where given an image a model predicts a graph with its objects and their relations. we use a pretrained sgg model @xcite to obtain and inject explicit relation information into image captioning, and investigate different image captioning model architectures that incorporate object and relation features, similarly to @xcite ; @xcite . we propose an extension to graph attention networks @xcite which we call conditional graph attention (c-gat), where we condition the updates of the scene graph features on the current image captioning decoder state. finally, we conduct an in-depth analysis of the captions produced by different models and determine if scene graphs actually improve the content of the captions. our approach is illustrated in figure 1 . our main contributions are: ## conditional graph attention (c-gat) standard graph neural architectures encode information about neighbour nodes n (v) into representations of node @xmath0. therefore, these gnns are contextual because they encode graph-internal context. we propose the conditional graph attention (c-gat) architecture, a novel extension for graph attention networks @xcite . 3 our goal is to make these networks conditional in addition to contextual. by conditional we mean that a c-gat layer is conditioned on external context, e.g. a vector representing knowledge that is not part of the original input graph. our motivation is that when using graph-based inputs such as a scene graph, a c-gat layer allows us to condition the message propagation between connected nodes in the graph on the current state of the model, e.g. on the decoder state in the figure 2: c-gat layer where we illustrate the update of vertex man by combining the features of all incoming relations (and objects) through an attention mechanism. the attention scores are conditioned on the external query vector q. in figure 2 , a c-gat layer is applied to an input scene graph g and conditioned on a query vector q, i.e. the decoder state. we illustrate the update of vertex v @xmath1. the self-relation is assumed to always be present and for readability is not shown. neighbour nodes' features are combined with an mlp attention mechanism @xcite and scores are computed using query q. as described in section 2.3, neighbours of an object vertex v @xmath2. to include neighbour object features as well as relation features, we collect features for all v @xmath3, defined as nodes accessible by all relation vertices ## model setup in this section, we first introduce image captioning models that do not explicitly use relation features (section 4.1) and contrast them with those that use explicit relation features (section 4.2). ## experimental setup and results we compare our models with the following external baselines with no access to scene graphs: (1) the adaptive attention model add-att which determines at each decoder time step how much of the visual features should be used @xcite ; we also compare with the following baselines that use scene graphs: (1) the "know more, say less" model kmsl extracts features for objects and relations based on the scene graph, which are passed through two attention heads and finally combined using a flat attention head @xcite ; and (2) the cascade model @xcite which is similar to our hierarchical attention model with a gat layer, but that instead uses a relational graph convolutional network @xcite . we do not discuss model variants/results that are trained with an additional reinforcement learning step @xcite and only compare single model results, since training and performing inference with such models is very costly and orthogonal to our research questions. our proposed models are: flat attention (fa), hierarchical attention with scene graph first (ha-sg) following equation 5 , hierarchical attention with objects detected first (ha-im) following equation 6, ha-sg with graph attention network (ha-sg+gat), and ha-sg with conditional graph attention (ha-sg+c-gat). we choose the last two variants to extend ha-sg following the setup used by @xcite . we evaluate captions generated by different models by investigating their spice scores @xcite , i.e. an f1 based semantic captioning evaluation metric computed over scene graphs. it uses the semantic structure of the scene graph to b4 c r s add-att * † 33.2 108.5 --nbt † 34.7 107.2 -20.1 butd † 36.2 113.5 56.4 20.3 butd 34.8 109.2 55.7 20.0 cascade † 34.1 108.6 55.9 20.3 kmsl † ## conclusions and future work in this work, we investigate the impact scene graphs have on image captioning. we introduced conditional graph attention (c-gat) networks and applied it to image captioning, and report promising results ( in an in-depth analysis, we found that the predicted scene graphs contain a large amount of noise which harms the captioning process. when this noise is reduced, large gains can be achieved across all image captioning metrics, e.g. 3.3 cider points in the high vg-coco split (