Transforming Visual Scene Graphs to Image Captions (2023)

Abstract: We propose to TransForm Scene Graphs into more descriptive Captions (TFSGC). In TFSGC, we apply multi-head attention (MHA) to design the Graph Neural Network (GNN) for embedding scene graphs. After embedding, different graph embeddings contain diverse specific knowledge for generating the words with different part-of-speech, e.g., object/attribute embedding is good for generating nouns/adjectives. Motivated by this, we design a Mixture-of-Expert (MOE)-based decoder, where each expert is built on MHA, for discriminating the graph embeddings to generate different kinds of words. Since both the encoder and decoder are built based on the MHA, as a result, we construct a simple and homogeneous encoder-decoder unlike the previous heterogeneous ones which usually apply Fully-Connected-based GNN and LSTM-based decoder. The homogeneous architecture enables us to unify the training configuration of the whole model instead of specifying different training strategies for diverse sub-networks as in the heterogeneous pipeline, which releases the training difficulty. Extensive experiments on the MS-COCO captioning benchmark validate the effectiveness of our TFSGC. The code is in: https://anonymous.4open.science/r/ACL23_TFSGC .

## introduction image captioning, which aims to generate one sentence for describing multi-aspects of an image, has made huge progress since the proposal of the encoder-decoder framework @xcite . such a framework contains one visual encoder to extract a series of visual features from the image and one language decoder to generate captions from the extracted visual features. since the visual encoder is usually well pre-trained by image classification and object detection, the extracted features contain abundant knowledge of the object categories, which enables the captioning model to generate object-abundant captions. however, object category is not the only visual pattern that matters for high-quality captions @xcite , object attributes and relations also play significant roles in generating descriptive captions, i.e., the caption containing multi-aspects of an image. motivated by this, researchers propose to incorporate additional semantic knowledge, e.g., object categories, attributes, and relations, into the captioning models by using the scene graph as the mediator @xcite . scene graphs assign each object node with certain attribute nodes and some pairwise objects with certain relation nodes. these nodes are represented by the corresponding semantic tags, e.g., as shown in fig. 1 , the object "dog" is assigned with the attribute "black" and the pairwise objects "dog" and "fish" have the relation "bite" in between. to exploit the scene graph, graph neural network (gnn) @xcite is deployed to embed the graphs and the output embeddings are input to the decoder for captioning. the top part of fig. 1 shows the pipeline of the previous popular gnn-based captioning model @xcite , which implements gnn as a few fully-connected (fc) and non-linear activation layers. to update the node embedding, this gnn maps the concatenated neighbour embeddings into the new one @xcite . then the updated graph embeddings are input into the language decoder that contains a few lstm layers and an attention module. the lstm layers are used to generate the context vector based on the partially generated captions. this context vector works as the query in the attention module for determining which graph embeddings should be used to generate the next word. compared with the models without gnn, this gnn-lstm pipeline usually gets better performance. to alleviate the above-mentioned flaws, we propose a novel homogeneous captioning model to transform scene graphs (tfsgc) into captions. our tfsgc is built based on the transformer @xcite since it is more powerful than lstm in image captioning @xcite @xcite . tfsgc is homogeneous since we use multi-head attention (mha) to design both the graph encoder to embed the scene graphs and the language decoder to generate the caption. our design principle is quite simple where we do not need to revise the self-attention operation but only need to reformulate the input data structure. specifically, to design gnn by mha, we first linearize the scene graph into a token sequence and introduce a binary mask to index which two nodes are connected in the graph. then we use the masked mha operation to deal with this linearized token sequence for graph embedding. in this process, each token embedding is added by a learnable type embedding to index the token type (e.g., object/attribute/relation) and we will show that such type embedding can help distinguish the edge type during the attention calculation. after graph operation, we get a series of object/attribute/relation embeddings, which will be used in the decoder for captioning. to make the decoder discriminate different embeddings for generating different words, we learn from moe networks @xcite @xcite to revise the original transformer decoder with two strategies. first, as fig. 1 bottom (d) shows, we use three encoder-decoder attention layers, which are built on mha, as three experts to address object/attribute/relation embeddings, respectively. second, we incorporate an attention-based soft routing network to discriminate which kinds of embeddings should be more responsible for generating the next word. both the moe-decoder and the type embedding in the encoder help distinguish node embeddings for better captions. we carry exhaustive ablation studies and comparisons to validate the effectiveness of tfsgc and it achieves 132.3/138.6/139.5 cider scores when using but-d/patch/vinvl features. ## related work image captioning. for a long time, the attentionbased cnn-lstm pipeline @xcite is the most popular backbone for captioning and various techniques have been added into it for better performance, including building stronger visual encoders @xcite , designing more advanced attention mechanisms @xcite , incorporating semantic knowledge @xcite @xcite , and exploiting language structure @xcite . recently, transformer @xcite has gradually substituted lstm as the mainstream language decoder in image captioning @xcite since it achieves better performances than the lstm-based models. based on this new backbone, researchers develop more advanced strategies for further improving the effectiveness, including designing more sophisticated attention mechanisms @xcite , introducing additional memory blocks @xcite , distilling knowledge from the large-scale pre-training models @xcite @xcite , and exploiting transformer-based visual encoders @xcite , modularized design for large-scale multi-modal pretraining @xcite @xcite . since the recently proposed sota models use transformer as the backbone, we also built tfsgc based on transformer for fair comparison. graph neural network (gnn). scene graph abstracts the major visual patterns in a visual scene as a graph. it is usually used as the mediator to narrow the gap between the vision and the language domains. to incorporate scene graphs into deep networks, @xcite is used to embed the discrete node labels into dense embeddings. however, most of the previous gnns are mlp-based @xcite @xcite @xcite , which may limit the effectiveness of embedding scene graphs in a transformer architecture. in our research, we design an mha-based gnn to remedy this limitation. moreover, noisy scene graphs may damage the performances @xcite , so we use better scene graph parsers to minimize the impact of noise on our model. ## revisiting of transformer we first revisit the transformer-based captioning model and then introduce how to revise it to get our tfsgc in the next section. transformerbased model contains a visual encoder to calculate the contexts of the extracted visual features and the output embeddings will be input into a language decoder for captioning. for both the encoder and the decoder, the most elemental building block is the multi-head attention (mha). given the query, key, and value matrices: besides mha, another important module in transformer is the feed-forward network (ffn): given mha and ffn, we can use them to build a transformer-based captioning model. for the encoder, it stacks 6 identical blocks where each one contains an mha and an ffn. given the output of the former block as the input x, the next block calculates its output as: note that the variables x, y , z used here are "local variables" for conveniently introducing the work flow of transformer architecture, whose values will be set to the specific values when introducing the concrete captioning model. in "self-att", q, k, v are set to the same value and this operation is named as self-attention @xcite . after stacking 6 blocks defined in eq. ( 3 ), a visual encoder is built. for the first block, its input is the extracted visual feature set of the given image. the output of the last block will be input into the language decoder. for the decoder, it also stacks 6 identical blocks where each one contains two mhas and an ffn. given the output of the former decoder block x d and the output of the visual encoder x e , the next decoder block calculates its output: note that in "ed-att", q is set to the output of the former decoder block while k, v are set to the output of the visual encoder, and such operation is called encoder-decoder attention @xcite . after stacking 6 blocks defined in eq. ( 4 ), a language decoder is built. for the first block in the decoder, x d in eq. ( 4 ) is set to the word embedding set of the partially generated captions @xmath0, ..., s t } at the t-th time step. for all the decoder blocks, the input x e is set to the same value, which is the output of the visual encoder. the output of the last decoder block @xmath1, ..., z t } is used to calculate the word distribution of the next word: ## transforming scene graphs in this section, we introduce how to revise the transformer to get our tfsgc. we will first show how to get an mha-based gnn and then introduce how to design an moe-based decoder. ## conclusion we proposed to transform the scene graph into captions (tfsgc) by a simple and homogeneous network. specifically, we use mha to design the gnn by linearizing the scene graph and remedying the lost topological knowledge with a binary mask matrix. furthermore, we add learnable type embedding and design an moe-based decoder to distinguish node embeddings for more descriptive captions. at last, we compared tfsgc with various sota models and demonstrated that our model can achieve comparable performances to some strong benchmarks.