Leveraging Partial Dependency Trees to Control Image Captions (2021)

Abstract: Controlling the generation of image captions attracts lots of attention recently. In this paper, we propose a framework leveraging partial syntactic dependency trees as control signals to make image captions include specified words and their syntactic structures. To achieve this purpose, we propose a Syntactic Dependency Structure Aware Model (SDSAM), which explicitly learns to generate the syntactic structures of image captions to include given partial dependency trees. In addition, we come up with a metric to evaluate how many specified words and their syntactic dependencies are included in generated captions. We carry out experiments on two standard datasets: Microsoft COCO and Flickr30k. Empirical results show that image captions generated by our model are effectively controlled in terms of specified words and their syntactic structures. The code is available on GitHub.

## introduction controllable image captioning emerges as a popular research topic in recent years. existing works attempt to enhance models' controllability and captions' diversity by controlling the attributes of image captions such as style @xcite , sentiments @xcite , contents @xcite @xcite and part-of-speech @xcite . however, some important attributes of image captions like words and syntactic structures, are ignored in previous works. for example, for the image in the figure 2 , the work @xcite specifies a set of objects like 'dog, man, frisebee' as a control signal, but there still exist lots of possibilities of composing them into different captions, such as 'a dog and a man play frisebee on grass' and 'a dog playing with a man catches frisebee', since both words and syntactic structures are not determined yet. to address this challenging issue, we propose a framework, which employs partial dependency trees as control signals. as shown in figure 1 , a partial dependency tree, a sub-tree of a syntactic dependency tree, contains words and their syntactic structures, and thus we can utilize it to specify control information about words and their syntactic structures. in addition, we develop a pipeline model called syntactic dependency structure-aware model (sd-sam) which first derives a full syntactic dependency tree and then flatten it into a caption. the motivation behind this pipeline model is that we assume explicitly generating syntactic dependency trees as intermediate representations can better help the model learn how to apply the specified syntactic information to the captions and the intermediate representations can give users an intuitive impression on which part of the captions' syntactic structures is controlled. finally, we propose a syntactic dependencybased evaluation metric which evaluates whether the generated captions have been controlled in terms of syntactic structures. our metric is computed based on the overlap of syntactic dependencies which is different from existing metrics like bleu @xcite , meteor (denkowski and lavie, 2014), rouge @xcite , cider @xcite and spice @xcite which rely on the overlap of ngrams or semantic graphs. empirical results show that image captions generated by our model are effectively controlled in terms of specified words and their syntactic structures. ## framework definition the task presented in this paper is defined as generating a caption sentence (i.e. word sequence) @xmath0, • • • , w |y| given an image i and a partial dependency tree p as input, so that the dependency tree t y of y includes p as far as possible. the syntactic dependency tree of a sentence, as shown in figure 1 , refers to a tree structure to represent syntactic relations between words. t x of a sentence @xmath1, • • • , w |x| is defined as a set of dependencies, {d 1 , d 2 , • • • , d |tx| }, where |t x | denotes the number of dependencies in t x . each dependency d k is expressed in the form of w i e i,j ## syntactic dependency structure aware model the syntactic dependency structure-aware model(sdsam) shown in figure 2 generates image captions in two steps: (1) the syntactic dependency tree generator on the left part derives a full syntactic dependency tree from the image and the partial dependency tree. (2) the caption generator on the right part will flatten the syntactic dependency tree into a caption. ## experiment preparing datasets with partial dependency trees for evaluation, we apply two methods to create partial dependency trees for on microsoft coco @xcite and flickr30k @xcite . the first method extracts partial dependency trees from reference captions. we parsing reference captions to syntactic dependency trees using spacy foot_0 and then randomly sample subsets from each syntactic dependency tree. sampled partial dependency trees are then paired with corresponding reference captions. the dataset created by this procedure is denoted as test gold in section 5. the other method creates partial dependency trees from images in two steps: (1) we first train a syntactic dependency classifier to predict syntactic dependencies for an input image. (2) predicted syntactic dependencies are combined to form a syntactic dependency graph for the input image, from which partial dependency trees are sampled. the dataset created by this procedure is denoted as test pred in section 5. for training, following the first method, we directly sample a partial dependency tree from one of the reference captions for each image and the paired reference caption is used as a training target. evaluation metric the evaluation metrics for image captioning fall into two categories: (1)quality: evaluating the relevance to human annotations with metrics including bleu @xcite and meteor (denkowski and lavie, 2014); rouge @xcite , and cider @xcite and spice @xcite . (2)control-ability: evaluating whether generated image captions are successfully controlled by partial dependency trees. we devise a new metric called dependency based evaluation metric (dbem) for this purpose. assuming that a partial dependency tree @xmath2, • • • , d |p | } is input, dbem calculates how many syntactic dependencies specified in the partial dependency tree are included in the dependency tree t y of generated caption y. the dbem score for the evaluation dataset is given as an average of this score for each input. formally, experiment setting the training of our model is split into two stages including training the syntactic dependency tree generator and training the caption generator. we set the size of hidden states to be 512, the word embedding size to be 512, and the dependency label embedding size to be 300. we train our model using the adam optimizer (kingma and ba, 2015) with a learning rate 5e -4 for the first stage and 1e -4 for the second stage. two models, including our sdsam model and the nic model @xcite (2) results on test pred : we show the evaluation results on test pred in controllability dbem scores on test gold and test pred are shown in when the models are given no control signals, sdsam has better dbem scores than nic. this is possibly because sdsam explicitly learns to generate syntactic dependency trees, and can bet- ## case study in figure 3 , we show an example of the output from our model on test pred . our syntactic dependency classifier first predicts a syntactic dependency graph from the input image. once the syntactic dependency graph is constructed, we sample three partial dependency trees with different node numbers as shown in the figure. finally, our sdsam model infers the captions from the input image and the partial dependency trees. from this example, it is obvious that all words and syntactic structures specified in partial dependency trees also appear in the generated captions. furthermore, the three generated captions are considerably different from each other, demonstrating that giving partial dependency trees as control signals can improve captions' diversity. ## conclusion we presented a framework for controlling image captions in terms of words and syntactic structures by giving partial dependency trees as control signals. we develop a syntactic dependency structure aware model to explicitly learn the syntactic structures in control signals. empirical results show that image captions generated by our model are effectively controlled in terms of specified words and their syntactic structures. furthermore, the results indicate that explicitly learning to generate the syntactic dependency trees of captions enhances the model's controllability.