Understanding Guided Image Captioning Performance across Domains (2021)

Abstract: Image captioning models generally lack the capability to take into account user interest, and usually default to global descriptions that try to balance readability, informativeness, and information overload. We present a Transformer-based model with the ability to produce captions focused on specific objects, concepts or actions in an image by providing them as guiding text to the model. Further, we evaluate the quality of these guided captions when trained on Conceptual Captions which contain 3.3M image-level captions compared to Visual Genome which contain 3.6M object-level captions. Counter-intuitively, we find that guided captions produced by the model trained on Conceptual Captions generalize better on out-of-domain data. Our human-evaluation results indicate that attempting in-the-wild guided image captioning requires access to large, unrestricted-domain training datasets, and that increased style diversity (even without increasing the number of unique tokens) is a key factor for improved performance.

## introduction describing the content of an image using natural language is generically referred to as image captioning, but there are a variety of ways in which this can be achieved: by focusing on the most salient aspects of an image, as in mscoco @xcite or conceptual captions @xcite ; on most of the groundable concepts in an image, as in image paragraphs @xcite or localized narratives @xcite ; or on a predefined set of objects, as in dense captioning @xcite . these various approaches acknowledge that a typical realworld image may contain a varying number of objects/concepts/actions that may be of interest to the caption consumer, and therefore the optimal description depends on the degree to which the * work done as a part of the google ai residency. figure 1 : an illustration comparing the difference between an image captioning model and a guided image captioning model. the guided image captioning model generates captions that focus on a specific object, concept or action of the image, provided to the model as free-form guiding text. however, the vast majority of captioning solutions lack the ability to take into account user interest, and usually default to global descriptions. such a description might say, for example "people attending a birthday party", which may not satisfy a user interested in a description of "the cake". in this paper, we capture user interest via the guiding text, a free-form text input that is assumed to be related to some concept(s) in the image; and we consider the guided image captioning task, where the guiding text is provided to the model as an additional input to control the concepts that an image caption should focus on 1 . this could, for instance, enable accessibility tools for visuallyimpaired users, who can select a guiding text produced from an upstream object/label detector to receive a guided description of their surroundings. note that guiding texts are not limited to a set of boxable objects, but include concepts (e.g. "vaca-tion", "love") and actions (e.g. "swimming"). a key question we ask in this work is what kind of training data can lead to models that work better in a real-world setting. ideally, the training data should contain multiple captions for a given image, each associated with a different guiding text. the visual genome dataset @xcite provides exactly that, with a total of 3.6m objectlevel captions created through human annotation. in contrast, the conceptual captions @xcite contains only image-level captions created through an automatic pipeline obtaining images and captions from the internet. we perform human evaluations that measure caption informativeness, correctness and fluency to test our approach in a real-world setting. interestingly, while the conceptual captions dataset contains a similar number of captions (3.3m), and has a lower number of unique tokens than visual genome, models trained on this dataset generalize better on out-of-domain data in human evaluations. the key contributions of this paper are summarized as follows: ## related work the task of image captioning has received considerable interest over the past decade @xcite @xcite @xcite @xcite . templatebased methods @xcite @xcite provide image grounding, but lack the ability to produce diverse captions, while early encoder-decoder methods @xcite @xcite @xcite produce diverse captions, but lack image grounding. contemporary encoder-decoder methods @xcite @xcite @xcite @xcite , including the method proposed in this work, can generate diverse captions with image grounding. most similar to our work, @xcite ) propose the use of a guiding object and produces a caption by using a forward and backward lstm to separately generate the caption text before and after the guiding object. in contrast, our approach involves a multi-modal transformer model that uses layers of self-attention and cross-attention to better ground the target caption using early-fused representations for the image and the guiding text, which in our work is not limited to boxable objects. ## method given an image i and guiding text @xmath0, ..., t l t } with l t tokens, the goal of guided image captioning is to generate an image description @xmath1, ..., y ly } with l y tokens, such that y focuses on the concepts provided in t . note that we do not necessitate our model to include the tokens of t inside y, yet we find that our trained model produces t inside y a majority of the time. interestingly, in some cases where t is absent from y, we find that t is paraphrased in y. see figure 5 for examples. both t i and y i are tokens in the shared model vocabulary. we employ a transformer @xcite ) based sequence to sequence model, with parameters θ where i and t are inputs to the encoder and y is the decoder output. we use a dataset with (i, t , y) tuples to search for the optimal model parameters θ * by maximizing the likelihood of the correct caption given below. in our model, the encoder input sequence comprises of image features followed by sub-token embeddings for the guiding text. we use a joint encoder for both image and guiding text to transform transformer encoder image guiding text graph rise rpn r-graph rise subword tokenizer fc fc fc subword embed trainable frozen caption r-cnn g r gr r frcnn t transformer decoder figure 2: our model architecture. thick arrows indicate a sequence, such as bounding boxes for image regions, or list of tokens in a text. the image features in the context of the guiding text concepts and vice versa, such that the decoder input is conditioned on the joint text-image representation. we represent the input image as a sequence of global and regional features, g and r, respectively. following @xcite , we use graph-regularized image semantic embedding (graphrise) as the global image features g. these 64-dimensional features, trained to discriminate o(40m) ultra-fine-grained semantic labels, have been shown to outperform other state of the art image embeddings @xcite . for regional features, we first compute bounding boxes using a region proposal network (rpn). following @xcite , we reimplement the faster r-cnn (frcnn) model trained to predict selected labels (1,600 objects and 400 attributes) in visual genome @xcite . this model returns k bounding box regions, each with a 2048-dimensional feature vector. @xcite observed that decoupling box proposal and featurization could improve downstream tasks. thus, we obtain two sets of regional features -r gr and r frcnn where the bounding box feature extractor are graphrise and frcnn respectively. we experimented with using either or both of these features for the top 16 regions. ## data we train our models on two datasets with different characteristics, we pick models according to automatic metrics computed over the validation (dev) split in each dataset. human evaluations were conducted on a separate test set which is out-of-domain for both training datasets. the sub-token vocabulary size is set to 4000 for both datasets, but a comparison of the number of unique tokens is reported in ## conclusion in this work, we compare conceptual captions and visual genome to analyze and determine which type of training set performs better guided image captioning on out-of-domain data. we show that although visual genome has humanannotated object-level captions and a higher number of unique tokens, training on the conceptual captions dataset with web-collected image-level captions that have a high diversity of writing styles and are image-dependent (unlike visual genome which has many image-independent target captions) produces better guided captions on out-ofdomain data. our results suggest that creating a good guided image captioning training set does not require laborious human annotations and that building these datasets by automatically scraping the internet which has the side effect of being more easily scaled-up can lead to better performance.