Cross-modal Coherence Modeling for Caption Generation (2020)

Abstract: We use coherence relations inspired by computational models of discourse to study the information needs and goals of image captioning. Using an annotation protocol specifically devised for capturing image–caption coherence relations, we annotate 10,000 instances from publicly-available image–caption pairs. We introduce a new task for learning inferences in imagery and text, coherence relation prediction, and show that these coherence annotations can be exploited to learn relation classifiers as an intermediary step, and also train coherence-aware, controllable image captioning models. The results show a dramatic improvement in the consistency and quality of the generated captions with respect to information needs specified via coherence relations.

## introduction the task of image captioning is seemingly straightforward to define: use natural language to generate a description that captures the salient content of an image. initial datasets, such as mscoco @xcite and flickr @xcite , approached this task directly, by asking crowd workers to describe images in text. unfortunately, such dedicated annotation efforts cannot yield enough data for training robust generation models; the resulting generated captions are plagued by content hallucinations @xcite that effectively preclude them for being used in real-world applications. in introducing the conceptual captions dataset, @xcite show that this dataset is large enough, at 3.3m examples, to significantly alleviate content hallucination. however, because the technique for creating such a large-scale resource relies on harvesting existing data from the web, it no longer guarantees consistent image-text relations. for example, along with descriptive captions (e.g.,"this is a person in a suit"), the dataset also includes texts that provide contextual background (e.g., "this is the new general manger of the team") and subjective evaluations (e.g., "this is stylish"). as a result, current captioning models trained on conceptual captions avoid content hallucination but also introduce different, more subtle and harderto-detect issues related to possible context hallucinations (i.e., is this actually the new general manager?) or subjective-judgement hallucinations (i.e., whose judgment is this anyway?). in this paper, we propose to tackle this issue of large-scale image-caption consistency using a coherence-aware approach inspired by the framework of discourse coherence theory @xcite . this framework characterizes the inferences that give discourse units a coherent joint interpretation using a constrained inventory of coherence relations. in multimodal presentations, discourse units can be images as well as text, so we appeal to new image-text coherence relations that capture the structural, logical, and purposeful relationships between the contributions of the visual modality and the contributions of the textual modality. for instance, a visible relation characterizes grounding texts that serve to make key aspects of the image content common ground (perhaps to a visually-impaired reader), analogous to restatement relations between one text unit and another; visible relations are key to traditional descriptive captions such as "this is a person in a suit." meanwhile, a story relation characterizes texts that develop the circumstances depicted in the image in pursuit of free-standing communicative goals, analogous to occasion or narration relations in text; story relations can go far beyond image content ("i hiked this mountain as we found it on a list for good hikes for kids") and so pinpoint one kind of risk for context hallucinations. the key contribution of our work is to show that image-text coherence can be systematized, recognized, and used to control image captioning models. to support our argument, we create a coherencerelation annotation protocol for image-caption pairs, which we use to annotate 10,000 imagecaption pairs over images coming from the conceptual captions @xcite and open images @xcite datasets. we release foot_0 this dataset, named clue, to facilitate follow-up research. by annotating these coherence relations in the context of image captioning, we open up the possibility of analyzing patterns of information in image-text presentations at web scale. in addition, we show that we can exploit these coherence-relation annotations by training models to automatically induce them, as well as by building models for coherence-aware image captioning. because they are driven by input coherence relations, these captioning models can be used to generate captions that are better suited to meet specific information needs and goals. ## prior work there are diverse ways to characterize the communicative functions of text and images in multimodal documents @xcite , any of which can provide the basis for computational work. some studies emphasize the distinctive cognitive effects of imagery in directing attention; engaging perceptual, spatial and embodied reasoning; or eliciting emotion @xcite . some look at contrasts across style and genre @xcite . others look holistically at the content of text and imagery as complementary or redundant @xcite . unlike our approach, none of these methodologies attempt to characterize information-level inferences between images and text, so none is suitable for building generation models that control the information that text provides. while coherence theory has been applied to a range of multimodal communication, including comics @xcite , gesture @xcite , film @xcite , and demonstrations and other real-world events @xcite , applying coherence theory specifically to text-image presentations is less well explored. the closest work to ours is @xcite , who explore coherence relations between images and text in a multimodal recipe dataset. their relations are specialized to instructional discourse and they do not build machine learning models combining imagery and text. we consider more general coherence relations and a broader range of machine learning methods. we use our relations and introduce a coherenceaware caption generation model that improves the rate of good visible captions by around 30%. this is a considerable improvement over the recent models that have tried to achieve more control over neural language generation using an enhanced beam search @xcite , a memory network with multiple context information @xcite , forced attentions @xcite and modeling and learning compositional semantics using fine-grained annotations of entities in mscoco @xcite . ## coherence in images and captions the first step toward our goals is to characterize image-text coherence and annotate a sizable corpus of image-text pairs with coherence relations. we use an overlapping set of high-level relations, inspired both by theoretical work linking discourse coherence to discourse structure and discourse goals @xcite , and by previous successful discourse annotation campaigns @xcite . crucially, following previous work on text @xcite and multimodal discourse @xcite , visible, meta ## predicting coherence relations in this section, we introduce the task of predicting cross-modal coherence relations. we describe a number of preliminary experiments that justify the potential of machine learning models in classifying coherence relations in text and imagery. to this end, we train and test different models on the clue dataset to automatically predict the coherence labels given an image and its caption. ## generating coherent captions we use the coherence label predictions on the conceptual captions dataset (section 4) to train a coherence-aware caption generation model. transformer encoder transformer decoder image features extractor object classifier coherence label start token image coherence-aware caption trainable components pre-trained components model outputs model inputs model we model the output caption using a sequence-generation approach based on transformer networks @xcite . the output is the sequence of sub-tokens comprising the target caption. the input is obtained by concatenating the following features. image features we obtain a 64 dimensional representation for the image using the graph-rise (?) feature extractor, which employs a resnet-101 network to classify images into some 40m classes. we do not fine tune this image encoder model. we use the 64-dimensional feature available immediately before the classification layer, and embed into the transformer encoder embedding space using a trainable dense layer. detected objects we obtain object labels for the image using google cloud vision api. 4 we represent each label using pre-trained 512-dimensional vectors trained to predict co-occurring objects on web pages, in a similar fashion as the word2vec model @xcite . we embed each of these into the transformer encoder embedding space using a trainable dense layer. coherence relation label this is an input label fed at training time, for which we use the inferred coherence relation for the image-caption pair; at inference time, the label input is used to control the information in the generated caption. embeddings for the coherence labels are trainable model parameters. additionally, the relationship label serves as the start token for the transformer decoder (figure 5 ), i.e., it is made available both for the encoder network and directly for the decoder network. when training and evaluating a coherence-agnostic model, this label is set to a special symbol, such as none, essentially running the model without coherence information. for all models described in this paper, the transformer network has 6 encoder layers, 6 decoder layers, 8 attention heads, and a 512-dimensional embedding space. ## results and evaluation in what follows, we discuss evidence for our hypotheses: (a) a coherence-aware model presents information that is aligned with the goal of the evaluation by expert annotators we train the model described above with the predicted discourse relation labels for image-caption pairs in the conceptual captions training and validation sets. the checkpoint with highest cider @xcite score on the validation set is selected for inference and human evaluations. we asked our annotators to annotate a subset of randomly selected image-caption pairs generated by this model. these evaluation images were selected from the conceptual captions evaluation set based on their predicted coherence label using the single-label classifier (section 4) on the captions generated by the coherence-agnostic model (section 5). according to our sensitivity power analysis, with a sample size of 1500 image-text pairs, 300 in each category, we are able to detect effect sizes as small as 0.1650 with a power and significance level of 95%. a similar trend is observed when asking for, e.g., meta: 46.49% of the captions are evaluated to fit the meta label (non-overlapping; the column under meta), up 15.3% over the coherence-agnostic model (with 31.18% story). a qualitative analysis of the generated captions shows that captions generated under the meta label include terms such as "screenshot" and "view", while subjective captions come with adjectives such as "beautiful" or "favorite". figure 6 shows several examples. crowdsouring and automatic metrics for the following experiments, a subset of the conceptual captions validation data was selected where the ground-truth captions are labeled as visible. to compare the quality of the generated captions using our framework with other models, we follow the same crowdsourcing protocol that @xcite employed for quality assessment. we asked subjects whether the generated captions are "good" or not. 86% of the captions generated by the coherence-aware model were selected as "good" captions, whereas only 74% of the captions generated by the coherence-agnostic model were selected as "good" captions. note that, based on the human-evaluation data published foot_4 for the conceptual caption workshop at cvpr 2019, this rate is on average 67% "good" captions for the participating state-of-the-art models in 2019. furthermore, in a follow-up experiment we ask subjects to choose between a caption generated by the coherence-aware model and one generated by the coherence-agnostic model: 68.2% of the time subjects preferred the coherence-aware result, versus 31.8% for the coherence-agnostic one. in addition, we study the quality and the relevance of the captions generated by our model as suggested by @xcite . on a scale of 0 to 5, the average scores of the quality of the captions generated by the coherence-aware and the coherence-agnostic model are, respectively, 3.44 and 2.83. the average score of the relevance for the coherence-aware and the coherence-agnostic conditions are, respectively, 4.43 and 4.40. note that subjects rated the quality and the relevance of the captions while seeing the questions on the same page. screenshots and code for the experiments can be found on our github page. with the exception of the relevance condition, the results of the other questions that we asked in the crowdsourcing experiments are statistically significantly different (@xmath2.05, @xmath3.1|), which indicates that subjects prefer captions generated by the coherence-aware model. we also mention here that this difference in quality, albeit significant from a human-rating perspective, is not reflected in the cider score computed on the same data (against the available reference captions). the cider score of the captions generated by the coherence-aware and the coherence-agnostic models are, respectively, 0.958 and 0.964. this is not surprising, as the reference captions used by cider are subject to the same distribution over coherence relations as the rest of the data, and therefore generating caption outputs with a different coherence-relation distribution ( ## conclusions and future work representing coherence in image-text presentations can provide a scaffold for organizing, disambiguating and integrating the interpretation of communication across modalities. we show that cross-modal coherence modeling significantly improves the consistency and quality of the generated text with respect to information needs. this is a step forward towards designing systems that learn commonsense inferences in images and text and use that to communicate naturally and effectively with the users. in addition, the presented dataset, clue, provides opportunities for further theoretical and computational explorations. the experiments described for the coherence relation prediction task set the stage for designing better models for inferring coherence for images-text pairs. the presented work has limitations that can be addressed in future research. according to the description of the conceptual captions dataset, its captions have been hypernymized. however, by studying the examples in the other category, we discovered an additional coherence relation that exists between an image and caption, in which the caption identifies an object or entity in the image-identification. examples of this relation involves a caption that mentions the brand of a product or the name of the person in the image. identification is easy to annotate but missing from this work due to the properties of the corpus we annotated. future work should study this additional relation in the context of caption annotation and generation.