Pragmatic Issue-Sensitive Image Captioning (2020)

Abstract: Image captioning systems need to produce texts that are not only true but also relevant in that they are properly aligned with the current issues. For instance, in a newspaper article about a sports event, a caption that not only identifies the player in a picture but also comments on their ethnicity could create unwanted reader reactions. To address this, we propose Issue-Sensitive Image Captioning (ISIC). In ISIC, the captioner is given a target image and an issue, which is a set of images partitioned in a way that specifies what information is relevant. For the sports article, we could construct a partition that places images into equivalence classes based on player position. To model this task, we use an extension of the Rational Speech Acts model. Our extension is built on top of state-of-the-art pretrained neural image captioners and explicitly uses image partitions to control caption generation. In both automatic and human evaluations, we show that these models generate captions that are descriptive and issue-sensitive. Finally, we show how ISIC can complement and enrich the related task of Visual Question Answering.

## introduction image captioning systems have improved dramatically over the last few years @xcite @xcite @xcite , creating new opportunities to design systems that are not just accurate, but also produce descriptions that include relevant, characterizing aspects of their inputs. many of these efforts are guided by the insight that high-quality captions are implicitly shaped by the communicative goal of identifying the target image up to some level of granularity @xcite @xcite . in this paper, we seek to more tightly control the issues target caption a small brown bird with a tan chest and a tan beak this bird has a brown crown a white eyebrow and a rounded belly what is the color of the bird? what is the head pattern of the bird? figure 1 : examples highlighting the power of an issuesensitive image captioner. four images are partitioned in two ways, each capturing different issues by grouping them into equivalence classes. the first row contrasts the brown and grey color of the bird, and the second contrasts the existence of white eyebrows. the target image is the same in both cases, but the partition leads to different captions that key into the structure of the input issue. to push captioners to produce more relevant texts, we propose the task of issue-sensitive image captioning (isic). in isic, the captioner's inputs are image/issue pairs, where an issue is a set of images partitioned in a way that specifies what information is relevant. in our first example above, we might define a partition that grouped players into equivalence classes based on their team positions, abstracting away from other facts about them. for the second example, we might choose a more fine-grained partition based on position and demographic features. given such inputs, the objective of the captioner is to produce a text that both accurately and uniquely describes the cell of the partition containing the target image. figure 1 illustrates with examples from our own models and experiments. in defining the task this way, we are inspired by visual question answering (vqa; @xcite , but isic differs from vqa in two crucial respects. first, we seek full image captions rather than direct answers. second, our question inputs are not texts, but rather issues in the semantic sense: partitions on subsets of the available images. the isic module reasons about the cells in these partitions as alternatives to the target image, and our notion of relevance is defined in these terms. nonetheless, vqa and isic complement each other: issues (as partitions) can be automatically derived from available image captioning and vqa datasets (section 6), opening up new avenues for vqa as well. our models are built on top of pretrained image captioners with no need for additional training or fine-tuning. this is achieved by extending those models according to the rational speech acts model (rsa; frank and goodman 2012; @xcite . rsa has been applied successfully to many nlp tasks (section 2.3). our key modeling innovation lies in building issues into these models. in this, we are inspired by linguistic work on question-sensitive rsa @xcite . our central experiments are with the caltech-uc san diego-bird dataset (cub; @xcite . this dataset contains extensive attribute annotations that allow us to study the effects of our models in precise ways. using cub, we provide quantitative evidence that our rsa-based models generate captions that both richly describe the tar-get image and achieve the desired kinds of issuesensitivity. we complement these automatic evaluation with a human evaluation in which participants judged our models to be significantly more issue-sensitive than standard image captioners. finally, we show how to apply our methods to larger image captioning and vqa datasets that require more heuristic methods for defining issues. these experiments begin to suggest the potential value of issue-sensitivity in other domains that involve controllable text generation. we share code for reproducibility and future development at https: //github.com/windweller/pragmatic-isic. ## task formulation in standard image captioning, the input i is an image drawn from a set of images i, and the output w is a sequence of tokens [w 1 , . . . , w n ] such that each w @xmath0, where v is the vocabulary. in isic, we extend standard image captioning by redefining the inputs as pairs (c, i), where c is a partition foot_0 on a subset of elements of i and @xmath1. we refer to the partitions c as issues, for the reasons discussed in section 2.4. the goal of isic is as follows: given input (c, i), produce a caption w that provides a true resolution of c for i, which reduces to w identifying the cell of c that contains i, as discussed in section 2.4. . . in principle, we could try to learn this kind of issue sensitivity directly from a dataset of examples ((c, i), w). we do think such dataset could be collected, as discussed briefly in section 7. however, such datasets would be very large (each image needs to collect |c| number of captions), and our primary modeling goal is to show that such datasets need not be created. the issue-sensitive pragmatic model we introduce next can realize the goal of isic without training data of this kind. "a red square" "a small square" ## ms coco and vqa 2.0 the annotations in the cub dataset allow us to generate nuanced issues that are tightly connected to the content of the images. it is rare to have this level of detail in an image dataset, so it is important to show that our method is applicable to less controlled, broader coverage datasets as well. as a first step in this direction, we now show how to apply our method using the vqa 2.0 dataset @xcite , which extends ms coco @xcite with the question and answer annotations needed for vqa. while ms coco does have instancelevel annotations, they are mostly general category labels, so the attribute-dependent method we used for cub isn't effective here. however, vqa offers a benefit: one can now control captions by generating issues from questions. dataset ms coco contains 328k images that are annotated with instance-level information. the images are mostly everyday objects and scenes. a subset of them (204,721 examples) are annotated with whole image captions. @xcite built on this resource to create a vqa dataset, and @xcite further extended that work to create vqa 2.0, which reduces certain linguistic biases that made aspects of the initial vqa task artificially easy. vqa 2.0 provides 1,105,904 question annotations for all the images from ms coco. ## conclusion we defined the task of issue-sensitive image captioning (isic) and developed a bayesian pragmatic model that allows us to address this task successfully using existing datasets and pretrained image captioning systems. we see two natural extensions of this approach that might be explored. first, the method we proposed can be used as a method for assessing the quality of the underlying caption model. using a dataset with issue annotations, if the model trained over the plain captions is more issue-sensitive, then it is better at decomposing the content of an image by its objects and abstract concepts. second, one could extend our notion of issuesensitivity to other domains. as we saw in section 6, questions (as texts) naturally give rise to issues in our sense where the domain is sufficiently structured, so these ideas might find applicability in the context of question answering and other areas of controllable natural language generation.