C ap WAP : Image Captioning with a Purpose (2020)

Abstract: The traditional image captioning task uses generic reference captions to provide textual information about images. Different user populations, however, will care about different visual aspects of images. In this paper, we propose a new task, Captioning with A Purpose (CapWAP). Our goal is to develop systems that can be tailored to be useful for the information needs of an intended population, rather than merely provide generic information about an image. In this task, we use question-answer (QA) pairs—a natural expression of information need—from users, instead of reference captions, for both training and post-inference evaluation. We show that it is possible to use reinforcement learning to directly optimize for the intended information need, by rewarding outputs that allow a question answering model to provide correct answers to sampled user questions. We convert several visual question answering datasets into CapWAP datasets, and demonstrate that under a variety of scenarios our purposeful captioning system learns to anticipate and fulfill specific information needs better than its generic counterparts, as measured by QA performance on user questions from unseen images, when using the caption alone as context.

## introduction the image captioning task typically selects for captions having high similarity with generic human references. while this task definition has driven much of the research in the field, the end-purpose of these captions is not always clearly articulated. we argue that (1) generic annotations may not be representative of users' information needs, (2) user questions are a more natural way of articulating information needs, and (3) optimizing captions to provide correct answers to those questions allows ## related work since the early days of the field, human-written references have been used for the supervised training and evaluation of text generation systems, including image captioning, summarization, and other related applications @xcite @xcite . recently, researchers have begun to consider a multitude of different objectives for reference comparison @xcite , or even parametric regressions trained on human judgements @xcite . though diverse in approach, each ultimately relies on designing a robust general-purpose metric. in practice, engineering such a metric is challenging-if at all possible @xcite . here we take a more empirical approach by relying on the information need expressed by users' questions. many studies have observed that reference-trained captioning models suffer from systematic usability issues-including being rigid, neglecting relevant image aspects, and regurgitating frequent phrases @xcite . as a result, much effort has been focused on developing secondary, corrective objectives-for instance, "discriminability" losses encouraging captions to be unique @xcite @xcite . while these measures provide some fixes, they do not necessarily reflect user information needs-a central concept in capwap. the idea of using qa for assessing information quality has been proposed in recent work for text summarization (arumae and @xcite @xcite . the primary distinctions with our work are both the domain (images) and how questions are obtained-both of which impact the task objective and learning procedure. in this prior work, questions are generated programmatically (e.g., following @xcite . such "questions" may not necessarily reflect real user preferences. our work focuses on qa not as just another method to improve standard referencebased metrics, but as a key, flexible way of formulating user information need-and as such we focus on challenging, real qa datasets. furthermore, we train on this signal, rather than rely on it solely for evaluation @xcite . efforts to leverage vqa resources to drive image captioning, and vice-versa, via variations of transfer learning, have also received extensive interest in recent years @xcite @xcite . as opposed to optimizing metrics for specific vqa or supervised captioning benchmarks, the primary focus in capwap is on modeling the target user population in order to anticipate the correct information-need. in a similar vein, vqa and textual qa resources have also been leveraged for active learning @xcite , where the model learns to query its environment for information it is uncertain about to help improve its performance on the given task. the key distinction with our work is the directionality of the questions. in capwap, the model uses questions posed by the users to infer their latent information need-which is a distinctly different, and quite challenging, setting. ## problem formulation we begin by formulating the capwap task. in our setting, questions and answers are the only source of direct supervision assumed during training. at test time, the model is not given questions in advance, but rather must anticipate the information need of the user, and generate captions that answer the forthcoming questions in expectation. task setting: given an image x the model must output a caption y, such that y entails the answer a for a question-answer pair (q, a) sampled from some underlying distribution d. examples from d are given during training, but are not known in advance by the generation model at test time. ## an approach to capwap given that we only have access to question-answer pairs during training, but not during inference, how can we learn a model for this task? eq. 1 naturally lends itself to a reinforcement learning (rl) framework where the model receives a reward r (e.g., @xmath0, q, a)) for each generated caption y and training qa pair (q, a). g θ (y|x) can be cast as a policy, and updated with policy gradients. optimizing such a policy, however, poses a technical challenge because the model is only rewarded for correct (or partially correct) answers, which is initially a rare event. transferring g θ (y|x) from generic captioning data can be a useful starting point. our method then follows this recipe: in sections 4.1 and 4.2 we detail our model for g θ (y|x), and the above training procedure. note that d generic is assumed to be out-of-domain for our intended captioning purpose, d target . since we are interested in diverse user-generated questions and information needs, the generic captioning data can often diverge dramatically from our end goal. to improve transfer, in section 4.3 we further develop a novel mechanism for automatically generating in-domain synthetic data that can be used as pre-training for guiding g θ (y|x) towards balanced areas of high reward in d target . ## results in the following, we address several key research questions relating to our approach to capwap, and the broader assumptions, strengths, and limitations of using qa to drive the process. evaluation of generic captions: we begin by empirically verifying our introductory claim that training on generic reference captions can poorly reflect the varying, user-specific information need. adaptation to information need: we next test the effectiveness of our proposed approach at tailoring captions to meet the specific information need stipulated by our datasets. our results in dataset input image generic caption output purposeful caption (rl + syn) output unseen question unseen answer capvqa a man playing tennis a man in white shirt and white shorts playing a game of tennis on a grass court what color is he wearing? white shirt and shorts capgqa a man riding on the back of a horse a man riding a horse next to a woman on the right side what color is the horse the man is to the right of? brown capvisual7w a couple of people that are standing in the snow two people posing for a picture taken at a ski slope why are the kids wearing coats ? it is cold capvizwiz a plate of food that is on a table this is a picture of a sweet corn frozen dinner what kind of tv dinner is this? lean cuisine these examples are representative of the way in which the various datasets ask about image content. tendencies include colors for capvqa, spatial relations for capgqa, higherlevel concepts in capvisual7w, and ocr in capvizwiz. note that our approach to capwap does not (and likely cannot ever perfectly) anticipate all unseen questions-but is distributionally closer in terms of content selection. importance of synthetic pre-training: a deficiency of qa-based rewards is that they neither explicitly enforce text fluency, nor penalize the system when content is produced that is either not relevant or not true. on the other hand, when reference captions are available, it is easy to learn a fluent language model. qualitative discussion: the qualitative effect of our method is quite intuitive (see figure 4 as well as in ablation studies: tables 7 and 8 show the effects of different design choices in our rl and rl + syn models. a significant challenge for cap-wap systems, as previously discussed and illustrated in future work: the capwap paradigm introduces new challenges for learning effective systems, some of which our approach solves, and others which it still leaves open (e.g., maintaining fluency and fidelty). while some may be addressed by large-scale multi-modal models @xcite , it is still unclear whether they would fully cover the diversity of information that real users are interested in (e.g., ocr). ## conclusion we defined and studied the capwap task, where question-answer pairs provided by users are used as a source of supervision for learning their visual information needs. our results indicate that measuring caption content by its ability to logically support the answers to typical qa pairs from a target audience is (1) not only feasible, but also (2) a good proxy for uncovering information need. we hope this work will motivate the image captioning field to learn to anticipate and provide for the information needs of specific user communities. yuke zhu, oliver groth, michael bernstein, and li fei-fei. 2016. visual7w: grounded question answering in images. in cvpr.