Pragmatic Inference with a CLIP Listener for Contrastive Captioning (2023)

Abstract: We propose a simple yet effective and robust method for contrastive captioning: generating discriminative captions that distinguish target images from very similar alternative distractor images. Our approach is built on a pragmatic inference procedure that formulates captioning as a reference game between a speaker, which produces possible captions describing the target, and a listener, which selects the target given the caption. Unlike previous methods that derive both speaker and listener distributions from a single captioning model, we leverage an off-the-shelf CLIP model to parameterize the listener. Compared with captioner-only pragmatic models, our method benefits from rich vision-language alignment representations from CLIP when reasoning over distractors. Like previous methods for discriminative captioning, our method uses a hyperparameter to control the tradeoff between the informativity (how likely captions are to allow a human listener to discriminate the target image) and the fluency of the captions. However, we find that our method is substantially more robust to the value of this hyperparameter than past methods, which allows us to automatically optimize the captions for informativity — outperforming past methods for discriminative captioning by 11% to 15% accuracy in human evaluations.

## introduction discriminative captioning provides a challenging testbed for generating context-sensitive grounded language. in this task, a model must produce a description of a target image (e.g., the green highlighted image in figure 1 ) that allows a person to correctly identify the target image from among a set of similar distractor images (e.g., the red highlighted images). good captions must strike a balance between two criteria: (1) being fluent figure 1 : illustration of the contrastive captioning task with a random example from the imagecode dataset. models are tasked with generating captions that distinguish the target image (a) from other very similar distractors images (b) to (d). (there are a total of 9 distractors in each set of images, we omit the rest of them for simplicity of illustration.) compared with baselines from previous work, our proposed approach, picl, generates informative captions that help clearly identify the target out of the distractors, while remaining natural and fluent. past work on discriminative captioning has successfully applied techniques from computational pragmatics to trade off between the two criteria above @xcite @xcite . possible captions are selected using a combination of two scoring functions: (1) the caption's probability under a standard image captioning model, or base speaker score, which measures the caption's fluency and faithfulness to the image, and (2) a base listener score, which predicts how likely a human listener would be to correctly identify the target image given the caption, i.e. measuring discriminativeness. these past works typically obtain the listener scores from the image captioning (speaker) model itself, for example using bayesian inference over the set of possible images @xcite . the relative weight of these two scores is controlled using a informativity hyperparameter, foot_1 whose value affects the tradeoff between producing captions that are predicted to be fluent and faithful, versus captions that are predicted to be discriminative. it is challenging to automatically choose a value for this hyperparameter, as captions that appear to be discriminative under a captioning model are frequently uninformative for people @xcite . our approach, picl (pragmatic inference with a clip listener) follows this same pragmatic framework, but scores discriminativeness using a listener model separate from the speaker. we implement the listener model using clip @xcite . as shown in previous work, the rich vision-language representation learned in clip (1) provides robust assessments of modelgenerated captions that highly correlate with human judgments @xcite , and (2) effectively quantifies the degree of discriminativeness/informativeness of visual referring expressions @xcite . to evaluate picl, we conduct experiments with sets of images from imagecode @xcite , a challenging dataset originally designed for contrastive retrieval: retrieving target images among a set of distractors given contextual descriptions. we perform contrastive captioning on this dataset for the first time. we compare picl to past work on two criteria: (1) informativeness and (2) fluency, evaluating both metrics using automatic as well as human evaluations. results show that our approach typically outperforms past methods on both criteria, and is substantially more robust to the value of the informativity hyperparameter. in particular, we are able to choose this hyperparameter automatically by maximizing how informative the captions are predicted to be to human evaluators. in contrast, we find that maximizing predicted informativity leads past methods to produce captions that are so disfluent that they are misleading for people. in this automatic hyperparameter selection setting, our method produces captions that are 11% to 15% easier for human annotators to interpret correctly than past work. ## related work contrastive captioning a variety of methods for contrastive captioning generate captions that optimize for discriminative objectives, e.g., minimizing the textual similarity between captions for the target and distractor images @xcite , using generated captions as input to image retrieval models @xcite , and computing clip similarity scores between captions and target images @xcite . other methods involve leveraging fine-grained image regional features to generate distinctive captions based on similar and/or unique objects among target and distractors @xcite , paraphrasing generic captions to enhance both diversity and informativeness @xcite , and finetuning rl-optimized caption models to encourage low-frequency words @xcite . most of the methods above require training a discriminative captioning model -either by designing an discriminative captioning architecture that takes multiple images as input, or fine-tuning a model using discriminative rewards. in contrast, our proposed approach is fully inference-time -it requires no training, and is applicable to any off-the-shelf generic captioning model. our approach builds on a family of inferencetime pragmatic-based contrastive captioning methods which have taken one of two approaches: (1) incrementally generating captions but using only a captioning model (our speaker model), where tokens are chosen that have high probability for the target image and low probability for the distractor @xcite @xcite or (2) using a separate discriminative model but selecting a discriminative caption from among a set of entire captions generated by the speaker model for the target image @xcite . our work shows that these approaches can be productively combined, using a strong off-theshelf discriminative model (clip) to guide the incremental generation of captions. this allows us to tackle a more challenging dataset and task than previous discriminative captioning work, containing a large number (10) of highly-similar distractor images. pragmatics our approach to contrastive generation follows a long line of work on computational pragmatics, particularly in the rational speech acts framework @xcite which models language generation as an interaction between speakers and listeners. prior work has found that pragmatic generation can improve performance on a variety of nlp tasks, including reference games @xcite , instruction generation @xcite , summarization @xcite , machine translation (cohn-gordon and goodman, 2019), and dialogue @xcite . tradeoff between discriminativeness and accuracy/fluency assessing the quality of image captions requires multifaceted evaluation. prior work on contrastive/discriminative captioning investigates the tradeoff of model performance between discriminativeness and accuracy/fluency @xcite @xcite @xcite . in this paper, we also perform an extensive study on the tradeoff between informativeness and fluency. specifically, we focus on analyzing the robustness of the proposed and baseline methods in the tradeoff according to the selection of hyperparameters. ## method our picl approach conducts incremental pragmatic inference at the token level by combining a base speaker and a clip listener to derive a pragmatic speaker. at each step of decoding, the base speaker selects a set of candidate tokens and adds them to partial captions. given candidate partial captions, the listener updates its beliefs on which is the target among the set of images based on clip similarity measurement. in particular, it contrasts each partial caption to all the images by calculating the clip similarity scores of partial caption-image pairs and normalizes over all images to derive the listener likelihood. finally, a pragmatic speaker reasons over both the base speaker and listener by combining their distribution to rerank partial captions, select a highly-scored subset and proceed to the next decoding step. ## experimental setup we evaluate picl on imagecode @xcite , a dataset originally designed for image retrieval with contextual descriptions. given the high visual similarity of the images in each problem in the dataset, we adopt it as a challenging testbed for discriminative captioning. we evaluate picl and competitive baseline methods on two criteria, informativeness and fluency, using both automatic and human evaluation. for informativeness, we follow previous work @xcite to automatically evaluate the performance of pragmatic models with an evaluating listener l eval . the discriminativeness of the method being evaluated is quantified by the retrieval accuracy of l eval with method-generated captions as input. for fluency, we score the well-formedness of generated captions with the perplexity (ppl) under gpt-2 @xcite . in addition to the automatic evaluation, we conduct human evaluation where annotators are tasked to a) retrieve the target image given the caption and b) score the fluency of the caption. ## conclusion we propose an incremental pragmatic inference approach with a clip listener, which combines the strengths of previous approaches that conduct incremental pragmatic reasoning with a separately modeled listener. we identify strengths and weaknesses of automatic model-based evaluation of discriminative captioning systems, and suggest that future work 1) control for the disfluency of generated captions and not solely optimize for predicted informativity and 2) use human evaluations. in human evaluations, our approach outperforms previous discriminative captioning methods, and is substantially more robust than previous approaches in trading off between the fluency and informativity of the captions to human listeners.