Informativity in Image Captions vs. Referring Expressions (2020)

Abstract: At the intersection between computer vision and natural language processing, there has been recent progress on two natural language generation tasks: Dense Image Captioning and Referring Expression Generation for objects in complex scenes. The former aims to provide a caption for a specified object in a complex scene for the benefit of an interlocutor who may not be able to see it. The latter aims to produce a referring expression that will serve to identify a given object in a scene that the interlocutor can see. The two tasks are designed for different assumptions about the common ground between the interlocutors, and serve very different purposes, although they both associate a linguistic description with an object in a complex scene. Despite these fundamental differences, the distinction between these two tasks is sometimes overlooked. Here, we undertake a side-by-side comparison between image captioning and reference game human datasets and show that they differ systematically with respect to informativity. We hope that an understanding of the systematic differences among these human datasets will ultimately allow them to be leveraged more effectively in the associated engineering tasks.

## introduction at the intersection between computer vision and natural language processing, there has been recent progress on two natural language generation tasks: dense image captioning and referring expression generation for objects in complex scenes @xcite @xcite @xcite @xcite . the former aims to provide a caption for a specified object in a complex scene for the benefit of an interlocutor who may not be able to see it, and may form part of a larger visual question answering (vqa) system @xcite @xcite . the latter aims to produce a referring expression that will serve to identify a given object in a scene that the interlocutor can see. the two tasks are designed for different assumptions about the common ground between the interlocutors, and serve very different purposes, although they both associate a linguistic description with an object in a complex scene. despite these fundamental differences, the distinction between these two tasks is sometimes overlooked @xcite @xcite . here, we undertake a side-by-side comparison between image captioning and reference game human datasets and show that they differ systematically with respect to informativity. we hope that an understanding of the systematic differences among these human datasets will ultimately allow them to be leveraged more effectively in the associated engineering tasks. ## background and predictions as the purpose of using a referring expression is to distinguish one referent from another, without being overly wordy, a naive expectation would be that referring expressions should contain as much information as is necessary to do that, and no more. in other words, descriptive modifiers are expected to be included only if they are informative in the sense of helping to narrow down on the set of potential referents. this kind of behavior is predicted by the rational speech act (rsa) framework (frank and goodman, 2012): speakers optimize their choice of expression through a tradeoff between accuracy and cost, and listeners use a bayesian reasoning process to identify a speaker's referent. in work on referring expression generation (reg; see @xcite , rsa has not been viewed entirely without skepticism. @xcite compare rsa to a probabilistic referential overspecification model (pro). they conclude that rsa is insufficient because it fails to consider overspecification and preference rankings when generating referring expressions. @xcite conduct production and interpretation studies that question the assumption that speakers aim to minimize production costs. their findings suggest that speakers may favor overspecification not only to help the listener, but to avoid the additional cognitive effort. amendments to rsa have been proposed in order to account for overinforativity. @xcite do so by adjusting the deterministic semantics that exists in the basic framework to continuous (fuzzy) semantics. @xcite leverage the captions from the visual genome corpus @xcite in order to define a semantics for an rsa-based referring expression generation system. the incremental nature of their system provides an alternative account of overinformativity, one which explains differences between languages with prenominal and postnominal adjectives @xcite . but overinformativity has its limits: there is still a basic trade-off between accuracy and cost at work in the realm of referring expressions. this basic premise predicts that referring expressions for objects in scenes with multiple objects of the same type will tend to be longer, as more content is necessary in order to distinguish one referent from another. captions are not subject to the same pressures. the purpose of a caption is not to distinguish one object from another, but rather to describe what is in the picture. hence we predict that the number of objects in a scene with the same type should have a significant impact on the length of a referring expression for an object of that type, but either less or no impact on the length of a caption. as we will show, this prediction is borne out by the data. we find furthermore that captions generally involve indefinite descriptions while referring expressions use definite descriptions, and referring expressions typically make use of more relational vocabulary (e.g. left, closest) than captions. ## approach the visual genome corpus @xcite provides a set of captions for objects in complex scenes, called region descriptions. we selected a subset of these images in order to construct a dataset of corresponding referring expressions. our dataset was constructed based on object types (e.g. horse, phone, vase) such that there exist images with one, two, and three objects of that type (e.g. one horse, two horses, and three horses). for each of the types satisfying this condition, we included two images with a single instance of the type, two with two instances (double), and two with three instances of the type (triple). a total of 198 images were included, comprising 33 sextuples. we developed an interactive web-based reference game in which a speaker was matched with a listener, and was told to complete the sentence draw a box around , for an object in a complex scene designated with a bounding box (see figure 1 ). participants were randomly assigned the role of speaker or listener and communicated through a modified chat window. the listener was instructed to draw a box around the entity indicated by the speaker, and the box drawn by the listener was shown to the speaker as feedback. we filtered out participants who did not attempt to distinguish one object from another in their responses (e.g. referring to one of three teddy bears as 'toy'), and we normalized the responses, taking into account self-corrections and variations in how speakers interpreted the task (e.g. 'the hose |i mean the horse' was normalized to 'the horse', and 'draw a box around the center horse' was normalized to 'the center horse'). our predictions about length are conditional on whether the referring expressions use a synonym or the same word as the target type; a hyponym would be an alternative strategy to include more specific information. we therefore analyzed the sense relation between the head noun of the description and the target type noun. we used a dependency parser to identify the head noun of the description, and categorized the head noun as a hyponym, synonym (or same word), or hy-pernym of the noun corresponding to the target type using wordnet foot_0 . we then carried out an analysis of the external syntax of these semi-normalized responses. some participants used full definite descriptions, as in the horse in the middle, while others left off the initial definite article horse in the middle, and others used an even more telegraphic style: horse in middle. the variation in style is of interest in its own right, but also makes the descriptions difficult to compare in terms of length. to resolve this, we normalized the responses to make them full noun phrases. we compared the length of the resulting fully normalized responses, comparing them to the captions in visual genome for the corresponding regions. ## results and discussion sample results are shown in figure 3 . in the image with a single (salient) plane, over-informative adjectives (red and white) are provided to describe the unique salient plane in the image (there is in fact another one in the background), while the referring expression provides just enough information to identify the salient plane (the plane). in the image with three polar bears, the caption is shorter than the referring expression; the caption simply describes the entity as a polar bear, while the referring expression provides enough information to distinguish the entity from other ones in the scene (the negation of a relational property, getting licked). in the image with two horses, the caption and the referring expression are of comparable length, but the caption provides non-distinguishing information; the referring expression uses the relational expression darker to uniquely identify a referent. in the image with three planes, again the caption and the referring expression are of comparable length, and the caption contains enough information to distinguish the referent from the other potential referents in the scene. however, the referring expression uses the relational term middle, while the caption describes a non-relational attribute of the object. and of course, the referring expressions use definite articles, while the captions tend to use indefinite articles. these images are representative of the overall set of patterns. let us turn now to a quantitative analysis. we note first that the overwhelming majority of the referring expressions we gathered (94.5%) were noun phrases headed by the same noun as the target type or a synonym; only 5.5% were a hyponym or a hypernym. we therefore predict for our dataset overall that in images with multiple instances of a given type, referring expressions picking out one of those instances should be longer, in comparison to images with only a single instance of the type. of the unique referring expressions we gathered, 63% used a definite description. less than 1% used an indefinite description. the remaining group was predominantly made up of descriptions lacking an initial article, e.g. horse on (the) left, with only a handful of exceptions. in contrast, in the corresponding region descriptions (captions), 4.7% used a definite description, and 39.6% used an indefinite description. the remaining set were predominantly noun phrases with no initial article (e.g. large brown bear by a rocky wall; notice here that the embedded noun phrase is indefinite, however). perhaps surprisingly, 11.9% of the region descriptions took the form of a sentence, e.g. pizza is thin crust or the zebra has stripes. it seems the region description data reflects a range of approaches to the annotation task; this is a source of noise in the data. we now compare the captions to the referring expressions with respect to length. the results are summarized in these results support the hypothesis that referring expressions and captions are subject to very different pressures with respect to informativity. referring expressions include descriptive information for the purpose of distinguishing one referent from another, while captions do not. finally, the kind of information that can help to discriminate among referents often consists in relations that instances of the type stand in to each other (e.g. darker brown, in the middle, closest, on the right). we defined a relational modifier narrowly as a modifier that specifies a characteristic of an object in relation to another instance of the type named by the head noun, excluding gradable size adjectives like big. even on this narrow definition, we find a strong difference between captions and referring expressions, with captions exhibiting such modifiers at a rate of less than one percent, and referring expressions exhibit- ## conclusion this comparison has shown that referring expressions and captions are subject to very different pressures with respect to informativity. when there is only a single instance of a given type (or only one instance that is visually salient), then it suffices to refer to it using 'the [noun]', where '[noun]' identifies the type. a caption, on the other hand, is there to tell someone about the object, so descriptive detail is more likely to be added even when it does not help to identify the referent. but captions are not systematically longer than referring expressions, either. descriptive modifiers will be added to a referring expression when they serve the purpose of distinguishing the referent from other ones, i.e., when they are informative. this is why expressions referring to objects of a type that is multiply instantiated within a scene tend to be longer. a caption and the corresponding referring expression may also be equally long, but the kind of information they contain is different: a caption is more likely to contain information that does not help to discriminate among the possible referents. relational vocabulary is for distinguishing among referents. we hope that these findings will enable image captioning datasets to be leveraged more effectively in systems for generating expressions that caption: 'red and white plane' ref. exp.: 'the plane' caption: 'a polar bear cub' ref. exp.: 'the bear that's not getting licked' caption: 'a brown and white horse' ref. exp.: 'the darker brown horse' caption: 'plane with a propeller on the front' ref. exp.: 'the airplane in the middle'