Referring Expression Generation in Visually Grounded Dialogue with Discourse-aware Comprehension Guiding (2024)

Abstract: We propose an approach to referring expression generation (REG) in visually grounded dialogue that is meant to produce referring expressions (REs) that are both discriminative and discourse-appropriate. Our method constitutes a two-stage process. First, we model REG as a text- and image-conditioned next-token prediction task. REs are autoregressively generated based on their preceding linguistic context and a visual representation of the referent. Second, we propose the use of discourse-aware comprehension guiding as part of a generate-and-rerank strategy through which candidate REs generated with our REG model are reranked based on their discourse-dependent discriminatory power. Results from our human evaluation indicate that our proposed two-stage approach is effective in producing discriminative REs, with higher performance in terms of text-image retrieval accuracy for reranked REs compared to those generated using greedy decoding.

## introduction a visually grounded dialogue is a conversation in which speakers refer to entities in a (shared) visual context. they do so by producing referring expressions (res). the listener is expected to use the re to identify the target entity, i.e., the referent. whether the listener is successful in doing so depends on several factors, one being how specific the description of the referent was. with regard to specification, there exists a trade-off between discriminatory power and efficiency. on the one hand, the aim is to produce an unambiguous expression with which a referent can be successfully identified, whereas on the other hand a cooperative speaker is expected to make their contribution as economical as possible, while still avoiding ambiguity @xcite . to illustrate, consider the three phones depicted in figure 1 . if the intention of a speaker was to produce a description based on visual content that uniquely identified the phone second from the left, "the phone with the qwerty keyboard" would be underspecified, as it applies to both the intended target as well as the right-most image. to avoid underspecification, additional content could be added to the re, possibly resulting in a description such as "the mostly black nokia e75 mobile phone with the side-sliding qwerty keyboard and keypad". this re does set apart the target from the distractors, but is overspecified, as the description contains more content than is strictly required for identification of the referent in this context, violating the gricean maxim of quantity @xcite . in determining form and lexical content of res, context plays a crucial role. we will again use figure 1 to illustrate this by example. a attempts to draw the attention of b to a specific phone by referencing its brand name. however, since b recognizes two phones to be from this brand, b asks a clarification question that focuses on color. there are two things to note here. first, the res produced by b, in particular "the black one", only work as discriminative references due to the mention of the brand name just prior, as "one" is here a proform of "nokia" (the right-most phone is also black). second is the symmetry between the res, showing conventional preservation of form. for a conversational agent to take part in visually grounded dialogue, it would preferably generate res in a similar, context-dependent manner, as this is expected by human conversational partners. the computational modeling of this process is the do-main of referring expression generation (reg), a core natural language generation (nlg) task for which a considerable body of work exists, spanning decades (see e.g., @xcite . however, reg has traditionally focused primarily on the discriminative properties of res, leaving discourse-appropriateness in the context of conversation a somewhat understudied problem. in this paper, we propose an approach to reg for visually grounded dialogue that is meant to satisfy the discriminative property, while simultaneously accounting for discourse-appropriateness. we frame the problem as a two-stage process: in the first stage, we model reg as a text-and imageconditioned next-token prediction task: given a dialogue history, i.e., a preceding linguistic context, and the image of a referent, we autoregressively generate an re as a continuation of the existing linguistic context, using a fine-tuned vision-language model (vlm). while at this stage we expect to generate an re that fits the dialogue context and is indicative of the target image, it is not necessarily discriminative with respect to distractors. we, therefore, propose to use comprehension guiding as part of a generate-and-rerank strategy (see e.g., @xcite in stage two; our goal being to select an re with discriminative properties. crucially, we introduce discourse-aware comprehension guiding as a way to estimate the discriminatory power of candidate res based on the dialogue context and incorporate this in the candidate selection process. our main contributions are as follows: although the bulk of the computation for most downstream tasks has been delegated to the pretraining of the base model, fine-tuning may still require significant computational resources. to combat this issue, parameter-efficient fine-tuning methods have been proposed, such as low-rank adaptation (lora, @xcite . by freezing the pretrained model weights and instead training rank decomposition matrices that have been added to the dense layers of the network, lora manages to reduce the number of trainable parameters by several orders of magnitude, often without considerable adverse effects to downstream performance. aside from language, transformers have shown promising results when it comes to modeling other modalities (e.g., @xcite . of particular interest here are multimodal models that combine vision and language. vlms such as clip @xcite have learned to jointly embed both modalities via contrastive pretraining objectives. their learned representations have shown to be useful for discriminative downstream vision-language tasks, such as text-image retrieval (tir). we will hereafter refer to these models as discriminative vlms. other vlms such as flamingo @xcite , blip-2 @xcite , kosmos-2 @xcite , llava @xcite , and internvl @xcite have been introduced to address generative downstream tasks, such as image captioning and (multi-turn) visual-question answering. these generative vlms, sometimes called multimodal llms (mllms), are able to autoregressively output text based on multimodal inputs, reg-with-guiding, doi:10.5281/zenodo.13225148 as they are built on (pretrained) llms with some form of visual input conditioning. this makes them particularly useful for inherently multimodal text generation problems such as reg for visually grounded dialogue. reg has been defined as a task that is chiefly concerned with identification @xcite . as such, most work in this area emphasizes the discriminative properties of res. the goal is to generate an expression with which a referent can be unambiguously identified. whether a candidate re possesses this property is context-dependent, where context represents a multi-faceted concept. one facet is the visual context in which the referent is embedded, often together with entities that may be mistaken for the referent, i.e., distractors. various strategies have been proposed to have neural models take into account the visual context and attempt to maximize discriminatory power of generated res, including discriminative decoding (e.g., @xcite and comprehension-guiding (e.g., @xcite . these methods typically incorporate some manner of scoring (partial) candidate res on the basis of their alignment with pragmatic principles, either at inference time to guide decoding, or as part of a generate-and-rerank strategy, a commonly used approach for a variety of nlg problems (e.g., @xcite @xcite . in the latter case, a reg model will generate a set of candidate res which are reranked on the basis of their discriminatory power according to some referring expression comprehension (rec) model. these strategies, however, tend to focus primarily on the generation of definite descriptions, disregarding other forms of res such as pronouns, and do not fully consider the dialogue context in which the res would be used. earlier work on rule-based reg did address some context-sensitive aspects, such as the by @xcite proposed extensions to the influential incremental algorithm @xcite , which included reduced descriptions of subsequent mentions and pronominalization. more recent work that explicitly considered the linguistic context in addition to the visual context has instead attempted to generate discriminative referring utterances @xcite , under the assumption, however, that each utterance only mentions a single referent. ## method in this work, we focus on generating res conditioned on a multimodal dialogue context for referents that are represented by independent images. this setting bares some resemblance to that of discriminative image captioning (see e.g., @xcite @xcite . reg more commonly attempts to describe objects or entities, represented by bounding boxes or segmentation masks, in single images or scenes. spatial relations frequently become part of distinguishing descriptions in such settings as a result. our method, however, focuses instead on generating res based on visual content in situations that have been specifically designed for this to be challenging. we leave extending the framework to incorporate spatial relations to future work. ## results our results are based on 1305 of the 1319 annotated mentions of single-image referents; 14 samples were excluded as their target referents were not part of the set of candidate referents as a consequence of evaluating with the reduced visual context. managed to closely replicate the results reported by @xcite despite our variant of the crdg being incremental. ## discussion in this paper, we explored the problem of reg in visually grounded dialogue. our aim was to realize the generation of res that were not only discriminative, but also appropriate for the dialogue context in which they would be used. we proposed to approach the problem from a causal language modeling perspective, where the generation of tokens would be conditioned on both text and images. by fine-tuning a generative vlm, idefics @xcite , we showed it is possible to generate res that are indicative of the referent while suitable for the dialogue context. notably, we were successful using a parameter-efficient fine-tuning approach @xcite and while having relatively limited data for training @xcite . in addition, we introduced discourse-aware comprehension-guiding to evaluate whether candidate res are discriminative given their linguistic context. by adding candidate res to the dialogue for which they were generated, we were able to use the crdg framework of @xcite to score candidate res on their discourse-dependent discriminatory power. finally, we showed that human tir accuracy using candidate res selected based on a weighted reranking of scores derived from this discourse-aware rec model was on average higher than for candidate res generated through greedy decoding. one of the main benefits of our approach is the ability for the reg model to generate res that visual context linguistic context [...] [...] [...] a: the poodle is the one that looks like a sheep right? a: the chocolate one now maybe? at least it has no cream, and some nuts a: didnt we say the white suv was more solid than grey and red? with respect to the human evaluation, what is notable is that the model-based res were generated based on a limited context window that included only the seven previous messages. the ground truth mentions, logically, were produced while the speakers had access to and knowledge of the entire dialogue history, the linguistic as well as the extralinguistic context. by evaluating using the unfolding dialogues in their entirety instead of limiting these to a rolling window of eight messages, we biased the human evaluation slightly towards the ground truth; this was a conscious design choice as not doing so would unfavorably bias results towards the models instead. in light of this, our results are arguably even more promising. furthermore, rather than incorporating the entire visual context, our reg model was only conditioned on an image of the referent when generating an re. as a result, the generated res were generally descriptive, but not necessarily discriminative. although we have now relied on our rec model to filter out such candidates, we suggest future research to consider the possibility of improving the generated candidates in terms of their discriminatory power by including the visual context as part of the input to the reg model. related, we suggest testing alternative decoding strategies, for example those that are sampling-based or, perhaps more appropriate, ones that aim to be discriminative (e.g., @xcite .