Resolving References in Visually-Grounded Dialogue via Text Generation (2023)

Abstract: Vision-language models (VLMs) have shown to be effective at image retrieval based on simple text queries, but text-image retrieval based on conversational input remains a challenge. Consequently, if we want to use VLMs for reference resolution in visually-grounded dialogue, the discourse processing capabilities of these models need to be augmented. To address this issue, we propose fine-tuning a causal large language model (LLM) to generate definite descriptions that summarize coreferential information found in the linguistic context of references. We then use a pretrained VLM to identify referents based on the generated descriptions, zero-shot. We evaluate our approach on a manually annotated dataset of visually-grounded dialogues and achieve results that, on average, exceed the performance of the baselines we compare against. Furthermore, we find that using referent descriptions based on larger context windows has the potential to yield higher returns.

## introduction visually-grounded dialogues are conversations in which participants make references to the visual world. referring in conversation is understood to be a collaborative process, with shared responsibility for ensuring the successful identification of the referent @xcite . it is not uncommon for a definite reference to be established over multiple turns, with each separate contribution unlikely to be a minimally distinguishable description of the referent. taken out of their use context, these referring expressions may be difficult, if not impossible, to resolve. consider the example dialogue in figure 1 . the underspecified description "the shiny one" leads to a clarification question, "do you mean that red one?". to resolve the expression "that red one" to its referent, we need information from earlier in the conversation to understand that "one" is a proform of "apple". without this linguistic context, the red strawberry and the red apple are equally likely referents. we can break the problem of reference resolution in visually-grounded dialogue down into three subproblems: (1) mention detection, or finding the expressions that can be grounded in the visual context ("that red one"); (2) aggregation of referentspecific information (linking "apple", "the shiny one", and "that red one"); and (3) referent identification, or the grounding of language (finding the referent that is best described by the three expressions from among a set of candidate referents). this final step requires bridging the gap between vision and language. for this purpose, we can turn to pretrained vision-language models (vlms), which have shown to be effective at zero-shot text-image retrieval when given a description of an image (e.g., @xcite @xcite . however, current vlms lack the discourse processing capabilities necessary for reference resolution in visually-grounded dialogue. although some vlms may correctly identify the red apple as the referent given the entire dialogue of figure 1 , dialogues are often vastly more complex than this hypothetical exchange. take, for instance, the dialogue in appendix a: with multiple mentions of different referents within the same utterance, such a brute-force method would immediately fail. it is clear that if we want vlms to be effective for this purpose, their discourse processing capabilities need to be augmented. to this end, we propose fine-tuning a causal large language model (llm) for the task of referent description generation. referent description generation can be regarded as a special case of referring expression generation with the goal of always generating the most complete expression possible. for a given mention, the model is trained to generate a definite description that summarizes all information that has been explicitly disclosed about the referent during a conversation. for example, for the mention "that red one" in figure 1 we would want the model to generate the description "the shiny red apple". we will refer to the fine-tuned model as the conversational referent description generator (crdg). the description generated by the crdg is then used by a pretrained vlm to identify the referent, zero-shot. our approach can be seen as an exploration of the limits of depending on linguistic context alone for generating referent descriptions, as the discourse processing and eventual grounding of the descriptions are entirely disjoint. for the experiments presented in this paper we use data from the collaborative image ranking task a game of sorts @xcite . referents are represented by separate, but visually similar images from a shared entity category. due to their largely unrestricted nature and with a focus on the collaborative referential process, the collected dialogues form a challenging test bed for visually-grounded language understanding in conversation. we manually annotate the dialogues by marking mention spans and aligning the spans with the images they denote, and provide both manually constructed and automatically derived "ground truth" referent descriptions based on our manual annotations for all marked mentions. our main contributions are as follows: visually-grounded language understanding is fundamental for conversational agents that engage in dialogue involving references to the visual world. researchers have introduced a variety of tasks that provide data for development and frameworks for evaluation of visually-grounded dialogue models. these tasks often take the form of goal-oriented, dyadic interactions but differ in terms of, for example, the visual stimuli used, e.g. abstract figures or realistic photos; the roles assigned to participants, e.g. whether symmetric or asymmetric; constraints on message content, e.g. a fixed vocabulary; and the nature of the task, e.g. navigation, identification, ranking, and multi-turn visual question answering (e.g. @xcite @xcite @xcite @xcite . it has been noted that the task configuration can significantly impact the extent to which certain dialogue phenomena, such as coreferences and clarification requests, are represented in the collected data, if at all @xcite @xcite @xcite . tasks that heavily constrain the interactions do not reflect the complex nature of dialogue to the same degree as tasks that have been designed for these phenomena to naturally emerge as part of the discourse, such as a game of sorts @xcite , which we use in this paper. the terms referring expression comprehension (e.g. @xcite , referring expression grounding (e.g. @xcite , referring expression recognition (e.g. @xcite , and reference resolution (e.g. @xcite have been used interchangeably to describe the problem of mapping the language that denotes a referent to a representation of that referent in the visual modality. prior work noted the importance of referring expressions to conversation, but often modeled the problem independent of the dialogue (e.g. @xcite @xcite . the granularity at which grounding occurs may differ between works, as the language may be mapped to bounding boxes of individual objects @xcite @xcite , objects or larger image regions represented by seg-mentation masks @xcite , or entire images altogether @xcite . to address the problem computationally, both modalities must in some way be encoded. engineered visual feature representations and simple language models such as those based on ngrams (e.g. @xcite @xcite have been mostly replaced with more powerful learned representations that embed the images and text in high-dimensional vector spaces @xcite . this has made it possible to resolve references by computing representational similarity between an encoding of the text that contains a mention and the embeddings of the candidate referents, where the candidate that has the highest matching score is assumed to be the referent @xcite . recent work on multimodal representation learning has shown that jointly embedding text and images can work at scale. trained using a contrastive objective, maximizing representational similarity between true pairings of images and text while simultaneously minimizing similarity of false pairs, vision-language models (vlms) such as clip @xcite , align @xcite , blip @xcite , and blip-2 @xcite , have shown to be effective zero-shot classifiers, outperforming the previous state-of-the-art on various benchmarks without the need for further fine-tuning on specific tasks. however, despite their noteworthy image-text matching performance based on simple text queries, these vlms lack the discourse processing capabilities required for reference resolution in visually-grounded dialogue. even a simplified example, such as shown in figure 1 , illustrates a fundamental challenge, namely that of coreference resolution. the interpretation of anaphoric pronouns, such as "it", is dependent on their antecedents. without resolving its coreferences first, identifying the referent based on the pronoun alone leads to a random guess. to improve downstream performance on discourse processing tasks involving coreference, prior work has approached the problem as one of transforming the original input based on linguistic context. this was done either via substitution, such as in @xcite where pronouns were substituted for more descriptive mentions of the same referent, or via generation, such as in @xcite where entire utterances were reconstructed in a pragmatically complete manner with coreferences and ellipses resolved. to the best of our knowledge, this approach has not yet been applied to reference resolution in visuallygrounded dialogue. most contemporary natural language processing (nlp) works use transformer-based language models @xcite . for text generation tasks, it is common to use (unidirectional) autoregressive, or causal, language models such as gpt @xcite . while processing sequences, causal language models mask the future, allowing the model to only attend to the current and previous tokens while predicting the next token. a persistent trend has been to scale up language models, both in terms of their parameter count and the size of their training datasets. these increasingly larger models, such as gpt-3 @xcite , opt @xcite , palm @xcite , and llama @xcite , have been dubbed large language models (llms). the current leading paradigm to modeling downstream nlp tasks is based on transfer learning, where a pretrained llm is fine-tuned for a specific task on a smaller, domain-specific dataset. ## method we treat visually-grounded reference resolution as a text-image retrieval task, where referents are represented by images. we leave finer-grained grounding of words and phrases to image regions or individual entities or parts thereof for future work. ## discussion we have presented an approach to visuallygrounded reference resolution that frames the discourse processing side of the task as a causal language modeling problem. by fine-tuning an llm to generate referent descriptions for marked mentions in dialogue segments from the collaborative image ranking task a game of sorts @xcite , we demonstrate the possibility of treating referent identification as a zero-shot text-image retrieval problem by using pretrained vlms for the grounding of the generated labels. as we have not in any way indicated coreferential relations in the fine-tuning training data, our results imply that certain pretrained llms, here gpt-3, may learn to resolve coreferences implicitly without the need for explicit supervision for this fundamental subtask. in this work, we have treated the processing of the discourse as entirely disjoint of the visual modality. as such, it has inherent limitations. the mentions we find in the dialogues have not been produced void of the extralinguistic context. the dialogue participants could rely on co-observed visual stimuli to help resolve otherwise ambiguous language use. from linguistic context alone, some ambiguities, such as prepositional phrase attachment, may be impossible to resolve. it is, therefore, noteworthy that the downstream zero-shot textimage retrieval performance using the generated descriptions from our unimodal approach far exceeds chance level accuracy, with the potential for results to improve further given access to the full dialogue history, as we found that the ground truth labels based on larger context windows achieve greater text-image retrieval performance. however, the results from our holistic human evaluation support the notion that a multimodal approach should ultimately prove even more effective. we found that a decrease in text generation metric scores did not necessarily indicate a similar decrease in text-image retrieval performance, suggesting that the generated descriptions captured sufficiently discriminative information about the referents and achieved similar grounding accuracy despite not approximating the ground truth labels to the same extent. it is also important to note that mentions may not have a single, canonical ground truth referent description due to lexical and syntactic variations between referring attempts. despite the relatively small size of the dataset collected by @xcite , we were still able to fine-tune gpt-3 to perform the task with greater accuracy than the baselines, which speaks to the sample efficiency of (certain) pretrained llms. in comparison, we find that the much smaller gpt-2 is prone to intrusions from the fine-tuning training data and more often fails to resolve the coreferences correctly. although the complexity of the discourse warrants the use of more powerful models, it is, nevertheless, likely that any llm used for the task would benefit from a larger fine-tuning dataset. related, benchmarking performance on other visually-grounded dialogue tasks would provide insights into the generalizability of the method. in addition to pursuing a multimodal approach, finer-grained grounding, and evaluating our method on other datasets, possible avenues for future work include expanding the annotations to include coreferential relations other than identity relations, addressing multi-image referents, and unifying the method with a mention proposal system.