Detecting Referring Expressions in Visually Grounded Dialogue with Autoregressive Language Models (2025)

Abstract: In this paper, we explore the use of a text-only, autoregressive language modeling approach for the extraction of referring expressions from visually grounded dialogue. More specifically, the aim is to investigate the extent to which the linguistic context alone can inform the detection of mentions that have a (visually perceivable) referent in the visual context of the conversation. To this end, we adapt a pretrained large language model (LLM) to perform a relatively course-grained annotation of mention spans in unfolding conversations by demarcating mention span boundaries in text via next-token prediction. Our findings indicate that even when using a moderately sized LLM, relatively small datasets, and parameter-efficient fine-tuning, a text-only approach can be effective, highlighting the relative importance of the linguistic context for this task. Nevertheless, we argue that the task represents an inherently multimodal problem and discuss limitations fundamental to unimodal approaches.

## introduction in conversation, speakers often make reference to objects, events, or concepts. words and phrases that are used with referential intent are known as referring expressions (res). effective communication hinges on the ability of the participants in the conversation to recognize these expressions and to determine what it is that they refer to, i.e., their referents. within the context of a discourse, identification of an intended referent for a given re may necessitate coreference resolution, i.e., the process of linking expressions that have the same referent. to illustrate this need, consider the following hypothetical exchange, with coreferring expressions underlined: figure 1 : visualization of the proposed mention detection method. the mention detector takes as input the most recent dialogue message, preceded by the available dialogue history, and autoregressively outputs an annotated reproduction of the last message while inserting mention span boundary tokens (the start and end of mention spans are represented @xmath0, respectively) where appropriate. excerpt shown is from a dialogue collected by @xcite . highlighted mentions in original dialogue and visual context with highlighted referent images are shown solely for illustrative purposes: the former is not available to the model at inference time, the latter neither at inference nor at training time. without access to the discourse context, "it" and "it" have indeterminate referents. by having knowledge of the prior contributions to the conversation, it is clear that both pronouns are anaphors with "schrödinger's cat" as their antecedent. the identification of res, or mentions 1 , in various types of discourse is a long-standing natural language processing (nlp) task commonly referred to as mention detection (md). simply put, when a given discourse is represented as a text document, the goal of md is to identify any and all spans of text that refer to some predetermined type of referent, such as named entities or events. in this paper, we explore the problem of md for conversation, specifically with a focus on the downstream purpose of reference resolution in visually grounded dialogue. that is, our aim is to identify the res that have a (visually perceivable) referent in the visual context of the conversation. of particular interest is the extent to which the linguistic context alone is able to inform predictions for what is arguably, inherently, a multimodal problem. in addition, we experiment with different context windows to investigate how dialogue history affects md performance. the expectation is that providing access to additional linguistic context in the form of preceding messages will generally lead to increased performance. to illustrate by example, whether the use of "that" in the exclaimed utterance "take that!" is referential or instead part of a nonreferential interjection cannot be known without additional context. in line with recent work on generative information extraction (see e.g., @xcite , we frame md in visually grounded dialogue as an autoregressive language modeling problem. more specifically, we propose to train a model to generate annotated reproductions of utterances: for a given utterance, in the process of generating a copy of the original message content, the model is expected to insert span boundary tokens indicating the start and end of mention spans, when and where appropriate. an illustration of the proposed approach is shown in figure 1 . our experiments involve the parameter-efficient fine-tuning @xcite of a large language model (llm) on annotated conversations from two different visually grounded dialogue datasets, namely a game of sorts (agos, @xcite and pho-tobook @xcite . for agos, we make use of the mention annotations from @xcite . for pb, we adopt a similar annotation protocol to manually create the required mention annotations for a subset of the dataset. 2 results of our experiments with the 8bparameter variant of llama 3.1 @xcite are promising, suggesting that the linguistic context can be relatively revealing for our purpose. note that our findings are in spite of the fact that our datasets are relatively small, our llm is relatively moderately sized, and our fine-tuning is parameter-efficient. nevertheless, we must contend with some limitations that are fundamental to unimodal approaches to multimodal problems, as well as the nature of the referential language in task-oriented dialogues. we provide additional discussion on these matters. ## background md has long been an essential component, or the central focus, of systems addressing various nlp tasks, such as named entity recognition (e.g., @xcite @xcite , event detection (e.g., @xcite , and coreference resolution (e.g., @xcite . earlier, rule-based approaches to md were frequently built atop a dependency parse of a text, and would, over time, incorporate increasingly more powerful statistical models into the pipeline (e.g., @xcite . the required sophistication of the approach generally depended on the downstream task. for coreference resolution, for example, simple heuristics leading to high recall would suffice if other parts of the system could compensate with higher precision (e.g., @xcite . interestingly, comparisons between different coreference resolution systems have often been conducted on the basis of gold, instead of predicted, mentions. this effectively side-steps md in an effort to focus on isolating the system's downstream performance. however, there tend to be notable performance gaps between these idealized and the realistic scenarios. as @xcite note, generally, the overall performance of a coreference resolution system has been contingent on the accuracy of the output from its md component. following advances in neural language modeling, approaches to md based on neural models (e.g., @xcite @xcite @xcite have gradually superseded the earlier methods. these increasingly more datadriven methods promised to do away with the need for extensive feature engineering. particularly consequential has been the adoption of general purpose, pretrained language models based on the transformer architecture @xcite , examples of which include the encoder-only bert @xcite and the decoder-only gpt @xcite . bert-based representations have been the backbone of numerous nlp systems, including those that deal with md (e.g., @xcite @xcite . of particular interest here are the autoregressive llms at the heart of most work on generative information extraction (see e.g., @xcite . various studies have shown that framing tasks involving structured predictions as autoregressive language modeling problems can be effective (e.g., @xcite @xcite . given an unstructured text, the model is trained to return, via next-token prediction, a structured representation of the input. although the feasibility of this approach has been shown for commonly used benchmarks that involve some form of md (e.g., @xcite , to the best of our knowledge, it has yet to be applied to visually grounded dialogue. in this paper, we explore to what extent we can adapt a pretrained llm via parameter-efficient fine-tuning @xcite to the task of md in visually grounded dialogue using this approach. ## experiments the language modeling experiments presented in this paper involve the fine-tuning of pretrained models on dialogues from two different, though closely related, visually grounded dialogue tasks, namely a game of sorts (agos, @xcite and photobook @xcite . we first perform cross-validation to score md performance on each dataset separately. we then assess cross-dataset transfer by training on one dataset and testing on the other. in addition, we investigate the effects of dialogue history on md performance, i.e., whether the model benefits from having access to preceding messages when making its predictions, by experimenting with different context window sizes, i.e., providing access to different numbers of preceding messages. finally, as points of comparison, we assess the md performance of a baseline based on noun phrase (np) extraction using constituency parsing, as well as that of an encoder-only llm fine-tuned for sequence labeling. ## results before reporting the results of our fine-tuning experiments, we first highlight some of the descriptive statistics reported in x p .881 .954 .934 .912 .933 .923 (.03) r .897 .842 .902 .924 .855 .884 (.03) f1 .889 .894 .918 .918 .892 .902 (.01) j .853 .823 .881 .874 .857 .858 (.02) ## discussion in this paper, we explored the potential of an approach to mention detection (md) in visually grounded dialogue based on autoregressive language modeling. results from our experiments on conversations from the visually grounded dialogue tasks a game of sorts (agos, willemsen et al., 2022) and photobook @xcite were promising, showing that a textonly approach that involves the parameter-efficient fine-tuning of llms to generate annotated reproductions of utterances can be effective. moreover, we showed that providing the models with additional context from the dialogue history-that is, any messages that preceded the utterance under consideration-generally benefits performance. although these findings were largely consistent between the competing methods presented in this work, within our experimental setup the generative approach to information extraction using the fine-tuned, decoder-only llama model was shown to consistently outperform the sequence labeling approach based on the fine-tuned, encoder-only modernbert. results from our cross-validation experiments showed that the models, on average, achieved better performance on the pb-gold than on the agos dataset. the cross-dataset transfer experiments revealed a notable performance gap between the two datasets; fine-tuning on the agos data seemed to result in the models being better able to generalize beyond their specific conversational domain than when fine-tuning on the pb-gold data. these findings suggest that agos offers a more challenging testbed when it comes to md, as it was explored in this work, than pb-gold. given that the primary focus of the pb task is the correct identification of images, participants' language use is disproportionally reserved for referential purposes. this was made apparent through a quantified characterization of the pb-gold mentions, indicating that mentions made up nearly two-thirds of the linguistic content of the dialogues. in contrast, with image identification being a secondary objective, mentions make up just shy of one-third of the linguistic content of the agos dialogues. in addition, mentions in the pb-gold dialogues are considerably longer, on average, than those in the agos dialogues. when qualitatively examining the mentions from both datasets, it becomes clear that the incidence rate of mentions that resemble image caption-like descriptions is notably higher for pb-gold than for agos. by and large, our findings suggest that agos offers its referring language use in a richer linguistic context than pb-gold, which aids the models' ability to generalize. that being said, it would be reasonable to assume that the incidence rate of mentions in these task-oriented dialogues from both datasets is high compared to that of organic, non-task-oriented conversations. conversations can go long stretches of time without the mention of a visually perceivable referent. our approach relies heavily on there being exploitable regularities in the linguistic context. the extent to which conversations with comparatively sparse mention occurrences, and that take place outside of task-oriented settings, still exhibit such actionable patterns is, as of yet, unclear. for both agos and pb-gold, the probability that a given linguistic expression (indirectly) points to a referent that is visually perceivable by at least one of the participants in the conversation is high, simply as a consequence of the situational context, as the images are the focal point of the conversations. discerning, from the linguistic context alone, whether an re has such a referent becomes far more challenging, if not impossible, when the configuration of the visual context of the conversation is less constrained, more dynamic, and cannot be anticipated ahead of time. in other words, we may still be able to extract mention candidates with a high degree of accuracy, but the number of false positives-by which we here mean any candidates that currently have no visually perceivable referents-is likely to be significantly higher; this outcome reminds of the high recall settings favored by aforementioned prior work on coreference resolution. inevitably, a general solution to the problem will require a cross-modal approach. although we make no assumptions regarding the manner of encoding, the visual information must somehow be incorporated to validate whether candidate men-tions indeed have a referent in the visual context; even when the linguistic context strongly implies the existence of such a referent, we simply cannot be certain without a review of the visual context. moreover, we are likely to see that end-to-end approaches will increasingly be favored over modular systems when it comes to addressing downstream tasks that have historically relied on some form of md, but for which md is simply a means to an end. nevertheless, we expect that md as a task in and of itself will remain relevant for niche applications for the foreseeable future. for one, it may continue to serve as a benchmark for the information extraction capabilities of models under varying conditions. perhaps more interestingly, however, are real-world applications, such as its use as an information extraction tool for corpus linguistics.