Grounding Language in Multi-Perspective Referential Communication (2024)

Abstract: We introduce a task and dataset for referring expression generation and comprehension in multi-agent embodied environments.In this task, two agents in a shared scene must take into account one another’s visual perspective, which may be different from their own, to both produce and understand references to objects in a scene and the spatial relations between them.We collect a dataset of 2,970 human-written referring expressions, each paired with human comprehension judgments, and evaluate the performance of automated models as speakers and listeners paired with human partners, finding that model performance in both reference generation and comprehension lags behind that of pairs of human agents.Finally, we experiment training an open-weight speaker model with evidence of communicative success when paired with a listener, resulting in an improvement from 58.9 to 69.3% in communicative success and even outperforming the strongest proprietary model.

## introduction language agents embodied in situated interactions alongside human users must be able to reason jointly about the space they occupy, the language they encounter, and their human partner's perception. for example, consider a home assistant robot that is assisting its human user in finding their lost keys. this system must take into account its previous and current observations of the space, as well as estimate what the user's current perspective is like in the shared environment. if the system generates a description of the keys' location that the user clearly and unambiguously understands, they have achieved communicative success. figure 1 shows an example of such a communicative task, where one person describes the location of an object to another person, whose view differs from their own. to correctly resolve and generate references to the surrounding environment, both the figure 1 : example scene from our environment and dataset. the center image shows the speaker on the left and the listener on the right with their respective fields of view @xcite . the speaker refers to the target object, distinguished by its blue color, and the listener selects the candidate referent they believe is described by the speaker's description, without access to its distinct color. we study human-human and human-agent referential communication in photorealistic 3d environments, introducing a platform that supports generating task instances with varying levels of difficulty. in contrast to most prior work on referring expression generation and comprehension, we focus on the setting where both agents are physically embodied in a scene but with different perspectives of the scene. we collect a dataset of 2,970 human-written referring expressions grounded in 1,485 generated scenes. we evaluate several recent vision-and-language models on the tasks of referring expression generation and comprehen-sion, including general instruction-tuned visionlanguage models, models designed for fine-grained vision-language processing, and a modular visionand-language reasoning system. when interpreting human-written referring expressions, the finegrained ferret model @xcite performs the best, successfully identifying 69.2% of intended referents. using human listeners, we find that the proprietary gpt-4o produces referring expressions that correctly identify the intended target referent for 64.9% of scenes, while the open-weight llava-1.5 @xcite is only successful for 55.7% of scenes. compared to the average human-human success rate of 87.6%, all models lag far behind humans when both generating and comprehending referring expressions. analyzing the language used by both automated and human speakers reveals significant differences in referential strategies; for example, human speakers use themselves or the listener agent as reference points much more frequently than automated models, which mostly rely on other objects in the scene. our scene-generation platform supports controlling two levels of task difficulty. first, it supports modifying the relative orientation of the agents. second, we train a referent placement policy to minimize communicative success between two automated agents. for scenes generated using this policy, we see a significant decrease in communicative success across nearly all agent combinations. finally, we fine-tune our weaker speaker model, llava-1.5 using data collected during deployment with both human and automated listeners. during learning, we first sample referring expressions from the speaker model, convert empirical observations of language interpretation by a listener into training examples @xcite , then apply proximal policy optimization to update model parameters on this data. we compare our fine-tuned models with gpt-4o, llava-1.5, and human speakers. with a single round of training and fewer than 200 sampled referring expressions, we see significant improvements in llava-1.5's ability to generate accurate referring expressions, with rates of communicative success with a human listener improving from 58.9 to up to 69.3, outperforming even the originally-stronger gpt-4o speaker. this demonstrates the strengths of learning from interaction to improve communicative success in multi-perspective referential communication. our contributions are as follows: 1. a platform for generating 3d scenes that encompass a two-player referential communication game, enabling the study of multi-perspective referring expression generation and comprehension (section 2). this platform supports controlling task difficulty through the placement of agents and referents. 2. a new dataset of comprising 27,504 sampled scenes, and 2,970 human-written referring expressions grounded in 1,485 generated scenes (section 3.1). 3. a comprehensive analysis of human-and model-written referring expressions, and benchmarking and analysis of different vision and language models on their communicative success (sections 3.2 and 4). 4. an approach for improving an open-source vision-language model on reference generation by learning from communicative success in interaction with human listener agents (section 5). our code, models, and dataset are released under an open-source license upon publication at the following url: https: //github.com/zinengtang/mulagentref . ## task and environment we study the task of embodied referential communication, where two agents coordinate their attention in a shared scene using referring expressions. to this end, we design a platform that for generating photorealistic 3d scenes that support this task at varying levels of difficulty. ## experimental setup we use our scene generation platform to evaluate embodied, multi-perspective referential communication with pairs of agents including humans and automated models. ## results we experiment with four configurations of agent dyads, combining humans and automated speakers and listeners. ## learning from communicative success we propose to further train our speaker model from learning signals acquired during referential communication. the basic premise that motivates this approach is that empirical observations of language interpretation provides evidence of utterance meaning, regardless of speaker intent @xcite . for instance, if the listener selects a different referent than the intended target, this indicates the speaker's referring expression describes (or at the very least, better describes) the chosen referent, even if the generated expression fails to describe the intended referent. in contrast to prior work that proposes methods that learn from communicative success (or failure) @xcite , we additionally explore the use of preference-based learning signals that explicitly pair the intended and chosen targets in case of communicative failure. learning. during training, we collect a dataset of m examples @xmath0, x (i) , t(i) ) m @xmath1, each consisting of a generated scene s (including the target referent index t), referring expression x ∼ p s (o s , r, t; θ) sampled from a pre-trained speaker and the referent t ∼ p l (o l , r, x; ϕ) selected by a listener. we fine-tune speaker parameters θ using our collected dataset of examples d. we experiment with four methods for using the collected data: (a) contrastive learning @xcite , (b) learning from successes only (lso), (c) creating positive examples from every example (pos. only), and (d) pairwise preference learning (ppl). in contrastive learning, for examples where t ̸ = t, we apply a contrastive objective to jointly maximize the probability of x given the chosen referent t and minimize the probability of x given the intended referent t. for all other methods, we use offline proximal policy optimization (ppo; @xcite , adjusting only the reward function. when learning from successes only, examples receive a reward of +1 when @xmath2. to create positive examples from every example, we assign a positive reward of +1 to each utterance x paired with the listener's selection t, which may or may not be equivalent to t. in pairwise preference learning, we take advantage of the fact that, especially in light of communicative failure, we can assume that the referring expression better describes the listener's guess than the speaker's intended referent. we formalize this with a reward function that maximizes the difference between the likelihoods of the speaker's referring expression x describing the listener's chosen target t versus the intended target t: in cases where @xmath3, the assigned reward is +1. finally, we also experiment with imitation learning, where we acquire human-written references. for each reference, we acquire three human listener selections. for each selection, we directly finetune the speaker model parameters to maximize the probability of the human reference conditioned on the scene and listener selection. experimental setup. we use the initial speaker model, pre-trained llava-1.5 @xcite , to generate referring expressions for 200 scenes sampled from the training split. we experiment with learning from both human and automated listener agents. we hypothesize that human listeners will provide higher-quality feedback in the form of referent selections than the automated listener model, given a human listener's superior languageunderstanding capability. however, using an automated listener is less costly, as it requires collecting no additional human data. for our automated listener, we also use pre-trained llava-1.5. we collect a single guess per referring expression from our automated listener, and three human listener guesses. this results in two datasets: d a containing 200 examples of automated listener selections, and d h containing 600 examples of human selections. both datasets contain the same 200 sampled speaker references. training results in eight models trained on model-generated references: for each of the training objectives (contrastive, lso, pos. only, and ppl), we learn from automated and human listener selections (d a and d h ). for the same 200 scenes, we also acquire one human-written referring expression and 3 listener selections for imitation learning. for evaluation, we acquire three human listener selections for generated referring expressions in a randomly-sampled but representative subset 195 scenes from the validation set. for the bestperforming and baseline models, we also evaluate on our test set of 1,375 scenes. results. manual analysis on the validation examples reveals that after training, the model generates fewer genuinely ambiguous descriptions (43.6 to 36.0% of analyzed descriptions), and shifts from a referential strategy that increasingly refers to the listener (3.2 to 20.6%) or speaker (8.5 to 21.3%) perspectives. we also analyze how training influences sentence length: prior to training, llava-1.5 produces lengthy descriptions at an average length of 61.1 tokens. for all training objectives, the fine-tuned model generates shorter expressions than the initial model. however, only after applying ppl-based learning does the sentence length decrease close to lengths of human references, without training on any human references. ## related work the meanings of relative spatial terms are highly dependent on the situated environment: the items participating in the relation and their intrinsic parts and affordances @xcite ; the relative perspectives of participants in an embodied scene @xcite ; and within-interaction conventions formed during multi-turn embodied dialogue @xcite , among other factors. in this work, we focus on the influence of relative perspective between multiple on the use of spatial language. production and comprehension of referring expressions has been studied in human-human dialogue @xcite @xcite , and in interactions between human and automated language users @xcite @xcite . however, most work has focused on disembodied referential communication, where agents tasked with communicating about sets of stimuli (hawkins et al., 2017; 9 using a paired t-test, we find @xmath4.05 when comparing lso and ppl for both fine-tuning dataset and @xmath5.05 when comparing pos. only and ppl. @xcite , or where agents are not physically situated within an environment @xcite . the problem of situated language grounding in multi-agent settings reflects an increasingly popular real-world scenario of embodied agents. in studies where interaction participants are both embodied with different visual perspectives on the same scene, they must either be literally physically embodied in a single scene @xcite , or are placed in synthetic environments @xcite . a small number of existing works have trained language-generation models using evidence of communicative success in interaction with another agent. for example, @xcite train an instruction-generating agent by observing humans follow generated instructions, and @xcite use signals from reference games with automated listeners to improve a speaker model. our work takes inspiration from the latter to improve our speaker model using referent selections from an automated listener; however, we explore a preferencebased objective that explicitly pairs the intended and empirically chosen referents. ## conclusion we study multi-agent referential communication in situated interactions. in this setting, a speaker and a listener are both embodied in a shared scene, but are placed in different locations, with different views of the scene. we design a platform that supports generation of photorealistic 3d scenes, with control for difficulty of the referential task. we evaluate both humans and automated agents as speakers and listeners in this task. while human-human dyads are successful at coordinating on a referent around 88.4% of the time, automated models fall far behind when used both as speakers and as listeners. however, we can substantially improve the performance of an open-weight speaker model by training it with evidence of communicative success in referential communication with both automated and human listeners. our findings suggest that despite the increasing relevance of multi-agent situated interactions between humans and automated agents, there is significant headroom for applying models that jointly process language and visual perception in this setting. however, they also show the promise of training such agents in interaction with people.