Speaking the Language of Your Listener: Audience-Aware Adaptation via Plug-and-Play Theory of Mind (2023)

Abstract: Dialogue participants may have varying levels of knowledge about the topic under discussion. In such cases, it is essential for speakers to adapt their utterances by taking their audience into account. Yet, it is an open question how such adaptation can be modelled in computational agents. In this paper, we model a visually grounded referential game between a knowledgeable speaker and a listener with more limited visual and linguistic experience. Inspired by psycholinguistic theories, we endow our speaker with the ability to adapt its referring expressions via a simulation module that monitors the effectiveness of planned utterances from the listener’s perspective. We propose an adaptation mechanism building on plug-and-play approaches to controlled language generation, where utterance generation is steered on the fly by the simulator without finetuning the speaker’s underlying language model. Our results and analyses show that our approach is effective: the speaker’s utterances become closer to the listener’s domain of expertise, which leads to higher communicative success.

## introduction speakers tend to adapt their language use to the perceived knowledge, information, and linguistic abilities of their interlocutors @xcite @xcite . when adults speak with children, for example, they use simplified expressions to ensure children are able to understand @xcite ; when computational linguists give a talk at a cognitive science conference, they (hopefully) avoid making extensive use of nlp jargon, as that would prevent their audience from following through the presentation. successful adaptation to the conceptual knowledge of conversational partners requires the ability to represent and reason about others' mental states * shared first authorship. ( @xcite , a socio-cognitive ability typically referred to as theory of mind (tom; @xcite . yet, speakers do not always resort to explicitly modelling the knowledge of their dialogue partner: due to different cognitive costs and pressures, they sometimes plan their utterances egocentrically, i.e., only taking into account their own knowledge and abilities @xcite . in this paper, we model a communicative situation where the interlocutors have asymmetric language abilities: a proficient speaker interacts with a listener characterised by limited semantic knowledge to complete a reference game, as illustrated in fig. 1 . our goal is to mimic a scenario in which, for example, a high school physics professor can make complex atomic models understandable to young students by using terminology familiar to them, such as culinary terminology to explain thomson's 'plum pudding model'. we focus on the speaker's referring expression generation (reg; @xcite in a multimodal dialogue setting and use reg models equipped with visual perception to generate discriminative image descriptions within a set of related image candidates. several psycholinguistic theories have proposed that language production is interwoven with comprehension via 'forward prediction'-i.e., producing an utterance involves predicting how a comprehender would understand it (e.g., @xcite . inspired by this idea, we equip our speaker model with a simulator, i.e., a module that 'simulates' whether a listener would be able to identify the target referent. based on this predicted behaviour (i.e., the expected effect of the planned utterance), the simulator modifies the generation plan on the fly to increase communicative success. these are the main contributions of our study: 1 ## problem formulation we provide an abstract overview of the problem we address and our approach. details on the data and the experimental pipeline are given in §4 and §5. scenario our setup is a classic referential game: two artificial agents, a speaker and a listener, share a visual context involving multiple images. the speaker produces an utterance to refer to one of the images (the target) and the listener attempts to identify the referent given that utterance. in particular, we model a scenario with knowledge asymmetry, where the speaker is more knowledgeable than the listener. we hypothesise that, in such a setup, for communication to be successful, the speaker will need to adapt its utterances to the listener's representational space and language. to make this possible, we endow the speaker with a simulation module and an adaptation mechanism. simulation we provide the speaker with a module that simulates how a listener would process a planned utterance. we assume that, by having interacted with listeners in the past, the speaker has learned a model of certain listener types (e.g., a prototypical idea of what a 3-year-old would understand). we operationalise this by pretraining several instances of the simulator, one per listener type, to predict how a listener is likely to resolve a referring utterance. we compare three settings: adaptation rather than finetuning the speaker's language model, we exploit the pretrained simulators to control utterance generation on the fly via a monitoring loop. the simulator checks whether planned utterances would be effective; if that is not the case, a loss is backpropagated to update the initial hidden state of the speaker's decoder and a new utterance is generated. our hypothesis is that such a mechanism will lead to referring utterances that are adapted to the listener's knowledge. ## data as a basis for our experiments, we use the pho-tobook dataset (pb; @xcite , a collection of task-oriented visually grounded english dialogues between pairs of participants who communicate via written chat. in a photobook game, two participants see their own private sets ('photobooks') of real-life images belonging to the same visual domain. the goal of the interaction is for them to find out which images they have in common. this elicits referring utterances such as "i have a little boy holding a phone to a teddy bear", where participants refer to an image that their dialogue partner needs to identify among six similar images. our focus is on the generation of such referring utterances, leaving aside the dialogue context for simplicity. we use the dataset of referring utterances automatically extracted from the pb dialogues by @xcite , which includes 41,340 utterances paired with their target image and the other five images in the visual context. we choose photobook because its visual contexts consist of realistic images and feature multiple challenging distractors, all selected from the visual domain of the target image. the original images are taken from the microsoft coco dataset @xcite and belong to 30 different visual domains (e.g., 'person-umbrella', 'car-motorcycle'). to model speaker adaptation to different semantic domains, we split the dataset of pb referring utterances according to the visual domain of each game. we cluster the image domains as a function of the similarity between their vocabulary vectors, constructed by counting word frequencies in the referring utterances belonging to a given domain. we obtain a set of 5 macro-domains (appliances, food, indoor, outdoor, vehicles), selected so that the domain vocabularies have minimal overlap. for each cluster of visual domains, we extract the corresponding referring utterance and visual context. we then randomly split these into training (70%), validation (15%), and test set (15%). we also merge the 5 domain-specific datasets into an 'all-domains' dataset to be used to train domain-general models as described in §5. see summary in ## experimental pipeline as described in §3, our experimental pipeline includes two agents-a speaker and a listenerimplemented as a generative language model instantiating the speaker, a discriminative model instantiating the listener, and a third model, a simulator used by the speaker to assess the forward effect of its planned utterance on the listener. the language model and the discriminator model are adapted from those by @xcite , and the simulator model is built on the discriminator's ## audience-aware adaptation in our framework, adaptation takes place at inference time building on our pretrained, frozen models for the language model, the discriminators and simulators described in §5. we first explain our adaptation mechanism ( §6.1) and then report the results obtained ( §6.2). ## analysis our experiments show that simulation-based adaptation leads to more successful communication. in this section, we analyse the speaker model and its generated utterances to understand which neural processing mechanisms and which production strategies are behind our main results. ## conclusion we focused on a standard reference game-a speaker produces an utterance, and a listener uses it target:food. listener domain:indoor gold (not adapted): green salad with a person holding up a portion with fork? generated (not adapted): i have one more maybe round you think that has a lime green shaped greens, a salad? adapted: must bookshelves in the salad? target:outdoor. listener domain:food gold (not adapted): i have the pink food truck again ... white shirt lady generated (not adapted): girl at black phone, red truck, brown hair, pink adapted: pink donuts for simplicity, we only show the target images and not the whole visual contexts. we report the final adapted utterances when the adaptation mechanism stops because the simulator predicts that the listener will select the correct image. to pick the referent from a visual context. however, our setup is asymmetric-the speaker has general semantic knowledge, while the listener has little knowledge of all domains but one (e.g., food). such a setting is a perfect scenario for studying adaptation, i.e., the common process in human communication by which a speaker tunes its language to that of a listener to achieve communicative success. we modeled this mechanism using a plug-andplay approach to controllable text generation: the speaker's output is conditioned on the effect of the planned utterance on the listener, as predicted by an internal simulator. our results show that speaking the language of a listener increases communicative success. through adaptation, the speaker's language becomes less tied to the input domain and more tied to the listener's vocabulary, revealing that audience-aware adaptation can be realized without irreversible changes to generation models. our approach and findings pave the way for pragmatic models that can account for different communicative scenarios. future work may study adaptation to other dimensions such as age group or sociocultural background. moreover, adaptation could be explored in multiple 'directions'-in our setup, only the speaker adapts. we also simplify the setup by abstracting away the online process that leads to the simulation of the listeners. it would be beneficial to allow the simulators to learn to predict listener behaviour during interaction in an online manner. finally, our approach could be applied to other and possibly more complex communicative tasks, perhaps in conjunction with a mechanism leveraging human feedback via reinforcement learning.