Know your audience: specializing grounded language models with listener subtraction (2023)

Abstract: Effective communication requires adapting to the idiosyncrasies of each communicative context—such as the common ground shared with each partner. Humans demonstrate this ability to specialize to their audience in many contexts, such as the popular game Dixit. We take inspiration from Dixit to formulate a multi-agent image reference game where a (trained) speaker model is rewarded for describing a target image such that one (pretrained) listener model can correctly identify it among distractors, but another listener cannot. To adapt, the speaker must exploit differences in the knowledge it shares with the different listeners. We show that finetuning an attention-based adapter between a CLIP vision encoder and a large language model in this contrastive, multi-agent setting gives rise to context-dependent natural language specialization from rewards only, without direct supervision. Through controlled experiments, we show that training a speaker with two listeners that perceive differently, using our method, allows the speaker to adapt to the idiosyncracies of the listeners. Furthermore, we show zero-shot transfer of the specialization to real-world data. Our experiments demonstrate a method for specializing grounded language models without direct supervision and highlight the interesting research challenges posed by complex multi-agent communication.

## introduction human language use is communicative, and thus involves substantial adaptation to each conversa-tional partner and context @xcite @xcite @xcite . we can adapt our speech to complex social settings, with multiple partners @xcite and competing constraints, such as politeness vs. explicitness @xcite . while current language and captioning models can increasingly imitate human language and scene descriptions @xcite @xcite @xcite , they generally are not explicitly adapted to a particular partner or to satisfy multiple constraints. common approaches to specializing the language generated by these models fall into two categories: finetuning on a supervised dataset (e.g. @xcite , which requires task-specific labeled examples, or prompt engineering, which is often brittle and may still require task-specific examples @xcite . here, we take inspiration from how humans adapt their language to their audience to offer an alternative approach to specializing grounded language models without direct supervision. our approach to specialization is inspired by the popular multi-player game dixit @xcite , which rewards each player's ability to adapt their communication to a particular audience. in a round of dixit, the "speaker" describes a chosen image in language. then, the rest of the players (two or more "listeners") attempt to identify the target image from a pool of distractors. importantly, the speaker is rewarded when some but not all listeners correctly identify the image. this leads to creative captions that target the common ground between a speaker and some (but not all) of the listeners (see example in figure 1a ). for more details, we refer readers to @xcite , who introduce dixit as a grand challenge for ai. using this inspiration, we formulate a setting where grounded language models can specialize their language without direct supervision, from rewards only. we build on prior work in emergent communication @xcite , and adapt a pretrained captioning model (the "speaker") by finetuning a small fraction of its parameters to maximize a dixit-like reward-this minimal adaptation helps reduce language drift @xcite . we go beyond prior work by investigating a complex, not fully cooperative, setting (figure 1b ). the speaker's goal is to communicate information to some listeners, but not others, by exploiting differences in listeners' idiosyncratic "personalities". we instantiate this setting by having a speaker model communicate with multiple (frozen) contrastive listener models differing along perceptual axes. we design careful experiments, with datasets, metrics, and controls to ensure quantifiable assessment of language specialization. we show that training a speaker with a pair of listeners in this dixit-inspired setting can lead to diverse specialization of natural language, with minimal language drift, and without direct supervision. from rewards, the speaker identifies and learns to cue to the difference between two listeners, which we call "listener subtraction". we show this adaptation across many pairs of perceptually-differing listeners, and various datasets (section 4.1, 4.2). for example, when trained with one listener which sees color, and another which sees only grayscale, the speaker learns to exclusively use color wordsit exploits the specialized information it shares with the first listener. furthermore, the speaker exhibits some zero-shot transfer of its language specialization from artificial to realistic datasets (section 4.3). to our knowledge, our work is the first to consider natural language communication in a grounded, multi-listener setting, without direct supervision. ## discussion we have demonstrated a method for specializing a grounded language model without direct supervision, by finetuning a small fraction of its parameters in a complex multi-agent setting. we have shown that this approach enables diverse types of specialization with minimal language drift, and have identified the essential aspects of this approach. we believe that our work offers a novel perspective on adapting models to new settings without supervised data. our approach to specializing a grounded language model rewards the speaker for using differences in the information it shares with two listener models. this formulation allows for flexible adap-tation of models to complex tasks. for example, our approach could potentially be used to diagnose biases in one contrastive listener model by comparing to another (e.g., trained on a different dataset), an area of considerable interest @xcite . a speaker optimized for the difference between two contrastive models could identify biases present in one listener, without the need for manually curated word lists. additionally, training against listeners of different skill could potentially improve language quality without relying on costly human annotations (e.g. @xcite . long term, we hope our work contributes to personalized ai. humans prefer interacting with others who share rare preferences @xcite , so an ai that personalizes its language to users would likely be preferred to one that uses standardized language. personalization could improve other kinds of communication too, like explanationse.g., adapting feedback to a particular student's idiosyncratic knowledge (cf. @xcite . our approach enables such adaptation through interaction, without explicit language supervision. towards dixit finally, while our experimental setup is inspired by dixit game, we only make the first steps towards a model that can play the full version of dixit effectively with humans-something that is an appropriate grand challenge for ai, as @xcite suggest. we highlight some of the next steps below. the present work focuses on perceptual differences, while human play often relies on differences in conceptual knowledge. finetuning listeners on different datasets (e.g., scenes from harry potter movies as in figure 1a ) might allow such conceptual differences to emerge. a challenge would be using datasets where this conceptual difference is relevant (dixit achieves this with abstract images). this could potentially be overcome by generating synthetic data using modern text-to-image models (e.g. @xcite ). given the robustness to different listener pairs already exhibited, we believe this is a promising future direction. moreover, human players adapt their language from a few interactions, while our speaker trains for thousands of iterations. one way to overcome this limitation would be to combine our work with concurrent progress in visual language models. @xcite introduce a model, flamingo, which can rapidly adapt to different visual language tasks in a few-shot setting. using such a high-quality base model as the speaker might enable reward-driven adaptation from just a few interactions. we hope that this work will inspire further research on settings like dixit, and help to enhance the capabilities of grounded language models and communicative agents more broadly.