Calibrate your listeners! Robust communication-based training for pragmatic speakers (2021)

Abstract: To be good conversational partners, natural language processing (NLP) systems should be trained to produce contextually useful utterances. Prior work has investigated training NLP systems with communication-based objectives, where a neural listener stands in as a communication partner. However, these systems commonly suffer from semantic drift where the learned language diverges radically from natural language. We propose a method that uses a population of neural listeners to regularize speaker training. We first show that language drift originates from the poor uncertainty calibration of a neural listener, which makes high-certainty predictions on novel sentences. We explore ensemble- and dropout-based populations of listeners and find that the former results in better uncertainty quantification. We evaluate both population-based objectives on reference games, and show that the ensemble method with better calibration enables the speaker to generate pragmatic utterances while scaling to a large vocabulary and generalizing to new games and listeners.

## introduction to be good conversational partners, language models (lms) should learn to produce fluent utterances that serve the needs of their listeners @xcite . however, they are often trained to capture the statistical, rather than communicative, properties of language via supervised learning @xcite . consider the reference game @xcite in figure 1 , where the goal of the speaker is to disambiguate a target image from distractor images. a naive lm may generate the literal description "the red shape"; this is semantically accurate, but fails to disambiguate the referent among its context. a more pragmatically useful description is "the red square". prior work in training communicative nlp systems has used explicit models of pragmatics @xcite to fine-tune traditional lms @xcite @xcite , or has trained lms with external reward signals that indicate the contextual utility of an utterance. human preferences are an ideal source of supervision for the latter approach @xcite , but are expensive to collect. one promising avenue is communication-based training (or self play), where a speaker learns to communicate with a learned model of a listener @xcite . however, this approach commonly suffers from the problem of semantic drift. our use of "semantic drift" follows the communication-based training literature @xcite : the speaker produces utterances that satisfy the listener but diverge from the semantics and conventions of natural language. it can be difficult to diagnose the nature of semantic drift in complex, open domain models. in this paper, we isolate one form of semantic drift that occurs when scaling communication-based training to larger domains: by increasing the size of the vocabulary available to the speaker and listener, the speaker fails to generalize to new listeners. we identify miscalibrated listener uncertainty as the source of this problem: the listener is highly confident in the interpretation of utterances outside its training domain, so the speaker overfits and produces nonsensical utterances that fail to generalize. we propose to correct the calibration problem by using populations of listeners, drawing from both computational and cognitive science work that suggests deficiencies in just training with a single listener @xcite @xcite and improvements in emergent communication protocols by regularization effects of ensembled models @xcite . we find that ensemble-based populations of listeners are better calibrated, helping speakers avoid semantic drift and generalize to new games and listeners. ## approach we study the problem of learning a pragmatic speaker for reference games with the shapeworld @xcite dataset. a reference game (i, t) consists of n images @xmath0, . . . , i n ) and a target image i t , with the index t known only to the speaker. the speaker s must produce an utterance u which allows the listener l to identify the target t given the images. formally, given u, the listener l is a distribution over possible targets in a reference game: we adopt the setting of white et al. (2020): we first train listener models on separate splits of shapeworld reference games, then train pragmatic speakers with the fixed listener models as the communication objective. the listeners and speakers do not share data splits among each other. training listener(s) are used as the internal listener model for the speaker in equation 3, and other validation listeners are held-out for speaker evaluation. all the speakers and listeners are trained on 15000 randomly generated reference games with a single target and 2 distractor images. more details on the model architecture, training, data, and the supplementary code can be found in appendices a and b. in the results reveal two noticeable discrepancies between the small and large vocabulary settings. first, speakers trained over the large vocabulary space can successfully communicate with the training listener on new reference games -that is, they generalize to new contexts. however, they are unable to generalize to held-out listeners and achieve only random chance accuracy (33%). this pattern of overfitting to the training listener despite generalizing to new contexts with the training listener indicates that the speaker suffers a form of semantic drift @xcite . this semantic drift is shown most clearly in the second discrepancy: speakers trained with large vocabularies rarely use tokens related to shapes and colors (i.e. the domain-relevant tokens), whereas those trained with small vocabularies do so (by design). upon qualitative examination of the speaker's utterances ( ## diagnosing semantic drift we hypothesize that the observed semantic drift arises from poor uncertainty calibration in the neural listener, which the speaker then overfits to. if this is correct, then populations of listeners-which have been shown to be better calibrated ( beluch et al., 2018; li and bowling, 2019 we compare the uncertainty measurements of three different internal listener implementations. one baseline is single-l 0 , which approximates the internal listener l with a single neural listener, i.e. exactly the setup in section 3 where existing work uses a single neural listener for communicationbased training. the second is an ensemble-based population of n neural listeners; this is equiva- in equation 3 where each listener l j is initialized randomly. they are trained on different data splits and different random seeds, and each achieves at least 92% accuracy on training and validation. while all listeners generalize to new in-distribution utterances, they generalize differently out-of-distribution. it is these disagreements among the ensemble that improve calibration compared to a single neural network like single-l 0 . the third is the dropout-based population, where we use a single listener to approximate a population of n listeners via mc-dropout @xcite ; this is equivalent to substituting l(t|u) = 1 n n @xmath1, d j ) in equation 3 where d j is a randomly sampled dropout mask. in our experiments, we use a dropout rate of @xmath2.1 but found no differences with using larger rates while maintaining high listener accuracy; please refer to appendix b for more details on the dropout implementation. intuitively, the ideal listener should be uncertain about utterances that are different from their training domain. to measure this, we evaluate listener uncertainty scores on utterances with varying degrees of overlap with their training domain. specifically, for high-overlap utterances, we sample messages directly from the ground truth target distractors method gt red shape single l0 suppressed suppressed mix suppressed pagan mix mix pagan mix dropout suppressed suppressed suppressedops suppressed mix pagan imprison mix ensemble rect rect rect rect target distractors method gt gray shape single l0 eseese fdawhat macedichickarios macedichick dropout ariosichickarios fda fdaarios brainsichick maced ensemble gray gray gray shapeworld language. for medium-overlap utterances, we sample language from a well-calibrated neural speaker ( ## countering semantic drift following our hypothesis, a well-calibrated listener should lead the speaker to generalize to new listeners and new contexts while reasoning pragmatically over a large vocabulary. we show that this is indeed the case in the following two experiments. first, we show that with increasing population size n, the ensemble-based population objective closes the accuracy gap between training and validation listeners, while still generalizing to new reference games (figure 3 ). this is not the case for the dropout listener with increasing the number of passes n. these results are averaged on 10 seeds, for n of {1, 10, 20, 30}. with ensemblebased population @xmath3, we found that the speaker token overlap increased to 48.4 ± 12.3%; this is a much larger token overlap than the original speaker trained on single-l0 (1.0 ± 0.6 from to explore the semantic content of utterance distributions, we investigate the topic overlap between different speakers and the ground truth utterances. we find that speakers trained with well-calibrated listeners produce messages which are semantically similar to the ground truth utterances, according to distances in glove @xcite embedding space ( ## conclusion the true objective for language use is communication. our work highlights the importance of wellcalibrated listeners in communication-based training. we show that it's important to understand the properties of speakers optimized to communicate with different listeners. we have found that naive communication-based training over unconstrained vocabularies is subject to a pernicious form of semantic drift that arises from poorly calibrated listener models. the overconfidence of listeners is exploited by the speaker during training and, as a result, speakers acquire niche linguistic properties, like conventions that fail to generalize to other listeners @xcite . by contrast, an ensemble of listeners shows better calibration. speakers trained to communicate with these listeners avoid semantic drift, generalizing to new games and new listeners. future research on communication-based training for language models will thus benefit from listener ensembles or other methods for training listeners with properly calibrated uncertainty.