Continual Adaptation for Efficient Machine Communication (2020)

Abstract: To communicate with new partners in new contexts, humans rapidly form new linguistic conventions. Recent neural language models are able to comprehend and produce the existing conventions present in their training data, but are not able to flexibly and interactively adapt those conventions on the fly as humans do. We introduce an interactive repeated reference task as a benchmark for models of adaptation in communication and propose a regularized continual learning framework that allows an artificial agent initialized with a generic language model to more accurately and efficiently communicate with a partner over time. We evaluate this framework through simulations on COCO and in real-time reference game experiments with human partners.

## introduction communication depends on shared conventions about the meanings of words @xcite , but the real-world demands of language use often require agents to go beyond fixed conventional meanings @xcite . recent work on pragmatic and context-aware models has approached this problem by equipping speaker and listener agents with the ability to explicitly reason about one another. pragmatic reasoning allows listeners to infer richer intended meanings by considering counterfactual alternatives, and allows speakers to be appropriately informative, not merely truthful @xcite @xcite @xcite . these models have largely focused on one-shot settings, where the context is the immediate visual environment. in common interactive settings, however, the relevant context for pragmatic competence also includes the history of previous interactions with the same communication partner. human in-a cat lying in the sunlight on a bed two cats are playing on a log trial 1 trial 2 ... here, we propose that the problem of ad hoc convention formation can be usefully re-formulated as an inference problem amenable to online domain adaptation. our approach is motivated by a growing body of evidence in cognitive science that humans quickly re-calibrate their expectations about how language is used by different partners @xcite . this empirical work highlights three key challenges facing a scalable adaptation approach. first, because the target data comes from intentional agents, pragmatic reasoning must be deployed throughout adaptation to strengthen inferences @xcite . second, because the data is sparse, strong adaptation risks catastrophic forgetting; yet, human speakers are able to revert to their background expectations for the next interlocutor @xcite . third, the ability to ground the meanings of later, shorter utterances (e.g. "back meds") in the use of earlier, longer utterances requires a compositional representation; otherwise the connection between the utterances is not clear @xcite . our primary contribution is an online continual learning framework for transforming pragmatic agents into adaptive agents that can be deployed in real-time interactions. this framework is shown schematically in fig. 1 : after each trial, we take a small number of gradient steps to update beliefs about the language model used by the current partner. to evaluate our framework, we first introduce a benchmark interactive repeated reference task (fig. 2 ) using contexts of natural images. in sec. 3, we introduce the three core components of our algorithm: (i) a contrastive loss objective incorporating explicit pragmatic reasoning, (ii) a kl regularization objective to prevent overfitting or catastrophic forgetting, and (iii) a data augmentation step for compositionally assigning credit to sub-utterances. in sec. 4, we report experiments demonstrating that this algorithm enables more effective communication with naive human partners over repeated interactions. finally, in sec. 5 we report a series of ablation studies showing that each component plays a necessary role, and close with a discussion of important areas for future research in sec. 6 2 related work personalizing language models. adapting or personalizing language models is a classic prob-lem of practical interest for nlp, where shifts in the data distribution are often found across test contexts @xcite @xcite . our approach draws upon the idea of dynamically fine-tuning rnns @xcite , which has successfully explained key patterns of human behavior in selfpaced reading tasks @xcite . we also draw on the regularization objectives proposed in this literatures @xcite . however, the interactive communicative setting we consider poses several distinct challenges from traditional speech recognition @xcite or text classification settings @xcite for which adaptation is typically considered. partnerspecific observations of language use are sparser, must be incorporated online, and are generated by intentional agents. ## approach we begin by recasting convention formation as an online domain adaptation problem. as in previous computational approaches to pragmatics (e.g. goodman and frank, 2016; andreas and klein, 2016), we formulate this problem as an inference about another agent. the key theoretical idea is to expand the scope of pragmatic inference from the single-utterance timescale to evidence accumulated over longer timescales of an interaction. in addition to inferring a partner's intended meaning (or interpretation) for each individual utterance, an adaptive agent pools across previous utterances to infer the distinct but stable way their partner uses language. under this inference framework, an agent must both (1) begin with background expectations about language shared across many partners, and (2) have a mechanism to rapidly learn the specific language model used by the current partner. our work assumes a conventional neural language model as the starting point and focuses on the partner-specific inference problem. in this section, we describe our repeated reference game benchmark task (3.1), review the underlying problem as it has been previously formulated in a bayesian framework (3.2), and finally describe our algorithm for adapting neural language models (3.3). ## interactive human evaluations in this section, we evaluate our model's performance in real-time interactions with human speakers. our artificial agent was paired with human partners to play a repeated reference game using images from the validation set of the coco corpus @xcite as the targets of reference. critically, we constructed contexts to create a diagnostic mismatch between the coco pre-training regime and the referential test regime. specifically, we chose contexts such that the model's accuracy -the probability of identifying the target -would be poor at the outset. to obtain appropriately challenging contexts, we used our pre-trained model's own visual encoder to find sets of highly similar images within the same category. we first extracted 256-dimensional feature vectors for each image from the final, fullyconnected layer of the encoder. we then used these features to partition the images into 100 groups using a k-means algorithm, sampled one image from each cluster, and took its 3 nearest neighbors in feature space, yielding 100 unique contexts of 4 images each. this adversarial process explicitly identified contexts that our pre-trained captioning model would be poorly equipped to distinguish. human baselines. we first investigated the baseline performance of human speakers and listeners. we recruited 108 participants (54 pairs) from amazon mechanical turk and automatically paired them into an interactive environment with a chatbox. for each pair, we sampled a context and constructed a sequence of 24 trials structured into 6 repetition blocks, where each of the 4 images appeared as the target once per block. we prevented the same target appearing twice in a row and scrambled the order of the images on each player's screen on each trial. we found that pairs of humans were highly accurate, with performance consistently near ceil-ing (fig. 3 , black lines). at the same time, their utterances grew increasingly efficient: their utterances reduced in length across repeated interaction (@xmath0.8, @xmath1.001). foot_2 ## analysis we now proceed to a series of ablation analyses that analyze the role played by each component of our approach. these analyses involve offline simulations conducted on the data we collected in the previous section. ## discussion relationship to human adaptation the theoretical ties between our approach and proposed cognitive models of human adaptation raises several questions. first, it is possible that improved performance could be driven by human speakers adapting in response to our listener agent's successes and errors rather than the other way around. while some degree of human adaptation is inevitablefor example, humans only seemed to shorten their utterances once our models' accuracy began to rise -human adaptation alone is insufficient to explain gains in accuracy. if these gains were due to human speakers gradually discovering utterances that a pre-trained (non-adapting) model could understand, we would expect some gains in the accuracy of our baseline non-adapting model over time. furthermore, we found that the handful of human speakers that dramatically changed their descriptions across rounds actually performed worse than those who adhered to consistent descriptions. with this said, the extent of adaptation in humancomputer dialogue is known to be affected by human participants' expectations about the artificial agent @xcite , potentially including expectations about whether it will be adaptive or not. bi-directional adaptation effects may be more pronounced in other dialogue settings where the human and model both speak, giving the human an opportunity to re-use utterances produced by the model. it will be important for future work to evaluate non-adaptive baselines online rather than offline, as we did, in order to observe exactly how humans respond to, or compensate for, non-adaptive agents. second, it is natural to ask how our model would perform in the speaker role with a human listener, using their (sparse) response success as feedback rather than their utterances. in ongoing work, we have found that the same approach allows a (pragmatic) model to converge to more efficient conventions in the speaker role (see appendix d in supplemental), such that the same language model can flexibly switch between speaker and listener roles with the same human partner. still, it is unlikely that this speaker model reduces in the same way as human speakers do (see supplemental fig. s1 for examples). differences may reflect additional accessibility, grammaticality, or compositionality biases in humans; direct comparisons remain an open question for cognitive science. third, scaling the principles of computationallevel bayesian cognitive models to neural networks capable of adapting to natural language in practice required several algorithmic-level innovations which are not yet plausible proposals for human cognition @xcite . while our local rehearsal mechanism may be consistent with replay mechanisms in human memory, our kl regularization mechanism implausibly requires earlier parameter values of the model to be held in memory. our data augmentation mechanism was introduced specifically to compensate for the inability of the lstm architecture to propagate the use of a referring expression to its entailments, but we expect that human language processing mechanisms achieve this effect by different means. we expect further work to refine these algorithmic components as neural language models continue to advance. relationship to language learning our work is also related to broader efforts to ground language learning and emergent communication in usage, where artificial agents are trained to use language from scratch by playing interactive reference games @xcite @xcite . rather than starting our agents from scratch, we have emphasized the need for continual, partner-specific learning even among mature language users with existing priors. this raises another question: how are these different timescales of learning related to one another? one possibility is that the need to quickly adapt one's language to new partners and contexts over short timescales may serve as a functional pressure shaping languages more broadly. recent theories in cognitive science have formalized this hypothesis in a hierarchical bayesian model @xcite . in this model, the prior θ that an agent brings into subsequent interactions is updated to reflect the overall distribution of partner-specific models θ i , thus balancing general and idiosyncratic language knowledge in a principled way. for neural language models, however, there is an apparent tension between the strong kl regularization required to prevent unwanted interference with background knowledge during partner-specific adaptation, leading to catastrophic forgetting, and the flexibility to generalize or transfer conventions to new communicative settings as required for language learning. we do not want to regularize so strongly that agents memorize conventions only applying to a single image that is completely reset after each interaction; instead, we wish to obtain a gradient of generalization across both referents and partners as a function of similarity @xcite . one promising solution to this problem, motivated by connections between hierarchical bayes and algorithms like maml @xcite @xcite , is to perform a meta-learning 'outer loop' updating the initialization θ, taking into account the regularized, partner-specific 'inner loop' of adaptation for each θ i . in principle, a meta-learning approach for neural language learning would distill abstract, shared aspects of language into a unified θ, while still allowing for rapid ad hoc conventionalization. still, cognitively plausible and scalable meta-learning algorithms remain an open area of research. limitations and future work while our evaluations were limited to a canonical cnn-rnn image captioning architecture, a key open question for future work is how our continual adaptation approach ought to be implemented for more complex, stateof-the-art architectures. one possibility, following the approach recently proposed by @xcite , is to allow context (e.g. partner identity) to control a low-rank transformation of the weight matrix such that online fine-tuning can take place in a more compact context embedding space @xcite . furthermore, while we adapted the entire parameterized rnn module end-to-end, future work should explore the effect of limiting adaption to subcomponents (e.g. only word embeddings) or expanding adaptation to additional model components such as attention weights or high-level visual representations. beyond possible consequences for engineering better adaptive models, each of these variants corresponds to a distinct cognitive hypothesis about exactly which representations are being adapted on the fly in human communication. a final area for future work is generalizing the forms of social feedback that can be used as data d i for updating representations beyond the sparse choices in a reference game. in particular, forms of repair through question-asking or other nonreferential dialogue acts may license stronger inferences about a partner's language model and allow misunderstandings to be resolved more quickly in challenging contexts @xcite @xcite . these forms of feedback may be particularly important for extending our approach beyond the benchmark task of repeated reference games to the more complex domains of real-world conversational tasks. conclusion human language use is remarkably flexible, continuously adapting to the needs of the current situation. in this paper, we introduced a challenging repeated reference game benchmark for artificial agents, which requires such adaptability to succeed. we proposed a continual learning approach allowing agents to form context-specific conventions by fine-tuning general-purpose representations. even when pre-trained models initially perform inaccurately or inefficiently, our approach allows such models to quickly adapt to their partner's language in the given context and thus become more accurate and more efficient using common ground.