Multi-agent Communication meets Natural Language: Synergies between Functional and Structural Language Learning (2020)

Abstract: We present a method for combining multi-agent communication and traditional data-driven approaches to natural language learning, with an end goal of teaching agents to communicate with humans in natural language. Our starting point is a language model that has been trained on generic, not task-specific language data. We then place this model in a multi-agent self-play environment that generates task-specific rewards used to adapt or modulate the model, turning it into a task-conditional language model. We introduce a new way for combining the two types of learning based on the idea of reranking language model samples, and show that this method outperforms others in communicating with humans in a visual referential communication task. Finally, we present a taxonomy of different types of language drift that can occur alongside a set of measures to detect them.

## introduction in this work, we aim at making agents communicate with humans in natural language. our starting point is a language model that has been trained on generic, not task-specific language data. we then place this model in a multi-agent communication environment that generates task-specific rewards, which are used to adapt or modulate the model, making it task-conditional. we thus propose to decompose the problem of learning language use into two components: learning "what" to say based on a given situation, and learning "how" to say it. the "what" is the essence of communication that underlies our intentions and is chosen by maximizing any given utility, making it a functional, utility-driven process. on the other hand, the "how" is a surface realization of our intentions, i.e., the words we use to communicate this "what" successfully. this factorization into content planning (here, "what") and surface realization (here, "how") moves us away from end-to-end neural generation systems and is in line with traditional methods of natural language generation @xcite . more importantly, it enables us to bring together two different strands of research: traditional data-driven natural language learning and multi-agent communication. traditional approaches to natural language learning @xcite @xcite @xcite are based on inferring structural properties of language from text corpora, often in a passive regime, dissociated from communication. while this type of learning is great for learning general statistical associations between symbols (e.g., adjectives come before nouns) and even inferring semantic relations, it ignores the functional aspects of communication, i.e., the fact that people use words to coordinate with others and make things happen in the world @xcite @xcite . on the other hand, multi-agent communication research @xcite @xcite @xcite puts communication at the heart of agents' (language) learning. implemented within a multi-agent reinforcement learning setup, agents start tabula rasa and form communication protocols that maximize task rewards. while this purely utilitarian framework results in agents that successfully learn to solve the task by creating a communication protocol, these emergent communication protocols do not bear core properties of natural language. @xcite show that protocols found through emergent communication, unlike natural language, do not conform to zipf's law of abbreviation; @xcite find that communication protocols do not follow composi-tionality patterns of natural language, and @xcite find emerged protocols to be sensitive to different experimental conditions. this growing set of alarming results on emergent communication raises doubts about the use of this type of functional learning as a viable alternative to language learning. concluding that neither approach on its own is adequate for learning language use, we propose a method for combining the best of both worlds. generic language data can be used effectively as a good prior model of language, encapsulating its intrinsic structural properties, i.e., are only used for the "how" in the form of generic language models. conversely, multi-agent interactions, that provide rewards specific to the task of interest, now only need to be used for the functional learning of language use, i.e., learning the "what". 1 the contributions of this paper are as follows. first, we propose a general research program of language learning that combines two learning signals coming from multi-agent communication and traditional data-driven natural language learning techniques. we present a concrete study in the context of a referential communication game (see section 2) between a speaker and a listener, where the traditional data-driven language learning takes the form of image captioning, and the functional learning takes the form of agent self-play (see section 3). we then present a new approach for combining the two learning signals, i.e., reward-learned rerankers (see section 4), and compare this to existing approaches using a human study (see section 5). we discuss shortcomings of this program with respect to different types of language drift that can occur, and introduce a number of automatic measures to detect them (see section 6). finally, we show how such a program under oracle rewards can be a viable approach moving towards learning language use from human rewards (see section 7). ## research framing our research can be framed in the following scenario. an agent needs to perform a functional communication task in a natural language (in this work, english). however, examples of linguistic communication about this functional task are not available -the only natural language data that can be used consist of examples of generic natural language, which are not grounded in the functional task. recasting the task as a multi-agent language game provides a way to obtain a reward that judges whether an utterance elicited the correct behaviour by a listener. ## methods for learning language use we describe ways to estimate the speaker's generative model p θ s (m|u, t) for message m, conditioned on target and distractor embeddings @xmath0, u d ] and target image index @xmath1, 1}. ## speakers trained jointly with listeners we conduct three different evaluations: at test time we play against the fixed listener, human listeners and the joint listener the speaker was trained with. while fixed listener is the same for all speakers, the joint listener is speaker-specific. we report results on two splits: for the easy and difficult split we report referential success of the joint listener, and for the latter split, we also report results of the fixed and human listener. to compute referential success using human listeners, we collect 400 annotations for each speaker model. to avoid annotators adapting to modelspecific strategies, we group predictions of similar models and collect annotations in three sessions (one for each group), during which we present annotators with predictions from a model sampled from that group. ## language drift and how to detect it we show that the multi-agent communication framework is prone to language drift @xcite , i.e., when protocols diverge from human language. we present a taxonomy of different types that occur in this framework, alongside a set of automatic measures to detect it. ## discussion and limitations we presented a method for teaching agents to communicate with humans in natural language, by combining two learning signals coming from multi-agent communication and traditional data-driven natural language learning techniques, which adds on recent efforts of blending emergent communication with natural language @xcite . self-play between speakers and listeners can result in language drift, the most severe of which being pragmatic drift. since speakers and listeners are learning concurrently, they can co-adapt to pair-specific policies that deviate from the policies that humans learn. this pathological behaviour of self-play is not specific to language and extends to other policies @xcite . finally, we introduced the reward-learned reranker approach which alleviates language drift and achieves the highest human performance, by constraining the functional learning to happen on the level of utterances generated by a pre-trained language model. however, since the functional signal is not currently influencing the sampling from the language model, this will lead to poor performance when using more general language models with weaker conditioning (e.g. gpt-2 @xcite ) whose samples potentially do not fit the functional context. moving towards integrating our findings into more realistic applications of self-play, e.g., user simulation in dialogue @xcite , these shortcomings need to be addressed.