Co-evolution of language and agents in referential games (2021)

Abstract: Referential games offer a grounded learning environment for neural agents which accounts for the fact that language is functionally used to communicate. However, they do not take into account a second constraint considered to be fundamental for the shape of human language: that it must be learnable by new language learners. Cogswell et al. (2019) introduced cultural transmission within referential games through a changing population of agents to constrain the emerging language to be learnable. However, the resulting languages remain inherently biased by the agents’ underlying capabilities. In this work, we introduce Language Transmission Simulator to model both cultural and architectural evolution in a population of agents. As our core contribution, we empirically show that the optimal situation is to take into account also the learning biases of the language learners and thus let language and agents co-evolve. When we allow the agent population to evolve through architectural evolution, we achieve across the board improvements on all considered metrics and surpass the gains made with cultural transmission. These results stress the importance of studying the underlying agent architecture and pave the way to investigate the co-evolution of language and agent in language emergence studies.

## introduction human languages show a remarkable degree of structure and complexity. in the evolution of this complex structure, several different intertwined pressures are assumed to have played a role. the first of these pressures concerns the function of language: as language is to communicate, it should allow effective communication between proficient language users (e.g. @xcite . this pressure is strongly intertwined with the nature of the proficient user: what features of language allow effective communication depends on the abilities of the user to use the language. a second pressure on the shape of human language stems from the fact that language must be learnable. unlike animal languages, which are taken to be mostly innate, human languages must be re-acquired by each individual @xcite . a language can only survive if it can successfully be transmitted to a next generation of learners. in the field of language evolution, this transmission process is referred to as cultural transmission, while the process of change that occurs as a consequence is called cultural evolution. 1 like the pressures arising from the function of language, the way that cultural evolution shapes language also depends on the language users: what is learnable depends on the inductive biases of the learner. computationally, the emergence of language can be studied through simulation with artificial agents and by investigating the resulting languages for structure, level of compositionality, and morphosyntactic properties @xcite . originally based on logic and symbolic representations @xcite , with the advent of modern deep learning methods, there has been a renewed interest in simulating the emergence of language through neural network agents (i.a. @xcite . such work typically involves the use of referential games @xcite , in which two or more agents have to emerge a language to obtain a shared reward. these studies are motivated by the first pressure: language as a tool for effective communication. however, they fail to consider the second pressure: language must be learnable by new agents. they also fail to study the impact of the learning biases of the artificial agents themselves, which underlies both pressures. in a recent study, @xcite proposed a method to include cultural evolution in a language emergence game. their approach is more naturally aligned with pressures in humans language evolution than single agent referential games (see e.g. @xcite , but fails to account for the fact that cultural evolution and the learning biases of the artificial agents are two sides of the same coin: what language is learnable depends on the learning biases of the learner. in this paper, we will therefore integrate the three components described above -communication, cultural evolution and learning biases -and setup a framework in which their interaction can be studied. this framework, which we refer to with the term language transmission simulator, consists of a referential game, played by a changing population of agents -simulating cultural evolution -which are subject to architectural evolution -simulating the learning biases of the learners and allowing them to co-evolve with the language. our contributions are three-fold fold: much work has been done on the emergence of language in artificial agents and investigating its subsequent structure, compositionality and morphosyntax @xcite . originally, such work was based on logic and symbolic representations @xcite , but with the advent of modern deep learning @xcite , there has been a renewed interest in simulating the emergence of language through neural network agents (i.a. @xcite @xcite @xcite . in the exploration of language emergence, different training approaches and tasks have been proposed to encourage agents to learn and develop communication. in a typical setup, two players aim to develop a communication protocol in which one agent must communicate information it has access to (typically an image), while the other must guess it out of a line-up @xcite . @xcite show that 'natural' language does not arise naturally in these communication games and it has to be incentivised by imposing specific restrictions on games and agents. @xcite first demonstrated that using straight-through estimators were more effective than reinforcement learning in a collaborative task, and that optimizing rewards can lead to structured protocols (i.e. strings of symbols) to be induced from scratch. @xcite find that syntactic structure emerges in the stream of symbol uttered by agents, where symbols and syntax can be mapped to specific meanings or instructions. @xcite use qualitative analysis, visualization and a zero-shot test, to show that a language with compositional properties can emerge from environmental pressures. chaabouni et al. (2019) find that emerged languages, unlike human languages, do not naturally prefer non-redundant encodings. chaabouni et al. (2020) further find that while generalization capabilities can be found in the languages, compositionality itself does not arise from simple generalization pressures. (rodríguez luna et al., 2020) encourage desirable properties of human languages, such as compositionality, to emerge through well-crafted auxiliary pressures. finally, harding graesser et al. (2019) demonstrate with experiments on populations of agents that language can evolve and simplify through the interactions of different communities. @xcite build upon the emergent language research by introducing cultural trans-mission as a pressure in referential games. they use a pool of agents with a resetting mechanism and show that this further encourages the emerging language to display compositional properties and structure allowing it to generalize better. pairing agents with one another in a larger population setting introduces cultural evolution, but it is the pressure introduced by the partial resetting which forces remaining agents to emerge a language that is quickly learnable by a new agent. while @xcite is the most related work to ours, an important difference is that they focus on cultural evolution only, without taking into account the learning biases of the agents via modelling architectural evolution. ## approach in this paper, we introduce architectural evolution in language emergence games and study the interaction between cultural and architectural evolution with a range of different metrics. below, we first give a definition of the referential game we consider (subsection 3.1). we then briefly explain our language transmission simulator (section 3.2 and how we model cultural and architectural evolution within it (section 3.3 and 3.4 respectively). ## experiments we test the lte framework on a compositionally defined image dataset, using a range of different selection mechanisms. ## results we now present a detailed comparison of our cultural and co-evolution setups. for each approach, we averaged over four random seeds, the error bars in all plots represent the standard deviation across these four runs. to analyse the evolution of both agents and languages, we consider the development of all previously outlined metrics over time. we then test the best converged languages and architectures in a single sender-receiver setup, to assess the impact of cultural and genetic evolution more independently. in these experiments, we compare also directly to a single sender-receiver baseline, which is impossible for most of the metrics we consider in this paper. finally, we briefly consider the emerged architectures from a qualitative perspective. ## analysis in this section we analyse the resulting behaviour and success of agents in language transmission simulator. we first use standard approaches such as average agent entropy and loss (convergence) to measure the success of agents with respect to their language and the task. secondly, we use other metrics to analyse the emergent language itself in terms of consistency and diversity by using jaccard similarity, the proportion of unique matches, the number of unique messages, and the topographic similarity. thirdly, we perform a qualitative analysis of the architecture that emerge from our language transmission simulator. finally, we design frozen experiments, in which we test the emerged languages and architectures in a 1v1 setting with a fresh agent. this allows us to compare and measure the the improvement gains made by the architecture and those made by the language which emerged. we show through these experiments that the coevolution setting leads to a language that is both more successful and easier to learn for a given new agent. ## conclusion in this paper, we introduced a language transmission bottleneck in a referential game, where new agents have to learn the language by playing with more experienced agents. to overcome such bottleneck, we enabled both the cultural evolution of language and the architectural evolution of agents, using a new language transmission simulator. using a battery of metrics, we monitored their respective impact on communication efficiency, degree of linguistic structure and intra-population language homogeneity. while we could find important differences in between cultural evolution strategies, it is when we included architectural evolution that agents scored best. in a second experiment, we paired new agents with evolved languages and agents and again confirmed that, while cultural evolution makes a language easier to learn, co-evolution leads to the best communication. in future research, we would like to apply the language transmission simulator on new, more complex tasks and further increase our understanding of the properties of the emerged languages and architectures. recent research has also found that relaxing the vocabulary size v and sequence length l constraints can lead to greater syntactic structure in emergent languages @xcite . we thus hope to investigate further relaxation of hyper-parameters and other neuro-evolution techniques in future work.