The Grammar of Emergent Languages (2020)

Abstract: In this paper, we consider the syntactic properties of languages emerged in referential games, using unsupervised grammar induction (UGI) techniques originally designed to analyse natural language. We show that the considered UGI techniques are appropriate to analyse emergent languages and we then study if the languages that emerge in a typical referential game setup exhibit syntactic structure, and to what extent this depends on the maximum message length and number of symbols that the agents are allowed to use. Our experiments demonstrate that a certain message length and vocabulary size are required for structure to emerge, but they also illustrate that more sophisticated game scenarios are required to obtain syntactic properties more akin to those observed in human language. We argue that UGI techniques should be part of the standard toolkit for analysing emergent languages and release a comprehensive library to facilitate such analysis for future researchers.

## introduction artificial agents parameterised by deep neural networks can learn to communicate using discrete symbols to solve collaborative tasks @xcite @xcite . a prime reason to conduct such studies, which constitute a new generation of experiments with referential games, is that they may provide insight in the factors that shaped the evolution of human languages @xcite . however, the emergent languages developed by neural agents are not human-interpretable, and little is known about their semantic and syntactic * shared senior authorship nature. more specifically, we do not know to what extent the structure of emergent languages resembles the structure of human languages, what the languages encode, and how these two things depend on choices that need to be made by the modeller. a substantial obstacle to better understanding emergent languages is the lack of tools to analyse their properties. previous work has concentrated primarily on understanding languages through their semantics, by studying the alignment of messages and symbolic representations of the meaning space (e.g. @xcite . a substantial downside of such approaches is that they are restricted to scenarios for which a symbolic representation of the meaning space is available. furthermore, they ignore a second important aspect of language: syntax, which is relevant not just for syntacticallyoriented researchers, but also for those that are interested in semantics from a compositional perspective. in this work, we aim to address this gap in the literature by presenting an analysis of the syntax of emergent languages. we take inspiration from unsupervised grammar induction (ugi) techniques originally proposed for natural language. in particular, we use them to investigate if the languages that emerge in the typical setup of referential games exhibit interesting syntactic structure, and to what extent this depends on the maximum message length and number of symbols that the agents are allowed to use. we first establish that ugi techniques are suitable also for our artificial scenario, by testing them on several artificial structured languages that are distributionally similar to our emergent languages. we then use them to analyse a variety of languages emerging from a typical referential game, with various message lengths and vocabulary sizes. we show that short messages of up to length five do not give rise to any interesting structure, while longer messages are significantly more structured than random languages, but yet far away from the type of syntactic structure observed in even simple human language sentences. as such, our results thus suggest that more interesting games scenarios may be required to trigger properties more similar to human syntax and -importantly -confirm that ugi techniques are a useful tool to analyse such more complex scenarios. their results are informative not only for those interested in the evolution of structure of human languages, but can also fuel further semantic analysis of emergent languages. ## related work previous work that focused on the analysis of emergent languages has primarily concentrated on semantics-based analysis. in particular, they considered whether agents transmit information about categories or objects, or instead communicate using low-level feature information @xcite @xcite @xcite . ## method we analyse the syntactic structure of languages emerging in referential games with ugi techniques. in this section, we describe the game setup that we consider ( §3.1), the resulting languages that are the subject of our analysis ( §3.2) and the ugi techniques that we use ( §3.3). lastly, we discuss our main methods of evaluating our ugi setups and the resulting grammars ( §3.4) as well as several baselines that we use for comparison ( §3.5). ## suitability of induction techniques as the grammar induction techniques we apply are defined for natural language, they are not trivially also suitable for emergent languages. in our first series of experiments, we therefore assess the suitability of the grammar induction techniques for our artificial scenario, evaluate to what extent the techniques are dependent on the exact sample taken from the training set, and we determine what is a suitable data set size for the induction techniques. the findings of these experiments inform and validate the setup for analysing the emergent languages in §5. ## analysing emergent languages having verified the applicability of both ccl-bmm and diora-bmm, we use them to induce grammars for all languages described in §3.2. we analyse the induced grammars and parses, comparing with the structured, shuffled, and random baselines introduced in §3.5. ## conclusion while studying language and communication through referential games with artificial agents has recently regained popularity, there is still a very limited amount of tools available to facilitate the analysis of the resulting emergent languages. as a consequence, we still have very little understanding of what kind of information these languages encode. in this paper, for the first time, we focus on syntactic analysis of emergent languages. we test two different unsupervised grammar induction (ugi) algorithms that have been successful for natural language: a pre-neural statistical one, ccl, and a neural one, diora. we use them to infer grammars for a variety of languages emerging from a simple referential game and then label those trees with bmm, considering in particular the effect of the message length and vocabulary size on the extent to which structure emerges. we first confirm that the techniques are capable of inferring interesting grammars for our artificial setup and demonstrate that ccl appears to be a more suitable constituency parser than diora. we then find that the shorter languages, with messages up to 5 symbols, do not contain any interesting structure, while languages with longer messages appear to be substantially more structured than the two random baselines we compare them with. interestingly, our analysis shows that even these languages do not appear to have a notion of word classes, suggesting that their symbols may in fact be more akin to letters than to words. in light of these results, it would be interesting to explore the use of unsupervised tokenisers that work well for languages without spaces (e.g. sentencepiece @xcite prior to our approach and to try other word embedding models for diora, such as the character-based elmo embeddings 8 @xcite or the more recent bert @xcite . our results also suggest that more sophisticated game scenarios may be required to obtain more interesting structure. ugi could provide an integral part in analysing the languages emerging in such games, especially since it -contrary to most techniques previously used for the analysis of emergent languages -does not require a description of the hypothesised semantic content of the messages.