The Effect of Efficient Messaging and Input Variability on Neural-Agent Iterated Language Learning (2021)

Abstract: Natural languages display a trade-off among different strategies to convey syntactic structure, such as word order or inflection. This trade-off, however, has not appeared in recent simulations of iterated language learning with neural network agents (Chaabouni et al., 2019b). We re-evaluate this result in light of three factors that play an important role in comparable experiments from the Language Evolution field: (i) speaker bias towards efficient messaging, (ii) non systematic input languages, and (iii) learning bottleneck. Our simulations show that neural agents mainly strive to maintain the utterance type distribution observed during learning, instead of developing a more efficient or systematic language.

## introduction the world's languages show immense variety, but linguistic patterns also show important universal tendencies @xcite . it has been argued that these common design features are shaped by human cognitive constraints and pressures during communication and transmission @xcite . a well-known example of these tendencies is the trade-off between case marking and word order as redundant strategies to encode the role of sentence constituents @xcite : flexible order typically correlates with the presence of case marking (e.g. in russian), while fixed order is often observed in languages with little or no case marking (e.g. english). researchers interested in language universals and their origins have extensively used agent-based modeling techniques to study the impact of social processes on the emergence of linguistic structures @xcite . besides the horizontal transmission that is often modeled in the referential game setup, the process of iterated learning, where signals are transmitted vertically from generation to generation, has been identified to shape language @xcite . recently, the advent of deep learning based nlp has triggered a renewed interest in agent-based simulations of language emergence. most existing studies simulate the emergence of language by letting neural network agents play referential games and studying the signals they use @xcite @xcite @xcite . by contrast, @xcite expose their agents to a pre-defined language, which is then learned and reproduced iteratively by a chain of agents. they analyze how specific properties of the initial languages affect learnability, and further investigate how agent biases affect the evolution across generations. among others, they studied whether neural agents tend to avoid redundant coding strategies as natural languages do. however, the case-marking/word-order trade-off did not clearly appear in their iterated learning experiments. in this work, we re-evaluate this finding in light of three factors that play an important role in comparable experiments from the language evolution field: (i) speaker bias towards efficient messaging @xcite , (ii) unpredictable variation in the initial languages @xcite , (iii) exposure to a limited set of example utterances, known as 'learning bottleneck' @xcite . we follow the iterated learning setup of @xcite where neural agents are trained to communicate about trajectories in a simple gridworld, exchanging instructions in miniature languages (fig. 1 ). foot_0 2 miniature languages word order and case marking are two different mechanisms to convey sentence constituent roles, both widely attested among world languages. both type utterance fix+marker m1 up 2 m2 left 3 m3 down 1 fix up 2 left 3 down 1 m1 up 2 m3 down 1 m2 left 3 free-order m2 left 3 m1 up 2 m3 down 1 +marker m3 down 1 m1 up 2 m2 left 3 ... have been shown to serve as equally valuable cues to learn grammatical roles during language learning by children @xcite , and by simple recurrent neural networks @xcite . to model these mechanisms we use simple artificial languages based on @xcite . the meaning space is composed of trajectories defined by random combinations of four oriented actions {left, right, up, down}. each utterance or sentence (s) consists of several phrases (p), which in turn are composed of a command (c) and a quantifier (q): fixed-order vs. free-order this concerns rule 1: in a fixed-order language, the order of phrases strictly corresponds to the temporal order of instructions in the trajectory. 2 free-order languages, instead, allow any permutation of phrases. for instance, there are in total six possible freeorder utterances for a 3-phrase trajectory (fig. 1 ). case marking in a case-marking language, each phrase is preceded by a temporal marker indicating its role. thus, rule 2 changes to: p @xmath0. note that a fully free-order language is unintelligible without markers. ## neural-agents iterated learning we strictly follow the iterated learning setup of @xcite unless explicitly noted. agent architecture agents are implemented as 1-layer attentional seq2seq @xcite lstm @xcite networks. each agent acts as both speaker (receiving trajectories and describing them with utterances) and listener (receiving utterances and trying to induce the corresponding trajectories). the vocabulary contains both actions and words; embeddings of the encoder input and decoder output are tied (press and wolf, 2017). individual and iterated learning given trajectory-utterance pairs, agents are trained by teacher forcing @xcite in both listening and speaking mode, using the early-stopping and optimizer settings of @xcite . 3 iterated learning @xcite is implemented by letting a trained adult agent teach a randomly initialized child agent, and repeating this process for a number of generations. at each generation, two steps are performed: (1) a trained adult agent receives a batch of trajectories and generates utterances by sampling from its own decoder outputs; evaluation in both speaking and listening mode, sequences are generated by greedy decoding and evaluated by sentence-level accuracy. listener evaluation is standard as the true meaning of an utterance is unique. for speakers in the first generation, instead, we consider all acceptable utterances according to the grammar as candidate targets; for later generations, we take @xmath1. validation for early stopping is performed similarly. these evaluation procedures allow a child agent's language to deviate from the parent language, even while achieving perfect accuracy. for each experiment, we report speaking accuracy, listening accuracy, as well as average utterance length across generations. to get more insight into language changes, we plot the distribution of utterance types in the adult speaking agents across generations. specifically, we count how often an utterance belongs to one of the basic language types (fix, fix_marker, free, free_marker), and how often markers are dropped for some of the phrases (fix_drop, free_drop). utterances that do not fall into any of these categories are labeled as 'other'. for more technical details on the agents and evaluation, see appendix a. example utterances at various generations are shown in appendix b. ## effect of least-effort bias a bias towards efficient messaging has been proposed as explaining factor for several tendencies observed in natural languages @xcite @xcite . could the lack of such a bias in neural networks explain the survival of redundant languages? to verify this, we design a simple mechanism simulating an agent's preference to minimize utterance length, based on @xcite 's framework: to teach the next generation, given a trajectory t, an adult agent generates n (possibly identical) utterances {û} = { û1 , û2 , ... ûn } by sampling from its trained network. instead of modifying the training process, we exploit the diversity occurring in the sampled utterances and hard-code a shorter-sentence selection bias into this adult language generation. as shown in algorithm 1, the sampling function is called n times to generate n samples. in turn, at each iteration, we ask the adult speaker to generate ℓ sentences and select each time the shortest one. thus, we can control the bias strength by varying the number of generated samples (ℓ) in each of the n iterations. as ℓ increases, the chances of sampling a shorter sentence increase, resulting in a stronger pressure; ℓ = 1 means no bias. we expect this least-effort bias will cause the redundant disambiguation mechanism to gradually disappear, and the fixed-order strategy to dominate as that always leads to shorter utterances. results fig. 2 shows the iterated learning results of the fixed+marker language with various levels of least-effort bias ℓ = {1, 3, 5, 8}, which represent no pressure, low-, medium-and highlevel pressure towards shorter utterances, respectively. the experiment without least-effort pressure 5 for @xmath2: ℓ do 6 @xmath3 we find that, while speaking accuracy remains stable (2a), our least-effort pressure leads to a severe drop in listening accuracy (2b) and a dramatic increase of other types in the speaking adult agent starting from the fifth generation (2d). stronger levels of pressure lead to a faster decrease of average utterance length (2c), which was expected. however, manual inspection of the utterances (appendix b) reveals that agents start dropping entire phrases, thereby losing information, instead of either dropping markers or changing the word order. ## effect of input language variability in the languages of @xcite , markers are either present and fully systematic, or not present at all. if there is no marker example in the initial language, it is unlikely an agent would suddenly invent it. conversely, a fully systematic use of markers may be perfectly learnable by the agent, unlikely to change or disappear over generations. unpredictable variation, instead, is a common feature of the artificial languages used in human learning studies. for example @xcite combine optional case marking with either fixed or free word order, while @xcite use two plural markers with different distributions over nouns. inspired by this, we experiment with unpredictable variations in the use of markers, namely: (i) variability among utterances, where each utterance is consistent with one of the basic language types chosen at random, and (ii) variability within utterances, where the use of markers is also unpredictable within a single utterance. ## effect of learning bottleneck although real languages support the production of enormous sets of utterances, human learners master them by exposure to a limited number of examples. this learning bottleneck acts as a pressure forcing languages to regularize during cultural transmission @xcite . human-based experiments and computational simulations have found that this pressure can lead to increased structure in emerging language systems @xcite . we apply a learning bottleneck to our mixed language experiment (sect. 5.1) by randomly sub-sampling, at each iteration, 50% of the data used to train the next generation. evaluation and training details are the same as in sect. 5.1. results we find that training data sub-sampling leads to a slightly steeper drop in listening accuracy (yellow vs. blue line in fig. 3b ), but the respective distributions of utterance types (fig. 3g vs. 3d) remain very similar, suggesting the learning bottleneck does not result in a more structured language. ## discussion and conclusions neural-agent iterating learning is a promising framework to study the impact of social processes on the emergence of patterns and language universals, like the word order/case marking tradeoff. however, previous work with lstm-based agents @xcite has failed to replicate this human-like pattern. we re-evaluated this finding by (i) hard-coding a least-effort bias into our agents, (ii) designing less systematic input languages, and (iii) introducing a learning bottleneck. in all cases, our agents proved to be accurate learners, but the patterns of language change over generations did not match our expectations. specifically, least-effort bias ( §4) and highly unpredictable input language ( §5.2) led to communication failure, whereas moderate input language variability ( §5.1) and learning bottleneck ( §6) led to a stable language distribution, confirming previous observations on the survival of redundant coding strategies in neural-agent iterated learning @xcite . only combining least-effort bias with moderate language variability ( §5.1) led to a temporary optimization of the language, but that was again followed by communication failure. in real language use, the pressure to reduce effort is balanced with communicative needs @xcite and does normally not lead to severe language degradation. future work should design subtler least-effort biases, for instance considering efficiency in terms of grammatical structure and cognitive effort. moreover, our results with non fully systematic languages show that agents strive to preserve the initial utterance type distribution. in human learning, this behavior is called probability matching and is affected by task complexity: more difficult tasks tend to regularization or over-matching @xcite , where the more frequent variant is chosen more often than it appeared in the input. over many generations, even a slight overmatching can lead to the emergence of linguistic regularities @xcite like the word order/case marking trade-off observed in human learners by @xcite . we conclude that the current neural-agent iterated learning framework is not yet ready to simulate language evolution processes in a human-like way. more natural cognitive biases supporting efficiency should be modeled, while the speaker objective should be balanced with a measure of communicative success, such as the likelihood of a message to be understood by the listener @xcite .