Communication Drives the Emergence of Language Universals in Neural Agents: Evidence from the Word-order/Case-marking Trade-off (2023)

Abstract: Artificial learners often behave differently from human learners in the context of neural agent-based simulations of language emergence and change. A common explanation is the lack of appropriate cognitive biases in these learners. However, it has also been proposed that more naturalistic settings of language learning and use could lead to more human-like results. We investigate this latter account, focusing on the word-order/case-marking trade-off, a widely attested language universal that has proven particularly hard to simulate. We propose a new Neural-agent Language Learning and Communication framework (NeLLCom) where pairs of speaking and listening agents first learn a miniature language via supervised learning, and then optimize it for communication via reinforcement learning. Following closely the setup of earlier human experiments, we succeed in replicating the trade-off with the new framework without hard-coding specific biases in the agents. We see this as an essential step towards the investigation of language universals with neural learners.

## introduction the success of deep learning methods for natural language processing has triggered a renewed interest in agent-based computational modeling of language emergence and evolution processes @xcite . an important challenge in this line of work, however, is that such artificial learners often behave differently from human learners @xcite @xcite . one of the proposed explanations for these mismatches is the difference in cognitive biases between human and neural network (nn) based learners. for instance, the neural-agent iterated learning simulations of @xcite and @xcite did not succeed in replicating the trade-off between word-order and case marking, which is widely attested in human languages @xcite and has also been observed in artificial language learning experiments with humans @xcite . instead, those simulations resulted in the preservation of languages with redundant coding mechanisms, which the authors mainly attributed to the lack of a human-like least-effort bias in the neural agents. besides human-like cognitive biases, it has been proposed that more natural settings of language learning and use could lead to more human-like patterns of language emergence and change @xcite @xcite . in this work, we follow up on this second account and investigate whether neural agents that strive to be understood by other agents display more human-like language preferences. to achieve that, we design a neural-agent language learning and communication (nellcom) framework that combines supervised learning (sl) with reinforcement learning (rl), inspired by @xcite and @xcite . specifically, we use sl to teach our agents predefined languages characterized by different levels of word order freedom and case marking. then, we employ rl to let pairs of speaking and listening agents talk to each other while optimizing communication success (also known as selfplay in the emergent communication literature). we closely compare the results of our simulation to those of an experiment with a very similar setup and miniature languages involving human learners @xcite , and show that a human-like trade-off can indeed appear during neural-agent communication. although some of our results differ from those of the human experiments, we make an important contribution towards developing a neural-agent framework that can replicate language universals without the need to hard-code any ad-hoc bias in the agents. we release the nellcom framework foot_0 to facilitate future work simulating the emergence of different language universals. various experiments with human participants @xcite @xcite were conducted to reveal the underlying cause of this correlation. in particular, @xcite , who highly inspired this work, applied a miniature language learning approach to study whether the trade-off could be explained by a human learning bias to reduce production effort while remaining informative. in their experiment, two groups of 20 participants were asked to learn one of two predefined miniature languages. both languages contained optional markers but differed in terms of word order (fixed vs. flexible). after three days of training, both groups reproduced the initial word order distribution, however the flexible-order language learners used case marking significantly more often than the fixed-order language learners. moreover, an asymmetric marker-using strategy was found in the flexible-order language learners, whereby markers tended to be used more often in combination with the less frequent language. thus, most participants displayed an inverse correlation between the use of constituent order and case marking during language learning, which the authors attributed to a unifying information-theoretic principle of balancing effort with robust information transmission. agent-based simulations of language evolution computational models have been used widely to study the origins of language structure @xcite @xcite . in particular, @xcite were able to mimic the human acquisition patterns of four languages with very different word order and case marking properties, using a simple recurrent network @xcite . modern deep learning methods have also been used to simulate patterns of language emergence and change @xcite @xcite @xcite . despite several interesting results, many report the emergence of languages and patterns that significantly differ from human ones. for example, @xcite found an anti-efficient encoding scheme that surprisingly opposes zipf's law, a fundamental feature of human language. rita et al. (2020) obtained a more efficient encoding by explicitly imposing a length penalty on speakers and pushing listeners to guess the intended meaning as early as possible. focusing on the order/marking trade-off, @xcite implemented an iterated learning setup inspired by @xcite where agents acquire a language through sl, and then transmit it to a new learner, iterating over multiple generations. the trade-off did not appear in their simulations. @xcite extended the study by introducing several crucial factors from the language evolution field (e.g., input variability, learning bottleneck), but no clear trade-off was found. to our knowledge, no study with neural agents has successfully replicated the emergence of this tradeoff so far. ## nellcom: language learning and communication framework this section introduces the neural-agent language learning and communication (nellcom) framework, which we make publicly available. our goal differs from that of most work in emergent communication, where language-like protocols are expected to arise from sets of random symbols through interaction @xcite @xcite @xcite . we are instead interested in observing how a given language with specific properties changes as the result of learning and use. specifically, in this work, agents need to learn miniature languages with varying word order distributions and case marking rules. while this can be achieved by a standard sl procedure, we hypothesize that human-like regularization patterns will only appear when our agents strive to be understood by other agents. we simulate such a need via rl, using a measure of communication success as the optimization objective. similar sl+rl paradigms have been used in the context of communicative ai @xcite @xcite . in particular, @xcite al. (2020) and lowe et al. (2020) explore different ways of combining sl and rl to teach agents to communicate with humans in natural language. a well-known problem in that setup is that languages tend to drift away from their original form as agents adapt to communication. in our context, we are specifically interested in studying how this drift compares to human experiments of artificial language learning. our implementation is partly based on the egg toolkit foot_1 @xcite . ## experimental setup we use nellcom to replicate the results of @xcite , who taught human subjects miniature languages with varying order distributions. subjects watched short videos of two actors performing simple transitive events (e.g., a chef hugging a referee) accompanied by spoken descriptions in the novel language. 7 we adopt the same setup, with two notable differences: (i) our agents do not take videos or images as input, but triplets of symbols representing agent, patient, and action, respectively (see section 3); ## supervised learning results we start by evaluating the agents' ability to learn to speak or listen in a fully supervised way, that is, using the generated meaning-utterance pairs from a specific language as labeled data. ## communication learning results we study the effect of communication learning on communication success and language properties. ## individual learners' trajectories all results so far were averaged over multiple randomly initialized agents. here, we look at possible variations among pairs of speaking-listening agents. we focus only on the flexible-order language, as it is more likely to undergo different optimization strategies. figure 4 shows 20 production distributions, each corresponding to a different random seed. most agents (no. 1 to 14) regularize their productions towards the osv order, as anticipated by the average results in figure 3 . however, we also find two agent pairs that take the opposite path and produce more sov (no. 19 and 20). the remaining four agents show no clear order preferences (no. 15, 16, 17, and 18). as for case marking, a clear preference to drop the marker from sov utterances can be found in 15/20 pairs (no. 4, 5, 9, 10, 16 are exceptions), which reflects the average trend of conditional case marking shown in figure 3h . this high degree of between-agents variability matches human results @xcite @xcite where learners often adopt different strategies to reach a common optimization objective. ## discussion and conclusion we studied the conditions in which the word order/case marking trade-off, a well established language universal example, could emerge in a small population of neural-network learners. we hypothesized that more naturalistic settings of language learning and use could lead to more human-like results, without the need to hard-code specific biases, such as least effort, into the agents. we then proposed a new neural-agent language learning and communication framework (nellcom) where pairs of speaking and listening agents learn a given language through supervised learning, and then use it to communicate with each other, optimizing a shared reward via reinforcement learning. we used nellcom to replicate the experiments of @xcite , where two groups of human participants were asked to learn a fixedand a flexible-order miniature language, respectively, and to use it productively after training. our results with rnn-based meaning-to-sequence and sequence-to-meaning networks confirm that sl is sufficient for perfectly learning the languages, but does not lead to any human-like regularization, in line with recent simulations of iterated learning @xcite . by contrast, communication learning leads agents to modify their production in interesting ways: firstly, optional markers are dropped more frequently in the redundant fixed-order language than in the ambiguous flexible-order language, which matches human learning results. moreover, one of the two equally probable word orders in the flexible-order language becomes clearly dominant and case marking starts to be used consistently more often in combination with one order than with the other. this conditional use of marking also matches human results. some interesting differences were also observed: for instance, nellcom agents showed, on average, a slightly stronger tendency to reduce effort rather than uncertainty. as another difference, several human subjects managed to 'break' the linear correlation by making the language more efficient and less uncertain, whereas this happened only in one of our agent pairs. despite these differences, agents' productions show a clear correlation between effort and uncertainty, which strongly matches the core finding of @xcite . we conclude that the word order/ case marking trade-off as a specific realization of the efficiency/informativity trade-off can, in fact, emerge in neural network learners equipped with a need to be understood. we made an important step towards developing a neural-agent framework that replicates patterns of human language change without the need to hard-code ad-hoc biases. future work includes extending the current framework with iterated learning, which might lead agents to further optimize the ambiguous language and improve communication success over generations. we also plan to experiment with different neural network architectures to study the impact of architecture-specific structural biases, and with different word order universals. we hope our framework will facilitate future simulations of language evolution at different timescales with the end goal of explaining why human languages look the way they do.