On the Relationship between Z ipf’s Law of Abbreviation and Interfering Noise in Emergent Languages (2021)

Abstract: This paper studies whether emergent languages in a signaling game follow Zipf’s law of abbreviation (ZLA), especially when the communication ability of agents is limited because of interfering noises. ZLA is a well-known tendency in human languages where the more frequently a word is used, the shorter it will be. Surprisingly, previous work demonstrated that emergent languages do not obey ZLA at all when neural agents play a signaling game. It also reported that a ZLA-like tendency appeared by adding an explicit penalty on word lengths, which can be considered some external factors in reality such as articulatory effort. We hypothesize, on the other hand, that there might be not only such external factors but also some internal factors related to cognitive abilities. We assume that it could be simulated by modeling the effect of noises on the agents’ environment. In our experimental setup, the hidden states of the LSTM-based speaker and listener were added with Gaussian noise, while the channel was subject to discrete random replacement. Our results suggest that noise on a speaker is one of the factors for ZLA or at least causes emergent languages to approach ZLA, while noise on a listener and a channel is not.

## introduction there has recently been a growing interest in simulating languages spontaneously emerging among artificial agents, by training them to solve some tasks requiring communications. a primary motivation in this area is to pursue the development of artificial intelligence that can interact or communicate with human beings (e.g., @xcite @xcite . in addition to this line of research, some studies have investigated the characteristics of emergent languages, mainly concerned with to what extent they are similar to human languages or what kind of factor forms language-like protocols (e.g., @xcite @xcite . @xcite , for example, studied the relationship between emergent languages and zipf's law of abbreviation (zla), which is a universal tendency in human languages, where frequent words tend to be shorter @xcite . to see whether emergent languages follow zla, they performed experiments in which agents played a signaling game. their results suggested that emergent languages have an opposite tendency against zla. in other words, more frequent inputs are encoded into longer messages. they also reported that by giving an additional penalty on message lengths @xcite , the emergence of a zla-like tendency was observed. zipf @xcite hypothesized that zla comes about between two conflicting pressures: one for accuracy and the other for efficiency. in a paradigm with human subjects using a simple artificial language, @xcite , for instance, introduced some external factors for simulating the competing pressures, namely, money reward for precise and quick communications. in emergentlanguage simulations, the explicit penalty on message lengths (eq. 6) of @xcite can also be considered an external factor for zla. however, we speculate that there might be not only such external factors but also internal factors (or implicit penalties) related to the cognitive abilities of human beings such as memory. inspired by some concepts in psychology, we hypothesize at first in the following way: hypothesis 1. zla appears due to some internal factors from the cognitive abilities of human beings, as well as external factors. in other words, human beings assign shorter codes to frequent words so that they can avoid difficulty in their internal processes as much as possible. some studies in psychology suggested that in human beings, there is an output buffer of some sort that temporarily reserves some words to be spoken @xcite @xcite @xcite . the output buffer might decay over time, be overwhelmed by incoming inputs one after another, or be exposed to other disturbances. such pressures, we thought, could be factors to shorten frequent words. but how should they be modeled in the simulations of language emergence? since artificial agents in simulations are not humans but often (recurrent) neural networks, it is not trivial to define equivalent pressures for them. to adopt such pressures into a signaling game, we propose modeling them into noise that interferes with the states of agents. although the potential factors described above might be the matter of a speaker in a signaling game, we also propose adding noise to a listener for comprehensive research. the listener's short-term memory might also be limited due to similar reasons as the speaker. besides, we try adding noise to a channel that spans the speaker and the listener, referring to a noisy-channel model @xcite . although a noisy channel is not probably pressure for efficiency but for accuracy, the assumption that redundancy contributes to accuracy seems to think implicitly of a listener as capable enough of correcting errors while maintaining necessary information, which is not trivial for neural agents. therefore it is worth a try. by the modeling and for the comprehensiveness, hypothesis 1 is revised as follows: hypothesis 2. zla appears due to some of the three types of noises: noise on a speaker, noise on a listener, and noise on a channel. in our experimental setup, speaker and listener agents are exposed to gaussian noise since they have continuous vectors as their states. on the other hand, the channel is exposed to discrete random replacements, as messages passing through it have discrete variables. our experiments suggest that noise on a speaker is one factor for zla or at least causes emergent languages to be closer to zla, whereas noise on a listener and a channel is not in our signaling game. rather, the noise on a channel strengthened redundancy. our analysis reveals the following things. first, when noise interferes with a speaker agent, noise accumulation can make it difficult to generate long consistent messages. second, when noise interferes with a listener agent, on the other hand, noise accumulation does not affect the overall tendency crucially: even if the listener agent "forgets" the prefix of a message, the suffix is sufficient for communications. third, noise on a channel can be thought of as a pressure for accuracy rather than efficiency, which is consistent with an information-theoretic point of view and zipf's hypothesis. ## discussion our experiments suggest that noise on a speaker is a factor for zla, while noise on a listener and a channel is not in our signaling game. one possible reason for the noise on a speaker is that noise accumulation matters as time goes. at each trial, the speaker agent gets an input i and transforms it into an initial hidden state h 0 . the hidden states need to maintain the input i in some way for emitting consistent symbols. but noise accumulates over time and is harmful to their memory, which may cause frequent messages to be shorter. however, the result per se shows a neutral tendency between zla and anti-zla. our implicit length pressure might not have been strong enough, or there might have been some problems with the agents' architectures. noise on a listener is not a crucial factor for zla in our setting. judging from symbol effectiveness, the latter halves of messages tend to be more informative than the former when noise interferes with the listener. it means that the listener could "forget" the former halves of messages. in the first place, however, the former halves are less informative even if there is no noise. that may be why noise on a listener did not affect the overall tendency. noise on a channel seems to facilitate the redundancy of messages, which is consistent with zipf's hypothesis and a noisy-channel model. to help agents with learning, we used the two auxiliary loss der (eq. 11) and sml (eq. 13) which are somewhat artificial. in particular, the usage of sml conflicts a bit with our original goal to give rise to zla by an implicit penalty, as sml is similar to an artificial length pressure (eq. 6). ## conclusion in this paper, we simulated the emergence of language and checked whether the emergent languages follow zipf's law of abbreviation (zla). inspired by some psychological concepts, we proposed exposing architectures to some noise during training. our experiments were conducted under several noise conditions. the results suggested that noise on a speaker agent is one factor for zla, whereas neither noise on a listener nor noise on a channel is in our signaling game. our main contribution is to propose a potential factor for zla instead of an external length pressure and to demonstrate that noise imposing internal difficulty on a speaker agent may cause zla. however, there are several problems and limitations in addition to what is discussed in section 5. first, we could not try the combination of noises. one might be interested in combining the noises on a speaker, listener, and channel, but we failed to train agents stably under such conditions. it is simply because it became much more difficult for agents to learn under several noises. second, our signaling game did not contain any contexts. as an input space was no more complex than having the order by frequency, emergent languages could only have a unigram-like structure. however, according to @xcite , word predictability considering contexts is a better predictor of word length than unigram probabilities. from a more realistic point of view, therefore, contexts should be considered in some ways. moreover, if agents are forced to remember contexts, noise on a listener may also be a factor for zla, making the listener impatient. we leave these issues for future work.