Emergent Language Generalization and Acquisition Speed are not tied to Compositionality (2020)

Abstract: Studies of discrete languages emerging when neural agents communicate to solve a joint task often look for evidence of compositional structure. This stems for the expectation that such a structure would allow languages to be acquired faster by the agents and enable them to generalize better. We argue that these beneficial properties are only loosely connected to compositionality. In two experiments, we demonstrate that, depending on the task, non-compositional languages might show equal, or better, generalization performance and acquisition speed than compositional ones. Further research in the area should be clearer about what benefits are expected from compositionality, and how the latter would lead to them.

## introduction there is a recent spike of interest in studying the languages that emerge when artificial neural agents communicate to solve a common task @xcite @xcite . a good portion of such studies looks for traces of compositional structure in those languages, or even tries to inject such structure into them @xcite @xcite @xcite @xcite @xcite . besides possibly providing insights on how compositionality emerged in natural language @xcite , this emphasis is justified by the idea that a compositional language has various desirable properties. in particular, compositional languages are expected to help agents to better generalize to new (composite) inputs @xcite , and to be faster to acquire @xcite @xcite . we engage here with this ongoing research pursuit. we step back and reflect on the benefits that compositionality can bring to the emergent languages: if there is none, then it is unlikely that agents will develop compositional languages on their own. indeed, several studies have shown that compositionality does not emerge naturally among neural agents (e.g. @xcite @xcite . on the other hand, understanding what benefits compositionality could bring to a language would help us in establishing the conditions for its emergence. compositionality is typically seen as a property of a language, independent of the task being considered. however, the task will likely influence properties such as generalization and ease of acquisition, that compositionality is expected to correlate with. our experiments show that it is easy to construct tasks for which a compositional language is equally hard, or harder, to acquire and does not generalize better than a non-compositional one. hence, language emergence researchers need to be clear about i) which benefits they expect from compositionality and ii) in which way compositionality would lead to those benefits in their setups. otherwise, the agents will likely develop perfectly adequate communication systems that are not compositional. ## operationalizing compositionality before we proceed, let us clarify our definition of compositionality. linguists and philosophers have extensively studied the topic for centuries (see @xcite , for a thorough review). however, the standard definition that a language is compositional if the meaning of each expression in it is a function of the meaning of its parts and the rules to combine them is so general as to be vacuous for our purposes (under such definition, even the highly opaque languages we will introduce below are compositional, contra our intuitions). in most current language emergence research, the input to language is composite in the sense that it consists of ensembles of elements. in this context, intuitively, a language is compositional if its symbols denote input elements in a disentangled way, so that they can be freely juxtaposed to refer to arbitrary combinations of them. more precisely, the following property might suffice for a limited but practical characterization of compositionality. given a set of atomic input elements (for example, a set of independent attribute values), each atomic symbol should refer to one and only one input element, independently of the other symbols it co-occurs with. foot_0 a language where all symbols respect this property is compositional in the intuitive sense that, if we know the symbols that denote a set of input elements, we can assemble them (possibly, following some syntactic rules of the language) to refer to the ensemble of those input elements, irrespective of whether we have ever observed the relevant ensemble. consider for example a world where inputs consist of two attributes, each taking a number of values. a language licensing only twocharacter sequences, where the character in the first position refers to the value of the first attribute, and that in the second position independently refers to the value of the second, would be compositional in our sense. on the other hand, a language that also licenses two-character sequences, but where both characters in a sequence are needed to decode the values of both the first and the second input attribute, would not be compositional. we will refer to the lack of symbol interdependence in denoting distinct input elements as naïve compositionality. 2 we believe that naïve compositionality captures the intuition behind explicit and implicit definitions of compositionality in emergent language research. for example, @xcite deem non-compositional those languages that either use single symbols to refer to ensembles of input elements, or where the meaning of a symbol depends on the context in which it occurs. @xcite looked for symbol-position combina-tions that encode a single concept in an image, as a sign of a compositional behavior. a naïvely compositional language will maximize the two recently proposed compositionality measures of residual entropy @xcite and positional disentanglement @xcite . naïve compositionality is also closely related to the notion of disentanglement in representation learning @xcite . interestingly, @xcite reported that disentanglement is not necessarily helpful for sample efficiency in downstream tasks, as had been previously argued. this resonates with our results below. ## communication game we base our experimental study on a one-episode one-direction communication game, as commonly done in the relevant literature @xcite @xcite . in this setup, we have two agents, sender and receiver. an input i is fed into sender, in turn sender produces a message m, which is consumed by receiver. receiver produces its output ô. comparing the output ô with the ground-truth output o provides a loss. we used egg @xcite to implement the experiments. 3 in contrast to the language emergence scenario, we use a hard-coded sender agent that produces a fixed, pre-defined language. this allows us to easily control the (naïve) compositionality of the language and measure how it affects receiver's performance. this setup is akin to the motivating example of @xcite . we study two receiver's characteristics: (i) acquisition speed, measured as the number of epoch needed to achieve a fixed level of performance on training set, and (ii) generalization performance on held-out data. ## experimental setup to demonstrate that compositionality of a language alone, detached from the task at hand, does not necessarily lead to higher generalization or faster acquisition speed, we design two experiments. the first experiment (attval) operates in an attribute-value world, similar to those of @xcite ; @xcite . we fix two languages, one compositional and one not, and build three tasks: (i) "easy" for compositional language and "hard" for non-compositional; (ii) equally "hard" for both; (iii) "hard" for compositional language and "easy" for non-compositional language. informally, we control the amount of computation needed by receiver to perform a task starting from a language, where it can be equally hard to rely on compositional or non-compositional languages, or the answers could even be readily available in a non-compositional language. in the second experiment (coordinates), we design a single task that is equally "easy" for an entire family of languages (parameterized by a continuous value), including compositional and noncompositional ones. the task is to transmit points on the 2d plane (thus, the input ensembles here are pairs of point coordinates). here, we leverage the observation that a typical neural model has a linear output layer, for which it is equally easy to learn any rotation of the ground-truth outputs. such rotation-group-invariance could play role in games where continuous image embeddings are used as input @xcite . ## discussion and conclusion our toy experiments with hand-coded languages make the possibly obvious but currently overlooked point that, in isolation from the target task, there is nothing special about a language being (naïvely) compositional. a non-compositional language can be equally or faster to acquire than a compositional one, and it can provide the same or better generalization capabilities. thus, if our goal is to let compositional languages emerge, we should be very clear about which characteristics of our setup should lead to its emergence. our concern is illustrated by the recent findings of @xcite , who observed that the degree of compositionality of emergent languages is not correlated with the generalization capabilities of the agents that rely on them to solve a task. indeed, lacking any specific pressure towards developing a (naïvely) compositional language, their agents were perfectly capable of developing generalizable but non-compositional communication systems. our experiments provide a plausible explanation of their findings. a stronger conclusion is that perhaps we should altogether forget about compositionality as an end goal. the current emphasis on it might just be a misguided effect of our human-centric bias. we should instead directly concentrate on the properties we want agent languages to have, such as fast learning, transmission and generalization.