On the Correspondence between Compositionality and Imitation in Emergent Neural Communication (2023)

Abstract: Compositionality is a hallmark of human language that not only enables linguistic generalization, but also potentially facilitates acquisition. When simulating language emergence with neural networks, compositionality has been shown to improve communication performance; however, its impact on imitation learning has yet to be investigated. Our work explores the link between compositionality and imitation in a Lewis game played by deep neural agents. Our contributions are twofold: first, we show that the learning algorithm used to imitate is crucial: supervised learning tends to produce more average languages, while reinforcement learning introduces a selection pressure toward more compositional languages. Second, our study reveals that compositional languages are easier to imitate, which may induce the pressure toward compositional languages in RL imitation settings.

## introduction compositionality, a key feature of human language, makes it possible to derive the meaning of a complex expression from the combination of its constituents @xcite . it has been suggested that more compositional languages are easier to acquire for both humans and artificial agents @xcite @xcite . therefore, to better understand the factors underlying language transmission, it is crucial to understand the relationship between ease-of-acquisition and compositionality. we study the link between compositionality and ease-of-acquisition in the context of emergent communication. in this setting, two deep artificial agents with asymmetric information, a sender and a receiver, must develop communication from scratch in order to succeed at a cooperative game @xcite @xcite . we will refer to this mode of language learning, in which agents develop language via feedback from mutual interaction, as communication-based learning. several studies have linked compositionality to ease-of-acquisition in communication-based learning. @xcite show compositionality predicts efficient linguistic transmission from senders to new receivers. conversely, li and bowling, 2019 re-pair a sender periodically with new receivers, and show this ease-of-teaching pressure improves compositionality. communication-based learning is not the only possibility for language learning, however. humans also crucially acquire language through imitation-based learning, in which they learn by observing other humans' language use @xcite . @xcite employ imitation learning, where in the first study, agents undergo a supervised imitation stage before communication-based learning, and where in the second, agents alternate between communication-based learning and imitating the best sender. however, the dynamics of imitation are not the focus in either study. for such an important vehicle of language acquisition, imitationbased learning thus remains under-explored in the emergent communication literature. we extend these lines of inquiry to systematically investigate compositionality in imitationbased learning. 1 our contributions are as follows: ## setup we study imitation in the context of referential communication games @xcite . in this setting, a sender agent observes an object x and transmits a message m to a second receiver agent. using this message, the receiver performs an action for which both agents receive a reward. over the course of the game, agents converge to a referential system (x, m), which we refer to as an emergent language. measuring compositionality evaluating compositionality in emergent languages is not straightforward given their grammars are a-priori unknown. therefore, we quantify compositionality using topographic similarity (topsim) @xcite , a grammar-agnostic metric widely applied to emergent languages in the literature. topsim is defined as the spearman correlation ρ between euclidean distances in the input space and levenstein distances in the message space-that is, it captures the intuition that nearby inputs should be described with similar messages. while we consider other compositionality metrics such as positional disentanglement @xcite , we focus on topsim due to its high correlation with generalization accuracy (ρ = 0.83) @xcite . see appendix a.3 for extended experiments on compositionality metrics and generalization. ## imitation and selection of compositionality we present results for imitation on mixtures of @xmath0. first, we generate 30 expert languages from the referential task, initially considering expert distributions corresponding to evenly-spaced percentiles of topsim, including the minimum and maximum (0.26, 0.43). for example, when @xmath1, we take the lowest, 50 th percentile, and highest-topsim languages. all results are aggregated over 5 random seeds after 2000 training epochs. we find that (1) whether imitators prefer compositional experts depends crucially on the learning algorithm: imitation by reinforcement results in marked compositional selection compared to supervision; and (2) compositional selection also depends on variance of expert topsims, λ entropy regularization coefficient, and number of experts. the distribution of learned expert weights in fig. 2 , as well as imitation validation accuracy curves in fig. c.2, evidence that in imitation by supervision, the empirical mixture is closer to uniform than when imitating by reinforcement. otherwise, when optimizing using reinforcement, the imitator selects more compositional languages. the shape of the expert weight distribution is tempered by the entropy regularization coefficient λ: smaller λ results in greater compositional selec-tion (that is, lower entropy and more negative skew) of the weight distribution (fig. 3 ). at the limit, imitation by supervision results in the highest entropy and the skew that is closest to zero. we then test the effect of expert topsim distribution asymmetry on the learned weights. to do so, for each @xmath2, we generate 10 expert topsim distributions with varying skew, following the procedure outlined in appendix d.2 (when @xmath3, skew is mechanically 0). we find that for both reinforce and supervision, holding k equal, the skew and entropy of the learned expert weight distribution are robust (i.e., not correlated) to the skew of the underlying expert topsim distribution (fig. d.2). this is desirable when imitating by reinforcement and undesirable when imitating by supervision: for example, consider expert topsim distributions [low high high] (@xmath4. in both cases, reinforce will select a high-topsim expert, and supervision will weight all experts equally, that is, supervision is unable to de-select poor topsims. using all expert topsim distributions generated so far (those where topsim ranks are evenly spaced, and those exhibiting varying skews), we investigate the effect of topsim distribution spread, quantified by standard deviation, on the learned weights. in fig. 4 , we note a significant negative effect of expert topsim standard deviation on the degree of compositional selection. that is, the more disperse the expert topsims, the more the imitator can differentiate between and select compositional experts (shown by a more negative skew in learned expert weights). though this correlation is highly statistically significant for both reinforce and supervision, the effect is ∼ 8x greater for rein-force, demonstrating that the spread between expert compositionalities plays a more important role in the degree of selection by reinforcement. finally, selection is less salient as the number of experts increases, seen by the increasing entropies and skews of expert weights (figs. 3 and d.3). results for @xmath5. ## speed-of-imitation may explain compositional selection thus far, we have seen that imitation by reinforcement selects compositional languages. this is likely because higher topsim languages are easier to imitate. we establish a positive and statistically significant relationship between topsim and ease-ofimitation, expanding the explorations in ren et al. (2020); li and bowling (2019); chaabouni et al. (2020) (see appendix c for experimental details). we evaluate ease-of-imitation using @xmath6, after @xmath7, where t is chosen based on validation accuracy convergence. correlations between topsims of 30 expert languages and imitator performance (averaged over three random seeds) are shown in moreover, correlations between topsim and easeof-imitation are stronger than those between expert validation accuracy and ease-of-imitation (table c.1). this suggests that the positive relationship between compositionality and ease-ofimitation is not due to a confound of high validation accuracy. ## discussion having (1) demonstrated a selection of compositional languages in imitation by reinforcement; (2) established a significant correlation between topsim and ease-of-imitation; we offer the following explanation for compositional selection: mode-seeking behavior in reinforcement learning exploits easeof-learning of compositional languages, resulting in a selection of compositionality. while both imitation and ease-of-learning of compositional languages have been instrumentalized in population training, they are engineered in a top-down way: in @xcite , agents imitate the best-accuracy agent, who is algorithmically designated as the teacher; in @xcite , imitation is stopped early to temporally select compositional features. 3 our work, using basic rl principles, proposes an alternative mechanism that selects compositional languages while requiring minimal engineering and assumptions. selection by rl imitation, using the same easeof-learning argument, applies to not only compositionality but also potentially to other traits, e.g., language entropy or message length. that is, rl imitation naturally promotes any learnability advantage among candidate languages without manual intervention, while agnostic to the signaling system. this may then be leveraged alongside communication-based learning in population-based emergent communication, where imitation would persist easy-to-learn linguistic features.