E MERGENT C OMMUNICATION AT S CALE
DeepMind
A BSTRACT
Emergent communication aims for a better understanding of human language evolution and building more efﬁcient representations.
We posit that reaching these goals will require scaling up, in contrast to a signiﬁcant amount of literature that focuses on setting up small-scale problems to tease out desired properties of the emergent languages.
We focus on three independent aspects to scale up, namely the dataset, task complexity, and population size.
We provide a ﬁrst set of results for large populations solving complex tasks on realistic large-scale datasets, as well as an easy-to-use codebase to enable further experimentation 1 .
In more complex tasks and datasets, we ﬁnd that RL training can become unstable, but responds well to established stabilization techniques.
We also identify the need for a different metric than topographic similarity, which does not correlate with the generalization performances when working with natural images.
In this context, we probe ease-of-learnability and transfer methods to assess emergent languages.
Finally, we observe that larger populations do not induce robust emergent protocols with high generalization performance, leading us to explore different ways to leverage population, through voting and imitation learning.
I NTRODUCTION
Language emergence is at the intersection of cognitive science and machine learning.
From a cognitive science view, researchers have been looking at artiﬁcial agents as another expressive species to shed light on the source of linguistic regularities (Wagner et al., 2003; Guo et al., 2019; Chaabouni et al., 2021).
From a machine learning view, language evolution is deemed as a promising direction to shape agents’ representation and design interactive AI (Steels, 1997; Lazaridou et al., 2020).
Most of the literature in this ﬁeld relies on different variants of the Lewis game (Lewis, 1969).
There, a speaker network must describe an object to a listener network, which must then retrieve it among a set of other objects.
To solve the game, the two agents need to settle on a communication protocol.
While deep agents manage to solve the Lewis game, their communication protocols are usually degenerate, lacking core properties of human languages (Bouchacourt & Baroni, 2018;
Chaabouni et al., 2019).
In response to these ﬁndings, several works proposed ad-hoc solutions by constraining the game and agents’ capacity (Kottur et al., 2017; Resnick et al., 2020).
While reducing problem complexity is tempting, it can lead to unexpected outcomes and may miss general language emergence behaviors (Hayes, 1985).
We take a different route and advocate that scaling up communication games is a prerequisite to building interactive AI (Baroni et al., 2017; Sutton, 2019) or modeling language evolution (Barsalou, 2008).
Indeed, contrary to other machine learning communities, the emergent communication ﬁeld mostly relies on small-scale games where only one speaker and one listener communicate about disentangled stimuli, which can hinder the generality of its conclusions.
In this paper, we focus on three scaling dimensions: the dataset, the task complexity, and the population size.
That is, we argue that making populations of deep agents communicate about larger and more realistic datasets and solve more complex tasks, is necessary if in the end we want these agents to interact with us or if we want to model human communication.
We study three properties of emergent languages: generalization, robustness to input noise, and ease of learning over transfer tasks, analyzing different facets of communication protocols.
The proposed scaling up remains computationally tractable, as most of the experiments can be done within a day on a single GPU.
The source code is based on the Jaxline pipeline (Babuschkin et al., 2020).
Overall, our experiments provide a large spectrum of observations, both positive and negative.
First, we found that scaling up the Lewis game quickly entails unstable RL optimization.
We propose KL regularization (Geist et al., 2019) as an effective way to overcome this issue.
Second, we observe that complexifying the task has two positive aspects: it better discriminates between methods and improves the generalization of the learned communication protocol.
Third, we note no correlation between generalization and the widely used topographic similarity metric, which suggests that the latter is not adequate to assess the compositionality of the languages in our more complex setup.
Instead, we take inspiration from the self-supervised learning literature and explore transfer learning as a new evaluation metric.
Fourth, unlike what was observed in human communication (Gary Lupyan, 2010; Raviv et al., 2019a;b), we ﬁnd little to no systematic beneﬁt on emergent languages’ properties when increasing the population size.
Instead, we propose alternative methods to leverage populations, namely voting and imitation among speakers (Hester et al., 2018; Vecerik et al., 2017).
In particular, we show that such population dynamics lead to more robust, productive, and in some cases easy-to-learn languages even when compared to our best seed without population, opening up new research opportunities.
In the end, we expect that these observations, baselines, and good practices would allow the language emergence community to beneﬁt further from deep RL advances and move the ﬁeld closer to its goals of improving representations and interactive systems.
S ETUP 2. D ISCRIMINATION GAME
The discrimination game involves two players, Speaker and Listener , and is decomposed into three sequential stages.
First, Speaker receives a target image x and outputs a message m that is sent to
Listener.
Second, Listener receives m and selects an image ˆ x among a set C of different images containing the target image x .
The set C is called candidates .
Finally, the target image x is revealed to Listener.
If Listener selects the target image, then both players receive a positive reward of 1 , and 0 otherwise.
Speaker and Listener are parameterized by a set of parameters θ and φ respectively.
The message m ( x, θ ) =
( w t ( x, θ ))
T − 1 t =0 is a sequence of length T of words w t ( x, θ ) .
When no confusion is possible, we omit the dependence on x and θ to simplify notations.
A word w t is an element of a ﬁnite vocabulary set W .
For the image selected by Listener ˆ x ( φ, m, C ) , we also omit the dependence on φ , m and C .
Finally, the reward for Listener and Speaker is denoted by R ( x, ˆ x ) =
1 x =ˆ x .
Communication in a population of agents.
A straightforward extension of the standard discrimination game is to train a population of agents.
In this case, given a population of N Speakers and N
Listeners, we sample S Speakers and S Listeners with replacement at each training step to construct
S random pairs; where each Speaker is paired with only one Listener.
All pairs observe the same examples and are trained independently to play the discrimination game as described in the 1-pair setting above.
In our case, S = N so that each agent is trained on average once per step, which allows a fair comparison with the baseline of 1 -pair.
At inference time, we take the ﬁrst P Speakers and Listeners and construct all the possible P 2 pairs to compute the accuracy, by averaging the rewards of all pairs as in (Mordatch & Abbeel, 2018).
Exploiting the population.
The training dynamic of the population of agents described above does not take advantage of the diversity of the population.
Indeed, we treat all agents in the same way.
Yet, one motivation for the effectiveness of the population is the variety between Speakers to invent new structures and between Listeners to avoid over-specialization (Bromham et al., 2015).
Prior works have looked at the impact of training agents with different dynamics (e.g., Ren et al., 2019;
Guo et al., 2019; Cogswell et al., 2019; Li & Bowling, 2019).
In this work, we introduce two different ways to exploit the population; on Speakers’ side, we add imitation, and on Listeners’ side, we allow Listeners to vote to get the ﬁnal prediction.
Mathematical details are found in Sec.
Imitation training consists of two different steps.
First, as described above, Speakers and Listeners are paired randomly for M interaction steps to learn to communicate.
Second, Speakers are trained for 1 imitation step as follows: we select the best Speaker (called “teacher”) among K randomly sampled Speakers without replacement, and use the remaining K - 1 samples as “students”.
We then train all students in a supervised way to imitate the teacher.
We alternate between the interaction and imitation steps.
Both M and K are hyper-parameters (see Appendix A.5).
Finally, Voting is only used at inference time, irrespective of the training mode.
Here, instead of averaging P 2 pairs’ rewards to compute accuracy, we now allow P Listeners to vote to get a unique prediction.
This is inspired by ensemble methods to reduce prediction errors (Polikar, 2012).
More details on imitation and voting are reported in Sec.
For completeness, we also reproduce ease-of-teaching protocol as another effective way to exploit population in Appendix B.4 (Li & Bowling, 2019).
2. D ATASETS AND N EURAL A RCHITECTURES
Datasets.
We use the ImageNet (Deng et al., 2009; Russakovsky et al., 2015), and CelebA datasets (Liu et al., 2015), which respectively contain 1400k and 200k labelled images.
CelebA contains 40 binary attributes per image, such as the presence of glasses, blond hair.
As the ofﬁcial
CelebA splits have non-overlapping identities, it is impossible to perform regular identity classiﬁcation on the test/valid sets.
We thus construct a new split, where we shufﬂe images to have overlapping identities across splits.
In both datasets, each image is center-cropped before being processed by a
ResNet-50 encoder f pretrained on ImageNet with the self-supervised method BYOL (Grill et al., 2020) to extract a representation of size 2048 .
Full implementation details are in Appendix A.6.
Speaker architecture.
Speaker is a neural network that receives an image x and outputs a message m = ( w t ) T − 1 t =0 .
It is composed of a ﬁxed (non-trainable) image encoder f
that transforms x into an embedding f ( x ) , before being projected by a state adapter c θ to an initial state of an
LSTM (Hochreiter & Schmidhuber, 1997), z − 1 ,θ =
c θ ( f ( x )) .
The LSTM receives as input a word embedding e t,θ =
g θ ( w t − 1 ) and outputs the next state z t,θ = h θ ( z t − 1 ,θ , e t,θ ) .
The ﬁrst word embedding e 0 ,θ = g θ ( sos ) is initialized with a start of sequence.
The state z t,θ is then fed to two different heads; the value head v θ estimates the value of the expected reward knowing z t,θ and the policy head π θ computes the probability of the chosen word given z t,θ .
Then, a sampling function s picks the word w t =
s ( π θ ( .
| z t,θ )) .
Finally w t is fed back to g θ to produce the next word and so on until the maximum length T is reached.
At training, the word w t is sampled according to the policy whereas it is greedily selected at evaluation.
Contrary to prior works (e.g., Kottur et al., 2017; Resnick et al., 2020), we do not restrict the channel capacity of agents to spur the emergence of systematic languages.
Instead, we endow Speaker with a message space of size | W | T = 20( | W | = 20 , T = 10 ), signiﬁcantly larger than the number of available training examples.
Listener architecture.
Listener is a neural network that receives Speaker’s message m and a set of image candidates C , containing the target image x .
It outputs the probability over each image ˜ x ∈ C of being the target image.
Listener is composed of an LSTM cell h φ , which is initialized to the null vector z − 1 ,φ .
The message m is decoded by processing the sequence of word embeddings e t,φ = g φ ( w t ) through the LSTM such as z t,φ = h φ ( z t − 1 ,φ , e t,φ ) .
The state z
T − 1 ,φ is then fed to the network p φ to output p m,φ = p φ ( z T − 1 ,φ ) .
In parallel, each image ˜ x is projected through the encoder f
and the network t φ to obtain the image embedding t ˜ x
= t φ ( f (˜ x )) .
Both message and image embeddings are then compared through a score function score ( m, ˜ x, φ ) = cos( p m,φ (cid:107) p m,φ (cid:107) 2 , t ˜ x (cid:107) t ˜ x (cid:107) 2 ) .
The scores over all images are normalized via a softmax to get a probability π φ ( .
| m, C ) .
Finally,
Listener selects an image by taking the best guess according to π φ , i.e. ˆ x ∈ arg max ˜ x ∈C π φ (˜ x | m, C ) .
Architecture details, hyper-parameters and graphical descriptions are provided in Appendices A. and A.4.
All experiments are run over 10 seeds.
2. O PTIMIZATION
Speaker training and loss.
The goal of Speaker is to optimize the message m sent to Listener such that the expected reward of the game is the highest possible.
This can be framed as a sequential decision making problem where the decision is the choice of each word w t .
Therefore, following the policy gradient approach with a baseline (Sutton et al., 2000), we train Speaker’s network by (i) minimizing a value loss L V ( θ ) to make the value head v θ ( z t,θ ) ﬁt the expected reward over a batch
X : L V ( θ ) =
1 |X| (cid:80) x ∈X (cid:80) T − 1 t =
( R ( x, ˆ x ) − v θ ( z t,θ )) 2 , (ii) maximizing the expected reward through minimizing the policy loss L π ( θ ) = 1 |X| (cid:80) x ∈X (cid:80) T − 1 t =0 sg
( R ( x, ˆ x ) − v θ ( z t,θ )) log( π θ ( w t | z t,θ )) , where sg ( . ) is the stop gradient function.
In addition, it is common practice in RL and emergent language literature to minimize an entropy loss L H ( θ ) encouraging to explore other choices of words by Speaker (Mnih et al., 2016; Williams & Peng, 1991; Espeholt et al., 2018).
Finally, several deep RL (Schulman et al., 2015; 2017) and theoretical RL papers (Geist et al., 2019; Vieillard et al., 2020a;b) argued that minimizing a KL loss L KL ( θ ) between the online policy π θ and a target policy π θ instead of or in addition to entropy regularization could be beneﬁcial for better ﬁnal performance as well as stabilizing the learning.
The policy π θ is obtained by doing an exponential moving average of the weights θ over training: θ ← (1 − η ) θ + ηθ where η is the exponential moving average parameter.
The KL minimization encourages the online policy to change slowly and smoothly.
To sum up, the speaker training loss L ( θ ) on a batch of images X is: L ( θ ) =
L V ( θ ) + L π ( θ ) + αL H ( θ ) + βL KL ( θ ) , where α and β are hyper-parameters.
Listener training and loss.
Listener is also trained to maximize the reward but acts by predicting the best guess for the game ˆ x ∈ arg max ˜ x ∈C π φ (˜ x | m, C ) .
For each image x in a batch X , a set of image candidates C is sampled randomly (uniform without replacement over X \ { x } ) chosen in X .
When |C| = |X| , we take C = X .
Finally, Listener’s goal is to retrieve x among C , i.e. outputting a probability π φ (˜ x | m, C )
=
if x = ˜ x and π φ (˜ x | m, C ) = 0 otherwise.
Therefore, we use a multiclass classiﬁcation loss where the correct class is the index of x in the set of candidates C also called
InfoNCE loss (van den Oord et al., 2018): L ( φ ) = − 1 |X| (cid:80) x ∈X log ( π φ ( x | m, C )) .
Imitation training among a group of Speakers.
In a training imitation step, a group of K speakers among the total population of N speakers is sampled without replacement.
Among those K speakers, one speaker plays the role of the teacher and K − 1 play the role of the students.
To choose the teacher, we compute, for each sampled speaker
i , the exponential moving average of the accuracies over each batch on which the speaker i has been trained on, σ i .
Then the teacher is simply the speaker with the highest σ i .
For convenience,we respectively note θ T and θ S the parameters of the teacher and student.
A student θ S is trained on a batch of data X by imitating the messages of the teacher θ T with the following loss: L I ( θ S )
= − 1 |X| (cid:80) x ∈X (cid:80) T − 1 t =0 log π θ S ( w t ( x, θ T ) | x, z θ S ,t ) , where L I ( θ S ) is a cross-entropy loss to encourage a student to output the same words as the teacher.
Listener’s voting at inference time.
At inference time, we can use all listeners ( φ i ) N − 1 i =0 of the population as an ensemble of networks.
Together, they vote for a joint guess ˆ x over a set of candidates
C of images for a message m coming from a speaker.
One simple way consists in averaging the score probabilities of the listeners and taking the best guess of this average.
Formally, for each listener φ
i , each message m and batch X , we have the score probability
π φ i ( .
| m, C ) .
Then the choice ˆ x ( m, C , ( φ i ) N − 1 i =0 ) of the population of listeners for the message m and the set C is the best guess of the average distribution which is ˆ x ( m, C , ( φ i ) N − 1 i =0 ) =
arg max ˜ x ∈C 1 N (cid:80) N − 1 i =
π φ i (˜ x | m, C ) .
More details about how we derive the different losses are found in Appendix A.3.
L ANGUAGE P ROPERTIES
Generalization.
It measures the ability of agents to communicate about never-seen inputs.
To compute generalization, we simply report test accuracy.
In the simple case where inputs are constructed by disentangled attributes, test inputs are new combinations of previously seen attributes at training.
Robustness.
We report agents’ drop of accuracy when faced with noisy inputs relatively to clean inputs at test time.
To construct noisy stimuli, we add a Gaussian noise for each batch, with mean 0 and a standard deviation that is half of the standard deviation of the batch.
Note that we sample different noises for Speakers’ and Listeners’ inputs.
That is, while Listeners are trained to ﬁnd the exact input of Speakers at train time, they are now required to uncover a different input at eval time.
Topographic Similarity (TopSim).
TopSim (Brighton & Kirby, 2006) is used as a proxy for compositionality (e.g., Li & Bowling, 2019; Lazaridou et al., 2018), and it is widely considered crucial for generalization (Fodor & Lepore, 2002; Marcus, 2003).
TopSim tests whether close objects in the input space are described by close messages by computing the Spearman correlation between the pairwise distances in the input and message spaces.
Following prior works, we use the edit-distance and the cosine-distance in the message and input spaces respectively.
Ease and transfer learning (ETL).
It captures how fast and well the emergent language is transmitted to new Listeners performing a distinct task.
It is similar to the ease-of-teaching (Li & Bowling, 2019) to the exception that Listeners are trained to solve a different task from the initial one that the emergent language was optimized for.
This task transfer is inspired by the self-supervised linear evaluation protocol (Chen et al., 2020).
To measure ETL , we take min( N, 5) ﬁxed Speakers from the population after convergence, and feed their deterministic language to 3 newly initialized Lis- teners.
We argmax from Speakers’ distribution to construct the deterministic languages.
Finally, we show how fast and well new Listeners perform in a given task when trained on ﬁxed languages by reporting the training curve for 10k steps.
We look at the ease of learning when Listeners are trained to solve harder discrimination, classiﬁcation, and image reconstruction tasks.
Hence, ETL not only captures the generality of the language to new Listeners but also its ability to transfer to new tasks.
E XPERIMENTS 3. T ASK SCALING UP : I NCREASING THE NUMBER OF C ANDIDATES
In the emergent communication literature, agents are typically required to discriminate between less than 20 objects (Mu & Goodman, 2021) or reconstruct hand-crafted attributes (Rita et al., 2020).
Yet, this simple training setting was shown to sometimes lead to degenerate communication protocols (Bouchacourt & Baroni, 2018).
As highlighted by (Dess`ı et al., 2021; Guo et al., 2021), the
Lewis game is a discrete variant of Contrastive Predictive Coding in the self-supervised learning ﬁeld (van den Oord et al., 2018).
There, it was observed that the intermediate representation quality improved when increasing the number of candidates (He et al., 2019).
We hence look at the impact of the number of candidates |C| on languages generalization and the technical challenges it introduces.
In this subsection, we consider the 1-pair setting trained on ImageNet while varying |C| .
Scaling up task difﬁculty requires to carefully tune optimization.
We ﬁrst train agents without
KL regularization ( β = 0 ) as it is commonly done in the ﬁeld (see e.g., Strub et al., 2017; Cao et al., 2018; Lu et al., 2020, among many others).
As displayed in Fig. 1(a), the training becomes unstable when increasing the number of candidates from 20 to 1024 candidates.
The mean train accuracy drops from 98.9% to 76.0%, and we observe a large variance across seeds.
This high variance demonstrates how crucial it is to run multiple seeds to state robust conclusions.
A remedy against such instability in RL consists in adding a KL regularisation L KL loss between the online policy and a target policy as described in Sec. 2.3.
Such a solution has yet not been explored in the emergent communication ﬁeld.
We add a KL regularization with a coefﬁcient β = 0 .
5 and display the results in Fig. 1(b).
We note that, with this better regularization, 20 and 1024 candidates converge to 99.8% and 98.3% mean accuracy respectively with small variance during training.
Adding L KL thus stabilizes the training and leads to better performances.
We illustrate further the effect of this regularization across multiple sweeps in Appendix B.3.
[Figure]
Scaling up task difﬁculty is necessary to differentiate representations and enhance protocol’s generalization.
In Fig. 2(a), we train 1-pair of agents on different number of candidates ranging from 64 to 1024 and evaluate them on 16 candidates.
3 Note that, in the literature, the number of distractors rarely exceeds a dozen both at train and eval times (Mu & Goodman, 2021; Li & Bowling, 2019).
In such settings, our experiments do not provide any interesting conclusions: all methods are above 99 .
7% test accuracy and within standard deviation.
However, when evaluating the same communication protocols now on harder tasks with 1024 candidates, as shown in Fig. 2(b), we note that these protocols are actually different.
In particular, agents achieve an accuracy of 87 .
25% , 96 . 09% when trained on 64 , 256 , and 1024 candidates respectively, differentiating the various representation’s generalization capacity.
While expected in light of the recent self-supervised results (Chen et al., 2020) - harder training tasks lead to better representation and harder evaluation tasks better discriminate algorithms - these results clearly illustrate how current emergent language experiments may be ill-scaled to have robust conclusions.
We report further experiments in Appendix B.5.
In the following, we use 1024 candidates at train and eval to fully leverage our ﬁndings.
[Figure]
D ATASET SCALING UP : R ETHINKING E VALUATION
In most emergent communication works, deep agents are situated in a simple disentangled environment where objects are one-hot vectors (Kottur et al., 2017; Li & Bowling, 2019).
As a result, the state space rarely goes beyond a few thousands of discrete samples.
Besides, such representations are unambiguous as they are inherently compositional.
We argue that such environments may be too simplistic to reach complex or diverse communication behaviors.
In contrast, the major success stories in machine learning collectively prove the importance of training neural networks on a rich and large amount of data to emulate complex distributions (Brown et al., 2020; Krizhevsky et al., 2012; Sutton, 2019).
Furthermore, if the goal is to reproduce human interaction, it is fundamental to incorporate complex stimuli to develop advanced concepts in language evolution (Miller &
Johnson-Laird, 1976; Barsalou, 2008).
In this subsection, we scale up the input space by using large natural image datasets, namely ImageNet and CelebA. In particular, we investigate further emergent language properties in the same setting of 1-pair of agents while varying |C| at train time.
[Figure]
[Table]
TopSim may fall short with natural images.
We compute TopSim using the pretrained image logits on ImageNet and CelebA as input representation while varying the task complexity at train, |C| .
Results are provided in Fig. 3(left).
We do not observe a consistent pattern between TopSim and |C| .
Furthermore, we note a non-signiﬁcant (p-value > 0.19) Spearman correlation of − 0 .
09 and 0 .
08 between TopSim and generalization for ImageNet and CelebA respectively.
Although pretrained logits are excellent linear features (Grill et al., 2020), they may not be compositional.
We thus reiterate the evaluation by using the attributes from CelebA as input representation to compute TopSim .
In that case too we do not note any signiﬁcant correlation (p-value = 0.13).
Similar ﬁndings have been observed in (Andreas, 2019; Chaabouni et al., 2020).
This could be due to different reasons.
First, agents could be communicating in a compositional way, to support good generalization, by encoding unexpected compositional features that are not labeled, like forehead shape or smile angle.
That is, human-labeled features may differ from the agents’ ones.
Second, TopSim relies on strong assumptions, such as the chosen distance, and the use of a linear correlation.
Such hypotheses may not hold when moving from artiﬁcial to realistic data.
Finally, due to the large channel capacity, agents could be using synonyms to generalize with low TopSim values.
In sum, our results demonstrate that
[Table]
TopSim is not a good predictor of communication protocol generalization for large-scale settings.
Instead, we investigate the information content via ETL .
ETL is a robust metric to evaluate languages with natural images.
ETL , contrary to TopSim , evaluates how useful the emergent language is for a target task beyond overﬁtting.
As such it is an important metric, since the goal of emergent communication is to endow agents with communication skills, rather than solving a speciﬁc game.
Results are reported in Table 1 over distinct tasks on CelebA.
We observe that, even though ETL only looks at the information content and not at compositionality, it is a better predictor for language generalization performances than TopSim , with a strong and signiﬁcant correlation > 0.90 (p-value ∼ 0) for all considered tasks.
As a visual sanity check, we generate predicted images of the reconstruction ETL task in Fig.
3(right).
There, a new
Listener is trained to minimize an l 2 loss with the target images given the precomputed messages.
We observe that gender, makeup, and hair color are fairly encoded, while others such as the smile are not.
More samples and training details are available in Appendix B.1.
Interestingly, some ETL tasks, like discrimination or reconstruction, do not depend on (human-)predeﬁned input representations, and are all more robust to channel capacity and linearity assumptions, as opposed to TopSim , which makes ETL a more general evaluation metric.
Zero-shot dataset transfer highlights emergent protocols’ differences and limits.
Scaling up to natural image opens a large diversity of visual distributions that we can leverage by using zeroshot dataset transfer (Yogatama et al., 2019; Lambert et al., 2020).
We hence measure how generic and high-level the emergent protocols are.
We take agents trained on ImageNet with |C| = 1024 , and evaluate them on CelebA in a zero-shot fashion.
There, the mean accuracy dropped from 95 .
96 to 36 .
73 , and the standard deviation raised from 0 .
49 to 12 . 86 .
Noticeably, the most different agents obtain a zero-shot accuracy of 27 . 88 and 68 . 49 , while their initial ImageNet accuracy only differed by 0 . 23 .
Firstly, such a standard deviation gap suggests that different emergent protocols have emerged between agent pairs.
Secondly, the accuracy drop indicates that the emergent protocols remain speciﬁc to the initial data-distribution, and no systematic language has yet emerged.
3. P OPULATION SCALING UP : E XPLOITING THE POPULATION DYNAMIC
The emergent communication literature often considers a single pair of agents (Lazaridou et al., 2017;
Foerster et al., 2016).
However, there is evidence in the multi-agent literature that such a setting may lead to extreme co-adaptation and overﬁtting (Lanctot et al., 2017).
One counter-measure consists of sampling agents within a population (Jaderberg et al., 2018).
From a linguistic perspective, different large-scale corpora analyses and human simulations support the importance of the population in shaping our languages (Gary Lupyan, 2010; Bromham et al., 2015; Raviv et al., 2019a;b).
In this context, Raviv et al.
(2019a) show that larger populations develop more systematic languages through human experimentation.
Nonetheless, a few artiﬁcial simulations tentatively consider population sizes up to 32 agents, but with mixed results about their advantage (Mordatch & Abbeel, 2018; Tieleman et al., 2018; Graesser et al., 2019; Cogswell et al., 2019).
In this part, we consider up to 100 agents.
Speciﬁcally, we explore the impact of the population size on languages properties when we deal with more complex tasks ( |C| =1024 at train and eval) and realistic inputs (CelebA & ImageNet) while varying the population size N ∈ { 1 , 10 , 50 } pairs.
The best single-pair seed should be the baseline against population.
A few works in the emergent communication framework looked at the beneﬁt of the population by comparing 2 N -agent’s performances to the 2 -agent baseline (Cogswell et al., 2019).
However, the former introduces × N more parameters, which could be responsible for its slight observed beneﬁt.
Here, we consider instead the “best 1 pair” setting to investigate whether it is computationally advantageous to train a population of size N rather than training N independent pairs.
That is, for a given computational budget, we ask whether it is beneﬁcial to train agents within a population as opposed to N pairs in parallel, as [Table] also done in (Tieleman et al., 2018).
“Best 1 pair” is constructed by selecting the best seed among the 10 seeds of “1 pair”, based on validation accuracy, offering a fairer baseline speciﬁcally for the “10 pairs” setting.
Results are shown in Table 2 and Fig.
We observe different behavior of “bestpair” across datasets.
Particularly, if it always achieves better generalization and ETL discrimination compared to “1 pair”, it shows, for ImageNet only, the worst Robustness and ETL classiﬁcation.
We hypothesize that in that speciﬁc case, the emergent protocol overspecializes to the task.
Still, “best 1 pair” outperforms the standard baseline “1 pair” 6/8 times, which makes it a stronger baseline.
In the following, we thus always compare the population setting with “best 1 pair” for a fair evaluation.
Population size does not lead to a systematic advantage.
We look at the language generalization and robustness performances when increasing the population size.
Results in Table 2 do not show a clear trend between population size and these metrics.
For example, for both datasets, “50 pairs” achieves lower Generalization and Robustness than “10 pairs”.
Furthermore, “best 1 seed” always achieves better Generalization compared to the largest population setting “50 pairs”.
In Fig.
4, we look at ETL for a further investigation of the languages’ properties.
CelebA results exhibit a beneﬁt of the population.
However, this beneﬁt does not always correlate with population size where “ pairs” outperforms “50 pairs” on the discrimination task.
This non-systematic beneﬁt is also found when experimenting with ImageNet.
Again, our results suggest no beneﬁt of the population size in the current setting.
We thus explore new approaches to leverage the population dynamics and improve the representation of the emergent protocol in the following.
A better use of the population is needed to see larger beneﬁt.
As population size does not improve the emergent language in itself, we took advantage of the richness of the population by considering imitation at training and vote at inference.
In Table 2, we observe that both imitation and vote introduce a systematic improvement compared to the standard training in a population and lead to better performances compared to “1 pair” in all metrics.
Moreover, when considering the stronger baseline “best 1 pair”, the latter only outperforms the imitation and vote settings 1/4 times ( Generalization for ImageNet).
However, as noted previously, this case is an over-specialized protocol.
Furthermore, if both imitation and vote introduce a systematic improvement of the protocol, their beneﬁt is even more noticeable for distribution-shifted settings.
For example, the imitation forpairs adds a relative improvement of 12.8% in Robusteness compared to 2.3% for Generalization on
CelebA. The same observation holds for the vote but with a lower degree.
Note that if vote induces only a slight beneﬁt, it does not incur any training cost.
When considering ETL on discrimination in Figs. 4(a)&4(c), we observe that imitation leads to a systematic improvement.
In CelebA, if the standard population was already beneﬁcial, adding the imitation strengthens the results.
In ImageNet, “10/50 pairs + imitation” perform similarly at the end of training than the overspecialized “best 1 pair” while having more stable optimization and it is considerably better than the standard training w/ and w/o population.
Yet, the results are more nuanced when considering ETL over classiﬁcation in Fig.
4(b)&4(d).
Speciﬁcally, while CelebA experiments suggest an advantage when training a population of agents w/o imitation as opposed w/ imitation or “1 pair”, this pattern is absent with ImageNet.
There all settings with population are on par.
In sum, both considered dynamics outperform in most cases the strong “best 1 pair” baseline (7/8 times), highlighting their viability.
D ISCUSSION AND C ONCLUSION
The emergent communication framework has been extensively studied for decades before the successes of deep RL.
In this context, many works have revisited this framework through deep RL settings (e.g., Kirby 2002 vs. Ren et al.
2019, Myers-Scotton et al.
2002 vs. Graesser et al.
2019, or
[Figure]
Batali 1998 vs. Choi et al. 2018).
However, the experimental settings have barely evolved for twenty years.
For instance, Kirby (2002) and Ren et al.
both used a binary input vector of size eight, and only a few papers try to go beyond such artiﬁcial input spaces or simple tasks (Lazaridou et al., 2017; Havrylov & Titov, 2017; Dess ` ı et al., 2021).
While computational constraints may have been a limiting factor, we show that our setting is possible with widely available hardware in Appendix B.2.
For example, Table 1 requires approximately 400 hours of compute, i [Figure] e. 16 GPUs for a single day.
Compared to other communities, this is equivalent to medium-scale studies in vision or NLP.
We argue that it is ﬁnally time for the emergent language community to scale up!
In this spirit, we start clearing the way by identifying some scaling up challenges: optimization instabilities, ill-adapted metrics, or lack of population synergy.
We show how to face these new difﬁculties by proposing: alternative optimization (KL regularization), new evaluation protocols (ETL, zero-shot dataset transfer, best seed baseline), and new dynamics to leverage populations (imitation, voting).
There are different theories about the necessity of complex tasks to model human communication (e.g., (Barrett & Skyrms, 2017) vs. (Bickerton, 2015; Dupoux, 2018)).
Hence, we endorse here Bickerton’s view and adopt performance-inspired solutions to scale up such as KLregularization and imitation.
An interesting future debate is whether our ﬁndings could inﬂuence the status quo of similar research in human communication.
Although we only examine three scaling up dimensions, many other directions can be pursued in the future: using larger or different architectures to improve agent abilities (Desai & Johnson, 2021), experimenting with multimodal data like video or sounds for more realistic stimuli (Arandjelovic &
Zisserman, 2017), or building symmetric communication channels to have emergent dialogues (Gao et al., 2019).
Another frontier would be to incorporate interaction within environments to ground language into actions (Bisk et al., 2020).
As such embodiment is crucial for human language understanding (Harnad, 1990; Barsalou, 2008), we may start exploring small grid world (Kaji´c et al., 2020) before scaling up to 3D-environments (Abramson et al., 2020).
R EPRODUCIBILITY S TATEMENT
In this paper we ensure the reproduciblity of the different ﬁndings through several (and voluntary redundant) ways, namely:
• The task is detailed in Section 2.1 and we provide a visual sketch in Appendix A.1.
• We use open-source datasets (ImageNet and CelebA).
Also, dataset processing is explained, with pseudo-code to reproduce our splits, in Appendix A.6.
• The speaker and listener architectures are ﬁrst explained in Section 2.2, we then detail model size values, and provide visual sketches in Appendix A.2.
The speciﬁcs for image reconstruction for ETL are in Appendix B.1 with a pseudo-code for the reconstruction head.
• The optimization is ﬁrst described in Section 2.3.
The equations are then fully detailed in Appendix A.3.
• All training and evaluation hyperparameters are listed in Appendix A.4.
• Through the paper, we voluntary provide training curves, test curves, and ﬁnal scores to have a global view of the training trends.
• We listed computation time and memory footprint for multiple experiments and multiple hardware in Appendix B. •
The source code should be released upon paper acceptance.
A CKNOWLEDGEMENT
We would like to thank Will Dabney, Remi Munos, Karl Tuyls, Nathalie Beauguerlange as well as the rest of the DeepMind Paris team for their continuous support.
We would also like to thank
Marco Baroni, Eugene Kharitonov, Olivier Pietquin and Mathieu Rita for their discussions and helpful feedback at the different stages of the project.
Finally, we thank Alison Reid and Saran
Tunyasuvunakool for their help in open-sourcing the codebase
A RCHITECTURE DETAILS
Speaker The speaker’s network architecture is composed of several components to transform the target image x into a message m = ( w t ) T − 1 t =0 :
• The encoder f is a ﬁxed Resnet- 50 architecture that has been previously trained on Imagenet with the BYOL algorithm.
The resulting embedding f ( x ) is of size 2048 .
• The RNN h
θ used is an LSTM of hidden size 256 .
Therefore the core state z t,θ is of size 512 .
• The core-state adapter c θ is a linear layer with input size 2048 and an output size of 512 that allows to transform the embedding f ( x ) into an appropriate core state z − 1 ,θ =
c θ ( f ( x )) .
We split z − 1 ,θ into two equal parts to obtain the initial hidden state z h, − 1 ,θ and the initial cell state z c, − 1 ,θ .
• The word embedder g θ associates to each discrete symbols in W ∪ { sos } an embedding of size 10 .
• The value head v θ ﬁrst selects the hidden part z h,t,θ of the core state z t,θ and then applies a linear layer of output size 1 .
• The policy head π θ ﬁrst selects the hidden part z h,t,θ of the core state z t,θ
and then applies a linear layer of output size |W| to obtain logits l t,θ .
Then a softmax layer transforms the logits into the output distribution π θ ( .
| z t,θ ) .
Listener The listener’s network architecture is
composed
of several
components
to transform the message m and the input image ˜ x into a score score ( m, ˜ x, φ ) :
• The encoder f is a ﬁxed Resnet- 50 architecture that has been previously trained on Imagenet with the BYOL algorithm.
The resulting embedding f ( x ) is of size 2048 .
• The RNN h φ used is an LSTM of hidden size 512 .
Therefore the core state z t,φ is of size 1024 and is composed of an hidden state z h,t,φ of size 512 and a cell state z c,t,φ of size 512 .
• The word embedder g φ associates to each discrete symbols in W an embedding of size 10 .
• The target projection t
φ is a linear layer with output size 256 .
• The core-state projection p φ is a linear layer with output size 256 .
As explained in Sec.2.2, the score function is deﬁned as score ( m, ˜ x, φ ) =
cos( p m,
[Equation]
The scores over all images are normalized via a softmax to get a probability π φ ( .
| m, C ) such that:
[Equation]
Finally, the listener selects an image by taking the best guess according to π φ , i.e.
ˆ x ∈
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
A. O PTIMISATION DETAILS
Speaker training and loss The goal of a speaker, parameterized by θ , is to optimize the message m sent to a listener, parameterized by φ , such that the mean reward of the game is the highest possible.
This can be framed as a sequential decision making problem where the decision is the choice of each word w t .
Thus, each word is sampled from a parameterized stochastic policy π θ ( .
| x, ( w k ) t − 1 k =0 ) that depends on the image x and past words ( w k ) t − 1 k =0 , where for t =
we have ( w k ) − 1 k =0 = ∅ .
Then, the goal is to maximize the expected reward J ( θ, φ ) by ﬁnding the best policy π θ :
[Equation]
where the expectation E x ∼ ρ is over the dataset of training images and the expectation E π θ ,x is over all possible sequences ( x, ( w t ) T − 1 t =0 ) that can be generated by π θ starting from x .
For a given image x , we deﬁne the value V π θ ( x ) =
E π θ ,x
[ R ( x, ˆ x )] as the expected reward for a given image, here we left the dependence over the choice of the listener ˆ x for ease of notations as the speaker cannot act on it.
For a parameterized policy π θ , we can use the policy gradient theorem to obtain the gradient:
[Equation]
where V π θ t − 1 ( x ) =
E π θ ,x (cid:2) R ( x, ˆ x ) | ( w k ) t − 1 k =0 (cid:3) is the value conditioned on all the information revealed at time t − 1 .
For our particular choice of speaker network, we encode the policy π θ ( .
| x, ( w k ) t − 1 k =0 ) and the value V π θ t − 1 ( x ) with the policy head π θ ( .
| z t,θ ) and the value head v θ ( z t,θ ) respectively.
Doing so is legitimate because by construction of our recurrent speaker network the embedding z t,θ is a function of the image x and the past words ( w k ) t − 1 k =0 .
We train then our speaker network by minimizing a value loss L V ( θ ) to make v θ ( z t,θ )
ﬁt V π θ t − 1 ( x ) over a batch X of images:
[Equation]
and a policy loss function L π ( θ ) to maximise the expected reward:
[Equation]
where sg ( . ) is the stop gradient function.
If we assume that v θ ( z t,θ ) ﬁts perfectly V π θ t − 1 ( x ) , one can easily verify that ∂ θ L π ( θ, x ) is an unbiased estimate of − ∂ θ V π θ ( x ) therefore ∂ θ L π ( θ ) is also an unbiased estimate of − ∂ θ J
( θ, φ ) .
In addition to following the gradient ∂ θ V π θ ( x ) , it is common practice in RL and emergent language literature to maximize an entropy term encouraging to explore other choices of words by Speaker:
[Equation]
In practice, a sampled version of the negative entropy L H ( θ ) is minimized with the speaker network:
[Equation]
Recently
[Equation]
several deep RL (Schulman et al., 2015; 2017) and theoretical RL papers (Geist et al., 2019; Vieillard et al., 2020a;b) argued that minimizing the KL between the online policy π θ and a target policy π θ instead of or in addition to entropy regularization could be beneﬁcial for better ﬁnal performance as well as stabilizing the learning.
Generally, π θ is an older policy or an exponential moving average of past policies.
Therefore
[Equation]
we also consider the following KL regularization
KL ( π θ , π θ ) :
[Equation]
In practice
[Equation]
the policy π θ is obtained by doing an exponential moving average of the weights θ over training.
Then, a sampled version of the KL is minimized L KL ( θ ) with our speciﬁc speaker network:
[Equation]
To sum up, the speaker training loss L ( θ ) on a batch of images X is:
[Equation]
where α and β are hyper-parameters.
Listener loss One important detail in Sec.
2.3 is that for each image x in a batch X , a set of image candidates C ( x, X ) is sampled randomly (uniform without replacement over X \ { x } ) chosen in X .
We omitted the dependencies of C ( x, X ) on x and X in the main text for ease of reading.
In addition, as explained in Sec. 2.3, the listener loss is a multi-class classiﬁcation loss where the correct class is the index of x in the set of candidates C also called InfoNCE loss (van den Oord et al., 2018):
L
[Equation]
log (
[Equation]
φ ( x | m, C )) .
This can be rewritten more explicitly:
[Equation]
Imitation training among a group of speakers.
In a training imitation step, a group of K speakers among the total population of N speakers is selected.
Among those K speakers, one speaker is going to play the role of the teacher and K − 1 are going to play the role of the students.
To choose the teacher among the K speakers, we use as metric the exponential moving average of the accuracies over each batch on which a given speaker has been trained on.
To be more precise, let θ i be the i -th speaker and X the present batch on which θ i has just been trained on, then σ i is updated according to the following rule:
[Equation]
and is the exponential moving average of coefﬁcient µ of the accuracies over each batch on which speaker i has been trained.
Then the teacher is simply the speaker with the highest σ i among the K speakers.
Now for convenience, let us note θ T the parameters of the teacher and θ S the parameters of a student.
A student θ S is going to be trained on a batch of data X by imitating the messages of the teacher θ T with the following loss:
[Equation]
The loss L I ( θ S ) is a cross-entropy loss that encourages the student to output the same words as the teacher.
Optimizer hyper-parameters Each member of the population (listener or speaker) uses its own optimiser.
All optimisers are Adam optimisers (Kingma & Ba, 2015) with the same set of hyperparameters:
• learning rate: 1 e − 4 , • β 1 : 0 . 9 , • β 2 : 0 . 999 , • (cid:15) : 1 e − 8 .
For the speakers’ loss, we use α = 0 .
0001 for the entropy regularization and β = 0 .
5 for the KL regularization.
See Sec.
B.3 for more details about the impact of these parameters.
A. H YPER - PARAMETERS
We use the same hyper-parameters across the different settings.
When experimenting with ImageNet vs. CelebA, we only vary the number of maximum steps.
Speciﬁcally, we use 300k maximum steps for CelebA (we already start observing some overﬁtting with this value) and 900k maximum steps for ImageNet.
In all cases, we select the best checkpoint for evaluation with respect to the listener loss at validation time.
For robust evaluation, we average the scores over 5 epochs to have different candidates for a given target and get a score less dependent on the sampling of the candidates.
Unless mentioned otherwise, the remaining training hyper-parameters are reported in Table
[Table] [Table]
A. I MITATION HYPER - PARAMETERS
In all our experiments
[Table]
we perform a grid search over the parameters of M (number of interaction steps) and K (number of sampled speakers at the imitation step).
For the case of a population of
N = 10 , K is chosen from { 1 , 4 , 9 } and M from { 10 , 50 , 100 } according to the validation accuracy.
For N = 50 , K is selected from { 4 , 9 , 24 } and M from { 1 , 10 , 50 } .
Table 4 shows the selected hyper-parameters ( K, M ) for the different settings.
[Table]
A. D ATASETS DETAILS
ImageNet ILSVRC2012, also known as ImageNet (Deng et al., 2009; Russakovsky et al., 2015) is among the historical largest natural RGB image dataset.
It mostly contains images of animals and objects over 1000 labels, e.g., camel, ostrich, hourglass, or bow tie.
In our experiments, we use 99% of the ofﬁcial train set for training, i.e., 1300k images, the last 1% of the train set for validation, i.e., 13k images, and the ofﬁcial validation set as our test set (i.e., 50k images).
CelebA CelebA is a large natural RGB dataset (Liu et al., 2015), which contains image of celebrity faces over 10,177 identities.
Noticeably, each face also contains 40 binary attributes, e.g., glasses and hair.
Such attributes are interesting to compute, for instance, language topography.
Although there is an ofﬁcial CelebA split, there are no overlapping identities between the train, validation, and test set.
We thus reshufﬂe the split as described in Listing 1.
In the end, we respectively obtain 169,444 training, 16,669 valid, and 16,486 test images.
Image Processing In both datasets, we resize images to 256 pixels along the shorter side using bicubic resampling before applying a 224 x 224 center crop.
We normalize the color channels subtracting the mean color and dividing by ImageNet std.
We then use the ResNet-50 encoder pretrained on ImageNet with the self-supervised method BYOL (Grill et al., 2020) to extract the ﬁnal representation of dimension 2048 .
Listing 1: CelebA Splits
SPLIT RATIO =
# the r a t
i o t r a i n : v a l i d i s 5: def c e l e b a s p l i t s ( data ) : ””” S p l
i t the data in t h r e e s e t s with overlapping i d e n t i t
i e s ”””
# I n i t i a l
i z e v a r i a b l e
s
i m a g e t r a i n , image valid , i m a g e t e s t =
[ ] , [ ] , [ ] c o u n t e r l a b e l = c
o l l e c t i o n s .
Counter ( ) for data in d a t a s e t .
values ( ) : l a b e l = data [ ’ l a b e l ’ ]
# l a b e l encode the face i d e n t
i t y count =
c o u n t e r l a b e l [ l a b e l ] i f count > 0 and count % SPLIT RATIO =
= 0:
# We e q u a l l y s p l i t v a l i d and t e s t by Even and Odd l a b e l s i f
l a b e l % 2 == 0: image valid .
append ( data ) e l s e : i m a g e t e s t . append ( data ) e l s e : i m a g e t r a i n .
append ( data ) c
o u n t e r l a b e l [ l a b e l ] +=return i m a g e t r a i n , image valid , i m a g e t e s t
B A DDITIONAL R ESULTS AND A NALYSIS B. F ACE RECONSTRUCTION
To visualize some of the features encapsulated in the messages used by the agent, we propose a face reconstruction procedure.
For a single speaker, parameterized by θ , and given an initial image x ∈ X , we produce a message m = ( w t ) T − 1 t =0 .
We feed the messages to a listener-like architecture, consisting of an embedding layer of size 10 followed by an LSTM with 512 units.
We keep the last output of the LSTM, h T − 1 as a message representation.
We then pass the message through a deconvolutional architecture, similar to the one in Radford et al.
(2016), but without batch normalizations.
Pseudocode for the architecture is provided in Listing 2.
The full network is trained by minimizing the (cid:96) 2 loss between the input image and the reconstructed one.
To optimize the loss, we use AdamW (Kingma & Ba, 2015) with a batch size of 128 , a learning rate of 3 · 10 − 4 , β 1 = .
9 , β 2 = .
9 , ε = 10 − 8 and a weight decay of 0 .
We clip gradients by norm with a clipping threshold of 500 , and skip gradient updates with norm superior to 2000 .
In Table 1, we report reconstruction losses for different number of candidates |C| .
The reconstruction loss is computed as a squared error, summed over both spatial and channel axes.
Figure 8 displays examples of input images, and corresponding reconstructions for two discrimination games with different number of candidates.
Optimally, each reconstruction should converge to the average face, given the corresponding message.
The more features a message contains, the better reconstructions should be.
Note ﬁrst that reconstructions are far from perfect, and much worse than reconstructions that an auto-encoder trained end to end on images would provide: ﬁrst, solving the discrimination game does not require fully reconstructing the input image, but only capturing enough features to identify uniquely the image in a batch of candidates; second, the discrete nature of the message, its limited size, and the fact that it is learnt using reinforcement learning act as strong bottlenecks, that prevents messages from containing all the information about input images.
Nonetheless, as qualitatively shown in Figure 8, messages contain semantic information about inputs images.
For instance, hair color, gender, or open mouth are mostly preserved throughout the reconstruction process.
Other features, such as face orientation, skin tone, or age are mostly ignored in messages.
Furthermore, both quantitative and qualitative difference are visible when going from low number of discriminators to high number of candidates.
Quantitatively, going from 16 tocandidates improves the reconstruction error from 2448 ± 16 to 2351 ± 14 (Table 1).
Qualitatively, messages with 1024 candidates seem to contain information about presence of eye-glasses (top left image), as well as presence of head cover (bottom right image), that are absent in messages from the
Speaker trained with 16 candidates.
Listing 2: Face reconstruction Head def upsample 2d ( x , f a c t o r : i n t = 2 ) : bs , height , width , channels = x .
shape x = image .
r e s i
z e ( x , ( bs , h e i g h t ∗ f a c t o r , width ∗ f a c t o r , channels ) , method= ’ n e a r e s t ’ ) return x def i m g r e c o n s t r u c t
i
o n ( x ) : ”””Take the LSTM output , and turn i t i n t o an image . ”””
# P r
o j e c t the f l a t embedding i n t
o a 2d t e n s o r of dim 4 x4x x = nn .
Linear (4 ∗ 4 ∗ 128)( x ) x = x .
reshape ( ( x . shape [ 0 ] , 4 , 4 , 128))
x = nn .
r e l u ( x ) x = upsample 2d ( x ) # 8x x
=
nn .
Conv3x3 (64 , w i t h b i a s =
False , padding=”VALID” )
( x ) x = nn .
r e l u ( x ) x = upsample 2d ( x ) # 16 x x = nn .
Conv3x3 (32 , w i t h b i a s =False , padding=”VALID” )
( x ) x = nn .
r e l u ( x ) x = upsample 2d ( x ) # 32 x x = nn .
Conv3x3 (16 , w i t h b i a s =
False , padding=”VALID” )
( x ) x = nn .
r e l u ( x ) x = upsample 2d ( x ) # 64 x x = nn .
Conv3x3 (16 , w i t h b i a s =
False , padding=”VALID” )
( x ) x = nn .
r e l u ( x ) x = nn . Conv3x
B. C OMPUTATIONAL REQUIREMENTS
As mentioned in the article, our approach remains computationally tractable with widely available hardware.
Table 5 summarizes these requirements to reproduce our experimental setup.
B. I MPACT OF THE KL REGULARIZATION ON TRAINING STABILITY
We showed in the main paper how training a pair of agents to solve a complex task (a discrimination game with 1024 candidates) becomes unstable when using common optimization algorithms.
In this section, we look at the training curves when agents are trained with and without KL regularization, while varying the entropy coefﬁcient α ∈ { 10 − 4 , 10 − 3 , 10 − 2 , 10 − 1 } .
Results are shown with a population of size 1 (Figure 9) and 10 (Figure 10).
In both cases, agents are trained to solve the complex discrimination task with 1024 candidates.
First, we observe that the lower α is, the more crucial applying a KL regularization is.
For example, in Figure 9(a), we go from a chaotic setting that converges to an accuracy of ∼ 40% when the
KL coefﬁcient β = 0 to a signiﬁcantly more stable optimization with almost perfect accuracy when β ≥ 0 .
Second, we observe that, if the KL regularization is useful for stable optimization, it comes at the cost of the rapidity of convergence.
This is clearly shown in Figures 9(c) and 10(c), where we observe that the larger β is, the slower the convergence is.
In other words, one should select the best β optimizing the stability/rapidity trade-off.
Third, in the presence of a KL regularization, training stability depends less on the entropy coefﬁcient α .
Indeed, we observe a stable and high training
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
curves for α ∈ { 10 − 4 , 10 − 3 , 10 − 2 } .
However, training without KL regularization (blue curves with β = 0 ) is very sensitive to the value of α and the most stable case is observed for α = 0 .
Still even in this best case without KL regularization, having a β > 0 is useful as shown in Figure 11.
[Figure] [Figure]
[Table]
B. S TUDY OF THE RESETTING ON THE C ELEB A DATASET
In this work
[Table]
we argue that training deep agents to communicate in a population beneﬁts from exploiting its richness, as opposed to a standard training (Mordatch & Abbeel, 2018).
In this context
[Table]
we introduced imitation and voting mechanisms.
However
[Table]
other prior works have stipulated a similar argument by focusing instead on the idea of the expressivity/compressibility trade-off (Kirby et al., 2014).
The latter attests that our language is shaped by the two competing pressures of expressivity and compressibility.
In practice, there were different methods to implement this tradeoff in deep agents communication such as iterated learning (Ren et al., 2019), cultural transmission (Cogswell et al., 2019)
[Table]
or ease-of-teaching (Li & Bowling, 2019).
In this section, we compare imitation and voting to the ease-of-teaching.
5 To encourage languages’ ease-of-teaching, Li & Bowling (2019) trained deep agents to solve a discrimination task, while periodically resetting listeners.
In this section
[Table]
we study the impact of this baseline on the different emergent languages’ properties introduced in the main paper.
Speciﬁcally, we consider 3 additional settings: (1) “1 pair + reset” that consists of resetting the only listener, (2) “10 pairs + reset” where we reset a randomly selected listener among the 10 available ones, and (3) “50 pairs + reset” where we reset a randomly selected listener among the 50 available ones.
In all cases, resetting takes place every 51 k steps.
Note that if (1) is identical to Li & Bowling (2019) setting, (2) and (3) present 10 and 50 speakers respectively unlike 1 speaker as it is the case in (Li & Bowling, 2019).
We observe in Table 6 that resetting has a noticeable beneﬁt only in the case of “1 pair” on the generalization and robustness of the languages.
In particular, we note an average Generalization of 89.00% for “1 pair” vs. 92.21% for “1 pair + reset” and an average Robustness of 18.34% vs. 35.82%.
However, there is no systematic improvement when agents are trained in a population, (“10 pairs” vs. “10 pairs + resetting”) and (“50 pairs” vs. “50 pairs + resetting”).
This is in line with Li &
Bowling (2019)’s results which show that an abrupt change during training leads to better results.
Still, “1 pair + reset” does not systematically improve the emergent languages’ properties compared to the population when imitation and voting are at play.
However, as mentioned in the original paper, the purpose of resetting is not to boost agents’ generalization or robustness, but instead to incentivize agents to develop easy-to-transmit languages.
In the following, we look at ETL shown in Figure 12.
Here, adding resetting does not lead to a signiﬁcant change showing curves almost identical to the ones with standard training in all cases.
Overall, our results suggest that resetting listeners is only beneﬁcial in the one-pair agents in some cases.
Furthermore, resetting does not induce faster or better easy-to-learn communication protocols over transfer tasks compared to the standard training of the Lewis game (with or without population).
[Figure]
B. I MPACT OF THE NUMBER OF CANDIDATES AT TRAINING AND EVALUATION TIME
In this part we look the impact of task complexity at train (Figure 14) and evaluation (Figure 13) times, by varying the number of candidates |C| .
The results conﬁrm our ﬁndings in the main paper.
That is, for a ﬁxed train |C| , the harder the evaluation task is, the lower the test accuracy is.
Still, if the task is complex enough at train time (e
[Figure]
g., |C| = 1024 ), agents reach overall higher accuracies for all evaluation settings, conﬁrming the importance of hard training tasks to see the emergence of communication protocols with good generalization performances.
Also, similarly to our results in the main paper, we observe in Figure 14 that harder evaluation task is necessary to discriminate between the communication protocols.
[Figure] [Figure]
[Table] [Table]
B. O UT - OF - DISTRIBUTION GENERALIZATION : THE O FFICIAL C ELEB A SPLIT
In the main paper
[Table]
we consider a new CelebA split where train and test sets have overlapping identities.
This allows us to test in-distribution performances.
In this Subsection, we look at out-ofdistribution generalization considering the ofﬁcial CelebA split, where train, validation, and test sets include distinct identities.
In other words, we now consider a harder form of generalization by testing agents on never-seen identities at training.
Results are shown in Table 8.
We observe a similar pattern to the in-distribution results shown in Table 2.
That is, we do not observe a beneﬁt for the population, where “best 1 pair” achieves better unseen-generalization than “10 pairs” and “50 pairs”.
However, when considering imitation and voting, agents develop languages that generalize better to never-seen identities.
The setting “50 pairs + imitation + vote” attains the best performance, with a 93.75% success when communicating about previously unseen identities at training time.
[Table]