Revisiting Populations in Multi-Agent Communication
Anonymous Author(s)
AffiliationAddress email
Abstract
Despite evidence from sociolinguistics that larger groups of speakers tend todevelop more structured languages, the use of populations has failed to yieldsignificant benefits in emergent multi-agent communication.
In this paper wereassess the validity of the standard training protocol and illustrate its limitations.
Specifically, we analyze population-level communication at the equilibrium insender-receiver Lewis games.
We find that receivers co-adapt to senders theyare interacting with, which limits the effect of the population.
Informed by thisanalysis, we propose an alternative training protocol based on “partitioning” agents.
Partitioning isolates sender-receiver pairs, limits co-adaptation, and results in a newglobal optimization objective where agents maximize (1) their respective "internal"communication accuracy and (2) their alignment with other agents.
In experiments,we find that agents trained in partitioned populations are able to communicatesuccessfully with new agents which they have never interacted with and tend todevelop a shared language.
Moreover, we observe that larger populations developlanguages that are more compositional.
Our findings suggest that scaling up topopulations in multi-agent communication can be beneficial, but that it mattershow we scale up.
Introduction
Uncovering the mechanisms that underlie our ability to communicate using language is an importantstepping stone towards developing machine learning models that are capable of coordinating andinteracting via natural language.
Over the last few years, there has been increasing interest insimulating the emergence of language using artificial agents trained with reinforcement learning tocommunicate to achieve a cooperative task [33].
Typically, agents are trained to perform a variantof the Lewis signaling game [37, 51] wherein a sender emits a message describing an object and areceiver attempts to reconstruct the object based on the description.
This line of work has applicationsto semi-supervised learning applied to concrete tasks such as image captioning or representationlearning [36, 18].
Most previous research has focused on communication between a single pair of agents.
However,there is mounting evidence that the communication protocols developed in this restricted settingbecome highly specialized and exhibit properties that are at odds with those found in human languages[4, 8]: for example agents are able to solve the task successfully while using languages that are notcompositional [32, 9].
As a possible solution, a growing body of work is advocating for scalingup the emergent communication literature to populations of more than two agents [24, 30, 49, 10].
Indeed, there is substantial evidence within the language sciences that population dynamics shapethe language structure [47, 42].
In spite of this fact, several negative results have been obtained,showing that training agents in population yield marginal benefits without explicit pressure towardse.g. population diversity [49] or emulation mechanisms [10].
In this paper, we call into question the way such populations are trained.
By studying a simplereferential game, we evaluate populations on two desirable features observed in natural language:
• Agents are able to communicate with new partners within the same population [23]• Larger populations tend to develop more structured languages [42].
We provide evidence that populations of artificial agents do not always possess these features (asalso attested by previous work, e.g. Kim and Oh [30], Chaabouni et al.
To shed light on thisphenomenon, we analyze the behaviour of agents in a population at the equilibrium.
We find that withthe standard training procedure, the functional form of the objective is the same as that of a single pairof agents, due to receivers co-adapting to their training partners.
As our main contribution, we proposean alternative training procedure which partitions sender-receiver pairs and limits co-adaptation ofreceiver agents.
We show that this new training paradigm maximizes a different objective at thepopulation level.
In particular, it explicitly promotes mutual-intelligibility across different agents.
In experiments, we find that agents trained in partitioned populations are able to communicatesuccessfully with new communication partners with which they have never interacted during training,and that languages spoken by various agents tend to be similar to one another.
In addition, we observethat (1) languages developed in partitioned populations tend to be more compositional and (2) thereis a population size effect whereby larger populations develop more structured languages.
Our resultsshow that there are multiple ways to generalize from single agent pairs to larger populations, and thatthese design choices matter when it comes to studying the emergent language.
Communication Game
We study communication in referential games, a variant of the Lewis signaling game [37] proposedby Lazaridou et al.
The game proceeds as follows: during each round, a sender agent π observesan object x ∈ X (e.g., an arbitrary categorical entity, or a natural images) sampled from input space X with probability p and generates a message m ∼ π ( · | x ) .
Messages consist of variable lengthsequences of tokens picked from a discrete vocabulary V .
Note that the tokens themselves arearbitrary and meaningless (typically they are represented as numbers from 1 to | V | ).
A receiver agentρ then observes message m and must predict the original object from among a set of candidates
C = { x, y 1 , . . .
y |C| } containing x and |C| distractors, where each distractor y is sampled uniformlywithout replacement from the input space excluding the original object, Xx .
Concretely, this is implemented by calculating a score f
( y, m ) for each candidate
y and definingthe probability of a candidate conditioned on the message ρ ( · | m, C ) as e f ( x,m ) (cid:80) y ∈C f ( y,m ) .
Based on thereceiver’s success, the sender agent receives a reward R ( x, ρ ( · | m, C )) .
In practice, both senders and receivers are implemented as neural networks π θ and ρ ψ with parametersθ and ψ estimated by gradient descent.
The sender is trained to maximize its expected reward usingthe REINFORCE algorithm [57], while the receiver maximizes the expected log-likelihood ofidentifying the original object, log ρ ψ ( x | m, C ) (also known as the InfoNCE objective; Oord et al.[45]).
Denoting as E x ∼ p the expectation over x sampled from p , the corresponding training objectivesare:
[Equation]
2.1 Population Level Training
The two-player referential game can be generalized to larger populations of agents [41, 10].
Inthe most general case, we consider a population of N s senders and N r receivers that are linked byan bipartite communication graph G defining connections between senders and receiver ( π θ i , ρ ψ j )[24, 30].
At training time, sender-receiver pairs are repeatedly sampled and trained to perform around of the game.
Importantly, only agent pairs that are connected in the communication graph aresampled.
Throughout this paper, we will refer to this type of training as Standard training.
With this training procedure, agents are trained to maximize their communicative success with alltheir neighbors in the communication graph.
Let N G ( i ) refer to the neighbors of the i -th node inthe graph, and J s,i → j (respectively J r,i → j ) denote the objective of π θ i (respectively ρ ψ j ) ) in the pairwise communication from sender i to receiver j .
We can write the overall objective for sender i(and receiver j , respectively) as:
[Equation]
At test time, the population is evaluated by averaging the referential accuracy across all possiblesender-receiver pairings.
Following previous work, in this paper we focus on populations with anequal number N :=
N s = N r of senders and receivers, meaning that there are up to N 2 possiblepairings.
2.2 What does Population-level Training Optimize?
To shed light on the differences between training a single agent pair and training a population ofagents, we analyze the objective optimized by the population.
Inspired by [1]’s analyses in thetwo-player case, we study the behaviour of the population at the optimum, that is when senders andreceivers have reached a Nash equilibrium [46].
In this section, we make the simplifying assumption that C = X .
In other words, receiver agents mustpick the correct candidate out of all possible objects in X .
This allows us to remove the conditioningon C and write ρ ψ ( x | m, C ) =
ρ ψ ( x | m ) .
We make this simplification to reduce clutter in notations.
Nevertheless, our key observations still hold for C ̸
= X (see Appendix B for a detailed discussion).
At a Nash equilibrium, the optimal receiver parameters ψ ∗ j satisfy
[Equation]
Assuming that receiver ρ ψ j has high enough capacity, and training is able to reach the global optimum,the solution of the optimization problem in Equation 4 has an analytical solution ρ ψ ∗ j which can bewritten as a function of π ∗N G ( j ) ( m | x ) :
= 1 |N G ( j ) | (cid:80) i [Equation] G ( j ) π θ ∗ i ( m | x ) , the mixture of all senderscommunicating with receiver j :
[Equation]
In other words, ρ ψ ∗ j is the posterior associated with π ∗N G ( j ) (full derivation in appendix A).
An important implication of this result is that when the population graph is fully connected (allsenders are connected to all receivers), each receiver converges to the same optimum π ∗ ( x | m ) =
[Equation]
=
[Equation]
π
[Equation]
x ) p
[Equation]
x ) E y ∼ p (cid:80) ni
=1 π θi ( m | x ) , the posterior of the mixture of all senders in the population.
Plugging this backinto each sender’s objective, we have
[Equation]
Summing across all senders, we can rewrite the global objective optimized by the senders as
[Equation]
In other words, at the equilibrium, the population maximizes the expected reward of the “senderensemble” π ∗ , rather than that of individual agents π θ ∗ i : the objective of a population N agents isfunctionally the same irrespective of N .
We postulate that this indifference to the population sizemay account for the surprising lack of effect of larger populations observed in some previous work[49, 10].
Differences in behaviour must be attributed to effects stemming from training dynamics( e.g. it becomes more difficult for receivers to learn the posterior π ∗ ( x | m ) ), or be imposed throughextraneous modifications of the population objective (for example explicit imitation components;
Chaabouni et al.
A second observation is that there is no direct pressure for agents that communicate at training timeto develop the same language.
Indeed, it is entirely possible that all senders develop different butnon-overlapping languages: it suffices that no two senders communicating with a shared receiveruse the same message m to describe a different object.
In this case receivers can simply learn theirneighboring sender’s languages and there is no need for the senders to converge to a unified language.
[Figure]
Partitioning Agents
[Figure]
A key difference between the usual population setting and populations of humans in laboratoryexperiments is that agents are not usually split into “senders” and “receivers”.
Rather, each participantin the experiment assumes both a sender and receiver role [21].
Our hypothesis is that, counter towhat is customary in the emergent communication literature, tying senders and receivers is key insurfacing useful population-level dynamics in multi-agent communication.
To operationalize this sender-receiver coupling, we identify an “agent” as a sender-receiver pair.
During training, we only train receiver ρ ψ i with its associated sender π θ i .
In other words, J r,i ( ψ i ) :=
J r,i → i ( ψ i ) .
In doing so, we “partition” the agents by preventing receiver i from co-adapting to othersenders j ̸
= i .
This procedure is illustrated in Figure 1.
Note that senders can nevertheless stilltrain with rewards from neighboring receivers, and so communication across agents can still emerge.
Importantly, partitioning prevents receivers from learning to recognize multiple languages, as theyare now only trained on messages emitted by a single sender.
Following a similar analysis as Section 2.2, we derive that at the optimum, receiver ρ ψ ∗ i ( x | m ) nowtakes the form of the posterior associated with its respective sender, π θ ∗ i ( x | m ) =
π θ ∗ i ( m | x ) p ( x ) E y ∼ p π m | y .
Wecan thus write the population-level objective at the equilibrium as
[Equation] [Table]
[Equation]
Note that the functional form of the objective can now be decomposed into two parts: an “internalcommunication” objective which takes the same form as that of a single pair of agents, and a “mutualintelligibility” objective which enforces that neighboring agents are able to communicate successfully.
In experiments, we show that this explicit pressure towards mutual intelligibility promotes theemergence of a single language within the population, which in turn enables agents to communicatewith new partners outside of their training neighborhood.
Experimental Setting
4.1 Datasets
We perform experiments on two datasets: a simple, synthetic “attribute/values” dataset and a morerealistic image dataset.
Attribute/Values In this dataset, each object is represent by a collection of abstract “attributes”.
Specifically, each input x is a vector of 4 attributes, each of which can take 10 total values.
This resultsin 10 4 total attribute/value combinations [32, 9].
In each setting we hold out 1 , 000 combinationsto be used as a validation, and 1 , 000 more for use as a test set.
We can thus ensure that we areevaluating the agents’ ability to generalize to unseen combinations of attributes.
ImageNet In addition to toy objects, we perform experiments with referential games based on morerealistic objects.
Following Chaabouni et al.
[10], we use the ImageNet [17] dataset of natural images.
The dataset consists of about 1.4M training images collected on the internet and annotated for 1,000labels from the WordNet database [40].
Images are first encoded as 2048-sized real-valued vectorswith a (frozen) ResNet pre-trained with BYOL [22] before being passed to sender and receivers.
4.2 Game Architecture
Both sender and receiver agents are based on 1 layer LSTMs [26] with embedding and hiddendimensions of size 256 .
Specifically, the sender first encodes the object x into a vector of size 256 ,which is concatenated to the input of the LSTM.
At each step, the output of the LSTM cell is passedthrough a fully connected layer to produce logits of size | V | .
A softmax function is then appliedto obtain normalized probabilities over the vocabulary.
During training, messages are generatedby sampling from the distribution whereas at test time we generate messages deterministically viagreedy decoding.
In both cases, generation stops whenever a special “ <EOS> ” is generated, or whenthe number of tokens reaches a fixed limit L .
The receiver encodes the message with an LSTM encoder, the output of which is the fed into afully connected layer to yield a vector of size 512.
The candidate objects C are then scored bycomputing the dot product of this vector with a 512-dimensional encoding of each candidate.
Theconditional distribution over candidates is then obtained by taking a softmax.
We set the rewardfunction for the sender to the log-likelihood assigned by the receiver to the correct candidate,
R ( x, ρ ψ ( · | m ))
= log ρ ψ ( x | m ) .
Throughout all experiments, we set the vocabulary size | V | to 20 and the maximum length of themessages, L , to 10 .
This means that the communication channel used by the agents has a capacityof about 20 10 which ensures that there is no communication bottleneck (the size of the channel isseveral orders of magnitude larger than the size of our datasets).
Our implementation, based on the
EGG toolkit [29], will be open-sourced upon de-anonymization.
[Figure]
4.3 Population training
[Figure] [Figure]
We train populations following the procedure outlined by
Chaabouni et al.
[10]: for each minibatch of data, wesample K pairs from the population (uniformly amongthe pairs linked in the communication graph).
Each pairplays an episode of the game, and the agents are updatedsimultaneously following the gradients of their respectiveobjectives.
We take K = max(10 , N ) to ensure that eachagent plays the game at least once at every step on average.
This procedure needs to be modified for partitionedpopulations: since receiver j is only with its respectivesender instead of with all of its neighbor, there is nowonly a 1 | N G ( j )
| chance that receiver j will be updated everystep (the probability that the pair ( j, j ) is sampled).
Forlarger populations, especially those that are fully-connected, this dramatically slows down training asreceivers are updated very infrequently.
To address this issue, we modify the procedure as follows:for every sampled agent pair ( π θ i , ρ ψ j ) , we calculate both J s,i → j and J r,i → i and update both π θ i andρ ψ i .
Note that this necessitates calculating both ρ ψ j ( x | m, C ) and ρ ψ i ( x | m, C ) and therefore weincur a small computational overhead.
However we only observe a ∼ 5% increase in training timedue to the fact that we are back-propagating through only one of the two receivers, ρ ψ i ( x | m, C ) .
With this modification, we recover the property that each agent (sender or receiver) is updated onceevery step on average.
In all experiments we train with a batch size of 1024 with the Adam optimizer [31] using a learningrate of 0 .
001 for the attribute/value dataset and 0 .
0001 for Imagenet.
The other parameters are set toβ 1 = 0 .
9 , β 2 = 0 .
999 and ε = 10 − 8 .
We apply ℓ 2 regularization with a coefficient of 10 − 5 .
We systematically augment the sender objectives with an entropy maximizing term, which hasbeen found to encourage exploration [58].
The coefficient for this entropy term is set to 0 .
1 in all
[Table] [Table]
experiments.
To reduce the variance of the policy gradient in REINFORCE, we substract a baselinecomputed by taking the average reward within a given mini-batch for each pair [54].
We evaluate the population every epoch (every 5 epochs for the Attribute/Value dataset) on thevalidation set.
We only evaluate on up to 100 unique pairs sampled uniformly within the population,this time without consideration for the communication graph.
We train for a fixed number of epochs,selecting the best model based on the average validation accuracy across all evaluation pairs.
Communication with New Partners
In our first set of experiments, we evaluate the ability of agents trained in populations to communicatewith partners they haven’t interacted with during training.
5.1 Circular Populations
Specifically, we study “circular” populations of agents arranged on a ring lattice.
Each agent (senderreceiver pair)
i is only trained with neighboring agents i − 1 , . . .
, i +
and the graph is cyclical (see
Figure 2b).
We choose this type of population because it is an extreme case of a population whereeach agent has the same, minimal amount of neighbors (two), yet there is still a path between any twoagents.
In this context, training partners are sender-receiver pairs that are connected in the graph andhave interacted during the training phase whereas new partners refers to pairs that have not interactedduring training.
We report results along two metrics:
• Communication Accuracy of sender/receiver pairs on an evaluation set.
This measures howsuccessful the pair is in communicating.
• Language Similarity between senders.
This metric (also called synchronization in Rita et al.
[49])is calculated as 1 − δ i,j , where δ i,j is the normalized edit distance between messages output bytwo senders, averaged across all objects in our evaluation set.
We report these metrics for both training partners and new partners.
Note that high communicationaccuracy does not always entail similar languages: it is possible for the receivers to achieve highaccuracy despite all senders sending different messages for any given object (it is only necessary fora given message to unambiguously refer to one object across senders).
5.2 Partitioning Enables Successful Zero-Shot Communication
In Table 1 and 2, we report accuracies and similarities for circular populations of 20 sender-receiverpairs trained on ImageNet and the Attribute/Values dataset.
All metrics are calculated on the test setand averaged across 3 independent experiments.
[Figure] [Figure]
We observe that in populations following the standard training paradigm ( Standard ), there is a starkdiscrepancy between training and new partners.
Indeed, on both datasets the accuracy with trainingpartners reaches a very high value, above 95% .
Yet, the accuracy when agents communicate withnew partners drops down to less than 10% .
On the other hand, in Partitioned populations, agentsreach a much higher accuracy with non-neighbors, up to 96% on ImageNet and 40% .
A similar trendholds for language similarity.
Note that all metrics on new partners exhibit high standard deviation.
An explanation is that amongnon-neighboring pairs there is a different behaviour depending on how far the two agents are in thepopulation.
This is verified in Figure 3, which displays a breakdown as a function of the distancebetween two agents in the communication graph (on ImageNet).
We find that without partitioning,accuracy drops off sharply to close to 0 for agents at a distance ≥ 2 , whereas it decreases almostlinearly with the distance in the partitioned case, down to about 95% for the most distant agents.
5.3 Training dynamics
We further investigate the evolution of accuracies during training.
In Figure 4, we plot the evaluationaccuracies of both standard and partitioned populations broken down by distance between pairs,focusing on the ImageNet dataset.
Note that there are two training phases in the standard case.
Up toepoch ≈ 10 , the accuracy for all training pairs increases, after which agents over-fit to their trainingpartners (distances 0 and 1) and the accuracy on other pairs decreases to a plateau.
On the other hand, Figure 4b illustrates the pressure for mutual-intelligibility in partitioned populations: as accuracy between training pairs reaches close to 99% accuracy (around epoch 20 ), accuraciesacross distant pairs increases rapidly before plateauing above 90% .
In fact, our results show that themost distant accuracies are still increasing after 150 epochs, albeit very slowly.
[Figure]
T o p o g r a p
h
i c
s i m il a r i t y
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
T o p o g r a p
h
i c
s i m il a r i t y
[Figure]
[Figure]
Partitioned Population Develop More Compositional Languages
In this section, we investigate the effect of partitioning on the structure of the language, with a focuson compositionality .
6.1 Measuring Compositionality
A language is said to be compositional when the meaning of a whole utterance can be systematicallydeduced from the meaning of its components ( i.e. words).
The notion of compositionality is widelyconstrued to underlay the near infinite productivity of human languages [55].
A common metric for measuring compositionality in emergent languages is the topographic similarity[5, 35].
Topographic similarity captures the intuition that a compositional language will map similar“meanings” to similar messages: the phrase “a red bird” is more similar to the phrase “a blue bird”than to “a powerful computer”.
In practice, the topographic similarity is computed by measuring the
Spearman rank correlation coefficient [52] between (1) the pairwise distances across all objects and(2) the pairwise distance across all messages.
6.2 Effect of Population Size on Compositionality
We run experiments on our Attribute/Values dataset, with both standard and partitioned populationsthat are fully-connected (see Figure 2a).
Population sizes range from 2 to 25 sender-receiver pairs.
We compute topographic similarity using the Hamming distance in the object space ( i.e. the distancebetween two objects is the number of attributes in which they differ) and the normalized edit distancebetween messages.
In Figure 5a, we observe that while standard population-level training does increase the topographicsimilarity of the language overall, population size has very little effect: populations of sizes 3 and 20both reach about the same value of 30 on average.
On the other hand, partitioning greatly increasesthe effect of population size on compositionality: populations of size 20 have a significantly highertopographic similarity than populations of size 5, with a ≈ 10 points difference.
6.3 Co-adaptation is Responsible for the Decrease in Compositionality
Up until this point, we have described partitioning (or lack thereof) as a binary choice.
However,it is possible to partition a population only partially, by allowing receiver j to train with sendersi ̸
= j occasionally with probability α > 0 .
In doing so, the optimal receiver now becomes a mixturebetween π θ ∗ i ( x | m ) and π ∗ ( x | m ) .
If 0 < α < 1 , receivers are now optimizing for a differentobjective (as in partitioned populations), but some amount of co-adaptation is still allowed.
We perform this experiment on the Attribute/Values dataset with a fully connected population ofsize 10 , varying the degree of co-adaptation α ranging in { 0 , 0 . 1 , 0 . 5 , 0 . 9 , 1 } .
α = 0 correspondsto partitioned training
whereas α = 1 is equivalent to standard training.
All populations convergeto > 99% accuracy.
However, in Figure 5c we find that topographic similarity drops as soon as weintroduce minimal amounts of co-adaptation ( α = 0 .
and decreases steadily to the level of standardpopulations as α grows to 1 .
This further corroborates our hypothesis that reducing co-adaptation promotes the emergence of a more structured language, and that eliminating it altogether (in apartitioned population) yields the best results.
6.4 Importance of Mutual Intelligibility
Recall that the objective of a partitioned population at the equilibrium (Equation 6) can be decomposedin two terms: an “internal communication” term which corresponds to the single agent pair objectiveand a “mutual intelligibility” term which encourages senders to align their languages.
Importantly,the latter is the only element that separates a partitioned population from a collection of isolatedagents.
To measure its effect on the compositionality of the emergent language, we train fully connectedpopulations of size 10 and decrease the relative weight of the mutual intelligibility term.
This isimplemented by making the pair ( π θ i , ρ θ i ) more likely to be sampled than other pairs ( π θ i , ρ θ j ) ,
j ̸
= i by a factor × 1 − ββ .
We let β range from 0 .
5 (partitioned population) to 0 .
(collection ofisolated sender-receiver pairs).
In Figure 5c, we find that emergent languages retain high topographicsimilarity even at small β , and the sharp drop-off occurs only when β is very close to 0, confirmingthat the mutual intelligibility term exerts a strong pressure towards compositionality.
Related Work
There is a rich history of modeling the emergence of language as the solution to a cooperative gamethat can be traced back to functional theories of language [59, 2, 13].
With a regain of interestfor the study of language evolution [15, 12], a rich literature has developed around computationalsimulations of the emergence of language based on simple language games [37, 51, 3, 6].
Examplesinclude studying evolutionary models of the emergence of grammar [44], the influence of culturaltransmission [5], game theoretical considerations [27] or linguistic diversity [39] among others.
The recent success of deep learning in natural language processing has spurred interest in studyingsignaling games between deep neural network trained with reinforcement learning to solve a signalinggame [34, 20].
Several follow-ups have taken this idea further by extending it to more complex gamesor environment [53, 25, 28, 16] or by adding an element of competition [50, 43] or negotiation [7] oreven explicit pressure towards certain desirable properties [32, 11, 38, 48].
In parallel, several effortshave been made to understand the properties of the emergent languages [4, 8, 9].
Within this growing literature, multiple authors have explicitly studied the use of populations of morethan two agents.
Various works have argued for augmenting populations with an explicit pressuretowards more structure languages, via e.g. generational transmission [14], adversarial regularization[56], varying learning speeds [49] or imitation learning and voting [10].
Although the focus is oftenon fully-connected populations, some authors have also explored more complex communicationgraphs, for the purpose of modeling contact linguistics [24] or the effect of social network structureon the language [19].
Recent work from Kim and Oh [30] is perhaps closest to our own: the authorsstudy the effect of population size and connectivity in the standard training paradigm.
In contrast, thepurpose of this paper is to highlight the impact of the training procedure on these very effects.
Conclusion
Empirical findings in socio-linguistics suggest that population dynamics should help in simplesender-receiver communication games.
In this paper, we observed that populations trained by naivelyextending the simple 1-1 protocol to N × N agent pairs fail to exhibit some of the properties thatare observed in human populations.
Motivated by an analysis of populations at the equilibrium, wedescribed an alternative training paradigm, based on agents partitioning to reduce co-adaptation.
Empirically, we find that partitioning enables us to recover some of the aforementioned properties.
Our findings call attention to the fact that there is more than one way to generalize two single to manyagents, and simple design choices can have a great impact on the training dynamics and ultimately theeffect of population on the emergent language.
Beyond emergent communication, we hope that thisobservation can inspire similar work in other cooperative multi-agent problems where co-adaptationbetween agents may counteract population effects.
A Derivation of the Optimal Receiver
We can write the objective of receiver j as
[Equation]
Note that by linearity of expectation we can pass the averaging over N G ( j ) inside of the expectations and rewrite the second expectation in terms of the mixture π ∗N G ( j ) ( m | x ) :=
[Equation]
N G
[Equation]
[Equation]
With slight abuse of notation, let us now denote by π ∗N G ( j ) ( m ) :
= E x ∼ p π ∗N G ( j ) ( m | x ) the marginaldistribution over messages and π ∗N G ( j ) ( x | m ) :
= π ∗N G ( j ) ( m | x ) p ( x ) π ∗N G ( j ) ( m ) the associated posterior.
Noticethat since by definition π ∗N G ( j ) ( m | x ) p ( x ) =
π ∗N G ( j ) ( x | m ) π ∗N G ( j ) ( m ) , we can rewrite thedouble expectation E x ∼ p E m ∼ π ∗N G ( j ) ( ·| x ) as E m ∼ π ∗N G ( j ) ( · ) E x ∼ π ∗N G ( j ) ( ·| m ) by inverting the order ofsummation.
We can therefore rewrite
[Equation]
where H ( p, q ) denotes the cross-entropy E q [ − log p ] of two distributions p and q .
Importantly thecross-entropy is non-negative and H ( p, q )
=
if and only if p = q .
Consequently, the receiver ρ ψ will be optimal ( J r,j ( ψ j ) = 0 )
if and only if for all m :
[Equation]
□
B The Case of Referential Games
In the analysis from Section 2.2 onward, we assumed C
=
X to simplify notation.
We can relax thisassumption without changing our key observation that all receivers are the same at the optimum.
Indeed, in this case the receiver’s objective in a standard population is:
[Equation]
This objective, called InfoNCE [45] also has an analytical solution that can be expressed as a functionof π ∗N G ( j ) , of the form:
[Equation]
Despite the more complicated form of the optimal receiver, the key ingredients to our analysisin Sections 2.2 and 3 are preserved: at the optimum, each receiver is a function of the posteriorπ N G ( j ) ( x | m ) associated with the communication partners to which it co-adapts.
A similar analysisin partitioned populations shows that the optimum for receiver j then only depends on the posteriorassociated with its respective sender π θ ∗ j instead.