Trading off Utility, Informativeness, and Complexity in Emergent Communication
Anonymous Author(s)
AffiliationAddress email
Abstract
Emergent communication research often focuses on optimizing task-specific utility as a driver for communication.
However, there is substantial evidence thathuman languages evolve under pressure to efficiently compress meanings intocommunication signals by optimizing the Information Bottleneck tradeoff betweeninformativeness and complexity.
In this work, we study how trading off these threefactors — utility, informativeness, and complexity — shapes emergent communication, including compared to human communication.
To this end, we propose
Vector-Quantized Variational Information Bottleneck (VQ-VIB), a method for training neural agents to compress inputs into discrete signals embedded in a continuousspace.
We train agents via VQ-VIB and compare their performance to previouslyproposed neural architectures, both in multi-agent reinforcement learning settingsand in a Lewis reference game.
Across all neural architectures and settings, takinginto account communicative informativeness benefits communication convergencerates, and penalizing communicative complexity leads to human-like lexicon sizeswhile maintaining high utility.
Additionally, we find that VQ-VIB outperformsother discrete communication methods.
This work demonstrates how fundamentalprinciples that are believed to characterize human language evolution may informemergent communication in artificial agents.
Introduction
Good communication is a critical component of successful teams of humans and artificial agents,but differing notions of “good” makes training agents to develop such communication challenging.
One view of communication focuses on utility by framing languages as successful to the extentthey enable high task performance.
Emergent communication literature, wherein agents learn tocommunicate while optimizing a task-specific reward or utility, emphasizes this view [1–6].
However,this approach may lead to communication systems that are too complex for humans to understand [7]or have slow convergence rates [8], implying that additional constraints are needed in order to guideagents toward human-like communication.
Finding such constraints is an open question, both forartificial intelligence and for cognitive modeling.
Here, we address this question by building on a recent line of work that argues that human languagesevolve under pressure to efficiently compress meanings into communicative signals [9–13].
Thisnotion of efficiency is formulated in terms of the Information Bottleneck (IB) principle [14], whichis a general information-theoretic principle with broad scope in machine learning [15–19], andin this context, it can be interpreted as a tradeoff between the complexity and informativeness of
[Figure]
Informativeness := − E
[ d ( X, ˆ X )]
Complexity :=
I ( X, C )
Utility :=
U ( X, Y )
[Figure]
communication [9].
Intuitively, complexity corresponds to the number of bits that are needed forcommunication, and informativeness corresponds to how well a listener can infer the speaker’sintended meaning regardless of a specific task (e.g., humans understand what the term “blue” meansregardless of any specific task, which may be “avoid the blue box” or “find a blue cup”).
Wehypothesize that taking into account complexity and informativeness, in addition to optimizinga task-specific utility function, may help guide artificial agents toward more natural, human-likecommunication systems.
We test this idea by training teams of artificial agents to optimize a tradeoff between maximizingtask utility, maximizing communicative informativeness, and minimizing communicative complexity,as depicted in Figure 1.
To this end, we extend prior work [20, 21] and propose Vector-Quantized Variational Information Bottleneck (VQ-VIB), a method for training neural agents to compress inputsinto signals, while maintaining a discrete (symbolic) signal representation that is embedded in acontinuous space.
In experiments, we find that training to increase informativeness improves theconvergence rate of emergent communication to high-performance communication, while annealingthe complexity loss generates a spectrum of learned communication, which we show aligns with avariety of human languages.
Additionally, we find that VQ-VIB appears better able to achieve highutility and informativeness for the same complexity than other discrete communication methods.
Related Work
We train neural agents, focusing on the effects of utility, informativeness, and complexity on emergentcommunication.
Here, we describe related work in emergent communication; we include approachesfor measuring informativeness and complexity in Section 3.
Generally, in emergent communicationsettings, researchers train agents to accomplish a cooperative task in which communication shouldenable better task performance [3, 22, 23].
For example, if one agent observes a goal locationthat another agent must navigate to, the team of agents would benefit from the “speaker” agentcommunicating with the “listener” agent.
Many emergent communication works have considered howenvironmental factors (such as reward, agent architecture, or more) affect learned communication [3,5, 22, 24].
We focus on subsets of the literature that explore aspects of informativeness and complexity,as well as specific applications in color reference games.
Notions of Complexity in Emergent Communication.
Recent emergent communication literature has taken initial steps beyond focusing only on utility by limiting complexity or encouraginginformativeness, although these terms are rarely explicitly acknowledged.
Numerous works simplifycommunication by discretizing it or by adding noise to the environment; these methods all correspondto limiting complexity [21, 25].
Wang et al. [26] explicitly limit complexity via a centralized messageprocessor that acts as an information bottleneck.
At the same time, other works find that auxiliarylosses designed to create “meaningful” communication improve convergence rates [8, 27].
For example, Lin et al.
[28] use an autoencoding loss to reconstruct states from communication; this istightly related to notions of informativeness.
Thus, while emergent communication literature hasbegun to rediscover the importance of complexity and informativeness in communication, theseterms have rarely been explicitly recognized.
We differ from prior art by including terms for utility,informativeness, and complexity to train each agent.
Emergent Communication and Color Reference Games.
We conduct experiments in colorreference games, which have been widely used in emergent communication literature [1, 29–31].
Typically, these reference games, or Lewis signalling games [32], comprise a speaker, who observesone color dubbed the “target,” and a listener, who observes the target color and distractor color(s).
Bycommunicating to the listener, the speaker seeks to have the listener identify the target.
We expandupon our specific experiment configuration in Section 4.4.
Color naming has been a central test-case for cognitive theories of semantic categorization andlanguage evolution [33].
It is a rare domain in which a large-scale cross-linguistic dataset, the World
Color Survey [WCS; 34], is available, documenting the wide diversity in color naming schemesacross languages.
Particularly relevant to our work is Zaslavsky et al.
[9]’s finding that color namingsystems across languages attain complexity–informativeness tradeoffs that are near-optimal in the
Information Bottleneck [IB; 14] sense.
Building on that, Chaabouni et al.
[31] showed that AI agentscan learn IB-efficient color naming systems by playing color reference games (and see [30] for arelated study using different notions of efficiency).
We similarly train agents in color reference games,but differ from prior art by directly tuning an objective function for trading off utility, informativenessand complexity, and by proposing a new agent architecture, which achieves higher informativenessfor a given complexity than other methods.
Technical Approach
Here, we show how to adapt standard neural architectures to enable informativeness- and complexitybased objectives.
We then introduce a new neural architecture, Vector-Quantized Variational Information Bottleneck (VQ-VIB), for learning complexity-limited discrete representations in a continuousspace.
Lastly, we combine utility, informativeness, and complexity terms into an overall objective.
3.1 The Information Bottleneck Framework for Semantic Communication
We adopt and extend Zaslavsky et al.
[9]’s IB framework for semantic communication, which hassubstantial empirical support across languages and domains [10–13].
In this framework, speakersand listeners jointly optimize the IB tradeoff between the complexity and informativeness of theircommunication system.
A speaker is characterized as a probabilistic encoder q ( c | x ) that, givenan input, x ∼ p ( x ) , generates a signal, c .
A listener is characterized as a probabilistic decoderq (ˆ x | c ) , that given c aims to reconstruct the speaker’s input.
Complexity is measured by the mutualinformation between the speaker’s inputs and signals, I ( X, C ) .
Informativeness is measured by thedegree to which the speaker’s mental representation of x matches the listener’s mental representationof ˆ x .
In general, these mental representations are defined by probability distributions, m and ˆ mrespectively, over some feature space.
In that case, minimizing the expected Kullback-Leibler (KL)divergence E [ D [ m ∥ ˆ m ]] amounts to maximizing the informativeness of the communication.
Thisyields the following optimization principle: min I ( X, C ) − β E
[ D [ m ∥ ˆ m ]] , where β is a tradeoffparameter.
It has been shown that this principle is equivalent to the IB principle [9].
Solving this optimization problem directly is challenging in large scale domains and with agentsthat are parameterized by neural networks.
To address this, we take a Variational Informationbottleneck [VIB; 18] approach, in which an upper bound is optimized, similar to ELBO in Variational
Autoencoders [VAEs, 35] and β -VAEs [36].
In practice, we also approximate informativeness by anautoencoding loss, d ( x, ˆ x ) , defined by the mean squared error (
[Figure] In the “sample-before” architecture (a), the speaker encodes an input, x , as parameters to a Gaussian, from which a communication vector is sampled and then discretized by mapping to the nearest encoding, to produce communication, c . In the “sample-after” architecture (b), the speaker deterministically maps x to the nearest quantized encoding, ζ , and then c is sampled by adding Gaussian noise to ζ .
3.2 Vector Quantized Variational Information Bottleneck (VQ-VIB)
Our proposed neural architecture, the Vector Quantized Variational Information Bottleneck (VQVIB), learns information-constrained discrete communication in a continuous space.
Our techniquecombines prior work in discrete representation learning [VQ-VAE, 20] with VIB [18].
3.2.1 Vector-Quantized Variational Autoencoder
First, we summarize a simplified Vector-Quantized Variational Autoencoder (VQ-VAE) architecture [20].
As in a standard VAE, our encoder is a conditional distribution over latent representations,given an input: q ( z | x ) .
A deterministic feedforward network maps from an input, x , to a latent representation, z : we dub such encodings, z ( x ) .
Next, for a VQ-VAE parametrized with K embeddingvectors, ζ , in the latent space, z ( x ) is discretized to a particular embedding vector, ζ i , by mapping tothe nearest embedding vector.
Thus, the discretized embedding is determined by Equation 1.
[Equation]
0 else (1)
3.2.2 Variational Information Bottleneck (VIB)
We also use notions from VIB, a variational approach for generating information-constrained continuous representations with neural nets.
Following Alemi et al.
[18], an encoder network mapsfrom an input to µ and σ , which are used as parameters to a Gaussian distribution from which alatent representation, z , is sampled.
VIB networks are trained via a weighted combination of aclassification loss and ( β times) the KL divergence µ ( x ) σ x loss corresponds to an upper bound of I ( X, Z ) .
3.2.3 VQ-VIB Architectures
Here, we integrate VQ-VAE and VIB for two methods of generating discrete, information-boundedrepresentations.
Our first architecture, VQ-VIB Before , corresponds to first using an encoder tosample a representation z in a continuous space, followed by a quantization layer that discretizes thesample by mapping to the nearest encoding ζ .
Our second architecture, VQ-VIB
After , correspondsto first using an encoder to generate a discrete representation ζ , whereupon a communication vectorc is sampled from a Gaussian in the continuous space centered around ζ .
These architectures aredepicted in Figure 2 and are formally defined in Equations 2 and 3.
[Equation]
In the VQ-VIB Before formulation (Eq. 2), the likelihood of a communication vector, c , given aninput, x , corresponds to the likelihood that an encoding z , sampled with parameters µ ( x ) and σ ( x ) ,is closest to an embedding vector equal to c . In the VQ-VIB After formulation (Eq. 3), the likelihoodof c is the Gaussian centered around a discrete embedding.
These two approaches reflect differingnotions of communicative uncertainty.
Sampling before corresponds to uncertainty over which “word”to use, whereas sampling after corresponds to choosing a particular “word” but corrupting it slightly. Regardless of the architecture, one may compute an upper bound on I ( X, C ) using the standard KLdivergence used for Gaussians in a VIB, simply using µ ( x ) (or ζ ) and σ ( x ) .
3.2.4 Complexity Equivalence Classes
Although one may approximate I ( X, C ) , simply penalizing this term in training was insufficient totrain VQ-VIB agents to use fewer unique discrete embeddings.
Fundamentally, two agents couldbelong to the same complexity equivalence class (with the same I
( X, C ) )
but have different numbersof embeddings if several embeddings are nearly identical (see [37] for analysis of these equivalenceclasses in IB).
Intuitively, this corresponds to notions of either communicating with many words, eachof which is difficult to disambiguate from other words, or fewer words, each of which is quite distinct.
We wish to bias agents to use fewer distinct clusters, which corresponds to the entropy over thecategorical distribution over embeddings.
Training with the exact entropy in the VQ-VIB architecturesis infeasible given the non-differentiablity of the argmin operation used in discretization.
Instead,we use the entropy of an approximation of the distribution over embeddings: the softmax of thenegative L2 distance from embeddings to each token.
(We leave exploration of other approximationsto future work.)
Because this loss is only used to change the number of clusters within a complexityequivalence class, we require only a small training signal.
In our experiments, we multiply this lossby 0.05, making it orders of magnitude smaller than other losses.
3.3 Overall Optimization
Here, we combine losses into an idealized objective function to maximize, shown in Equation 4.
This objective function is the weighted combination of three terms, wherein we seek to maximize atradeoff between high utility, high informativeness (low expected distortion d ), and low complexity.
Intuitively, these three scalar weights form a simplex that describe a range of communicationfrom silence (minimizing complexity) to task-optimized communication that does not generalize(maximizing utility) to complete descriptions (maximizing informativeness).
[Equation]
In practice, we train neural agents via gradient descent to maximize differentiable approximations ofthese terms.
Utility is represented via policy gradient methods (for MARL settings) or crossentropyloss.
Informativeness is computed via the negative mean squared error between the speaker’s inputand a decoder’s output, given the communication.
Lastly, an upper bound of complexity is computedas the KL divergence between the speaker’s distribution over communication and a fixed marginal(either a uniform distribution for onehot communication, or a unit Gaussian for others).
Experiment Preliminaries
We formulated hypotheses about the roles of utility, informativeness, and complexity in emergentcommunication, and designed experiments to test them.
4.1 Hypotheses
We expected that increasing informativeness is a bias for learning useful communication, but that doing so would likely increase complexity, which must be bounded to align with human communication.
More specifically, our two hypotheses were:
[Figure]
• Training to increase informativeness will cause agents to converge to successful communica-
tion systems more quickly than without such training, although increasing informativenesswill increase complexity.
• Complexity-bounded agents will generate more human-like discrete communication systems than unbounded agents.
4.2 Trials and Baselines
We compared 5 types of neural architectures in our experiments: continuous, onehot, Proto (based onthe prototype-based communication from Tucker et al.
[21]), VQ-VIB Before, and VQ-VIB After.
Note that both VQ-VIB architectures resolve to VQ-VAE when sampling with zero variance, whichagents approach for λ C = 0 . All except the first architecture pass through a discrete bottleneck,although only onehot and VQ-VIB Before communicate via a finite set of symbols.
For each domain,we trained agents from scratch using 5 random seeds, starting at 0.
Full details of the architectures,hyperparameters, and environments are included in Appendix A.
4.3 Particle Worlds
We trained agents in two environments inspired by those introduced by Tucker et al.
[21]: 9 pointsand Uniform.
Both environments consisted of two-dimensional worlds in which a speaker observeda target location that a listener had to reach but could not directly observe, as depicted in Figure 3.
In the 9 points environment, there were 9 landmarks at fixed locations, one of which was chosen atrandom as the target for each episode.
In the Uniform environment, two landmarks were spawned inlocations drawn uniformly at random in the world; one landmark was specified as the target.
Agentsreceived identical reward equal to the negative distance from the listener to the target and were trainedusing Multi-Agent Deep Deterministic Policy Gradient [3].
The listener could not observe the targetlocation, and the speaker could only communicate at the first timestep by broadcasting a vector thatwas observable at all future timesteps; thus, an optimal team policy would consist of the speakercommunicating about the target at the first timestep, and the listener navigating straight to the targetbased on the message.
The two environments differ in their complexity: in the 9 points environment,the speaker had a finite amount of information about which to communicate, whereas in the Uniformenvironment, the location could be any continuous value (up to the computer’s float precision).
4.4 Color Reference Games
We also tested agents in reference games, or Lewis signalling games [32], which are often used inemergent communication studies; by focusing on a color domain, which has corresponding dataon human communication [33, 34], we hoped to identify similarities between agent and humanlanguages [30, 31].
Human languages use a finite set of words, so we only tested onehot and VQ-VIB
Before communication.
Other architectures (continuous, Proto, and VQ-VIB After) sample fromcontinuous spaces and could therefore generate infinite unique communication vectors. [Figure]
In these reference games, we trained agents within a supervised learning framework, differentiatingthrough both the speaker and listener.
A diagram of the network architecture is included in Figure 6in Appendix A. A speaker observed a target color and (if in the “Common Ground” setting, explainedlater) a distractor color and output a communication vector.
A listener observed the speaker’scommunication and the target and distractor colors and predicted which of the colors was the target.
Colors were represented by their 3D CIELAB coordinates (illustrated in Figure 6 in Appendix A),and speaker observations were corrupted by zero-mean Gaussian noise with a standard deviation of 8,as motivated by human perception [9].
Both colors were sampled from the set of 330 World Color
Survey (WCS) colors according to a human-based prior from [9].
1 Only the speaker agent observedwhich color was the target, so the agents had to learn to convey information from the speaker to thelistener.
Agents were trained via binary crossentropy loss computed on the listener’s output, whichworked more reliably than MARL methods (corroborating Chaabouni et al.
[31]’s findings).
We trained agents in two settings: Isolated, in which the speaker only observed the target, and
Common Ground, in which the speaker observed both the target and distractor colors.
The Isolatedscenario resembles the WCS setting, in which participants were asked to name individual colors.
Conversely, the Common Ground scenario was inspired by work in pragmatics and Rational Speech
Act, wherein humans communicate less information than fully describes a scene, but enough toinduce listeners to behave in the desired manner [38, 39].
For example, when comparing a brightorange color to a dark purple one, a person might say just “orange” instead of “bright orange.”
Experiment Results
Results from our experiments indicated that an informativeness loss improved communicationconvergence, and annealing λ C allowed us to generate a suite of agents of different complexities.
5.1 Particle Worlds
In the particle worlds, increasingly penalizing complexity led agents to lower-utility communication,although architecture choices and training to increase informativeness could increase reward.
First,we conducted experiments in which we did not penalize complexity, but varied λ I across trials.
Across all architectures, higher λ I increased the mean team reward and improved convergence rates,as shown in the learning curves of VQ-VIB Before in Figure 4 a.
Next, for a fixed λ I = 1 . 0 (to improve convergence), we trained agents while annealing λ C ; thissmoothly decreased the complexity of communication.
Results for the Uniform environment alignedwith our expectations Figure 4 b.
Each curve indicates the cubic best fit to the data generated across 5 random seeds for each configuration (discarding outliers for reward less than -120), withcomplexity and reward measured at each increment of 1,000 training episodes from 3,000 to 10,000.
As complexity decreased (moving right to left in the plot), reward decreased as well.
Visualizations ofagent behavior confirmed that low-complexity VQ-VIB Before agents communicated via increasinglycoarse discretizations of the space and are included in Appendix B.
Lastly, these results suggest that neural architecture choice was important.
In both 9 points and
Uniform, the VQ-VIB agents tended to achieve higher reward, for the same complexity, as onehot or
Proto agents.
(Continuous communication reflected an optimal upper bound.)
Furthermore, addingmore encodings (9 discrete encodings vs. 100) significantly increased VQ-VIB performance, whilechanging onehot agents to communicate in 9 or 100 dimensions had a smaller effect.
This sectionincludes partial results, focusing on VQ-VIB Before in the Uniform environment; full results are in Appendix B. We observed similar benefits from high λ I , similar complexity-reward tradeoffs, andsimilar advantages of VQ-VIB over other architectures in the 9 points environment.
5.2 Reference Games
In color domains, encouraging informativeness and penalizing complexity generated a range ofcommunication that closely approximated optimal compression and human languages.
5.2.1 Isolated Color Reference Game
In the Isolated color reference game, an informativeness training loss appeared necessary for onehot,while a complexity-based loss was useful for generating a spectrum of communication conventions.
First, we trained agents with a range of λ I values while leaving λ C = 0 .
We briefly report two mainfindings: 1) for a given λ I , agents converged to a small range of complexity and informativenessand 2) VQ-VIB Before agents typically learned more complex and informative communication thanonehot agents did, although increasing λ
I shrank the gap (indicating a novel finding, building upon Chaabouni et al.
[31], whose experiments we approximated by setting λ I = 0 ).
Full results for theseexperiments are included in Appendix C.
Second, we tested agents for λ
I = 1 .
while linearly annealing λ C to penalize complexity.
As λ Cincreased, the communication complexity decreased, as shown in Figure 5 (left).
Over 5 randomseeds
, the points in the plot, generated at each training epoch, closely approximated the IB upperbound for maximum informativeness for a given complexity.
Inspection of the learned color namingschemes at different complexity levels revealed important trends (Figure 5 right).
As complexitydecreased, the number of communication clusters emitted by VQ-VIB agents decreased (visualized [Table] via 2-dimensional principle component analysis).
Clustering in the communication space was reflected in the color space (e.g., “yellow” and “orange” merging).
More generally, a least-squareslinear correlation between the distance between pairs of communication vectors and the distancebetween associated colors (in CIELAB space) showed r 2 = 0 .
97 ± (0 . 01) for VQ-VIB, whereas foronehot communication r 2 = 0 .
76 ± (0 . 03) .
That is, for VQ-VIB, if two communication vectors werefar apart in communication space, they described colors that were far apart in CIELAB space.
Thisshows a fundamental connection between the form and meaning of communication used by VQ-VIBagents, which onehot-based communication is architecturally unable to support.
Lastly, we compared the agent color naming schemes to human language data.
For each of the 110languages in the WCS Dataset [34], we found the best-matching agent as determined by minimizinggNID, a measure of dissimilarity of naming distributions introduced by Zaslavsky et al.
Examplenearest languages are shown in Figure 5 in dashed boxes.
Despite having no supervised data, theartificial agents learned similar color naming schemes to humans.
More generally, we analyzed themean gNID for each language to the best agent, as well as deviations of agents from the IB bound(calculated as in Zaslavsky et al.
[9] and reported as ϵ ) in Table 1.
Together, these results indicate that annealing complexity is a critical component in recreatinghuman-like color naming systems.
Furthermore, VQ-VIB Before matched the form and meaning ofcommunication by merging clusters in the communication space that represented nearby colors.
5.2.2 Common Ground Color Reference Game
We tested agents in the Common Ground color reference game to evaluate contextualized communication.
For a fixed, gray distractor, both VQ-VIB Before and onehot demonstrated nearly optimalinformativeness for a given complexity. Agents also learned to condition their communication notonly on the target color, but also on the distractor.
Complete results and figures are included in
Appendix C, confirming that agents learned efficient, contextualized communication.
Contributions
In this work, we explore the interplay of utility, informativeness, and complexity in emergent communication.
To the best of our knowledge, we are the first to directly optimize agents to communicateaccording to a weighted combination of all three terms.
In our experiments, we found that trainingagents while rewarding informativeness led to faster communication convergence, and penalizingcomplexity created more human-like communication (supporting our two hypotheses).
Further, wefound that our novel neural architecture, VQ-VIB, achieved greater utility and informativeness forthe same complexity than other discrete methods.
Natural extensions of our work include furtherimprovements to neural architectures, incorporation of new, cognitively-inspired terms to guidecommunication, or applications to more complex domains.
Our work represents a step towards reconciling principles from cognitive science with emergentcommunication literature, which we hope better supports human preferences.
However, we recognizethat, by quantifying properties of languages, our results may be misinterpreted to assign relative valuesto one language over another (e.g., higher complexity being better).
We insist that our framework 1)has only been applied to narrow domains that do not capture any language’s entirety and 2) does notprioritize
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section TODO.•
Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In
your paper
, please
delete
this
instructions block and only keep the Checklist section heading above along with
the questions/answers
below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] See abstract, introduction, and findings
(b) Did you describe the limitations of your work?
[Yes] See Section 6, in which wenote that we tested in limited domains and Section 3.2.4, in which we note that anapproximation we made may be improved upon in future work.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Section 6 in which we discuss how readers may misconstrue quantitative measures of languages as assigning value to one over the other.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See footnote
for link to anonymized code
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix A
(c) Did you report error bars (e g., with respect to the random seed after running exper- iments multiple times)?
[Yes] See error bars in plots and Table 1, in Section 5.
Allscatter plots directly included the raw results from all 5 random trials.
(d) Did you include the total amount of compute and the type of resources used (e
g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] Training times and computational
resources are listed in Appendix A.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We use the World
Color Survey (WCS) data, as well as the publicly-available code by Zaslavsky et al.
Both sources are cited.
(b) Did you mention the license of the assets?
[Yes] We note that both are publicly-available
and link to them in Footnote 1.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
Anonymized code is linked to in a footnote.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A] We only used the existing WCS dataset.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A] We only used the existing WCS dataset.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots
, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Appendix: Implementation Details
Here, we include implementation details that we omitted from the main paper that can be used toreproduce our experiments.
A.1 Multi-Agent Reinforcement Learning Experiments
We trained agents in the 9 points and Uniform environments with Multi-Agent Deep Deterministic
Policy Gradient (MADDPG), a standard method for multi-agent reinforcement learning [3].
Agent architectures were identical in both environments.
A speaker agent consisted of 2, 64-neuronfully-connected layers with ReLU activations, followed by a speaker “head.”
The head implementedthe specific communication method (e.g., continuous, VQ-VIB Before, etc.) by taking the 64dimensional representation from the speaker and going through one of 5 possible architectures toproduce a vector in the communication space, R c .
(As discussed later, c varied by environment andarchitecture slightly.
• Continuous: The continuous communication head consisted of two fully connected layersbranching off the speaker encoding to generate a µ and logvar (the log of the variance) fromwhich the communication vector was sampled using the reparametrization trick.
Complexitywas measured as the KL divergence between the parameters µ and logvar and a unitGaussian.
[Table]
• Onehot: The onehot communication head consisted of one fully connected layer from thespeaker encoding to R c .
Onehot vectors were generated using gumble softmax sampling,with temperature 1.
Complexity was measured as the KL divergence between the distributionfrom which the onehot vector was sampled and a uniform categorical distribution.• VQ-VIB Before: The VQ-VIB Before communication head first passed the speaker encoding through two fully connected layers, branching off the encoding, to generate a µ andlogvar , as in the continuous communication case.
A communication vector was sampledfrom the distribution, whereupon it was discretized to the nearest quantized vector, ζ , in thecommunication space (as done in a standard VQ-VAE, without the sampling).
Complexitywas measured as the KL divergence between the parameters µ and logvar and a unitGaussian.• VQ-VIB After: The VQ-VIB After communication head first passed the speaker encodingthrough two fully connected layers to generate an encoding, z , and logvar .
The encoding,z , was discretized to the nearest quantized vector, ζ .
Lastly, a communication vector, c ,was sampled from a normal distribution centered at ζ with variance specified via logvar .
Complexity was measured as the KL divergence between the parameters ζ and logvar anda unit Gaussian.
• Proto:
The Proto head mapped the speaker encoding through a single fully-connected layerto a categorical distribution over k neurons, for k representing the number of prototypes.
A kdimensional onehot vector was sampled using gumble softmax; a prototype, p was selectedby multiplying the prototype matrix by the onehot vector.
A different fully-connectedlayer mapped from the speaker encoding to logvar .
Lastly, a communication vector, c ,was sampled from a Gaussian centered at p with variance via logvar .
Complexity was
measured as the KL divergence between the parameters p and logvar and a unit Gaussian.
This closely resembles the VQ-VIB After architecture, but with a different discretization
mechanism.
All agents were trained with an Adam optimizer with learning rate 0.01 and batch size 1024.
A.1.1 9 points Environment
For all agent architectures except for onehot, the communication dimension was 2; onehot usedcommunication dimension 9 so that the number of discrete representations allowed by the architecturewas not the limiting factor for complexity.
Similarly, the VQ-VIB and Proto agents used 9 discreteembeddings.
In our main results (presented in Figure 4), agents were trained for 10,000 episodes, with each episodelasting 100 timesteps.
The listener agent was always spawned at the origin, equidistant from eachlandmark.
The informativeness loss weight, λ I , was constant at 1.0.
For the continuous and VQ-VIBcommunication methods, the complexity loss weight, λ C increased by 8 ∗ 10 − 6 per episode.
Theonehot and Proto methods suffered from communication collapse at that rate, so results λ C increasedhalf as quickly: at 4 ∗ 10 − 6 per episode.
Training a team in the 9 points environment for 10,000 episodes took approximately 15 minutes on adesktop computer with 16 Intel i9 cores for all architectures, except for onehot, which took roughly20 minutes.
This difference in time is attributable to the larger communication dimension needed foronehot.
A.1.2 Uniform Environment
In the Uniform environment, we used identical parameters as in the 9 points environment, except forthe changes noted below.
For all agent architectures except for onehot, the communication dimension was 3; for onehot, weexperimented with communication dimensions of 9 and 100.
Similarly, we tested VQ-VIB and
Proto agents when using 9 or 100 discrete embeddings.
The listener agent was spawned at a startinglocation sampled uniformly at random from the map (bounded between -200 and 200 for both x andy ).
Otherwise, we used the same number of training episodes, episode duration, and complexityannealing rates as in 9 points.
Training a team in the Uniform environment for 10,000 episodes took approximately 15 minutes,except for onehot, which took roughly 30 minutes for 100-dimensional communication.
A.2 Color Reference Game Experiments
[Figure]
Here, we include the implementation details used in our color reference game experiments.
Agents were trained in the overall architecture depicted in Figure 6.
The speaker agent wasparametrized with 3 fully connected layers with ReLU activations and hidden dimension 64, followedby a communication head.
The communication heads used the same architectures as specified inthe particle world experiments: gumbel softmax was used for generating onehot vectors, and astraight-through estimator was used for discretization in VQ-VIB architectures.
In all experiments,the communication dimension was set to 32 for all agents except onehot, for which the communicationdimension was set to 330 (to match the expressive power of other discrete agents, that each had 330discrete tokens in the 32 dimensional space).
Without annealing the complexity loss, 50 training epochs sufficed for agents to converge to a stablesolution.
Thus, in our experiments without complexity loss, we only trained for 50 epochs.
Forlater experiments, wherein we annealed complexity, we first trained agents for 50 epochs beforebeginning to anneal the complexity term for the next 40 epochs (for a total of 90 epochs).
We notethat, for VQ-VIB Before, we set the complexity loss to 0 during this “burnin” period, but for onehotcommunication, it appeared necessary to set the loss to 0.05 to prevent numerical instability duringthe first epoch after the burnin period.
During complexity loss annealing, we increased λ C by 0.01during each epoch for VQ-VIB Before, and by 0.1 for onehot.
These different rates were necessaryto induce similar ranges in complexity of learned communication.
In the Isolated color reference game, the speaker observed a only the 3-dimensional, corrupted targetcolor.
In the Common Ground setting, the speaker observed a 6-dimensional vector, generated by concatenating noisy versions of the target and distractor colors.
In both settings, the listener observedthe communication output by the speaker, as well as the target and distractor colors (ordered randomly,without added noise).
Results were analyzed using code partially-based on publicly-available codereleased by Zaslavsky et al.
We trained agents on a desktop computer with 16 Intel i9 CPUs and 1 NVIDIA GeForce RTX2080.
Training for 90 epochs in both the Isolated and Common Ground settings took approximately3 minutes for all agent architectures.
Agents were trained with an Adam optimizer with defaultparameters, with batch size 1024.
B Appendix: Full Particle World Results
For brevity, we only included partial results from the particle-world experiments in the main paper.
Here, we include the full results.
First, we experimented with varying λ
I for different environments and neural architectures, withoutannealing complexity.
Training curves for both environments, using varying λ
I for each neuralarchitecture, are included in Figures 7 and 8.
For each configuration (a particular speaker architectureand λ I ), we trained 5 teams from scratch for 5,000 episodes and plotted the median and quartilerewards, smoothed over a 50 episode window.
A consistent trend emerged: higher λ
I led to faster convergence to a higher reward.
Indeed, thiscorroborates findings by Lin et al.
[28], who used an autoencoder architecture for communicationin multiagent settings, but here we showed good convergence properties when mixing utility andinformativeness terms (whereas they reported worse results when adding utility losses).
Overall,these findings motivated training agents to convergence with high λ I before beginning to penalizecomplexity with λ C .
Next, we include the results of annealing complexity in both the 9 points and Uniform environments,as depicted in Figure 9.
In both environments, as we increased λ C , complexity and reward decreased.
Comparing both VQ-VIB architectures to other discrete communication methods showed that VQVIB agents tended to obtain a higher mean reward, for a given complexity, than other methods.
In the Uniform environment, we also compared different variants of VQ-VIB, Proto, and onehotagents based on the maximum number of discrete messages allowed by the neural architectures.
Wetested agents parametrized with either 100 or 9 unique messages; we hypothesized that initializingagents with many discrete messages and then gradually decreasing complexity would result in betterperformance than training with few messages from the start.
Indeed, we observed that was the case.
As seen in Figure 9 b, VQ-VIB agents parametrized with 100 embeddings consistently achievedhigher reward for the same complexity than VQ-VIB agents parametrized with only 9 embeddings.
This trend held even when the complexity was lower than log 2 (9) bits.
Lastly, we visually analyzed VQ-VIB Before communication in the 9 points and Uniform environments by tracking listener movement.
If agents used less complex communication (i.e., fewer distinctmessages from the speaker to the listener), one would expect the listener to end in a smaller set ofpossible destinations.
We therefore traced, over 50 evaluation episodes, the listener’s location inblack, and plotted its location after 100 timesteps in red.
We performed this analysis for snapshots ofthe agents taken at episode 3,001 (corresponding to relatively high complexity) and 10,000 (at theend of training, with lower complexity).
Our results, depicted in Figure 10, supported our intuition:agents later in training ended in fewer final locations, corresponding to decreased communicationcomplexity.
C Appendix: Color Reference Game Full Results
We omitted some of the color reference game results from the main paper due to space constraints.
Here, we include our complete results.
[Figure]
C.1 Isolated Color Reference Game
First, we include results from the Isolated setting, in which we trained agent with different valuesof λ I without penalizing complexity.
These results are included in Figure 11.
Setting λ
I = 0induced similar conditions to those used by Chaabouni et al.
[31]; any positive λ I favored moreinformative (and therefore more complex) communication.
Given the monotonically increasingnature of the IB bound, an optimal agent trained with any positive λ I would maximize utility andinformativeness by maximizing complexity, up to a maximum of log 2 (330) bits.
Instead, we foundthat agents appeared to converge to lower complexity values, although increasing λ
I tended tosomewhat increase complexity and informativeness.
For example, onehot communication increasedin complexity from roughly 2.3 bits for λ I = 0 .
to 3.0 bits for λ
I = 1 . 0 .
This alone is anovel result compared to Chaabouni et al. [31], who trained agents with no informativeness lossand observed lower-complexity communication.
Curiously, VQ-VIB Before agents automaticallyconverged to informativeness around 3.5 bits regardless of the λ I value, although increasing λ
I didimprove training stability.
This difference between onehot and VQ-VIB indicates that the differingarchitectures themselves have different inductive biases.
C.2 Common Ground Color Reference Game
Here, we include the full results from our Common Ground color reference game experiments.
Giventhe lack of human-labeled data for such a setting, we were unable to compare agent communicationwith human languages.
However, by analyzing communication when varying the target for a fixed
[Figure]
[Figure]
distractor, or by varying the distractor for a fixed target, we established that agents conditionedcommunication both on the target and distractor colors.
We trained agents in the Common Ground setting by randomly sampling target and distractor colors,as in the Isolated setting, but here the speaker observed (noisy) versions of the distractor as wellas the target.
Based on insights that training to increase informativeness produced more complexcommunication, we trained agents for 30 episodes with λ I = 1 . 0 . For the next 70 episodes, we
[Figure] As complexity decreased, the number of distinct end locations also decreased, indicating a coarser of the annealed the complexity weight, λ C , using the same rates as in the Isolated setting; this producedcommunication at different levels of complexity.
We then analyzed communication in two ways: using a fixed distractor with varying targets and usinga fixed target with varying distractors.
First, for a fixed, gray distractor (#46 in the WCS data), wemeasured the standard metrics of informativeness and complexity.
That is, we set the distractor colorto gray, and we recorded, for each of the other 329 colors as the target, the speaker’s communication.
We then performed the same complexity and informativeness analysis as in the Isolated setting, andfound that both VQ-VIB Before and onehot communication remained near-optimally efficient, asshown in Figure 12 a.
Second, we measured whether agents conditioned their communication on the distractor color.
Unlikethe prior evaluation, in which we varied the target color for a fixed distractor, here, we evaluatedcommunication for different distractors.
Plots of a VQ-VIB Before agent’s naming scheme, usingeither an orange (color #212) or purple (color #325) distractor are shown in Figure 12 b.
As expected, the agent displayed different resolutions in regions of color depending upon the distractor.
Specifically, for an orange distractor, the speaker had finer resolution around shades of orange andyellow; conversely, the speaker had finer resolution around shades of purples for a purple distractor.
This aligns with expectations that, when trying to differentiate between colors, one must use morecomplex or accurate words to distinguish similar colors.
Lastly, we measured agents’ utility in the Common Ground task (measured as the listener’s accuracy)and, as expected, agents maintained higher utility at lower complexity than in the Isolated setting, as
Without positive λ I , onehot learns low-informativeness communication, but λ
[Figure]
shown in Figure 12. Both methods achieving roughly 88% accuracy with only one bit of complexity,whereas in the Isolated setting, 1 bit of complexity achieved approximately 82% accuracy. Furthermore, especially for low-complexity communication, VQ-VIB achieved higher utility, for the samecomplexity, than onehot communication. Jointly, these findings indicate that many of the themes identified in the Isolated setting – nearlyoptimal communicative efficiency, the importance of annealing λ C to generate varied complexity,and the advantages of VQ-VIB over onehot – held in the Common Ground setting as well.