Trading off Utility, Informativeness, and Complexity in Emergent Communication
Anonymous Author(s)
AffiliationAddress email
Abstract
Emergent communication research often focuses on optimizing task-specific utility as a driver for communication.
However, there is substantial evidence thathuman languages evolve under pressure to efficiently compress meanings intocommunication signals by optimizing the Information Bottleneck tradeoff betweeninformativeness and complexity.
In this work, we study how trading off these threefactors — utility, informativeness, and complexity — shapes emergent communication, including compared to human communication.
To this end, we propose
Vector-Quantized Variational Information Bottleneck (VQ-VIB), a method for training neural agents to compress inputs into discrete signals embedded in a continuousspace.
We train agents via VQ-VIB and compare their performance to previouslyproposed neural architectures, both in multi-agent reinforcement learning settingsand in a Lewis reference game.
Across all neural architectures and settings, takinginto account communicative informativeness benefits communication convergencerates, and penalizing communicative complexity leads to human-like lexicon sizeswhile maintaining high utility.
Additionally, we find that VQ-VIB outperformsother discrete communication methods.
This work demonstrates how fundamentalprinciples that are believed to characterize human language evolution may informemergent communication in artificial agents.
Introduction
Good communication is a critical component of successful teams of humans and artificial agents butdiffering notions of “good” makes training agents to develop such communication challenging.
Oneview of communication focuses on utility by framing languages as successful to the extent they enablehigh task performance.
Emergent communication literature, wherein agents learn to communicatewhile optimizing a task-specific reward or utility, emphasizes this view [1–6].
However, training viautility alone may lead to slow convergence rates [7] and aspects of learned communication are highlysensitive to environment design (at times diverging from human naming patterns [8]) implying thatadditional constraints are needed in order to guide agents toward human-like communication.
Findingsuch constraints is an open question, both for artificial intelligence and for cognitive modeling.
Here, we address this question by building on a recent line of work that argues that human languagesevolve under pressure to efficiently compress meanings into communicative signals [9–13].
Thisnotion of efficiency is formulated in terms of the Information Bottleneck (IB) principle [14], whichis a general information-theoretic principle with broad scope in machine learning [15–19], andin this context, it can be interpreted as a tradeoff between the complexity and informativeness of
[Figure]
Informativeness :=
[Equation]
E
[Equation]
Complexity
[Equation]
=
[Equation]
Utility :
[Equation]
[Figure]
communication [9].
Intuitively, complexity corresponds to the number of bits that are needed for communication, and informativeness corresponds to how well a listener can infer the speaker’sintended meaning regardless of a specific task (e.g., humans understand the term “blue” in “avoidthe blue box” and “find a blue cup”).
We hypothesize that taking into account complexity andinformativeness, in addition to optimizing a task-specific utility function, may help guide artificialagents toward more natural, human-like semantic systems (e.g., learning a single word for “blue”rather than 20 unique words for very similar shades of blue).
We test this idea by training teams of artificial agents to optimize a tradeoff between maximizing taskutility, maximizing communicative informativeness, and minimizing communicative complexity inemergent communication settings, as depicted in Figure 1.
To this end, we extend prior work [20,21] and propose Vector-Quantized Variational Information Bottleneck (VQ-VIB), a method fortraining neural agents to compress inputs into signals, while maintaining a discrete (symbolic) signalrepresentation that is embedded in a continuous space.
In experiments, we find that training to increaseinformativeness improves the convergence rate of emergent communication to high-performancecommunication, while annealing the complexity loss generates a spectrum of learned communication,which we show aligns with a variety of human languages.
Additionally, we find that VQ-VIB appearsbetter able to achieve high utility and informativeness for the same complexity than other discretecommunication methods.
Related Work
We train neural agents, focusing on the effects of utility, informativeness, and complexity on emergentcommunication.
Here, we describe related work in emergent communication; we include approachesfor measuring informativeness and complexity in Section 3.
Generally, in emergent communicationsettings, researchers train agents to accomplish a cooperative task in which communication shouldenable better task performance [3, 22, 23].
For example, if one agent observes a goal locationthat another agent must navigate to, the team of agents would benefit from the “speaker” agentcommunicating with the “listener” agent.
Many emergent communication works have considered howenvironmental factors (such as reward, agent architecture, or more) affect learned communication [3,5, 22, 24].
We focus on subsets of the literature that explore aspects of informativeness and complexity,as well as specific applications in color reference games.
Notions of Complexity in Emergent Communication.
Recent emergent communication literaturehas taken initial steps beyond focusing only on utility by limiting complexity or encouraging informativeness, although these terms are rarely explicitly acknowledged.
There are many notions of alanguage’s complexity, including aspects of grammar, syntax, morphology, length, or compositionality [25–28]; we do not position ourselves within literature that seeks to define complexity.
Rather, weadopt a specific, existing metric of complexity, which we define more fully in Section 3.
Numerous works simplify communication by discretizing it, adding noise to the environment, or penalizinglarge vocabulary sizes; these methods all align with limiting complexity [21, 29, 30].
Wang et al.[31] explicitly limit complexity via a centralized message processor that acts as an informationbottleneck.
At the same time, other works find that auxiliary losses designed to create “meaningful”communication improve convergence rates [7, 32].
For example, Lin et al.
[33] use an autoencodingloss to reconstruct states from communication; this is tightly related to notions of informativeness.
However, because Lin et al. [33] do not penalize communication complexity, they note that theirmethod “may fail... where agents need to be more selective of the information communicated.”
Lastly, Chaabouni et al.
[8] measure (but do not directly control) emergent communication complexity whilechanging environment parameters to induce different forms of communication.
Thus, while emergentcommunication literature has begun to rediscover the importance of complexity and informativenessin communication, these terms have rarely been explicitly recognized, especially in training.
Emergent Communication and Color Reference Games.
We conduct experiments in colorreference games, which have been widely used in emergent communication literature [1, 34, 35, 8].
Typically, these reference games, or Lewis signalling games [36], comprise a speaker, who observesone color dubbed the “target,” and a listener, who observes the target color and distractor color(s).
Bycommunicating to the listener, the speaker seeks to have the listener identify the target.
We expandupon our specific experiment configuration in Section 4.4.
Color naming has been a central test-case for cognitive theories of semantic categorization andlanguage evolution [37].
It is a rare domain in which a large-scale cross-linguistic dataset, the World
Color Survey [WCS; 38], is available, documenting the wide diversity in color naming schemesacross languages.
Particularly relevant to our work is Zaslavsky et al.
[9]’s finding that color namingsystems across languages attain complexity–informativeness tradeoffs that are near-optimal in the
Information Bottleneck [IB; 14] sense.
Building on that, Chaabouni et al.
[8] showed that AI agentscan learn IB-efficient color naming systems by playing color reference games (and see [35] for arelated study using different notions of efficiency).
Unlike Chaabouni et al.
[8], who studied theeffect of environment design on communicative informativeness and complexity, we directly trainagents via a tradeoff of these terms without changing the environment.
In addition, we propose a newagent architecture that achieves higher utility for a given complexity than other methods.
Technical Approach
Here, we show how to adapt standard neural architectures to enable informativeness- and complexitybased objectives.
We then introduce a new neural architecture, Vector-Quantized Variational Information Bottleneck (VQ-VIB), for learning complexity-limited discrete representations in a continuousspace.
Lastly, we combine utility, informativeness, and complexity terms into an overall objective.
3.1 The Information Bottleneck Framework for Semantic Systems
We adopt and extend Zaslavsky et al.
[9]’s IB framework for semantic systems, which has substantialempirical support across languages and domains [10–13].
In this framework, speakers and listenersjointly optimize the IB tradeoff between the complexity and informativeness of their communicationsystem.
A speaker is characterized as a probabilistic encoder q ( c | x ) that, given an input, x ∼ p ( x ) ,generates a signal, c .
A listener is characterized as a probabilistic decoder q (ˆ x | c ) , that given c aimsto reconstruct the speaker’s input.
Complexity is measured by the mutual information betweenthe speaker’s inputs and signals, I ( X, C ) .
Informativeness is measured by the degree to whichthe speaker’s mental representation of x matches the listener’s mental representation of ˆ x .
Ingeneral, these mental representations are defined by probability distributions, m and ˆ m respectively,over some feature space.
In that case, minimizing the expected Kullback-Leibler (KL) divergence 113 E [ D [ m ∥ ˆ m ]] amounts to maximizing the informativeness of the communication.
This yields thefollowing optimization principle: min I ( X, C ) − β E
[ D [ m ∥ ˆ m ]] , where β is a tradeoff parameter.
Ithas been shown that this principle is equivalent to the IB principle [9].
Solving this optimization problem directly is challenging in large scale domains and with agentsthat are parameterized by neural networks.
To address this, we take a Variational Informationbottleneck [VIB; 18] approach, in which an upper bound is optimized, similar to ELBO in Variational
Autoencoders [VAEs, 39] and β -VAEs [40].
In practice, we also approximate informativeness by anautoencoding loss, d ( x, ˆ x ) , defined by the mean squared error (MSE) loss of the speaker’s input andthe listener’s reconstruction.
We next explain our approach in more detail.
3.2 Vector Quantized Variational Information Bottleneck (VQ-VIB)
Our proposed neural architecture, the Vector Quantized Variational Information Bottleneck (VQVIB), learns information-constrained discrete communication in a continuous space.
Our techniquecombines prior work in discrete representation learning [VQ-VAE, 20] with VIB [18].
3.2.1 Vector-Quantized Variational Autoencoder
First, we summarize a simplified Vector-Quantized Variational Autoencoder (VQ-VAE) architecture [20].
As in a standard VAE, our encoder is a conditional distribution over latent representations,given an input: q ( z | x ) .
A deterministic feedforward network maps from an input, x , to a latent representation, z : we dub such encodings, z ( x ) .
Next, for a VQ-VAE parametrized with K embeddingvectors, ζ , in the latent space, z ( x ) is discretized to a particular embedding vector, ζ i , by mapping tothe nearest embedding vector.
Thus, the discretized embedding is determined by Equation 1.
[Equation]
3.2.2 Variational Information Bottleneck (VIB)
We also use notions from VIB, a variational approach for generating information-constrained continuous representations with neural nets.
Following Alemi et al.
[18], an encoder network mapsfrom an input to µ and σ , which are used as parameters to a Gaussian distribution from which alatent representation, z , is sampled.
VIB networks are trained via a weighted combination of aclassification loss and ( β times) the KL divergence of µ ( x ) and σ ( x ) from a unit Gaussian; this KLloss corresponds to an upper bound of I ( X, Z ) .
3.2.3 VQ-VIB Architecture
Here, we integrate VQ-VAE and VIB for a novel method of generating discrete, information-boundedrepresentations.
In the VQ-VIB method, an encoder is used to sample a representation z in acontinuous space, followed by a quantization layer that discretizes the sample by mapping to thenearest encoding ζ .
This architecture is depicted in Figure 2 and is formally defined in Equation 2.
In the VQ-VIB formulation, the likelihood of a communication vector, c , given an input, x , corresponds to the likelihood that an encoding z , sampled with parameters µ ( x ) and σ ( x ) , is closest to anembedding vector equal to c .
Intuitively, this approach corresponds to uncertainty over which “word”to use.
One may compute an upper bound on I ( X, C ) using the standard KL divergence used for
Gaussians in a VIB, simply using µ ( x ) and σ ( x ) .
[Equation]
We proposed an additional network architecture, dubbed VQ-VIB After , that is related to our main
VQ-VIB method but relates to different forms of bounded complexity and does not generate discretecommunication vectors.
We included discussion of this approach in Appendix B; in our main paper,we include results of using VQ-VIB After as a baseline, but again emphasize that the approach is notrelated to the core focus of our work on discrete emergent communication.
To prevent ambiguityin presenting our results, at times we label VQ-VIB results as “VQ-VIB Before” to distinguish ourstandard method, in which sampling is performed before discretization, from VQ-VIB After.
3.2.4 Complexity Equivalence Classes
Although one may approximate I ( X, C ) , simply penalizing this term in training was insufficient totrain VQ-VIB agents to use fewer unique discrete embeddings.
Fundamentally, two agents couldbelong to the same complexity equivalence class (with the same I
( X, C ) )
but have different numbersof embeddings if several embeddings are nearly identical (see [41] for analysis of these equivalenceclasses in IB).
Intuitively, this corresponds to notions of either communicating with many words, eachof which is difficult to disambiguate from other words, or fewer words, each of which is quite distinct.
We wish to bias agents to use fewer distinct clusters, which corresponds to the entropy over thecategorical distribution over embeddings.
Training with the exact entropy in the VQ-VIB architectureis infeasible given the non-differentiablity of the argmin operation used in discretization.
Instead,we use the entropy of an approximation of the distribution over embeddings: the softmax of thenegative L2 distance from embeddings to each token.
(We leave exploration of other approximationsto future work.)
Because this loss is only used to change the number of clusters within a complexityequivalence class, we require only a small training signal.
In our experiments, we multiply this lossby 0.05, making it orders of magnitude smaller than other losses.
3.3 Overall Optimization
Here, we combine losses into an idealized objective function to maximize, shown in Equation 3.
This objective function is the weighted combination of three terms, wherein we seek to maximize atradeoff between high utility, high informativeness (low expected distortion d ), and low complexity.
Intuitively, these three scalar weights form a simplex that describe a range of communicationfrom silence (minimizing complexity) to task-optimized communication that does not generalize(maximizing utility) to complete descriptions (maximizing informativeness).
[Equation]
In practice, we train neural agents via gradient descent to maximize differentiable approximations ofthese terms.
Utility is represented via policy gradient methods (for MARL settings) or crossentropyloss.
Informativeness is computed via the negative mean squared error between the speaker’s inputand a decoder’s output, given the communication.
Lastly, an upper bound of complexity is computedas the KL divergence between the speaker’s distribution over communication and a fixed marginal(either a uniform distribution for onehot communication, or a unit Gaussian for others).
Experiment Preliminaries
We formulated hypotheses about the roles of utility, informativeness, and complexity in emergentcommunication, and designed experiments to test them.
4.1 Hypotheses
We expected that increasing informativeness is a bias for learning useful communication, but that doing so would likely increase complexity, which must be bounded to align with human communication.
More specifically, our two hypotheses were:
[Figure]
• Training to increase informativeness will cause agents to converge to successful communica-
tion systems more quickly than without such training, although increasing informativeness
will increase complexity.
• Complexity-bounded agents will generate more human-like discrete communication systemsthan unbounded agents.
4.2 Trials and Baselines
We compared 5 types of neural architectures in our experiments: continuous, onehot, Proto (based onthe prototype-based communication from Tucker et al.
[21]), VQ-VIB, and VQ-VIB After.
We againemphasize that only VQ-VIB and onehot communicate via a finite set of symbols, whereas othermethods fall out of the scope of this work of complexity-bounded, discrete emergent communication.
For each domain, we trained agents from scratch using 5 random seeds, starting at 0.
As a highlevel overview, Proto.
agents internally compute a onehot vector and then multiply it by a learned“embedding” matrix, which generates a finite set of vectors in continuous space.
Full details of thearchitectures, hyperparameters, and environments are included in Appendix A.
4.3 Particle Worlds
We trained agents in two environments inspired by those introduced by Tucker et al.
[21]: 9 pointsand Uniform.
Both environments consisted of two-dimensional worlds in which a speaker observeda target location that a listener had to reach but could not directly observe, as depicted in Figure 3.
In the 9 points environment, there were 9 landmarks at fixed locations, one of which was chosen atrandom as the target for each episode.
In the Uniform environment, two landmarks were spawned inlocations drawn uniformly at random in the world; one landmark was specified as the target.
Agentsreceived identical reward equal to the negative distance from the listener to the target and were trainedusing Multi-Agent Deep Deterministic Policy Gradient [3].
The listener could not observe the targetlocation, and the speaker could only communicate at the first timestep by broadcasting a vector thatwas observable at all future timesteps; thus, an optimal team policy would consist of the speakercommunicating about the target at the first timestep, and the listener navigating straight to the targetbased on the message.
The two environments differ in their complexity: in the 9 points environment,the speaker had a finite amount of information about which to communicate, whereas in the Uniformenvironment, the location could be any continuous value (up to the computer’s float precision).
4.4 Color Reference Games
We also tested agents in reference games, or Lewis signalling games [36], which are often used inemergent communication studies; by focusing on a color domain, which has corresponding dataon human communication [37, 38], we hoped to identify similarities between agent and humanlanguages [35, 8].
Human languages use a finite set of words, so we only tested onehot and VQ-VIB
[Figure]
communication.
Other architectures (continuous, Proto, and VQ-VIB After) sample from continuousspaces and could therefore generate infinite unique communication vectors.
In these reference games, we trained agents within a supervised learning framework, differentiatingthrough both the speaker and listener.
A diagram of the network architecture is included in Figure 7in Appendix A. A speaker observed a target color and (if in the “Common Ground” setting, explainedlater) a distractor color and output a communication vector.
A listener observed the speaker’scommunication and the target and distractor colors and predicted which of the colors was the target.
Colors were represented by their 3D CIELAB coordinates (illustrated in Figure 7 in Appendix A),and speaker observations were corrupted by zero-mean Gaussian noise with a standard deviation of 8,as motivated by human perception [9].
Both colors were sampled from the set of 330 World Color
Survey (WCS) colors according to a human-based prior from [9].
1 Only the speaker agent observedwhich color was the target, so the agents had to learn to convey information from the speaker to thelistener.
Agents were trained via binary crossentropy loss computed on the listener’s output, whichworked more reliably than MARL methods (corroborating Chaabouni et al.
[8]’s findings).
We trained agents in two settings: Isolated, in which the speaker only observed the target, and
Common Ground, in which the speaker observed both the target and distractor colors.
The Isolatedscenario resembles the WCS setting, in which participants were asked to name individual colors.
Conversely, the Common Ground scenario was inspired by work in pragmatics and Rational Speech
Act, wherein humans communicate less information than fully describes a scene, but enough toinduce listeners to behave in the desired manner [42, 43].
We note that, unlike Chaabouni et al.[8], we do not change the subset of sampled distractor colors to induce different complexity; ourframework results in more stable training and allows us to directly vary communication complexitywithin the same environment.
Experiment Results
Our results indicated that an informativeness loss improved convergence, annealing λ C generateda suite of agents of different complexities that more closely approximated human languages, and
VQ-VIB agents achieved greater utility, for the same complexity, than onehot agents.
5.1 Particle Worlds
In the particle worlds, increasingly penalizing complexity led agents to lower-utility communication,although architecture choices and training to increase informativeness could increase reward.
First,we conducted experiments in which we did not penalize complexity, but varied λ I across trials.
Across all architectures, higher λ I increased the mean team reward and improved convergence rates,as shown in the learning curves of VQ-VIB in Figure 4 a.
We note that successfully training agentswith a mix of informativeness and utility loss represents an improvement over Lin et al.
[33], whostruggled to train agents with a combination of losses.
Next, for a fixed λ I = 1 . 0 (to improve convergence), we trained agents while annealing λ C ; thissmoothly decreased the complexity of communication.
Results for the Uniform environment alignedwith our expectations Figure 4 b.
Each curve indicates the cubic best fit to the data generatedacross 5 random seeds for each configuration (discarding outliers for reward less than -120), withcomplexity and reward measured at each increment of 1,000 training episodes from 3,000 to 10,000.
As complexity decreased (moving right to left in the plot), reward decreased as well.
Visualizations ofagent behavior confirmed that low-complexity VQ-VIB agents communicated via increasingly coarsediscretizations of the space and are included in Appendix C.
The plots align with notions of howhumans might communicate in natural language about a continuous space via coarse discretizationslike “north” or “south” rather than hundreds of specific words for any possible location.
Lastly, these results suggest that neural architecture choice was important.
In both 9 points and
Uniform, the VQ-VIB agents tended to achieve higher reward, for the same complexity, as onehot or
Proto agents.
(Continuous communication reflected an optimal upper bound but violated our desire tolearn discrete communication.)
Furthermore, adding more encodings (9 discrete encodings vs. 100)significantly increased VQ-VIB performance, while changing onehot agents to communicate in 9or 100 dimensions had a smaller effect.
Parametrizing VQ-VIB or onehot agents with even morediscrete tokens (200 or 500) did not improve mean utility beyond 100 tokens, suggesting that greaterparametrization will not decrease the gap.
This section includes partial results, focusing on VQ-VIBin the Uniform environment; full results are in Appendix C. We observed similar benefits from highλ I , similar complexity-reward tradeoffs, and similar advantages of VQ-VIB over other architecturesin the 9 points environment.
5.2 Reference Games
In color domains, encouraging informativeness and penalizing complexity generated a range ofcommunication that closely approximated optimal compression and human languages.
5.2.1 Isolated Color Reference Game
In the Isolated color reference game, an informativeness training loss appeared necessary for onehot,while a complexity-based loss was useful for generating a spectrum of communication conventions.
[Table]
First, we trained agents with a range of λ I values while leaving λ C = 0 .
We briefly report two mainfindings: 1) for a given λ I , agents converged to a small range of complexity and informativenessand 2) VQ-VIB agents typically learned more complex and informative communication than onehotagents did, although increasing λ
I shrank the gap.
These experiments are similar to those conductedby Chaabouni et al.
[8], who did not train with informativeness or complexity losses.
Full results forthese experiments are included in Appendix D.
Second, we tested agents for λ
I = 1 .
while linearly annealing λ C to penalize complexity.
As λ Cincreased, the communication complexity decreased, as shown in Figure 5 (left).
Over 5 randomseeds
, the points in the plot, generated at each training epoch, closely approximated the IB upperbound for maximum informativeness for a given complexity.
(A zoomed-in version of similaranalysis, conducted with 1024 possible tokens, is included in Figure 14.)
Inspection of the learnedcolor naming schemes at different complexity levels revealed important trends (Figure 5 right).
Ascomplexity decreased, the number of communication clusters emitted by VQ-VIB agents decreased(visualized via 2-dimensional principle component analysis).
Clustering in the communication spacewas reflected in the color space (e.g., “yellow” and “orange” merging).
More generally, a least-squareslinear correlation between the distance between pairs of communication vectors and the distancebetween associated colors (in CIELAB space) showed r 2 = 0 .
97 ± (0 . 01) for VQ-VIB, whereas foronehot communication r 2 = 0 .
76 ± (0 . 03) .
That is, for VQ-VIB, if two communication vectors werefar apart in communication space, they described colors that were far apart in CIELAB space.
Thisshows a fundamental connection between the form and meaning of communication used by VQ-VIBagents, which onehot-based communication is architecturally unable to support.
Lastly, we compared agent color naming schemes to each other and to human language data.
For eachof the 110 languages in the WCS Dataset [38], we found the best-matching agent as determined byminimizing gNID, a measure of dissimilarity of naming distributions introduced by Zaslavsky et al.[9] .
Example nearest languages are shown in Figure 5 in dashed boxes.
Despite having no superviseddata, the artificial agents learned similar color naming schemes to humans, but only when agentschanged complexity to match human languages’ (shown via the gNID metrics in Table 1).
Thishighlights the importance of controlling complexity in emergent communication: only by changingcomplexity could we generate naming systems similar to human communication.
Thus, methodslike Lin et al.
[33] that encourage informativeness without penalizing complexity would be unableto recreate our results, and Chaabouni et al.
[8] would have to change the environment to indirectlycontrol complexity.
Finally, when comparing VQ-VIB and onehot naming schemes, we found thatagents were similarly efficient (deviation from optimality is calculated as ϵ in Table 1), but that
VQ-VIB achieved higher utility, for the same complexity, as shown in Figure 6.
Varying the numberof tokens from 330 to 1024 or replacing the supervised training mechanism with REINFORCEbased training did not change results, as examined in Appendix D.
This corroborates patterns fromthe particle world environments, wherein VQ-VIB agents achieved higher rewward, for the samecomplexity, as other methods.
Together, these results indicate that annealing complexity is a critical component in recreating humanlike color naming systems.
Furthermore, VQ-VIB matched the form and meaning of communicationby merging clusters in the communication space that represented nearby colors, and outperformedonehot communication by achieving higher utility for the same complexity.
[Figure]
5.2.2 Common Ground Color Reference Game
We tested agents in the Common Ground color reference game to evaluate contextualized communication.
Crucially, these experiments tested agent behavior when utility and informativeness werenot identical: utility could be maximized by the speaker only communicating about the target color,whereas informativeness was maximized by communicating about both target and distractor.
In thisdistinct setting, the same trends as in the Isolated experiments held.
An informativeness loss improvedconvergence, and, while both VQ-VIB and onehot were nearly optimally efficient, VQ-VIB achievedgreater utility than onehot did, as shown in Figure 6.
Complete results and figures are included in
Appendix D, confirming that agents learned efficient, contextualized communication.
Contributions
In this work, we explore the interplay of utility, informativeness, and complexity in emergentcommunication.
To the best of our knowledge, we are the first to directly optimize agents tocommunicate according to a weighted combination of all three terms.
In our experiments, we foundthat training agents while rewarding informativeness led to faster communication convergence, andpenalizing complexity created more human-like communication (supporting our two hypotheses).
Further, we found that our novel neural architecture, VQ-VIB, achieved greater utility for the samecomplexity than other discrete methods.
VQ-VIB combines the strengths of the methods it buildsupon: it has a symbolic structure, defined by quantized vectors, but it is also explicitly uses an IBencoder to sample in a continuous space.
Natural extensions of our work include further improvementsto neural architectures, incorporation of new, cognitively-inspired terms to guide communication, ormore complex domains, which could better illustrate advantages of VQ-VIB.
Our work represents a step towards reconciling principles from cognitive science with emergentcommunication literature, which we hope better supports human preferences.
However, we recognizethat, by quantifying properties of languages, our results may be misinterpreted to assign relative valuesto one language over another (e g., higher complexity being better).
We insist that our framework 1)has only been applied to narrow domains that do not capture any language’s entirety and 2) does notprioritize any term and that any tradeoff between these terms is equally valid.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section TODO.
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes] See abstract, introduction, and findings
(b) Did you describe the limitations of your work?
[Yes] See Section 6, in which wenote that we tested in limited domains and Section 3.2.4, in which we note that anapproximation we made may be improved upon in future work.(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] SeeSection 6 in which we discuss how readers may misconstrue quantitative measures oflanguages as assigning value to one over the other.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes] See footnote
[Table]
for link to anonymized code
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Appendix A
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See error bars in plots and Table 1, in Section 5.
Allscatter plots directly included the raw results from all 5 random trials.
(d) Did you include the total amount of compute and the type of resources used (e.g., type [Table] of GPUs, internal cluster, or cloud provider)?
[Yes] Training times and computational [Table] resources are listed in Appendix A. [Table]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes] We use the World
Color Survey (WCS) data, as well as the publicly-available code by Zaslavsky et al.
[Table]
Both sources are cited.
[Table]
(b) Did you mention the license of the assets?
[Yes] We note that both are publicly-available [Table]
and link to them in Footnote 1.
[Table] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
Anonymized code is linked to in a footnote.
[Table] (d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A] We only used the existing WCS dataset.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A] We only used the existing WCS dataset.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Appendix: Implementation Details
Here, we include implementation details that we omitted from the main paper that can be used toreproduce our experiments.
A.1 Multi-Agent Reinforcement Learning Experiments
We trained agents in the 9 points and Uniform environments with Multi-Agent Deep Deterministic
Policy Gradient (MADDPG), a standard method for multi-agent reinforcement learning [3].
Agent architectures were identical in both environments.
A speaker agent consisted of 2, 64-neuronfully-connected layers with ReLU activations, followed by a speaker “head.”
The head implementedthe specific communication method (e.g., continuous, VQ-VIB, etc.) by taking the 64-dimensionalrepresentation from the speaker and going through one of 5 possible architectures to produce a vectorin the communication space, R c .
(As discussed later, c varied by environment and architectureslightly.)
• Continuous: The continuous communication head consisted of two fully connected layers
branching off the speaker encoding to generate a µ and logvar (the log of the variance) from
which the communication vector was sampled using the reparametrization trick.
Complexity
was measured as the KL divergence between the parameters µ and logvar and a unit
Gaussian.
[Table] • Onehot: The onehot communication head consisted of one fully connected layer from the speaker encoding to R c .
Onehot vectors were generated using gumble softmax sampling,
[Table]
with temperature 1.
Complexity was measured as the KL divergence between the distri-
[Table]
bution from which the onehot vector was sampled and a uniform categorical distribution.
[Table]
Thus, complexity corresponds to the difference between the conditional and unconditional
[Table]
entropies of messages, unlike non-IB frameworks that often penalize unconditional entropy.
• VQ-VIB:
The VQ-VIB communication head first passed the speaker encoding through two
fully connected layers, branching off the encoding, to generate a µ and logvar , as in the
continuous communication case.
A communication vector was sampled from the distribution,
whereupon it was discretized to the nearest quantized vector, ζ , in the communication space
(as done in a standard VQ-VAE, without the sampling).
Complexity was measured as the
KL divergence between the parameters µ and logvar and a unit Gaussian.
• VQ-VIB
After: The VQ-VIB After communication head first passed the speaker encoding through two fully connected layers to generate an encoding, z , and logvar .
The encoding, z , was discretized to the nearest quantized vector, ζ .
Lastly, a communication vector, c , was sampled from a normal distribution centered at ζ with variance specified via logvar .
Complexity was measured as the KL divergence between the parameters ζ and logvar and
a unit Gaussian.
• Proto:
The Proto head mapped the speaker encoding through a single fully-connected layer
to a categorical distribution over k neurons, for k representing the number of prototypes.
A k -
dimensional onehot vector was sampled using gumble softmax; a prototype, p was selected
by multiplying the prototype matrix by the onehot vector.
A different fully-connected
layer mapped from the speaker encoding to logvar .
Lastly, a communication vector, c ,was sampled from a Gaussian centered at p with variance via logvar .
Complexity was
measured as the KL divergence between the parameters p and logvar and a unit Gaussian.
This closely resembles the VQ-VIB After architecture, but with a different discretization
mechanism.
All agents were trained with an Adam optimizer with learning rate 0.01 and batch size
A.1.1 9 points Environment
For all agent architectures except for onehot, the communication dimension was 2; onehot usedcommunication dimension 9 so that the number of discrete representations allowed by the architecturewas not the limiting factor for complexity.
Similarly, the VQ-VIB and Proto agents used 9 discreteembeddings.
In our main results (presented in Figure 4), agents were trained for 10,000 episodes, with each episodelasting 100 timesteps.
The listener agent was always spawned at the origin, equidistant from eachlandmark.
The informativeness loss weight, λ I , was constant at 1.0.
For the continuous and VQ-VIBcommunication methods, the complexity loss weight, λ C increased by 8 ∗ 10 − 6 per episode.
Theonehot and Proto methods suffered from communication collapse at that rate, so results λ C increasedhalf as quickly: at 4 ∗ 10 − 6 per episode.
Training a team in the 9 points environment for 10,000 episodes took approximately 15 minutes on adesktop computer with 16 Intel i9 cores for all architectures, except for onehot, which took roughly20 minutes.
This difference in time is attributable to the larger communication dimension needed foronehot.
A.1.2 Uniform Environment
In the Uniform environment, we used identical parameters as in the 9 points environment, except forthe changes noted below.
For all agent architectures except for onehot, the communication dimension was 3; for onehot, weexperimented with communication dimensions of 9 and 100.
Similarly, we tested VQ-VIB and
Proto agents when using 9 or 100 discrete embeddings.
The listener agent was spawned at a startinglocation sampled uniformly at random from the map (bounded between -200 and 200 for both x andy ).
Otherwise, we used the same number of training episodes, episode duration, and complexityannealing rates as in 9 points.
Training a team in the Uniform environment for 10,000 episodes took approximately 15 minutes,except for onehot, which took roughly 30 minutes for 100-dimensional communication.
A.2 Color Reference Game Experiments
Here, we include the implementation details used in our color reference game experiments.
Agents were trained in the overall architecture depicted in Figure 7.
The speaker agent wasparametrized with 3 fully connected layers with ReLU activations and hidden dimension 64, followedby a communication head.
The communication heads used the same architectures as specifiedin the particle world experiments: gumbel softmax was used for generating onehot vectors, anda straight-through estimator was used for discretization in VQ-VIB architectures [44, 45].
In allexperiments, the communication dimension was set to 32 for all agents except onehot, for which thecommunication dimension was set to 330 (to match the expressive power of other discrete agents,that each had 330 discrete tokens in the 32 dimensional space).
[Equation]
Without annealing the complexity loss, 50 training epochs sufficed for agents to converge to a stablesolution.
Thus, in our experiments without complexity loss, we only trained for 50 epochs.
For laterexperiments, wherein we annealed complexity, we first trained agents for 50 epochs before beginningto anneal the complexity term for the next 40 epochs (for a total of 90 epochs).
We note that, for
VQ-VIB, we set the complexity loss to 0 during this “burnin” period, but for onehot communication,it appeared necessary to set the loss to 0.05 to prevent numerical instability during the first epochafter the burnin period.
During complexity loss annealing, we increased λ C by 0.01 during eachepoch for VQ-VIB, and by 0.1 for onehot.
These different rates were necessary to induce similarranges in complexity of learned communication.
In the Isolated color reference game, the speaker observed a only the 3-dimensional, corrupted targetcolor.
In the Common Ground setting, the speaker observed a 6-dimensional vector, generated byconcatenating noisy versions of the target and distractor colors.
In both settings, the listener observedthe communication output by the speaker, as well as the target and distractor colors (ordered randomly,without added noise).
Results were analyzed using code partially-based on publicly-available codereleased by Zaslavsky et al.
We trained agents on a desktop computer with 16 Intel i9 CPUs and 1 NVIDIA GeForce RTX2080.
Training for 90 epochs in both the Isolated and Common Ground settings took approximately3 minutes for all agent architectures.
Agents were trained with an Adam optimizer with defaultparameters, with batch size 1024.
B Appendix: VQ-VIB After
We proposed an additional architecture, VQ-VIB After , that is closely related to our main VQVIB method.
In VQ-VIB After, an encoder to generates a discrete representation ζ , whereupona communication vector c is sampled from a Gaussian in the continuous space centered around ζ(Equation 4).
This “sample-after” method is depicted in Figure 8, and corresponds to an intuitivenotion of choosing a word (a discrete embedding) deterministically but then being uncertain aboutthe form (by sampling in the continuous space around it).
This architecture fits into the standard Information Bottleneck framework, as complexity can bepenalized via standard variational methods.
However, it is distinct from discrete communication [Figure] methods because communication vectors are ultimately sampled from a continuous space.
Thus, thiswork does not fit naturally into the main focus of this work, but we believe it could have applicationsin other settings, such as speech processing (wherein discrete words are pronounced in a continuousauditory space).
C Appendix: Full Particle World Results
For brevity, we only included partial results from the particle-world experiments in the main paper.
Here, we include the full results.
First, we experimented with varying λ
I for different environments and neural architectures, withoutannealing complexity.
Training curves for both environments, using varying λ
I for each neuralarchitecture, are included in Figures 9 and 10.
For each configuration (a particular speaker architectureand λ I ), we trained 5 teams from scratch for 5,000 episodes and plotted the median and quartilerewards, smoothed over a 50 episode window.
A consistent trend emerged: higher λ
I led to faster convergence to a higher reward.
Indeed, thiscorroborates findings by Lin et al.
[33], who used an autoencoder architecture for communicationin multiagent settings, but here we showed good convergence properties when mixing utility andinformativeness terms (whereas they reported worse results when adding utility losses).
Overall,these findings motivated training agents to convergence with high λ I before beginning to penalizecomplexity with λ C .
[Figure] [Figure] other discrete higher Next, we include the results of annealing complexity in both the 9 points and Uniform environments,as depicted in Figure 11.
In both environments, as we increased λ C , complexity and reward decreased.
Comparing both VQ-VIB architectures to other discrete communication methods showed that VQVIB agents tended to obtain a higher mean reward, for a given complexity, than other methods.
In the Uniform environment, we also compared different variants of VQ-VIB, Proto, and onehotagents based on the maximum number of discrete messages allowed by the neural architectures.
Wetested agents parametrized with either 100 or 9 unique messages; we hypothesized that initializingagents with many discrete messages and then gradually decreasing complexity would result in better [Figure] performance than training with few messages from the start.
Indeed, we observed that was the case.
As seen in Figure 11 b, VQ-VIB agents parametrized with 100 embeddings consistently achievedhigher reward for the same complexity than VQ-VIB agents parametrized with only 9 embeddings.
This trend held even when the complexity was lower than log 2 (9) bits.
Lastly, we visually analyzed VQ-VIB communication in the 9 points and Uniform environments.
As agents used less complex communication (i.e., fewer distinct messages from the speaker to thelistener), one would expect each message to refer to coarser regions of space and for the listener totherefore end in a smaller set of possible destinations.
We recorded, over 1,000 episodes, the targetlocation, the speaker’s communication, and the listener’s trajectory.
In the Uniform environment,using a discretized representation of the continuous space (for visualization purposes), we performedsimilar “modemap” analysis as in the color reference game.
That is, we recorded the most likelytoken associated with each target location and colored all locations associated with the same targetwith the same color.
(The exact color was determined via a color map designed for visualizationpurposes, or left as white if no single token was used for that location more than 50% of the time.)
Inthe 9 points environment, we did not perform such coloring because the set of possible destinationswas already discrete.
In both environments, however, we visualized listener agent’s trajectories.
Visualizations of VQ-VIB teams are included in Figure 12.
Figure 12 is consistent with expected behavior.
As communication complexity decreased, the speakeragent used fewer distinct tokens and the listener moved to fewer distinct destinations.
Furthermore,as seen in the Uniform environment, the speaker communicated about increasingly coarse locations(indicated by larger spatial regions referred to by the same token, visualized with color).
This visualanalysis confirms similar trends as in the color domain, highlighting the general applicability of ourframework for shaping communication complexity.
D Appendix: Color Reference Game Full Results
We omitted some of the color reference game results from the main paper due to space constraints.
Here, we include our complete results.
D.1 Isolated Color Reference Game
First, we include results from the Isolated setting, in which we trained agent with different values ofλ I without penalizing complexity.
These results are included in Figure 13.
Setting λ
I = 0 inducedsimilar conditions to those used by Chaabouni et al.
[8]; any positive λ I favored more informative(and therefore more complex) communication.
Given the monotonically increasing nature of the IBbound, an optimal agent trained with any positive λ I would maximize utility and informativeness bymaximizing complexity, up to a maximum of log 2 (330) bits.
Instead, we found that agents appeared toconverge to lower complexity values, although increasing λ
I tended to somewhat increase complexityand informativeness.
For example, onehot communication increased in complexity from roughly 2.3bits for λ I = 0 .
to 3.0 bits for λ
I = 1 . 0 .
This alone is a novel result compared to Chaabouni et al.[8], who trained agents with no informativeness loss and observed lower-complexity communication.
Curiously, VQ-VIB agents automatically converged to informativeness around 3.5 bits regardless
ofthe λ I value, although increasing λ
I did improve training stability.
This difference between onehotand VQ-VIB indicates that the differing architectures themselves have different inductive biases.
Without positive λ I , onehot learns low-informativeness communication, but increasing λ
I generated more complex agents, especially for onehot
We conducted further experiments to compare agents with the neural agents presented by Chaabouniet al.
[8], wherein onehot agents were parametrized with a communication dimension of 1024.
Wetherefore tested VQ-VIB and onehot agents with 1024 tokens, instead of the 330 used for the resultsin our main paper.team This hyperparameter change largely did not alter results.
As shown in
Figure 14, both VQ-VIB and onehot achieved nearly optimal informativeness for a given complexity,and VQ-VIB achieved higher utility than onehot (results zoomed in to complexity values between 1and 2 bits for clearest plotting of these trends).
Lastly, we performed additional experiments to investigate the effect of training with REINFORCE,instead of the supervised learning framework used in our main experiments (e.g., backpropagatingthrough gumbel softmax) [46].
Chaabouni et al.
[8] had found that training with REINFORCEtypically resulted in lower-complexity communication than when training with messages generatedvia a gumbel softmax layer, and that training with REINFORCE was less stable.
Results from our REINFORCE experiments, using 330 tokens for onehot and VQ-VIB, are included in Figure 15.
Wetrained agents with λ U = 1 . 0 , λ I = 50 . 0 , and λ C was annealed from 0 to 0.1, and our code wasadapted from Rita et al.
[Figure] [Figure]
Our results from these experiments align with all prior findings: VQ-VIB and onehot achieved highcommunicative efficiency, and VQ-VIB produced greater utility than onehot did.
Beyond those corroborating results, our experiments aligned with Chaabouni et al.
[8]’s findings and further illustratedthe importance of informativeness losses and the VQ-VIB architecture.
With zero informativenessloss (i.e., λ I = 0 ) onehot agents often failed to converge to any useful communication (as indicatedin prior art), but by increasing λ
I , agents consistently converged to useful communication.
Indeed,with λ I = 50 and λ C = 0 , onehot agents learned to communication with complexity of up to 2.4bits, greater than any complexity value found by Chaabouni et al.
[8] using REINFORCE over 180seeds (see Fig.
5 in that paper).
Furthermore, we observed that VQ-VIB agents typically convergedto higher informativeness (and higher complexity) communication than onehot when complexity wasnot penalized, indicating that further investigation of inductive biases is a promising direction forfuture work.
Overall, this experiment largely replicates our findings from the main paper.
Thus, findings from our REINFORCE experiments indicate that the main conclusions of our paperhold true across several training mechanisms, and furthermore that our framework (including inparticular the informativeness loss) can address issues already noted in
D.2 Common Ground Color Reference Game
Here, we include the full results from our Common Ground color reference game experiments.
Giventhe lack of human-labeled data for such a setting, we were unable to compare agent communicationwith human languages.
However, by analyzing communication when varying the target for a fixeddistractor, or by varying the distractor for a fixed target, we established that agents conditionedcommunication both on the target and distractor colors.
[Figure]
We trained agents in the Common Ground setting by randomly sampling target and distractor colors,as in the Isolated setting, but here the speaker observed (noisy) versions of the distractor as wellas the target.
Based on insights that training to increase informativeness produced more complexcommunication, we trained agents for 30 episodes with λ I = 1 . 0 .
For the next 70 episodes, weannealed the complexity weight, λ C , using the same rates as in the Isolated setting; this producedcommunication at different levels of complexity.
In the main paper, we showed how VQ-VIBachieved higher utility, for the same complexity, than onehot (Figure 6).
We further analyzed communication in two ways: using a fixed distractor with varying targets andusing a fixed target with varying distractors.
First, for a fixed, gray distractor (#46 in the WCSdata), we measured the standard metrics of informativeness and complexity.
That is, we set thedistractor color to gray, and we recorded, for each of the other 329 colors as the target, the speaker’scommunication.
We then performed the same complexity and informativeness analysis as in the
Isolated setting, and found that both VQ-VIB and onehot communication remained near-optimallyefficient, as shown in Figure 16 a.
Second, we measured whether agents conditioned their communication on the distractor color.
Unlikethe prior evaluation, in which we varied the target color for a fixed distractor, here, we evaluatedcommunication for different distractors.
Plots of a VQ-VIB agent’s naming scheme, using either anorange (color #212) or purple (color #325) distractor are shown in Figure 16 b.
As expected, the agent displayed different resolutions in regions of color depending upon the distractor.
Specifically, for an orange distractor, the speaker had finer resolution around shades of orange andyellow; conversely, the speaker had finer resolution around shades of purples for a purple distractor.
This aligns with expectations that, when trying to differentiate between colors, one must use morecomplex or accurate words to distinguish similar colors.
Jointly, our findings indicate that many of the themes identified in the Isolated setting – nearly-optimalcommunicative efficiency, the importance of annealing λ C to generate varied complexity, and theadvantages of VQ-VIB over onehot – held in the Common Ground setting as well.