Trading off Utility, Informativeness, and Complexity in Emergent Communication
Abstract
Emergent communication (EC) research often focuses on optimizing task-specific utility as a driver for communication.
However, there is increasing evidence that human languages are shaped by task-general communicative constraints and evolve under pressure to optimize the Information Bottleneck (IB) tradeoff between the informativeness and complexity of the lexicon.
Here, we integrate these two approaches by trading off utility, informativeness, and complexity in EC.
To this end, we propose Vector-Quantized Variational Information Bottleneck (VQ-VIB), a method for training neural agents to encode inputs into discrete signals embedded in a continuous space.
We evaluate our approach in multi-agent reinforcement learning settings and in color reference games and show that: (1) VQ-VIB agents can continuously adapt to changing communicative needs and, in the color domain, align with human languages; (2) the emergent VQ-VIB embedding spaces capture semantically meaningful structures; and (3) encouraging informativeness leads to faster convergence rates and improved utility, both in VQ-VIB and in prior neural architectures for symbolic EC, with VQ-VIB achieving higher utility for any given complexity.
This work offers a new framework for EC that is grounded in information-theoretic principles that are believed to characterize human language evolution and that may facilitate human-agent interaction.
Introduction
Good communication is a critical component of successful teams of humans and artificial agents, but differing notions of “good” makes training agents to develop such communication challenging.
One view of communication focuses on utility by framing languages as successful to the extent they enable high task performance.
Emergent communication (EC) literature, wherein agents learn to communicate while optimizing the utility associated with a task-specific goal, such as reaching a target landmark, emphasizes this view [1–6].
However, training via utility alone may lead to undesirable properties of the EC system, such as violations of linguistic universals [7], poor human interpretability [8], slow convergence rates [9], or high sensitivity to environment design [10].
This implies that some constraints may be needed to guide agents toward human-like communication.
Another view of communication, often considered in the cognitive science literature, focuses on task-agnostic communicative objectives as major forces shaping human languages [11, 12].
Most relevant to our work is a recent line of work [13–17] that argues that languages evolve under pressure to efficiently compress meanings into communicative signals by optimizing the Information
Bottleneck (IB) principle [18].
In this context, IB can be interpreted as a tradeoff between the
[Figure]
Task agnostic ( ! "
=0)
T o o s p e c
i f i c , s l o w c o n v e
r g e n c
e
( ! # = 0 )
T oo c o m p l e x f o r hu
m a n s ( ! $ = 0 ) !
$ ! #
Complexity ! "
# #$
Informativeness
Utility % & ' complexity and informativeness of the lexicon [13].
Intuitively, complexity corresponds to the number of bits allocated for communication, and informativeness corresponds to how well a listener can understand a speaker regardless of any task-specific goal (e.g., humans understand the term “blue” in “avoid the blue box” and “find a blue cup”).
This information-theoretic framework for grounded semantic categories has been gaining broad empirical support across hundreds of languages and multiple semantic domains [13–16], and has been directly tied to language change over time [17].
However, the process of inducing such of IB-efficient communication systems among artificial agents is less clear.
While it has been shown that such systems may emerge in a color reference game via utility maximization [10], language variation in that study is implicitly controlled by environmental factors, leaving open the question of how efficient communication may evolve in a given environment and agent population, and in the context of tasks other than reference games.
In this work, we integrate these two views of communication by training teams of interactive artificial agents to optimize a tradeoff between maximizing utility, maximizing informativeness, and minimizing complexity, as depicted in Figure 1.
We hypothesize that taking into account complexity and informativeness, which are cognitively-motivated task-general communicative constraints, in addition to task-specific utility, may help guide artificial agents toward communication systems that can be more naturally aligned with and adapted to human languages.
To this end, building on prior work [19–21], we propose Vector-Quantized Variational Information Bottleneck (VQ-VIB), a method for training neural agents to encode inputs into signals, while maintaining a discrete (symbolic) signal representation that is embedded in a continuous space, inspired by word embeddings [22, 23].
We evaluate our approach in two multi-agent reinforcement learning environments and in two variants of a color reference game.
We show that: (1) by annealing the complexity weight, VQ-VIB agents can continuously adapt to changing communicative pressures and, in the color domain, align with human languages without changing the agents’ architecture or the structure of the environment; (2) the emergent VQ-VIB embedding spaces capture semantically meaningful structures, which are grounded in the agents’ embodiment; and (3) encouraging agents to be informative generally leads to faster convergence rates and improved utility, with VQ-VIB achieving higher utility for a given complexity compared to prior neural architectures that support symbolic communication.
Taken together, this work offers a new information-theoretic framework for EC that is grounded in general optimization principles that are believed to characterize human language evolution, and that has the potential of improving human-agent interaction.
Related Work
In this work, we train neural agents in cooperative settings, focusing on the effects of utility, informativeness, and complexity in three EC environments.
Below we review prior work that considers related measures and test environments in EC, as well as in human languages.
Communicative constraints in emergent communication While traditional EC literature focuses on task utility, some studies also consider communicative constraints that are closely related to our measures of complexity and informativeness (defined in Section 3).
For example, numerous works consider discretization, environmental noise, or penalty on large vocabulary sizes as proxies for reducing complexity [21, 24, 25].
Wang et al.
[26] use an information bottleneck in a centralized message processor to explicitly limit complexity.
Our notion of complexity is grounded in ratedistortion theory [27], capturing the degree of semantic compression seen across languages [13].
Other notions of linguistic complexity often focus on grammar, word forms, or compositionality, and offer promising directions for future work [28–31].
Additionally, some works consider auxiliary reconstruction losses designed to create “meaningful” communication and improve convergence rates [9, 32].
In particular, Lin et al.
[33] use an autoencoding loss which is closely related our notion of informativeness.
However, because they do not penalize complexity, they note that their method “may fail... where agents need to be more selective of the information communicated.”
Lastly, several studies consider reinforcement learning settings with informational constraints similar to ours [e.g., 34–38], however not in the context of EC as we do here.
Emergent communication environments and color naming
We explore particle-world environments that agents must navigate, as in [3, 21], as well as reference games (related to Lewis signalling games [39]), both of which are popular types of environments for evaluating EC.
While reference games may be played with varied inputs, we focus here on the color domain for several reasons.
First, it has been a central test-case for cognitive theories of semantic categorization and language evolution [1, 40, 41].
Second, it enables comparison with the World Color Survey [WCS; 42], a dataset of color naming systems from 110 languages of non-industrialized societies.
Third, it has been a focus of numerous studies in EC and machine learning more generally [1, 43–45, 10].
In particular, Chaabouni et al.
[10] use the IB framework from [13] to evaluate the emergent color signaling systems in artificial agents, and found similarities to human languages.
However, to capture the diversity observed across languages, they vary characteristics of the training environment by selecting dissimilar distractor colors, or by adding noise to the agents’ communication.
Rather than focusing on environment-based factors, we adopt a principled approach by directly optimizing theoretically- and cognitively-motivated terms of complexity and informativeness.
Information-Theoretic Emergent Communication
Here, we propose an information-theoretic EC framework that integrates task-general communicative objectives with task-specific utility maximization.
We begin with reviewing the theoretical framework on which we build (Section 3.1.1) and the set of variational inference tools that we use in order to extend that framework to EC settings (Sections 3.1.2 and 3.1.3).
We then define our integrative objective function for training EC agents (Section 3.2) and present Vector-Quantized Variational
Information Bottleneck (VQ-VIB, Section 3.3), a method for training agents with respect to our objective function while maintaining symbolic representations that can be used for communication.
The experimental setup and results for evaluating this new method are presented in Sections 4 and 5.
3.1 Preliminaries 3.1.1 The Information Bottleneck framework for grounded semantic systems
We build on and extend Zaslavsky et al.
[13]’s information-theoretic framework for grounded semantic systems.
In this framework, speakers and listeners jointly optimize the IB tradeoff [18] between the complexity and informativeness of their communication system.
A speaker is characterized as a probabilistic encoder S ( c | x ) that, given an input, x ∼ p ( x ) , mentally represents it as m x and based on that generates a communication signal, c .
m x is the speaker’s belief state, which is grounded in the agents’ representation of the world and captures any uncertainly the speaker may have about the input.
More formally, it is defined by a probability distribution over a set of possible inputs or relevant features that characterize the input space U , i.e, m x ( u ) =
p ( u | m x ) .
For example, x may be a target color and m x a Gaussian distribution over the CIELAB perceptual color space (Figure 3c), capturing the speaker’s perceptual noise.
Given c , the listener constructs ˆ m c to infer the speaker’s belief state.
Complexity is measured by the mutual information between the speaker’s inputs and signals, I S ( x, c ) , which, as in rate-distortion theory [27, 46], is roughly the number of bits that are needed for communication [18].
Informativeness corresponds to the similarity between the speaker’s and listener’s belief states.
Because these states are defined by probability distributions, maximizing informativeness amounts to minimizing the expected Kullback-Leibler (KL) divergence E S [ D [ m x ∥ ˆ m c ]] .
Taken together, in this view, for any given tradeoff parameter β ≥
the speaker and listener should jointly
[Equation]
which is equivalent to optimizing the IB principle [see 13, 47, for derivation].
For β ≤ 1 , the optimal systems are non-informative and minimally complex ( I S ( x, c ) = 0 ), yielding no communication.
Gradually annealing β yields optimal systems with greater complexity, and as β → ∞ , the optimal systems become maximally informative but also maximally complex.
Zaslavsky et al.
[13] and colleagues have shown that human languages achieve near-optimal IB tradeoffs across multiple semantic domains, and that the continuous evolution of the optimal IB systems via annealing the tradeoff parameter β [48] captures important aspects of language change over time [13, 17].
3.1.2 Variational Information Bottleneck
Variational Information Bottleneck [VIB, 19] is a variational approximation method for IB with continuous representations, using a neural network parameterization and the reparameterization trick [49].
In its typical form, a neural encoder q θ ( z | x ) maps an input x to parameters of a ddimensional Gaussian distribution, µ ( x ) and Σ( x ) , from which a latent representation, z ∈ R d , is sampled.
Given z , VIB reconstructs a target feature u using q ϕ ( u | z ) , parameterized by a soft-max function.
VIB optimizes a variational bound on the IB objective function, based on the fact that
[Equation]
for any distribution r ( z ) , which is often taken to be N (0 , I d ) .
VIB also bounds the IB informativeness term.
However, because we consider here a different instantiation of IB, based on unsupervised rather than supervised learning, we use a different informativeness bound, as explained in Section 3.3.
3.1.3 Vector-Quantized Variational Autoencoder
While VIB learns continuous representations, Vector-Quantized Variational Autoencoder [VQ-VAE, 20] learns discrete, but potentially highly complex, representations in an unsupervised manner.
More specifically, VQ-VAE defines a latent embedding space of K embedding vectors ζ i ∈ R d , i = 1 , . . .
, K .
An encoder maps an input x to a discrete variable k ∈ { 1 , . .
, K } by first passing x through a deterministic network to obtain a latent representation
z ( x ) ∈ R d , and then discretizing it to the index of the nearest embedding vector, i.e., k = argmin i
∥
z ( x ) − ζ i || 2 .
The decoder receives the corresponding embedding vector, ζ k ( x ) , and passes it through another deterministic network to reconstruct the input.
The set of trainable parameters Θ consists of the latent embedding space, { ζ 1 , . . .
, ζ K } , and the parameters of the encoder and decoder networks.
Because gradients cannot be passed through the argmin operator, a straight-through estimator [50] method is used, where the gradients from the decoder’s input, ζ k ( x ) , are copied to the encoder’s output z ( x ) .
Similar to
VAE [49], VQ-VAE’s training loss, shown in Equation (3), uses the evidence lower bound (ELBO) on the log-likelihood of the model, log p ( x ; Θ) .
However, because the encoder is deterministic and the prior over k is taken to be uniform, the ELBO is reduced to the reconstruction loss (first term in Equation (3)).
VQ-VAE adds to it two VQ terms, which are designed for learning the embedding space, yielding the overall training loss
[Equation]
where sg stands for the stopgradient operator, and following [20], α = 0 .
3.2 Integrating utility, informativeness, and complexity
In traditional EC settings, agents are trained to maximize the expected reward, or utility [51].
In a simplified setting, we assume that a speaker, S maps from inputs, x , to communication c , while a listener, L , maps from c to an output y .
Utility-based approaches to EC aim to find S, L that maximize the expected U ( x, y ) .
Similar utility-based framings apply to more complex settings with discounted rewards, more agents, or agents that can communicate and act.
One may integrate utility with the IB framework into an idealized objective function we seek to maximize, shown in Equation 4.
This objective function is the weighted combination of three terms, wherein we seek to maximize a tradeoff between high utility, high informativeness (low
KL divergence between m x and ˆ m x ), and low complexity.
Intuitively, these three scalar weights form a simplex (depicted in Figure 1) that describe a range of communication from uninformative communication (minimizing complexity) to task-optimized communication that does not generalize (maximizing utility) to complete descriptions (maximizing informativeness).
[Equation]
Directly training to maximize this objective is intractable in larger domains, but variational methods, including one we propose in the next section, support useful approximations.
3.3 Vector-Quantized Variational Information Bottleneck
Our proposed method, Vector Quantized Variational Information Bottleneck (VQ-VIB), learns information-constrained discrete communication embedded in a continuous space, reminiscent of the notion of word embeddings [22, 23].
This method combines prior work on Variational Information
Bottleneck [VIB, 19] with discrete representation learning via Vector-Quantized Variational Autoencoders [VQ-VAE, 20].
It is also closely related to prototype-based EC [21], which aims to link EC with word embeddings, although using a different objective function and architecture than VQ-VIB.
3.3.1 VQ-VIB encoder (speaker) architecture
Here, we combine VIB and VQ-VAE for a novel method of generating discrete, information-bounded representations, as depicted in Figure 2.
Similar to VQ-VAE, VQ-VIB maintains a trainable codebook of K embedding vectors in R d , C = { ζ 1 , . .
, ζ K } , which in our case will be used for communication.
Given an input x , a neural encoder q θ ( z | x ) is used to sample a latent representation z ∈ R d , followed by a quantization layer that discretizes the sample by mapping to the nearest quantized vector ζ ∈ C .
This generates a communication vector c = argmin ζ i ( z − ζ i ) 2 .
Similar to VIB, we define q θ ( z | x ) by mapping x to parameters of a Gaussian distribution over the latent space.
Overall, the VQ-VIB encoder, or speaker agent, is given by
[Equation]
We train VQ-VIB agents using Equation 6.
We assume access to some utility-based loss function such as the REINFORCE training loss [52].
Informativeness is computed via the negative mean squared error between the speaker’s input and a decoder’s output, given communication.
(MSE is related to the KL divergence between two equal-variance Gaussians).
Communication complexity is bounded via the KL divergence between the speaker’s distribution for z and a unit Gaussian.
(Bounding I S ( x, z ) implicitly bounds I S ( x, c ) via the data processing inequality).
Lastly, we reuse vector quantization losses from Equation 3.
In all experiments, we set λ VQ = 1; α = 0 .
[Equation] [Equation]
Thus, VQ-VIB may be thought of as generating a complexity-bounded continuous representation followed by a vector quantization process.
We also considered a variant of a VQ-VIB speaker, called
VQ-VIB After , which quantized representations and then sampled around them, but it does not perform as well as our main VQ-VIB speaker.
We leave the discussion of VQ-VIB After to Appendix A.
3.4 Decoder (listener) architecture
Regardless of the speaker architecture, we trained agents in the overall network architecture depicted in Figure 1 b. A decoder, D (ˆ x | c ) , reconstructed the speaker’s input, given c , and a listener output actions or decisions y based on communication (and any additional observations specific to the listener:
L ( y | c, x L ) .
Using a separate decoder, as introduced by Lin et al.
[33], allows backpropagation through the decoder and speaker even when gradients cannot flow from the listener to the speaker.
3.4.1 Complexity equivalence classes
For a given communication complexity level, there is a set of IB-optimal systems with different effective lexicon sizes k
= |{ c ∈ C : (cid:80) x p ( x ) S ( c | x ) > 0 }| [48].
We wish to bias agents within a complexity class towards the canonical system, with the minimal k , by regularizing the entropy of the speaker’s communication vectors.
Regularizing the entropy in VQ-VIB is problematic given the non-differentiablity of the argmin operation used in discretization.
Instead, we use the entropy of an approximated categorical distribution based on the euclidean distance to each quantized vector, p cat ( ζ i | z ) ∝ e − ( z − ζ i ) 2 .
Because this entropy term is only used to break the symmetry between equally optimal systems, we use a small regularizing weight (0.05 in our experiments, which is orders of magnitude smaller than the weights we use for the other terms in our objective function).
Experimental Setup
In designing our experiments, we relied upon the following intuitions (confirmed in findings) about the roles of utility, informativeness, and complexity in emergent communication.
Increasing informativeness should cause agents to converge to successful communication systems more quickly than without such training.
However, increasing informativeness will also increase complexity, so limiting complexity will be necessary to induce human-like discrete communication systems.
Trials and baselines We compared VQ-VIB with four types of neural architectures: continuous (e.g., Sukhbaatar et al.
[53]), onehot (e.g., Lowe et al.
[3]), prototype-based [Proto., 21], and VQ-VIB
After (defined in Appendix A).
All speaker architectures were trained with the same decoder and listener architectures, and in all domains agents were trained from scratch using 5 random seeds.
We strengthened the baseline architectures relative to prior art to support variational bounds on complexity.
For onehot agents, we bounded complexity via the KL divergence between the speaker’s categorical distribution and from a uniform distribution.
Typical Proto.
agents deterministically map from inputs to a onehot vector that is then multiplied by a learnable “embedding” matrix; in our variational method, we then sampled around such embeddings with variance also output by the speaker (much like a VAE sampling according to a µ and σ ).
These changes strictly improved baselines, as with no complexity regularization, agents could learn deterministic policies.
See Appendix B for full details of the architectures, hyperparameters, and environments.
Particle worlds We train agents in two particle world environments adapted from Tucker et al.
[21], 9 points and Uniform, shown in Figure 3.
Both environments consist of two-dimensional worlds in which a speaker observes the exact target location that a listener needs to reach but cannot directly observe.
In the 9 points environment, there were 9 landmarks at fixed locations, one of which was chosen at random as the target for each episode.
In the Uniform environment, two landmarks were spawned in locations drawn uniformly at random in the world; one landmark was specified as the target.
Agents received identical reward equal to the negative distance from the listener to the target and were trained using Multi-Agent Deep Deterministic Policy Gradient [3], which is a standard policy-gradient method for training agents without backpropagating through one agent into another.
The listener could not observe the target location, and the speaker could only communicate at the first timestep by broadcasting a vector that was observable at all future timesteps; thus, an optimal team policy would consist of the speaker communicating about the target at the first timestep, and the listener navigating to the target based on the message.
The two environments differ in their complexity: in the 9 points environment, the speaker had a finite amount of information about which to communicate, whereas in the Uniform environment, the location could be any continuous value.
Color reference games In our two color reference game experiments, we trained agents by differentiating through both the speaker and listener, as depicted in Appendix B, Figure 8.
In the
ABCDEFGHIJ
[Table] [Table]
[Figure]
“Basic” reference game, a speaker observed a target color and output a communication vector.
A listener observed the speaker’s communication and the target and distractor colors and predicted which of the colors was the target.
Colors were represented by their 3D CIELAB coordinates (illustrated in Figure 8 in Appendix B), and speaker observations were corrupted by zero-mean
Gaussian noise with a standard deviation of 8, as motivated by human perception [13].
Both colors were sampled from the set of 330 World Color Survey (WCS) colors according to a human-based prior from [13].
1 Only the speaker agent observed which color was the target, so the agents had to learn to convey information from the speaker to the listener.
Agents were trained via binary crossentropy loss computed on the listener’s output.
Additional experiments, wherein agents were trained with REINFORCE, achieved slightly worse results (corroborating Chaabouni et al.
[10]’s findings)
and are included in Appendix D [52].
In addition to the Basic setting, we also used a “Common Ground” setup which differed only by letting the speaker also observe the distractor color.
Thus, while the Basic scenario resembles the
WCS setting, in which participants were asked to name individual colors, the Common Ground scenario was inspired by work in pragmatics and Rational Speech Act, wherein humans communicate less information than fully describes a scene, but enough to induce listeners to behave in the desired manner [44, 54].
We note that, unlike Chaabouni et al.
[10], we do not change the subset of sampled distractor colors to induce different complexity; our framework results in more stable training and allows us to directly vary communication complexity within the same environment.
Experiment Results
Our results indicated that an informativeness loss improved convergence, annealing λ C generated a suite of agents of different complexities that more closely approximated human languages, and
VQ-VIB agents achieved greater utility, for the same complexity, than onehot agents.
5.1 Particle Worlds
In the particle worlds, increasingly penalizing complexity led agents to lower-utility communication, although architecture choices and training to increase informativeness could increase reward.
First, we conducted experiments in which we did not penalize complexity, but varied λ I across trials.
Higher λ I increased the mean team reward and improved convergence rates, as shown for VQ-VIB in Figure 4 a. Similar trends appeared for all speaker architectures in both environments (Appendix C).
We note that successfully training agents with a mix of informativeness and utility loss represents an improvement over Lin et al.
[33], who struggled to train agents with a combination of losses.
[Figure]
Next, for a fixed λ I = 1 . 0 , we trained agents while annealing λ C ; this smoothly decreased the complexity of communication.
Results for the Uniform environment aligned with our expectations (Figure 4 b).
Each curve indicates the cubic best fit to the data generated across 5 random seeds for each configuration (discarding outliers for reward less than -120), with complexity and reward measured at increments of 1,000 training episodes.
As complexity decreased (moving right to left in the plot), reward decreased as well, although VQ-VIB agents tended to achieve greater utility, for the same complexity, as onehot or Proto agents.
(Continuous communication reflected an optimal upper bound but violated our desire for discrete communication.)
Adding more encodings (9 discrete encodings vs. 100) significantly increased VQ-VIB performance, while changing onehot agents to communicate in 9 or 100 dimensions had a smaller effect.
Parametrizing agents with even more tokens (200 or 500) did not improve utility beyond 100 tokens, suggesting that greater parametrization will not decrease the gap.
Similar trends held in the 9 points environment (Appendix C). Lastly, visualizations of VQ-VIB agent behavior (Figure 5) confirmed that low-complexity VQVIB agents communicated via increasingly coarse discretizations of the space.
(Animations of the evolution of such changes are available in our codebase.) In both the Uniform and 9 points environments, low-complexity systems used fewer tokens to refer to broader targets, much as how humans might communicate via works like “north” or “south” rather than use hundreds of specific words for any possible location. Interestingly, 2D PCA of VQ-VIB embeddings (Figure 5 b and d) reveal a similar structure in the communication space as in the environment, indicating a strong relationship between form and meaning of VQ-VIB communication, like word embeddings for natural language. While Tucker et al.
[21] created similar “semantic spaces” for communication by adding environmental noise, VQ-VIB allows us to generate such rich representations at a variety of complexity levels, without changing the environment.
5.2 Color reference games
In the Basic color reference game, an informativeness training loss appeared necessary for onehot, while a complexity-based loss was useful for generating a spectrum of communication conventions.
First, we trained agents with a range of λ I values while leaving λ C = 0 .
We briefly report two main findings: 1) for a given λ I , agents converged to a small range of complexity and informativeness and 2) VQ-VIB agents typically learned more complex and informative communication than onehot agents did, although increasing λ
I shrank the gap.
These experiments are similar to those conducted by Chaabouni et al.
[10], who did not train with informativeness or complexity losses.
Full results for these experiments are included in Appendix D.
Second, we tested agents for λ
I = 1 .
while annealing λ C to penalize complexity.
As λ C increased, the communication complexity decreased, as shown in Figure 6 (left).
Over 5 random seeds, the points in the plot, generated at each training epoch, closely approximated the IB upper bound for maximum informativeness for a given complexity.
Inspection of the learned color naming schemes at different complexity levels revealed important trends (Figure 6 right).
As complexity decreased, the number of communication clusters emitted by VQ-VIB agents decreased (visualized via 2-dimensional principle component analysis).
Clustering in the communication space was reflected in the color space (e.g., “green” and “blue” merging).
More generally, for any two randomly selected pairs of colors, the pair of colors that was closer together in CIELAB space resulted in vectors that were closer together in the communication space 70% of the time.
As in the particle-world domains, these results indicate a fundamental connection between the form and meaning of communication used by VQ-VIB agents, which onehot-based communication (with all tokens equidistant) is architecturally unable to support.
Lastly, we compared agent color naming schemes to each other and to human language data.
For each of the 110 languages in the WCS Dataset [42], we found the best-matching agent as determined
[Table]
by minimizing gNID, a measure of dissimilarity of naming distributions introduced by Zaslavsky et al.
Example nearest languages are shown in Figure 6.
Despite having no supervised data, the artificial agents learned similar color naming schemes to humans, but only when agents changed complexity to match human languages’ (shown via the gNID metrics in Table 1).
This highlights the importance of controlling complexity in emergent communication: only by changing complexity could we generate naming systems similar to human communication.
Thus, methods like Lin et al.
[33] that encourage informativeness without penalizing complexity would be unable to recreate our results, and Chaabouni et al.
[10] would have to change the environment to indirectly control complexity.
Finally, when comparing VQ-VIB and onehot naming schemes, we found that agents were similarly efficient (deviation from optimality is calculated as ϵ in Table 1), but that VQ-VIB achieved higher utility, for the same complexity, as shown in Figure 6 b (corroborating results from particle world experiments).
Varying hyperparameters such as the number of tokens or training scheme did not change results, as examined in Appendix D.
Together, these results indicate that annealing complexity is a critical component in recreating humanlike color naming systems.
Furthermore, VQ-VIB matched the form and meaning of communication by merging clusters in the communication space that represented nearby colors, and outperformed onehot communication by achieving higher utility for the same complexity.
Common ground color reference game We tested agents in the Common Ground color reference game to evaluate contextualized communication (results in Appendix D).
The same trends as in the
Basic experiments held: an informativeness loss improved convergence and, while both VQ-VIB and onehot were nearly optimally efficient, VQ-VIB achieved greater utility than onehot did.
Contributions
In this work, we explore the interplay of utility, informativeness, and complexity in emergent communication.
To the best of our knowledge, we are the first to directly optimize agents to communicate according to a weighted combination of all three terms.
We found that training agents while rewarding informativeness led to faster communication convergence, and penalizing complexity created more human-like communication.
Further, we found that our novel neural architecture,
VQ-VIB, achieved greater utility for the same complexity than other discrete methods.
VQ-VIB combines the strengths of the methods it builds upon: it has a symbolic structure, defined by quantized vectors, but it also explicitly uses an IB encoder to sample in a continuous space.
Natural extensions of our work include incorporation of new, cognitively-inspired terms to guide communication, or more complex domains, which could better illustrate advantages of VQ-VIB.
Our work represents a step towards reconciling principles from cognitive science with emergent communication literature, which we hope better supports human preferences.
However, we recognize that, by quantifying properties of languages, our results may be misinterpreted to assign relative values to one language over another (e.g., higher complexity being better).
We insist that our framework 1) has only been applied to narrow domains that do not capture any language’s entirety and 2) does not prioritize any term and that any tradeoff between these terms is equally valid.
Acknowledgments and Disclosure of Funding
M.T. was supported by an Amazon Science Hub Fellowship.
N.Z. was supported by a K. Lisa
Yang Integrative Computational Neuroscience (ICoN) Postdoctoral Fellowship.
We thank Rahma
Chaabouni
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section TODO.
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] See abstract, introduction, and findings
(b) Did you describe the limitations of your work?
[Yes] See Section 6, in which we note that we tested in limited domains and Section 3.4.1, in which we note that an approximation we made may be improved upon in future work.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Section 6 in which we discuss how readers may misconstrue quantitative measures of languages as assigning value to one over the other.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See footnote for link to code (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix B (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See error bars in plots and Table 1, in Section 5.
All scatter plots directly included the raw results from all 5 random trials.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] Training times and computational resources are listed in Appendix B. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We use the World Color Survey (WCS) data, as well as the publicly-available code by Zaslavsky et al.
Both sources are cited.
(b) Did you mention the license of the assets?
[Yes] We note that both are publicly-available and link to them in Footnote 1.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] Code is linked to in a footnote.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] We only used the existing WCS dataset.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] We only used the existing WCS dataset. 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Appendix: VQ-VIB After
We proposed an additional architecture, VQ-VIB After , that is closely related to our main VQVIB method.
In VQ-VIB After, an encoder to generates a discrete representation ζ , whereupon a communication vector c is sampled from a Gaussian in the continuous space centered around ζ .
This “sample-after” method is depicted in Figure 8, and corresponds to an intuitive notion of choosing a word (a discrete embedding) deterministically but then being uncertain about the form (by sampling in the continuous space around it).
This architecture fits into the standard Information Bottleneck framework, as complexity can be penalized via standard variational methods.
However, it is distinct from discrete communication methods because communication vectors are ultimately sampled from a continuous space.
Thus, this work does not fit naturally into the main focus of this work, but we believe it could have applications in other settings, such as speech processing (wherein discrete words are pronounced in a continuous auditory space).
B Appendix: Implementation Details
Here, we include implementation details that we omitted from the main paper that can be used to reproduce our experiments.
B.1 Multi-Agent Reinforcement Learning Experiments
We trained agents in the 9 points and Uniform environments with Multi-Agent Deep Deterministic
Policy Gradient (MADDPG), a standard method for multi-agent reinforcement learning [3].
Agent architectures were identical in both environments.
Regardless of the speaker architecture used, all teams used common decoder and listener agents (except for variations due to different communication dimensions, explained later).
The decoder used 2 fully-connected ReLU layers with hidden dimension 64; the listener used 3 fully-connected ReLU layers with hidden dimension 64.
Because of the common architecture for the decoder and listener agents across teams, variation arose due to differences among the speaker agents.
A speaker agent consisted of 2, 64-neuron fully-connected layers with ReLU activations, followed by a speaker “head.”
The head implemented the specific communication method (e.g., continuous, VQ-VIB, etc.) by taking the 64-dimensional representation from the speaker and going through one of 5 possible architectures to produce a vector in the communication space, R d .
(As discussed later, d varied by environment and architecture slightly.)
• Continuous: The continuous communication head consisted of two fully connected layers branching off the speaker encoding to generate a µ and logvar (the log of the variance) from which the communication vector was sampled using the reparametrization trick.
Complexity was measured as the KL divergence between the parameters µ and logvar and a unit Gaussian.
• Onehot: The onehot communication head consisted of one fully connected layer from the speaker encoding to R d .
Onehot vectors were generated using gumble softmax sampling, with temperature 1.
Complexity was measured as the KL divergence between the distribution from which the onehot vector was sampled and a uniform categorical distribution.
Thus, complexity corresponds to the difference between the conditional and unconditional entropies of messages, unlike non-IB frameworks that often penalize unconditional entropy.
We used greater communication dimensionality for onehot agents and therefore rescaled the first layer of the decoder and listener agents to accept such communication.
• VQ-VIB:
The VQ-VIB communication head first passed the speaker encoding through two fully connected layers, branching off the encoding, to generate a µ and logvar , as in the continuous communication case.
A communication vector was sampled from the distribution,
whereupon it was discretized to the nearest quantized vector, ζ , in the communication space (as done in a standard VQ-VAE, without the sampling).
Complexity was measured as the KL divergence between the parameters µ and logvar and a unit Gaussian.
• VQ-VIB
After: The VQ-VIB After communication head first passed the speaker encoding through two fully connected layers to generate an encoding, z , and logvar .
The encoding, z , was discretized to the nearest quantized vector, ζ .
Lastly, a communication vector, c , was sampled from a normal distribution centered at ζ with variance specified via logvar .
Complexity was measured as the KL divergence between the parameters ζ and logvar and a unit Gaussian.
• Proto:
The Proto head mapped the speaker encoding through a single fully-connected layer to a categorical distribution over k neurons, for k representing the number of prototypes.
A kdimensional onehot vector was sampled using gumble softmax; a prototype, p was selected by multiplying the prototype matrix by the onehot vector.
A different fully-connected layer mapped from the speaker encoding to logvar .
Lastly, a communication vector, c , was sampled from a Gaussian centered at p with variance via logvar .
Complexity was measured as the KL divergence between the parameters p and logvar and a unit Gaussian.
This closely resembles the VQ-VIB After architecture, but with a different discretization mechanism.
All agents were trained with an Adam optimizer with learning rate 0.01 and batch size 1024.
B.1.1 9 points Environment
For all agent architectures except for onehot, the communication dimension was 2; onehot used communication dimension 9 so that the number of discrete representations allowed by the architecture was not the limiting factor for complexity.
Similarly, the VQ-VIB and Proto agents used 9 discrete embeddings.
In our results with annealed complexity weight (presented in Figure 11), agents were trained for 10,000 episodes, with each episode lasting 100 timesteps.
The listener agent was always spawned at the origin, equidistant from each landmark.
The informativeness loss weight, λ I , was constant at 1.0.
For the continuous and VQ-VIB communication methods, the complexity loss weight, λ C increased by 8 ∗ 10 − 6 per episode.
The onehot and Proto methods suffered from communication collapse at that rate, so results λ C increased half as quickly: at 4 ∗ 10 − 6 per episode.
Training a team in the 9 points environment for 10,000 episodes took approximately 15 minutes on a desktop computer with 16 Intel i9 cores for all architectures, except for onehot, which took roughly 20 minutes.
This difference in time is attributable to the larger communication dimension needed for onehot.
B.1.2 Uniform Environment
In the Uniform environment, we used identical parameters as in the 9 points environment, except for the changes noted below.
For all agent architectures except for onehot, the communication dimension was 3; for onehot, we experimented with communication dimensions of 9 and 100.
Similarly, we tested VQ-VIB and
Proto agents when using 9 or 100 discrete embeddings.
The listener agent was spawned at a starting location sampled uniformly at random from the map (bounded between -200 and 200 for both x and y ).
Otherwise, we used the same number of training episodes, episode duration, and complexity annealing rates as in 9 points.
Training a team in the Uniform environment for 10,000 episodes took approximately 15 minutes, except for onehot, which took roughly 30 minutes for 100-dimensional communication.
B.2 Color Reference Game Experiments
Here, we include the implementation details used in our color reference game experiments.
Agents were trained in the overall architecture depicted in Figure 8.
The speaker agent was parametrized with 3 fully connected layers with ReLU activations and hidden dimension 64, followed by a communication head.
The communication heads used the same architectures as specified in the particle world experiments: gumbel softmax was used for generating onehot vectors, and a straight-through estimator was used for discretization in VQ-VIB architectures [55, 56].
In all experiments, the communication dimension was set to 32 for all agents except onehot, for which the communication dimension was set to 330 (to match the expressive power of other discrete agents, that each had 330 discrete tokens in the 32 dimensional space).
decoder and listener the same feed-forward architecture in the particle world experiments.
Without annealing the complexity loss, 50 training epochs sufficed for agents to converge to a stable solution.
Thus, in our experiments with λ C = 0 , we only trained for 50 epochs.
For later experiments, wherein we annealed complexity, we first trained agents for 50 epochs before beginning to anneal the complexity term for the next 40 epochs (for a total of 90 epochs).
We note that, for VQ-VIB, we set the complexity loss to 0 during this “burnin” period, but for onehot communication, it appeared necessary to set the loss to 0.05 to prevent numerical instability during the first epoch after the burnin period.
During complexity loss annealing, we increased λ C by 0.01 during each epoch for VQ-VIB, and by 0.1 for onehot.
These different rates were necessary to induce similar ranges in complexity of learned communication.
In the Basic color reference game, the speaker observed a only the 3-dimensional, corrupted target color.
In the Common Ground setting, the speaker observed a 6-dimensional vector, generated by concatenating noisy versions of the target and distractor colors.
In both settings, the listener observed the communication output by the speaker, as well as the target and distractor colors (ordered randomly, without added noise).
Results were analyzed using code partially-based on publicly-available code released by Zaslavsky et al.
We trained agents on a desktop computer with 16 Intel i9 CPUs and 1 NVIDIA GeForce RTX 2080.
Training for 90 epochs in both the Basic and Common Ground settings took approximately 3 minutes for all agent architectures.
Agents were trained with an Adam optimizer with default parameters, with batch size 1024.
C Appendix: Full Particle World Results
For brevity, we only included partial results from the particle-world experiments in the main paper.
Here, we include the full results.
First, we experimented with varying λ
I for different environments and neural architectures, without annealing complexity.
Training curves for both environments, using varying λ
I for each neural architecture, are included in Figures 9 and 10.
For each configuration (a particular speaker architecture and λ I ), we trained 5 teams from scratch for 5,000 episodes and plotted the median and quartile rewards, smoothed over a 50 episode window.
A consistent trend emerged: higher λ
I led to faster convergence to a higher reward.
Indeed, this corroborates findings by Lin et al.
[33], who used an autoencoder architecture for communication in multiagent settings, but here we showed good convergence properties when mixing utility and informativeness terms (whereas they reported worse results when adding utility losses).
Overall,
U tilit y
[Figure]
these findings motivated training agents to convergence with high λ I before beginning to penalize complexity with λ C .
[Figure] [Figure]
Next, we include the results of annealing complexity in both the 9 points and Uniform environments, as depicted in Figure 11.
In both environments, as we increased λ C , complexity and utility decreased.
Comparing both VQ-VIB architectures to other discrete communication methods showed that VQVIB agents tended to obtain a higher mean reward, for a given complexity, than other methods.
In the Uniform environment, we also compared different variants of VQ-VIB, Proto, and onehot agents based on the maximum number of discrete messages allowed by the neural architectures.
We tested agents parametrized with either 100 or 9 unique messages; we hypothesized that initializing agents with many discrete messages and then gradually decreasing complexity would result in better performance than training with few messages from the start.
Indeed, we observed that was the case.
As seen in Figure 11 b, VQ-VIB agents parametrized with 100 embeddings consistently achieved higher reward for the same complexity than VQ-VIB agents parametrized with only 9 embeddings.
This trend held even when the complexity was lower than log 2 (9) bits.
Ultimately, results from both particle world environments corroborate the findings from the color reference games: encouraging informativeness improved convergence, VQ-VIB achieved greater utility than onehot, and decreasing complexity led to coarser discretizations (main paper, Figure 5).
The varied domains and training mechanisms (MARL vs. backpropagation for reference games) leading to consistent results suggest broad generalizability of our framework and VQ-VIB.
D Appendix: Color Reference Game Full Results
We omitted some of the color reference game results from the main paper due to space constraints.
Here, we include our complete results.
D.1 Basic Color Reference Game
First, we include results from the Basic setting, in which we trained agent with different values of λ I without penalizing complexity.
These results are included in Figure 12.
Setting λ
I = 0 induced similar conditions to those used by Chaabouni et al.
[10]; any positive λ I favored more informative (and therefore more complex) communication.
Given the monotonically increasing nature of the IB bound, an optimal agent trained with any positive λ I would maximize utility and informativeness by maximizing complexity, up to a maximum of log 2 (330) bits.
Instead, we found that agents appeared to converge to lower complexity values, although increasing λ
I tended to somewhat increase complexity and informativeness.
For example, onehot communication increased in complexity from roughly 2.
bits for λ
I = 0 .
to 3.0 bits for λ
I = 1 . 0 .
This alone is a novel result compared to Chaabouni et al.
[10], who trained agents with no informativeness loss and observed lower-complexity communication.
Curiously, VQ-VIB agents automatically converged to informativeness around 3.5 bits regardless of the λ I value, although increasing λ
I did improve training stability.
This difference between onehot and VQ-VIB indicates that the differing architectures themselves have different inductive biases.
Without positive λ I , onehot learns low-informativeness communication, but increasing λ
I generated more complex agents, especially for onehot.
We conducted further experiments to compare agents with the neural agents presented by Chaabouni et al.
[10], wherein onehot agents were parametrized with a communication dimension of 1024.
We therefore tested VQ-VIB and onehot agents with 1024 tokens, instead of the 330 used for the results in our main paper.
This hyperparameter change largely did not alter results, which is unsurprising given that agents with 330 tokens did not use their full vocabulary.
As shown in Figure 13, both
VQ-VIB and onehot achieved nearly optimal informativeness for a given complexity, and VQ-VIB achieved higher utility than onehot (results zoomed in to complexity values between 1 and 2 bits for clearest plotting of these trends).
Lastly, we performed additional experiments to investigate the effect of training with REINFORCE, instead of the supervised learning framework used in our main experiments (e.g., backpropagating through gumbel softmax) [52].
Chaabouni et al.
[10] had found that training with REINFORCE typically resulted in lower-complexity communication than when training with messages generated via a gumbel softmax layer, and that training with REINFORCE was less stable.
Results from our
REINFORCE experiments, using 330 tokens for onehot and VQ-VIB, are included in Figure 14.
We trained agents with λ U = 1 . 0 , λ I = 50 . 0 , and λ C was annealed from 0 to 0.1, using code adapted from Rita et al.
Our results from these experiments align with all prior findings: VQ-VIB and onehot achieved high communicative efficiency, and VQ-VIB produced greater utility than onehot did.
Beyond those corroborating results, our experiments aligned with Chaabouni et al.
[10]’s findings and
[Figure] [Figure]
further illustrated the importance of informativeness losses and the VQ-VIB architecture.
With zero informativeness loss (i.e., λ I = 0 ) onehot agents often failed to converge to any useful communication (as indicated in prior art), but by increasing λ
I , agents consistently converged to useful communication.
Indeed, with λ I = 50 and λ C = 0 , onehot agents learned to communication with complexity of up to 2.4 bits, greater than any complexity value found by Chaabouni et al.
[10] using REINFORCE over 180 seeds (see Fig.
5 in that paper).
Furthermore, we observed that VQ-VIB agents typically converged to higher informativeness (and higher complexity) communication than onehot when complexity was not penalized, indicating that further investigation of inductive biases is a promising direction for future work.
Overall, this experiment largely replicates our findings from the main paper.
Thus, findings from our REINFORCE experiments indicate that the main conclusions of our paper hold true across several training mechanisms, and furthermore that our framework (including in particular the informativeness loss) can address issues already noted in prior work.
D.2 Common Ground Color Reference Game
Here, we include the full results from our Common Ground color reference game experiments.
Crucially, these experiments tested agent behavior when utility and informativeness were not identical: utility could be maximized by the speaker only communicating about the target color, whereas informativeness was maximized by communicating about both target and distractor.
Given the lack of human-labeled data for such a setting, we were unable to compare agent communication with human languages.
However, by analyzing communication when varying the target for a fixed distractor, or by varying the distractor for a fixed target, we established that agents conditioned communication both on the target and distractor colors.
[Figure]
We trained agents in the Common Ground setting by randomly sampling target and distractor colors, as in the Basic setting, but here the speaker observed noisy versions of the distractor as well as the target.
Based on insights that training to increase informativeness produced more complex communication, we trained agents for 30 episodes with λ I = 1 . 0 .
For the next 70 episodes, we annealed the complexity weight, λ C , using the same rates as in the Basic setting; this produced communication at different levels of complexity.
As in the Basic setting, we found that VQ-VIB achieved higher utility, for the same complexity, than onehot (Figure 16).
We further analyzed communication in two ways: using a fixed distractor with varying targets and using a fixed target with varying distractors.
First, for a fixed, gray distractor (#46 in the WCS data), we measured the standard metrics of informativeness and complexity.
That is, we set the distractor color to gray, and we recorded, for each of the other 329 colors as the target, the speaker’s communication.
We then performed the same complexity and informativeness analysis as in the Basic setting, and found that both VQ-VIB and onehot communication remained near-optimally efficient, as shown in Figure 15 a.
Second, we measured whether agents conditioned their communication on the distractor color.
Unlike the prior evaluation, in which we varied the target color for a fixed distractor, here, we evaluated communication for different distractors.
Plots of a VQ-VIB agent’s naming scheme, using either an orange (color #212) or purple (color #325) distractor are shown in Figure 15 b.
As expected, the agent displayed different resolutions in regions of color depending upon the distractor.
Specifically, for an orange distractor, the speaker had finer resolution around shades of orange and yellow; conversely, the speaker had finer resolution around shades of purples for a purple distractor.
This aligns with expectations that, when trying to differentiate between colors, one must use more complex or accurate words to distinguish similar colors.
Jointly, our findings indicate that many of the themes identified in the Basic setting – nearly-optimal communicative efficiency, the importance of annealing λ C to generate varied complexity, and the advantages of VQ-VIB over onehot – held in the Common Ground setting as well.