E MERGENCE OF L INGUISTIC C OMMUNICATION FROM R EFERENTIAL G AMES WITH S YMBOLIC AND P IXEL
I NPUT
Anonymous authors
Paper under double-blind review
A BSTRACT
The ability of algorithms to evolve or learn (compositional) communication protocols has been traditionally measured in the language evolution literature by emergent communication tasks.
In this work, we scale up this research by using contemporary deep learning materials and train reinforcement learning neuralnetwork agents on referential communication games.
We extend previous work in this direction in which agents learn from symbolic environments to those learning from using raw pixel input data, a more challenging and realistic input representation.
We ﬁnd that the degree of structure found in the input data affects the nature of the emerged protocols and thereby corroborate the hypothesis that structured compositional language is most likely to emerge when agents perceive the world as being structured.
I NTRODUCTION
Research on emergent communication is inﬂuenced by the view that language derives meaning from its use (Wittgenstein, 1953).
This perspective especially motivates the study of language emergence through problems where co-operative agents try to achieve shared goals in games (Steels, 2003;
Brighton & Kirby, 2006; Mordatch & Abbeel, 2017).
This is also related to the study of multi-agent and self-play methods that have found great success in other areas of machine learning (Bansal et al., 2017; Silver et al., 2017).
The study of emergent communication is important for two related problems in language development, both human and artiﬁcial: language evolution, the development of communication protocols from scratch (Nowak & Krakauer, 1999); and language acquisition, the ability of an embodied agent to learn an existing language.
In this paper we focus on the problem of how environmental or pre-linguistic conditions affect the nature of the communication protocol that an agent learns.
Considering the realism and complexity of environments used for grounded language learning today (Brockman et al., 2016; Hermann et al., 2017), it is important to analyse these effects in detail.
One of the most important properties of natural language is that it is compositional.
Smaller building blocks (words, morphemes, . . . ) are used to generate inﬁnite numbers of more complex forms (sentences, multi-word expressions, . . . )
, with the semantic meaning of the larger form being a function of the meaning of its parts (Frege, 1892).
Compositionality is an advantage in both language and other communication protocols as it allows inﬁnite expression through a ﬁnite dictionary.
In emergent communication research, work has shown that agents can produce (somewhat) compositional protocols when engaging in language games (Steels, 2003).
However, in those cases the computational agents were placed in artiﬁcial worlds containing just a handful of objects represented as disentangled, structured, and sometimes even atomic symbols, e.g., attribute-based vectors or one-hot vectors (Batali, 1998; Brighton, 2002; Franke, 2015; Andreas & Klein, 2017; Mordatch & Abbeel, 2017).
However, humans do not receive symbolic but rather raw sensorimotor input.
Hence, while these results are impressive, they can potentially produce misleading conclusions about human language (both from an applied and a scientiﬁc perspective).
To date, little work has tested to what hank you!
hank you!
[Figure]
A
[Figure]
hank you!
hank you!
player A player B
CAT speaker listener 22 2the third one!
h a s _ w
h
i s k e r
s h a s _ s e a t h a s _ w h e e l s
CAT
SOFA
CAR h a s _ l e g s m a d e _ o f _ m e t a l pixel data symbolic data
[Figure]
[Figure]
degree (if at all) these ﬁndings carry over when agents exist in less perfect and more realistic worlds that bear more similarity to the kind of entangled and noisy signals that humans are exposed to.
In this work and in the context of referential communication games (see Figure 1), we contrast the results of two studies that lie at extremes in terms of how much structure is provided by their environments.
The ﬁrst study (Section 3) focuses on symbolic representations, where objects are represented as bags-of-attributes.
This representation is inherently disentangled since dimensions encode individual properties.
The second study (Section 4) considers raw perceptual input, hence data that more closely resembles what humans are exposed to.
Clearly, the latter is a more challenging scenario as computational agents should operate on entangled inputs with no pre-coded semantics.
Crucially, both studies use the same referential game setup, the same learning procedure (policy learning methods) and the same neural network agent architectures.
We show that reinforcement learning agents can communicate not only when presented with symbolic and highly structured input data, but—more importantly—even when presented with raw pixel input.
This opens up avenues to more realistic simulations of language emergence.
From an applied perspective, we successfully use the learning signal from the referential game to train agents end-to-end, including in cases where they need to perform visual processing of images with a convolutional neural network.
On a more theoretical note, we ﬁnd that the agents struggle to produce structured messages when presented with entangled input data (Bengio et al., 2013) due to the difﬁculty of uncovering the true factors of variation, corroborating the hypothesis of Smith et al.
(2003) that structured (compositional) language is most likely to emerge when agents perceive the world as structured.
R EFERENTIAL G AMES AS M ULTI - AGENT CO - OPERATIVE R EINFORCEMENT L EARNING
The referential game is implemented as an instance of multi-agent co-operative reinforcement learning; two agents take discrete actions in their environment in order to maximize a shared reward.
2. G AME AND T ERMINOLOGY
The agents engage in referential games, a variant of the Lewis signaling game (Lewis, 1969), which have been extensively used in linguistic and cognitive studies in the context of language evolution (e.g., Briscoe, 2002; Cangelosi & Parisi, 2002; Steels & Loetzsch, 2012; Spike et al., 2016; Lazaridou et al., 2017).
See Figure 1 for a schematic description of our setup.
First, a speaker is presented with a target object (highlighted CAR or right image) and by making use of an alphabet consisting of primitive discrete symbols (“22”, “10”, “0”,“2”), constructs a message describing that object (“ 2 0”).
We will refer to the set of all distinct messages generated by the speaker as the their lexicon or protocol .
Finally, the listener is presented with the target and a set of distractor objects, and—by making use of the speaker’s message—has to identify the target object from the set of candidate objects.
Communicative success is deﬁned as the correct identiﬁcation of the target by the listening agent.
Formally, the attribute-based object vectors (disentangled) or the pixel-based images (entangled) are the set of pre-linguistic items W = { o 1 , . .
, o N } .
From this set we draw a target t ∈ W and subsequently K − 1 distractors D = { d 1 , . . .
, d K − 1 } ⊂ W s.t.
∀
j t (cid:54) =
d j .
The speaker has only access to the target t , while the listener receives candidate set C = t ∪ D , not knowing which of the set elements is target t .
2. A GENTS
The speaker encodes t to a dense representation u with f S ( t, θ Sf ) .
The function of this encoder depends on the type of pre-linguistic data used and is discussed separately for each study.
Given an alphabet A of discrete unit symbols (akin to words) and u , the speaker next generates a discrete, variable-length, bounded message m by sampling symbols from a recurrent policy π S derived by a decoder g S ( u, θ Sg ) .
The sequence generation is terminated either by the production of a stop symbol or when maximum length L has been reached.
We implement the decoder as a single-layer
LSTM (Hochreiter & Schmidhuber, 1997).
Note that the symbols in the agents’ alphabet A have no a priori meaning.
Instead, these symbols get grounded during the game.
The listening agent uses a similar encoder as the speaker but has independent network weights ( θ Lenc ).
Applying this encoder to all candidates results in a set U = f L ( C, θ Lf ) .
We use a single-layer LSTM to encode the message m , s.t.
z = h L ( m , θ Lh ) .
Given encoded message z and candidates U the listener predicts a target object t (cid:48) ∈ C following a policy π L implemented using a non-parametric pointing module which samples the predicted object from a Gibbs distribution computed via the dot product between vectors z and all encoded candidates u ∈ U .
See Appendix B for information regarding the agents’ architecture.
We note that at inference time, we replace the stochastic sampling process of the speaker’s message generation and the listener’s pointing with a deterministic one based on selecting the message and object with the highest probability.
2. L EARNING
All weights of speaker and listener agent θ = (cid:104) θ
Sf , θ Sg , θ Lf , θ Lh (cid:105) are jointly optimized while playing the game.
We emphasize that no weights are shared between the speaker and the listener, and that the only supervision used is communication success, i.e., whether the listener identiﬁed the correct target.
The objective function that the two agents maximize for one training instance is
R (cid:16)(cid:80) L log p ( m lt | m <lt , u ) + log p ( u t (cid:48) | m , U ) (cid:17) where R is the reward function returning 1 if t = t (cid:48)
(if the listener pointed to the correct target) and 0 otherwise.
To maintain exploration in the speaker’s policy π S of generating a message and the listener’s policy π L of pointing to the target, we add to the loss an entropy regularization term (Mnih et al., 2016).
The parameters are estimated using the
REINFORCE update rule (Williams, 1992).
See Appendix B for information regarding learning.
S TUDY 1: R EFERENTIAL GAME WITH SYMBOLIC DATA
We ﬁrst present experiments where agents are learning to communicate when presented with structured and disentangled input.
We use the Visual Attributes for Concepts Dataset (ViSA) of Silberer et al.
(2013), which contains human-generated per-concept attribute annotations for 500 concrete concepts (e.g., cat , sofa , car ) spanning across different categories (e.g., mammals , furniture , vehicles ), annotated with 636 general attributes (e.g., has tail , is black , has wheels ).
We disregarded homonym concepts (e.g., bat ), thus reducing our working set of concepts to 463 and the number of attributes to 573 (after eliminating any attribute that did not occur with working concepts). On [Table]
average, each concept has 11 attributes.
All pre-linguistic objects are represented in terms of binary vectors o ∈ { 0 , 1 } 573 .
Note that these representations do carry some inherent structure; the dimensions in the object vectors are disentangled and so each object can be seen as a conjunction of properties.
Speaker and listener convert the pre-linguistic representations to dense representations u by using a feed-forward linear encoder followed by a sigmoid activation function.
In all our experiments, we set the number of candidate objects K to ﬁve, meaning there were four wrong choices per correct one (resulting in a 20% random baseline).
Inspired by Kottur et al.
(2017), who show that non-compositional language emerges in the case of overcomplete alphabets, we set the size of alphabet A to a number smaller than the set of objects (i.e., A has 100 symbols).
3. A GENT P ERFORMANCE AND A MBIGUITY
We ﬁrst report model performance on the training data, comparing different settings for the maximal allowed message length (2, 5 or 10 symbols).
Results are presented in Table 1 (ignore the last row topographic ρ which will be explained in later sections).
In the case of the shortest message settings (maximum length set to 2), our trained agents on average only develop a protocol of 31 unique messages used to describe 363 training concepts.
This indicates high ambiguity, with on average each message used to denote 11 concepts.
Interestingly, recent ﬁndings suggest that ambiguity is a design feature of language that prevents the inefﬁcient use of redundant codes, since some the message content can be extracted from context: “the most efﬁcient communication system will not convey information already provided by the context” (Piantadosi et al., 2012).
In our case, we do no explicitly encode any bias towards ambiguity.
We hypothesize that ambiguity arises due to the hard exploration problem that agents are faced with, in combination with the fact that ambiguous protocols present a good “local optimum” that is over-represented in the hypothesis search space.
As a result, in the absence of environmental pressures (e.g., a high number of carefully constructed distractors) a suboptimal policy can still achieve a reasonably high accuracy (92%), making it even harder during training to escape from such a solution.
In classic signaling games, this polysemy phenomenon manifests itself as different states receiving the same signal and is termed partial pooling equilibrium (Skyrms, 2010).
Perhaps rather counterintuitively, Skyrms (p.131) suggests that a way to obtain communication protocols that are robust to this type of local communication minima is to allow the invention of new signals, essentially increasing the search space of signals.
Motivated by this, we play variants of the game in which we allow the agents to produce messages of greater maximum length (5 and 10), which leads to improved communicative success (98.2% and 98.5% respectively).
We observe that the number of messages in the protocol increases from 31 to 293 and 355 respectively, reducing the average number of concepts a message can denote from 11 concepts to (approximately) 1 concept.
3. R EALISTIC CONTEXT DISTRIBUTION
In the real world, when speakers refer to cats, listeners would more often be in a situation where they had to discriminate a cat in the context of a couch or a dog rather than in the context of a mirror or a cow.
3 Simply put, things in the world do not appear in random contexts, but rather there is regularity in the distribution of situational and visual co-occurrence.
This property of the world is typically not captured in referential games studied in the language emergence literature, with distractors typically drawn from a uniform distribution.
We address this issue and design an additional experiment with distractors sampled from a target-speciﬁc context distribution reﬂecting normalized object co-occurrence statistics.
Co-occurrence data is extracted from the MSCOCO caption dataset (Lin et al., 2014).
This leads to more plausible distractor sets with, for instance, the target GOAT more likely being mixed with SHEEP and COW as distractors rather than with BIKE or
EGGPLANT .
Note that the distractor selection process (uniform vs context-dependent) affects the language learning dynamics (see Figure 2 for training curves for different experimental conﬁgurations).
While the non-uniform distractor sampling of the context-dependent process can be exploited to learn a degenerate strategy —giving up to 40% communicative success shortly after the start of training— subsequently learning under this scenario takes longer.
This effect is likely a combination of the local minimum achieved by the degenerate strategy of picking a target at random from the topically relevant set of distractors together with the fact that the co-occurrence statistics tend to align with the feature vectors, meaning that very similar objects are more likely to appear as distractors and the overall game thus becomes slightly more difﬁcult.
We now consider the question of how objects denoted by the same (ambiguous) message are related.
When the context is drawn uniformly, object similarity is a predictor of object confusability , as similar objects tend to be mapped onto the same message (0.26 and 0.43 median pairwise cosine similarities of objects that received the same message as computer on the ViSA space, for maximum message length 2 and 5, respectively).
In the non-uniform case, we observe object confusability to be less inﬂuenced by object similarity (0.15 and 0.17 median pairwise cosine similarities of objects that received the same message, for maximum message length 2 and 5, respectively), but rather driven by the visual context co-occurrences.
Simply put, in the non-uniform case confusability is less inﬂuenced by similarity since the agents must learn to distinguish between objects that naturally co-occur ( SHEEP and GOAT ).
Thus, the choice of distractors, an experimental design decision that in existing language emergence literature has been neglected, has an effect on the organization (and potentially the naturalness) of the emerged language (e.g., what is the semantics of ambiguous or homonym words in language).
3. S TRUCTURAL PROPERTIES OF EMERGED PROTOCOLS
Quantifying the degree of compositionality and structure found in the emerged language is a challenging task; to the best of our knowledge, there is no formal mathematical deﬁnition of compositionality that could allow for a deﬁnitive quantitative measure.
Thus, research on this topic usually relies on deﬁning necessary requirements that any language claiming to be compositional ought to adhere to, such as the ability to generalize to novel situations (Batali, 1998; Franke, 2015; Kottur et al., 2017).
We adopt a similar strategy and report results (see Section 3.3.1) where we test the generalization ability of an emerged language to novel objects.
Moreover, we also report results (see
[Table] Section 3.3.2) on a quantitative measure of message structure proposed in the language evolution literature (Brighton & Kirby, 2006; Carr et al., 2017).
3.3. G ENERALIZATION TO NOVEL OBJECTS
We perform experiments where trained agents from Section 3.1 are exposed to different types of unseen objects, each of them differing to the degree to which the unseen objects resemble the objects found in the training data.
In the test scenario, objects come from the same data distribution as the training data, but were not presented to the agents during training (e.g., a mouse); in the unigram chimeras scenario, the novel objects are constructed by sampling properties from a property-based distribution inferred from the training data, thus breaking any feature correlation (e.g., a mouselike animal with wheels); in the uniform chimeras scenario, the novel objects are constructed by uniformly sampling properties (e.g., a square red furry metallic object).
Table 2 reports the communicative success.
While there is a drop in performance for unseen objects, agents are performing above random chance (i.e., 20%).
The emerged language is indeed able to generalize to unseen objects; however, the degree of generalization is a function of similarity between the training and unseen objects, thus resulting in the uniform chimeras obtaining the lowest performance.
Moreover, we observe examples of productivity , a key feature of compositionality.
At test time, speakers are able to come up with novel messages on-the-ﬂy (i.e., messages that are not part of their lexicon inferred during training) to describe unseen objects.
See Table 2 last 3 rows and lexicon size column for the percentage of novel messages.
Even though listeners were not trained to associate the novel messages with the novel objects, they are still able comprehend them and correctly identify the target object.
In the test data and length 10 cases, the novel messages account for almost all generated messages with performance being at 81.6%, providing evidence of the structure found in the messages.
3.3. T OPOGRAPHIC SIMILARITY
Given a set of objects, their meanings and the associated signals, Brighton & Kirby (2006) deﬁne topographic similarity to be the correlation of the distances between all the possible pairs of meanings and the corresponding pairs of signals.
Figure 3 shows mappings between states and signals for examples of holistic (b) and compositional (c,d) languages, with the topographic similarity of compositional languages being higher than that of holistic.
The intuition behind this measure is that, given the deﬁnition of compositionality under which “the meaning of the whole is a function of the meaning of its parts”, semantically similar objects should have similar messages.
To compute this measure, we ﬁrst compute two lists of numbers: (i) the Levenshtein distances between all pairs of objects’ messages and (ii) the cosine similarity between all pairs of all objects’
ViSA vectors.
Given these two lists, the topographic similarity is deﬁned as their negative Spearman ρ correlation (note, that since we are correlating distances with similarities negative values of correlation indicate topographic similarity of the two spaces.
Intuitively, if similar objects share much of the message structure (e.g., common preﬁxes or sufﬁxes), and dissimilar objects have little common structure, then the topographic similarity should be high, the highest possible being 1.
Results presented in Table 1 column topographic ρ show that topographic similarity is positive in all experimental setups, indicating that similar objects receive similar messages ( p < 0 .
01 , permutation test). As an example, a qualitative analysis of the messages generated in the length 10 and training data cases showed that 32% of the mammal objects had as preﬁx the bigram 95#10 , 36% of vehicle objects had 68#95 and 11% of tool objects had 0#61 , suggesting that these preﬁx bigrams encode category-speciﬁc information.
Next, for each object pair, we calculate their Levenshtein message distance and respective cosine similarity, and plot in Figure 3 ( right ) for each distance the average cosine similarities of the pairs with that distance (this is done for the length 10 and training data experiment).
We observe that there is a clear relation between message similarity and meaning similarity.
In Figure 3, we also plot a similar correlation curve for an emerged language obtained by producing messages with a randomly initialized and ﬁxed speaker/listener architectures.
This emerged language is at random in terms of communicative success; however, it does show signs of structure since similar objects obtain similar messages.
This seems to suggest that structured and disentangled pre-linguistic representations are, perhaps, a sufﬁcient condition for the emergence of structured language, especially in neural network-based agents which, due to the nature of representation and information ﬂow, favor similar inputs to trigger similar outputs.
S TUDY 2: R EFERENTIAL GAME WITH RAW PIXEL DATA
In this section, we present experiments in which agents receive as input entangled data in the form of raw pixel input and have to learn to perform visual conceptual processing guided from the communication signal.
We use a synthetic dataset of scenes consisting of geometric objects generated using the MuJoCo physics engine (Todorov et al., 2012).
We generate RGB images of resolution 124 × 124 depicting single object scenes.
For each object we pick one of eight colors ( blue, red, white, black, yellow, green, cyan, magenta ) and ﬁve shapes ( box, sphere, cylinder, capsule, ellipsoid ) resulting incombinations, for each of which we generate 100 variations, varying the ﬂoor color and the object location in the image.
Moreover, we introduce different variants of the game: game A withdistractors; game B with 1 distractor; game C with 1 distractor and with speaker and listener having different viewpoints of the target object (thus the target object on the listener’s side is in a different location); game D with 1 distractor, with speaker and listener having different viewpoints, and with balanced number of shapes and color (obtained by downsampling from 8 colors to 5 and removing any image containing objects of the 3 disregarded objects).
For each game, we create train and test splits with proportions 75/25 (i.e., 3000/1000 for games A and B and 1850/650 for games C and D ).
[Figure] Pre-linguistic objects are presented in the form of pixel input, o ∈ [0 − 255] 3 × 124 × 124 .
Speaker and listener convert the images o to dense representations u , each of them using an 8-layer convolutional neural network (ConvNet).
Crucially, we do not pre-train the ConvNets on an object classiﬁcation task; the only learning signal is the communication-based reward.
Despite this fact, we observe that the lower layers of the ConvNets are encoding similar information to ConvNet pre-trained on ImageNet (Deng et al., 2009).
Conceptually, we can think of the whole speaker/listener architecture as an encoder-decoder with a discrete bottleneck (the message).
Given our initial positive ﬁndings, this reward-based learning signal induced from communication games could be used for class-agnostic large-scale ConvNet training.
Moreover, we ﬁnd that even though no weights were shared, the agents’ conceptual spaces get aligned at different levels, reminiscent of theories (Garrod & Pickering, 2004) of interactive conceptual alignment during dialogue (see Appendix A for the related experiment).
4. C OMMUNICATIVE SUCCESS AND EMERGENT PROTOCOLS
Unlike the experiments of Section 3 where agents start from disentangled representations, starting from raw perceptual input presents a greater challenge; the agents have to establish naming conventions about scenes, while at the same time learning to process the input with their own visual conceptual system.
Since we do not pre-train their ConvNets on an object recognition task, the dense representations u used to derive the message contain no bias towards any image- or scene-speciﬁc information (e.g, object color, shape and location).
The extraction of visual properties is thus driven entirely by the communication game.
This contrasts with the cases of Havrylov & Titov (2017) and Lazaridou et al.
(2017) who are using pre-trained visual vectors, and qualitatively observe that the induced communication protocols encode information about objects.
Table 3 presents the results in terms of communicative train and test success (see AppendixC for additional experiments when having access to gold object attribute classiﬁers).
Moreover, we also report the topographic similarity (see column topographic ρ ) between the symbolic attribute-based representations of scenes (ﬂoor color, object color, shape and location) and the generated messages.
Overall, despite the challenges posed in this setup due to the raw nature of data, performance across all games is well above chance, indicating that reinforcement learning agents trained end-to-end can manage to establish a communication protocol in this grounded environment.
In game A , the agents reach 93.7% accuracy with their lexicon consisting of 1068 messages to describe 3000 training objects.
Most importantly, as captured by the positive topographic similarity, agents produce messages that respect (even to a limited degree) the compositional nature of scenes (i.e., objects as bags-of-attributes), indicating that similar scenes receive similar messages. Indeed, by examining their protocol (see Table 4), we ﬁnd messages to encode in a structurally consistent way information about absolute location of objects, with the message preﬁx and sufﬁx denoting the horizontal and vertical co-ordinate respectively.
Interestingly, this communication strategy is usually followed by human players of referential games (Kazemzadeh et al., 2014).
[Table]
However, we ﬁnd the emerged protocols to be very unstable and too grounded in the speciﬁc game situation.
Small modiﬁcations of the game setup, while having close to no negative impact on the communicative performance, can alter radically the form, semantics and interpretability of the communication protocol.
In game B , performance remains at the same level (93.2%).
However, we observe that the protocol consists of 13 unique messages which do not reﬂect the objects’ attributes (as indicated by the close to zero topographic similarity), making thus the messages harder to interpret (see Figure 4 for randomly sampled examples).
When we change the viewpoint of the agents in game C , thus biasing them against communicating about absolute object location, the players derive a compact communication protocol consisting of 8 unique messages that describe primarily color.
Finally, when color and shape are balanced as in game D , we still observe a bias of agents towards the color of objects, with the ﬁve induced messages providing a perfect clustering of the objects according to their colors.
In an entangled world, agents do not possess a priori visual biases and knowledge of concepts.
Since objects can be conceptualized in indeﬁnitely many ways, the type of information encoded in the messages is tied to the environmental pressures; communication behaviour is a function of the environment, which also dictates what data structures can emerge.
The implication of this observation is that protocols essentially overﬁt to the particular game situation, to the degree that they become specialized ad-hoc naming conventions.
Interestingly, the emergence of ad-hoc naming conventions has also been observed during humanhuman interaction; when participants engage in some speciﬁc game situation (e.g., communicating about abstract tangram shapes), they tend to form highly specialized naming conceptions (i.e., conceptual pacts ) that allow them to communicate with maximum efﬁciency (Brennan & Clark, 1996).
While in this study we do not address this issue of how a stable and general language could emerge in entangled worlds, we believe that to alleviate the formation of such ad-hoc communication protocols it is essential to increase the complexity of the games as well as introducing the notion of game multi-tasking.
4. P ROBE MODELS
In order to investigate what information gets captured by the speaker’s ConvNet, we probe the inferred visual representations u used to derive the message.
Speciﬁcally, we design 4 probe classiﬁers for the color and shape of object, object position which is derived by discretizing each co-ordinate into 3 bins, and ﬂoor color which is obtained by clustering the RGB color representation of the ﬂoor.
For each probe, we performed 5-fold cross validation with a linear classiﬁer and report accuracy results in Table 4.
Overall, different games result in visual representations with different predictive power.
object position is almost always encoded in the speaker’s visual representation, even in situations where location of the object is not a good strategy for communication.
On the other hand, object shape seems to provide less salient information, despite the fact that it is relevant for communication, at least in the C&D games. [Table]
As expected, the structure and semantics of the emergent protocols are a function of the information captured in the visual representations.
The degree to which the agents are able to pull apart the objects’ factors of variation impacts their ability to communicate about those factors, with the most extreme case being game D , where the message ignores the shape entirely.
Thus, disentanglement seems to be a necessary condition for communication, at least in the case of pixel input.
C ONCLUSION
We presented a series of studies investigating the properties of protocols emerging when reinforcement learning agents are trained end-to-end on referential communication games.
We found that when agents are presented with disentangled input data in the form of attribute vectors, this inherent compositional structure is successfully retained in the output.
Moreover, we showed that communication can also be achieved in cases where agents are presented with raw pixel data, a type of input that aligns better with the raw sensorimotor data that humans are exposed to.
At the same time, we found that their ability to form compositional protocols in these cases is hampered by their ability to pull apart the objects’ factors of variations.
Altogether, we provide positive results of scaling up traditional research of language evolution literature on emergent communication tasks to the contemporary deep learning framework, thus opening avenues to more realistic (and large scale) computational simulations of language emergence with complex image stimuli.
A C ONCEPTUAL ALIGNMENT OF SPEAKER AND LISTENER
During conversation, communication allows interlocutors to achieve interactive conceptual alignment (Garrod & Pickering, 2004).
We are able to communicate because we have established a common ground and our representations at different levels become aligned (e.g., participants mutually understand that “he” in the conversation refers to Bob).
We investigated whether the agents’ conceptual systems achieve a similar structural alignment.
We measure the alignment in terms of
Spearman ρ correlation of the intra-agent pairwise object cosine similarities as calculated via representing objects as activations from ConvNet layers.
Interestingly, we observe a gradual increase in the structural similarity as we represent the objects with layer activations closer to the pixel space.
Conceptual spaces are more aligned the closer they are to the raw pixel input ( ρ = 0.97-0.91, depending on the game) and become more dissimilar as the representations become more abstract.
We can draw the analogy to language processing, as ﬁrst ConvNet layers perform some low-level processing analogous to phoneme recognition or word segmentation (and are thus more objective) while higher layers perform more abstract processing, vaguely analogous to semantics and pragmatics (thus, represent more subjective knowledge).
In cases of successful communication, speakers’ and listeners’ conceptual spaces closer to the communication point are structurally very similar ( ρ = 0.85-0.62, depending on the game), however this similarity drops dramatically in cases of failure of communication ( ρ = 0.15).
B H YPERPARAMETER DETAILS
All LSTM hidden states of the “speaking” and “listening” module as well and the “seeing” prelinguistic feed-forward encoders (see Section 3), have dimension 50.
The “seeing” pre-linguistic
ConvNet encoders (see Section 4) has 8 layers, 32 ﬁlters with the kernel size 3 for every layer and with strides [2 , 1 , 1 , 2 , 1 , 2 , 1 , 2] for each layer.
We use ReLU as activation function as well as batch normalization for every layer.
For learning, we used the Rmsprop optimizer, with learning rate 0.0001. We use a separate value of entropy regularization for each policy. For π S we use 0.01 and for π L we use 0.001. We use a mini-batch of 32.
C C OMMUNICATIVE SUCCESS USING GOLD ATTRIBUTE CLASSIFIERS
We assume a model which has access to perfect attribute classiﬁers for color , shape and object position , for the latter using a classiﬁer operating on the discretized annotations we obtained in
Section 4.2 after quantazing the real-valued object location.
For computing the performance of this model using gold attribute classiﬁers, we ﬁrst remove from the distractors any candidate not matching the target’s attributes and them pick at random.
We repeat this experiment for single attribute classiﬁers and their pairwise combinations.
Table 5 reports the communicative success results obtained empirically by averaging across 1000 simulations, alongside the training and test accuracies of the trained agents of Section 4.1 for comparison.
[Table]