M ULTI -A GENT
C OMPOSITIONAL
C OMMUNICATION
L EARNING FROM R AW V ISUAL I NPUT
Anonymous authors
Paper under double-blind review
A BSTRACT
One of the distinguishing natures of human language is its compositionality, which allows us to describe the complex environment with limited vocabulary.
Previously, it has been shown that neural network agents can learn to communicate in compositional language based on disentangled input ( e.g. hand-engineered features).
Humans, however, perceive the environment in raw signals, not well-summarized features.
In this work, we train neural network agents to communicate in compositional language based only on raw image pixels.
The agents play an image description game where the image contains compositional factors such as colors and shapes.
We train the agents using the obverter technique where an agent introspects to generate a message that best makes sense to itself.
During the zero-shot test, agents could communicate with the accuracy as high as 97% even for objects not seen during the training, providing evidence that neural network agents are capable of learning compositional language even from raw image pixels.
I NTRODUCTION
One of the key requirements for artiﬁcial general intelligence (AGI) to thrive in the real world is its ability to communicate with humans in natural language.
The recent natural language processing (NLP) research, especially after the introduction of deep learning methods (LeCun et al., 2015), have shown great progress in training machines to process and generate natural language (Cho et al., 2014;
Bahdanau et al., 2014; Vinyals et al., 2015; Karpathy & Fei-Fei, 2015; Hu et al., 2017; Serban et al., 2016; Lewis et al., 2017).
However, training machines in a supervised manner with a large dataset has its limits when it comes to communication.
Supervised methods are effective for capturing statistical associations between discrete symbols ( i.e. words, letters).
However, the essence of communication is more than just predicting the most likely word to come next.
It is a means to coordinate with others and potentially achieve a common goal (Austin, 1975; Clark, 1996; Wittgenstein, 2010).
An alternative path to teaching machines the art of communication is to give them a speciﬁc task and encourage them to learn how to communicate on their own.
This approach will encourage the agents to use languages grounded to task-related entities as well as communicate with other agents, which resembles how humans learn to communicate (Bruner, 1981).
Recently, there have been several notable works that demonstrated the emergence of communication between neural network agents (Sukhbaatar et al., 2016;
Foerster et al., 2016; Jorge et al., 2016;
Lazaridou et al., 2017).
Even though each work produced very interesting results of its own, in all works, communication was either achieved with a single discrete symbol (as opposed to a sequence of discrete symbols) or via a continuous value.
Needless to mention the latter, even using a single discrete symbol is quite far from natural language communication.
One of the key features of human language is its compositional nature (Hurford, 2004); we do not express our intentions with a single word, but rather use words as building blocks to compose a complex meaning.
More recently, Mordatch & Abbeel (2017) and
Kottur et al.
(2017) trained the agents to communicate in grounded, compositional language.
In both studies, however, inputs given to the agents were hand-engineered features ( disentangled input ) as opposed to raw signals we receive as humans.
In this work, we train neural agents to develop compositional language based on raw signals, speciﬁcally image pixels.
This is especially difﬁcult for the agent since it has to learn at the same time to disentangle visual input and use discrete symbols to describe it unequivocally.
We place the agents in a two-person image description game, where images contain objects of various color and shape.
Inspired by a pioneering work of Batali (1998), we employ a communication philosophy named obverter to train the agents.
Having its root in the theory of mind (Premack & Woodruff, 1978) and human language development (Milligan et al., 2007), obverter technique motivates an agent to generate messages that make sense to itself.
We conduct various experiments including the zero-shot test to evaluate the compositionality of the learned language.
The contribution of our work can be summarized as follows:
• We train artiﬁcial agents to learn to communicate in compositional language based only on raw image pixels.
• We describe how obverter technique could be employed in a communication with raw visual input, and show it is a promising alternative to RL-based approaches to train communicating agents.
• We visualize how the agents are perceiving the images and show that they learned to disentangle color and shape without any explicit supervision.
• Experiment results, including the zero-shot test accuracy as high as 97% , suggest that the agents could develop, out of raw image input, a compositional language, given a proper pressure from the environment ( i
e. the image description game).
In the following section, we describe the two-person image game and the model architecture.
We also describe how we employ the obverter philosophy to train the agents, and how the game should be sufﬁciently challenging for the agents to be motivated to learn compositional language.
In section 3, we analyze the experiment results in various aspects.
We conclude this paper with a discussion on the importance of the quantitative measure of compositionality in section 4.
M ETHOD 2. T HE TWO - PERSON
[Figure]
Listener
ABACD
Speaker Listener ABACD box, sphere, cylinder, capsule, ellipsoid ), giving us totalcombinations.
In a single round of the two-person image description game, one agent becomes the speaker and the other the listener.
The speaker is given a random image, and generates a message to describe it.
The listener is also given a random image, possibly the same image as the speaker’s.
After hearing the message from the speaker, the listener must decide if it is seeing the same image as the speaker (Figure 1).
In other words, the listener decides whether the message and the given image are consistent or not.
In each round the agents change roles of being the speaker and the listener.
In order to develop a compositional language, there must be compositional features to the images.
We generated synthetic images using Mujoco physics simulator 1 .
The example images are shown in
Figure 2.
Each image depicts a single object with a speciﬁc color and shape in 128 × 128 resolution.
There are eight colors ( blue, red, white, gray, yellow, green, cyan, magenta ) and ﬁve shapes ( box, sphere, cylinder, capsule, ellipsoid ), giving us 40 combinations.
We generated 100 variations for each of the 40 object type.
Note that the position of the object varies in each image, changing the object size and the orientation.
Therefore even if the speaker and the listener are given the same object type, the actual images are very likely to be different, preventing the agents from using pixel-speciﬁc information, rather than object-related information to win the game.
2. M ODEL ARCHITECTURE
[Figure]
Aside from using disentangled input, another strong assumption made in previous works(Batali, 1998;
Mordatch & Abbeel, 2017) was that the agents had access to the true intention of the speaker.
In
Batali (1998), the listener was trained to modify its RNN hidden vector as closely to the meaning vector as possible.
In Mordatch & Abbeel (2017), each agent had an auxiliary task to predict the goals of all other agents.
In both cases, the true meaning/goal vector was used to update the model parameters, exposing the disentangled information to the agents.
In order to relax this assumption and encourage the agents to develop compositional language with minimal guidance, our model uses no other signal than whether the listener made a correct decision.
Figure 3 depicts the agent model architecture.
We use a convolutional neural network followed by a fully-connected layer to process the image.
Messages are generated and consumed by the RNN, speciﬁcally, gated recurrent units (GRU) (Cho et al., 2014).
Then the image embedding from the visual module and the message embedding from the language module are concatenated and processed by another fully-connected layers ( i.e. decision module) with the sigmoid output ˆ y , 0 being “My (listener) image is different from the speaker’s” and 1 being “My image is the same as the speaker’s”.
Further details of the model architecture ( e.g. number of layers) are described in Appendix C.
2. O BVERTER TECHNIQUE
Batali (1998) used the obverter technique to train the neural agents, which has its root in the theory of mind (see Appendix A for the description of Batali (1998)).
Theory of mind (Premack & Woodruff, 1978) observes that a human has direct access only to one’s own mind and not to the others’.
Therefore we typically assume that the mind of others is analogous to ours, and such assumption is reﬂected in the functional use of language (Bruner, 1981).
For example, if we want to convey a piece of information to the listener 2 , it is best to speak in a way that maximizes the listener’s understanding.
However, since we cannot directly observe the listener’s state of mind, we cannot exactly solve this optimization problem.
Therefore we posit that the listener’s mind operates in a similar manner as ours, and speak in a way that maximizes our understanding , thus approximately solving the optimization problem.
This is exactly what the obverter technique tries to achieve.
When an agent becomes the teacher ( i.e. speaker), the model parameters are ﬁxed.
The image is converted to an embedding via the visual module.
After initializing its RNN hidden layer to zeros, the teacher at each timestep evaluates ˆ y for all possible symbols and selects the one that maximizes ˆ y .
The RNN hidden vector induced by the chosen symbol is used in the next timestep.
This is repeated until ˆ y becomes bigger than the predeﬁned threshold, or the maximum message length is reached (see Appendix D for algorithm).
Therefore the teacher, through introspection, generates a message such that the consistency between the image and the message is as clear to itself as possible.
When an agent becomes the learner ( i.e. listener), its parameters are updated via back-propagating the cross entropy loss between its output ˆ y and the true label y .
Therefore the agents must learn to communicate only from the true label indicating whether the teacher and the learner are seeing the same object.
We remind the reader that only the learner’s RNN parameters are updated, and the teacher uses its ﬁxed RNN.
Therefore an agent uses only one RNN for both speaking and listening, guaranteeing self-consistency (see Appendix B for a detailed comparison between the obverter technique and the
RL-based approach).
Another feature of the obverter technique is that it observes the principle of least effort (Zipf, 2016).
Because the teacher stops generating symbols as soon as ˆ y reaches the threshold, it does not waste any more effort trying to perfect the message.
The same principle was implemented in one way or another in previous works, such as choosing the shortest among the generated strings (Kirby & Hurford, 2002) or imposing a small cost for generating a message (Mordatch & Abbeel, 2017).
2. E NVIRONMENTAL
During the early stages of research, we noticed that randomly sampling object pairs (one for the teacher, one for the learner) lead to agents focusing only on colors and ignoring shapes.
When the teacher’s object is ﬁxed, there are 40 (8 colors × 5 shapes) possibilities on the learner’s side.
If the teacher only talks about the color of the object, the learner can correctly decide for 36 out ofpossible object types.
The learner makes incorrect decisions only when the teacher and the learner are given objects of the same color but different shapes, resulting in 90% accuracy on average.
This is actually what we observed; the accuracy plateaued between 0 . 9 and 0 .
92 during the training, and the messages were more or less the same for objects with the same color.
Therefore when constructing a mini-batch of images, we set 25% to be the object pairs of the same color and shape, 30% the same shape but different colors, 20% the same color but different shapes.
The remaining 25% object pairs were picked randomly 3 .
This suggests that the environment plays as important, if not more, role as the model architecture in the emergence of complex communication, and should be a main consideration for future efforts.
Further details regarding hyperparameters are described in Appendix E.
E XPERIMENTS
In this section, we ﬁrst study the convergence behavior during the training phase.
Then we analyze the language used by the agents and evaluate its compositionality.
Lastly, we conduct a zero-shot test to see if the agents can compose consistent messages for objects they have not seen during the training.
We also visualize the image embeddings from the visual modules of both agents to understand how they are recognizing colors and shapes, the results of which, for a better view of the ﬁgures, are provided in Appendix H.
3. C ONVERGENCE BEHAVIOR
Figure 4 shows the convergence behavior during the training.
Training accuracy was calculated by rounding the learner’s sigmoid output by 0 .
Message distinctness was calculated by dividing the number of unique messages in the mini-batch by the size of the mini-batch.
Ideally there should be, on average, 40 distinct messages in the mini-batch of 50 images, therefore giving us 0 .
8 distinctness.
equally work well, but we empirically found this ratio to be effective in many experiments
Every 10 round, both agents were given the same 1 , 000 randomly sampled images to generate 1 ,message pairs.
Then perplexity was calculated for each object type and averaged, thus indicating the average number of distinct messages used by the agents to describe a single object type (note that perplexities in the plot was divided by 20).
Jaccard similarity between both agents’ messages was also calculated for each object type and averaged.
At the beginning, the listener ( i.e. learner) always decides it is not seeing the same object as the speaker, giving us 0 .
75 accuracy 4 .
But after 7 , 000 rounds, accuracy starts to go beyond 0 .
Loss is negatively correlated with accuracy until round 15 , 000 , where it starts to ﬂuctuate.
Accuracy, however, remains high due to how accuracy is measured; by rounding the learner’s output by 0 . 5 .
Although we could occasionally observe some patterns in the messages when both accuracy and loss were high, a lower loss generally resulted in stronger compositionality and better zero-shot performance.
The loss ﬂuctuation also indicates some instability in the training process, which is a potential direction for future work.
Message distinctness starts at near 0 , indicating the agents are generating the same message for all object types.
After round 7 , 000 , where both message distinctness and message length reach their maximum, both start to decrease.
But message distinctness never goes as high as the ideal 0 .
8 , meaning that the agents are occasionally using the same message for different object types, as will be shown in the following section.
Both perplexities and Jaccard similarity show seemingly meaningless ﬂuctuation at early rounds.
After round 7 , 000 , perplexities and Jaccard similarity show negatively correlated behavior, meaning that not only is each agent using consistent messages to describe each object type, but also both agents are using very similar messages to describe each object type.
We found perplexity and
Jaccard similarity to be an important indicator of the compositionality of the messages.
During rounds 7 , 000 ∼ 8 , 000 , performance was excellent in terms of loss and accuracy, but perplexity was high and Jaccard similarity low, indicating the agents were assigning incoherent strings to each object type just to win the game.
Similar behavior was observed in the early stages of language evolution simulation in Kirby & Hurford (2002) where words represented some meanings but had no structure ( i.e. protolanguage).
It seems that the compositional language comes after the emergence of protolanguage regardless of whether the input is entangled or disentangled.
3. C OMPOSITIONALITY ANALYSIS
We choose agents from different training rounds to highlight how the language becomes more structured over time.
Table 1 shows agents’ messages in the beginning (round 40), when the training accuracy starts pushing beyond 90% (round 6 , 940 ), when agents settle on a common language (round 16 , 760 ).
[Table]
aeceaeaeaaaeeeeeeeeeaacceeaaaaeeeeeeeeeeaaccceaaaaaaaaeeeeeeaeeeeaaaaeeeeeeeeeeeaeaceeaeaeeeeeeeeeeeaceacacaaaaaaeeeeeeeabeeeeaeeeeeeeeeeeeeaacceeaeeeeeeeeeeeeeaacceeaeaaeeeeeeeeeeaeeacacaaaeeccceeeee aaedacacaaaaaaaaaaaaaaccdaacaaaaaaaaaaaaaaabcdadaaaaaaaaaaaaaaeeacaeaaaaaaaaaaaaaaccdadaaaaaaacaaaaaaaedaceaaaaaaaaaaaaaaaeaccaeaaaaaaaaaaaaaaceacaacaaaaaaaaaaaaacdacdaaaaaaaaaaaaaaaccdadaaaaacaaaaaaa
[Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table] [Table] [Table] [Table]
In round 40 , both agents are respectively producing the same message for all object types as mentioned in section 3.1.
We might say the messages are structured, but considering that the listener always answers 0 in early rounds, we cannot say the agents are communicating .
In round 6 , 940 , which is roughly when the agents begin to communicate more efﬁciently, training accuracy is signiﬁcantly higher than round 40 .
However, perplexities show that both agents are assigning many names to a single object type (40-80 names depending on the object type), indicating that the agents are focusing on pixel-level differences between images of the same object type.
Table 1 shows, as an example, the messages used by both agents to describe the red sphere .
Due to high perplexity, it is difﬁcult to capture the underlying structure of the messages even with regular expression.
Furthermore, as
Jaccard similarity indicates, both agents are generating completely different messages for the same object type.
In round 16 , 760 , as the perplexities and Jaccard similarity tell us, the agents came to share a very narrow set of names for each object type (1-4 names depending on the object type).
Moreover, the names of the same-colored objects and same-shaped objects clearly seem to follow a pattern.
We found the messages in round 16 , 760 could be decomposed in a similar manner as Table 6 in
Appendix A.
The top of Table 2 shows a possible decomposition of the messages from round 16 ,and the bottom shows the rules for each color and shape derived from the decomposition.
According to our analysis, the agents use the ﬁrst part of the message ( i.e. preﬁx) to specify a shape, and the second part ( i.e. sufﬁx) to specify a color.
However, they use two different strings to specify a shape.
For example, the agents use either aaaa or bbbbb to describe a box .
The strings used for specifying colors show slightly weaker regularity.
For example, red is always described by either the sufﬁx c or sufﬁx e, but magenta is described by the sufﬁx bb, bd, and sometimes b or bc.
¯a used for gray objects represents deletion of preﬁx a .
Note that removing preﬁxes beyond their length causes the pattern to
Box
Sphere
Cylinder
Capsule
Ellipsoid
Blue bbbbb bb { b,d } bb { b,d } bbbb bb { b,d } bbb bb { c,d } bb bb { ,c,d } Red aaaa { c,e } aa { c,e } aaa { c,e } a { c,e } { c,e } White bbbbb b { b,d } bbbb { b,d } bbb { b,d } bb { ,c,d } Gray aaaa ¯a¯aa aa ¯a¯aa aaa ¯a¯aa a ¯a¯ac ¯a¯a { b,bd } Yellow aaaa aa aa aa aaa aa a aa a { a,e } Green aaaa { a,ad } aa { a,ad } aaa { a,ad } a { a,ad } a Cyan a ×bbb { b,d } bbbb bbb { b,d } bbb bbbd bb bbb { b,d } Magenta bbbbb bb b { b,d } bbbb b { b,d } bbb bd bb b { ,c,d }
Deﬁne Blue, White, Cyan, Magenta as color group 0, rest as color group 1.
Rule
Blue
End with bbb or bbd Red
End with c or e White
End with b or d Gray
End with
[Table]
End with
[Table]
End
[Table]
or
[Table]
Box
[Table]
[Table] [Table]
break.
For example, Gray box, gray sphere and gray cylinder use the same ¯a¯aa to express the color, but gray capsule and gray ellipsoid use irregular sufﬁxes.
Despite some irregularities and one exceptional case ( cyan box ), the messages follow a clear pattern, providing strong evidence that the agents learned to properly recognize color and shape from entangled input, mapped each color and shape to preﬁxes and sufﬁxes, and are able to compose meaningful messages to describe a given image to one another.
Communication accuracy for each object type is described in Appendix F. Communication examples and their analysis are given in Appendix G. [Table] EVALUATION
[Table]
Blue eeeeee e { e,ee } eeee ee eeeee e { e,ed } eee e { e,ed } ee e { e,a } Red eeeeee ¯ e ¯ e ¯ e ¯ e { e,a } eeee ¯ e ¯ e ¯ e ¯ e { e,ea } eeeee ¯ e ¯ e ¯ e ¯ e { e,a } eee ¯ e ¯ e ¯ e ¯ e { b,ba } ee ¯ e ¯ e ¯ e ¯ e { a,c } White bbbb b b b bbb { b,d } bb { b,d } bbb { d,c }
Gray eeeeee ¯ e ¯ e ¯ ee eeee ¯ e ¯ e ¯ e { ,a } eeeee ¯
e
¯
e
¯ e { e,a } eee ¯ e ¯ e ¯ e { e,a } ee ¯
e
¯
e
¯ e { b,d }
Yellow bbbb ¯ b ¯ b { c,d } b ¯ b ¯ be { e,ec } bbb ¯ b ¯ b { c,d } bb ¯ b ¯ b { a,c } bbb ¯ b ¯ b { a,e }
Green eeeeee { e,a } eeee { e,a } eeeee { e,a } eee { e,a } ee { e,a }
Cyan eeeeee eeea eeee ee { e,a } eeeee ee { e,ea } eee ee { e,a } ee eea
Magenta bbbb ¯ b { b,d } b ¯ b { c,d } bbb ¯ b { b,d } bb ¯ bb bbb ¯ b ¯ b { c,a }
If the agents have truly learned to compose a message that can be divided into a color part a shape part, then they should be able to accurately describe an object they have not seen before, which is one of the marks of compositional language.
Therefore, we hold out ﬁve objects (the shaded cells in Table 3) from the dataset during the training and observe how agents describe ﬁve novel objects during the test phase.
The agents were chosen from round 19 , 980 , which showed a high accuracy ( 97 . 8% ), low perplexities ( 1 . 48 , 1 . 65 ) and a high Jaccard similarity ( 0 . 75 ).
Table 3 shows a potential decomposition of the messages used by the agents (original messages are described by Table 8 in
[Table]
Appendix I).
We can observe the overall compositional tendency, although some messages show somewhat weaker patterns compared to when the agents were trained with all object types (Table 2).
Sufﬁxes for specifying yellow and magenta are especially irregular, even when we consider the effects of ¯b and ¯e.
However, the messages describing the held-out object types show clear compositional patterns with the exception of yellow ellipsoid .
In order to assess the communication accuracy when held-out objects are involved, we conducted another test with the agents from round 19 , 980 .
Each held-out object was given to the speaker, the listener, or both.
In the ﬁrst two cases, the held-out object was paired with all 40 object types and each pair was tested 10 times.
In the last case, the held-out object was tested against itself 10 times.
In all cases, the agents switched roles after 5 times.
Table 4 shows communication accuracies for each case.
We can see the agents can successfully communicate most of the time even when given novel objects.
The last column shows that the listener is not simply producing 0 to maximize its chance to win the game.
It is also notable that the objects described without ¯ b or ¯ e show better performance in general.
We noticed the communication accuracy for held-out objects seems relatively weak considering the messages used to describe them seem to strongly follow compositional patterns (Table 3).
This, however, results from the compositional pattern itself being less clear than Table 2, especially with short messages ( i.e. frequent use of ¯b and ¯ e).
The same tendency can be observed for non-held-out objects as described by the per-object communication accuracy Table 9 in Appendix J.
D ISCUSSION AND F UTURE W ORK
In this work, we trained the neural network agents to successfully communicate based on entangled visual input, using the obverter technique and the two-person image game.
The ultimate aim of this work was to motivate the agents to develop compositional language.
Through qualitative analysis (Table 2), visualization and zero-shot test, we have shown that the emerged language had strong ﬂavor of compositionality.
However, we consider our evaluations as different ways to check if the necessary condition is met for the language to be considered compositional.
Even passing the zero-shot test, which researchers typically use to evaluate compositionality, does not guarantee the communication is compositional, as some tasks can be still be achieved when agents are speaking partial truth or using negations.
During the evaluation process we tried to design an evaluation method to undeniably conﬁrm the compositionality of the emerged language.
But we realized that the exact deﬁnition of compositional language is somewhat debatable, and, to the best of our knowledge, there was no reliable way to check for the compositionality of an arbitrary language.
Therefore, in order to encourage active research and discussion among researchers in this domain, we propose for future work, a quantitatively measurable deﬁnition of compositionality and ways to measure the compositionality of any language.
We believe compositionality of language is not binary ( e.g. language A is compositional/not compositional), but a spectrum.
Even some aspect of human language is compositional, but some are not ( e.g. irregular verb tenses in English).
It is also important to clearly deﬁne grounded language and compositional language.
If one agent says abc (eat red apple) and another says cba (apple red eat) , and they both understand each other, are they speaking compositional language?
We believe such questions should be asked and addressed to shape the deﬁnition of compositionality.
In addition to the deﬁnition/evaluation of compositional language, there are numerous directions of future work.
Observing the emergence of compositional language among more than two agents is an apparent next step.
Designing an environment to motivate the agents to disentangle more than two factors is also an interesting direction.
Training agents to consider the context ( i.e. pragmatics), such as giving each agent several images instead of one, is another exciting future work.
A E MERGENCE OF GRAMMAR , B ATALI (1998)
In Batali (1998), the author successfully trained neural agents to develop a compositional language using disentangled meaning vectors as the input.
Using 10 subject vectors and 10 predicate vectors, all represented as explicit binary vectors, total 100 meaning vectors could be composed(Table 5).
Each
[Table]
digit in the subject vector 5a serves a clear role, respectively representing speaker( sp ), hearer( hr ), other( ot ), and plural( pl ).
The predicate vector values, on the other hand, are randomly chosen so that each predicate vector will have three 1 ’s and three 0 ’s.
The combination of ten subject vectors and ten predicate vectors allows 100 meaning vectors.
The author used twenty neural agents for the experiment.
Each agent was implemented with the vanilla recurrent neural networks (RNN), where the hidden vector h ’s size was 10, same as the size of the meaning vector m in order to treat h as the agent’s understanding of m .
In each training round a single learner ( i.e. listener) and ten teachers ( i.e. speaker) were randomly chosen.
Each teacher, given all 100 m ’s in random order, generates a message s 5 for each m and sends it to the learner.
The messages are generated using the obverter techinque, which is described in Algorithm 1.
The learner is trained to minimize the mean squared error (MSE) between h (after consuming the s ) and m .
After the learner has learned from all ten teachers, the next round begins, repeating the process until the error goes below some threshold.
When the training was complete, the author was able to ﬁnd compositional patterns in the messages used by the agents (Table 6).
Note that the messages using predicates tired, scared, sick and happy especially follow a very clear pattern.
Batali also conducted a zero-shot test where the agents were trained without the diagonal elements in Table 6 and tested with all 100 meaning vectors.
The agents were able to successfully communicate even when held-out meaning vectors were used, but the
- one they you yall yup me we mip yumi
all tired cda cdab cdc cdcb cdba cd cdd cddb
cdcd cdb scared caa caab cac cacb caba ca cad cadb cacd cab sick daa daab dac dacb daba da dad dadb dacd dab happy baa baab bca bcab baac ba badc bab bac babc sad aba abab ac acb abac a abdc abb abc abbc excited cba cbab cca cacb cbca c ccdc cb ccb cbc angry bb bbb bc bcb bbc b bddc bdb bdc bdbc silly aa aaab aca acab adba add addc adad adc
adbc thirsty dbaa dbab dca dcba dbca dda ddac dbad dcad dbacd hungry dbb dbbd dc dcb dbc dd ddc dbd dcd dbcdone they you
yall
yup me
we mip yumi all tired cd a cd ab cd c cd cb cd ba cd cd d cd db cd cd cd b scared ca a ca ab ca c ca cb ca ba ca ca d ca db ca cd ca b sick
da a da ab da c da cb da ba da da d da db da cd da b happy ba a ba ab
b c a b c a b ba ac ba ba dc ba b ba c ba bc sad a ba a bab a c a cb a bac a a bdc a bb
[Table]
messages used for the held-out meaning vectors did not show as strong compositional patterns as the non-zero-shot case.
B C OMPARISON BETWEEN THE OBVERTER TECHINQUE AND THE REINFORCEMENT LEARNING - BASED APPROACH
The obverter technique allows us to generate messages that encourage the agents to use a shared language, even a compositional one, via using a single RNN for both speaking and listening.
This is quite different from other RL-based related works (Lazaridou et al., 2017; Mordatch & Abbeel, 2017; Foerster et al., 2016; Jorge et al., 2016; Kottur et al., 2017) where each agent has separate components ( e [Table] g. two RNNs) for generating messages and consuming messages.
This is necessary typically because the message generation module and the message consumption module have different input/output requirements.
The message generation module accepts some input related to the task ( e.g. goal description vector, question embedding [Table] or image embedding) and generates discrete symbols.
The message consumption module, on the other hand, accepts discrete symbols ( i.e. the message) and generates some output related to the task ( e.g. some prediction or some action to take).
Therefore, when a neural agent speaks in the RL-based approach, its message generation process is completely separated from its own listening process, but tied to the listening process of another agent ( i.e. listener) 6 .
This means an agent may not have internal consistency; what an agent speaks may not make sense to itself.
However, agents in the RL-based setting do converge on common language because during the training the error signal ﬂows from the listener to the speaker directly.
Obverter approach, on the other hand, requires that each agent has a single component for both message generation and message consumption.
This single component accepts discrete symbols and generates some output related to the task.
This guarantees internal consistency because an agent’s message generation process is tied to its own message consumption process; it will only generate messages that make sense to itself.
In the obverter setting, the error signal does not ﬂow between agents directly, but agents converge on a common language by taking turns to be the listener; the listener tries to understand what the speaker says, so that when the listener becomes the speaker, it can generate messages that make sense to itself and, at the same time, will be understood by the former speaker (now listener).
The advantage of obverter approach over RL-based approach is that it is motivated by the theory of mind and more resembles the acquisition/development process of human language.
Having a single mechanism for both speaking and listening, and training oneself to be a good listener leads to the emergence of self-consistent, shared language.
However, obverter technique requires that all agents perform the same task, which means all agents must have identical model architectures.
This is because, during the message generation process, the speaker internally simulates what the listener will go though when it hears the message.
Therefore we cannot play an asymmetrical game such as where the speaker sees only one image and generates a message but the listener is given multiple images and must choose one after hearing the message.
RL-based approaches do not have this problem since there are separate modules for speaking and listening.
We believe obverter technique could be the better choice for certain tasks regarding human mind emulation.
But it certainly is not the tool for every occasion.
The RL-based approach is a robust tool for any general task that may or may not involve human-like communication.
We conclude this section with a possible future research direction that combines the strengths of both approaches to enable communication in more interesting and complicated tasks.
C M ODEL ARCHITECTURE DETAILS
We used TensorFlow and the Sonnet library for all implementation.
C. V ISUAL MODULE
We used an eight-layer convolutional neural network.
We used 32 ﬁlters with the kernel size 3 for every layer.
The strides were [2 , 1 , 1 , 2 , 1 , 2 , 1 , 2] for each layer.
We used rectiﬁed linear unit (ReLU) as the activation function for every layer.
Batch normalization was used for every layer.
We did not use the bias parameters since we used Batch normalization.
For padding, we used the TensorFlow
VALID padding option for every layer.
The fully connected layer that follows the convolutional neural network was of 256 dimensions, with ReLU as the activation function.
C. L ANGUAGE MODULE
We used a single layer Gated Recurrent Units (GRU) to implement the language module.
The size of the hidden layer was 64.
C. D ECISION MODULE
We used a two-layer feedforward neural network.
The ﬁrst layer reduces the dimensionality towith ReLU as the activation function, then the second layer generates a scalar value with sigmoid as the activation function.
D M ESSAGE GENERATION ALGORITHM USED IN OUR WORK E T RAINING DETAILS
Both agents’ model parameters are randomly initialized.
The training process consists of rounds where teacher/learner roles are changed, and each round consists of multiple games where learner’s model parameters are updated.
In each game, the teacher, given a mini-batch of images, generates corresponding messages.
The learner, given a separate mini-batch of images and the messages from the teacher, decides whether it is seeing the same object type as the teacher.
Learner’s model parameters are updated to minimize the cross entropy loss.
After playing a predeﬁned number of games, we move on to the next round where two agents change their roles.
We found twenty games per round, with ﬁfty images per mini-batch to work well.
We repeat the rounds for 20 , 000 times.
Further rounds did not improve the results, or even degraded the performance.
For vocabulary size ( i.e. number of unique symbols) and the maximum message length, we used 5 and 20 respectively, similar to what Batali (1998) used.
We noticed that a larger vocabulary and a longer message length helped the agents achieve a high communication accuracy more easily.
But the resulting messages were challenging to analyze for compositional patterns.
Note that when generating a message using the obverter technique, the generation process stops as soon as the speaker’s ( i.e. teacher) output ˆ y becomes bigger than some threshold.
In our work, we experimented with various values from 0 .
5 to 0 .
95 , and found higher values to work better than lower values.
We used 0 .
95 for all our ﬁnal experiments.
F C OMMUNICATION ACCURACY FOR EACH OBJECT TYPE
[Table]
50 Red [Table] [Table] We conducted a separate test with the agents from round 16 , 760 to assess the communication accuracy for each object type.
The agents were given 1 , 600 total object pairs ( 40 × 40 ).
Each object pair was tested 10 times, where after 5 times the agents switched speaker/listener roles.
The average accuracy was 95 .
4% , and only 88 out of 1 , 600 object pairs were communicated with accuracy lower than 0 . 8 .
Table 7 describes the accuracy when each object type was given the speaker.
We can observe that the accuracy is higher for objects that are described with less overlapping messages.
For example, yellow box is communicated with the accuracy of 98% , and it is described with aaaaaa , which is not used for any other object types.
Gray box , on the other hand, is communicated with accuracy 93% .
It is described with aaa , which is also used for yellow capsule and green sphere , both of are communicated with low accuracies as well.
[Figure]
G C OMMUNICATION EXAMPLE AND ANALYSIS
Figure 5 provides ten examples of communication when the speaker is given a blue box and the listener is given various object types.
The listener’s belief ( i.e. score) that it is seeing the same image as the speaker changes each time it consumes a symbol.
It is notable that most of the time the score jumps between 0 and 1 , rather than gradually changing in between.
This is natural given that messages that differ by only a single character can mean different objects ( e.g. blue box and blue cylinder ).
This phenomenon can also be seen in human language.
For example, blue can and blue cat differ by a single alphabet, but the semantics are completely different.
Object types that are described by similar messages as blue box , such as blue cylinder and magenta box cause marginal confusion to the listener such that prediction scores for both objects are not complete zeros.
There are also cases where two completely different objects are described by the same message as mentioned in Section 3.1.
From Table 2 we can see that blue box and cyan cylinder are described by the same message bbbbbbb { b,d } , although the underlying composition rules are different.
Therefore the listener generates high scores for both objects, occasionally losing the game when the agents are given this speciﬁc object pair (1 out of 40 chance).
This can be seen as a side effect coming from the principle of least effort which motivates the agents to win the game most of the time while minimizing the effort to generate messages.
The same phenomenon is observed in human language in the form of homonyms 7 .
[Figure]
Section 3.2 provides strong evidence the agents are properly recognizing the color and shape of an object.
In this section, we study the visual module of both agents to study how they are processing the pixel input.
We give each agent 1600 images, 40 per object type, and take their image embeddings (output of the fully-connected layer in the visual module).
We use t-SNE (Maaten & Hinton, 2008) to reduce the dimensionality to 2D, and generate scatter plots as shown by Figure 6.
The top row and the bottom row are the same scatter plots, but the dots are colored with different labels; the top row shows the color of each object, and the bottom row shows the shape of each object.
It is notable that the image embeddings form clear clusters, and all clusters are quite disentangled from one another.
This means that the agents have learned to differentiate objects by their color and shape, and each color and shape have a speciﬁc place in the agents’ mind.
It is also impressive that not only both agents learned similar relationships between colors and shapes(Figure 6 (a) and (b) show similar clusters, as do (d) and (e)), but also they learned similar absolute values for colors and shapes.
Even when we plot image embeddings from both agents together (Figure 6 (c) and (f))
the cluster qualities are kept almost identically (with slightly higher number of clusters) to when we plot them separately.
Therefore when one agent thinks of color red , and utters a message to describe it, the other agent hears the message and think of something red as well.
This is, of course, what we wanted to achieve by using the obverter technique.
The fact that a couple of color or shapes are occasionally in the same cluster suggests the agents have not perfectly disentangled colors and shapes.
For perfect disentanglement, we believe some modiﬁcations to the image description game is required, which can be an important topic for future work.
However, studying the cluster sheds some light on why the agents generate speciﬁc messages for each object type.
For example, in Figure 6 (a) and (b), blue, white, cyan and magenta are often intertwined or their respective clusters are located nearby, as is the case for red, gray, yellow and green .
This suggests the reason agents use preﬁx b ’s to specify shapes for former color group and preﬁx a ’s to specify shapes for the latter color group.
Additionally, in Figure 6 (d) and (e), Box and cylinder are often located nearby, and sphere and ellipsoid show similar behavior 8 .
We conjecture that this is the reason the messages describing box and cylinder are similarly long, and the messages describing sphere and ellipsoid tend to be similarly short.
I O RIGINAL MESSAGES FROM THE
- SHOT TEST
[Table]
eeeeeee { e,ee } eeeeee eeeeee { e,ed } eeee { e,ed } eee { e,a }
[Table]
ee { e,a } e,ea e { e,a } b,ba a,c
[Table]
bb bbb {
[Table]
,d } bb { b,d } bbb { d,c }
[Table]
eeee e,ea ee { e,a } e,a b,d
[Table]
[Table]
d } e { e,ec } b { c,d } a,c a,e
[Table]
eeeeee { e,a } eeee { e,a } eeeee { e,a } eee { e,a } ee { e,a
[Table]
eeeeeeeeea
[Table]
eeeeee {
[Table]
[Table]
} eeeeeee { e,ea } eeeee {
[Table]
,a } eeeea
[Table]
bbb
[Table]
[Table]
[Table] [Table]
bb
[Table] [Table]
J C OMMUNICATION ACCURACY FOR EACH OBJECT TYPE IN ZERO - SHOT TEST
[Table]
In the same manner as Appendix F, we conducted a separate test with the agents from round 19 ,to assess the communication accuracy for each object type.
The agents were given 1 , 600 total object pairs ( 40 × 40 ).
Each object pair was tested 10 times, where after 5 times the agents switched speaker/listener roles.
The average accuracy was 94 .
73% , and 103 out of 1 , 600 object pairs were communicated with accuracy lower than 0 . 8 .
Table 9 describes the accuracy when each object type was given the speaker.
Shaded cells indicate objects not seen during the training.
Here we can observe the same tendency as the one seen in
Appendix F; the accuracy is higher for objects that are described with less overlapping messages.