V ARYING MEANING COMPLEXITY TO EXPLAIN AND MEASURE COMPOSITIONALITY
A BSTRACT
Compositionality is assumed to be a key property of language, but it is hard to observe in language emergence simulations.
Following De Beule & Bergen (2006), we posit that the meaning of the datapoints that agents discuss must vary in com plexity.
We extend their work in different directions.
First, we argue that this vari ation in the task is realistic and underlies the emergence of intersective adjectives and argument structure.
Secondly, we show promising results for this hypothesis with attention based neural networks.
Thirdly, we argue that languages learned on tasks where meaning complexity varies are easier to analyse, and propose an intuitive metric called concatenability to illustrate this claim.
Natural language is often thought to be compositional , so that “the meaning of a complex expression is determined by its structure and the meaning of its constituents” (Szabó, 2020).
However, we do not fully understand the nature of compositionality, how it emerges and how it can be integrated as inductive biases in artiﬁcial agents.
For Zipf (1949), language maximizes information transmission while minimizing the effort of the speakers and listeners.
This optimisation process is constrained by factors internal to the agents (memory constraints, preferences for iconicity, etc.)
as well as by external factors such as the distribution of the meanings, pragmatics or discourse factors (Kirby, 1999).
We can tease apart these factors using computer simulations of language evolution and acquisition (Steels, 1997; Lazaridou & Baroni, 2020).
However, such simulations are often not very successful, unless they use unrealistic assumptions, in which case they also lose part of their explanatory value.
For example, Kottur et al.
(2017) restrict the vocabulary size of the speaker to be exactly the number of different values to denote, and also erases its memory after each utterance.
Korbak et al.
(2019) proposes that compositionality evolves from simpler communication protocols where the primitives (“blue”, “circle”) are learned in a separate training process, but the syntax of the agents is hardcoded through very speciﬁc training procedure involving several agents.
The difﬁculty of quantifying compositionality is yet another hurdle to progress.
When agents exchange variable-length utterances without an explicit model of grammar, it is not clear whether there are any meaningful subconstituents or not, and if there are any, how to segment them in order to measure the quantities of interest.
Thus it is often implicitly assumed that each symbol bears meaning.
However, this can lead to strange conclusions: if we assume that each letter in “cat” is meaningful, then English is not compositional.
To tackle these problems, we propose to study language emergence simulations where the complexity of the meanings to convey vary.
Combined with a loss function encoding Zipf (1949)’s least-effort principle, we posit that grammatical structures will grow in complexity to accommodate for the variations in complexity.
A similar hypothesis was proposed by De Beule & Bergen (2006) and we build on their work in the following ways.
Firstly, we argue that such variations are present in everyday communicative situations.
We propose two very different setups to explain the emergence of intersective adjectives and argument structure.
In this view, it is not a drawback of the method that task complexity must be low for compositionality to emerge, since it is realistic.
Secondly, we show how to reproduce their conclusions using neural architectures.
In their work, agents use explicit grammars, whereas ours use neural models based on attention mechanisms.
Our work adds weak evidence for the learnability (and not only the existence) of systematic connectionist architectures, contra Fodor et al.
T 1 2 3 F 2 
 F 1 2 3 T 2
T 1 2 3 T 2
T F F F 1
T 2 1 1 F 1
F F T F 1
F 2 1 1 T 1
T F T T 2
F 2 1 1 F 1
[Figure]
Message a bcc abc ≈ bcca ?
≈
abcc ?
[Figure]
Entities α
A
B − , 8 ,0 , 1 ,24 , 79 ,105 ,0 , 0 ,105 , 16 ,34 ,0 , 1 ,105 , 79 ,34 ,− , 8 , [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
18 ,− , 8 ,0 ,
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table] [Table] [Table] [Table]
[Table]
[Table] [Table] [Table] Finally, we show how meaning complexity variation helps to study artiﬁcial languages.
Having access to messages denoting objects or properties in isolation, we can sidestep the need to complex artiﬁcial messages into smaller constituents.
This framework opens the door to a variety of novel metrics.
We conclude with an overview of the opportunities afforded, outlining some concrete next steps.
T WO REALISTIC SETUPS WITH MEANING COMPLEXITY VARIATION
Compositionality is typically studied mathematically as follows (Montague, 1970; Szabó, 2020).
Consider a meaning function m that maps expressions taken from a language L (the space of grammatically correct expressions) to a space of meanings X (all the things that the language can denote).
A language is compositional iff m is an homomorphism, that is, if there are two binary operators, · on L , and × on X , such that for any expression made of two constituents e 1 and e 2 in L , we have
[Equation]
In language emergence simulations, L is the set of variable-length sequences of symbols produced by an agent (the speaker).
In our case, symbols are sampled from a vocabulary { 1 , . . .
, n V } and the length of the messages are bounded.
The operator · is the concatenation operator, so we write e 1 · e 2 = e 1 e 2 .
The meaning space X contains sets of ﬁller-role pairs, with the constraint that each role appears at most in one pair.
As we will see, the ﬁllers and roles differ slightly the two tasks.
Since we use sets, we can take × to be the union on sets.
We deﬁne simple meanings as sets from X containing a single pair, while complex meanings contain at least two pairs.
A task with meaning complexity variation contains both simple and complex meanings to convey.
We assume a functional view of language seen primarily as a communication device, shaped by its use.
The agents jointly optimize a loss function which has two terms, balancing the maximization of information transmission while minimizing the efforts of the speaker.
This system is a partial, particular model of the functional view put forward by Zipf (1949).
We further assume that this objective is the cause of various pragmatic effects such as the maxim of quantity of Grice (1975), according to which the speaker convey information that is necessary to solve a given task, and nothing more.
We found that inferring what part of the input should be conveyed is hard for current architectures and optimisation techniques.
Therefore, we help the agents by providing pragmatic annotations as inputs.
Agents can use these annotations to respect the Gricean maxim and communicate optimally, and they are encouraged to do so by a least-effort penalty, but they do not have to do so.
1. T HE EMERGENCE OF COMPOSITIONALITY IN INTERSECTIVE ADJECTIVES
The ﬁrst task is a discrimination task (Lazaridou et al., 2017; Havrylov & Titov, 2017).
The speaker observes a set of objects from X and one of these is marked as a target, while the others are distractors.
The speaker produces a message that characterizes the target.
The listener observes the same set of objects and must pick out the target using the message.
Each object x ∈ X is a set of attribute-value pairs.
For example, a blue circle can be described as x 1 = { ( COLOR , BLUE ) , ( SHAPE , CIRCULAR ) } .
Crucially, the agents are exposed to simple meanings, for which a single attribute-value pair is necessary to pick out the target, as well as to complex meanings, for which two pairs are necessary (cf.
ﬁrst column in Figure 1).
In other words, the meaning complexity varies.
This task simulates hypothetical conditions of emergence of intersective adjectives.
In natural language, these adjectives are concatenated sequentially, as in “the blue circular thing”.
2 There are several important parallel between frequent, real-life communicative situations and this task, but we need to focus on the realism of our innovation: the variable meaning complexity.
The primary function of adjectives is probably to distinguish between similar enough objects.
One does not use adjectives to discriminate between different kind of concrete things such as a car and a tree, as these things are more naturally associated to basic level terms, which probably have a lesser cognitive cost.
Thus, in our task, some of the meanings are such that a single attribute-value pair is necessary to solve the task.
By contrast, works from Lazaridou et al.
(2017); Havrylov & Titov (2017) seem to sample objects independently.
To help the speaker to be concise and thus obtain more natural languages, we provide it with the minimal set of attributes that distinguish the target from the distractors.
This set of attributes is given to the speaker via a mask over features, as illustrated in Figure 1 (column 1).
1. T HE EMERGENCE OF COMPOSITIONAL ARGUMENT STRUCTURE
In the second task, meanings in X are events which contain entity-role pairs.
We can think of these as representations of clauses.
For instance, one such event would be e =
{ ( ABOVE , x 1 ) , ( BELOW , x 2 ) , ( REL , ABOVENESS ) } , where x 1 and x 2 can be the geometric shapes of the previous task.
To simplify the experiments and the analysis of the results, the pair with the REL attribute is observed by both agents and needs not be conveyed.
The complete description of this setup will be published soon in a separate publication.
Crucially, each relation takes a variable number of roles.
Simple meanings require a single entityrole pair to be conveyed while complex meanings require several.
Furthermore, the listener observes a partial view of the input of the speaker.
This is illustrated in the second column of Figure 1: a meaning made up of two entities can be partially observed (ﬁrst 2 rows) or completely unobserved by the listener (last row).
When all but one entity is observed by the listener, the meaning is simple, since only the entity hidden to the listener must be communicated.
Otherwise, the meaning is complex.
Thus meaning complexity varies across inputs in this task as well.
This task simulates the emergence of argument structure.
Indeed, in all natural languages, the way arguments are composed seems to be highly systematic.
In simple sentences, arguments are simply concatenated to the verb.
This seems to be true, regardless if roles are encoded via word order (analytic languages), case marking (synthetic languages) or verb afﬁxes (polysynthetic languages).
The most important characteristic of this task is probably partial observability, and it corresponds to a fundamental aspect of the human experience.
Most events that have happened in the past or will take place in the future are not directly witnessed and are only partially represented in our minds.
Yet we often want to represent them more completely, which prompts one to ask questions, or spontaneously share information that is ignored by the interlocutor.
For instance, upon seeing a broken window, one can ask “who broke the window?”.
A knowledgeable interlocutor could answer “John” or “John did”.
In our experiments, the speaker is this knowledgeable person, answering such questions about unobserved entities.
Even if the agents do not engage in dialogues but in one-time interactions, the mask models an inference made by the speaker about the listener’s knowledge, and abstracts away over all the possible reasons to share this information.
Going back to our example, “John did” is an acceptable answer which does not contain the phrase “the window”.
This parsimony is encouraged by the least-effort penalty.
In Mordatch & Abbeel (2018)’s and Bogin et al.
(2019)’s works, agents also need to express such events.
However, the complexity of the meanings do not vary.
A formal and more detailed comparison between the two tasks is available in Section A.
S IDESTEPPING THE NEED FOR SEGMENTATION
To verify if compositionality holds as formalized in equation 1, we need to access the constituents e 1 and e 2 that make up the complex expressions of the form e 1 e 2 .
In some works, this structure is known (Andreas, 2018), but it is more typically assumed that every symbol is meaningful (for example, in Bogin et al.
and Chaabouni et al.
(2020)’s works).
However, neither of these options seems appropriate to analyze the messages exchanged by neural agents.
This problem is also identiﬁed by Baroni (2020), but to our knowledge, we are the ﬁrst to address it, or rather to sidestep it.
Using tasks with meaning complexity variation, we can encode simple meanings { x 1 } and { x 2 } in isolation to obtain meaningful constituents e 1 and e 2 .
This is possible because some of the simple meanings are part of the training data.
Let e 1 , 2 be the message produced by the speaker to discuss { x 1 , x 2 } .
To quantify compositionality, and in particular, systematicity, we would like to measure the similarity between m ( e 1 e 2 ) or m ( e 2 e 1 ) and m ( e 1 , 2 ) (cf.
third column in Figure 1).
One way to do so is to compare the loss incurred by the listener when it interprets e 1 e 2 versus e 1 , 2 .
Formally, we deﬁne the information transmission loss (reconstruction or discrimination) incurred when the complex meaning { x 1 , x 2 } is conveyed via e as l ( e ) = − log p ( { x 1 , x 2 }| e ) , where p is the conditional distribution of the listener.
Then, the listener’s concatenability is deﬁned as
[Equation]
Following (Lowe et al., 2019), we propose two metrics that capture the understanding of the two agents separately.
We deﬁne u ( e ) =
log q
( e |{ x 1 , x 2 } ) , the log probability (according to the speaker’s conditional distribution q ) of an expression e when the complex meaning is observed.
The speaker’s concatenability is deﬁned similarly as
[Equation]
These metrics are positive iff the concatenation e 1 e 2 (or e 2 e 1 ) is preferred to the actually sent message e 1 , 2 .
For the speaker, this means that the concatenation e 1 e 2 (or e 2 e 1 ) is more likely; for the listener, that e 1 e 2 (or e 2 e 1 ) allows it to recover the information better.
The min operator picks the lowest reconstruction loss among the two possible concatenation orders.
Thus, the method is agnostic to the order of the subexpressions, similarlty to bosdis in Chaabouni et al.
(2020)’s work.
However, we emphasize that these metrics are only examples of what one can do with our framework. For instance, to study the importance of the order of the subexpressions, one could study the gap | l ( e 1 e 2 ) − l ( e 2 e 1 ) | . We believe many more insightful and intuitive metrics can be imagined when one has access to subconstituents e 1 and e 2 .  E XPERIMENTS
The commonalities between the two tasks (inputs, targets and loss functions) are given in Section A.
In all our architectures, attention mechanisms (Bahdanau et al., 2014) plays a crucial role of selecting what information to send next.
In particular, the models used are based on Transformers (Vaswani et al., 2017).
The main difference between the architectures in the two tasks is that the sender attends over objects in the argument structure task, while it attends over properties of the target object in the intersective adjective setup.
Since our experiments on the second task will be published soon as a separate publication, we focus on the ﬁrst task, the discriminative task.
We use synthetic data in order to carefully control the complexity of the meanings, as described in Section B.1.
The sender represents the target object as a set of vectors (one for each property), and produces a message by attending over these vectors.
The properties that are pragmatically relevant to distinguish the target from the distractors are marked with a speciﬁc embedding.
The listener embeds all the objects independently and represent the message in a single vector using a Transformer encoder block.
Then, a probability over objects is given simply by computing the dot product between the message vector and all the objects representations.
This process is detailed in Section B.2.
Once we have trained our runs and ﬁltered out the runs that failed (cf.
Section C)
, we perform a qualitative and quantitative analysis.
To give an intuition about the concatenatibility metrics, we show messages produced by a sender-listener pair with a low C L = − 1 .
19 (model A ) versus one that yields a high C L = − 0 .
(model B ) in the discriminative task.
We encode all utterances from the training set, compute the average C L and compute the most frequent message for each pragmatically relevant features (simple and complex).
We report (non-cherrypicked) examples.
In Table 3, we see that B ’s sender produces messages that are often the concatenation of messages for simple meanings in isolation.
By contrast, A ’s sender produces messages that are related to either or both messages, but the messages seem to be less structured and predictable.
Thus concatenability seems to capture an intuitive deﬁnition of systematicity that 1) is correct regardless of the lengths of the expressions to compose and 2) is invariant with regards to the order of the sub-expressions.
More details are provided in Section D.1.
D ISCUSSION AND CONCLUSION
Meaning complexity variation is a simple and promising functional explanation for the emergence of compositionality.
We have argued that in conjunction with the least-effort principle, it could be a common cause behind the emergence of intersective adjectives and simple argument structure.
This explanation is orthogonal to other characteristics of the tasks (reconstruction vs discriminative tasks) and could operate at different linguistic levels (phrase-level, sentence-level).
In addition to the functional explanation, we obtain a whole class of methods to analyze compositionality (and in particular, systematicity), as well as the ability to apply existing metrics such as those proposed by Chaabouni et al.
We think that it is not a coincidence if meaning complexity variation both explains compositionality and helps us to study it.
The easier languages are to analyze for scientists and machines, the easier they should be acquired by children as well, a reasoning that resonates with Gopnik (1996)
’s theory.
However, this double role makes it harder to verify that our hypothesis is different from two alternative explanations.
The ﬁrst alternative explanation is that what matters is that inputs have the “right” meaning complexity.
For instance, does compositionality emerge when all examples are complex, or when all examples are simple?
The second alternative explanation is that what matters is the variation in the quantity of information (i.e. the non-uniform character of the distribution of the inputs), not the structure or complexity of meanings.
In our setup, the two go hand in hand.
We need to verify such claims as rigorously as possible in the future.
We also need to study whether there is a relationship between generalisation and compositionality.
For instance, Chaabouni et al.
(2020)’s
metrics are not necessary for generalisation.
Is it also the case for concatenability?
Finally, the two tasks could be merged within a single simulation.
The meanings of the argument structure task are made of several entity-role pairs, but the entities themselves are meanings of the adjective task.
By training agents to solve these two tasks at once, one expects adjectives to start appearing within arguments.
More generally, this opens the door to simulations where various parts of speech emerge.
A CKNOWLEDGEMENTS
We thank Pascal Vincent for his work on the preceding paper, NSERC for ﬁnancial support, the
Mila IDT team for the computational infrastructure, and the reviewers for their useful and thorough feedback.
A T ASKS A. M EANING SPACE
We assume that language is evolved and acquired only after other skills are mastered.
In particular, we assume that agents already conceptualize the world in terms of objects with properties.
This seem like a reasonable assumption, given what we know about individuation in infants (Xu & Carey, 1996) and apes (Mendes et al., 2008).
Each meaning contains one or several objects with several properties, which we represent hierarchically using ﬁller-role pairs.
In turn, these discrete representations can be embedded and represented as sets of vectors used by neural architectures.
In theory, such representations could alternatively be obtained directly using perceptual neural architectures which segment the inputs into objects and categorize each object’s representation along several axes, yielding properties
We will abbreviate the reconstruction task by REC and the discriminative task by DIS .
The REC task will be described in depth in another publication, so we omit the details here.
We ﬁrst deﬁne a set of roles R 0 for the properties of objects.
For example, R 0 can be { COLOR , SHAPE , . . . } .
For each role, there is a corresponding set of ﬁllers, for example, the ﬁllers corresponding to COLOR are { BLUE , RED , . . . } .
In DIS , these roles are arbitrary and meaningless but in REC , they have a clear interpretation.
An object o ∈ O is a set of ﬁller-role pairs such as a blue circle, represented as o
= { ( COLOR , BLUE ) , ( SHAPE , CIRCULAR ) } .
In Figure 1, each object is represented by a row of 3 integers, where each column correspond to a role and each integer to a ﬁller for that role.
In both tasks, the inputs contain several objects, which will be indexed by different sets of roles.
In
DIS , we deﬁne R 1 = { 1 , 2 , 3 , . . .
} , which could be interpreted as a position in space and embedded using Cartesian coordinates.
An object of the meaning space X is a set of ﬁller-role pairs, where the ﬁller space is O and the role space is R 1 .
For a meaning x ∈ X , one of the pair ( r, o ) ∈ x is the target .
It is this piece of information that the speaker should communicate to the listener to succeed the task.
In both tasks, the listener observations are called context .
In this task, the context is simply the meaning x .
In REC , the roles R 1 = { A GENT , P ATIENT , M ISC } correspond to θ -roles.
The meaning is partially observed by the listener: the context is c (cid:40) x .
A. S IMULATING PRAGMATIC INFERENCES
We further assume that agents follow the Gricean maxim of quantity (Grice, 1975): agents communicate exactly the information that is needed for solving the task, not more, not less.
If we take the
REC task, there is no need to send the entire meaning to the listener, but the complementary of the context.
In the DIS task, there is a minimal set of pairs that uniquely distinguish the target from the other objects.
Ideally, the agents should be able to perform pragmatic inferences themselves.
Indeed, the maxim of quantity probably stems from a simpler objective trading off communication efﬁciency and the efforts of the agents, as formulated by Zipf (1949).
However, as a ﬁrst step towards this more complete simulation, the pragmatic inference that determine what to send is given to the senders.
In
REC , the objects that are not observed by the listener are indicated to the sender by a speciﬁc ﬂag, which is embedded and added to the object representations.
In DIS , the properties of the target object are embedded and given to the sender, which therefore does not see the distractors.
Moreover, the relevant properties are marked with a speciﬁc ﬂag as well.
We are currently working on architectures which would compute this set of features by observing all the objects.
In Figure 1, we show pragmatic inferences more abstractly as boolean indicators that are observed by the speaker.
Importantly, some least-effort pressure is necessary for the speaker to only send information that is marked as pragmatically relevant.
Without this pressure, if the message space is large enough (if the vocabulary size and the maximum message length are large enough), the sender can encode its entire input.
The pressure encourages the sender to leverage these pragmatic markers.
Moreover, there are many more or less economical ways to send the pragmatically relevant information.
A. L OSS
Let ( x, t ) be a meaning along with a target.
The context c is uniquely determined by ( x, t ) .
The speaker observes ( x, t ) and produces a message e ∼ q ( ·| x, t ) which is then transmitted to the listener.
The loss function is composed of an information transmission term and a least-effort penalty,
[Equation]
Information transmission term: In the discriminative case, the listener produces a conditional probability p over the possible targets T , and the information transmission term to minimize is
[Equation]
In the reconstruction case, the listener produces a conditional probability over the meaning space X , and the information transmission term is
[Equation]
Recall that in REC , x ∈
X is a set.
The distribution over this set is obtained by predicting for each role r ∈ R 1 all the features of all the properties r (cid:48) ∈ R 0 , as well as predicting whether ( o, r ) ∈ x .
That is, p ( x | e, c ) =
(cid:81) r ∈ R 1 p ( o r | r, e, c ) p ( γ r | r, e, c ) where γ r is a binary random variable indicating whether there is a pair in x containing the role r .
Least-effort penalty:
The second term, r ( x, e, t ) , penalizes the speaker for using long messages.
As a ﬁrst approximation, the length of the message is proportional to the energy spent by the speaker to produce the message.
In Chaabouni et al.
(2019), the function r was deﬁned as
[Equation]
where λ is a hyperparameter, and | e | is the number of symbols (except the end-of-sentence symbol) in the message.
However, we noticed that messages collapse to empty messages early on during training.
This is similar to the well-known posterior collapse , where the approximate posteriors of latents of sequenceto-sequence VAEs collapse to their priors (Bowman et al., 2016).
We ﬁx the issue by adapting two well-known tricks: Pelsmaeker & Aziz (2020)’s minimum desired rate and Kingma et al.
(2016)’s free bits.
The penalty term becomes
[Equation]
where 111 is the indicator function.
For this term to be non-zero, two conditions need to be fulﬁlled.
Firstly, the reconstruction error must be below τ , which is analogous to a minimum desired rate.
The interpretation is that the speakers only care about minimizing their efforts once they know how to get their point across.
Secondly, the penalty is above
only if the message contains more than n min symbols.
This gives models n min “free” symbols for each datapoint.
Without this factor, we found that speakers often utter empty messages.
B D ISCRIMINATIVE TASK
In this section, we describe the data used in the DIS task and the architecture of the agents.
B. D ATA
The data is synthetic.
Each role R 0 is also called a property or feature since it characterizes an object and is numbered from 1 to n f .
Each object is composed of all roles from R 0 and a ﬁller for that role which is simply number from { 1 , . . .
, n v } .
Thus each object can be represented easily as a dense vector.
In our experiments, we use n v = 8 and n f = 5 .
The data is generated as follow.
First, we draw the number of objects N from a categorical distribution p k
= P ( N = k ) deﬁned by p 2 =
p 3 = 18 , p 4 = 18 , p 5 = 14 .
Then, the ﬁrst object object o 1 is obtained by sampling a value uniformly from { 1 , . . .
, n v } for each feature.
The next object u 2 is obtained by sampling a feature f 1 ∈ R 0 and resampling a different value.
u 3 is obtained by sampling a different feature f ∈ R 0 { f 1 } and by resampling another value, etc.
This process has an interesting property: the ﬁrst and the last objects are distinguishable from all the other objects by a single feature while the objects in the middle are distinguishable from all the other objects by two features at most.
We call these features the necessary features, denoted by the random variable K .
This is illustrated in Table 2.
Furthermore, in order to create an out-of-distribution ( OoD ) test set, we reject some datapoints.
When K = 2 , we reject datapoints where feature 1 is necessary, and we also reject datapoints where features 2 and 3 are necessary (but feature 2 and 3 can be necessary features along with another feature).
As a result of the distribution of N and the OoD rejection procedure, we obtain
P ( K = 1) ≈ 0 .
We have not yet tested our models on this test sets but intend to do so in a future publication.
[Equation]
{+4 , 5 , 1 , 5 ,] u 
+}
=
{+[ 4 , 5 , 1 , 3 ,] u 
+}
=
[ 4 , 5 , 1 , 3 ,] u 5 = [ 4 , 5 , 6 , 3 ,]
B. M ODEL
In order to use neural models, we represent the inputs x and contexts c as tensors.
In all our architectures, attention mechanisms (Bahdanau et al., 2014) plays a crucial role of selecting what information to send next.
In particular, the models used are based on Transformers (Vaswani et al., 2017).
The main difference between the architectures in the two tasks is that the sender attends over objects in the argument structure setup, while it attends over properties of the target object in the intersective adjective setup.
In DIS , the sender observes a representation of the target object as a set of properties over which it can attend.
The necessary properties (that are pragmatically relevant) are marked.
Formally, for each pair of the target object ( r, f ) corresponds a row V i in the matrix V , deﬁned as
[Equation]
where Role and Val r are learned embedding matrices.
This matrix V is attended over by a Transformer decoder, which predicts log probabilities over the vocabulary auto-regressively, via causal masking.
We use the Gumbel-Softmax straight-through (Jang et al., 2017; Maddison et al., 2016) to backpropagate through the discrete decisions.
The listener is made of two parts.
Firstly, a Transformer encoder processes the message and produces a sentence representation z by linearly transforming the contextualized end-of-sentence embedding.
Secondly, each object o i = { ( r 1 , f 1 ) , . . .
, ( r n , f n ) } is represented vectorially as
[Equation]
where [ · ; · ] denotes the concatenation of vectors, Val (cid:48) r is an embedding matrix speciﬁc to the role r ∈ R 0 and W is a matrix that lowers the dimensionality of the vector from d × n to d .
Finally, the probability of the object i being the target is simply p ( t = i | e, c ) ∝ z T u i .
C E XPERIMENTAL SETUP
We perform a random search (Bergstra & Bengio, 2012) over a variety of hyperparameter values.
The agents of interest use non-empty messages, but also messages that have a maximal size, which would mean that the least-effort pressure is not active and properly tuned.
Thus, we set the maximum length to 8 symbols (excluding the end-of-sentence token) and ﬁlter out runs which on average have less than 1 symbol or more than 6 symbols.
The following values are speciﬁc to the DIS task.
We use the Adam optimizer (Kingma & Ba, 2014) with a learning rate of 0.0003, with β 1 ∼ U ( { 0 . 3 , 0 . 9 } ) , β 2 ∼ U ( { 0 . 9 , 0 . 99 , 0 . 999 } ) and the batch size follows U ( { 32 , 128 } ) .
The training, validation and test set are all drawn from the same distribution and contain 3000, 1000 and 1000 examples each.
We validate every 5 epochs and compute the loss.
If the loss hasn’t improved for 15 validations in a row, we stop the optimisation.
The temperature of the Gumbel-Softmax estimator lies in { 0 . 9 , 1 . 0 , 1 . 2 } .
As for the loss, the number of free symbols is either 0 or 1 , λ ∼ U { 0 . 1 , 0 . 3 , 1 . 0 } , and τ ∼ U { 0 . 1 , 0 . 3 } .
Finally, the vocabulary size is drawn from U ( { 32 , 128 } ) .
We use the EGG framework (Kharitonov et al., 2021) to run our experiments and in particular,
PyTorch (Paszke et al., 2019) for implementing our models.
The dropout parameter is drawn from { 0 . 2 , 0 . 3 } .
There is a shared number of layers in every Transformer used in the sender drawn from { 1 , 2 , 3 } , and similarly for the message decoder in the Listener (the only Transformer in the listener).
D R ESULTS D. C ONCATENABILITY AND QUALITATIVE ANALYSIS
Concatenability metrics are information-theoretic quantities measured in nats.
When C L = − 0 .
05 , on average, the cross-entropy loss incurred by the listener is 0 .
05 higher when it receives the best concatenated messages ( e 1 e 2 or e 2 e 1 ) compared to when the actual message e 1 , 2 is received.
The loss of information of 0 .
05 appears very small, when considered in comparison with the loss incurred by a random baseline in a discriminative task with uniform distribution over two objects that are the targets with equal probability, log 2 ≈ 0 .
On the other hand, 1 . 19 is very large for a task with at most 4 distractors, since it is above log 3 = 1 . 09 .
C S is in general a much bigger quantity, since it is deﬁned relatively to log probabilities of messages, which live in a much bigger space.
Moreover, C S and C L are very correlated (Spearman coefﬁcient of 0 . 63 ).
This is not very surprising since the sender and the listener are trained jointly.
B also has the highest C S = − 5 .
among the runs, meaning that the concatenated messages (obtained systematically) have a probability relatively close to the actually sent messages.
Meaning
A
B − , 1 , − , − , −  − , − , − , 1 , − 24 , 24 , − , 1 , − , 1 , − 24 , 3 ,13 ,− , 1 , − , − , −  − , − , − , 2 , −  − , 1 , − , 2 , − 6 ,− , 1 , − , − , −  − , − , − , − ,  − , 1 , − , − ,  − , − , − , 1 , − 24 , 24 , − , − , − , − ,  − , − , − , 1 ,24 , 27 ,13 , 12 ,− , − , − , 1 , − 24 , 24 , − , − , − , − ,31 , − , − , − , 1 ,11 , 24 , 30 ,6 ,− , − , − , 2 , −  − , − , − , − ,  − , − , − , 2 , 22 ,− , − , − , 2 , −  − , − , − , − ,31 , − , − , − , 2 ,11 , − , 1 , − , − , −  − , − , − , − ,31 , − , 1 , − , − ,31 , 3 ,6 ,