The Emergence of Argument Structure in Artificial Languages (2022)

Abstract: Computational approaches to the study of language emergence can help us understand how natural languages are shaped by cognitive and sociocultural factors. Previous work focused on tasks where agents refer to a single entity. In contrast, we study how agents predicate, that is, how they express that some relation holds between several entities. We introduce a setup where agents talk about a variable number of entities that can be partially observed by the listener. In the presence of a least-effort pressure, they tend to discuss only entities that are not observed by the listener. Thus we can obtain artificial phrases that denote a single entity, as well as artificial sentences that denote several entities. In natural languages, if we ignore the verb, phrases are usually concatenated, either in a specific order or by adding case markers to form sentences. Our setup allows us to quantify how much this holds in emergent languages using a metric we call concatenability. We also measure transitivity, which quantifies the importance of word order. We demonstrate the usefulness of this new setup and metrics for studying factors that influence argument structure. We compare agents having access to input representations structured into pre-segmented objects with properties, versus unstructured representations. Our results indicate that the awareness of object structure yields a more natural sentence organization.

## model and objective we present two transformer-based @xcite variants of the model of the agents: one that encodes an object-centric bias and one that does not. before delving into their differences, let us describe their common features. ## general experimental setup in the next sections, we review various properties of natural languages, and introduce metrics to quantify these in artificial languages and compare the effect of using oc versus fa on these metrics. the training set contains 60% of the data, the validation set 10%, and the test set the rest. we denote the entire data set by d and denote by d k the subsets of d composed of examples for which i α @xmath0, that is, the examples where k objects are hidden. all the experiments use the egg framework @xcite based on the pytorch library @xcite . 3 the neural agents are trained using adam @xcite . there is a large number of hyperparameters so we resort to random search @xcite . 4 our hyperparameter search is deliberately broad since we do not know a priori which hyperparameter choices are realistic. we expect to obtain results with high-variance, but a major advantage is that we get more robust conclusions by averaging over unknowns. we perform linear regressions to predict the value of each metric given a binary variable indicating whether oc is used. when the coefficient for this variable is significantly different from 0 according to a t-test, then oc has a significant effect. 5 additionally, we consider that the entropy of the messages is a mediator that we control for. for instance, the reconstruction error is indirectly influenced by the vocabulary size and the sampling temperature via the entropy. however, if we observe that oc improves the generalization error, we want to exclude the possibility that this is because oc agents send messages with higher entropy than fa agents, since it should be trivial to also increase the entropy of the fa models by modifying hyperparameters. we discard models with messages of average length below 1 and above 6. indeed, when the average length is too small, many messages are empty, and when it is too long, artificial sentences are barely or not longer than artificial phrases. these cases are considered a priori unnatural. this leaves us with 100 out of 136 runs. 3 the proto-role dataset is available here: http:// decomp.io/projects/semantic-proto-roles/ . the code (including on-the-fly preprocessing of the dataset) is available at https://github.com/tombosc /egg_f/tree/r1/egg/zoo/vd_reco . note that the length penalty works as expected. without the penalty, the messages all contain the maximum number of symbols. with the penalty, the average message length grows as the speaker needs to send more and more information (on d 1 : 4.19, d 2 : 5.24, d 3 : 5.89). ## generalization performance natural languages are often said to be productive and systematic: there is an infinity of utterances which we can understand without having encountered them before (productivity), in particular when we understand constituents of the novel sentence in other contexts (systematicity) (szabó, 2020). do emergent languages exhibit these characteristics? in this section, we study such generalization abilities. we measure how well the listener can reconstruct the inputs when the sender communicates about datapoints unseen at train time. firstly, we test our models in distribution. secondly, we test our models out of distribution (ood), following @xcite . we compute the empirical marginal distributions over the number of hidden entities, the entities, the roles, and the relations. then, the ood test set is sampled from these marginals independently. we measure the reconstruction losses on subsets where 1, 2, and 3 entities are hidden for a finer-grained analysis. results: however, the performance difference between oc and fa does not tell us much: both oc and fa agents could exchange messages that are structured in very unnatural manners. in the next two sections, we introduce metrics to shed light on how the information about different entities is packaged into a single message. ## concatenability in natural languages, the verb encodes the relation while arguments refer to entities, but roles do not have direct equivalents in all languages. they are encoded using three strategies, typically using a mix of strategies within a single language. in analytic languages like english or chinese, roles are encoded in word order and possibly using adpositions or clitics, but the role does not change the form of the arguments. for example, in sentences (1a) and (1b), the arguments are identical but are placed in a reverse order, so that their roles are inverted, too: in more synthetic languages like russian or turkish, case markings code for the roles. in russian, these markings are suffixes on nouns, adjectives, and so forth, as can be seen in ( 2a ) and ( 2b): (2) finally, in polysynthetic languages (caucasian languages, samoan languages, etc.), arguments typically look like those in analytic languages, but the roles are encoded using markers on the verb. 6 since, in this work, relations are not communicated by agents, there is no artificial equivalent of the verb. therefore, this strategy cannot emerge and we consider it no further. crucially, simple sentences are obtained by concatenating a verb and one or several noun phrases that refer to entities, whether word order matters or word order does not matter and cases are marked. for a single event, by varying what information is available to the listener through the mask α, we get messages describing two entities in isolation (phrases) as well as messages describing two entities at once (sentences). for example, consider (i s , (1, 1, 0), r s , β) drawn from d 2 , the subset of the data with two hidden objects. let g be the function that transforms this speaker's inputs into a message via greedily decoding, and define we obtain the messages sent when l observes the first or the second object in isolation as we define concatenated messages to be m 12 = m 1 ⊕ m 2 and m 21 = m 2 ⊕ m 1 , where ⊕ is the concatenation operator. this is shown in figure 1 . we define p 2 as the empirical distribution on the subset of d 2 such that neither m 1 or m 2 are empty messages, implying that m 12 = m 21 . as argued above, in natural languages, m 12 or m 21 (or both, if word order is irrelevant) should convey information at least as well as m * . denote by l(m) the reconstruction loss incurred by l if l had received the message m, that is, l(m) =log p(i s |i l , m, β). then, concatenability from the listener's point of view is defined as when close to 0, on average, one of the two concatenated messages (or both) is as informative as the message actually uttered by the speaker for reconstructing the inputs. l can correctly reconstruct s's inputs from a concatenated message that s is unlikely to utter. inversely, a concatenated utterance can be highly likely for s even if l might fail to reconstruct s's input from it. therefore, there are actually two symmetrical measures of concatenability, one from the point of view of s and the other from the point of view of l. a similar proposition was made by @xcite in the context of interactive games. they have shown the usefulness of distinguishing these two points of view. the metric is defined similarly on the speaker's side with a slight subtlety. since sampled messages have a maximum message length of n l , the probability of a sequence longer than n l is 0. however, concatenated messages are sometimes longer than n l . we define q ∞ as the distribution generated by s without the constraint that probable sequences have length below n l . we denote the conditional log-probability of a message given a certain input by u(m) = log q ∞ (m|i s , α, β). then, concatenability from the speaker's point of view is defined as it is close to 0 when, on average, one concatenation of the two messages (or both) has roughly the same probability as the actual message. to give an intuition, let us go back to our examples. take the speaker of an hypothetical language, english without verbs. suppose that this speaker, when exposed to a given input x @xmath3, (1, 1, 0), r s , β), produces a sentence m * corresponding to (1a), ''the old lady the dog''. by exposing the speaker to the same input, but by changing the mask to (1, 0, 0), they produce m 1 = ''the lady'', while using the mask (0, 1, 0), they produce m 2 = ''a golden retriever''. c s compares the log probability of m * with that of m 12 = ''the lady a golden retriever'' and m 21 = ''a golden retriever the lady'', whichever is more probable. since english without verbs is rather concatenable, the speaker judges that m 12 is roughly as likely as m * given the inputs. thus, the value inside the expectation of c s will be high, close to 0. now, take an identical speaker, except that they assign a very high probability to m 1 = ''a shoebox'', while the new m 12 and m 21 are unlikely conditioned on x s . then c s will be low and negative. perhaps (i) ''a shoebox'' has different semantics when it is used alone in a sentence, as compared to when it is used with a second referent; or perhaps (ii) ''a shoebox'' is never used with another referent in a sentence, and the speaker would use ''a lady'' instead. in any case, concatenability for this speaker would be low, which corresponds to the intuition that their language is unnatural and unsystematic. 7 7 this example only illustrates the intuition. in reality, it is not straightforward to apply these metrics on natural language, because they require probability distributions for the agents. we could learn models that map back and forth between the semantics and the ground-truth utterances, but c l ↑ c s ↑ fa -6.1 ± 3.8 -29 ± 13 oc -3.2 ± 2.4 * * * -26 ± 15 the same illustration holds for c l , and it can be adapted to show why c s and c l should also be high for more synthetic languages. results: we measure these metrics on the test set. in our experiments, they always take negative values: the concatenated messages are on average worse than the actual messages. some models yield values close to 0, but this depends on the choice of hyperparameters. first, note that it is difficult to detect the presence of case markings directly. even for the simplest forms of morphology, we are hindered by the segmentation problem of identifying the root and the affix, as mentioned in section 1.3. 8 the models would add some bias. moreover, we only have ground-truth utterances for english and any attempts to use machine translation would add some more bias. yet we can quantify on average how much referential phrases (messages about a single hidden object) encode roles. we train a bigram classifier on the training set and measure its test error, the role prediction error (rpe). if there are case markings, this error will be low (but the opposite is not true). moreover, we introduce two transitivity metrics, to directly measure the importance of word order. t s is defined as: this metric is 0 if the two concatenated messages are equally probable for s; and it is large if one word order is much more likely than the other for s. similarly, t l is defined as these metrics are only interpretable when concatenability metrics are high enough, so we measured t s only for runs where c s is above the median and similarly for t l . results: as can be seen on figure 3 , when transitivity is low and rp e is high, the reconstruction loss is poor (top-left corner), because there is no efficient strategy to encode roles. there is a lot of variance both for oc and fa, but oc models tend to have higher transitivity, both on average and in terms of maximal values. thus word order is more important for oc runs than for fa runs. this is also confirmed by ## qualitative analysis one can gain intuition about the metrics by looking at messages exchanged by agents. in particular, we compare two models a and b which both have relatively high concatenability, but a has high transitivity scores whereas those of b are low. the chosen models also have relatively close reconstruction loss, so that the messages convey roughly as much information about their inputs. to simplify, we focus on one entity vector and see how it is transmitted when it is in different roles and in different contexts. since feature vectors are slightly sparse (with many na values), vectors which have many nas are sometimes not conveyed at all (the penalty makes it costly to do so). we search for an entity that appears in many different roles and that is sufficiently not sparse. the 8th most frequent vector (u 8 ) is the most frequent vector that fits these criteria. first, let us examine the left-hand side of despite using a large vocabulary of 128 symbols, only a few symbols are used. this is due to the difficulty of discrete optimization. we were puzzled to find so many common symbols in the two models, but it turns out that the selected models have the same hyperparameters except for the length-penalty coefficient (a: λ = 1, b: λ = 10). each last row of each block of three lines shows an artificial sentence, where two entities are hidden. we can see that most symbols in these sentences also frequently appear in phrases that denote individual entities (identified by their colors). some symbols from phrases are omitted or in a different order in the sentence, but the precise structure of these phrases is out of scope for our work. a is more consistent in its use of word order than b: a almost always refers to misc before patient, whereas the order varies for b. this is evidence that the transitivity metrics correctly measure the importance of word order, at least when concatenability is high enough. on the right-hand side of ## conclusion we have presented an experimental setup for studying how probabilistic artificial agents predicate, that is, how they convey that a relation holds between entities. in our daily lives, events are partially observed and predication is used to share information about what is not observed, often in a parsimonious manner. our task and loss realistically reflect this function of language. at the same time, this setup allows us to directly study argument structure while ignoring the internal structure of phrases. indeed, we can easily obtain artificial phrases, that is, utterances that refer to single entities, as well as artificial sentences, utterances which express the relation holding between different entities. then, we can study whether and how artificial phrases are systematically composed to form artificial sentences, via our concatenability and transitivity metrics. thus we completely sidestep the need to segment artificial sentences into phrases, a complicated problem that is unfortunately ignored in previous works. more precisely, we have argued that all natural languages should have high concatenability, while transitivity is not necessarily high and merely quantifies the importance of word order. equipped with this setup and these metrics, we have compared a cognitively plausible architecture that leverages the structure of the inputs into objects with properties (oc) against an implausible baseline that ignores this structure (fa). object-centric models yield more natural languages in terms of concatenability, while also relying more on word order. moreover, they generalize better than their implausible counterparts, both in distribution and out of distribution. these results confirm the importance of the input representations and of the architectures leading to the discretization bottleneck, also reported by @xcite and @xcite . in our experiments, discrete low-dimensional inputs were processed by task-specific architectures. however, we believe that one can use high-dimensional representations obtained from pretrained models, as long as these representations are prelinguistic, as object-centric representations seem to be. our methods could be extended to investigate other aspects of sentences. for instance, how would agents convey relations? to answer this question, we could use the representations learned via relational unsupervised learning algorithms as inputs. we could study how different relations are discretized into one or several symbols, perhaps the equivalent of verbs and adverbs. we could also analyze how relation-specific roles cluster in abstract roles (like θ-roles) and structure grammar.