L INKING E MERGENT AND N ATURAL L ANGUAGES VIA C ORPUS T RANSFER
Anonymous authors
Paper under double-blind review
A BSTRACT
The study of language emergence aims to understand how human languages are shaped by perceptual grounding and communicative intent.
Computational approaches to emergent communication (EC) predominantly consider referential games in limited domains and analyze the learned protocol within the game framework.
As a result, it remains unclear how the emergent languages 1 from these settings connect to natural languages or provide beneﬁts in real-world language processing tasks, where statistical models trained on large text corpora dominate.
In this work, we propose a novel way to establish such a link by corpus transfer , i.e. pretraining on a corpus of emergent language for downstream natural language tasks, which is in contrast to prior work that directly transfers speaker and listener parameters.
Our approach showcases non-trivial transfer beneﬁts for two different tasks – language modeling and image captioning.
For example, in a low-resource setup (modeling 2 million natural language tokens), pre-training on an emergent language corpus with just 2 million tokens reduces model perplexity by 24.6% on average across ten natural languages.
We also introduce a novel metric to predict the transferability of an emergent language by translating emergent messages to natural language captions grounded on the same images.
We ﬁnd that our translation-based metric highly correlates with the downstream performance on modeling natural languages (for instance ρ = 0 .
on Hebrew), while topographic similarity, a popular metric in previous work, shows surprisingly low correlation ( ρ = 0 . 003 ), hinting that simple properties like attribute disentanglement from synthetic domains might not capture the full complexities of natural language.
Our ﬁndings also indicate potential beneﬁts of moving language emergence forward with natural language resources and models.
I NTRODUCTION
The remarkable recent progress in NLP (Devlin et al., 2018; Radford et al., 2019; Brown et al., 2020) beneﬁts from huge text corpora, on which powerful models learn rich statistical patterns tabula rasa and adapt to downstream tasks.
But in stark contrast, human language acquisition (De Villiers et al., 1978) does not start with passive and complex corpora like Wikipedia.
Instead, children learn language through a cumulative series of interactions with other people (Bruner, 1985; Barton, 1994) grounded in the physical and social world (Harnad, 1990; Vogt, 2002; Alomari et al., 2017).
Incorporating such cognitive insights with recent machine learning advances holds great promises towards more functional (Wittgenstein, 1958; Wagner et al., 2003) and generalizable (Lake et al., 2017) language agents.
The study of emergent communication (EC) (Cangelosi & Parisi, 2002; Kirby, 2002; Wagner et al., 2003; Lazaridou & Baroni, 2020) is one promising direction toward this motivation, where a communication protocol is shaped through multi-agent interactions with perceptual grounding.
A typical setup is the image referential game (Figure 1(a)), where a speaker generates a discrete sequence of tokens based on an input image, a listener is challenged to select the input out of distractors based on the message, and both networks are optimized jointly via game success signals.
By studying these games, researchers are interested in the emergence of desirable properties resembling natural
[Figure] down
[Figure]
view
[Figure]
skyline
A
[Figure]
from
[Figure] [Figure]
Captioned
[Figure]
[Figure]
Pre
[Figure]
train
LM
These
[Figure]
Fine
[Figure]
from LM
Pre
[Figure]
train 13 34
[Figure]
IC
Fine
[Figure]
A view of skyline IC 13 87 213
[Figure] [Figure] [Figure]
Speaker
[Figure]
language, such as game success generalization (Kharitonov & Baroni, 2020; Lazaridou & Baroni, 2020) and compositionality (Smith et al., 2003; Andreas & Klein, 2017; Kirby et al., 2015; Lazaridou et al., 2018; Li et al., 2020b).
However, these properties are mostly deﬁned and analyzed within each individual game framework.
For example, generalization is usually measured as game success with novel inputs, but it remains unknown how the emergent coding can be useful beyond the particular game (e.g. useful for natural language tasks).
Moreover, compositionality is often approximated by attribute-wise disentanglement in simple environments, which could be drastically different from the structural properties of natural language (Dankers et al., 2021).
Such gaps pose difﬁculty in objectively understanding the emergent properties with respect to the full complexities of natural language, and assessing the universality of various claims.
In this work, we aim to address these issues by linking emergent languages with natural languages.
Concretely, we investigate if modeling an emergent language provides transferable beneﬁts for downstream natural language tasks — language modeling (Figure 1(b)), image captioning (Figure 1(c)) — and if downstream performances in turn allow us to understand and analyze properties of emergent language beyond the EC game and toward natural language. The key technique is corpus transfer , i.e. by pre-training a model on a corpus of emergent messages produced by a trained emergent speaker, and ﬁne-tuning the model on downstream tasks with natural language data.
This approach integrates inspirations from prior work like Papadimitriou & Jurafsky (2020), which proposes the transfer scheme from synthetic to natural corpora, and Li et al.
(2020b), which aims to improve few-shot translation by transferring the trained EC speaker and listener model weights.
Through a series of experiments, we ﬁnd that corpus transfer is helpful when the downstream natural language resource is limited.
For example, in a low-resource setup of modeling two million natural language tokens, such a transfer scheme reduces the test perplexity by 24 .
6% on average versus training from scratch, across ten different downstream languages.
We also establish the non-triviality of such a transfer performance by comparing to other synthetic and natural source corpora, as well as multiple ablation studies on the EC and downstream transfer setups to help understand the transferability of emergent language.
Notably, our method of corpus transfer signiﬁcantly outperforms directly transferring the trained emergent speaker model (Li et al., 2020b), demonstrating that modeling the emergent language could yield greater usefulness than directly transferring the EC agents.
While our transfer experiments demonstrate that emergent languages can be leveraged and understood outside the game and towards real-world tasks, the pre-training and ﬁne-tuning pipeline is computationally expensive as a scalable evaluation scheme for a population of emergent languages.
Instead, we are interested in developing a simpler metric to predict the transferability of emergent language, which pertains to properties of natural language and may challenge metrics deﬁned within the game framework (e.g. game accuracy).
Therefore, we propose a novel metric for emergent languages based on how easily it can be translated to the corresponding natural language grounded on the same perceptual input (Figure 1(d)).
Concretely, we employ the trained speaker to produce emergent messages based on a small set of unseen images with natural language captions, and train a translation model on paired emergent and natural captions.
We show that the translation score (such as ROUGE L (Lin, 2004)) can better predict (e.g. Pearson correlation ρ = 0 . 83 for modeling Hebrew) the transfer beneﬁts of different emergent languages than two commonly adopted metrics in prior literature – game success with novel inputs ( ρ = 0 . 74 ), and topographic similarity (Brighton & Kirby, 2006; Lazaridou et al., 2018)
( ρ = 0 . 003 ).
These results call for a re-thinking of what properties we should evaluate for an emergent language corpus, and how we should evaluate them.
For example, game accuracy may be confounded by the listener performance, and topographic similarity, a metric of attribute disentanglement, can be ﬂawed for measuring the compositional structure of real language (Kirby, 2001; Goldberg, 2015; Steinert-Threlkeld, 2020).
In summary, our work takes a step toward bridging the gap between emergent and natural languages.
We believe that shifting from synthetic setups and in-game evaluations to leveraging downstream natural language tasks and state-of-the-art language models could potentially lead to more general insights and greater practical impact for EC and NLP research.  R ELATED W ORK Emergent Communication and Evaluation
A large body of prior work qualitatively analyzes the evolved message structures when the EC games are set up with synthetic environments (Mordatch &
Abbeel, 2018; Lazaridou et al., 2018) or even with real images (Havrylov & Titov, 2017).
Alternative evaluations and examinations include ease of teaching (Li & Bowling, 2019) and symbol usage purity (Lazaridou et al., 2016).
While these schemes align the emergent language with game success or different aspects of game input, our work evaluates the emergent language outside the game framework, instead via the corpus transfer performance to natural language tasks and the translation performance to natural language. Linking Synthetic and Natural Language Papadimitriou & Jurafsky (2020) ﬁnd that recurrent networks pretrained on non-linguistic corpora (e.g. music, code, regular languages) facilitate natural language modeling. In an opposite direction, Lu et al. (2021) observe that language pre-training improves Transformer’s performance with several non-language tasks (logical, vision, protein) even when most weights are frozen, indicating that learning language structure could yield generally useful computational subroutines.
Andreas et al.
(2017) propose to translate a differentiable communication channel in deep multi-agent policies to natural language by collecting human-human communication in the same games.
However, their approach focused on interpreting a continuous channel, while we focus on evaluating emergent messages with discrete structures.
Use of Emergent Communication beyond games There have been efforts to incorporate communication signals and natural language supervision to better communicate (Lowe et al., 2020), avoid language drift (Lazaridou et al., 2020), or to improve NLP tasks like vision-language navigation (Fried et al., 2018), translation (Lee et al., 2018), and image captioning (Havrylov & Titov, 2017).
Different from these setups where the speaker is also grounded on natural language annotations, Li et al.
(2020b) propose to separate the EC game, where no natural language is involved, and ﬁne-tuning on a downstream translation task with limited natural language data.
Despite being the closest work to ours, such a model transfer approach is fundamentally different from our corpus transfer in several ways.
First, while Li et al. (2020b) mainly aim to improve downstream tasks, we envision transfer as a novel means to evaluating and analyzing emergent languages.
Also, the emergent language is a representation with properties beyond the parameters of speciﬁc EC models (Section 3.1).
Last but not least, corpus transfer has several practical advantages over model transfer (Section 4.3).
As shown in Figure 1(a), we consider a typical speaker-listener referential game (Lazaridou et al., 2016; 2018; Li & Bowling, 2019) on a set of N image features
D I
=
{ I 1 , · · · , I N } .
At each training step, the speaker takes an input image feature
I i and generates a discrete message M i ∈
[ V ] T using the Gumbel-Softmax trick (Jang et al., 2016), where V is the vocabulary size and T is the sequence length limit.
For simplicity denote m =
M i so that m t =
M i,t denotes the t -th token of the message
M i .
Then
[Equation]
Here hs t denotes speaker hidden states.
Next, the listener takes the message m , and tries to guess the right image
I i out of a set of K confounding images C
i =
{ I j 1 , · · · , I j K } ⊂ D I − { I i } .
To do so, it uses another GRU layer to turn the message m into a hidden vector hl T
[Equation]
Based on hl T , the listener assigns a score for each candidate image based on inverse square error (Lee et al., 2018), then selects the image by Softmax sampling across the scores.
The speaker and listener are jointly optimized by minimizing the cross-entropy loss of image selection:
[Equation]
After the game is trained, the speaker can be employed to sample a corpus of emergent language
D M = { M 1 , · · · , M N } based on input images.
Though the emergent language is generated by a particular EC speaker, the representation could be potentially learned and used by new agents, with more general properties beyond the parameters of a particular speaker architecture.
Thus, our work focuses on evaluating and analyzing the properties of the emergent language (e.g. corpus transfer) rather than the emergent communication models (e.
g. model transfer)
3. E VALUATING E MERGENT L ANGUAGE
A central question in emergent communication is how to evaluate the quality of the emergent corpus
D M .
Two commonly used metrics in prior work are game accuracy and topographic similarity.
Game Accuracy with Novel Objects The most straightforward metric is the accuracy of playing the referential game E I i ,C i , M i [ 1 guess =
I i ] .
Prior work (Cogswell et al., 2019; Li & Bowling, 2019) usually considers how well the speaker and listener generalize to novel inputs, and ﬁnds that such a generalization ability might not correlate with other properties such as compositionality (Chaabouni et al., 2020; Kharitonov & Baroni, 2020).
Note that this metric involves both the speaker and listener, while the emergent language corpus D M is generated solely by the speaker.
Topographic Similarity
The topographic similarity (Brighton & Kirby, 2006; Lazaridou et al., 2018) measures how messages align with the meaning representations.
More concretely, let CS ij = − I i · I j /
( || I
i ||
2 ||
I
j || 2 ) be the negative cosine similarity of image features
I i and I j , and ED ij be the Levenshtein distance (Levenshtein et al., 1966) between messages M i and M j .
The metric is deﬁned as the Spearman rank correlation of the two distance metrics:
[Equation]
Here R ( · ) denotes the rank of the element in the list.
This metric intuitively measures disentanglement : different attributes of the input should be expressed in different token positions, and each token only describes one attribute (e.g. “big red cube” and “big blue cube”).
Such a property has been claimed (Lazaridou et al., 2018; Li & Bowling, 2019) as connected to compositionality , a key structural property of natural language.
However, this metric is too rigid in its deﬁnition of compositionality, ignoring aspects like argument structure, context or morphology which play a key role in determining the combination of word semantics (Goldberg, 2015).
We note that experiments around both these metrics are within the game framework – accuracy is based on game success , while topographic similarity measures how emergent messages align with game input representations .
This leaves two questions unsolved: (i) can emergent languages be used outside the game?
(ii) if so, would these metrics predict the usefulness of emergent languages for downstream tasks?
We tackle these two questions in Sections 4 and 5 respectively.
E MERGENT C ORPUS T RANSFER FOR N ATURAL L ANGUAGE T ASKS
In this section, we present initial evidence that a corpus of emergent language D M can beneﬁt two types of downstream tasks involving natural language: language-only task (language modeling,
Section 4.1) and vision-language task (image captioning, Section 4.2).
Moreover, we show such a beneﬁt is most signiﬁcant when the natural language resources are limited, which indicates that emergent language is potentially valuable for low-resource languages and tasks.
We then analyze what contributes to such a transfer via ablation studies in Section 4.3.
4. L ANGUAGE M ODELING
Inspired by Papadimitriou & Jurafsky (2020), we consider the task of natural language modeling, and explore if pretraining a language model on a corpus of emergent messages can reduce the downstream perplexity in a low-resource setup (Figure 1(b)), and how emergent language compares to other sources of synthetic and even natural language corpora.
Pre-training Corpora We compare three source corpora:
(i) a Spanish Wikipedia corpus ( es ) culled to 50,000 vocabulary size using the pre-processing script from Papadimitriou & Jurafsky (2020), (ii) a regular language of well-balanced brackets ( paren-zipf ) with the Zipf unigram distribution as es , and (iii) an emergent language corpus ( ec ).
paren-zipf is the best-performing source corpus from Papadimitriou & Jurafsky (2020) not created by humans (e.g. natural language, music, or code) with the inductive bias of hierarchical structure , and es should set an upper bound of the transfer performance for both paren-zipf and ec .
We vary the source corpus size from 2, 5, 10, 15 tomillion tokens to understand how the transfer results change with respect to the pre-training scale.
Fine-tuning Corpora
We scrape Wikipedia corpora of 10 languages to test downstream transfer: Danish ( da , IE-Germanic), Basque ( eu , Basque), Japanese ( ja , Japanese), Romanian ( ro ,
IE-Romance), Finnish ( ﬁ , Uralic), Indonesian ( id , Austronesian), Kazakh ( kk , Turkic), Hebrew ( he ,
Afro-Asiatic), Urdu ( ur , IE-Indic), and Persian ( fa , IE-Iranian).
They come from diverse linguistic families with different levels of resource richness.
Each corpus has 2 million tokens and a vocabulary size of 50,000, and is pre-processed (to remove noise) via the same script to process es .
Implementation We train the EC game and generate ec based on the Conceptual Captions dataset (Sharma et al., 2018), using more than 2.8 million natural images in the wild.
Note that we do not use the natural language captions in the dataset for EC game training.
We take the 512-dim pre-trained ResNet-18 (He et al., 2016) features before the classiﬁcation head as input image features
I i .
Other architecture and training details mainly follow Li et al.
(2020b), and by default V = 4035 ,
T = 15 , K = 256 .
For language modeling, we adopt a Transformer (Vaswani et al., 2017) withdecoder layers and 6 attention heads, and pre-train on each source corpus for 3,000 steps with batch size 32, input length 1,000, and learning rate 5 × 10 − 4 .
For ﬁne-tuning and training from scratch on downstream corpora, the batch size is 8 and learning rate is 10 − 4 .
Hyperparameters are chosen after grid search.
We report the test perplexity at the best validation loss.
Results As can be seen from Figure 2, pre-training on ec (blue) signiﬁcantly reduces the perplexity compared with training from scratch (red, dotted line with constant value), with an average reduction of 24 . 6% with merely 2 million pre-training tokens.
This is a positive signal that a corpus of emergent language could be useful beyond the communication game itself .
Comparing two synthetic pre-training corpora, ec performs better than or comparable to paren-zipf throughout different source sizes and downstream languages, with the exception of Basque and
Finnish when pre-training size is more than 15 million.
Given the strong inductive bias in paren-zipf (hierarchical structure and unigram information from real language), and the fact that ec is generated by just a single-layer GRU speaker without such explicit bias, such a result hints that ec may possess non-trivial structural properties due to the communication signals and perceptual grounding.
On the other hand, the natural language corpus es usually achieve a lower perplexity than both ec and paren-zipf as may be expected.
Most surprisingly, when pre-training only on 2M tokens, ec consistently beats es in terms of transfer performance, and their performances are comparable on most languages using 5 million source tokens. To offer one plausible explanation, ec could contain [Figure] Metrics coco ( 5k ) coco ( 50k ) coco ( full ) base +
[Table] +nl base +ec +nl base +ec +nl BLEU [Table]
[Table]
ROUGE
[Table]
[Table]
[Table]
[Figure]
CIDEr
[Table]
[Table]
[Table]
[Figure]
properties closer to everyday conversations – smaller vocabulary, simpler structures (generated by onelayer GRU) and repetitive (all about image references) – easier for language modeling, compared to the complicated structures and statistical patterns of large text corpora like Wikipedia es .
However, the relative simplicity of ec also prevents the transfer from scaling as well as es - comparing pretraining on 30 versus 2 million tokens, ec only reduces perplexity by 2 . 4% on average, while es reduces 23 . 8% .
Such a mixed ﬁnding points to potential in combining the strengths of simpler emergent corpora (easier to acquire and beneﬁt from) and noisy natural language corpora (more structures and complexities) for more efﬁcient language learning. Such an idea of mixing NLP and EC training has been useful for some NLP (Fried et al., 2018; Lee et al., 2018) and EC (Lowe et al., 2020;
Lazaridou et al., 2020) tasks. Another important direction would be on how to evolve emergent languages with more complexities that resemble and transfer to natural language, where existing approaches to improving and regularizing EC (Mu & Goodman, 2021; Guo et al., 2019; Li & Bowling, 2019; Dess ` ı et al., 2021; Luna et al., 2020) could be tested through our transfer scheme.
4. I MAGE C APTIONING
The language modeling transfer experiment leverages the structural properties of the source corpus, but has little to do with the semantics of the source corpus.
For example, a regular language like paren-zipf has no real meanings, but still reduces downstream perplexity due to its hierarchical structure.
Therefore, we also consider transfer to image captioning, a classical vision-language task where a model has to ground meaning of the words to the images.
As shown in Figure 1(c), we pre-train an image captioning model on unlabeled images to generate EC captions, then ﬁne-tune on downstream annotated images to generate natural language captions.
We note that such a transfer scheme cannot work for other synthetic corpora such as music, code, or paren-zipf as they are not grounded on perceptual stimuli.
Pre-training Data We use the EC speaker trained on Conceptual Captions to generate an EC caption M i based on each ResNet-18 image feature
I i in the dataset.
As image captioning requires ﬁne-grained information beyond a global feature, we extract detection features I (cid:48)
i for each image via pre-trained Faster R-CNN (Ren et al., 2015) and ResNet-101 (He et al., 2016), and pre-train an image captioning model to generate EC captions based on detection features ( I (cid:48)
i (cid:55)→ M i ).
To set up an upper [Table] [Table] bound for transfer performance, we also use the English captions NL i in the Conceptual Captions dataset for pre-training ( I (cid:48) i (cid:55)→ NL i ).
Fine-tuning Data
We use the MS-COCO dataset (Lin et al., 2014) for ﬁne-tuning, which consists of around 500,000 pairs of images and English captions.
We use the full training set, or a subset with 5,000 or 50,000 samples to study the transfer beneﬁt when natural language annotation is limited.
Implementation We use a Transformer with 3 encoder layers and 6 decoder layers as the image captioning model, and perform pre-training and ﬁne-tuning in a seq2seq framework (Ott et al., 2019).
Notably, we only transfer the encoder weights, and ﬁne-tune the whole network end-to-end with a learning rate of 3 × 10 − 4 , as we notice transferring the full weights leads to worse performance.
We report BLEU4 (Papineni et al., 2002), ROUGE L (Lin, 2004), and CIDEr (Vedantam et al., 2015) scores for the test split based on best validation loss.
Results As shown in Table 1, when ﬁne-tuning on 5,000 or 50,000 samples, pre-training on either
EC or NL captions both signiﬁcantly improve the image captioning performance.
For example, with 50,000 samples, EC pre-training can improve the BLEU-4 score by +0 . 9 points, while NL pre-training further improves +1 .
0 points.
The fact that EC pre-training can provide half the beneﬁt of NL pre-training is surprising, because the former requires no language annotation - both training the EC games and generating the EC captions can be done on unlabeled images in the wild, while the
NL pretraining leverages more than 2.8 million English sentences.
However, when the full MS-COCO dataset is used, even pretraining on 2.8 million English captions is not signiﬁcantly helpful, possibly because Conceptual Captions collect noisy Internet sentences while MS-COCO annotates captions with less diversity (Wang et al., 2020) so a model trained from scratch on the full dataset can capture most of the output patterns.
Other vision-language tasks where natural language annotations are harder or more limited (e.g. instruction following, navigation) would potentially beneﬁt more from EC pre-training, where we can turn abundant unlabeled visual stimuli into annotations in the emergent language.
4. W HAT C ONTRIBUTES TO S UCCESSFUL C ORPUS T RANSFER ?
We have shown that emergent languages can provide transfer beneﬁt for natural language tasks, but it is still unknown what properties of emergent languages contribute to such a beneﬁt.
Hence we ablate several key aspects of the EC game training and EC corpus generation, and examine the effect via transfer (with 15 million pre-training tokens) to two example languages, Romanian ( ro ) and Hebrew ( he ).
We leave ablation studies on the image captioning experiment in Section A.3.
Communication and Perceptual Stimuli in EC Game
The emergent language stems from two elements in the game that human language acquisition relies on: perceptual grounding (input image features) and communication (referential game).
For ablation, we ﬁrst consider an untrained GRU speaker with randomized parameters ( random speaker ), and use it to sample tokens conditioned on image features according to (1).
As Table 2 shows, pre-training on such a corpus leads to even worse performance than training from scratch, because an untrained speaker has highly random generations with no clear structure.
On the other hand, to ablate the effect of visual stimuli, we sample a set of N random vectors from
N ( µ ( D I ) , σ ( D I )) , and replace it for D
I as inputs of the EC game ( random input ).
We then use the trained speaker to generate a EC corpus based on the same random vectors.
It turns out only one out of three game trials develop a game accuracy > 90% , and other two trials cannot learn an accuracy beyond 5% .
On Hebrew, the “successful” trial achieves a transfer perplexity of 388, which is still worse than EC corpus on visual inputs, while other two trials yield signiﬁcantly worse perplexities (573, 509).
So it turns out communicating visual stimuli instead of random inputs is more robustly successful, and more helpful for transfer even when communication can be successful.
Sentence Structure in EC Corpus To investigate how structured EC sentences are, we independently shufﬂe each EC sentence in D M to form a new-pretraining corpus ( permuted EC ), and ﬁnd in Table 2 that it indeed leads to a consistent worse performance, though better than training from scratch.
This indicates the speaker develops a coding scheme with sentence structures beyond bag-of-words.
Model vs. Corpus Transfer Li et al.
(2020b) propose to transfer the EC models for a downstream translation task, whereas we leverage a speaker-generated EC corpus to pre-train with architectures that could be more ﬂexible and powerful, which leads to several signiﬁcant practical advantages.
First, EC model architectures might not apply to many downstream tasks (e.g. image captioning with multiple input features), but corpus transfer could still be readily deployed (Section 4.2).
On other tasks (e.g. language modeling) where model transfer could be implemented, its performance is still constrained by the simple EC model architectures 2 .
As shown in Table 2, directly ﬁne-tuning the one-layer speaker GRU ( model transfer (GRU) ) indeed leads to much worse perplexities than corpus transfer with a Transformer ( EC pretrain ), but more interestingly, is also worse than corpus transfer with the same GRU architecture ( corpus transfer (GRU) ), possibly due to the beneﬁt of self-distillation (Zhang et al., 2019; Allen-Zhu & Li, 2020). In summary, our corpus transfer approach better utilizes the emergent communication for downstream tasks than model transfer.
E MERGENT -N ATURAL L ANGUAGE T RANSLATION AS A M ETRIC
Section 4.3 presents initial evidence that the quality of an emergent language correlates with the transfer performance, by showing ablated emergent languages transfer worse.
However, directly using transfer as an evaluation scheme for emergent languages has certain deﬁciencies.
First, the pretraining and ﬁne-tuning optimizations are computationally expensive.
Second, transfer performance across different downstream tasks might induce inconsistency.
Hence we propose a cheaper and simpler evaluation for emergent languages by measuring how easily it translates to the corresponding natural language grounded on the same input, and show that it better correlates with the transfer performance than existing metrics across different tasks.
More concretely, we use a small set of captioned images, where image I i is associated with a natural language caption NL i .
We also use the trained speaker to generate emergent message M i based on each
I i , and train a translation model to map emergent sentences to natural sentences M i
(cid:55)→ NL i .
Intuitively, a higher translation score means the emergent and natural sentences are closer in structure and semantics, similar to how French-English translation might be easier than Chinese-English.
Implementation To estimate the correlation between metrics and transfer performance requires a population of emergent languages with different speakers.
Thus we choose 5 EC game setups with varying vocabulary and sequence length limits (( V, T ) ∈ { (4035 , 5) , (4035 , 15) , (4035 , 25) , (1000 , 15) , (10000 , 25) } ) , and for each setup we run 4 trials for 2,000 steps.
We consider checkpoints every 200 steps, summing to 5 × 4 × 10 = 200 checkpoints.
Due to computational limits, we train EC games on a small subset of 50,000 MS-COCO images, and use another subset of 50,000 MS-COCO image-caption pairs to calculate the translation metric.
For translation, we use the same seq2seq Transformer for image captioning, but only train for 2 epochs, so that an emergent language more similar to the natural language can perform better with limited [Figure] training.
We use ROUGE L (Lin, 2004) as the translation metric, and ﬁnd other metrics with similar results.
For downstream language modeling, we use ImageNet (Deng et al., 2009) to generate a corpus of 15 million tokens and ﬁne-tune on Romanian ( ro ) and Hebrew ( he ).
Results We compare with the two existing metrics, validation accuracy and topographic similarity (see Section 3.2).
Table 3 shows how different metrics correlate with downstream performance across all checkpoints form all trials 3 .
Surprisingly, we ﬁnd little correlation between topographic similarity and downstream performance, while our translation metric has a better correlation ( ρ ro = 0 .
757 , ρ he = 0 . 829 ) than validation accuracy ( ρ ro = 0 .
672 , ρ he = 0 .
To better understand the results, we take all 4 trials from a speciﬁc game setup ( V = 10000 , T = 15 ) and plot in Figure 3 how the three metrics (accuracy, topographic, translation ROUGE L) as well as the downstream Romanian performance change with training steps (similar analysis of more setups (vocabulary size, sequence length) in Section B.4).
We observe that our metric (Figure 3(c)) aligns with the downstream performance (Figure 3(d)) in that both reach a high level around 300 steps (red dash line) and maintains a similar level afterwards.
The validation accuracy ((Figure 3(a))) has a generally similar tendency as these two metrics, with the difference being that it is only around 50% at step 300 and far from convergence, possibly due to the underdevelopment of the listener.
A similar observation was made in Li et al.
(2020b). Such a dependency on the listener renders game success unreliable for the evaluating emergent languages produced by the speaker alone.
On the other hand, the topographic similarity (Figure 3(b)) peaks as early as step 300, after which the degradation fails to explain the downstream performance.
To illustrate the misalignment, consider the “best” language with respect to topographic, i.e. a fully disentangled coding scheme (e.g. “red/blue cube/sphere”).
Such a corpus would essentially have a maximum possible entropy, pre-training on which should not help with any downstream tasks.
Our ﬁnding suggests that simple measurements of rigid disentanglement fall short of estimating the compositionality of real language , and we may need to take into account other structural properties of natural language, e.g. stable irregularity (Kirby, 2001), or the role of argument structures, context and morphology (Goldberg, 2015;
Steinert-Threlkeld, 2020).
C ONCLUSION
While previous research in language emergence tends to investigate different setups and how they affect certain metrics, such a paradigm cannot provide robust and valuable insights if the metrics are ﬂawed.
Thus, our work calls for a paradigm shift in how we evaluate and analyze language emergence — by stepping from metrics within each EC framework or over-approximate NL properties, and instead directly linking to NL by transfer or translation.
Such a paradigm shift enables several brand new and exciting future directions — how to improve or regularize EC through the lens of our metrics, how to push EC to be more useful for NLP 4 , and how to design more ﬁne-grained metrics for speciﬁc
NL properties.
We believe progress in these directions will further develop a synergy between EC and NLP research and bring beneﬁts and insights for both sides.
E THICS S TATEMENT
Our work is a preliminary step toward aligning the language of machines and humans, which has potential positive social impacts in several ways.
First, shaping machine communications towards the structure of natural language contributes to human and AI alignment in terms of values and understanding of the world.
Second, emergent languages could yield transfer beneﬁts for lowresource languages and tasks, contributing to linguistic diversity and inclusion.
We could not think of negative ethics impact for now, as the emergent language is still very different from natural language
R EPRODUCIBILITY S TATEMENT
Our research in based on public codebases and datasets, with implementation details described throughout the paper.
We have conducted reasonable hyperparameter searches and ablation studies to make sure experiment conclusions are fair.
In the Appendix, we further provide links to all used code and data resources and additional details including hyperparameter search setups and resource requirements for each experiment.
We will clean and release the code upon acceptance
A I MPLEMENTATION D ETAILS A. E MERGENT C OMMUNICATION G AME
We adapt the public code 5 from Li et al.
(2020b)
and mostly follow their default setups.
For training, we use a batch size of 256, and each batch element contains one input images and other 255 distractor images.
Since the Conceptual Captions dataset has more than 2.8 million images, random sampling a batch of data is computationally costly.
So for each batch, we ﬁrst sample 50,000 images from the whole dataset, then sample input and distractor pairs from this subset.
We use an Adam optimizer with learning rate 10 − 3 .
We use a soft version of Gumbel-softmax with temperature 1 , and have tried hard Gumbel-softmax and found it not further helpful for downstream performance.
Each game training only takes less than 12 hours using one GeForce RTX 2080 GPU.
A. L ANGUAGE M ODELING
We use the public script 6 from Papadimitriou & Jurafsky (2020) to pre-process Wikipedia corpora of different languages, using the default setup of culling to 50,000 vocabulary size.
We hand-pick downstream languages to make sure they represent different linguistic families.
We use the language modeling script 7 from Huggingface (Wolf et al., 2019) for both pre-training and ﬁne-tuning.
We have tried grid search for the pre-training learning rate ( 10 − 3 , 5 × 10 − 4 , 10 − 4 ) and batch size ( 4 , 32 ), which checkpoint to transfer ( 1000 , 2000 , 3000 ), as well as the ﬁne-tuning learning rate ( 10 − 4 , 5 × 10 − 5 , 10 − 5 ) and batch size ( 8 , 32 ).
We ﬁnd that for all three source corpora ( es , ec , paren-zipf ), it works best to pre-train with learning rate 5 × 10 − 4 and batch size ( 32 ), transfer using the checkpoint with 3000 training steps, and ﬁne-tune with learning rate 10 − 4 and batch size 8 .
For training from scratch, we have tried grid search for the learning rate ( 10 − 3 , 5 × 10 − 4 , 10 − 4 , 5 × 10 − 5 ) and batch size ( 4 , 32 ), and ﬁnd that learning rate 10 − 4 and batch size 8 work best for different downstream languages.
An pre-training experiment can ﬁnish within one hour using one GeForce
RTX 3090 GPU, while a ﬁne-tuning or training-from-scratch experiment can ﬁnish within one hour using one GeForce RTX 2080 GPU.
A. I MAGE C APTIONING
We use the pre-processed detection features 8 of Conceptual Captions from the codebase of Li et al.
(2020a).
For both pre-training and ﬁne-tuning, we use a public codebase 9 for image captioning based on
FAIRSEQ (Ott et al., 2019), and mostly follow their default setups.
Pre-training on Conceptual
Captions takes 8 GeForce RTX 3090 GPU for around two days.
Fine-tuning takes 1 GeForce RTX 2080 GPU for one hour.
B A DDITIONAL R ESULTS B. L ANGUAGE U NIGRAMS
As shown in Figure 4, the es and paren-zipf corpora have a larger vocabulary size (5000) and a larger entropy (6.48).
While ec is set with vocabulary limit 4,035, its corpus only uses around 2,500 words with smaller entropy (3.7).
The ec corpus with random speaker almost has a large entropy (7.98).
[Figure]
B. I MAGE C APTIONING
We visualize the ﬁne-tuning process of image captioning experiments in Figure 5.
Interestingly, we ﬁnd that under different natural language resource conditions (5,000, 50, or all samples in the MS-COCO (Lin et al., 2014) training set) the training progress is different.
Speciﬁcally, with 5,000 samples, EC or NL pre-training and training from scratch ﬁrst learn similarly well, then gaps gradually appear with more training epochs.
In contrast, when more han 50,0000 samples are used, the gap between pre-training methods and training from scratch is most signiﬁcant when trained for only one epoch
and it starts to diminish with more training epochs.
It suggests that even when downstream natural language resources are abundant, pre-training on an EC corpus might still help in a fast adaption setup.
We also perform additional ablation studies to conﬁrm the non-triviality of image captioning results.
Due to computation limits we use 450k MS-COCO samples (instead of 3M Conceptual Captions samples used in the main paper) to pre-train (image (cid:55)→ emergent caption / shufﬂed emergent caption / random paren-zipf string with the same unigram as emergent captions), and the other 50k MS-COCO samples to ﬁnetune (image (cid:55)→ English caption).
As shown in Table 4, pre
-training with shufﬂed emergent captions or ungrounded strings leads to worse performance than even no pre-training, while pre-training with emergent captions still improves the downstream performance.
This helps address the importance of having a semantically grounded and structural language for vision-language pretraining.
However, we do note that in the case of MS-COCO image captioning, ﬁne-tuning usually does the heavy-lifting and pre-training contributes much less than the case of language modeling.
That is why our work focuses analysis on the language modeling experiments.
B. C ORRELATIONS OF M ETRICS AND D OWNSTREAM P ERFORMANCE
Figure 6 plots the 200 data points in terms of their metric values and downstream performance on
Romanian and Hebrew modeling, respectively.
We note that topographic similarity is usually very
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
low (less than 0.1), and a higher value does not correlate with better downstream performance.
Also, we note that data points around the beginning of training (e
[Figure]
g. 200 steps) have very low accuracy or translation score, and their downstream performance is worse but signiﬁcantly more varied than the rest of data points.
B.
[Figure]
AND D OWNSTREAM P ERFORMANCES
Figure 3 in Section 5 mainly studies how metrics and downstream performance change with respect to the training steps, while controlling the other setups.
Following a similar logic, in Figure 7 we take those checkpoints with 1000 training steps and sequence lengths being 15 and plot the downstream performances and metrics with respect to the vocabulary size (1000, 4035, 10000), and in Figurewe take those checkpoints with 1000 training steps and vocabulary size being 4035 and plot the downstream performances and metrics with respect to the sequence length (5, 15, 25).
For the vocabulary size setup (Figure 7), we ﬁnd that both validation accuracy and our metric capture that the performance when vocabulary size is 1000 is worse and has signiﬁcantly more variance.
In contrast, topographic similarity is highest when vocabulary size is smallest (1000), which disagrees with downstream performances.
However, all metrics are higher when vocabulary size is 4035 instead of 10000, which is the opposite for downstream performances.
For the sequence length setup (Figure 8), we ﬁnd that our metric captures the fact that a larger sequence length leads to better and less varied downstream performance.
In contrast, validation accuracy with sequence length being 25 is worse than sequence length being 15, and topographic similarity fails to capture the decreasing trend of performance variance.
In summary
[Figure]
we ﬁnd that a larger vocabulary size or sequence length within our setup choices leads to better downstream performance with less variance, which is best captured by our translation metric.
[Figure]
[Figure]
Vocabulary size is 4035 and training step is 1000.
B. Q UALITATIVE E XAMPLES
Figure 9 includes some COCO images (not Conceptual Captions training images for EC) and their natural and emergent captions.
Manual check might reveal some patterns, e.g. transportation tools might start with 2340 (a, b, j), indoor scenes might start with two 1777 tokens and indoor home scenes might start with even more 1777 tokens (d, e, k, l).
Still, we note that our paper intentionally aims to move away from previous manual-check paradigms in EC papers and tries to establish a more standardized, scalable, and robust quantitative metric to enable more progress.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]