Emergent Communication Pretraining for Few-Shot Machine Translation (2020)

Abstract: While state-of-the-art models that rely upon massively multilingual pretrained encoders achieve sample efficiency in downstream applications, they still require abundant amounts of unlabelled text. Nevertheless, most of the world’s languages lack such resources. Hence, we investigate a more radical form of unsupervised knowledge transfer in the absence of linguistic data. In particular, for the first time we pretrain neural networks via emergent communication from referential games. Our key assumption is that grounding communication on images—as a crude approximation of real-world environments—inductively biases the model towards learning natural languages. On the one hand, we show that this substantially benefits machine translation in few-shot settings. On the other hand, this also provides an extrinsic evaluation protocol to probe the properties of emergent languages ex vitro. Intuitively, the closer they are to natural languages, the higher the gains from pretraining on them should be. For instance, in this work we measure the influence of communication success and maximum sequence length on downstream performances. Finally, we introduce a customised adapter layer and annealing strategies for the regulariser of maximum-a-posteriori inference during fine-tuning. These turn out to be crucial to facilitate knowledge transfer and prevent catastrophic forgetting. Compared to a recurrent baseline, our method yields gains of 59.0% 147.6% in BLEU score with only 500 NMT training instances and 65.1% 196.7% with 1,000 NMT training instances across four language pairs. These proof-of-concept results reveal the potential of emergent communication pretraining for both natural language processing tasks in resource-poor settings and extrinsic evaluation of artificial languages.

## introduction zero-shot and few-shot learning are notoriously challenging for neural networks @xcite @xcite . however, they are a prerequisite for natural language processing in most languages, which suffer from the paucity of annotated data @xcite . state-of-the-art models rely on knowledge transfer, whereby an encoder is pretrained via language modeling on texts from multiple languages, and subsequently 'fine-tuned' on labelled examples of resource-rich languages @xcite or few examples in a target resourcepoor language @xcite . however, even raw texts required for pretraining are scant @xcite : for instance, wikipedia dumps cover 278 languages out of the 7,097 spoken world-wide @xcite . for this reason, we push the idea of cross-lingual knowledge transfer even further, exploring and profiling a setting where not even raw natural language data for a target language are available for unsupervised pretraining. in their stead, we exploit artificial languages emerging from a referential game on raw images @xcite . in particular, we encourage agents to cooperate in identifying images among distractors by communicating over vocabularies whose meanings are unknown. the key intuition is that, whereas lexicalisation is mostly arbitrary @xcite , communication grounded in a real-world environment does constrain what languages are likely or figure 1 : an overview of the model architecture. dashed lines denote parameter transfer from the ec pretraining task to the mt fine-tuning task. we stress that during ec pretraining, we do not leverage any image-caption pairs; instead, only unlabelled images are used. during mt fine-tuning, standard seq2seq nmt models are trained on src and trg sentence pairs without any visual information available. possible @xcite . hence, we hypothesise that communication over raw images offers a favourable inductive bias for natural language tasks. in particular, we experiment with initialising an encoder-decoder model for few-shot neural machine translation with parameters pretrained on emergent communication. in the past, emergent communication has mostly attracted theoretical interest as a tool to shed light on cooperative behaviours, the compositional properties of emergent communication protocols @xcite @xcite @xcite , and natural language evolution @xcite . to our knowledge, this is the first preliminary study on deploying artificial languages from emergent communication in natural language applications. conversely, our method also constitutes an extrinsic evaluation protocol to probe the properties of different emergent languages. the underlying assumption is that they should facilitate downstream tasks only to the extent that they share common characteristics with natural languages. in particular, we run in-depth analyses on the impact that the rate of communication success and maximum sequence length have on nmt performance. for the sake of fully leveraging the pretrained parameters and ameliorating overfitting, we also explore several new strategies to perform knowledge transfer. in particular, we customise the adapter layer @xcite and propose annealing strategies for the regularisation term of map inference during fine-tuning. we run experiments in nmt between english and four languages (german, czech, romanian, and french) in both directions. by virtue of emergent communication pretraining and the proposed transfer strategies, we report gains in bleu scores when simulating few-shot mt setups for the four target languages: 59.0%∼147.6% over a standard encoder-decoder baseline when 500 training instances are available, and 65.1%∼196.7% when 1, 000 training instances are available. our code is available online at https://github.com/cambridgeltl/ecnmt . ## related work our work lies at the intersection of several prominent research areas such as pretraining for transfer learning, emergent communication, few-shot machine translation, and inductive biases for language. to all of these we cannot do full justice given space constraints. pretraining for transfer learning. unsupervised pretraining on large collections of unlabelled text yields general-purpose contextualized word representations @xcite that are beneficial across a range of downstream nlp tasks. the current dominant paradigm is training a transformer-based deep model @xcite relying on masked language modeling or a similar objective, as proposed in the omnipresent bert model @xcite and its extensions @xcite @xcite , and then fine-tuning the model further on a downstream task @xcite . often this approach exploits large textual data and deep models spanning even billions of parameters @xcite @xcite . in this work, we refrain from chasing task leaderboards (linzen, 2020) and posit a fundamental question about language learning instead. ## model architecture the proposed method comprises the standard two stages of transfer learning. first, as detailed in § 3.1, we pretrain two speaker-listener agents via emergent communication on image referential games. we then recombine foot_0 the pretrained ec agents to construct nmt encoder-decoder networks (see figure 1 ), and fine-tune the networks on a small number of parallel sentence pairs, as we describe in § 3.2. at the fine-tuning stage, we also add an adapter layer between the translation encoder and decoder, and further leverage two variants of regularisation with annealing, which are outlined in § 3.3. an illustrative overview of the proposed method is provided in figure 1 . ## experimental setup ec pretraining is based on the ms coco data set @xcite . we randomly select 50,000 images for training and 5,000 for validation. 5 for each image, a 2,048-dimensional feature vector is extracted from resnet-50 @xcite . the input vocabulary size for ec is equal to the bpe vocabulary size during mt fine-tuning. however, since human language data are excluded, note that there is no alignment between ec and mt bpe vocabularies. 6 the maximum message length, l max , is set to an integer around the average length, in terms of bpe tokens, of the mt training sets: 15-18 in our experiments. we do not impose additional constraints on the generated messages' length. 7 we later profile the impact of l max on mt performance in §5. the layer size is 256 for the input embeddings and 512 for the hidden layers. we use adam (kingma and ba, 2015) with a learning rate of @xmath0.001. the dropout rate is set to 0.1 and the gumbel-softmax temperature is set to 1. the number of distracting images is @xmath1, and @xmath2. experiments on the validation set achieve the prediction accuracy @xmath3, i.e., the listener is able to guess the single correct image from a set of 128 images almost always. we analyse the impact of ec prediction accuracy on few-shot mt performance later in §5. machine translation experiments are conducted on two standard datasets: multi30k and europarl. the multi30k data set @xcite , originally devised for multi-modal mt, contains multilingual captions for ≈ 30k images. we discard images and run text-only fine-tuning and evaluation on english-german (en-de) and english-czech (en-cs) in both directions. we rely on the default training set of 29,000 pairs of parallel sentences, which we also subsample to simulate true few-shot scenarios: we randomly select 500, 1,000, and 10,000 sentence pairs for the lower-resource setups. in all experimental runs, we use the original validation set spanning 1,014 sentence pairs and the default test set spanning 1,000 pairs. we also run experiments on europarl data @xcite from opus @xcite for two language pairs: english-romanian (en-ro) and english-french (en-fr), again in both directions. we retain only sentences with a length between 5 and 15 words to construct data sets whose average sentence length is similar to that of multi30k. we then randomly sample 10,000 parallel sentences as our (largest) training set, while two other disjoint random samples of 1,500 sentence pairs are used for validation and test, respectively. as with multi30k, we again sample 500 and 1,000 training instances from the full set of 10k examples to simulate few-shot settings. for each language pair, we lowercase and tokenise the data using byte-pair encoding (bpe) @xcite . our bpe vocabularies are derived from all 29,000 training pairs (for the multi30k language pairs) and 10,000 training pairs (for the europarl language pairs). we again use adam in the same configuration as ec pretraining, except for setting the dropout rate to 0.2. the hyper-parameters of the annealed regulariser are set to α = 5 and λ = 0.998 based on the scores on the en-de validation set (in the 1k training setup) and fixed to those values in all other experiments and for all other language pairs. for a fair comparison, the other hyper-parameters for fine-tuning are set identically to the nmt baseline introduced in the next paragraph. ## results and analysis in what follows, we report the nmt results of our proposed model on all language pairs. we then perform an ablation study highlighting the individual contributions-of the customised adapter layer, the strategies for annealing the regulariser, and emergent communication pretraining-to the final results. finally, we assess the impact of the rate of communication success and maximum sequence length on downstream nmt performances. ## conclusion and future work we have demonstrated that an extreme pretraining paradigm without any human language data, but rather based on emergent communication (ec) in referential games, provides an inductive bias for learning natural languages. in theory, it makes this paradigm applicable to any of the world's languages, most of which suffer from the paucity of annotated data. in particular, we focused on neural machine translation (nmt) with limited parallel data as a downstream task. our results across several language pairs and in different few-shot setups indicate that parameter initialisations informed by ec pretraining, in combination with adapter modules and annealed regularisation, yield higher accuracy and sample efficiency than baselines without any ec pretraining. vice versa, we argued that nmt performance can also be interpreted as an extrinsic evaluation protocol for emergent communication models: it can assess to which extent emergent languages reflect properties found in natural languages. in particular, we discovered that communication success rate is well correlated with bleu scores, whereas maximum sequence length is not impactful. in the future, we plan to experiment with other state-of-the-art nmt architectures, apply our method to extremely low-resource languages, and extend the scope of our work to other tasks beyond nmt.