X fer B ench: a Data-Driven Benchmark for Emergent Language (2024)

Abstract: In this paper, we introduce a benchmark for evaluating the overall quality of emergent languages using data-driven methods. Specifically, we interpret the notion of the “quality” of an emergent language as its similarity to human language within a deep learning framework. We measure this by using the emergent language as pretraining data for a downstream NLP tasks in human language—the better the downstream performance, the better the emergent language. We implement this benchmark as an easy-to-use Python package that only requires a text file of utterances from the emergent language to be evaluated. Finally, we empirically test the benchmark’s validity using human, synthetic, and emergent language baselines.

## introduction neural language models learn many things in pretraining, but research suggests @xcite that a substantial part of that knowledge is not simply knowledge of a particular language or domain, but rather knowledge of "how to language." we currently teach models to "language" using vast quantities of text dredged from the dark recesses of the web-text that is full of bias, toxicity, and potential intellectual property violations. ideally, we would be able to teach models to "language" without such compromises through the use of synthetic data, but mainstream approaches to synthesizing data produce outputs that do not have the same structural and social properties as human language. emergent communication (ec), also called emergent language (el), is a potential solution to this problem @xcite @xcite . emergent languages are communication systems developed de novo among multiple agents in a reinforcement learning simulation. because the conditions under which they develop mirror, reductively, the conditions under which languages develop among humans, there is reason to believe that els will ultimately be more like human language than other sources of synthetic data. however, up to this point, there is no way of quantifying-in a holistic way-how much like human languages any particular el really is, or to what extent it may provide useful pretraining signals. research on deep learning-based emergent communication has seen the introduction of many metrics to measure various aspects of the language. these metrics quantify notions such as compositionality @xcite , expressivity @xcite , ease-of-teaching @xcite , and zero-shot transfer @xcite , to name a few. despite this proliferation of metrics, emergent language largely lacks evaluation metrics. an evaluation metric is specifically one that measures the overall quality of an emergent language and not simply a particular property. thus, we introduce xferbench, a data-driven benchmark for evaluating the overall quality of emergent languages using transfer learning with deep neural models. evaluation metrics are critical in gauging progress in technical fields since they quantify otherwise vague notions of improvement over time. benchmarks, in particular, pair evaluation metrics with specific data and evaluation procedures to compare various systems on common ground. benchmarks and shared tasks have been critical to the development of nlp from the penn treebank @xcite to the wmt datasets @xcite to glue @xcite . in the field of emergent communication specifically, @xcite introduced the idea of using corpus transfer as means of practically applying emergent communication to deep learning-based nlp via transfer learning. in corpus transfer, a language model is pretrained on a corpus of emergent language utterances before being tuned on real data for a human languagebased downstream task. as a corollary, they suggest that the effectiveness of this transfer can serve as a means of evaluating the quality of the emergent in a more general sense. this is based on the intuition that the more similar two language are, the better transfer learning works from one to the other (observed in @xcite , for example). this paper takes the transfer learning-as-anevaluation metric idea from @xcite and expands it into a full benchmark, xferbench, for emergent languages (illustrated in figure 1 ). an evaluation metric for emergent languages in a benchmark format is the first of its kind. additionally, xferbench is unique within emergent communication for being primarily data-driven instead of relying on particular handcrafted emergent language corpus pretrain language model tune on human language 1 cross-entropy on hl 1 tune on hl 2 . . . tune on hl n ce on hl 2 . . . ce on hl n final score (average) figure 1: illustration of the architecture of xferbench. algorithms for quantifying a given phenomenon. this means that xferbench can be easily scaled up in the future as the field of emergent communication advances and requires expanded means of evaluating emergent languages. finally, xferbench is distributed as a userfriendly python package, allowing researchers from across the field of emergent communication to apply xferbench to their own work on emergent communication. contributions this paper makes the following contributions: (1) introduces xferbench, a data-driven benchmark for evaluating the overall quality of an emergent language, the first of its kind in emergent communication. emergent communication this paper is situated in the field of emergent communication (a.k.a. emergent language) which is generally covered by the review @xcite . the field centers around the invention of language by deep neural networks typically using multi-agent reinforcement learning techniques. the study of emergent communication is intended to ( 1 ) shed light on the origin and nature of the human language (lacroix, 2019; @xcite and ( 2 ) provide an alternative approach to problems in nlp and multi-agent reinforcement learning which relies on constructing language from the ground up and not just pre-existing (human) languages alone @xcite @xcite . transfer learning transfer learning for deep neural networks is a key component of xferbench and follows in general tradition of @xcite . specifically, this paper draws heavily from @xcite (see also @xcite ; @xcite ) which introduce the technique of corpus transfer for emergent language, that is, pretraining a neural model on an emergent language corpus before tuning it on a downstream human language task. in particular, this paper takes @xcite 's idea of using corpus transfer as a metric and adapts it into a benchmark pipeline which can easily be applied to new emergent languages. benchmarks work such as @xcite and @xcite have looked at benchmarking particular aspects of emergent languages, but xferbench is the first of its kind in benchmarking the overall quality of an emergent language. @xcite also explicitly provide a metric for emergent language quality, but this metric is restrictive in that it can only be applied to emergent languages derived from a model that takes images (that have captions available) as input; this conflicts with the design goals of xferbench discussed below. outside of emergent communication, xferbench is more analogous to benchmarks for generative models (e.g., fréchet inception distance @xcite for image generation) than more traditional nlp benchmarks like glue @xcite or squad @xcite . this is because emergent communication is a generative enterprise, where one of the main goals is to create samples (emergent languages) which resemble a target distribution (human languages) either generally or in some particular respect. furthermore, metrics like fid are primarily self-supervised, data-driven measures of similarity in the same vein as xferbench. this is in contrast to more traditional nlp benchmarks which combine data-driven methods with many human judgments (i.e., through labeled examples). ## conclusion in this paper we have introduced xferbench, a firstof-its-kind benchmark for evaluating the quality of an emergent language corpus based on its transfer learning performance on human languages. this approach to evaluating emergent language scales with data and compute as opposed to requiring increasingly complex handcrafted rules to measure the desirable qualities of emergent language. we provide empirical results of xferbench across human, synthetic, and emergent languages and demonstrate that these results correlate with downstream performance on a machine translation task. xferbench is implemented as an easy-to-use python package that will permit researchers in the field to easily apply xferbench to new emergent languages.