Concept-Best-Matching: Evaluating Compositionality In Emergent Communication (2024)

Abstract: Artificial agents that learn to communicate in order to accomplish a given task acquire communication protocols that are typically opaque to a human. A large body of work has attempted to evaluate the emergent communication via various evaluation measures, with **compositionality** featuring as a prominent desired trait. However, current evaluation procedures do not directly expose the compositionality of the emergent communication. We propose a procedure to assess the compositionality of emergent communication by finding the best-match between emerged words and natural language concepts.The best-match algorithm provides both a global score and a translation-map from emergent words to natural language concepts. To the best of our knowledge, it is the first time that such direct and interpretable mapping between emergent words and human concepts is provided.

## introduction artificial agents that learn to communicate for accomplishing a given task acquire communication protocols. in the common setting, a sender observes a set of objects and sends a message to a receiver, which then needs to identify the correct target objects out of a set of distractors. the two agents learn a communication protocol over discrete vocabulary of atoms, termed "words". the emergent communication (ec) is typically opaque to a human. as a result, a large body of work has attempted to characterize the emergent communication in light of natural language (nl) traits, such as compositionality @xcite , systematic generalization @xcite , pragmatism @xcite , and more. chief among these is compositionality, which enables the construction of complex meanings from the meaning of parts @xcite . however, quantifying compositionality in ec is notofor the blue-triangle (turn 1) and [w 2 , w 3 ] for the red-triangle (turn 2). (c) during evaluation, we construct a bipartite graph of words generated by the sender and concepts provided by the oracle for each turn. (d) the best-match algorithm matches ec words to nl concepts and provides the cbm score. in this example, all ec words are matched with nl concepts, resulting in a cbm score of 1.0. this work develops a method that characterizes how compositional an ec is. our method is founded on the key insight that, in an ec setting, a communication is compositional if agents communicate successfully via complex messages formed of simple atoms and these atoms are mapped to nl concepts. our evaluation is based on the classical best-match algorithm @xcite . given a set of ec atoms and a set of nl concepts, we construct a bi-partite graph and seek the optimal one-to-one mapping between words and concepts. a perfectly compositional ec would yield a perfect match, where every ec atom is mapped to exactly one nl concept. see figure 1 for an illustration. we experiment with the proposed procedure in two ec settings and compare it to two popular evaluation measures -topographic similarity and adjusted mutual information (ami). we demonstrate that our approach provides a fine-grained characterization of emergent protocols, exposing their strengths and weaknesses, while other measures only provide coarse, opaque scores. according to our evaluation, state-of-the-art communication methods do not achieve satisfactory results. ## concept best matching our key insight is that compositionality requires a mapping between words (atomic parts of ec messages) and concepts (a set of fvps) that compose nl phrases. given an evaluation set of examples d, consider their corresponding ec messages m and nl phrases l. let w denote the set of unique words in m, and v the set of unique fvps in l. let {w} i , {v} i be the sets of words and fvps in sample i, and with words on one side and fvps on the other side. edges e are defined by the evaluation set. the weight q e of edge e ij is the number of times word i appeared in a message that the sender transmitted for a labeling rule with fvp j. this construction reflects the intuition that we do not know the correct mapping between words and fvps (concepts). see figure 1 for an example. ## experimental setup we experiments with agents that learn to play multitarget referential games, introduced by @xcite and described in section 2.1. all experiments reported in datasets. we experiment with two datasets, described briefly here; appendix b has more details. ## results the table also shows the rate of ambiguities and paraphrases, exposed by our method. ambiguities happen when the same ec word is mapped to different concepts. finally, unmatched concepts have no matching ec word. they result from a too narrow or underutilized channel. as example match. beyond global scores, the cbm provides an interpretable translation graph, which maps ec words to nl concepts, facilitating insights on reasons for sub-optimal communication. figure 2 shows an example graph for the shape game with gs communication. 4 as seen, the algorithm successfully matched 10 ec words to 10 concepts. the sender generated 11 unique words in this experiment, indicating that one word is a paraphrase of an already matched concept, even within this narrow-channel setup. ## conclusion we proposed a new procedure for assessing compositionality of emergent communication. in contrast to other evaluations, our procedure provides a human-interpretable translation map of emerged words to natural concepts. moreover, our approach provides detailed insights into the reasons for suboptimal translation. we demonstrated it on two ec games with two different communication types. our evaluation reveals even that quantized communication performs better than gumbel-softmax, yet none exhibits compositionality in a level similar to natural language.