Multi-Agent Language Learning: Symbolic Mapping (2023)

Abstract: The study of emergent communication has long been devoted to coax neural network agents to learn a language sharing similar properties with human language. In this paper, we try to find a ‘natural’ way to help agents learn a compositional and symmetric language in complex settings like dialog games. Inspired by the theory that human language was originated from simple interactions, we hypothesize that language may evolve from simple tasks to difficult tasks. We propose a curriculum learning method called task transfer, and propose a novel architecture called symbolic mapping. We find that task transfer distinctly helps language learning in difficult tasks, and symbolic mapping promotes the effect. Further, we explore vocabulary expansion, and show that with the help of symbolic mapping, agents can easily learn to use new symbols when the environment becomes more complex. All in all, we find that a process from simplicity to complexity can serve as a natural way to help multi-agent language learning, and the proposed symbolic mapping is effective for this process.

## introduction agent communication has been a popular research field in the context of multi-agent reinforcement learning @xcite @xcite . recent work has focused on the emergence of language in cooperative tasks where neural network agents learn a communication protocol from scratch to solve problems together @xcite @xcite @xcite . an array of work has empirically shown that agents can make use of their developed language to successfully complete the tasks. beyond that, some work probed into the process of language emergence, and tried to figure out † corresponding author whether the learned language could share similar properties with human language like compositionality @xcite @xcite and symmetry @xcite @xcite . most of these studies on emergent communication are based on referential games @xcite and have shown that compositionality can be induced with suitable environmental pressures. some have explored the influential factors on the symmetry of protocols among a group of agents. however, tasks in these studies are often simple, and some of these methods are hard to implement in complex settings like dialog games. @xcite found that in a two-agent multi-round dialog game, language with compositionality does not naturally emerge, unless strict conditions are imposed to agents, such as deprivation of memory. language emergence only in simple games is obviously not satisfactory. in this paper, we tend to find a new way to make compositional and symmetric language emerge 'naturally' in complex settings. psychological studies suggest that human language was originated from simple gestures like pointing and pantomiming @xcite . this may explain why referential games are suitable for emergent language studies: these games are similar to 'pointing' in pragmatic process. however, from another perspective, the theory may also imply that communication protocols like human language cannot be formed directly from complex interactions. instead, a natural process is probably that a language is first formed in simple tasks, and then applied in more complex tasks, meanwhile it evolves to become more complicated and complete, similar to the concept of curriculum learning @xcite . hence, we propose a method called task transfer to implement this process on emergent communication between neural network agents, and explore whether the process could help language learning in complex settings through empirical experiments. we also design two tasks for the experiment, including a two-player referential game and a multi-round dialog game involving a group of agents. straightforward task transfer may not work well, since agents, even if using a same language all the time, can have different speaking policies across tasks. so instead of transferring the policies directly, we tend to make agents learn a common function for communication. we propose a novel architecture called symbolic mapping, which maps the input to related symbols, as a basic component of communication system of agent. the intuition is that when presented with the same input, we always associate it with the same pile of words and concepts, and this kind of association is consistent across tasks, so can be transferred. our experiments show that agents with symbolic mapping perform better in task transfer. as we explore the learning process of agents from simple tasks to difficult tasks, we are also curious about how the language evolves if old conventions are not enough in new environments. language learning should not be accomplished overnight. in a more natural scene, agents should first learn a simple language in a initial environment, and after entering a more complicated environment, they will learn something new and the language develops. we conduct the experiment about vocabulary expansion, also in a curriculum learning manner. we find that through vocabulary expansion, agents can accomplish tasks in complex environments where they would fail if they are asked to learn a language directly. this result reveals again that a process from simplicity to complexity is crucial for multi-agent language learning. and we also find that symbolic mapping agents perform better in vocabulary expansion. ## related work cooperative games. different kinds of cooperative games have been proposed in emergent communication literature. a popular one is referential game @xcite , where one agent, often noted as the speaker, has to send a message describing a target (e.g., an image) which it has just observed to the other agent. then the other agent, often noted as the listener, must select the target from several candidates containing the target and some distractors, after receiving the message @xcite . we use a variant of referential game to serve as the simple task in our experiments, similar to the game in @xcite where the listener should reconstruct the target. the difference is that we train the listener model by reinforcement learning, while they use the cross-entropy loss. our difficult task is inspired by the task & talk game proposed by @xcite , which is a multi-round dialog game. in the task & talk game, there are two agents, one always asks questions while the other answers these questions. however, our task involves a group of homogeneous agents who do not play specific roles.besides, our task has unfixed number of rounds, making the game more realistic while more complex. other studies @xcite @xcite also concern emergent language in a group of agents, and @xcite proposed a multi-step referential game. ## experimental setting for each attribute a, we represent it as a n adimension one-hot vector, where n @xmath0. an input object o from d n,m is then represented by the concatenation of all its attributes. symbolic mapping map(•) chooses symbols for o and gets the word bank w . the hidden state of the lstm h t serves as the memory. when speaking at time step t, one-hot encodings of symbols in w are concatenated to the hidden state h t and passed to speaking network g sp (•) to get the probability distribution π sp (•) to produce the symbol. in discrimination game, we initialize the hidden state h 0 as a zero vector, and each time a symbol s is transmitted in the dialog, s is fed into lstm f (•). symbols transmitted in the dialog are encoded as one-hot embeddings. to differentiate the speaker of each symbol, we concatenate a flag to the embeddings. if the speaker is the agent itself, the flag is zero; otherwise the flag is one. note that the agent does not know the identity of its partner. whenever receiving a symbol from the partner at time step t, the concatenation of hidden state h t and input object o is fed to the decision network π j dec (•), realized by a 2-layer mlp followed by a softmax activation, which outputs an action v t . the action means continuing the dialog or an answer. in description game, the speaking process is the same. we fix the message length to n, corresponding to one symbol per attribute. to do this, after the speaker produces a symbol s t at time step t, the symbol is fed into its lstm f (•), and the next symbol s t+1 is sampled at time step t + 1. this process proceeds until the fixed message length is reached. the listener is instantiated by n linear layers, which are called reconstruction networks. the message sent by the speaker is represented by the bag-of-words model and consumed by the listener. then each of its reconstruction network outputs an action to predict the value of each attribute. we use reinforce @xcite to train each agent. we apply entropy regularization in the loss function to encourage exploration, and use the adam optimizer with a learning rate of 0.001 in all settings. we run all our experiments three times with different random seeds and present the mean and standard deviation of the results. ## metrics compositionality. in our setting, the evaluation criterion of compositionality is whether agents can communicate different attributes independently. note that compositionalty in natural language has more complicated forms, but we only consider the juxtaposition of independent symbols to represent an overall meaning because we hypothesize that compositionality was rather simple when language was formed in the early stage and thus the proposed form is adequate for our research. inspired by positional disentanglement in chaabouni et al. ( 2020 ), we propose a metric called referential disentanglement (refdis), which measures whether a specific symbol refers to a specific attribute. we ignore the positional information because we need a language suitable for different kinds of interactions, and if symbols' positions are informative, the language is hard to transfer to dialogs. for each symbol s, we denote a s 1 the attribute that has the lowest conditional entropy given s : a s 1 = arg min a h(a|s). similarly, we denote a s 2 = arg min a̸ =a s 1 h(a|s). then we define refdis as: ## conclusion in this paper, we hypothesize that a process from simplicity to complexity is a natural way to help multi-agent language learning. we propose a curriculum learning method called task transfer, which uses referential games as the starting point of language learning. we propose symbolic mapping and implemented it in lstm-based agents. this architecture can be applied in different kinds of interactions, so that it can help realize language transfer across different tasks. we also explore another curriculum vocabulary expansion. our results show that learning from simplicity to complexity indeed helps, while symbolic mapping greatly promotes the effect of both task transfer and vocabulary expansion. in summary, we verify our hypothesis from two aspects, language transfer and language development, and our proposed architecture symbolic mapping shows remarkable effect.