Paper: Emergent Robust Communication for Multi-Round Interactions in Noisy Environments (ICLR-2024)

Reviewer comment (Novelty):
There are several works introducing the multi-round Lewis Game and its corresponding listener architectures [1,2]. Justification would be better to underscore your contribution by comparing and contrasting these literatures.

Author response:
We thank Reviewer VCLK for the suggestions. We added these works to the extended related work section in Appendix A. Nevertheless, there are significant differences between [1,2] and our work: In [1], the Listener processes one candidate at a time, and the vocabulary used contains only two tokens. In our case, our vocabulary has 20 tokens, and the Listener tries to discriminate between all candidates at each round, which substantially increases game complexity. In [2], the game design is similar to ours, where the Listener receives a message and decides to play another round or select a candidate as the final answer. Nonetheless, our approach is far more complex and realistic than in [2] since we have a discrete communication channel that can suffer modifications from an external source instead of a continuous communication channel [2]. Complementarily, [2] simplifies the problem by using a DIAL approach, allowing gradients to flow between agents. In contrast, our method uses a RIAL approach, where agents perceive others as part of the environment, meaning no gradient flows between agents. In this case, the Speaker does not know that the message is being modified, complicating the coordination task.

Reviewer comment (Experiments):
From the experimental results (Table 3,4,7,8), the accuracies of game variants with/without multiple rounds do not differ a lot. More elaboration or experiments are expected to ablate the contribution of the multi-round setting.

Author response:
With these results, we observe no clear advantage to using multiple rounds; the agents have enough predictive power to play the game successfully with only one round.

Reviewer comment (Experiments):
Training and testing in the same noisy environment may not be enough to show that “robust communication” emerges, as noise may come from the observation besides the communication channel, for example from the sender or receiver sides [3].

Author response:
We thank Reviewer VCLK for the suggestion. This is an interesting experiment to explore. We already have further experiments in Appendix F; one of them (discrimination task) does this, adding noise to the observation of the Speaker and the Listener's candidates.

Reviewer comment (Experiments):
Instead of also replacing message tokens during testing, other interfering methods to the communication channel can be applied, for example randomly dropping tokens in the messages.

Author response:
This is what we do, the noise we introduce acts by replacing message token by a pre-defined token called the *unk* token.

Reviewer comment (Experiments):
It is unclear what the result would be when the agents are trained with $\lambda=0.75$ and tested with 0.5, and vice versa.

Author response:
We thank Reviewer VCLK for the suggestion. We can easily add further results considering this (experiments are already running).

Reviewer comment (Experiments):
It is unclear how the results would compare with previous works on adding noise to the communication channel [4,5].

Author response:
We thank Reviewer VCLK for the suggestions. We added these works to the extended related work section in Appendix A. Still, there are some major differences. [4] uses a DIAL setting, and the noise is introduced in the continuous space (before sampling discrete tokens). [5] also implements a DIAL setting and the introduction of noise is adversarial, meaning message tokens are changed by other valid tokens, trying to deceive the Listener. In contrast, our method uses a RIAL approach, where agents perceive others as part of the environment, meaning no gradient flows between agents. In this case, the Speaker does not know that the message is being modified, complicating the coordination task.

Reviewer comment (Experiments):
What is the maximum number of round $I$ set to? Throughout the training process, does the average number of communicative rounds vary a lot? For example, from more rounds to fewer rounds? This may help validate your multi-round design.

Author response:
Yes, the avg. number of rounds increases as the training progresses. We can add a section to the appendix displaying this information. Although the number of rounds increases, the overall performance remains the same.

Reviewer comment (Writing):
There are several references missing on page 6.

Author response:
This was a problem when splitting the document into 2 parts (main paper and appendix). We fixed the problem, thank you.

Reviewer comment (Experiments):
This paper does not explore the type of messages learned by the agents. It would be interesting to visualize in some way some of the messages learned by the agents to communicate.

Author response:
We thank Review zpnz for the interesting suggestion. Since the agents learn an abstract language using abstract tokens, we do not have a good way to visualize the messages sent, but it could be an interesting future work direction.

Reviewer comment (Theory):
The entropy term in the loss is not defined in the paper and the motivations for the use of this term do not seem good enough.

Author response:
The loss functions used in both agents are fully detailed in Appendix E.1. Additionally. The entropy term was fully evaluated in [Chaabouni et al., 2022]. In the text, we point to this paper for further details (see Section 2.2.2).

Reviewer comment (Evaluation):
In table 2, I wonder about the significance of these results; of course the agents are trained with noise, they will perform better in noisy testing than agents that were not trained with noise.

Author response:
We can see a clear difference in the emergent communication protocol learned. Looking at Table 1 and Table 2, we can understand that the agents trained in the NLG (with noise) can create a robust communication protocol to handle noisy and noiseless messages.

Reviewer comment (Novelty):
The use of noise is framed as a major contribution. However, using random noise in the messages is not new; the authors give the example of the work with the zipfs law to analyse language properties (page 2) but this is not the only place where noise is used and analysed [1, 2, 4].

Author response:
We thank Reviewer zpnz for the suggestions. We added these works to the extended related work section in Appendix A. Nevertheless, we maintain our claim for novelty for the following reasons. All works [1, 2, 4] simplify the problem by having a DIAL method, meaning gradient flows between agents. In our case, we implement a RIAL architecture where each agent sees the other as part of the environment, substantially increasing the problem's complexity. As such, the Speaker does not even know that the message is suffering modifications, making the coordination problem extremely complicated. [1] also uses a continuous communication problem, making it distant from the scope of our work.

Reviewer comment (Writing):
In section 2.2, it is unclear to me what is the $unk$ token and how the noise is applied. From equation (1) I understand it results from the noise function if $p\leq\lambda$, but I fail to understand where $unk$ comes from. Is it a fixed token? If so, I cannot agree that noise is being applied to the message, also because, according to the equation, the message $m$ is not affecting any of the generated noise. Could the authors elaborate on this?

Author response:
Yes, *unk* is a fixed token. We apply noise to the message by substituting a variable number of message tokens with the *unk* token. The noise applied is externally introduced in the message. The Speaker does not even know if the message contains noise, adding additional complexity.

Reviewer comment (Writing):
It is unclear to me what $\hat{x}$ means. In page 3, it is both stated "where the goal is to try to identify the image $x \in C$ that the Speaker received, $\hat{x}= x$" and "round when the Listener plays the I don't know (idk) action, $\hat{x} = \hat{x}_{idk}$". It seems to have different meaning in each case. In the first case it seems to denote the guess of the listener and in the second it seems to define an action. Could the authors clarify?

Author response:
$\hat{x}$ is the action selected by the Listener. This action can be an image or the action ``I don't know'' ($x_{\text{idk}}$). If an image is selected by Listener, $\hat{x}$ is also one of the images of a set (candidates set) given to the Listener.

Reviewer comment (Theory):
In page 6: "where we linearly increase the noise level in the communication channel from 0 to $\lambda$.". From equation (1) $\lambda$ represents a probability of whether to insert noise or not. How does increasing $\lambda$ increase the level of noise? While it can happen, it does not seem necessarily true that it will happen.

Author response:
For each token in the message, we draw a number from the uniform distribution $l\sim\mathcal{U}(0,1)$. If $l>\lambda$ we change the message token by the \textit{unk} token. In expectation the number of removed token is $\lambda\cdot T$, where $T$ is the message length.

Reviewer comment (Evaluation):
In tables 1 and 2, since LG(RL) and NLG are the same but NLG uses noise (as described in section 3.1); how is the accuracy of NLG much higher? Is the existence of noise ($\lambda\geq 0$) beneficial for learning?

Author response:
Yes, from table 1 and 2 we can see that when adding a noisy communication channel allows for the agent to process more diverse information and create robust communication channel that work in both cases (Lewis Game with and without noise).

Reviewer comment (Novelty):
Is the variation LG(RL) a contribution (as stated in section 3.1)? REINFORCE has been used before in emergent communication games [3, 4].

Author response:
We thank Reviewer zpnz for the suggestions. We added these works to the extended related work section in Appendix A. The architectures applied for the games we propose are novel. The proposed games use natural images as input and discrete messages. For example, in [3], the input has very few degrees of freedom; [4] uses a DIAL method, meaning gradient flows from the Listener to the Speaker, which implies severe architectural differences. In our case, we propose a new Listener architecture that uses an attention mechanism to combine information from the noisy message and each candidate's information. Additionally, we also modify the Listener's policy to take into account time-dependent environments, where we give the possibility to the Listener reason if it prefers to play another round (gather more information) or make a final decision.

Reviewer comment (Reproducibility):
Do the authors allow the gradients to flow across agents as it happens in works such as DIAL [5], or are they fully independent?

Author response:
As described in the beginning of section 2 (and Appendix A), we impose a RIAL setting, no gradients flow between the Speaker and Listener.

Reviewer comment (Novelty):
The authors' reliance on established paradigms like the Lewis Game, even with modifications, raises concerns about true innovation. Is this just a repackaging of old concepts?

Author response:
The Lewis Game is a well-established task in emergent communication since it presents several crucial challenges to overcome to understand and study how communication can emerge to solve a cooperative task. Most related work presented in the paper and the related work proposed by all reviewers has experiments based on a derivation of the original Lewis Game. In our proposed work, we follow the work proposed by [Chaabouni et al., 2022] that complexifies the original Lewis Game by using huge natural image datasets for the input distribution and increases the number of candidates given to the Listener, increasing the difficulty of the discrimination task. The main focus of our work is to study and explore how noisy conditions can affect the emergence and quality of the communication protocol in such tasks. As described in Section 3, previous methods are not robust to noise in the communication channel. As such, we proposed new agent architectures and training schemes, creating robust communication channels.

Reviewer comment (Novelty):
The emphasis on noise adaptation, though relevant, is hardly novel in the field of emergent communication. The authors fail to differentiate their approach sufficiently from existing solutions.

Author response:
In the introduction, we talk about [Ueda and Washio, 2021]. Even though previous works consider noisy environments in emergent communication, the difference to our work is two-fold. First, we only consider discrete messages as [Ueda and Washio, 2021]. Second, much like  [Chaabouni et al., 2022], we take into consideration more realistic datasets (discriminate natural images) applied to the context of having noisy communication channels. We added to appendix a section for related work, see Appendix A.

Reviewer comment (Novelty):
The comparative approach with Ueda & Washio 2021 feels superficial, lacking in-depth analysis on fundamental differences.

Author response:
Please check Appendix A, for a more in-depth comparison (extended related work).
    
As explained in the introduction, the main difference is that [Ueda and Washio, 2021] promotes an adversarial setting. The noise is introduced by substituting a token with another one. In this setting, the main focus is to trick the Listener by giving another plausible message different from the one created by the Speaker. By creating a problem like this, the authors want to force a pair of agents to create messages as short as possible to test the ZLA hypothesis. Additionally, [Ueda and Washio, 2021] also simplifies the environment by giving as input one-hot encoded inputs to the Speaker.

In our case, not only do we use a more realistic dataset increasing complexity, but we also focus on a different problem. Our focus is to create robust communication protocols that can handle external interference in the communication channel, for example, simulating the loss of information in the communication channel. As such, we apply noise by substituting tokens into a fixed and extra token called the unknown token. Another particularity of our work is that we want our method to function in the noiseless and noisy cases at the same time. We added an extended related work section to the appendix where we fully detail the differences between our work and [Ueda and Washio, 2021].

Summarizing, in the underlying assumptions, we differ from [Ueda and Washio, 2021] since we assume that the noisy communication channel has external interference, which translates into the loss of information. On the other hand, [Ueda and Washio, 2021] assumes an adversarial setting where the objective is trying to deceive the Listener. Regarding the problem formulation, the game proposed by [Ueda and Washio, 2021] is a simpler version of our NLG game. In our case, we use natural images as input. Additionally, the Listener's discrimination task takes into account up to 4096 images to discretize, where in [Ueda and Washio, 2021] the input is a simple one-hot vector with a maximum size of 256. Moreover, since the input is one-hot encoded, the Listener does not need to receive candidates and just outputs a number between 1 and 256. For the expected outcomes, we show that our methods work on the noiseless and noisy case, creating robust communication protocols. On the other hand, [Ueda and Washio, 2021] only wants to force the communication protocol to create small messages by introducing adversarial noise, where a test between settings (train without noise and test with noise, and vice-versa) was never made or even intended.

Reviewer comment (Novelty):
Given the modifications you've introduced to the Lewis Game, how do you justify that these changes bring genuine innovation in the context of multi-agent reinforcement learning, as opposed to simply adding complexity to an existing framework?

Author response:
We would like to emphasize that the primary purpose of our work is not to create a MARL system. The main trends in MARL focus on studying centralized training with decentralized execution, how partial observability affects performance, and how to exchange information through communication. In contrast, emergent communication studies aim to assemble cooperative tasks to study emergent communication and its properties.

We now clarify the novelty of our work. First, we present a series of experimental evaluations that give new insights into how noise affects the emergence of communication when using more realistic datasets (in our case, ImageNet and CelebA). Secondly, we merge noisy and time-dependent environments, creating highly complex environments that generalize the original Lewis Game. Finally, we derive new agent architectures to play these realistic and more challenging games. For example, most (if not all) methods proposed in previous works could not take advantage of the Multi-Round Indecisive Lewis Game (MRILG) because these methods are not prepared to have time dependencies in their architecture.

Reviewer comment (Novelty):
The concept of noise adaptation in emergent communication is not new. How does your approach fundamentally differ from existing solutions, and what unique challenges does it address in the multi-agent reinforcement learning landscape?

Author response:
In the introduction, we talk about [Ueda and Washio, 2021]. Even though previous works consider noisy environments in emergent communication, the difference to our work is two-fold. First, we only consider discrete messages as [Ueda and Washio, 2021]. Second, much like  [Chaabouni et al., 2022], we take into consideration more realistic datasets (discriminate natural images) applied to the context of having noisy communication channels. We added to appendix a section for related work, see Appendix A.

Reviewer comment (Novelty):
Your comparison with the Ueda & Washio 2021 study appears to lack depth. Could you elucidate the fundamental differences in the underlying assumptions, problem formulations, and outcomes between their work and yours, especially in the context of multi-agent dynamics?

Author response:
Please check Appendix A, for a more in-depth comparison (extended related work).
    
As explained in the introduction, the main difference is that [Ueda and Washio, 2021] promotes an adversarial setting. The noise is introduced by substituting a token with another one. In this setting, the main focus is to trick the Listener by giving another plausible message different from the one created by the Speaker. By creating a problem like this, the authors want to force a pair of agents to create messages as short as possible to test the ZLA hypothesis. Additionally, [Ueda and Washio, 2021] also simplifies the environment by giving as input one-hot encoded inputs to the Speaker.

In our case, not only do we use a more realistic dataset increasing complexity, but we also focus on a different problem. Our focus is to create robust communication protocols that can handle external interference in the communication channel, for example, simulating the loss of information in the communication channel. As such, we apply noise by substituting tokens into a fixed and extra token called the unknown token. Another particularity of our work is that we want our method to function in the noiseless and noisy cases at the same time. We added an extended related work section to the appendix where we fully detail the differences between our work and [Ueda and Washio, 2021].

Summarizing, in the underlying assumptions, we differ from [Ueda and Washio, 2021] since we assume that the noisy communication channel has external interference, which translates into the loss of information. On the other hand, [Ueda and Washio, 2021] assumes an adversarial setting where the objective is trying to deceive the Listener. Regarding the problem formulation, the game proposed by [Ueda and Washio, 2021] is a simpler version of our NLG game. In our case, we use natural images as input. Additionally, the Listener's discrimination task takes into account up to 4096 images to discretize, where in [Ueda and Washio, 2021] the input is a simple one-hot vector with a maximum size of 256. Moreover, since the input is one-hot encoded, the Listener does not need to receive candidates and just outputs a number between 1 and 256. For the expected outcomes, we show that our methods work on the noiseless and noisy case, creating robust communication protocols. On the other hand, [Ueda and Washio, 2021] only wants to force the communication protocol to create small messages by introducing adversarial noise, where a test between settings (train without noise and test with noise, and vice-versa) was never made or even intended.

Reviewer comment (Experiments):
The evaluation of the proposed architecture is focused on the newly developed referential game. It would be valuable to evaluate the architecture on a wider range of tasks and compare its performance to other architectures to assess its generalizability.

Author response:
In Appendix F, we introduce and evaluate the agents trained in every Lewis Game variation in other transfer learning tasks with the main purpose of testing their capability to generalize to new tasks.
    
Additionally, the vast majority of works in emergent communication are based fundamentally only on this task - a variation of the Lewis Game. The LG is a simple but highly customizable task that evaluates several properties of emergent languages, allowing the connection to human languages.

Reviewer comment (Writing):
While the paper mentions the capability of the proposed architecture to handle noise, there is limited discussion on how the architecture specifically addresses and mitigates the impact of noise in the communication channel. Providing more details on this aspect would enhance the clarity and understanding of the proposed method.

Author response:
In section 3.2, we motivate that our novel agent architecture perform better than the baseline. In section 3.2.1, we demonstrate that if we generalize the Lewis Game, where the communication channel can modify messages by introducing noise, our proposed architecture can still be robust and have high performance in both games (Lewis Game without noise and Lewis Game with noise). The second architectural change appears in section 2.2, where we define the MRILG (multi-round indecisive Lewis Game) and further extend the Listener architecture to contain an extra action, the *I don't know* action. When the Listener chooses this action, the pair of agents play a subsequent round in the Lewis Game, which allows the Listener to receive and process more information.
    
To give some additional information, the original Lewis Game (LG) is played only in one round - the Speaker receives an image and creates a message describing the image. The Listener tries to guess the image received by the Speaker over a set of candidate images taking only the message into account. In MRILG, several rounds of the LG can be played as long as the Listener chooses the idk action. During these several rounds, the Speaker's input image and the candidates' set are always the same.

Reviewer comment (Experiments):
Ablation studies, where different components or modules of the architecture are systematically removed or modified, could provide insights into the contribution and importance of each component. This would strengthen the analysis and understanding of the proposed architecture.

Author response:
Due to space constraints, ablation studies were left in the Appendix. Precisely, we focus on the Listener's attention mechanism, where we compare how introducing non-linearity affects the overall performance (Appendix D). Additionally, in Appendix E.4, we ablate different head configurations, taking into account the actor and critic heads.