Paper: Emergent Language based Dialog for Collaborative Multi-agent Navigation (ICLR-2024)

Reviewer comment (Evaluation):
The paper does not provide a detailed analysis of the underlying rules that the emergent language adheres to, which may limit the understanding of the method.

Author response:
Thank you for pointing out this concern. There is no explicit underlying rules or heuristics that the emergent language adheres to. Actually, we performed intensive analysis of the emergent language, by detecting the alignment of emergent language with visual surrounding, and of emergent language with task, which we consider as two essential characters of language. In order to make further study on whether there are implicit rules underlying the emergent language, following [2], we analyzed the compositionality of emergent language, by calculating the topographic similarity between observation pairs and corresponding message pairs under two different settings. 

In the first setting, we calculate the editing distances of two dialogue histories up to the same turn $t$ as following:

$$LevenshteinDistance(H_{utterance, t}^{T, i}, H_{utterance, t}^{T, j})$$

$$H_{utterance, t}^{T, i} = \langle U_1^{T, i}, U_2^{T, i}, ..., U_t^{T, i}\rangle$$

$$H_{utterance, t}^{T, j} = \langle U_1^{T, j}, U_2^{T, j}, ..., U_t^{T, j}\rangle$$

In the second setting, we calculate the editing distances of two messages in the same turn $t$:

$$LevenshteinDistance(U_t^{T, i}, U_t^{T, j})$$

The curves of topographic similarity are shown in the Figure 5 in the paper. The first setting observes an uptrend in topographic similarity (colored in orange), while the second setting a downtrend (colored in blue). The uptrend suggests that the longer history conveys the more information. The downtrend suggests that each message progressively enriches information given the previous turns, and the messages at the beginning of the task carry more information than that of the later steps. As a metric evaluating the compositionality of the emergent language, this score suggests that the emergent language could composite task-specific features helpful for task solving.

Reviewer comment (Novelty):
The design of each module of the method is relatively conventional, and no particular contribution was found.

Author response:
Please refer to the response to Question 2.

Much existing research on navigation task concerns only language understanding ability of one agent. The most similar work is RMM [1], which focuses on training the natural language generation and understanding of two agents, under the supervision of a large scale of labelled dialogues. While our work concerns about the emergent language generation and understanding, without any labelled language data. The intrinsic advantage of our design is to train a multi-agent navigation model all from the scratch, without any human intervention, which is, to the best of our knowledge, the first work that proves the feasibility of multi-pass emergent communication in realistic dynamic environments, and provides an empirical method for the implementation.

Reviewer comment (Evaluation):
Can you provide some text examples to compare the differences between the text learned from the best baseline and the method learned from your own method?

Author response:
Among the mentioned baselines, RMM [1] is the only work concerning language generation ability. Unfortunately, it’s hard to reproduce RMM to get sufficient text examples for comparison. We take the example provided in the RMM paper, run our model using the same start position and the same target, and compare the emergent texts of our model with the natural language of RMM, as depicted in Appendix Table 5. There is no obvious alignment between the natural language and the emergent language. This observation draws a similar conclusion as that of Appendix A, i.e., the emergent language captures different views about the surrounding and the task ontology with the natural language learned from a large scale of human-annotated data. This will lead to an interesting content of our future work.

Reviewer comment (Novelty):
Can you explain again what advantages the design of each module in the method has compared to the previous method?

Author response:
Much existing research on navigation task concerns only language understanding ability of one agent. The most similar work is RMM [1], which focuses on training the natural language generation and understanding of two agents, under the supervision of a large scale of labelled dialogues. While our work concerns about the emergent language generation and understanding, without any labelled language data. The intrinsic advantage of our design is to train a multi-agent navigation model all from the scratch, without any human intervention, which is, to the best of our knowledge, the first work that proves the feasibility of multi-pass emergent communication in realistic dynamic environments, and provides an empirical method for the implementation.

Reviewer comment (Evaluation):
The major weakness of the paper is the lack of deep analysis of the emergent language. Computing metrics of emergent languages like topographic similarity (using environment encodings and agent messages) [1] could give an idea of the structure of the agents' protocols.

Author response:
Thanks very much for the suggestion, inspired by your suggestions we have conducted some analysis about the emergent language, and have added it to the paper. More efforts will be made to have a deeper analysis of the language. 

we conducted an experiment to analyze the compositionality of emergent language, by calculating the topographic similarity between observation pairs and corresponding message pairs under two different settings. 

In the first setting, we calculate the Levenshtein distances of two dialogue histories generated by the Tourist up to the same turn $t$.

$$LevenshteinDistance(H _{utterance, t}^{T, i}, H _{utterance, t}^{T, j})$$

$$H _{utterance, t}^{T, i} = \langle U _1^{T, i}, U _2^{T, i}, ..., U _t^{T, i}\rangle$$

$$H _{utterance, t}^{T, j} = \langle U _1^{T, j}, U _2^{T, j}, ..., U _t^{T, j}\rangle$$

In the second setting, we calculate the Levenshtein distances of two messages in the same turn $t$.

$$LevenshteinDistance(U _t^{T, i}, U _t^{T, j})$$

The curves of Topographic similarity are shown in Figure 5 in the paper. The first setting observes an uptrend in topographic similarity (colored in orange), while the second setting is a downtrend (colored in blue). The uptrend suggests that the longer history conveys the more information. The downtrend suggests that each message progressively enriches information given the previous turns, and the messages at the beginning of the task carry more information than those of the later steps. As a metric evaluating the compositionality of the emergent language, this score suggests that the language could composite task-specific features helpful for task solving.

Those results and more details have been added to the paper and we are trying hard to explore more characteristics of the emergent language.

Reviewer comment (Theory):
At the end of the related work section the authors claim: "we also empower the oracle agent to provide instructions progressively". It is unclear how it is guaranteed that messages are sent progressively and not all at the beginning. The reviewer is wondering whether this is backed by any analysis work or just by the nature of the sequential communication modules.

Author response:
The claim “we also empower the oracle agent to provide instructions progressively “is backed by the experiment results on different settings of dialogue strategy which is shown in Table 4. When comparing the “only in beginning” setting with other settings with more dialogue frequency, it observes that the “only in. beginning” setting has a poorer performance, indicating that providing all information only at the beginning is not enough, and the message in each turn conveys additional helpful information for the following task. To better verify this point, we conducted experiments of “direct communication” (i.e., sending the shortest path repeatedly in each turn) and observed a similar performance with that of the “only in beginning” setting (tl=10.09, ne=6.88, sr=25.74, spl=22.78, vs tl=10.79, ne=6.44, sr=25.37, spl=21.13), proving that the subsequent utterances provide additional helpful information instead of repeating the same information. Our further study also found that especially when the Tourist made wrong movement actions, the multi-turn communication could provide helpful instructions, leading to a much better performance.

Reviewer comment (Experiments):
In the training setup section, it is unclear why the two objective “pre-training” tasks are called pre-training. The reviewer questions whether they are actually used jointly to train the agents, as the division between pre-training and training is not clear from the paper.

Author response:
At the very beginning of our research, we tried to directly train a multi-agent navigation model in an end-to-end manner using reinforcement learning, and observed a discouraging non-convergence of the training. Under this setting, we jointly trained the language generation module and understanding module for both agents (i.e., four modules in total) by simultaneously maximizing the whole accumulated rewards. However, the model collapsed into a poor status since the generation and understanding are not coordinated at the start of training which can cause the state to quickly become large and chaotic. Therefore, we formulate the navigation task as a two-pass task and train each pass separately in a supervised learning manner: In the first pass (the so-called “Localization” task), the Tourist was encouraged to learn how to generate language accurately describing its visual surrounding, and the Guide to correctly understand the message from the Tourist to predict where the Tourist is, with the goal of making proper guideline for future steps. In the second pass (the so-called “Movement” task), the Guide was encouraged to generate language accurately describing its guideline, and the Tourist to correctly understand the message conveying the guideline, in order to make proper movement. Then we returned back to jointly training for the navigation task as mentioned at the beginning, but with parameters initialized by the two-pass training tasks. Therefore, we call each pass a pre-training task of the targeted navigation task, the principled fundamentals of which are explained in Section 4.5.

Reviewer comment (Experiments):
The choice of a vocabulary size of 26 is unclear. The reviewer initially thought it was to draw a similarity with the English language, but then considered that it might be the result of a hyperparameter optimization following an ablation study, as suggested by Figure 3b. Clarification is requested.

Author response:
Yes, your thought is right. We do choose the vocabulary size of 26 to draw a similarity with the English language. The experiment in Figure 3b is to explore the effect of the vocabulary size on the performance and the 26 may not be the best choice among all possible choices of vocabulary size in the proposed navigation task.

Reviewer comment (Reproducibility):
In Section 5.5, the method of computing the reduced 2D space is not explained. Additional details are requested, which could be added to the appendix for the camera-ready version.

Author response:
In Figure 4 (a) of section 5.5, we use PCA as a dimension reduction technique to reduce the encoded Tourist’s visual image $\bar{e} _o^t \in \mathbb{R}^{dim}$, and encoded dialog history $\bar{e} _{H _{dialog}}^G \in \mathbb{R}^{dim}$ into 2D space. Similarly, in Figure 4 (b), the encoded dialogue history $\bar{e} _{H _{dialog}}^T \in \mathbb{R}^{dim}$, and the encoded next step action $\bar{e} _p^{pos _i \rightarrow pos _j } \in \mathbb{R}^{dim}$ are reduced via PCA dimension reduction tool. Then, in Figure 4 (a), we cluster the encoded visual images using KMeans and connect the center of the clusters and the center of the corresponding messages, then we find the parallel. Similarly, in Figure 4 (b), we cluster the encoded next step action using KMeans and connect the center of the clusters and the center of the corresponding messages. Thanks for your advice, and we have added those details to the appendix.

Reviewer comment (Evaluation):
The paper does not sufficiently assess the quality and characteristics of the emergent language, such as its compositionality and its relation to the plan set out by the Guide.

Author response:
Thank you for your suggestion, and we had noticed this problem before but did not make much attempt at it. We performed language analysis by detecting the alignment of emergent language with visual surroundings, and that of emergent language with task, which we consider as the essential characters of language. Motivated by your suggestion, we conducted an experiment to analyze the compositionality of emergent language, by calculating the topographic similarity between observation pairs and corresponding message pairs under two different settings. 

In the first setting, we calculate the Levenshtein distances of two dialogue histories generated by the Tourist up to the same turn $t$.

$$LevenshteinDistance(H_{utter, t}^{T, i}, H_{utter, t}^{T, j})$$

$$H_{utter, t}^{T, i} = \langle U_1^{T, i}, U_2^{T, i}, ..., U_t^{T, i}\rangle$$

$$H_{utter, t}^{T, j} = \langle U_1^{T, j}, U_2^{T, j}, ..., U_t^{T, j}\rangle$$

In the second setting, we calculate the Levenshtein distances of two messages in the same turn $t$.

$$LevenshteinDistance(U_t^{T, i}, U_t^{T, j})$$

The curves of Topographic similarity are shown in the Figure 5 of the paper. The first setting observes an uptrend in topographic similarity (colored in orange), while the second setting is a downtrend (colored in blue). The uptrend suggests that the longer history conveys the more information. The downtrend suggests that each message progressively enriches information given the previous turns, and the messages at the beginning of the task carry more information than those of the later steps. As a metric evaluating the compositionality of the emergent language, this score suggests that the language could composite task-specific features helpful for task solving.

Those results and more details have been added to the paper and we are trying hard to explore more characteristics of the emergent language.

Reviewer comment (Presentation):
The figures contain very small text, making them hard to understand.

Author response:
Thanks for your suggestion. We do find the font size is too small when we zoom in, which is unfriendly to readers. We have adjusted (and further improvements will be made) Figures 2 & 3 carefully and hope it could improve the readability of this paper.

Reviewer comment (Writing):
While the main contribution seems to be the proposal of the task, a large portion of the description is dedicated to the network architecture (Section 3). The authors should provide a more intuitive explanation of the general task framework. Adding pseudocode could help potential readers grasp the proposal more effectively.

Author response:
Thank you for your suggestion and the old version is indeed not clear enough. We have added the pseudocode and the descriptions of the baseline methods to the Appendix (and more improvements will be made). And hope the revised version could be more friendly to readers.

Reviewer comment (Experiments):
The characteristics and details of the compared baseline methods in the experiments are not clear. It would be beneficial to include these descriptions in the Appendix.

Author response:
Thank you for your suggestion and the old version is indeed not clear enough. We have added the pseudocode and the descriptions of the baseline methods to the Appendix (and more improvements will be made). And hope the revised version could be more friendly to readers.

Reviewer comment (Experiments):
The most straightforward communication approach the Guide could adopt is to repeatedly send the shortest path. How does the emergent language compare with such direct communication? If this hasn't been explored, it would be worthwhile to discuss.

Author response:
Thank you for your thought-provoking question. We had conducted experiments in a similar setting, referring to “only in beginning” in Table 4. We let the Tourist encode the shortest path instruction as a part of dialog history in each turn. Inspired by your question, we immediately conducted experiments using “direct communication” (i.e., sending the shortest path repeatedly in each turn) and observed a similar performance with that of the “only in beginning” setting (tl= 10.09, ne=6.88, sr=25.74, spl=22.78, vs tl = 10.79, ne=6.44, sr=25.37, spl=21.13). By comparing with other results in Table 4, both performance of “direct communication” and “only in beginning” are poorer than that under other settings (“soft”, “20%”,  “50%”,  “100%”), suggesting that the dialogue progressively provides additional information. Our further study also found that especially when the Tourist made wrong movement actions, the multi-turn communication could provide helpful instructions, leading to a much better performance.

Reviewer comment (Experiments):
Based on the motivation of this work, CVDN is a natural task for this method. It is better to show the performance on the test splits (unseen) comparing with SoTA methods, rather than only showing the val split.

Author response:
we thank the reviewer for giving us valuable advice. The R2R and CVDN datasets are initially designed for natural language navigation between two agents, therefore their test data are used to evaluate the alignment quality of natural language with the environment, which are demonstrated on the corresponding leaderboards (but with the environment settings unknown to the public). Our work focuses on emergent language communication therefore it is infeasible to draw a comparison with related work on the leaderboard using test data. However, in order to assess the language understanding and generation ability of our method, we train and evaluate it under paralleled environment settings (i.e., with the same visual groundings of train and val dataset) with the related work on R2R and CVDN. We have mentioned this in section 5.1 and will give more explanations in the future revised version.

Reviewer comment (Experiments):
Is it possible to show the results for Test Unseen on R2R? besides the results on val seen and unseen. Similarly for CVDN dataset.

Author response:
we thank the reviewer for giving us valuable advice. The R2R and CVDN datasets are initially designed for natural language navigation between two agents, therefore their test data are used to evaluate the alignment quality of natural language with the environment, which are demonstrated on the corresponding leaderboards (but with the environment settings unknown to the public). Our work focuses on emergent language communication therefore it is infeasible to draw a comparison with related work on the leaderboard using test data. However, in order to assess the language understanding and generation ability of our method, we train and evaluate it under paralleled environment settings (i.e., with the same visual groundings of train and val dataset) with the related work on R2R and CVDN. We have mentioned this in section 5.1 and will give more explanations in the future revised version.

Reviewer comment (Presentation):
Figure 2 & 3, the font is too small to read.

Author response:
Thanks for your suggestion. We do find the font size is too small when we zoom in, which is unfriendly to readers. We have adjusted (and further improvements will be made) Figures 2 & 3 carefully and hope it could improve the readability of this paper.