Paper: Emergent Communication with Conversational Repair (ICLR-2024)

Reviewer comment (Novelty):
Section 2.2 on page 2 — Travis LaCroix's paper on reflexivity could be relevant.

Author response:
Regarding Travis LaCroix's paper on reflexivity
- Thanks, this paper offers new interesting perspectives. We integrated it into the discussion of the updated manuscript. The lack of compositionality found in our study could indeed provide further evidence that other aspects of human languages, such as reflexivity, should be taken into account.

Reviewer comment (Writing):
On page 5 — Replace "instable" with "unstable".

Author response:
Other points/typos:
- "instable" -> "unstable" and "lack vision-and-language" -> "lack of vision-and-language": Done

Reviewer comment (Evaluation):
On page 5 — It would be better to use 95% confidence intervals instead of raw standard error.

Author response:
Other points/typos:
- It would be better to use 95% confidence intervals instead of raw standard error: Done

Reviewer comment (Novelty):
On page 5 in the Discussion — "jointly co-constructing" seems relevant to Kottur et al. (2017); maybe include a sentence connecting the two works.

Author response:
Regarding the discussion: "jointly co-constructing" seems pretty relevant to Kottur et al. (2017), maybe a sentence connecting the two works would be appropriate
- This is indeed very related, we added a footnote (Footnote 7) to the updated version of the manuscript.

Reviewer comment (Writing):
On page 5 — Replace "lack vision-and-language" with "lack of vision-and-language".

Author response:
Other points/typos:
- "instable" -> "unstable" and "lack vision-and-language" -> "lack of vision-and-language": Done

Reviewer comment (Experiments):
The current experiments only show that communication with the receiver’s feedback will generalize better in a noisy environment. It would be more comprehensive to understand this feedback behavior with further analysis of the messages updated by the sender — based on the receiver’s feedback, how the sender’s messages vary across different time steps.

Author response:
This is very related to the previous question, and is depending on a better understanding of the feedback signal of the receiver.

Reviewer comment (Experiments):
The current experiments only show that communication with the receiver’s feedback will generalize better in a noisy environment. It would be more comprehensive to understand this feedback behavior with further analysis of whether, through multiple iterations, the receiver will provide less feedback or less informative feedback while they gradually build their common ground.

Author response:
This is a very important point. The nature of feedback during the training iterations is surely developing, however, we do not have a notion of what is “more” or “less” feedback as the feedback signal of the receiver is not directly interpretable (also the receiver is always sending a feedback message, there is currently no option to stay silent implemented). More generally, in the current setup there is always noise on the communication channel, which means that some form of feedback will stay useful even for 2 agents who already established a common ground. Future work could additionally explore agent populations, in which some agents speak more often to some than to others, in order to explore effects of pre-established common ground between speakers. (see also [1])

Reviewer comment (Evaluation):
The current experiments only show that communication with the receiver’s feedback will generalize better in a noisy environment. It would be more comprehensive to understand this feedback behavior with further analysis of whether the emerged languages with additional feedback make the sender’s messages of different objects more separatable.

Author response:
More distinct messages for different objects allow for a higher discrimination performance. As we indeed observe a higher test accuracy, we conclude that most likely these senders produce more separatable messages.

Reviewer comment (Experiments):
How does the game end? Will they communicate a fixed number of time steps?

Author response:
Yes, limited by the message length |M|

Reviewer comment (Experiments):
The drops in the compositionality are interesting. Since the sender’s message now depends on both the symbolic input and the receiver’s feedback, the semantic spaces could be disturbed. Have you tried to let the sender and the receiver share the same vocabulary?

Author response:
We do not intend to have the sender and receiver share the same vocabulary, as the idea is to create independent agents that only exchange information using the communication channel. As we are attempting to model human language evolution, we do not consider that a shared vocabulary is pre-existing between different agents.
It is true that the sender’s messages in the feedback case depend on both the input object and the receiver’s feedback. We tried calculating topsim scores that also take into account the receivers messages (by concatenating the sender messages and the receiver messages) and found that these scores are even lower than the original topsim scores (e.g. model with $p_{noise}=0.7$, $(|A|, |V|) =  (4, 4)$: topsim_sender_receiver=0.15+-0.02 | topsim=0.19+-0.02). 
More recently, a number of works also started criticizing the use of topsim to measure compositionality in emergent communication [2,3]. Our work adds additional evidence for the weakness of this metric.

Reviewer comment (Novelty):
While some details of the specific implementation are novel, a number of prior works have also introduced interactive repair mechanisms into signaling games and are not discussed. Even the classic Steels (1995) simulations included a form of (binary) repair. Several salient examples from more recent literature are omitted, including:  
- van Arkel, Woensdregt, Dingemanse, & Blokpoel. (2022). *A simple repair mechanism can alleviate computational demands of pragmatic reasoning: simulations and complexity analysis*. CoNLL.  
- Tria, Galantucci, & Loreto (2012). *Naming a structured world: A cultural route to duality of patterning*. PLoS ONE.  
- de Ruiter & Cummins. (2012). *A model of intentional communication: AIRBUS (Asymmetric Intention Recognition with Bayesian Updating of Signals)*. SemDial.  
- Silva & Roberts. (2016). *Exploring the role of interaction in the emergence of linguistic structure*. EVOLANG.  
- White, Poesia, Hawkins, Sadigh, & Goodman. (2022). *Open-domain clarification question generation without question examples*. EMNLP.

Author response:
Thanks for the very relevant references. We integrated them in the related work section of the updated manuscript and contextualize our work respectively. Our main contribution in comparison to these works is that we explicitly investigate the effect of a feedback channel by directly comparing models with and without feedback. Further, we employ deep-learning based modeling which scales to more realistic input, instead of only small-scale toy-language setups. This enabled us to perform the experiments described in Section 4.3.

Reviewer comment (Experiments):
The feedback channel only allows binary signals, and it is unclear how the results would be affected by a higher-dimensional space of feedback.

Author response:
It is true that the nature of the feedback will most likely change if the dimensionality of the feedback channel is increased. Instead of simple open clarification requests and/or acknowledgements, the model could leverage mechanisms such as restricted clarification requests or restricted offers. However, as the vocabulary size of the speaker is currently set to 2, we don’t think that an increase in the feedback channel dimensionality is actually useful in our setup. (We verify this claim by running a model with a feedback vocabulary size of 10, and find that it does not improve generalization performance: For a noise level of 0.9, the test accuracy drops from 0.75 (stddev: 0.07) to 0.70 (stddev: 0.07). As this slight drop is within the standard deviation, we assume it's caused by effects of the random initialization. Future work should investigate models with higher vocabulary sizes combined with increased feedback channel sizes.

Reviewer comment (Experiments):
Noise is only added to sender messages, but realistically noise would affect receiver feedback too, possibly shrinking the gap between the feedback vs. no-feedback models.

Author response:
Regarding noise on the feedback messages from the receiver: Yes, this has been discussed as a meaningful extension in Section 5, and will definitely be subject of future work.

Reviewer comment (Writing):
Typo: “defined by the number of symbols in the vocabulary V” — the vocabulary was previously denoted by X; V was defined as the set of possible values the attributes may take.

Author response:
Thanks as well for spotting the typo, this has been fixed.