Paper: It's About Time: Temporal References in Emergent Communication (ICLR-2024)

Reviewer comment (Reproducibility):
My biggest concern with this work is about understanding the actual approach the authors used for the neural implementation. Fundamentally, I do not understand how the authors order inputs over time and the structure of communication. I explain my question about this under "Questions" - generally, I view this uncertainty as a weakness given how central such details are to the paper. Certainly, I could be misunderstanding information presented in the paper, but I am very knowledgeable about EC, so I do not think I am missing something obvious.

Author response:
We respond to this weakness in the questions section.

Reviewer comment (Presentation):
The authors dedicate a fair amount of effort to establishing some notation based on LTL, but then do not seem to use it fully. For example, many of the figures (e.g., Figure 2) have almost natural-language y axis labels, whereas I would have expected a single mathematical symbol.

Author response:
Thank you for this suggestion. We will convert our notation for the figures to use the LTL based notation, as well as in other points in the text.

Reviewer comment (Evaluation):
The authors mostly focus on the emergence of temporal references, but further metrics would help situate the broader network performance. Notably, the authors do evaluate validation set accuracy in Figure 6, and they show a small decrease in performance for temporal networks relative to non-temporal networks. This is good analysis but then makes me 1) worry about the utility of this approach, if accuracy goes down and 2) slightly contradicts some of the claims in the main paper, which motivate temporal references to improve performance.

Author response:
While the accuracy does go down, we would consider this a negligible performance decrease, and only occurs when using the temporal prediction loss. This is especially true, considering that Temporal networks without the temporal prediction loss perform extremely similarly to our baselines, if not slightly better on the `RG Hard` dataset. Our claims about the increase in performance refer to improvements in bandwidth efficiency, were more complex environments considered, or if a length loss was applied. In these cases, temporal messages could 1) enable communication about previous events which would not be otherwise possible and 2) temporal references could be shorter when bandwidth efficiency is encouraged. In the revised version, we can further include metrics around compositionality, showing that temporal references do not impact it significantly (except for Temporal networks with the temporal prediction loss).

Reviewer comment (Reproducibility):
Footnote 1 appears to indicate that the authors will share code upon acceptance. I suggest the authors use https://anonymous.4open.science/ in future submissions to link to anonymized code during the review phase.

Author response:
We apologise for the confusion. The code is indeed hosted at <https://anonymous.4open.science/>, and the link in Footnote 1 is hyperlinked to our repository at https://anonymous.4open.science/r/TRG-E137.

Reviewer comment (Writing):
The notation around equation 2 is not right. In particular, typically I read the right text in case-based equations as specifying the conditions under which that behavior is selected. I think the cases should just be $c < 0.5$ and $c >= 0.5$, therefore, without any of the $x$ parts.

Author response:
We will remove the $x$ parts from the right side of the equation in the revised version.

Reviewer comment (Writing):
Equation 3 seems to have a few things off about it. First, "objectRepeated" seems misleading as a name. I believe it just returns true if $x_n$ is present $h_v$ episodes ago, correct? What does that have to do with repetition? Second, the variable $n$ is very overloaded in the expression, which impedes readbility. It is simulatenously used to represent a fixed number of previous timestep (see left side of the equation) and is used as a counter variable (see right side of the equation).

Author response:
We will rename the `objectRepeated` function to `objectSame` as that is what it would be checking, i.e., if the object $x_n$ is the same as the object $x_{n-h_v}$. We will also split the variable $n$ to improve readability.

Reviewer comment (Experiments):
My main question, which I hope the authors address in their rebuttal, is about better understanding the exact form of the inputs and outputs of the speaker network. The key idea of this paper is that speakers should be able to refer back to previous communication if they see similar inputs. Thus, the speaker must somehow see a series of inputs, right? How is that represented? Is it a batch of inputs? I'm guessing that is the case, because the temporal module in the speaker reshapes the inputs based on batch size, so that makes me think it allows the temporal module to represent at which point in "time" an input appears. If it is the case, however, that the temporal order of inputs is represented by position in the batch, I am somewhat disappointed in the baselines used for comparison. It seems obvious non-temporal networks, which observe inputs in parallel over a batch, cannot represent temporal relations. Overall, I am unsure about what the authors are actually doing to represent sequentially-ordered inputs, though, and really need clarity on this topic to evaluate this paper.

Author response:
Your description of the way we represent the series of inputs is correct. We indeed represent the temporal order of inputs by their position in the batch.

   However, we would disagree with this being disappointing. We agree that non-temporal networks cannot represent temporal relations. However, we include them as both a baseline in terms of the temporal metric, and **to prove** this intuition. We only include this simple trick, instead of more complex ways to learn temporal relationships (such as using attention networks, single LSTMs, or transformers, which we touch on in our response to Reviewer `RdRa`), to show that with just a simple modification to the LSTM the agents learn to exploit temporal relationships. We also show that no other losses are necessary. We consider the simplicity of both our setup and the ease of emergence of temporal relationships a positive aspect of our work, showing that the learnings are easily transferable to more complex settings and agent architectures.

Please refer to our response above.

Reviewer comment (Writing):
The notations in this paper need to be further standardised and unified. The current ones sometimes make readers (at least me) hard to follow. Here are the ones that I have spotted: (1) a vector in this paper sometimes means the feature vector of a target and sometimes a target sequence. Both of them can have arbitrary lengths and often appear without further explanation. (2) things like x \in V are not conditions in equation 2, IMO, they should not be connected with the "real" conditions with AND. Also, see my questions below.

Author response:
Thank you for the suggestion. We will unify the notation across the paper, and remove the extraneous information in Equation 2 in the revised version.

Reviewer comment (Experiments):
The paper spends quite a lot of space describing the details of the agents (e.g., the dimension of each vector during the implementation). Contrary to this is the absence of motivations for many choices made in the current study. For example, why the environment was defined in this way? Why a second LSTM in both sender and receiver? Why does there have to be an additional LSTM to capture temporal information? Actually, in terms of the architectural design, the two additional LSTMs in the sender and receiver have very different roles as the two LSTMs in the sender are parallel, and the two in the receiver are stacked.

Author response:
We define our `TRG Previous` environment to allow agents to develop temporal references. We additionally use notation from PLTL to bring rigour to our definitions. 
   In terms of the LSTMs — the second, temporal, LSTM is needed to capture temporal information as it is batched differently to the regularly employed LSTM, which in our case is the meaning LSTM. There does not necessarily need to be an additional LSTM, but it does improve performance, with one LSTM focusing on the meaning in the objects, and the other on their temporal relationships. We explore this briefly in `Section 3.3, p7`. 

    Indeed, the two LSTMs perform very different roles, as the sender agent has to understand the temporal relationships and the meaning of the object separately using two parallel LSTMs. The receiver LSTM first decodes the messages and then observes the temporal relationships between them. This was done to stay as close as possible to our base agents, as presented in EGG [1].

Reviewer comment (Experiments):
Related to my second concern, the experiment design in the present study also cannot validate the rationalities of these choices. This makes the conclusions from the experiments, namely a different batching strategy is sufficient, look narrow.

Author response:
We disagree that the experiment design makes our conclusions narrow. As touched on in our response to Reviewer `KopV`, we present this simple architecture to show that no significant changes are needed for the temporal references to emerge. While we have tested other architecture, such as transformer or attention-based, the simplicity of our model and heuristics allows for an easy transfer of these learnings to other settings.

Reviewer comment (Writing):
Page 3: c \leq 0.5 -> c < 0.5.

Author response:
Thank you for pointing out the typos. We will correct them in the revised version.

Reviewer comment (Writing):
I do not fully understand the definition of feature space in section 2.1. If F is a feature space of a feature, then why its size N_F is the number of features rather than the number of values that the associated property can take?

Author response:
We apologise for the confusion. We will clarify our definitions of features and properties by moving to the more standard attribute-value notation. In this case, the feature space would translate to the values that each attribute can have.

Reviewer comment (Novelty):
This paper does not make a substantial contribution to the field. While the TRG could elicit potentially interesting behaviors, the primary thrust of this paper is that it introduces a new architecture which is able to solve this game without much further analysis about temporality.

Author response:
We apologise for the confusion. Our focus in this paper was not to introduce a new architecture, but rather show how the emergence of temporal references could occur. While the architecture we present is indeed just an LSTM, we consider it a good baseline because it is a simple architectural change, and works to show that temporal references do not require new complex architectures. In addition, we evaluated architectures which contain _only_ the differently batched LSTM (`Section 3.3, p7`) which shows that allowing the agents to learn temporal references is the only required factor. Additionally, while we could not include this in the full text, we also considered different agent models, including transformers and attention-based architectures. While transformers would be very inefficient (workarounds have to be used to exploit the batching trick, as passing in a sequence of `batch_size` objects would generate a single `batch_size` message), attention-based networks work just as well as the simple LSTM.

In terms of the temporal loss, we introduce it to verify that additional losses are not required, by showing that `Non-Temporal-NL` networks can also develop temporal references. We modify the loss function to show the behaviour is not tied to a specific one, and will occur with just the regular loss function. We consider our exploration of the possible impact of the loss function similar to that of [1-4].

Reviewer comment (Novelty):
The new architecture is, to the best of my knowledge, just an LSTM where each timestep is a single round of the referential game. This seems like an appropriate baseline to work with and does not seem like a contribution in and of itself.

Author response:
We apologise for the confusion. Our focus in this paper was not to introduce a new architecture, but rather show how the emergence of temporal references could occur. While the architecture we present is indeed just an LSTM, we consider it a good baseline because it is a simple architectural change, and works to show that temporal references do not require new complex architectures. In addition, we evaluated architectures which contain _only_ the differently batched LSTM (`Section 3.3, p7`) which shows that allowing the agents to learn temporal references is the only required factor. Additionally, while we could not include this in the full text, we also considered different agent models, including transformers and attention-based architectures. While transformers would be very inefficient (workarounds have to be used to exploit the batching trick, as passing in a sequence of `batch_size` objects would generate a single `batch_size` message), attention-based networks work just as well as the simple LSTM.

In terms of the temporal loss, we introduce it to verify that additional losses are not required, by showing that `Non-Temporal-NL` networks can also develop temporal references. We modify the loss function to show the behaviour is not tied to a specific one, and will occur with just the regular loss function. We consider our exploration of the possible impact of the loss function similar to that of [1-4].

Reviewer comment (Novelty):
In Sec 1, "social deduction game" should cite related work (e.g., [this](https://www.semanticscholar.org/paper/RLupus%3A-Cooperation-through-emergent-communication-Brandizzi-Grossi/d94504e5319ea1a2dc7df8a1ad25a48a4c3ae650)).

Author response:
* Sec 1
    * > social deduction game: cite related work`
    * Thank you for pointing this out, we have added the citations.

Reviewer comment (Writing):
In Sec 2.1, "attribute-value" seems more standard than "property-feature"; the latter sounds like "property" and "feature" could refer to the same thing.

Author response:
* Sec 2.1
    * > "attribute-value" seems more standard than "property-feature"; the latter sounds like "property" and "feature" could refer to the same thing`
    * We have changed our wording to be more standard by using "attribute-value" terminology.

Reviewer comment (Theory):
In Sec 2.3, it seems like you could just parameterize the threshold for \(c\).

Author response:
* Sec 2.3
    * > Seems like you could just parameterize the threshold for $c$?`
    * > Alternatively, it would probably be even clearer to say that we sample $c$ from a Bernoulli distribution since that is in effect what is happening. Also, saying is uniformly sampled integer from $[0,1]$ is sort of confusing since interval notation refers to real values, not integers.`
    * Thank you for this suggestion. We will reword the description to use the Bernoulli distribution with $p=0.5$ for the base case and $p=0.25$ or $p=0.75$ for `Figure 3b`.

Reviewer comment (Writing):
In Sec 2.3, it would probably be even clearer to say that we sample \(c\) from a Bernoulli distribution since that is in effect what is happening. Also, saying \(c\) is uniformly sampled integer from \([0,1]\) is sort of confusing since interval notation refers to real values, not integers.

Author response:
* Sec 2.3
    * > Seems like you could just parameterize the threshold for $c$?`
    * > Alternatively, it would probably be even clearer to say that we sample $c$ from a Bernoulli distribution since that is in effect what is happening. Also, saying is uniformly sampled integer from $[0,1]$ is sort of confusing since interval notation refers to real values, not integers.`
    * Thank you for this suggestion. We will reword the description to use the Bernoulli distribution with $p=0.5$ for the base case and $p=0.25$ or $p=0.75$ for `Figure 3b`.

Reviewer comment (Writing):
In Sec 2.3, "verify whether the messages" should be revised to "determine whether or not the messages".

Author response:
* Sec 2.3
    * > "verify whether the messages" -> "determine whether or not the messages"`
    * We will change this wording in the revised version.

Reviewer comment (Reproducibility):
In Sec 2.4, "same as the EGG framework" is not a good way to specify the architecture since it is a whole framework and not a particular model. Maybe just take a sentence or two to specify the defaults that are being used.

Author response:
* Sec 2.4

    * > "same as the EGG framework" is not a good way to specify the architecture since it is a whole framework and not a particular model. Maybe just take a sentence or two to specify the defaults that are being used. p6
    * Thank you for this suggestion. We will include a short description in the revised version.

Reviewer comment (Theory):
In Sec 3.1, the \(100\%\) seems too restrictive to determine a word is properly temporal reference. For example, in English, is "yesterday" temporally referent? Because it can be used in a non-temporally referent way (e.g., "TensorFlow is yesterday's ML library.").

Author response:
* Sec 3.1

    * > The $100\\%$ seems too restrictive to determine a word is properly temporal reference. For example, in English, is "yesterday" temporally referent? Because it can be used in a non-temporally referent way (e.g., "TensorFlow is yesterday's ML library.").
    * We agree that $100\\%$ is restrictive. However, as we mention in our response to Reviewer $Avm5$, we use $100\\%$ as the cut-off to be able to conclude with certainty that we do observe temporal references. We agree with the example, that in English some words that are usually temporal references can be used in non-temporal contexts. However, this would be challenging to separate from accidental temporal references using the $M_{\ominus^n}$  metric (as we discuss in `Section 4, p2`). Instead, we look for words which would represent, for example, `2 minutes ago`, which would not be used in a non-temporal context.

Reviewer comment (Experiments):
In Sec 3.2, it seems like the non-temporal network is strictly precluded from performing temporal reference. How is this a useful baseline, then? Are we seeing if it can perform just as well or what the random baseline of temporal reference might be?

Author response:
* Sec 3.2

    * > It seems like the non-temporal network is strictly precluded from performing temporal reference? How is this a useful baseline, then? Are we seeing if it can perform just as well or what the random baseline of temporal reference might be?
    * As we touch on in response to Reviewer `KopV`, we include them as both a baseline in terms of the temporal metric, and to prove the intuition that they will not be able to form temporal references. Through this, we can show that only a simple modification is needed to the base agent to allow them to develop temporal references.

Reviewer comment (Writing):
In Sec 3.3, is \(M_{\ominus^4}\) the max value of any message? It seems like the temporality is per-message, not per-language.

Author response:
* Sec 3.3

    * > Is $M_{\ominus^4}$ the max value of any message? It seems like the temporality is per-message, not per-language.
    * The $M_{\ominus^n}$ indeed calculates temporality per message. We use the metric to calculate the value for each message in a given language, and these are the values we present in the figures. This is as we do not aim to find the average value for $M_{\ominus^n}$ for a language, as that could be misleading, due to the reasons outlined in `Section 4, p2`. Instead, we look for messages which achieve $100\\%$ on the $M_{\ominus^n}$ metric.

Reviewer comment (Evaluation):
In section 3.3, it is said “only networks that have the sequential LSTM are capable of producing temporal references”: From Figure 2a, we can observe all networks have cases “reaching % used as previous” above 80%. A clarification on how you make a judgment on “producing temporal references” should be made.

Author response:
Our cut-off for classifying a network as producing temporal references is $100\\%$, as a strong requirement, to ensure our reported results are significant. We choose to use the $100\\%$ to ensure that our positive observation is not a result of chance, such as in the case of the $80\\%$, where the temporal messages are misclassified due to naturally occurring repetitions in the dataset. We further explore this in `Section 4, p2`. We will clarify this in the revised version.

Reviewer comment (Evaluation):
In Figure 3b, only the results of Temporal networks are presented because “non-temporal networks learn no temporally specialised messages”. The threshold and significance between temporally and no temporally specialised messages should be made clear as well.

Author response:
We present the Non-Temporal variants in `Figure 3b`, so we assume this refers to `Figure 2b`.
   
    The threshold for `Figure 2b` is the same as we described in our response above - $100\\%$. Therefore, as the Non-Temporal networks never reach this threshold (exactly $0\\%$ of the networks) they are not included in this Figure. We will clarify this in revised version.
   
    Could you please clarify what you mean by `significance between temporally and no temporally specialised messages`?

Reviewer comment (Presentation):
In Figure 2, which of the six environments used for the legends about “trained RGs” and “TRGs” are not specified.

Author response:
Where we refer to the training environments `RGs` and `TRGs` in `Figure 2` we use the base environment, i.e., `RG Classic` and `TRG Previous`. The other variants (i.e., `RG Hard`, `TRG Hard`, `Never Same`, `Always Same`) are only used for evaluation, and not training. We will clarify this in the revised version.

Reviewer comment (Experiments):
For the temporal loss, I assume it is a categorical classification about the last position the object shows up. I think it could be misleading when trained with target object classification together. There are some circumstances when the objects are the same but the temporal labels are different, which may cause the network to be confused. Have you tried to add a smaller weight to the temporal loss or try to add this loss on top of the temporal LSTM?

Author response:
The loss indeed functions as you describe. However, we do not think it should interfere with the object classification as the two losses are computed independently, and use a different branch of the network. We have not experimented with different weighting of this loss, as it is not required for the emergence of temporal references, nor does it improve performance. We have also added this loss to the temporal LSTM, which is the `Temporal` variant, as opposed to the `Temporal-NL` variant which does not feature this loss. We have experimented with using this loss with just the temporal LSTM, removing the meaning LSTM, as we briefly discuss in `Section 3.3, p7`.

Reviewer comment (Evaluation):
More analysis of the features that the temporal LSTM provides would be helpful to demonstrate its vital role.

Author response:
As we touch on in our response to Reviewer `KopV` we can also include metrics regarding the language compositionality in the appendix of the revised version.