Paper: Online Continual Learning for Interactive Instruction Following Agents (ICLR-2024)

Reviewer comment (Theory):
conceptually, the task boundariless scenario in embodied instruction following AI is very limited, since the user will know when to switch to a new task

Author response:
$\to$ First, we guess the ‘task boundariless’ you refer to is the ‘task free’ setup (in the second paragraph of Sec.4 and the ‘Ours vs. Distillation-based models.’ paragraph in Sec.5.1, and we will add more details about task-free setup in Sec.2 of the revision). We respectfully argue that it may not be trivial for the user to know when to switch to a new task. For example, when the data is injected in a Gaussian scheduled *continuous* scenario [1], the data distribution imperceptibly changes over time as there are no explicit task boundaries, and even the user cannot tell where the task boundary is. To address a large set of scenarios including the ones mentioned above, we devise a *task-free* method.

[1] Murray Shanahan, Christos Kaplanis, and Jovana Mitrovic. Encoders and ensembles for task-free continual learning. arXiv preprint arXiv:2105.13327, 2021.

Reviewer comment (Experiments):
another baseline I could think of is to use the loss as the weight to dynamically change the old logits. What are some intuition on why confidence is better than loss to use to dynamically weight?

Author response:
$\to$ Interesting suggestion! While using loss values can potentially serve as a strong baseline as the reviewer mentioned, we choose to use confidence scores considering the aspects below.

First, while a confidence score ranges from 0 to 1, the loss functions usually have unbounded ranges. For the weighted moving average, the weights are expected to be in the range from 0 to 1 and the sum of the weights becomes 1. Therefore, it is not suitable to use the unbounded loss value as the weight. As an alternative, we use the confidences which are bounded.

Second, employing the loss may result in marginal updates to the stored logit, especially in the early stages of training. This leads to a slow improvement of teacher logits for distillation. On the other hand, using confidence enables quicker updates with well-learned logits, leading to a rapid update to more informative logits. This is because widely used loss functions adopt log-based functions of probability such as cross-entropy losses, while confidences are probability values themselves. Due to the logarithmic scale of loss functions, loss values obtained by low probabilities could be quite small, and thus, the moving average with such small values might lead to marginal updates on the logits. As we can see in Figure 3, the confidence of a stored logit and a newly encountered logit is near 0, *i.e.*, the prediction probability of a stored logit and the newly introduced logit is $p$ and $p+\epsilon$, respectively, where $p$ and $\epsilon$ is positive but near 0.  In this case,  a stored logit needs to be updated using a newly introduced logit (*i.e.*, logits obtained by a model that is slightly more trained) since a logit with a high probability of the ground truth label implies that it learned well about that class, as we mentioned above. However, if we use the loss as a criterion of CAMA, the stored logit will be marginally updated since ($log(p)$) becomes nearly infinite when $p$ is near 0 in  (*i.e.*, $\gamma \approx 0$). In contrast, since confidences are probability values themselves, they can be updated more quickly with better logits during the early stages of training, leading to a faster improvement in distillation performance.

Reviewer comment (Writing):
equation (1), the $\gamma$  in front of $z$ is it missing the subscript $a$ and $c$? or is it a fixed value?

Author response:
$\to$ Yes, the subscripts of $a$ and $c$ were missing in $\gamma$ and we corrected them in the revision. Thank you for pointing this out.

Reviewer comment (Reproducibility):
I am not too familiar with the benchmark, but when is the object class used when making decisions? if you need to heat a mug, does your agent also need to recognize the object in front of it is a mug too?

Author response:
$\to$ Yes, the agent needs to recognize the object (*i.e.*, a mug) in front of it and predict the object class, “mug,” if the agent intends to interact (*e.g.*, pick up, put, *etc.*) with it.

Reviewer comment (Evaluation):
Depending on the availability of compute it would be beneficial to increase the number of seeds (3->5) to reduce the standard errors of the results. At the moment the standard errors are often larger than the gaps in mean performance.

Author response:
$\to$ As suggested, we are running our experiments in 2 more random seeds (*i.e.*, total of 5 different random seeds) for the Environment-IL setup, compare the best-performing baselines (*i.e.*, CAMA w/o D.C. and DER++), and add the result in Table 5 of the revision. With smaller standard errors, we observe that our CAMA still outperforms the baselines by noticeable margins. We will update the 5-seed results for the other baselines in the final manuscript.

**Table A. The Environment-IL setup.**
| Model | Val. Seen $SR_{last}\uparrow$ | Val. Seen $GC_{last}\uparrow$ | Val. Seen $SR_{avg}\uparrow$ | Val. Seen $GC_{avg}\uparrow$ | Val. Unseen $SR_{last}\uparrow$ | Val. Unseen $GC_{last}\uparrow$ | Val. Unseen $SR_{avg}\uparrow$ | Val. Unseen $GC_{avg}\uparrow$ |
|----------------------|---------|---------|---------|----------|-----------|-------------|---------------|---------------|
| CAMA | $29.42 \pm 0.34$ | $38.14 \pm 1.10$ | $34.87 \pm 2.46$ | $43.79 \pm 2.84$ | $14.37 \pm 0.47$ | $30.31 \pm 0.41$ | $15.90 \pm 0.95$ | $33.34 \pm 1.87$ |
| CAMA w/o D.C. | $30.95 \pm 0.72$ | $38.36 \pm 0.51$ | $33.70 \pm 2.48$ | $42.48 \pm 2.34$ | $12.58 \pm 0.89$ | $28.24 \pm 0.63$ | $13.13 \pm 0.52$ | $29.44 \pm 0.90$ |
| DER++ | $26.06 \pm 1.35$ | $35.83 \pm 0.60$ | $28.51 \pm 2.93$ | $37.66 \pm 3.63$ | $11.93 \pm 1.21$ | $28.63 \pm 1.51$ | $10.85 \pm 1.87$ | $26.38 \pm 2.21$ |

Reviewer comment (Evaluation):
As shown in Figure 3, when a new tasks arrives confidence drops. However, there is also evidence that in OOD cases a model can be confidently wrong (Do Deep Generative Models Know What They Don't Know?, Nalisnick et. al.). How detrimental would that be for performance?

Author response:
$\to$ The concern raised by Nalisnick et al. is about models exhibiting overconfidence in *incorrect classes*. We utilize confidence scores of only the *correct* classes (*i.e.*, ground-truth classes), thus there are no detrimental occurring due to the overconfidence problem.

Reviewer comment (Writing):
Abstract, last sentence: "outperforms prior arts", was that supposed to be "outperforms prior state of the art"?

Author response:
$\to$ Prior art and the state of the art are regarded as synonym (Please refer to the ‘Patent Law’ section of https://en.wikipedia.org/wiki/State_of_the_art). We will clarify it by “outperforms state of the arts.” Thank you!

Reviewer comment (Evaluation):
The performance of the "finetune" baseline with respect to the $SR_{last}$ and $GC_{last}$ metrics. If I understand the baseline correctly is that the currently trained policy is finetuned on the data of the next task without any memory. Should the performance on the final task not be . Why is the performance on the .

Author response:
$\to$ Yes, as the reviewer mentioned, the “finetune” baseline is finetuned on the next task’s data without memory (*i.e.*, without any previous tasks’ data).

$\to$ We assume that the last part of the question is “Should the performance on the final task not be $SR_{avg}$. Why is the performance on the $SR_{avg}$.” (please correct us if we are wrong). The $SR_{avg}$ and $GC_{avg}$ metrics *do not* require the usage of memory. In other words, we measure metrics (*i.e.*, SR and GC) whenever right before receiving samples of a new behavior or environment and take the average of the metrics obtained at the end of each behavior or environment.

Reviewer comment (Reproducibility):
Which episodes are stored in the memory M? Is it simply a queue? If yes then given that its capacity is such a small percentage of the total number of episodes (as said in the text that is the standard) and given the number of tasks for each of the evaluation episodes will most episodes not be about the same same task such that the logits there should be replaced with the current logits of the model?

Author response:
$\to$ The memory used is not a queue but an $M$-dimension array that does not have any temporal dependency on the order of incoming and outcoming examples. To address the concern that the current episodes may use the majority of memory space as the reviewer pointed out, we adopt reservoir sampling (Vitter et al., 1985, Rolnick et al., 2019, Buzzega et al., 2020) that enables us to maintain the stored episodes in memory approximately uniform across sequential tasks (*i.e.*, behaviors and environments).

Reviewer comment (Experiments):
In the CAMA w/o D.C. baseline, was the fixed coefficient tuned for optimal performance of the baseline?

Author response:
$\to$ Yes, we tuned the coefficient to obtain the best valid unseen $SR_{last}$ for an arbitrarily chosen random seed in a setup (here, Environment-IL), among $\gamma={0.99, 0.9, 0.79, 0.5, 0.39, 0.01}$ and use it across all random seeds.

To find the coefficient tuned for Environment-IL setup, we grid-search the best coefficients for the best performance (*i.e.*, the highest Val. Unseen $SR_{last}\uparrow$, following the same evaluation protocol of the ALFRED benchmark (Shridhar et al., 2021)), and summarize the results in Table A below. We observe that 'CAMA w/o D.C.’ achieves slightly better performance values than 'CAMA’ in some metrics (*e.g.*, Val. Unseen $SR_{last}\uparrow$). However, it underperforms CAMA in the other metrics by noticeable margins and requires additional searches for the coefficients for different setups.

As we mentioned above, please note that it is often impractical to find the coefficients for the best performance considering all setups due to limited access to continuous data (*i.e.*, we do not know which data will come in). Thus, we used the same hyperparameter $\gamma$ for all setups including the  Environmental-IL and Behavior&Environment-IL setups.

**Table A. The Environment-IL setup.** "CAMA w/o D.C." uses $\gamma_a = \gamma_c = 0.9$ found to yield the highest Val. Unseen $SR_{last}\uparrow$.
| Model | Val. Seen $SR_{last}\uparrow$ | Val. Seen $GC_{last}\uparrow$ | Val. Seen $SR_{avg}\uparrow$ | Val. Seen $GC_{avg}\uparrow$ | Val. Unseen $SR_{last}\uparrow$ | Val. Unseen $GC_{last}\uparrow$ | Val. Unseen $SR_{avg}\uparrow$ | Val. Unseen $GC_{avg}\uparrow$ |
|----------------------|---------|---------|---------|----------|-----------|-------------|---------------|---------------|
| CAMA | $32.47 \pm 5.53$ | $41.22 \pm 3.68$ | $38.75 \pm 3.52$ | $47.77 \pm 3.24$ | $13.89 \pm 1.75$ | $31.17 \pm 1.81$ | $14.92 \pm 2.75$ | $34.18 \pm 2.15$ |
| CAMA w/o D.C. | $30.27 \pm 1.61$ | $38.91 \pm 1.95$ | $38.44 \pm 3.89$ | $47.64 \pm 2.89$ | $14.08 \pm 1.32$ | $31.21 \pm 1.51$ | $14.31 \pm 2.89$ | $33.12 \pm 1.17$ |

Reviewer comment (Experiments):
I think some of the assumptions this paper makes are not realistic. E.g., the paper indicates that this approach is task-free and that task-identifiers are not available, but then appears to assume that the data is balanced. In a real-world continual learning setup there is no guarantee that the tasks/data the agent encounters will be balanced, and so if data balance is enforced via sub-sampling I think this is implicitly leaking information regarding task identifiers (as they would be needed to balance an unbalanced dataset in practice). An analysis on how class/environment/behavior imbalance affects the model would be beneficial here.

Author response:
> I think some of the assumptions this paper makes are not realistic. E.g., the paper indicates that this approach is task-free and that task-identifiers are not available, but it appears to assume that the data is balanced. But in a real-world continual learning setup, there is no guarantee that the tasks/data the agent encounters will be balanced, and so if data balance is enforced via sub-sampling I think this is implicitly leaking information regarding task identifiers (as they would be needed to balance an unbalanced dataset in practice).

$\to$ For data balance, we follow common practice in previous continual learning literature (Wu et al., 2019, Aljundi et al., 2019, Bang et al., 2021, Boschini et al., 2022).  However, we agree with the reviewer in that we have no guarantee that the data that the agent encounters will be balanced. To investigate the imbalance scenario as suggested, we are performing additional experiments by removing the subsampling process in the Environment-IL setup and we will upload the result in the revision.

Although we balance the training and evaluation episodes, we *do not* exploit any task-specific information or identifiers during the training and evaluation process, thus believe there is no information leakage. Since we denote by ‘task-free’ the absence of task identifiers *during the training process* (Aljundi et al., 2019), not task identifiers themselves. We added a discussion about task-free continual learning setups in “Task-free continual learning setups” in Sec. 2 of the revision.

To consider a data-imbalance scenario, we consider *intra-task* imbalance, *i.e.*, an imbalance between object and action classes in each task to construct our proposed incremental setups. For *inter-task* balance, we followed previous works (Wu et al., 2019, Aljundi et al., 2019, Bang et al., 2021, Boschini et al., 2022) that assume balanced distribution between tasks (*i.e.*, assuming inter-task balance) for simplicity of setups. To mitigate severe inter-task imbalance in the environments in the ALFRED benchmark, we subsample the same number of episodes across all environments for inter-task balance.

However, we agree that there is no guarantee of encountering inter-task-balanced data. To investigate our approach in such inter-task imbalance, we construct a modified version of the Environment-IL setup by omitting subsampling and summarize the result in Sec. E.1 of the revision. In this setup, an environment type (*i.e.*, Kitchens) has three times more episodes than the other environment types and thus, this setup considers inter-task imbalance as well.

> An analysis on how class/environment/behavior imbalance affects the model would be beneficial here.

$\to$ As the reviewer suggested, we are performing additional experiments on the Environment-IL setup with imbalances between environments (*i.e.*, without subsampling). We will upload the revision with the results soon.

Reviewer comment (Evaluation):
Is there a concern about over/under-confidence with such an approach? Presumably, such a model is not calibrated since the model is continually out-of-distribution. Given that modern deep neural networks are particularly susceptible to confidence issues, an analysis on this aspect would be useful.

Author response:
> Is there a concern about over/under-confidence with such an approach? Presumably, such a model is not calibrated since the model is continually out-of-distribution. Given that modern deep neural networks are particularly susceptible to confidence issues, an analysis on this aspect would be useful.

$\to$ To investigate the confidence issue that the reviewer mentioned, we calibrate the confidence scores of our CAMA using temperature scaling [1] and summarize the results below. As shown in the table, we do not observe significant performance differences, implying that the confidence issues may not be a primary concern in the proposed approach.

[1] Guo et al. On calibration of modern neural networks. ICML, 2017.

**Table A. The Environment-IL setup.**
| Model | Val. Seen $SR_{last}\uparrow$ | Val. Seen $GC_{last}\uparrow$ | Val. Seen $SR_{avg}\uparrow$ | Val. Seen $GC_{avg}\uparrow$ | Val. Unseen $SR_{last}\uparrow$ | Val. Unseen $GC_{last}\uparrow$ | Val. Unseen $SR_{avg}\uparrow$ | Val. Unseen $GC_{avg}\uparrow$ |
|----------------------|---------|---------|---------|----------|-----------|-------------|---------------|---------------|
| CAMA | $32.47 \pm 5.53$ | $41.22 \pm 3.68$ | $38.75 \pm 3.52$ | $47.77 \pm 3.24$ | $13.89 \pm 1.75$ | $31.17 \pm 1.81$ | $14.92 \pm 2.75$ | $34.18 \pm 2.15$ |
| CAMA w/ calibrated | $29.80 \pm 1.90$ | $38.73 \pm 1.63$ | $37.70 \pm 3.39$ | $47.44 \pm 2.71$ | $13.83 \pm 3.66$ | $30.49 \pm 0.63$ | $13.36 \pm 2.54$ | $31.85 \pm 2.27$ |

Reviewer comment (Experiments):
Does performance depend on the order of the tasks/environments? I see reference to random ordered sequences in Sec. B.3, but it's not clear to me how this ties into the training/evaluation methodology.

Author response:
> Does performance depend on the order of the tasks/environments? I see reference to random ordered sequences in Sec. B.3, but it’s not clear to me how this ties into the training/evaluation methodology.

$\to$ Yes, performance depends on the order of the tasks and this is also widely observed in continual learning literature (Prabhu et al., 2020, bang et al., 2021, koh et al., 2022, ). To avoid favoring a particular order of the behaviors and environments, we validate our method with multiple random orders and report their mean and standard error ( sentence in Sec. 3.2.1).

For the training/evaluation methodology, we use the same training and evaluation method across the entire randomly ordered sequences (*i.e.*, it does not require any additional tuning for a specific order), which enables our method to be applied to arbitrary sequences of the behaviors and environments.

Reviewer comment (Experiments):
Is there any insight on how the size of the memory buffer relates to the number of learning tasks? Is 500 an arbitrary number or was this empirically chosen?

Author response:
> Is there any insight on how the size of the memory buffer relates to the number of learning tasks? Is 500 an arbitrary number or was this empirical chosen?

$\to$ We set the memory size to *less than 5%* of the training dataset following a common practice on continual learning literature (Prabhu et al., 2020; Bang et al., 2021; Koh et al., 2022; 2023). Specifically, we set our memory size to 500, which is approximately 2.38% of the training episodes in the ALFRED benchmark.

Reviewer comment (Theory):
While the empirical results seem promising, the reviewer is not fully convinced by the proposed confident-aware moving average approach. Specifically, it is unclear why the average of most recent N confidence scores could be a good indicator of the ‘quality’ of the newly obtained logits. More concretely, the behavior $t_i$ of the current episode is different from the behavior $t_j$ that is associated with the sampled past experience $x’$. Why would confidence scores of current episodes with behavior $t_i$ provide information on a different behavior $t_j$.

Author response:
$\to$ We believe that the confidence scores of the current episodes with a behavior, $t_i$, can provide information on a different behavior, $t_j$, because the actions (*e.g.*, turn left, pick up, *etc.*) and object classes (*e.g.*, apple, pen, *etc.*) learned for the current task (*i.e.*, behavior and environment) can be also used for the previous tasks and therefore, achieving high accuracy of the actions and object classes for the current task can positively affect the previous tasks. For example, an agent that previously learned how to heat a mug may now learn to move two mugs on the table for the current task. To complete the current task, the agent should first learn to plan actions (*e.g.*, turn left/right, move ahead, pick up, *etc.*) and predict relevant object classes (*e.g.*, mug) to reach and interact with a mug. It is noteworthy that these learned actions and object classes are also used for the previous task (*i.e.*, heat a mug) as the agent should also reach and interact with a mug to heat it.

Reviewer comment (Writing):
In equation (1), subscript $a, c$ are missing in $\gamma_a$ and $\gamma_c$.

Author response:
$\to$ Thank you for the comment!. We revised the equation (1) ($\gamma$ -> $\gamma_a$, $\gamma$ -> $\gamma_c$) in the revision.

Reviewer comment (Experiments):
This paper uses imitation learning to train the agents. Is the proposed approach extendable to a reinforcement learning setting?

Author response:
$\to$ Great suggestion! Yes, our proposed approach can be extendable to a reinforcement learning setting. Once an action logit, $z_{old,a}'$, is stored in episodic memory for the current state, $s$, and action, $a$, the next state, $s’$, and the corresponding reward, $r$, resulting in the form of $(s, a, s’, r)$, we can maintain $(s, a, s’, r, z_{old,a}')$ and given a new logit, $z_a'$, for the same current state, we can update the previous logit, $z_{old,a}'$, in the same way as CAMA, resulting in $(s, a, s’, r, z_{new,a}')$ where $z_{new,a}'$$ = (1-\gamma_a) z_{old,a}' + \gamma_a z_a'(z_{new,c}'$ can be updated in the same way if applicable). Note that we do not assume the usage of imitation learning for training.