From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: This paper proposes a simple algorithm in a meta-continual learning scenario where the task boundaries are not known and distribution shift can occur. Based on the observation that the loss values sharply increase at the boundary, the paper introduces two threshold values for detecting task boundaries and distributional shifts selectively control between task-specific learning and meta-updates. Empirical studies on three different benchmarks demonstrate the effectiveness of the proposed algorithm compared to related baseline approaches.

The empirical results of the proposed algorithm are promising. Moreover, the paper claims that this choice "is also well supported by our theoretical analysis of dynamic regret in online meta-learning [...]" (see abstract and introduction). However, the main concern pointed out by a majority of expert reviewers is the mismatch between the proposed algorithm and theoretical analysis. The paper makes a simplifying assumption of known task boundaries, which effectively renders the theoretical analysis irrelevant to the proposed algorithm (Reviewer ZoyY, 1HZs, and HHCs). The reviewers also question the technical novelty of the proof techniques used under this simplifying assumption (Reviewer ZoyY and 1HZs). Finally, there remains some confusions on how this setting is related to existing working in the online/meta learning literature. Hence, the clarity issue remains at large. 

As a result, I cannot recommend this paper in its current form for publication at ICLR 2023.
justification_for_why_not_higher_score: The paper requires a major revision and thereby another round of peer review.
justification_for_why_not_lower_score: N/A
summary_of_AC-reviewer_meeting: N/A

From: ICLR.cc/2023/Conference/Paper3799/Authors
title: Response to further questions 
comment: **Q:** From my understanding your MetaOGD baseline is basically OGD, which also does not specify how to update meta parameters, and your theoretical analysis is of an algorithm which also needs to know the task boundaries. The question here can be thought of as an ablation: does your theoretical analysis matter, or is the empirical improvement over say MetaOGD due to pretraining or some other aspect of your method? 

**A:** We clarify that the setting we analyze in theory does not fully capture all novel designs in our algorithms. As we mentioned before, our theory focuses only on justifying the OGD perspective. Some main novel elements of our algorithm such as the detection of task boundaries and domain boundaries, and how the meta parameters are updated with respect to task and domain changes are left out. The advantages of those designs are reflected mainly in Section 5 where our experiments demonstrate that our algorithm significantly outperforms the state-of-the-art baselines. Hence, we kindly request the reviewer to focus on **Section 3** to evaluate the novelty of our algorithm, and **Section 5** to evaluate how our algorithm outperforms the state-of-the-art baselines.

The baseline MetaOGD is a natural **extension** of OGD to the **online meta learning** setting. It computes the meta loss for the current task as in MAML (i.e., adapt on a support set and compute the meta loss on a query set using adapted parameters) and takes a gradient step based only on that current meta loss. MetaOGD always updates the **meta parameters** in that manner regardless of whether there is a task/distribution switch or not. We also kindly let the reviewer know that MetaOGD is not considered as the state-of-the-art in online meta learning. 

The advantage of our algorithm over MetaOGD is not due to the pretraining (in fact we always perform pretraining whenever it is helpful and this is for all baselines). Instead, the empirical advantage of our algorithm over MetaOGD comes from two key aspects of our algorithm: (1) Our algorithm can effectively reuse the current task knowledge whenever there is no task switch, instead of adapting from the meta parameters at every time instance; (2) Our algorithm adopts different updating schemes for the meta parameters based on whether the task at hand belongs to the pretraining distribution or not. The second point allows our algorithm to keep knowledge of pretraining tasks while also effectively improving performance on out-of-distribution tasks. Our experimental results indeed confirm these key advantages of our algorithm over not just MetaOGD but all compared baselines.

From: ICLR.cc/2023/Conference/Paper3799/Reviewer_ZoyY
title: Response
comment: From my understanding your MetaOGD baseline is basically OGD, which also does not specify how to update meta parameters, and your theoretical analysis is of an algorithm which also needs to know the task boundaries. The question here can be thought of as an ablation: does your theoretical analysis matter, or is the empirical improvement over say MetaOGD due to pretraining or some other aspect of your method?

From: ICLR.cc/2023/Conference/Paper3799/Authors
title: Response to further questions 
comment: **Q1:** To me, the most interesting part is the proposed detection mechanism that helps to identify the change in the task boundary, but its effectiveness is only justified by experiments. I agree that this is not a theoretical paper and the paper enjoys its empirical values. But, given the proposed detection criterion is very similar to the previous work Liu et al, 2020, its empirical success would not be that surprising. The lack of theoretical justification on the detection mechanism could make this paper less appealing. 

**A1:** We can analyze the detection scheme by itself. However, incorporating such an analysis to regret analysis for online meta learning is out of capacity of the online learning community. Besides, while the detection of distribution shift is based on the previous work Liu et al. 2020, it hasn’t been used in online meta learning to detect task boundaries before. More importantly, it serves as only one block in our proposed algorithm, which cannot guarantee superior performance by itself. It is not clear apriori how to design a lightweight system to efficiently update the meta model and online task model under distribution shifts, even if the task boundaries are detected accurately. 

**Q2:** Besides, it seems to me the proposed bound can be obtained by applying Lemma 1 of Jadbabaie et al.,2015 when $P_T$ is known. Note that their bound holds for any benchmark sequence $u_t$, and one can choose $u_t$ as the piece-wise static sequence to obtain the proposed bound in the paper with the optimal step size. In this sense, I still find the paper somewhat lacking in novelty. 

**A2:** We agree with the reviewer’s interpretation and our theorem can be obtained by applying the result in Jadbabaie et al. (2015). Please note that we have been emphasizing a few times that our theory simply serves a supporting role, and our main contribution to the paper is on the experimental side. 

We want to call the reviewer’s attention to the fact that while in theory the inner-loop of meta-learning was captured together with outer-loop function **generically** as a convex function, online meta learning has its own empirical challenges. Indeed the experimental considerations in online meta learning go much further beyond such an abstraction, e.g., how many steps to update variables in the inner loop, how frequently to update the meta variable in the outer-loop, how to determine task boundary, how to determine domain boundary, how to improve memory efficiency, etc. These important issues are not captured by the abstract theoretical model but addressed by our proposed algorithm design, which consequently leads to our main contributions for the meta-learning community in terms of addressing the challenging non-stationary online meta-learning without knowing the task boundaries.

From: ICLR.cc/2023/Conference/Paper3799/Authors
title: Response to further questions 
comment: **Q1:** While setting Phi to the identity in the result of Hall & Willett (2015) indeed ignores the dynamical aspect, it still yields a dynamic regret bound that I believe is as good or better than the one here. See their statement: Notice also that if Φt(θ) = θ in for all t, then Theorem 2 gives a novel, previously unknown tracking regret bound for COMID. Furthermore, as pointed out by Reviewer 1HZs, Lemma 1 of Jadbabaie et al. (2015) is indeed applicable here. 

**A1:** We agree with Reviewer 1HZs’ interpretation and our theorem can be obtained by applying the result in Jadbabaie et al. (2015). Please note that we have been emphasizing a few times that our theory simply serves a supporting role, and our main contribution to the paper is on the experimental side. So we request the reviewer again to evaluate our paper only based on our experimental contributions. In fact, it is common that experimental online meta learning papers have only experiments (Caccia et al. (NeurIPS’20), Yao et al. (NeurIPS’20), Rajasegara et al. (2022)).  

**Q2:** It is unclear to me why your solutions to experimental considerations could not be applied to the task-boundary-agnostic algorithms of Hall & Willett (2015) and Jadbabaie et al. (2015) or the task-boundary-aware online-within-online algorithms of Denevi et al. (2019) and Khodak et al. (2019). I imagine the former would result in something similar to what you call MetaOGD, except presumably with pre-training. 

**A2:** We don’t exactly understand the question. Perhaps the reviewer can further elaborate the point here. Hall & Willett (2015) and Jadbabaie et al. (2015) are on vanilla online learning, and their algorithms won’t be directly applicable to our online **meta** learning setting, because they don’t specify how to update meta parameters. Algorithms in Denevi et al. (2019) and Khodak et al. (2019) need to know the task boundary, and are not applicable to our setting.

From: ICLR.cc/2023/Conference/Paper3799/Reviewer_ZoyY
title: Response
comment: While setting Phi to the identity in the result of Hall & Willett (2015) indeed ignores the dynamical aspect, it still yields a dynamic regret bound that I believe is as good or better than the one here. See their statement: *Notice also that if Φt(θ) = θ in for all t, then Theorem 2 gives a novel, previously unknown tracking regret bound for COMID.* Furthermore, as pointed out by Reviewer 1HZs, Lemma 1 of Jadbabaie et al. (2015) is indeed applicable here. 

It is unclear to me why your solutions to experimental considerations could not be applied to the task-boundary-agnostic algorithms of Hall & Willett (2015) and Jadbabaie et al. (2015) or the task-boundary-aware online-within-online algorithms of Denevi et al. (2019) and Khodak et al. (2019). I imagine the former would result in something similar to what you call MetaOGD, except presumably with pre-training.

From: ICLR.cc/2023/Conference/Paper3799/Authors
title: Response to further questions
comment: **Q:** Fair, Jadbabaie et al. (2015) compare to the optimal action at each step. Perhaps more relevant is the result of Hall & Willett (2015) with Phi=Identity. More broadly, the concern with the theoretical analysis extends to the empirical results, because if the analysis is used to justify the method, and if there are existing analyses with fewer assumptions or better guarantees, then why are those methods not baselines in the experiments?

References:

Hall, Willett. Online Optimization in Dynamic Environments. IEEE Journal of Selected Topics in Signal Processing - Signal Processing for Big Data 9(4). 2015.

**A:** We respectfully disagree with the reviewer about the suggested reference. The approach in Hall & Willett (2015) assumes that the shifts in the environment can be taken care of by modulating the online parameters with known dynamic models $\phi_t$. For example in their Algorithm 1 (DMD method), setting the modulating functions $\phi_t$ to IDENTITY (see eq. 1b in [Hall ‘15]) would make their method ignore the dynamical aspect of the environment. In fact when $\phi_t$ = IDENTITY, their Algorithm 1 is equivalent to the Composite Objective Mirror Descent (COMID) [Duchi ‘10] method which is not designed for the dynamic environments setting and achieves sublinear regret for the stationary case. 

On the other hand, their algorithm 2 (DFS method) requires the knowledge of the number $N$ of domains to be encountered during online learning as well as their corresponding dynamics $\phi$’s, and uses the Fixed Share [Herbster ‘98] algorithm to average the predictions obtained from the different domains. However, such a method (DFS) has an unfair advantage (in terms of resource budget) over methods that use a single fixed capacity predictor (the setup considered in our work), because DFS requires knowing and storing the dynamic models for all domains to be encountered during online learning. 

More importantly, for all references suggested by the reviewer (including those in the reviewer’s previous comments), none of them consider online **meta** learning, which is the central topic of this paper. Hence, none of those algorithms can serve as baselines for online **meta** learning. While in theory the inner-loop of meta-learning was captured together with outer-loop function **generically** as a convex function, the experimental considerations go much further beyond such an abstraction, e.g., how many steps to update variables in the inner loop, how frequently to update the meta variable in the outer-loop, how to determine task boundary, how to determine domain boundary, how to improve memory efficiency, etc. These important issues are not captured by the abstract theoretical model but addressed by our proposed algorithm design, which consequently leads to our main contributions for the meta-learning community in terms of addressing the challenging non-stationary online meta-learning without knowing the task boundaries. For experimental comparisons, the online **meta** learning community has well established experimental baselines that have publicly available code for researchers to use and compare, particularly for online few-shot learning. In our experiments, we compare with those baselines except for recent FOML [Rajasegaran ‘22], which doesn’t have code available online, and for which we compare with our own implementation of the algorithm. 

References:

[Hall ‘15] Hall & Willett. Online Optimization in Dynamic Environments. IEEE Journal of Selected Topics in Signal Processing - Signal Processing for Big Data 9(4). 2015. 

[Duchi ‘10] J. Duchi, S. Shalev-Shwartz, Y. Singer, and A. Tewari, “Composite objective mirror descent”, in Conf. on Learning Theory (COLT), 2010. 

[Herbster ‘98] M. Herbster and M. K. Warmuth, “Tracking the best expert”, Machine Learning, vol. 32, pp. 151-178, 1998. 

[Rajasegaran ‘22] Jathushan Rajasegaran, Chesea Finn, and Sergey Levine. Fully online meta-learning without task boundaries. arXiv preprint arXiv:2202.00263, 2022.

From: ICLR.cc/2023/Conference/Paper3799/Reviewer_1HZs
title: Update
comment: After reading the authors' feedback and other reviews, my concerns about the mismatch between the proposed algorithm and regret analysis still remain.

To me, the most interesting part is the proposed detection mechanism that helps to identify the change in the task boundary, but its effectiveness is only justified by experiments. I agree that this is not a theoretical paper and the paper enjoys its empirical values. But, given the proposed detection criterion is very similar to the previous work Liu et al, 2020, its empirical success would not be that surprising. The lack of theoretical justification on the detection mechanism could make this paper less appealing.

Besides, it seems to me the proposed bound can be obtained by applying Lemma 1 of Jadbabaie et al.,2015 when $P_T$ is known. Note that their bound holds for any benchmark sequence $u_t$, and one can choose $u_t$ as the piece-wise static sequence to obtain the proposed bound in the paper with the optimal step size. In this sense, I still find the paper is somewhat lacking of novelty. Given the above concerns, I still hope to keep my score.

From: ICLR.cc/2023/Conference/Paper3799/Reviewer_ZoyY
title: Response
comment: Fair, Jadbabaie et al. (2015) compare to the optimal action at each step. Perhaps more relevant is the result of Hall & Willett (2015) with Phi=Identity. More broadly, the concern with the theoretical analysis extends to the empirical results, because if the analysis is used to justify the method, and if there are existing analyses with fewer assumptions or better guarantees, then why are those methods not baselines in the experiments?

### References:
- Hall, Willett. *Online Optimization in Dynamic Environments*. IEEE Journal of Selected Topics in Signal Processing - Signal Processing for Big Data **9** (4). 2015.

From: ICLR.cc/2023/Conference/Paper3799/Authors
title: Response to further questions 
comment: Q1: With respect to the removed assumption (Q2), it seems all discussion of when Assumptions 1 and 2 hold were removed entirely. Conditions under which such (notably non-standard, unlike claimed in the paper) assumptions hold should be stated in the paper. 

A1: To clarify, we did not remove assumption 2 but we removed an inaccurate statement which claimed when this assumption could hold. Thanks for the suggestion. We fully agree with your suggestion that the conditions under which the assumptions hold should be kept. We will add it to the paper in the later version (we are unable to upload a new revision now). 

What we plan to add is as follows: 

Assumption 2 will hold in the following case. Following from Theorem 1 in Finn et. al (2019), when the function $f_k^d$ is strongly-convex (with the additional smoothness conditions in Assumption 1 in Finn et. al (2019)) and $U_k^d$ is the one step gradient descent map, then the composition $f_k^d \circ U_k^d$ will be convex (it will also be strongly-convex). 

Q2: With respect to the regret analysis (Q3), based on the updates to the theorem to remove the square from the path length, it seems the result here is basically the same as Jadbabaie et al. (2015), except they do not need to know the task-boundaries. In-fact, based on the fact that the algorithm being analyzed is the one in (8), I believe their result can just directly be applied without doing any math. 

A1: We respectfully disagree with the reviewer’s comments. The setting studied in Jadbabaie et al. (2015) is different from ours. In Jadbabaie et al. (2015), the task distribution changes every time instance, so that the comparator $f_t(x_t^*)$ is applied to every time instance. However, in our setting, we have a hierarchy of tasks. Namely, tasks follow the same distribution under a certain domain $d$, which spans over $K_d$ time instances. Hence, over such  $K_d$ time instances, the static regret is applied. Then tasks can switch to a different domain $d+1$, where they follow a different distribution. Hence across domains, dynamic regret is applied. Thus, our theoretical analysis combines techniques from the static and dynamic regret analysis. For any given domain index $d$, we do a static regret analysis and then we telescope the results for different domain $d$’s using dynamic regret analysis techniques. Even though our analysis follows by adapting the previous analysis tools, it appears that such a result cannot be obtained by directly applying previous results. On the other hand, we don’t take this theory as the main contribution of this work, rather a useful support of the value of considering some overlooked aspects in empirical online meta-learning study. Hence, we want to call the reviewer’s attention  to the practical aspect of the paper, which significantly improves the performance of the SOTA of online meta learning.

From: ICLR.cc/2023/Conference/Paper3799/Reviewer_ZoyY
title: Response
comment: Thank you for the update. I have some further comments:
1. With respect to the removed assumption (Q2), it seems all discussion of when Assumptions 1 and 2 hold were removed entirely. Conditions under which such (notably non-standard, unlike claimed in the paper) assumptions hold should be stated in the paper. 
2. With respect to the regret analysis (Q3), based on the updates to the theorem to remove the square from the path length, it seems the result here is basically the same as Jadbabaie et al. (2015), except they do not need to know the task-boundaries. In-fact, based on the fact that the algorithm being analyzed is the one in (8), I believe their result can just directly be applied without doing any math.

From: ICLR.cc/2023/Conference/Paper3799/Authors
title: Response to Reviewer HHCs 
comment: We thank the reviewer for providing the helpful review! We have addressed the reviewer’s helpful comments and modified the paper accordingly. Please note that in the revised paper, we highlighted our changes by blue-colored texts in both the main body and the appendices of the paper. 

Q1: Task switch detection heuristic does not correspond to dynamic regret algorithm techniques from the online learning literature. The paper lacks details on how this heuristic (threshold) is tuned in practice. 

A1:  (1) For the energy threshold $\tau$: we follow the strategy in [Liu 20’], i.e., we set the threshold $\tau$ using the pre-training tasks. More specifically, we set $\tau$ so that 95% of the pre-training inputs are correctly detected as pre-training data. 

(2) For the threshold $\ell$: in standard few-shot learning experimental setups, the true labeling for each individual task is usually randomly chosen. When there is a task switch, the task specific model learnt from the previous task generally does not fit the new task anymore, where the learning performance could be similar to that of a random model. Thus motivated, we find that a good heuristic to choose a starting value for the threshold $\ell$ is to use the loss value evaluated on a random model. For example, for 10-ways classification tasks that value would be $\ell_r = -\log(1/10) = 2.3$. 

(3) In our revision we provide new experiments in Appendix A on the sensitivity of our algorithm w.r.t. the thresholds $\tau$ and $\ell$. We found that when the threshold $\ell$ is too small, the algorithm tends to over detect task switches, which results in inferior performance of the algorithm due to ineffective reuse of task knowledge. On the other hand when $\ell$ is too large, the high miss detection rate results in the algorithm mostly fine-tuning the online task model $\phi_t$ with the current task support data. As expected, this results in a failure mode due to the adversariality of different tasks (please see Appendix A for more details). 
The results also show that larger values of $\tau$, which collapse to updating the meta-model at each step (even for pretraining task distribution), does not substantially improve the performance. This demonstrates the advantage of the distinct meta-update scheme proposed for in- and out-of-distribution tasks, which avoids unnecessary frequent meta-updates for the pretraining tasks and thus allows a more judicious usage of computational budget. Lower values of $\tau$ (e.g. $\tau=15$) tends to detect all task distributions as the pertaining one, and thus corresponds to eliminating the domain adaptation component of our algorithm. 

Q2: The paper emphasizes the setting where task boundaries are not known, but without comment the analysis (e.g., Theorem 1) appears to assume task distributions are switching at a regular cadence. 

A2: In our theory, each distribution changes after some online steps $K_d$ (which may vary w.r.t. the distribution index $d$), and thus the frequency of the switching is not fixed. 

Q3: Algorithm 1 line 6 uses $\ell$ which was not listed as an input or indicate how it is computed. 

A3: We thank the reviewer for pointing this out. $\ell$ should be listed as input and we corrected this in our revision. 

Q4: what is $g_k$ ("the prediction model's output?") mentioned after equation (4)? 

A4:  $g_k$ is the  $k$-th entry of the logits vector predicted by the neural network model.

From: ICLR.cc/2023/Conference/Paper3799/Authors
title: Continued response to Reviewer HHCs 
comment: Q5: when describing the experimental setup in section 5, the moderately stationary case has a higher task switch probability than the low stationary case. Also the phrase "where $p$ is very small" makes no sense. 

A5: We thank the reviewer for noticing this, but in fact $p$ corresponds to the probability of NOT switching as defined towards the end of section 2 of the paper. We mistakenly called it probability of switching in the specific sentence the reviewer was referring to, but we corrected this in our revision. We hope this clarifies the reviewer’s doubts. 

Q6: Better describe wtf you are actually doing to set the temperature. Provide some experimental commentary on the sensitivity of the technique to the choice of temperature. 

A6: We thank the reviewer for this important suggestion. In our revision, we include new experiments in Appendix A on the sensitivity of our algorithm w.r.t. to the temperature as well as the thresholds $\ell$ and $\tau$. For the temperature, we found that simply setting it to $\delta =1$ yields the best performance, and large values of $\delta$ eliminates the effectiveness of the energy-based detection module (figure 6 (b) in Appendix A). 

Q7: quantify how homogenous (or not) the task loss distributions are in the setups you are using to give an idea of why the heuristic is working and when it would fail. 

A7: In figure 1, we provide the variations of the task loss during online learning, which is the observation that supports our heuristic for task boundaries detection. We hope that this answers the reviewer’s question but if the reviewer is asking for a different plot, we kindly ask the reviewer to be a little more specific about the requested plot and we will do our best to obtain it in our revision. 

Q8: provide some commentary relative to what dynamic regret algorithms actually do to detect environment drift: they do regret bound arithmetic, e.g. [Wei 21’].

A8: The paper [Wei 21’] uses a simple detection mechanism based on value function changes (for a Reinforcement Learning setting), so that the value function change can be incorporated into the path length (which has a similar form as value function change). This can then be included in the regret bound. However, in our case the detection metrics (e.g.  the energy metric) is very different from the form of path length as well as the regret metric. So it is much harder to be incorporated into the regret analysis compared to the detection scheme in [Wei 21’] which uses function values. Further, even for their detection scheme based on function values, it requires a complicated design of the algorithm and it is not clear whether their resulting algorithm can be implemented efficiently in practice (in fact their paper did not provide any experiments for that matter). Also their regret analysis requires some assumptions on the task changes which may not hold for the online meta-learning case. 

Finally, we thank the reviewer again for the helpful comments and suggestions for our work. If our response resolves your concerns to a satisfactory level, we kindly ask the reviewer to consider raising the rating of our work. Certainly, we are more than happy to address any further questions that you may have.

References: 

[Liu 20’] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. Advances in Neural Information Processing Systems, 33:21464–21475, 2020. 

[Wei 21’] Chen-Yu Wei and Haipeng Luo. Non-stationary Reinforcement Learning without Prior Knowledge: An Optimal Black-box Approach. Arxiv 2102.05406

From: ICLR.cc/2023/Conference/Paper3799/Authors
title: Response to Reviewer 1HZs
comment: We thank the reviewer for providing the helpful review! We have addressed the reviewer’s helpful comments and modified the paper accordingly. Please note that in the revised paper, we highlighted our changes by blue-colored texts in both the main body and the appendices of the paper. 

Q1: My main concern about the paper is the mismatch between the theoretical analysis and the proposed methods. The authors have proposed two heuristics to detect the shift on tasks and task distribution, which are the main algorithm contributions of this paper. But, the theoretical analysis is irrelevant to the proposed mechanism. 

A1: We thank the reviewer for the comments. We would like to make the following clarifications. 

(i) Our theoretical analysis assumes that the detection mechanisms are correct every time. Thus, our theory is to support that our algorithm will have theoretically grounded performance as long as the detection mechanisms are accurate enough, which is indeed the case as demonstrated in our experiments (please see ablation studies on detection accuracies in section 5.2, and we also provide new sensitivity experiments in Appendix A). We also want to emphasize that an important goal of our regret analysis is to motivate some aspects of our algorithm design choices, such as updating the meta model based only on the current data is sufficient for achieving sublinear regret, which is usually not adopted in the online meta-learning community. Our results suggest that the community could also learn from careful explorations of simpler designs, besides emphasizing algorithmic complexities. 

(ii) It appears that the current regret analysis tools for online learning are simply not yet ready to capture such detection mechanisms. In our case the detection metrics (e.g.  the energy metric) is very different from the regret metric, which makes it very hard to be incorporated into the regret analysis. The problem may be simplified if we assume the detection component as a blackbox that yields the miss detection and false alarm rates. In this case it might be possible to incorporate the detection parameters into the regret analysis, which is an interesting and open problem to explore in the future and could be very useful to the broader online learning community. 

Q2: About the novelty of the dynamic regret bound. The analysis for the dynamic regret seems a direct application of that for the OGD algorithm [Zinkvich, 2003] under the convexity assumption (Assumption 2). In this sense, the theoretical technical novelty of this is somewhat limited to me. 

A2: We are aware that the dynamic regret analysis may not be brand new to the online optimization community. But our analysis still has some novelty as it combines techniques from the static and dynamic regret analysis. For any given domain index $d$ we do a static regret analysis and then we telescope the results for different $d$’s using dynamic regret analysis techniques. 
We also call the reviewer’s attention that the goal of our paper is mainly practice driven and the theory is to support some overlooked aspects in online meta-learning. As such we propose a simple algorithm that can detect task boundaries and distribution shifts and achieves superior performance, compared to the state-of-the-art methods for online meta-learning in changing environments and without knowing task-boundaries. Thus, we kindly ask the reviewer to evaluate our paper with an emphasis on the empirical contributions and consider our theory as an extra effort. 

Q3: About the correctness of the dynamic regret bound. It seems to me that $P_T$  in Theorem 1 should be defined as $P_T=\sum‖\theta_d^∗−\theta_{d+1}^∗‖$ instead of the second-order formulation. It is unclear to me why the second inequality of Eq. (13) holds. 

A3: We thank the reviewer for noticing this. The “square” was in fact a typo and we corrected it in our revision. We hope this clarifies the reviewer’s doubts. 

Q4: About the claim on the sublinear regret: It seems that the proposed regret bound will suffer from a linear regret when $P_T=O(T)$. Therefore, the claim on the sublinear dynamic regret is somewhat an overclaim to me. 

A4: If the nature of the problem is such that $P_T=O(T)$, then one cannot expect to have an algorithm with sublinear regret. We do not believe such a statement is an overclaim and this is also a common claim in the dynamic regret literature.

From: ICLR.cc/2023/Conference/Paper3799/Authors
title: Continued response to Reviewer 1HZs 
comment: Q5: About the experiments: To validate the effectiveness of the proposed detection scheme, it would be better to provide a more detailed inspection of the shift detection instead accuracy instead of just providing the comparison on the overall performance. 

A5: We compare the precision and recall of the detection schemes used in the relevant methods, in order to validate the effectiveness of the proposed detection schemes. This is also standard in the related literature [Caccia 20’]. We kindly ask the reviewer to be more specific about the detailed inspection they suggested and we will do our best to include them in our revision. 

Q6: Should it be $L(θ_{t−1};S_t)$  in Line 14 of Algorithm 1?  

A6: Yes. We thank the reviewer for pointing this out. We corrected it in our revision. 

Q7: The axis and legend of the figures are not clear enough. 

A7: We thank the reviewer for the suggestions. We are still working on improving the quality of the paper. We will make sure to make everything look as clear as possible in our final version. 

Q8: The proposed detector for distribution shift detection is novel to me, but they are not brand new since it is established on the OOD detection proposed by previous works. The theoretical analysis is also a standard analysis for the OGD algorithm. 

A8: The task switch detection is based on our empirical observation depicted in figure 1. The domain shift detection uses an energy-based approach from the OOD literature, but this is the first time that this mechanism is adapted to online meta learning. Building upon these two detection schemes allows us to propose a novel online meta-learning approach that can effectively reuse the current task knowledge while also incorporating the new meta-knowledge whenever necessary.  

Our theoretical analysis combines techniques from the static and dynamic regret analysis. For any given domain index $d$ we do a static regret analysis and then we telescope the results for different $d$’s using dynamic regret analysis techniques. 
We also call the reviewer’s attention that the goal of our paper is mainly practice driven and the theory is to support some overlooked aspects in online meta-learning. As such we propose a simple algorithm that can detect task boundaries and distribution shifts and achieves superior performance, compared to the state-of-the-art methods for online meta-learning in changing environments and without knowing task-boundaries. Thus, we kindly ask the reviewer to evaluate our paper with an emphasis on the empirical contributions and consider our theory as an extra effort. 

Finally, we thank the reviewer again for the helpful comments and suggestions for our work. If our response resolves your concerns to a satisfactory level, we kindly ask the reviewer to consider raising the rating of our work. Certainly, we are more than happy to address any further questions that you may have.

From: ICLR.cc/2023/Conference/Paper3799/Authors
title: Response to Reviewer ZoyY
comment: We thank the reviewer for providing the helpful review! We have addressed the reviewer’s helpful comments and modified the paper accordingly. Please note that in the revised paper, we highlighted our changes by blue-colored texts in both the main body and the appendices of the paper. 

Q1: The switch detection mechanisms seem interesting, although I do not know if they are novel; are they connected to change-point detection algorithms? A comparison with past work such as CMAML++ would also be useful. 

A1: The task switch detection is based on our empirical observation depicted in figure 1. The domain shift detection uses an energy-based approach from the OOD literature, but this is the first time that this mechanism is adapted to online meta learning. Building upon these two detection schemes allows us to propose a novel online meta-learning approach that can effectively reuse the current task knowledge while also incorporating the new meta-knowledge whenever necessary.  
Differently from the change-point detection algorithms, where the goal is to detect distribution change of data, here we detect the change of tasks, i.e., whether the current online model can be applied to a new task. 

Q2: The statement “as shown in Finn et al. (2019), Assumption 2 holds when the function is convex and the adaptation mapping corresponds to the one-step gradient descent” is incorrect; the relevant statement in that paper (Theorem 1) holds only for strongly convex, strongly smooth, and Hessian smooth functions, and it is unclear if the result can be extended to convex functions without making the step-size of the adaptation mapping zero and thus uninteresting in this context. 

A2: We agree with the reviewer about the inaccuracy of this statement. The function should be strongly convex (with additional smoothness assumptions), and we have fixed it in our revision.

From: ICLR.cc/2023/Conference/Paper3799/Authors
title: Continued response to Reviewer ZoyY
comment: Q3: It is a bit difficult to understand exactly which algorithm Theorem 1 gives guarantees for, but my understanding is that they are for Algorithm 1 assuming the detection mechanisms are correct every time. What this means is that, while the paper is advertised as not knowing the task boundaries, for the actual guarantee the task boundaries are indeed known. The issue then becomes that the authors do not compare to the trivial baseline, which is restarting gradient descent whenever a new task is detected; in the regime where all tasks have at most K rounds doing this has dynamic regret O(D*sqrt(K)), which is better than the provided guarantee unless the path-length is o(sqrt(D)). If the task-boundaries are known, the setting studied by the authors is in-fact more akin to that of online-within-online meta-learning (Denevi et al., 2019; Khodak et al., 2019), where you have a sequence of tasks with some number of rounds per task; the only difference is here the loss functions are composite (but convex, as in those papers). Notably, Theorem 3.3 of the latter gives a guarantee of O(D^(3/4)*sqrt(K)) as a special case, which is better than the provided guarantee unless the path-length is o(D^(1/4)). Even if Theorem 1 were providing guarantee for unknown task-boundaries, work by Jadbabaie et al. (2015) shows a dynamic regret of O(sqrt((P+1)*T) for a similar notion of path-length P (basically without the square on the norm) and to my understanding without needing to know the path-length to set the step-size. Applying their algorithm to the convex composite losses here would yield what I would view as a better result than the proposed approach (and in-fact can adapt to other types of regularity in the sequence). 

A3: We thank the reviewer for the very useful comments. We would like to make the following clarifications. 

(i) Indeed our theoretical analysis assumes that the detection mechanisms are correct every time. Thus, our theory is to support that our algorithm will have theoretically grounded performance as long as the detection mechanisms are accurate enough, which is indeed the case as demonstrated in our experiments (please see ablation studies on detection accuracies in Section 5.2, and we also provide new sensitivity experiments in Appendix A). We also want to emphasize that an important goal of our regret analysis is to motivate some aspects of our algorithm design choices, such as updating the meta model based only on the current data is sufficient for achieving sublinear regret, which is usually not adopted in the online meta-learning community. Our results suggest that the community could also learn from careful explorations of simpler designs, besides emphasizing algorithmic complexities. 

(ii) We understand that we may not always achieve the best possible regret bound for the setting considered (for example, there may be a way to combine our analysis with the references suggested by the reviewer to further improve our bound), but the goal of our paper is mainly practice driven and the theory is to support some overlooked aspects in online meta-learning. As such we propose a simple algorithm that can detect task boundaries and distribution shifts and achieves superior performance compared to the state-of-the-art methods for online meta-learning in changing environments and without knowing task-boundaries. Thus, we kindly ask the reviewer to evaluate our paper with an emphasis on the empirical contributions and consider our theory as an extra effort. 

(iii) It appears that the current online learning tools are simply not yet ready to capture such detection mechanisms. In our case the detection metrics (e.g.  the energy metric) is very different from the regret metric, which makes it very hard to be incorporated into the regret analysis. The problem may be simplified if we assume the detection component as a blackbox that yields the misdetection and false alarm rates. In this case it might be possible to incorporate the detection parameters into the regret analysis, which is an interesting and open problem to explore in the future and could be very useful to the broader online learning community.

From: ICLR.cc/2023/Conference/Paper3799/Authors
title: Continued response to Reviewer ZoyY 
comment: Q4: The discussion of memory-efficiency is a bit overstated; theoretically at least, the O(T) issue in Finn et al. (2019) is the result of the odd choice of using FTL to optimize the sequence of strongly convex functions, when adaptive OGD (Hazan et al., 2007) would have achieved the same regret and with O(1) memory. Methods studying the online-within-online setting (Denevi et al., 2019; Khodak et al., 2019) also use O(1) memory. Notably, memory efficiency is not a “theoretical advance in online learning” that occurred in 2016. 

A4: We agree with the reviewer that memory-efficiency may not be a “theoretical advance” in online learning community, but most of the existing work in online meta-learning literature follow a FTL-like approach or store/summarize previous data in one way or the other, which indeed makes memory-efficiency worth exploring in online meta-learning. Part of the goal of this paper is to call attention to this highly overlooked issue by the online meta-learning community. 

Q5: It is strange to motivate an adversarial theoretical analysis and algorithm with a discussion of distribution shifts, which suggests a stochastic problem. 

A5: In this work, we consider optimal meta comparators for the worst-case scenario (the standard adversarial case in online learning), instead of the optimal meta comparators with respect to the task distributions (the stochastic case) due to the following reasons:

(1) Since meta-learning usually focuses on the few-shot setup where each task only has a very small dataset, the optimal task model for a single task may vary substantially when small changes occur in the dataset (especially in practice). To take this `sensitivity’ into consideration, we study the adversarial formulation in this work.

(2) On the other hand, the optimal meta comparator for the finite sum formulation is also an estimation of that for the stochastic formulation, especially when the number of tasks is large for the corresponding task distribution.

As suggested by the reviewer, it is also of great interest to study the stochastic formulation. Since the paper is mainly practice driven and the theory is to support some overlooked aspects in online meta-learning, we will consider the stochastic formulation in the future work with a theoretical focus. 

Finally, we thank the reviewer again for the helpful comments and suggestions for our work. If our response resolves your concerns to a satisfactory level, we kindly ask the reviewer to consider raising the rating of our work. Certainly, we are more than happy to address any further questions that you may have.

From: ICLR.cc/2023/Conference/Paper3799/Authors
title: Response to Reviewer 8xrn 
comment: We thank the reviewer for providing the helpful review! We have addressed the reviewer’s helpful comments and modified the paper accordingly. Please note that in the revised paper, we highlighted our changes by blue-colored texts in both the main body and the appendices of the paper. 

Q1: The proposed method seems to be largely based on empirical observation that the loss value sharply increases when the task switches. The corresponding thresholding scheme is also very naive, although it seems intuitive and empirically effective. I wonder how they set the threshold value in practice - Have those hyperparameters been found with a holdout meta-validation set? or meta-test set?  

A1: (1) For the energy threshold $\tau$: we follow the strategy in [Liu 20’], i.e., we set the threshold $\tau$ using the pre-training tasks. More specifically, we set $\tau$ so that 95% of the pre-training inputs are correctly detected as pre-training data. 

(2) For the threshold $\ell$: in standard few-shot learning experimental setups, the true labeling for each individual task is usually randomly chosen. When there is a task switch, the task specific model learnt from the previous task generally does not fit the new task anymore, where the learning performance could be similar to that of a random model. Thus motivated, we find that a good heuristic to choose a starting value for the threshold $\ell$ is to use the loss value evaluated on a random model. For example, for 10-ways classification tasks that value would be $\ell_r = -\log(1/10) = 2.3$. 

(3) In our revision we provide new experiments in Appendix A on the sensitivity of our algorithm w.r.t. the thresholds $\tau$ and $\ell$. We found that when the threshold $\ell$ is too small, the algorithm tends to over detect task switches, which results in inferior performance of the algorithm due to ineffective reuse of task knowledge. On the other hand when $\ell$ is too large, the high miss detection rate results in the algorithm mostly fine-tuning the online task model $\phi_t$ with the current task support data. As expected, this results in a failure mode due to the adversariality of different tasks (please see Appendix A for more details). 
The results also show that larger values of $\tau$, which collapse to updating the meta-model at each step (even for pretraining task distribution), does not substantially improve the performance. This demonstrates the advantage of the distinct meta-update scheme proposed for in- and out-of-distribution tasks, which avoids unnecessary frequent meta-updates for the pretraining tasks and thus allows a more judicious usage of computational budget. Lower values of $\tau$ (e.g. $\tau=15$) tends to detect all task distributions as the pertaining one, and thus corresponds to eliminating the domain adaptation component of our algorithm (figure 6 (a) in Appendix A). 

Q2: The quality of the paper is a bit low. \citep and \citet are not clearly distinguished. Also, the learning curves are just copy and past of wandb plots, making the quality of the paper look very low. Some of the baselines have not been fully run (e.g., FOML in Fig 3).

A2: We thank the reviewer for these important remarks. We have further improved the quality of the paper per the reviewer’s suggestion. We are still working on making the quality of the paper as good as possible. 
For the questions on FOML, we compare all baselines on a 16GB GPU memory budget and FOML runs out of memory due to its linear growth in memory requirement. But up to the point it runs out of memory, it's clear that the FOML algorithm is not competitive to the best performing methods. This poor performance is essentially due to the fact that FOML considers mutually non-exclusive tasks which may not hold for the standard few-shot learning benchmarks.

From: ICLR.cc/2023/Conference/Paper3799/Authors
title: Continued response to Reviewer 8xrn 
comment: Q3: I'm not sure whether some of the existing baselines such as FTML by Finn et al. and OSML by Yao et al. are not applicable to the setting the authors considered. Could you compare against FTML and OSML as well? 

A3: There are two main reasons that we do not consider FTML and OSML as baselines: 
(1) FTML (Finn et al.) and OSML (Yao et al.) use MAML at each step on all previously seen tasks to find a good meta-model for the current task, which is computationally expensive. Hence these methods are only applicable up to a few dozens of tasks (less than 100 in their papers). In our setting, the number of tasks could be several thousands (we consider up to 20000 steps in our experiments) and thus it would be unrealistic to apply them as baselines.  

(2) Further, a method such as OSML has an unfair advantage (in terms of resource budget) over methods that use a single fixed capacity meta-network (the setup considered in our work), because OSML allows the meta-hierarchical graph to grow by incorporating new knowledge blocks whenever discovered. 

Finally, we thank the reviewer again for the helpful comments and suggestions for our work. Certainly, we are more than happy to address any further questions that you may have. 

References: 

[Liu 20’] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. Advances in Neural Information Processing Systems, 33:21464–21475, 2020.

From: ICLR.cc/2023/Conference/Paper3799/Authors
title: Response to Reviewer R5Rh 
comment: We thank the reviewer for providing the helpful review! We have addressed the reviewer’s helpful comments and modified the paper accordingly. Please note that in the revised paper, we highlighted our changes by blue-colored texts in both the main body and the appendices of the paper. 

Q1: How is the threshold $\ell$  in Algorithm 1 chosen? When the algorithm is used online, it seems unclear to users about how to choose an appropriate $\ell$. Some practical guidance will be very useful. 

A1:  In standard few-shot learning experimental setups, the true labeling for each individual task is usually randomly chosen. When there is a task switch, the task specific model learnt from the previous task generally does not fit the new task anymore, where the learning performance could be similar to that of a random model. Thus motivated, we find that a good heuristic to choose a starting value for the threshold $\ell$ is to use the loss value evaluated on a random model. For example, for 10-ways classification tasks that value would be $\ell_r = -\log(1/10) = 2.3$. 
In our revision, we also provide a new experiment in appendix A on the sensitivity of our algorithm w.r.t. the threshold $\ell$. We found that when the threshold $\ell$ is too small, the algorithm tends to over detect task switches, which results in inferior performance of the algorithm due to ineffective reuse of task knowledge. On the other hand when $\ell$ is too large, the high miss detection rate results in the algorithm mostly fine-tuning the online task model $\phi_t$ with the current task support data. As expected, this results in a failure mode due to the adversariality of different tasks (please see Appendix A for more details). 

Q2: Why doesn't the regret bound depend on $\ell$? It seems if we choose $\ell = 0$, then the regret should grow linearly in $\Sigma_d K_d$. 

A2: The purpose of the regret analysis is to motivate some aspects of our algorithm design such as updating the meta model based only on the current data is sufficient for achieving sublinear regret. Hence, the analysis is based on the simplified version of the proposed algorithm, as described in eq. 7, where we implicitly assume that the task switches can be accurately detected. Therefore, the regret bound does not depend on $\ell$.

Q3: In the regret definition (6), does $\Sigma_d K_d$  equal $T$? 

A3: Yes, $\Sigma_d K_d$ will correspond to the total number of online steps $T$. 

Q4: Consider two scenarios with two very different tasks (data from each task are drawn from a stationary distribution). The first scenario is that from time 1 to T/2, it is the first task, and from time T/2+1, ..., T, it is the second task. The second scenario is that two tasks frequently switch between each other. Clearly, the proposed algorithm performs well in the first scenario, but not in the second. I don't see how the regret varies between these two scenarios. Some clarification will be helpful. 

A4: We thank the reviewer for the insightful comments on these two special scenarios. The second scenario is clearly more challenging in general due to its nature. In fact, this is captured in the regret bound by the path-length $P_T = \sum‖\theta_d^∗−\theta_{d+1}^∗‖^2$, where $\theta_d^∗$ is the optimal meta-model for the $d$-th encountered distribution. Thus, $P_T$ captures the accumulative model shift which further reflects the domain distribution shift over time. 
Hence, more frequent switches between two very different tasks (e.g., the second scenario) will lead to a bigger  $P_T$, and thus the regret will be larger. 

Finally, we thank the reviewer again for the helpful comments and suggestions for our work. Certainly, we are more than happy to address any further questions that you may have.

From: ICLR.cc/2023/Conference/Paper3799/Reviewer_R5Rh
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper proposes a new algorithm for task-agnostic online meta-learning when data is non-stationary. The algorithm has two components. First, the algorithm can detect task switches. Second, the algorithm can adapt to distribution shifts. The paper provides a regret analysis of the algorithm.
strength_and_weaknesses: Strength:

This paper seeks to solve a challenging problem, and provides a nice solution. The solution makes intuitive sense and is supposed by a regret analysis.

Weakness:

1. How is the threshold $\ell$ in Algorithm 1 chosen? When the algorithm is used online, it seems unclear to users about how to choose an appropriate $\ell$. Some practical guidance will be very useful. 

2. What doesn't the regret bound depend on $\ell$? It seems if we choose $\ell = 0$, then the regret should grow linearly in $\sum_d K_d$.  In the regret definition (6), does $\sum_d K_d$ equal $T$?

3. Consider two scenarios with two very different tasks (data from each task are drawn from a stationary distribution). The first scenario is that from time 1 to T/2, it is the first task, and from time T/2+1, ..., T, it is the second task. The second scenario is that two tasks frequently switch between each other. Clearly, the proposed algorithm performs well in the first scenario, but not in the second. I don't seem how the regret varies between these two scenarios. Some clarification will be helpful.


clarity,_quality,_novelty_and_reproducibility: This paper provides a nice solution to the challenging problem in online meta-learning when tasks are agnostic and data is non-stationary. The paper demonstrates the good performance of its proposed method through a number of experiments. The paper is well-written, and I enjoy reading the paper. 
summary_of_the_review: This paper proposes a new algorithm for online meta-learning to simultaneously address the problem of agnostic tasks and the problem of distribution shift. The paper is well-written. However, I think the paper can benefit from more discussion on the theoretical results.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3799/Reviewer_8xrn
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper propose to investigate meta-continual learning scenario where the task boundaries are not known and whether distribution shift occur or not. To solve this challenging problem, they focus on some empirical findings that the loss values or free energies sharply increases when the boundary meets. Thus they introduce two threshold values for detecting task boundaries and distributional shifts, in order to selectively control between task-specific learning and meta-updates. The empirical results demonstrate the efficacy and effectivenss of their methodologies on various learning scenarios.
strength_and_weaknesses: Strength
- The target problem is largely important for real-world applications.
- the paper is well written and method easy to understand.
- Method performs well on various learning scenarios, outperforming current state-of-the-arts on the given problem.
- Theoretical results support their claim.

Weaknesses
- The proposed method seems to largely based on empirical observation that the loss value sharply increases when the task switches. The corresponding thresholding scheme is also very naive, although it seems intuitive and empirically effective. I wonder how they set the threshold value in practice - Have those hyperparameters been found with a holdout meta-validation set? or meta-test set?
- The quality of the paper is a bit low. \citep and \citet are not clearly distinguished. Also, the learning curves are just copy and past of wandb plots, making the quality of the paper looking very low. Some of the baselins have not been fully run (e.g., FOML in Fig 3)
- I'm not sure whether some of the existing baselines such as FTML by Finn et al. and OSML by Yao et al. are not applicable the setting the authors considered. Could you compare against FTML and OSML as well? 

clarity,_quality,_novelty_and_reproducibility: Clarity 
- The paper is easy to understand and mostly clear

Quality
- The paper needs to improve the quality by a lot

Novelty
- I'm not very sure about the novelty part (the method seems to be a single thresholding scheme)

Reproducibility
- They provided their code.
summary_of_the_review: In summary, although there are some concerns regarding the novelty and quality of the submission. I think the strength outweights its weaknesses which I think can be easily addressed. Thus I recommend (weak) acceptance.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3799/Reviewer_ZoyY
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper considers the problem of online meta-learning and proposes an algorithm to handle the case where the task boundaries are unknown and the the comparator can change over time. They propose detection mechanisms for task switches and a method that adapts the meta-model when the switches occur and the fine-tuned model when they do not. The proposal is supported with theoretical and experimental analysis.
strength_and_weaknesses: ### Strengths:
1. The problem is well-motivated as being able to adapt quickly in change environments is a core aspect of learning.
2. The switch detection mechanisms seem interesting, although I do not know if they are novel; are they connected to change-point detection algorithms? A comparison with past work such as CMAML++ would also be useful
3. Code is provided in the supplementary material.

### Weaknesses:
1. My main concern with this paper is in the theory, where there are several issues:
    1. The statement “as shown in Finn et al. (2019), Assumption 2 holds when the function is convex and the adaptation mapping corresponds to the one-step gradient descent” is incorrect; the relevant statement in that paper (Theorem 1) holds only for strongly convex, strongly smooth, and Hessian smooth functions, and it is unclear if the result can be extended to convex functions without making the step-size of the adaptation mapping zero and thus uninteresting in this context.
    2. It is a bit difficult to understand exactly which algorithm Theorem 1 gives guarantees for, but my understanding is that they are for Algorithm 1 assuming the detection mechanisms are correct every time. What this means is that, while the paper is advertised as not knowing the task boundaries, for the actual guarantee the task boundaries are indeed known. The issue then becomes that the authors do not compare to the trivial baseline, which is restarting gradient descent whenever a new task is detected; in the regime where all tasks have at most K rounds doing this has dynamic regret O(D*sqrt(K)), which is better than the provided guarantee unless the path-length is o(sqrt(D)). If the task-boundaries are known, the setting studied by the authors is in-fact more akin to that of online-within-online meta-learning (Denevi et al., 2019; Khodak et al., 2019), where you have a sequence of tasks with some number of rounds per task; the only difference is here the loss functions are composite (but convex, as in those papers). Notably, Theorem 3.3 of the latter gives a guarantee of O(D^(3/4)*sqrt(K)) as a special case, which is better than the provided guarantee unless the path-length is o(D^(1/4)).
    3. Even if Theorem 1 were providing guarantee for unknown task-boundaries, work by Jadbabaie et al. (2015) shows a dynamic regret of O(sqrt((P+1)*T) for a similar notion of path-length P (basically without the square on the norm) and to my understanding without needing to know the path-length to set the step-size. Applying their algorithm to the convex composite losses here would yield what I would view as a better result than the proposed approach (and in-fact can adapt to other types of regularity in the sequence).
2. The discussion of memory-efficiency is a bit overstated; theoretically at least, the O(T) issue in Finn et al. (2019) is the result of the odd choice of using FTL to optimize the sequence of strongly convex functions, when adaptive OGD (Hazan et al., 2007) would have achieved the same regret and with O(1) memory. Methods studying the online-within-online setting (Denevi et al., 2019; Khodak et al., 2019) also use O(1) memory. Notably, memory efficiency is not a “theoretical advance in online learning” that occurred in 2016.
3. It is strange to motivate an adversarial theoretical analysis and algorithm with a discussion of distribution shifts, which suggests a stochastic problem.


### References:
- Denevi, Stamos, Ciliberto, Pontil. Online-within-online meta-learning. NeurIPS 2019.
- Finn, Rajeswaran, Kakade, Levine. Online meta-learning. ICML 2019.
- Hazan, Rakhlin, Bartlett. Adaptive online gradient descent. NeurIPS 2007.
- Jadbabaie, Rakhlin, Shahrampour, Sridharan. Online optimization: Competing with dynamic comparators. AISTATS 2015.
- Khodak, Balcan, Talwalkar. Adaptive gradient-based meta-learning methods. NeurIPS 2019.
- Mokhtari, Shahrampour, Jadbabaie, Ribeiro. Online optimization in dynamic environments: Improved regret rates for strongly convex problems. CDC 2016
clarity,_quality,_novelty_and_reproducibility: Clarity: aspects of the theory are unclear, such as assumptions on the loss functions and for what precise algorithm the theory holds.  
Quality: the theoretical component of the work is of poor quality due to incorrect statements and lack of comparisons. The experimental results are interesting.  
Novelty: the work has some novelty in the task detection algorithm and setting studied.  
Reproducibility: good.
summary_of_the_review: I do not believe should be accepted in its current form due to the issues with the theoretical results outlined above.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper3799/Reviewer_1HZs
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This work considers online meta-learning in a non-stationary environment where new tasks might come and task distribution can change. To solve these problems, this work designs two heuristic detectors to detect the task switches and distribution shifts. It also provides a theoretical analysis of the dynamic regret under certain assumptions.
strength_and_weaknesses: ### Strength:
- This paper considers an interesting but hard problem that the new task/distribution could appear in the online meta-learning process.
- The paper is clearly structured and well-written.
- Theoretical analysis of the dynamic regret is provided.

### Weakness:
- My main concern about the paper is the mismatch between the theoretical analysis and the proposed methods. The authors have proposed two heuristics to detect the shift on tasks and task distribution, which are the main algorithm contributions of this paper. But, the theoretical analysis is irrelevant to the proposed mechanism.
- About the novelty of the dynamic regret bound. The analysis for the dynamic regret seems a direct application of that for the OGD algorithm [Zinkvich,2003] under the convexity assumption (Assumption 2). In this sense, the theoretical technical novelty of this is somewhat limited to me.
- About the correctness of the dynamic regret bound. It seems to me that $P_T$ in Theorem 1 should be defined as $P_T = \sum \Vert \theta_d^*-\theta_{d+1}^*\Vert$ instead of the second-order formulation. It is unclear to me why the second inequality of Eq. (13) holds.
- About the claim on the sublinear regret: It seems that the proposed regret bound will suffer from a linear regret when $P_T = O(T)$. Therefore, the claim on the sublinear dynamic regret is somewhat an overclaim to me. 
- About the experiments: To validate the effectiveness of the proposed detection scheme, it would be better to provide a more detailed inspection of the shift detection instead accuracy instead of just providing the comparison on the overall performance.


clarity,_quality,_novelty_and_reproducibility: Quality: The proposed method and the theoretical analysis are mismatched. Besides, it seems that the definition of the $P_T$ is not correct to me. 

Clarity: The paper is overall well-written. But there are still some minor points. 
- I think it would be better to provide more explanation on what task boundaries means in Section 1 for readers who are not familiar with meta-learning.
- It’s better to distinguish the expected loss and the empirical loss in the paper.
- Should it be $\mathcal{L}(\theta_{t-1};S_t)$ in Line 14 of Algorithm 1?
- The axis and legend of the figures are not clear enough.

Originality: The proposed detector for distribution shift detection is novel to me, but they are not brand new since it is established on the OOD detection proposed by previous works. The theoretical analysis is also a standard analysis for the OGD algorithm.
summary_of_the_review: This paper has studied an interesting but challenging problem. But I am concerned about the mismatch between the proposed algorithm and the theoretical analysis. Besides, though the shift detector is novel to me, the criterion is mainly established by the previous work [Liu et al., 202], and the theoretical analysis is standard. Overall, I think this paper has some interesting parts, but the flaws outweigh its merit. So, I would like to reject this paper.

correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3799/Reviewer_HHCs
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: Authors propose an algorithm for task-agnostic online meta-learning in nonstationary environments.
strength_and_weaknesses: Strengths:
  * The insight that the simplicity of the task-adaptation step (one single application of batch gradient) convexifies the problem despite the use of deep architectures, allowing the application of online learning techniques.
  
Weaknessess:
  * Task switch detection heuristic does not correspond to dynamic regret algorithm techniques from the online learning literature.  The paper lacks details on how this heuristic (threshold) is tuned in practice.
  * The paper emphasizes the setting where task boundaries are not known, but without comment the analysis (e.g., Theorem 1) appears to assume task distributions are switching at a regular cadence.  
  
clarity,_quality,_novelty_and_reproducibility: The clarity is:
  * occassionally excellent: this reviewer was initially disinclined to believe that dynamic regret analysis would apply to a deep architecture, but the discussion preceeding Theorem 1 was quite convincing.
  * occassionally bad: 
    * Algorithm 1 line 6 uses $\ell$ was doesn't list it as an input or indicate how it is computed.
	* what is $g_k$ ("the prediction model's output?") mentioned after equation (4)?  
	* when describing the experimental setup in section 5, the moderately stationary case has a higher task switch probability than the low stationary case.  Also the phrase "where $p$ is very small" makes no sense.
	
Novelty looks good: as previously indicated, the semantics of Theorem 1 were not apriori expected, so this reviewer learned something valuable (reviewing is mostly thankless, so this is a nice upside surprise).
summary_of_the_review: **Don't panic**: This paper wants to be accepted.  You just need to address my concerns in order for me to raise my score.

My primary concern is the task switching heuristic.  Both the heuristic and the description of the heuristic are lacking.  Most of the clarity issues I identified were related to this aspect.  Just fix them.
  * Step 1: better describe what you are actually doing to set the temperature.  
  * Step 2: provide some experimental commentary on the sensitivity of the technique to the choice of temperature.
  * Step 3: quantify how homogenous (or not) the task loss distributions are in the setups you are using to give an idea of why the heuristic is working and when it would fail.
  * Step 4: provide some commentary relative to what dynamic regret algorithms *actually* do to detect environment drift: they do regret bound arithmetic, e.g., [1]
	
The other concerns are lower in importance but to the extent you address them you'll have a better paper.

**Updated**: 5->6 after author response

	
[1] https://arxiv.org/abs/2102.05406
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold