Paper: Improved Regret Bounds for Non-Convex Online-Within-Online Meta Learning (ICLR-2024)

Reviewer comment (Theory):
Although improved regret bounds are presented, the authors do not explain why their analysis and algorithm led to these improvements.

Author response:
**Q1. The authors should explain why their analysis and algorithm led to the improvements in the regret bounds.**\
A1: Thanks for your comments. We have explained our three technical novelties in deriving improved regret bounds for OWO meta learning in Remark B.2 of Appendix B in the original version, and the details are as follows: \
$\mathbf{(1)}$ **The first novelty** lies in deriving improved regret bound for the online algorithm that runs over the functions $\\{h _{t}(v)=v+f _{t}(\rho _{t1})/v\\} _{t \in [T]}$ on the domain $[B^{2}, D^{2}]$ to learn the step size $v _{t}$ of EWA algorithm. Throughout the whole paper, we choose the efficient Follow-The-Leader (FTL) algorithm to learn the step size $v _{t}=argmin _{v \in [B^{2}, D^{2}]}\sum _{s=1}^{t-1}h _{s}(v)=\big({\sum _{s=1}^{t-1}f _{s}(\rho _{s1})/(t-1)}\big)^{{1}/{2}}$ and use the primal-dual analysis from [1, Cor 1] to derive a logarithmic regret bound of $O(\log{T})$. The key step in obtaining the logarithmic regret $O(\log{T})$ is to show that $\min _{t \in [T]}f _{t}(\rho _{t1})$ is strictly positive (i.e. $B>0$) to guarantee the strong-convexity of $h _{t}(v)$ and the boundedness of $\partial h _{t}(v _{t})$ (i.e. the Lipschitz property of $h _{t}(v)$ at the point $v _{t}$). The positiveness of $\min _{t \in [T]}f _{t}(\rho _{t1})$ is guaranteed for piecewise Lipschitz functions in Proposition 3 and for non-Lipschitz functions in Proposition 5 respectively, via a fine-grained estimation of the lower bound $B^{2}$ of $f _{t}(\rho _{t1})$. In contrast, existing works [2, Prop B.2] and [3, Cor 3.2] both choose $\epsilon$-FTL algorithm to optimize the functions $\\{h _{t}(v)=v+(f _{t}(\rho _{t1})+\epsilon^{2})/v\\} _{t \in [T]}$ on the domain $[0, D^{2}]$ (where $D^{2}\geq\max _{t}f _{t}(\rho _{t1})$) to learn the step size $v _{t}$, and lead to the regret $O(T\epsilon^{2}+(\log{T})/\epsilon^{2})$, which is of $O(\sqrt{T}\log{T})$ if we set $\epsilon={1/T^{1/4}}$ and is slower than our bound. \
$\mathbf{(2)}$ **The second novelty** lies in deriving improved regret bound for the online algorithm that runs over the functions $\{f _{t}(\rho)=V(\rho, \rho _{t}^{\*})\} _{t\in [T]}$ in the piecewise Lipschitz case. We use Follow-The-Regularized-Leader (FTRL) algorithm to achieve the regret of $O(T^{1/2+\alpha})$, $\alpha \in (0, \frac{1}{2})$, and the mixture parameter $\gamma = m^{d\beta}/T^{\alpha}$ is irrelevant to the optimal $\boldsymbol{\rho}^{\*}$ (hence $\gamma$ can be set in advance). Existing work [3, Thm 3.2] also uses FTRL algorithm, but [3] leverages a more complicated analysis, and attains a larger regret bound $O(m^{d/2}T^{3/4})$. Nevertheless, their choice of $\gamma=\frac{\operatorname{KL}(\boldsymbol{\rho}^{\*}||\hat{\boldsymbol{\nu}})^{1/4}m^{d\beta/2}}{T^{1/4}V _{d}^{1/2}}$ in [3, Thm 3.2]  depends on the knowledge of optimal distribution $\boldsymbol{\rho}^{\*}$ that contains information of $T$ training tasks, which is unfeasible in the sequential online meta learning setting. \
$\mathbf{(3)}$ **The third novelty** lies in deriving improved regret bound for the online algorithm that runs over the functions $\\{f _{t}(\rho)=\operatorname{KL}(\rho _{t}^{\*}||\rho)\\} _{t\in [T]}$ in the non-Lipschitz case. Obtaining regret bounds for FTL algorithm run over the functions $\\{\operatorname{KL}(\rho _{t}^{*}||\rho)\\} _{t\in [T]}$ is hard, because $\operatorname{KL}(\rho _{t}^{\*}||\rho)$ is the functional of the probability distribution $\rho$, and we are unable to use traditional regret analysis for the functions over Euclidean space (e.g. the gradient boundedness and strong convexity analysis of the functions). Instead, we leverage the insightful Lemma E.2 to obtain the analytic form $\rho _{t1}=argmin _{\rho \in \mathcal{P}(\theta)}\sum _{s=1}^{t-1}\operatorname{KL}(\rho _{s}^{\*}||\rho)=\frac{1}{t-1}\sum _{s=1}^{t-1}\rho _{s}^{\*}$ of the solution of FTL algorithm. Then, we use this analytic form, as well as technical properties of RN derivative to estimate the upper bound of the regret and ultimately obtain a non-trivial logarithmic regret $O(\log{T})$, achieving so far the tightest regret bound for learning the initialization of EWA algorithm.

Reviewer comment (Theory):
Moreover, it seems that the second improved regret bound is tighter than the first one, and holds in a more general case (non-Lipschitz). So, the value of the first one is not clear.

Author response:
**Q2. It seems that the second improved regret bound is tighter than the first one, and holds in a more general case (non-Lipschitz). So, the value of the first one is not clear.**\
A2: Sorry for the confusion. The value/advantages of the first improved regret bound lie in the following 4 aspects, and we have made its value clearer in Remark B.5 in Appendix B of the revised version: \
$\mathbf{(1)}$ The first improved regret bound in our Theorem 2 is not a special case of our Theorem 3. The main reason is that Theorem 2 uses $V^{2}=min _{\rho:\Theta \mapsto \mathbb{R} _{\geq 0}, \int _{\Theta}\rho(\theta)\mathrm{d}\theta=1}-\frac{1}{T}\sum _{t=1}^{T}\log{\int _{\mathcal{B}(\theta _{t}^{*},m^{-\beta})}\rho(\theta)\mathrm{d}\theta} $ as the similarity notion between different tasks, but Theorem 3 uses the task similarity notion $V^{2}=min _{\rho \in \mathcal{P}(\Theta)}\frac{1}{T}\sum _{t=1}^{T}\operatorname{KL}(\rho _{t}^{\*}||\rho)$. Besides, the task similarity in Theorem 2 is defined according to the specific property (i.e. $\epsilon$-radius) of the piecewise-Lipschitz function, and hence is particularly applicable to the piecewise-Lipschitz setting.\
$\mathbf{(2)}$ The action-space-discretization technique (described in Section 4.1 to obtain the first regret bound in Theorem 2) is of independent interest. The defined task similarity in Theorem 2 also requires a novel action-space-discretization method to translate the minimization problem $\min _{\rho \in \Theta}f _{t}(\rho)$ over the set of distributions into a tractable online convex optimization problem. \
$\mathbf{(3)}$ At present, the first regret bound has higher application value than the second regret bound. The Algorithm 1 (corresponding to the first regret bound in our Theorem 2) can be applied in the continuum domain, but Algorithm 2 (corresponding to the second regret bound in our Theorem 3) at the current stage is still not easy to be applied in the continuum domain (see more explanations in our Remark H.1). \
$\mathbf{(4)}$ The first improved regret is obtained under the same assumptions (i.e. piecewise-Lipschitz and bounded loss functions) as that in [3], via a more technical analysis. We list this improved regret in our work to make a fair comparison and show rigorous improvements over the existing work [3].

Reviewer comment (Novelty):
The transfer risk and PAC-Bayes generalization bounds are derived by using the online-to-batch and online-to-PAC techniques, which are incremental to some extent.

Author response:
**Q3. The transfer risk and PAC-Bayes generalization bounds are derived by using the online-to-batch and online-to-PAC techniques, which are incremental to some extent.**\
A3: Thanks for your comments. But we believe our generalization bounds for statistical meta learning are not incremental, due to the following 2 main reasons (which are also added as Remark G.1 in Appendix G of the revised version): \
$\mathbf{(1)}$ **For the transfer risk bound in Theorem 4:** the novelties of our online-to-batch technique lie in 2 aspects: $\mathbf{(i)}$ The first novelty lies in bounding $\operatorname{KL}(\rho^{\*}||\rho _{T+1,1})$ with Proposition 4 and Proposition F.2, both of which require the technical analysis of the properties of the Radon-Nikodym derivative. To the best of our knowledge, such analysis does not exist in the previous online-to-batch literature. $\mathbf{(ii)}$ We further use concentration inequality to bound $ \mathbb{E} _{\mu \sim \tau}\frac{\operatorname{KL}(\rho^{\*}||\rho _{T+1,1})}{H}$, leading to the transfer risk bound in the non-convex setting. Such transfer risk bound, as shown in our Remark 3, has almost the same convergence rate when compared with the latest transfer risk bound in the convex setting (i.e. [2, Thm E.1], which is obtained by Jensen’s inequality of convex loss), demonstrating the novelty of the online-to-batch analysis developed in our Theorem 4. \
$\mathbf{(2)}$ **For the PAC-Bayes generalization bound in Theorem 5:** first we need to admit that our Proposition 6 is a direct corollary of recent online-to-PAC result in [4, Thm 1] (i.e. extent the result [4] from single-task learning to our multi-task learning setting), but the novelty of our Theorem 5 lies in the combination of the online-to-PAC analysis and the task-averaged regret analysis developed in our Theorem 3. Such combination pioneers a new research direction to demonstrating PAC-Bayes generalization error bounds for statistical multi-task learning. If we do not combine the aforementioned two analysis tools, but just use the online-to-PAC analysis (i.e. [4, Thm 1]) and traditional regret analysis for single-task learning (i.e. [5, Thm 2.1]), we can only obtain the following PAC-Bayes bound for statistical multi-task learning:
$$\frac{1}{T}\sum _{t=1}^{T}\mathbb{E} _{\theta \sim \mathcal{A} _{t}(S _{t})}\Big[\mathbb{E} _{z\sim \mu _{t}}\ell(\theta, z)-\frac{1}{m}\sum _{i=1}^{T}\ell(\theta,z _{ti})\Big]\leq \frac{\sqrt{m}M(1+\frac{1}{T}\sum _{t=1}^{T}\operatorname{KL}(\rho _{t}^{\*}||\rho _{t1}))}{m} + M\sqrt{\frac{2\log{\frac{1}{\delta}}}{Tm}},$$which is less informative than the generalization bound in our Theorem 5.

Reviewer comment (Experiments):
Although some experimental results are provided, no existing algorithms are compared and discussed in the experiments.

Author response:
**Q4. Although some experimental results are provided, no existing algorithms are compared and discussed in the experiments.**\
A4: Thanks for your suggestion. We have discussed more differences between our Algorithm 1 and the state-of-the-art algorithm from [3, Alg 3] in Remark H.2 of Appendix H of the revised version. The detailed explanations are three-fold: \
$\mathbf{(1)}$ The comparisons with the single-task algorithm. Actually, we choose single-task EWA algorithm as our baseline in all experiments. In both Figure 1 ($m$-shot multi-task method v.s. single-task baseline) and Figure 2 ($T =5$ multi-task method v.s. $T=1$ single-task baseline), we show advantages of the meta-learning based EWA algorithm over the single-task EWA algorithm baseline, and verify the convergence performance of task-averaged regret of our meta learning algorithm. \
$\mathbf{(2)}$ The comparisons with the state-of-the-art algorithm for non-convex piecewise-Lipschitz OWO meta learning [3, Algorithm 3]. Our Algorithm 1 is actually based on the modification of [3, Algorithm 3], and the two main differences are as follows: \
$\mathbf{(i)}$ For learning the initialization $\rho$, we use FTRL algorithm to achieve the regret of $O(T^{1/2+\alpha})$, $\alpha \in (0, \frac{1}{2})$, and the mixture parameter $\gamma = m^{d\beta}/T^{\alpha}$ is irrelevant to the optimal $\boldsymbol{\rho}^{\*}$ (hence $\gamma$ can be set in advance). Existing work [3] also uses FTRL algorithm, but [3] leverages a more complicated analysis, attaining a larger regret bound $O(m^{d/2}T^{3/4})$. Nevertheless, their choice of $\gamma=\frac{\operatorname{KL}(\boldsymbol{\rho}^{\*}||\hat{\boldsymbol{\nu}})^{1/4}m^{d\beta/2}}{T^{1/4}V _{d}^{1/2}}$ in [3, Thm 3.2] depends on the knowledge of optimal distribution $\boldsymbol{\rho}^{\*}$ that contains information of $T$ training tasks, which is unfeasible in the sequential online meta learning setting. \
$\mathbf{(ii)}$ For learning the step size $v$, we choose FTL algorithm to learn the step size $v _{t}=argmin _{v \in [B^{2}, D^{2}]}\sum _{s=1}^{t-1}h _{s}(v)=\big({\sum _{s=1}^{t-1}f _{s}(\rho _{s1})/(t-1)}\big)^{{1}/{2}}$ to derive a logarithmic regret bound of $O(\log{T})$. In contrast, [3] chooses $\epsilon$-FTL algorithm to optimize the functions $\\{h _{t}(v)=v+(f _{t}(\rho _{t1})+\epsilon^{2})/v\\} _{t \in [T]}$ on the domain $[0, D^{2}]$ (where $D^{2}\geq\max _{t}f _{t}(\rho _{t1})$) to learn the step size $v _{t}=argmin _{v \in [0, D^{2}]}\sum _{s=1}^{t-1}h _{s}(v)=\big({\sum _{s=1}^{t-1}f _{s}(\rho _{s1})/(t-1)}+\epsilon^{2}\big)^{{1}/{2}}$, and lead to the regret $O(T\epsilon^{2}+(\log{T})/\epsilon^{2})$, which is of $O(\sqrt{T}\log{T})$ if we set $\epsilon={1/T^{1/4}}$ and is slower than our regret bound.\
$\mathbf{(3)}$ Practical implementation of meta learning algorithm. However, [3] used a heuristic method to learn (instead of meta learn) the step size $\lambda _{t}$, so their implementation is not rigorously the same as the pseudo code in their paper (i.e. [3, Algorithm 3]). We follow almost the same implementation details as in [3] (which is also explained in detail in the Experimental Setting part in our Section H), with the main difference being that we verify the regret bound over the training tasks but [3] verify the regret bound over the test tasks. Therefore, we did not compare the convergence rate of regret of [3] in our experiments.

Reviewer comment (Theory):
The authors should explain why their analysis and algorithm led to the improvements in the regret bounds.

Author response:
**Q1. The authors should explain why their analysis and algorithm led to the improvements in the regret bounds.**\
A1: Thanks for your comments. We have explained our three technical novelties in deriving improved regret bounds for OWO meta learning in Remark B.2 of Appendix B in the original version, and the details are as follows: \
$\mathbf{(1)}$ **The first novelty** lies in deriving improved regret bound for the online algorithm that runs over the functions $\\{h _{t}(v)=v+f _{t}(\rho _{t1})/v\\} _{t \in [T]}$ on the domain $[B^{2}, D^{2}]$ to learn the step size $v _{t}$ of EWA algorithm. Throughout the whole paper, we choose the efficient Follow-The-Leader (FTL) algorithm to learn the step size $v _{t}=argmin _{v \in [B^{2}, D^{2}]}\sum _{s=1}^{t-1}h _{s}(v)=\big({\sum _{s=1}^{t-1}f _{s}(\rho _{s1})/(t-1)}\big)^{{1}/{2}}$ and use the primal-dual analysis from [1, Cor 1] to derive a logarithmic regret bound of $O(\log{T})$. The key step in obtaining the logarithmic regret $O(\log{T})$ is to show that $\min _{t \in [T]}f _{t}(\rho _{t1})$ is strictly positive (i.e. $B>0$) to guarantee the strong-convexity of $h _{t}(v)$ and the boundedness of $\partial h _{t}(v _{t})$ (i.e. the Lipschitz property of $h _{t}(v)$ at the point $v _{t}$). The positiveness of $\min _{t \in [T]}f _{t}(\rho _{t1})$ is guaranteed for piecewise Lipschitz functions in Proposition 3 and for non-Lipschitz functions in Proposition 5 respectively, via a fine-grained estimation of the lower bound $B^{2}$ of $f _{t}(\rho _{t1})$. In contrast, existing works [2, Prop B.2] and [3, Cor 3.2] both choose $\epsilon$-FTL algorithm to optimize the functions $\\{h _{t}(v)=v+(f _{t}(\rho _{t1})+\epsilon^{2})/v\\} _{t \in [T]}$ on the domain $[0, D^{2}]$ (where $D^{2}\geq\max _{t}f _{t}(\rho _{t1})$) to learn the step size $v _{t}$, and lead to the regret $O(T\epsilon^{2}+(\log{T})/\epsilon^{2})$, which is of $O(\sqrt{T}\log{T})$ if we set $\epsilon={1/T^{1/4}}$ and is slower than our bound. \
$\mathbf{(2)}$ **The second novelty** lies in deriving improved regret bound for the online algorithm that runs over the functions $\{f _{t}(\rho)=V(\rho, \rho _{t}^{\*})\} _{t\in [T]}$ in the piecewise Lipschitz case. We use Follow-The-Regularized-Leader (FTRL) algorithm to achieve the regret of $O(T^{1/2+\alpha})$, $\alpha \in (0, \frac{1}{2})$, and the mixture parameter $\gamma = m^{d\beta}/T^{\alpha}$ is irrelevant to the optimal $\boldsymbol{\rho}^{\*}$ (hence $\gamma$ can be set in advance). Existing work [3, Thm 3.2] also uses FTRL algorithm, but [3] leverages a more complicated analysis, and attains a larger regret bound $O(m^{d/2}T^{3/4})$. Nevertheless, their choice of $\gamma=\frac{\operatorname{KL}(\boldsymbol{\rho}^{\*}||\hat{\boldsymbol{\nu}})^{1/4}m^{d\beta/2}}{T^{1/4}V _{d}^{1/2}}$ in [3, Thm 3.2]  depends on the knowledge of optimal distribution $\boldsymbol{\rho}^{\*}$ that contains information of $T$ training tasks, which is unfeasible in the sequential online meta learning setting. \
$\mathbf{(3)}$ **The third novelty** lies in deriving improved regret bound for the online algorithm that runs over the functions $\\{f _{t}(\rho)=\operatorname{KL}(\rho _{t}^{\*}||\rho)\\} _{t\in [T]}$ in the non-Lipschitz case. Obtaining regret bounds for FTL algorithm run over the functions $\\{\operatorname{KL}(\rho _{t}^{*}||\rho)\\} _{t\in [T]}$ is hard, because $\operatorname{KL}(\rho _{t}^{\*}||\rho)$ is the functional of the probability distribution $\rho$, and we are unable to use traditional regret analysis for the functions over Euclidean space (e.g. the gradient boundedness and strong convexity analysis of the functions). Instead, we leverage the insightful Lemma E.2 to obtain the analytic form $\rho _{t1}=argmin _{\rho \in \mathcal{P}(\theta)}\sum _{s=1}^{t-1}\operatorname{KL}(\rho _{s}^{\*}||\rho)=\frac{1}{t-1}\sum _{s=1}^{t-1}\rho _{s}^{\*}$ of the solution of FTL algorithm. Then, we use this analytic form, as well as technical properties of RN derivative to estimate the upper bound of the regret and ultimately obtain a non-trivial logarithmic regret $O(\log{T})$, achieving so far the tightest regret bound for learning the initialization of EWA algorithm.

Reviewer comment (Theory):
The authors should explain whether there exist some advantages of the first improved regret bound when it is compared with the second one.

Author response:
**Q2. It seems that the second improved regret bound is tighter than the first one, and holds in a more general case (non-Lipschitz). So, the value of the first one is not clear.**\
A2: Sorry for the confusion. The value/advantages of the first improved regret bound lie in the following 4 aspects, and we have made its value clearer in Remark B.5 in Appendix B of the revised version: \
$\mathbf{(1)}$ The first improved regret bound in our Theorem 2 is not a special case of our Theorem 3. The main reason is that Theorem 2 uses $V^{2}=min _{\rho:\Theta \mapsto \mathbb{R} _{\geq 0}, \int _{\Theta}\rho(\theta)\mathrm{d}\theta=1}-\frac{1}{T}\sum _{t=1}^{T}\log{\int _{\mathcal{B}(\theta _{t}^{*},m^{-\beta})}\rho(\theta)\mathrm{d}\theta} $ as the similarity notion between different tasks, but Theorem 3 uses the task similarity notion $V^{2}=min _{\rho \in \mathcal{P}(\Theta)}\frac{1}{T}\sum _{t=1}^{T}\operatorname{KL}(\rho _{t}^{\*}||\rho)$. Besides, the task similarity in Theorem 2 is defined according to the specific property (i.e. $\epsilon$-radius) of the piecewise-Lipschitz function, and hence is particularly applicable to the piecewise-Lipschitz setting.\
$\mathbf{(2)}$ The action-space-discretization technique (described in Section 4.1 to obtain the first regret bound in Theorem 2) is of independent interest. The defined task similarity in Theorem 2 also requires a novel action-space-discretization method to translate the minimization problem $\min _{\rho \in \Theta}f _{t}(\rho)$ over the set of distributions into a tractable online convex optimization problem. \
$\mathbf{(3)}$ At present, the first regret bound has higher application value than the second regret bound. The Algorithm 1 (corresponding to the first regret bound in our Theorem 2) can be applied in the continuum domain, but Algorithm 2 (corresponding to the second regret bound in our Theorem 3) at the current stage is still not easy to be applied in the continuum domain (see more explanations in our Remark H.1). \
$\mathbf{(4)}$ The first improved regret is obtained under the same assumptions (i.e. piecewise-Lipschitz and bounded loss functions) as that in [3], via a more technical analysis. We list this improved regret in our work to make a fair comparison and show rigorous improvements over the existing work [3].

Reviewer comment (Theory):
In relation to the issue in Point 1, Remark B.1 mentions that alternative update rules of $\rho_{t1}$ may address the problem, but questions remain: (i) are there candidate update rules in mind? and (ii) how badly might they impact the regret bound? Specifically, is it plausible that an analog to Proposition 4 could yield the same or nearly the same good task-averaged upper-regret bound as Theorem 3?

Author response:
**Q2. Is it plausible that an analog to Proposition 4 could be found that would still yield the same or almost the same good task-averaged upper-regret bound as Theorem 3?**\
A2: Thanks. Our explanations for the possible analog to Proposition 4 are three-fold: \
$\mathbf{(1)}$ Its existence: we believe that the analog likely exists, because our regret bound for FTL in Proposition 4 only holds for the sequence of $\operatorname{KL}$-divergence $\operatorname{KL}(\rho_{t}^{\*}||\rho)$. However, we can obtain for generalized EWA algorithm (see [3, Thm 2.1]) regret bounds that have a similar form as that in our Eq.(3) but replace the $\operatorname{KL}$-divergence with other kinds of $f$-divergence (e.g. the $\chi^{2}$-divergence in [3, Cor 2.4]). Using online algorithms to minimize the sequence of $f$-divergence may lead to a new regret bound. \
$\mathbf{(2)}$ Its convergence rate w.r.t. $T$: the convergence rate of regret bound in the analog will not be better than the logarithmic regret in our Proposition 4. Because without additional information of the loss function, logarithmic regret is optimal (see more explanations for online convex optimization in Appendix A). \
$\mathbf{(3)}$ Its tightness when setting optimal distribution $\rho _{t}^{\*}$ as a Dirac measure: In online non-convex setting where the regret is defined as the gap between the cumulative loss (w.r.t. a sequence of distributions $\\{\rho _{ti}\\} _{i \in [m]}$)  and the minimal loss (w.r.t. the optimal distribution $\rho _{t}^{\*}$), the regret bounds of online algorithm always involve a divergence between the optimal distribution $\rho _{t}^{\*}$ and the initial distribution $\rho _{t1}$ (see examples in [3, Thm 2.1]). All of these $f$-divergences may be vacuous if we set $\rho _{t}^{\*}$ as a Dirac measure, indicating to some extent the limitation of the regret  (i.e. the gap between the cumulative loss and the minimal loss w.r.t. the optimal distribution $\rho _{t}^{\*}$) defined for EWA-type algorithm.

Reviewer comment (Theory):
The appendix (Remark H.1, which does not appear to be referenced in Section 4) notes that the analytic form of the RN derivatives cannot be computed in practice, even for uniform or Gaussian distributions, which further raises concerns over the significance of Theorem 3.

Author response:
**Q3. It is noted in the appendix (Remark H.1, which was not referenced in Section 4) that the analytic form of the RN derivatives can’t be computed in practice. This also leads to concerns over the significance of Theorem 3.**\
A3: Thanks for this suggestion, we have referenced Remark H.1 at the end of Remark 2 in Section 4 in the revised version. Our explanations for the significance of our Theorem 3 lie in 3 aspects: \
$\mathbf{(1)}$ It is truly that at the current stage we are unable to compute the analytic form of RN derivative of probability measure over continuum domain, because of the initialization update rule $\rho_{t1} = \frac{1}{t-1}\sum_{s=1}^{t-1}\rho_{s}^{\*}$. We treat this issue as one of the ongoing research directions of this work. \
$\mathbf{(2)}$ We also need to point out that our Algorithm 2 is applicable to the discrete domain (e.g. to the Expert Advice problem for online model selection) where the updating rule $\rho_{t1} = \frac{1}{t-1}\sum_{s=1}^{t-1}\rho_{s}^{\*}$ corresponds to the averaging of previous $t-1$ discrete probability density vectors, which is more feasible and applicable than over continuum domain. \
$\mathbf{(3)}$ Theorem 3 is not only useful to derive task-averaged regret bound for OWO meta learning algorithms, but also necessary to be applied to give PAC-Bayes generalization bounds for statistical meta learning model in our Theorem 5. This is also one of the significances of our Theorem 3.

Reviewer comment (Experiments):
The experiments only show Algorithm 1’s performance and do not include results from (Balcan et al. (2021)) or any other baseline, even though the experiments are set up identically to those in (Balcan et al. (2021)).

Author response:
**Q4. It was good to include some experiments, but only Algorithm 1’s performance is shown, not (Balcan et al. (2021)) or any other baseline.**\
A4: Thanks for your comments. More differences between our Algorithm 1 and the state-of-the-art algorithm from (Balcan et al. (2021)) have been discussed in Remark H.2 of Appendix H of the revised version, and our explanations are two-fold: \
$\mathbf{(1)}$ Actually, we choose single-task EWA algorithm as our baseline in all experiments. In both Figure 1 ($m$-shot multi-task method v.s. single-task baseline) and Figure 2 ($T =5$ multi-task method v.s. $T=1$ single-task baseline), we show advantages of the meta-learning based EWA algorithm over the single-task EWA algorithm baseline, and verify the convergence performance of regret of our meta learning algorithm. \
$\mathbf{(2)}$ It is true that our experimental setting is the same as that in (Balcan et al. (2021)). However, (Balcan et al. (2021)) used a heuristic method to learn (instead of meta learn) the step size $\lambda _{t}$, so their implementation is not rigorously the same as the pseudo code in their paper (i.e. their Algorithm 3). We follow almost the same implementation details as in (Balcan et al. (2021)) (which is also explained in detail in the Experimental Setting part of our Section H), with the main difference being that we verify the task-averaged regret bound over the training tasks but (Balcan et al. (2021)) verify the regret bound over the test tasks.

Reviewer comment (Experiments):
Can you add some discussion for problem parameter sizes? For some potential applications, would the number of iterations $m$ be large while the number of tasks $T$ be much smaller, or vice versa?

Author response:
**Q6. Can you add some discussion for problem parameter sizes --- eg for some potential applications would the number of iterations be large while the number of tasks be much smaller or vice versa?**\
A6: Yes. Consider the autonomous vehicles application, a self-driving car may drive in different environments (for example in a superhighway, in the miry road of the rural area, at the crossroads of a city, etc) and these environments can be considered as sequential different tasks. In each task, the self-driving car takes photos of the environment, uses the photos for training, and finally adjusts its route. The photos taken in different environments can be considered as the training samples available in a sequential order. In the above application, the number of environments is the number $T$ of training tasks in OWO meta learning, and the number of photos taken in each environment is the number $m$ of training samples per task. If the self-driving car encounters many environments, then maybe $T >> m$; if the car drives in the same environment for a long time and takes many photos, then maybe $m >> T$.

Reviewer comment (Writing):
In the related works section, the statement “These regret bounds are irrelevant to the sample size m per task…” is ambiguous. Does this mean ‘the regret bounds do not depend on the sample size m per task,’ or something else?

Author response:
**Q7. (Related works) “These regret bounds are irrelevant to the sample size m per task…” does that mean ‘the regret bounds do not depend on the sample size m per task’ or something else?**\
A7: Yes. The regret bounds (e.g. $O(\log{T})$ or $O(\sqrt{T})$) for OWB meta learning only depend on the number $T$ of training tasks, and the properties of loss functions (e.g. $L$-Lipschitzness, smoothness, etc), but do not depend on the sample size $m$ per task.