Paper: Meta-Learning with Task-Environment Interaction (ICLR-2024)

Reviewer comment (Experiments):
The authors only conducted experiments on the MAML model and its variant, which cannot show the improvement of TIML. There should be some experiments on the recent models to show the improvement of the TIML, e.g., SKD, PAL.

Author response:
The two methods proposed, SKD (Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks) and PAL (Partner-Assisted Learning for Few-Shot Image Classification), fall under the category of knowledge distillation approaches. They exhibit significant differences from the overall framework of MAML (Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks). However, all three methods belong to the realm of few-shot and meta-learning.  
    The approach presented in this paper can be applied to both SKD and PAL. In the field of meta-learning, the Task Interaction Mechanism (TIML) aims to make better decisions under the assumption of random task sampling. Knowledge distillation, on the other hand, involves creating a teacher model and a student model at the sample level, which contrasts with the random sampling approach proposed in this paper. The choice of random sampling is motivated by the need to adapt to the unpredictability of nature and foster learning in situations where the future environment cannot be foreseen.  
    The Task Interaction Mechanism proposed in this paper is applicable in scenarios with multiple parallel choices. SKD and PAL exhibit characteristics of mixing various tasks during task operations. Therefore, applying the proposed mechanism is likely to result in improved performance in these methods.  
   Thank you very much for your review. I will update the PDF in a few days, and I welcome your feedback and corrections.

Reviewer comment (Experiments):
Can your model be combined with other models besides MAML? Would the experimental result improve in other model?

Author response:
The two methods proposed, SKD (Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks) and PAL (Partner-Assisted Learning for Few-Shot Image Classification), fall under the category of knowledge distillation approaches. They exhibit significant differences from the overall framework of MAML (Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks). However, all three methods belong to the realm of few-shot and meta-learning.  
    The approach presented in this paper can be applied to both SKD and PAL. In the field of meta-learning, the Task Interaction Mechanism (TIML) aims to make better decisions under the assumption of random task sampling. Knowledge distillation, on the other hand, involves creating a teacher model and a student model at the sample level, which contrasts with the random sampling approach proposed in this paper. The choice of random sampling is motivated by the need to adapt to the unpredictability of nature and foster learning in situations where the future environment cannot be foreseen.  
    The Task Interaction Mechanism proposed in this paper is applicable in scenarios with multiple parallel choices. SKD and PAL exhibit characteristics of mixing various tasks during task operations. Therefore, applying the proposed mechanism is likely to result in improved performance in these methods.  
   Thank you very much for your review. I will update the PDF in a few days, and I welcome your feedback and corrections.

Reviewer comment (Novelty):
The method does not do what it sets out to do. The authors claim that their method has an advantage over previous task-selection methods, in that it can handle the incoming tasks as they are provided, rather than constructing synthetic tasks as in other methods ("In the process of natural selection, we cannot choose whether tomorrow will be sunny or rainy;"). But that is not what is happening. The method requires that a batch of tasks can be sampled, and then that some of these tasks may be prioritized and others ignored during the meta-learning phase, which is equivalent to choosing whether you're seeing rain or sun, and basically similar to synthesizing tasks. Thus it is not clear what advantage is provided over existing task selection methods, including those mentioned by the authors themselves.

Author response:
Thank you very much for your review. During the training of a batch task group randomly sampled, only a few steps of gradient descent are employed. This process does not continuously consume a significant amount of resources for training. Additionally, to address the concern of random/uninformative tasks in a batch, the proposed task environment interaction mechanism incorporates a random selection process. This mechanism avoids persistent learning in tasks with poor performance.
   Your characterization of the method presented in this paper as similar to those constructing synthetic tasks is not accurate. Currently, the synthesis of tasks that I am familiar with involves preprocessing the entire task pool, evaluating and judging tasks from various perspectives, and combining them to create new tasks. This approach incurs substantial resource costs. In contrast, the method proposed in this paper introduces a training decision during the training process without affecting the overall framework. Task selection is based on the premise that some tasks already demonstrate good performance on the query set, and thus, there is no need to train on them. The meta-learning discussed in this paper aims to find a starting point that is effective for the majority of tasks and quickly adapt it towards tasks with poorer performance.

Reviewer comment (Writing):
The method is poorly described, making it difficult to understand (see below).

Author response:
I apologize if the wording has caused confusion. Some aspects of this paper may have been abrupt. The PDF will be updated in the next few days to address these issues. I hope this clarifies any confusion.

Reviewer comment (Novelty):
Selecting tasks by difficulty is hardly a novel idea. Juergen Schmidhuber has long emphasized the need to concentrate on tasks that offer optimal difficulty (allowing for maximal learning, neither too hard nor too easy). More recently, the POET algorithm of Wang et al. (arXiv 1901.01753) largely selects new environments by their difficulty for the current generation of learners. This past research suggests that the method proposed by the authors (just favor the task with the highest intermediate loss) would fail badly on a very simple case: the presence of impossible or random tasks, which always generate a high loss without allowing for significant learning. This would largely eliminate learning under the method proposed here (but not standard MAML, which would still benefit from the non-impossible tasks).

Author response:
Thank you very much for your review. During the training of a batch task group randomly sampled, only a few steps of gradient descent are employed. This process does not continuously consume a significant amount of resources for training. Additionally, to address the concern of random/uninformative tasks in a batch, the proposed task environment interaction mechanism incorporates a random selection process. This mechanism avoids persistent learning in tasks with poor performance.

Reviewer comment (Experiments):
The actual algorithm, although confusingly reported, seems to be quite different from standard MAML. In particular, it seems that the gradient updates to the initialization parameters are computed and accumulated over every step of the inner loop. Then the sum of these computed updates is applied to the initialization parameters in line 20 of Algo 1. This is quite different from MAML, which first goes through an entire inner loop (resulting in a single updated parameter theta_i), then computes a single loss over the "query set" (using theta_i), and takes the gradient of *that* over initialization parameters theta. This makes the experiments uninformative about the benefit of the approach. If the standard MAML baselines were implemented by this same incremental method, they do not represent actual MAML performance. If, on the other hand, MAML was implemented in the standard way, we do not know whether the tiny improvements result from the task selection or from the novel (and puzzling) scheme of incremental outer-loop gradients.

Author response:
The summing method used in line 20 of Algorithm 1 is consistent with MAML's Algorithm 1, line eight, both employing a summation. You may refer back to the original MAML paper for confirmation. The core innovation of the proposed task environment interaction mechanism lies in two aspects. Firstly, adjusting more optimal training strategies under the condition of random sampling. Secondly, altering the traditional outer loop update of MAML. Instead of using the sum of losses for all tasks in a batch to update the outer loop meta-parameters, we only use the summed losses of the selected training tasks. This accelerates the convergence towards tasks with poorer performance, enhancing overall generalization. Tasks that do not participate in the outer loop meta-parameter update are excluded because the meta-parameters already perform well on these tasks, and there is no need to bring them closer. We aim to find a starting point that is favorable for the majority of tasks.

Reviewer comment (Writing):
Minor: the paper is poorly written, with many typos. The algorithm is unclear (see below), and the discussion of related work is actually spread over sections 1, 2 and 3.

Author response:
I apologize if the wording has caused confusion. Some aspects of this paper may have been abrupt. The PDF will be updated in the next few days to address these issues. I hope this clarifies any confusion.

Reviewer comment (Experiments):
How do you think the algorithm would behave in the presence of impossible or random/uninformative tasks, as mentioned above?

Author response:
Thank you very much for your review. During the training of a batch task group randomly sampled, only a few steps of gradient descent are employed. This process does not continuously consume a significant amount of resources for training. Additionally, to address the concern of random/uninformative tasks in a batch, the proposed task environment interaction mechanism incorporates a random selection process. This mechanism avoids persistent learning in tasks with poor performance.

Reviewer comment (Reproducibility):
Confirm whether the outer-loop update (to the initialization parameters) is computed as the sum of intermediate gradients at each time step of the inner loop, which is what lines 14 and 20 in Algo 1 denote in their current form. How were the MAML baselines implemented?

Author response:
The summing method used in line 20 of Algorithm 1 is consistent with MAML's Algorithm 1, line eight, both employing a summation. You may refer back to the original MAML paper for confirmation. The core innovation of the proposed task environment interaction mechanism lies in two aspects. Firstly, adjusting more optimal training strategies under the condition of random sampling. Secondly, altering the traditional outer loop update of MAML. Instead of using the sum of losses for all tasks in a batch to update the outer loop meta-parameters, we only use the summed losses of the selected training tasks. This accelerates the convergence towards tasks with poorer performance, enhancing overall generalization. Tasks that do not participate in the outer loop meta-parameter update are excluded because the meta-parameters already perform well on these tasks, and there is no need to bring them closer. We aim to find a starting point that is favorable for the majority of tasks.

Reviewer comment (Theory):
A critical aspect of the manuscript that requires enhancement is the depth of rationale and exposition provided for the central idea, particularly concerning the reward function in Equation 4. While the concept as presented appears somewhat intuitive, the narrative lacks a detailed explanation or theoretical background that justifies this specific formulation.

Author response:
In Formula 4, $\mathcal{L}_{\mathcal{T}_i}^{\mathcal{D}_i^Q}(t)$ represents the model's performance on the selected task in the query set. $R_i(t-1)$ denotes the cumulative reward for this task. A larger difference between these two values indicates a more significant improvement in training effectiveness. To ensure fair comparisons across different tasks, this difference is normalized by dividing it by the accumulated selection count for the chosen task. The updated future reward value is obtained by adding this normalized difference to the previous cumulative reward value. 
    The core idea of self-paced learning is to simulate human cognitive mechanisms by initially learning simple and universally applicable knowledge structures and gradually increasing difficulty, transitioning to more complex and specialized knowledge. However, the task interaction mechanism mentioned in this paper involves directly selecting more challenging tasks for learning, skipping the process of training on all tasks from easy to difficult. We adopt a selective training approach, balancing training resources among tasks with greater future returns while keeping the number of training iterations constant. The goal of meta-parameters is to establish a good starting point on different tasks. We begin with task assessments, prioritizing the training of tasks with higher difficulty and future returns. Information from tasks involved in the training is used to update meta-parameters, allowing them to converge more rapidly toward tasks with less favorable outcomes. This approach differs significantly from self-paced learning.

Reviewer comment (Novelty):
The paper significantly underperforms in establishing its novelty, particularly within the saturated domain of task scheduling algorithms, well-documented by sources [1-6]. A striking omission is the lack of a comprehensive comparison or substantive discussion related to these established works. Both the experimental and related work sections of the paper are noticeably thin on comparative analysis, undermining the paper's credibility and scholarly rigor.

Author response:
In the three articles, "Adaptive Task Sampling for Meta-Learning," "Probabilistic Active Meta-Learning," and "Meta-Learning with an Adaptive Task Scheduler," as well as in the proposed Task Interaction Mechanism (TIML) in this paper, the methods share the perspective that tasks randomly sampled are suboptimal. The distinction lies in the fact that these three methods operate on a task pool containing all tasks, assuming that all tasks are known. They employ different approaches to assess and evaluate tasks before feeding them into the training process. In contrast, TIML, as proposed in this paper, makes superior choices for task training under the premise of random sampling, aiming to adapt to the environment without altering it. Meta-learning, belonging to the field of few-shot learning, faces challenges in acquiring samples, and the sample landscape continuously evolves over time. Continuous learning is required when new samples emerge. If operations are performed on the entire task pool, incorporating new samples requires mixing all samples for task sampling to achieve meaningful results, rendering previous training outcomes obsolete. With TIML, incorporating new samples into the previously trained results is sufficient for continued learning.

Reviewer comment (Experiments):
In the experimental design, the authors restrict their focus to MAML-based methods, neglecting a host of other task scheduling approaches documented in [1-6]. This narrow scope fails to justify the proposed method's advantages within the broader context of the field. By not contrasting their results with these established methodologies, the authors miss the opportunity to demonstrate the superiority or the distinctive aspect of their approach.

Author response:
In the three articles, "Adaptive Task Sampling for Meta-Learning," "Probabilistic Active Meta-Learning," and "Meta-Learning with an Adaptive Task Scheduler," as well as in the proposed Task Interaction Mechanism (TIML) in this paper, the methods share the perspective that tasks randomly sampled are suboptimal. The distinction lies in the fact that these three methods operate on a task pool containing all tasks, assuming that all tasks are known. They employ different approaches to assess and evaluate tasks before feeding them into the training process. In contrast, TIML, as proposed in this paper, makes superior choices for task training under the premise of random sampling, aiming to adapt to the environment without altering it. Meta-learning, belonging to the field of few-shot learning, faces challenges in acquiring samples, and the sample landscape continuously evolves over time. Continuous learning is required when new samples emerge. If operations are performed on the entire task pool, incorporating new samples requires mixing all samples for task sampling to achieve meaningful results, rendering previous training outcomes obsolete. With TIML, incorporating new samples into the previously trained results is sufficient for continued learning.

Reviewer comment (Novelty):
At its core, the proposed method appears to be an oversimplified version of its predecessors, rather than a groundbreaking innovation. However, this perception may be somewhat influenced by the paper's lack of clarity, making it difficult to ascertain the full scope of the proposed methodology.

Author response:
The core idea of self-paced learning is to simulate human cognitive mechanisms by initially learning simple and universally applicable knowledge structures and gradually increasing difficulty, transitioning to more complex and specialized knowledge. However, the task interaction mechanism mentioned in this paper involves directly selecting more challenging tasks for learning, skipping the process of training on all tasks from easy to difficult. We adopt a selective training approach, balancing training resources among tasks with greater future returns while keeping the number of training iterations constant. The goal of meta-parameters is to establish a good starting point on different tasks. We begin with task assessments, prioritizing the training of tasks with higher difficulty and future returns. Information from tasks involved in the training is used to update meta-parameters, allowing them to converge more rapidly toward tasks with less favorable outcomes. This approach differs significantly from self-paced learning.