Original: Set-based Meta-Interpolation for Few-Task Meta-Learning
Revised: Set-based Meta-Interpolation for Few-Task Meta-Learning

Original: Abstract
Revised: Abstract

Original: Meta-learning approaches enable machine learning systems to adapt to new tasks given few examples by leveraging knowledge from related tasks.
Revised: Meta-learning approaches enable machine learning systems to adapt to new tasks given few examples by leveraging knowledge from related tasks.

Original: However, a large number of meta-training tasks are still required for generalization to unseen tasks during meta-testing, which introduces a critical bottleneck for real-world problems that come with only few tasks, due to various reasons including the difficulty and cost of constructing tasks.
Revised: However, a large number of meta-training tasks are still required for generalization to unseen tasks during meta-testing, which introduces a critical bottleneck for real-world problems that come with only few tasks, due to various reasons including the difficulty and cost of constructing tasks.

Original: Recently, several task augmentation methods have been proposed to tackle this issue using domain-specific knowledge to design augmentation techniques to densify the meta-training task distribution.
Revised: Recently, several task augmentation methods have been proposed to tackle this issue using domain-specific knowledge to design augmentation techniques to densify the meta-training task distribution.

Original: However, such reliance on domain-specific knowledge renders these methods inapplicable to other domains.
Revised: However, such reliance on domain-specific knowledge renders these methods inapplicable to other domains.

Original: While Manifold Mixup based task augmentation methods are domain-agnostic, we empirically find them ineffective on non-image domains.
Revised: While Manifold Mixup based task augmentation methods are domain-agnostic, we empirically find them ineffective on non-image domains.

Original: To tackle these limitations, we propose a novel domain-agnostic task augmentation method, Meta-Interpolation, which utilizes expressive neural set functions to densify the meta-training task distribution using bilevel optimization.
Revised: To tackle these limitations, we propose a novel domain-agnostic task augmentation method, Meta-Interpolation, which utilizes expressive neural set functions to densify the meta-training task distribution using bilevel optimization.

Original: We empirically validate the efficacy of Meta-Interpolation on eight datasets spanning across various domains such as image classification, molecule property prediction, text classification and speech recognition.
Revised: We empirically validate the efficacy of Meta-Interpolation on eight datasets spanning across various domains such as image classification, molecule property prediction, text classification and sound classificattion.

Original: Experimentally, we show that Meta-Interpolation consistently outperforms all the relevant baselines.
Revised: Experimentally, we show that Meta-Interpolation consistently outperforms all the relevant baselines.

Original: Theoretically, we prove that task interpolation with the set function regularizes the meta-learner to improve generalization.
Revised: Theoretically, we prove that task interpolation with the set function regularizes the meta-learner to improve generalization.

Original:  Introduction
Revised:  Introduction

Original: The ability to learn a new task given only a few examples is crucial for artificial intelligence.
Revised: The ability to learn a new task given only a few examples is crucial for artificial intelligence.

Original: Recently, meta-learning [39, 3] has emerged as a viable method to achieve this objective and enables machine learning systems to quickly adapt to a new task by leveraging knowledge from other related tasks seen during meta-training.
Revised: Recently, meta-learning [39, 3] has emerged as a viable method to achieve this objective and enables machine learning systems to quickly adapt to a new task by leveraging knowledge from other related tasks seen during meta-training.

Original: Although existing meta-learning methods can efficiently adapt to new tasks with few data samples, a large dataset of meta-training tasks is still required to learn meta-knowledge that can be transferred to unseen tasks.
Revised: Although existing meta-learning methods can efficiently adapt to new tasks with few data samples, a large dataset of meta-training tasks is still required to learn meta-knowledge that can be transferred to unseen tasks.

Original: For many real-world applications, such extensive collections of meta-training tasks may be unavailable.
Revised: For many real-world applications, such extensive collections of meta-training tasks may be unavailable.

Original: Such scenarios give rise to the few-task meta-learning problem where a meta-learner can easily memorize the meta-training tasks but fail to generalize well to unseen tasks.
Revised: Such scenarios give rise to the few-task meta-learning problem where a meta-learner can easily memorize the meta-training tasks but fail to generalize well to unseen tasks.

Original: The few-task meta-learning problem usually results from the difficulty in task generation and data collection.
Revised: The few-task meta-learning problem usually results from the difficulty in task generation and data collection.

Original: For instance, in the medical domain, it is infeasible to collect a large amount of data to construct extensive meta-training tasks due to privacy concerns.
Revised: For instance, in the medical domain, it is infeasible to collect a large amount of data to construct extensive meta-training tasks due to privacy concerns.

Original: Moreover, for natural language processing, it is not straightforward to split a dataset into tasks, and hence entire datasets are treated as tasks [30].
Revised: Moreover, for natural language processing, it is not straightforward to split a dataset into tasks, and hence entire datasets are treated as tasks [30].

Original: [Figure]
Revised: [Figure]

Original: Several works have been proposed to tackle the few-task meta-learning problem using task augmentation techniques such as clustering a dataset into multiple tasks [30], leveraging strong image augmentation methods such as vertical flipping to construct new classes [32], and the employment of Manifold Mixup [44] for densifying the meta-training task distribution [49, 50].
Revised: Several works have been proposed to tackle the few-task meta-learning problem using task augmentation techniques such as clustering a dataset into multiple tasks [30], leveraging strong image augmentation methods such as vertical flipping to construct new classes [32], and the employment of Manifold Mixup [44] for densifying the meta-training task distribution [49, 50].

Original: However, majority of these techniques require domain-specific knowledge to design such task augmentations and hence cannot be applied to other domains.
Revised: However, majority of these techniques require domain-specific knowledge to design such task augmentations and hence cannot be applied to other domains.

Original: While Manifold Mixup based methods [49, 50] are domain-agnostic, we empirically find them ineffective for mitigating meta-overfitting in few-task meta-learning especially in non-image domains such as chemical and text, and that they sometimes degrade generalization performance.
Revised: While Manifold Mixup based methods [49, 50] are domain-agnostic, we empirically find them ineffective for mitigating meta-overfitting in few-task meta-learning especially in non-image domains such as chemical and text, and that they sometimes degrade generalization performance.

Original: In this work, we focus solely on domain-agnostic task augmentation methods that can densify the meta-training task distribution to prevent meta-overfitting and improve generalization at metatesting for few-task meta-learning.
Revised: In this work, we focus solely on domain-agnostic task augmentation methods that can densify the meta-training task distribution to prevent meta-overfitting and improve generalization at metatesting for few-task meta-learning.

Original: To tackle the limitations already discussed, we propose a novel domain-agnostic task augmentation method for metric based meta-learning models.
Revised: To tackle the limitations already discussed, we propose a novel domain-agnostic task augmentation method for metric based meta-learning models.

Original: Our method,
Revised: Our method,

Original: Meta-Interpolation, utilizes expressive neural set functions to interpolate two tasks and the set functions are trained with bilevel optimization so that a meta-learner trained on the interpolated tasks generalizes to tasks in the meta-validation set.
Revised: Meta-Interpolation, utilizes expressive neural set functions to interpolate two tasks and the set functions are trained with bilevel optimization so that a meta-learner trained on the interpolated tasks generalizes to tasks in the meta-validation set.

Original: As a consequence of end-to-end training, the learned augmentation strategy is tailored to each specific domain without the need for specialized domain knowledge.
Revised: As a consequence of end-to-end training, the learned augmentation strategy is tailored to each specific domain without the need for specialized domain knowledge.

Original: For example, for K -way classification, we sample two tasks consisting of support and query sets and assign a new class k to each pair of classes { σ ( k ) , σ ′ ( k ) } for k = 1 , . . .
Revised: For example, for K -way classification, we sample two tasks consisting of support and query sets and assign a new class k to each pair of classes { σ ( k ) , σ ′ ( k ) } for k = 1 , . . .

Original: , K , where σ, σ ′ are permutations on { 1 , . . .
Revised: , K , where σ, σ ′ are permutations on { 1 , . . .

Original: , K } as depicted in Figure 1a.
Revised: , K } as depicted in Figure 1a.

Original: Hidden representations of the support set with classes σ ( k ) and σ ′ ( k ) are then transformed into a single support set using a set function that maps a set of two vectors to a single vector.
Revised: Hidden representations of the support set with classes σ ( k ) and σ ′ ( k ) are then transformed into a single support set using a set function that maps a set of two vectors to a single vector.

Original: We refer to the output of the set function as the interpolated support set and these are used to compute class prototypes.
Revised: We refer to the output of the set function as the interpolated support set and these are used to compute class prototypes.

Original: As shown in Figure 1b, the interpolated support set is paired with a query set (query set 1 in Figure 1a)), randomly selected from the two tasks to obtain a new task.
Revised: As shown in Figure 1b, the interpolated support set is paired with a query set (query set 1 in Figure 1a)), randomly selected from the two tasks to obtain a new task.

Original: Lastly, we optimize the set function so that a meta-learner trained on the augmented task can minimize the loss on the meta-validation tasks as illustrated in Figure 1c.
Revised: Lastly, we optimize the set function so that a meta-learner trained on the augmented task can minimize the loss on the meta-validation tasks as illustrated in Figure 1c.

Original: To verify the efficacy of our method, we empirically show that it significantly improves the performance of Prototypical Networks [40] on the few-task meta-learning problem across multiple domains.
Revised: To verify the efficacy of our method, we empirically show that it significantly improves the performance of Prototypical Networks [40] on the few-task meta-learning problem across multiple domains.

Original: Our method outperforms the relevant baselines on eight few-task meta-learning benchmark datasets spanning image classification, chemical property prediction, text classification, and automatic speech recognition.
Revised: Our method outperforms the relevant baselines on eight few-task meta-learning benchmark datasets spanning image classification, chemical property prediction, text classification, and sound classification.

Original: Furthermore, our theoretical analysis shows that our task interpolation method with the set function regularizes the meta-learner and improves generalization performance.
Revised: Furthermore, our theoretical analysis shows that our task interpolation method with the set function regularizes the meta-learner and improves generalization performance.

Original: Our contribution is threefold:
Revised: Our contribution is threefold:

Original: • We propose a novel domain-agnostic task augmentation method, Meta-Interpolation, which leverages expressive set functions to densify the meta-training task distribution for the few-task meta-learning problem.
Revised: • We propose a novel domain-agnostic task augmentation method, Meta-Interpolation, which leverages expressive set functions to densify the meta-training task distribution for the few-task meta-learning problem.

Original: • We theoretically analyze our model and show that it regularizes the meta-learner for better generalization.
Revised: • We theoretically analyze our model and show that it regularizes the meta-learner for better generalization.

Original: • Through extensive experiments, we show that Meta-Interpolation significantly improves the performance of Prototypical Network on various domains such as image, text, and chemical molecule classification, and speech recognition on the few-task meta-learning problem.
Revised: • Through extensive experiments, we show that Meta-Interpolation significantly improves the performance of Prototypical Network on various domains such as image, text, and chemical molecule, and sound classification on the few-task meta-learning problem.

Original:  Related Work
Revised:  Related Work

Original: Meta-Learning The two mainstream approaches to meta-learning are gradient based [10, 33, 14, 24, 12, 36, 37] and metric based meta-learning [45, 40, 42, 29, 26, 6, 38].
Revised: Meta-Learning The two mainstream approaches to meta-learning are gradient based [10, 33, 14, 24, 12, 36, 37] and metric based meta-learning [45, 40, 42, 29, 26, 6, 38].

Original: The former formulates meta-knowledge as meta-parameters such as the initial model parameters and performs bilevel optimization to estimate the meta-parameters so that a meta-learner can generalize to unseen tasks with few gradient steps.
Revised: The former formulates meta-knowledge as meta-parameters such as the initial model parameters and performs bilevel optimization to estimate the meta-parameters so that a meta-learner can generalize to unseen tasks with few gradient steps.

Original: The latter learns an embedding space where classification is performed by measuring the distance between a query and a set of class prototypes.
Revised: The latter learns an embedding space where classification is performed by measuring the distance between a query and a set of class prototypes.

Original: In this work, we focus on metric based meta-learning with fewer number of meta-training tasks, i.e., few-task meta-learning.
Revised: In this work, we focus on metric based meta-learning with fewer number of meta-training tasks, i.e., few-task meta-learning.

Original: We propose a novel task augmentation method that densifies the meta-training task distribution and mitigates overfitting due to the fewer number of meta-training tasks for better generalization to unseen tasks.
Revised: We propose a novel task augmentation method that densifies the meta-training task distribution and mitigates overfitting due to the fewer number of meta-training tasks for better generalization to unseen tasks.

Original: Task Augmentation for Few-Task Meta-learning Several methods have been proposed to augment the number of meta-training tasks to mitigate overfitting in the context of few-task meta-learning.
Revised: Task Augmentation for Few-Task Meta-learning Several methods have been proposed to augment the number of meta-training tasks to mitigate overfitting in the context of few-task meta-learning.

Original: Ni et al.
Revised: Ni et al.

Original: [32] apply strong data augmentations such as vertical flip to images to create a new class.
Revised: [32] apply strong data augmentations such as vertical flip to images to create a new class.

Original: For text classification, Murty et al.
Revised: For text classification, Murty et al.

Original: [30] split meta-training tasks into latent reasoning categories by clustering data with a pretrained language model.
Revised: [30] split meta-training tasks into latent reasoning categories by clustering data with a pretrained language model.

Original: However, they require domain-specific knowledge to design such augmentations, and hence the resulting augmentation techniques are inapplicable to other domains where there is no well-defined data augmentation or pretrained model.
Revised: However, they require domain-specific knowledge to design such augmentations, and hence the resulting augmentation techniques are inapplicable to other domains where there is no well-defined data augmentation or pretrained model.

Original: In order to tackle this limitation, Manifold Mixup-based task augmentations have also been proposed.
Revised: In order to tackle this limitation, Manifold Mixup-based task augmentations have also been proposed.

Original: MetaMix [49] interpolates support and query sets with Manifold Mixup [44] to construct a new query set.
Revised: MetaMix [49] interpolates support and query sets with Manifold Mixup [44] to construct a new query set.

Original: MLTI [50] performs Manifold Mixup [44] on support and query sets from two tasks for task augmentation.
Revised: MLTI [50] performs Manifold Mixup [44] on support and query sets from two tasks for task augmentation.

Original: Although these methods are domain-agnostic, we empirically find that they are not effective in some domains and can degrade generalization performance.
Revised: Although these methods are domain-agnostic, we empirically find that they are not effective in some domains and can degrade generalization performance.

Original: In contrast, we propose to train an expressive neural set function to interpolate two tasks with bilevel optimization to find optimal augmentation strategies tailored specifically to each domain.
Revised: In contrast, we propose to train an expressive neural set function to interpolate two tasks with bilevel optimization to find optimal augmentation strategies tailored specifically to each domain.

Original:  Method
Revised:  Method

Original: Preliminaries In meta-learning, we are given a finite set of tasks {T t } Tt =1 , which are i.i.d samples from an unknown task distribution p ( T ) .
Revised: Preliminaries In meta-learning, we are given a finite set of tasks {T t } Tt =1 , which are i.i.d samples from an unknown task distribution p ( T ) .

Original: Each task T t consists of a support set D st = { ( x st,i , y st,i ) } N s i
Revised: Each task T t consists of a support set D st = { ( x st,i , y st,i ) } N s i

Original: = and a query set D qt = { ( x qt,i , y qt,i ) } N q i =1 , where x t,i and y t,i denote a data point and its corresponding label respectively.
Revised: = and a query set D qt = { ( x qt,i , y qt,i ) } N q i =1 , where x t,i and y t,i denote a data point and its corresponding label respectively.

Original: Given a predictive model ˆ f θ,λ :=
Revised: Given a predictive model ˆ f θ,λ :=

Original: f Lθ L ◦ · · · ◦ f l +1 θ l +1 ◦ φ λ ◦ f lθ l ◦ · · · ◦ f 1 θ 1 with L layers, we want to estimate the parameter θ that minimizes the meta-training loss and generalizes to query sets D q ∗ sampled from an unseen task T ∗ using the support set D s ∗ , where λ is a hyperparameter for the function φ λ .
Revised: f Lθ L ◦ · · · ◦ f l +1 θ l +1 ◦ φ λ ◦ f lθ l ◦ · · · ◦ f 1 θ 1 with L layers, we want to estimate the parameter θ that minimizes the meta-training loss and generalizes to query sets D q ∗ sampled from an unseen task T ∗ using the support set D s ∗ , where λ is a hyperparameter for the function φ λ .

Original: In this work, we primarily focus on metric based meta-learning methods rather than gradient based meta-learning methods due to efficiency and empirically higher performance over the gradient based methods on the tasks we consider.
Revised: In this work, we primarily focus on metric based meta-learning methods rather than gradient based meta-learning methods due to efficiency and empirically higher performance over the gradient based methods on the tasks we consider.

Original: Problem Statement
Revised: Problem Statement

Original: In this work, we focus solely on few-task meta-learning.
Revised: In this work, we focus solely on few-task meta-learning.

Original: Here, the number of meta-training tasks drawn from the meta-training distribution is extremely small and the goal of a meta-learner is to learn meta-knowledge from such limited tasks that can be transferred to unseen tasks during meta-testing.
Revised: Here, the number of meta-training tasks drawn from the meta-training distribution is extremely small and the goal of a meta-learner is to learn meta-knowledge from such limited tasks that can be transferred to unseen tasks during meta-testing.

Original: The key challenges here are preventing the meta-learner from overfitting on the meta-training tasks and generalizing to unseen tasks drawn from a meta-test set.
Revised: The key challenges here are preventing the meta-learner from overfitting on the meta-training tasks and generalizing to unseen tasks drawn from a meta-test set.

Original: Metric Based Meta-Learning The goal of metric based meta-learning is to learn an embedding space induced by ˆ f θ,λ , where we perform classification by computing distances between data points and class prototypes.
Revised: Metric Based Meta-Learning The goal of metric based meta-learning is to learn an embedding space induced by ˆ f θ,λ , where we perform classification by computing distances between data points and class prototypes.

Original: We adopt Prototypical Network (ProtoNet) [40] for ˆ f θ,λ , where φ λ is the identity function.
Revised: We adopt Prototypical Network (ProtoNet) [40] for ˆ f θ,λ , where φ λ is the identity function.

Original: Specifically, for each task T t with its corresponding support D st and query D qt sets, we compute class prototypes { c k } Kk =1 as the average of the hidden representation of the support samples belonging to the class k as follows:
Revised: Specifically, for each task T t with its corresponding support D st and query D qt sets, we compute class prototypes { c k } Kk =1 as the average of the hidden representation of the support samples belonging to the class k as follows:

Original: [Equation]
Revised: [Equation]

Original: where N k denotes the number of instances belonging to the class k .
Revised: where N k denotes the number of instances belonging to the class k .

Original: Given a metric d ( · , · ) : R D ×
Revised: Given a metric d ( · , · ) : R D ×

Original: R D (cid:55)→ R , we compute the probability of a query point x qt,i being assigned to the class k by measuring the distance between the hidden representation ˆ f θ,λ ( x qt,i ) and the class prototype c k followed by
Revised: R D (cid:55)→ R , we compute the probability of a query point x qt,i being assigned to the class k by measuring the distance between the hidden representation ˆ f θ,λ ( x qt,i ) and the class prototype c k followed by

Original: [Table]
Revised: [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: dation
Revised: dation

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: 9: v 1 − =
Revised: 9: v 1 − =

Original: α · grad ( ∂ L tr ∂θ , θ, grad_outputs = v 1 ) 10: p += v 1 11: end for 12: v 2 ← grad ( L tr ∂θ , λ, grad_outputs = α p ) 13: return ∂ L V ∂λ − v 2 14: 15: softmax.
Revised: α · grad ( ∂ L tr ∂θ , θ, grad_outputs = v 1 ) 10: p += v 1 11: end for 12: v 2 ← grad ( L tr ∂θ , λ, grad_outputs = α p ) 13: return ∂ L V ∂λ − v 2 14: 15: softmax.

Original: With the class probability, we compute the cross-entropy loss for ProtoNet as follows:
Revised: With the class probability, we compute the cross-entropy loss for ProtoNet as follows:

Original: [Equation]
Revised: [Equation]

Original: where 1 is an indicator function.
Revised: where 1 is an indicator function.

Original: At meta-test time, a test query is assigned a label based on the minimal distance to a class prototype, i.e., y q ∗ = arg min k d
Revised: At meta-test time, a test query is assigned a label based on the minimal distance to a class prototype, i.e., y q ∗ = arg min k d

Original: ( ˆ f θ,λ ( x q ∗ ) , c k ) .
Revised: ( ˆ f θ,λ ( x q ∗ ) , c k ) .

Original: However, optimizing 1 T (cid:80) Tt =
Revised: However, optimizing 1 T (cid:80) Tt =

Original: 1 L singleton ( λ, θ ; T t ) w.r.t θ is prone to overfitting since we are given only a small number of meta-training tasks.
Revised: 1 L singleton ( λ, θ ; T t ) w.r.t θ is prone to overfitting since we are given only a small number of meta-training tasks.

Original: The meta-learner tends to memorize the meta-training tasks, which limits its generalization to new tasks at meta-test time [51, 35].
Revised: The meta-learner tends to memorize the meta-training tasks, which limits its generalization to new tasks at meta-test time [51, 35].

Original: Meta-Interpolation for Task Augmentation
Revised: Meta-Interpolation for Task Augmentation

Original: In order to tackle the meta-overfitting problem with a small number of tasks, we propose a novel data-driven domain-agnostic task augmentation framework which enables the meta-learner trained on few tasks to generalize to unseen few-shot classification tasks.
Revised: In order to tackle the meta-overfitting problem with a small number of tasks, we propose a novel data-driven domain-agnostic task augmentation framework which enables the meta-learner trained on few tasks to generalize to unseen few-shot classification tasks.

Original: Several methods have been proposed to densify the meta-training tasks.
Revised: Several methods have been proposed to densify the meta-training tasks.

Original: However, they heavily depend on the augmentation of images [32] or need a pretrained language model for task augmentation [30].
Revised: However, they heavily depend on the augmentation of images [32] or need a pretrained language model for task augmentation [30].

Original: Although Manifold Mixup based methods [49, 50] are domain-agnostic, we empirically find them ineffective in certain domains.
Revised: Although Manifold Mixup based methods [49, 50] are domain-agnostic, we empirically find them ineffective in certain domains.

Original: Instead, we optimize expressive neural set functions to augment tasks to enhance the generalization of a meta-learner to unseen tasks.
Revised: Instead, we optimize expressive neural set functions to augment tasks to enhance the generalization of a meta-learner to unseen tasks.

Original: As a consequence of end-to-end training, the learned augmentation strategy is tailored to each domain.
Revised: As a consequence of end-to-end training, the learned augmentation strategy is tailored to each domain.

Original: Specifically, let φ λ : R n × d → R d be a set function which maps a set of d dimensional vectors with cardinality n to a d dimensional vector.
Revised: Specifically, let φ λ : R n × d → R d be a set function which maps a set of d dimensional vectors with cardinality n to a d dimensional vector.

Original: In all our experiments, we use Set Transformer [23] for φ λ .
Revised: In all our experiments, we use Set Transformer [23] for φ λ .

Original: Given a pair of tasks T t 1 = {D st 1 , D qt 1 } and T t 2 = {D st 2 , D qt 2 } with corresponding support and query sets for K way classification, we construct new classes by choosing K pairs of classes from the two tasks.
Revised: Given a pair of tasks T t 1 = {D st 1 , D qt 1 } and T t 2 = {D st 2 , D qt 2 } with corresponding support and query sets for K way classification, we construct new classes by choosing K pairs of classes from the two tasks.

Original: We sample permutations σ t 1 and σ t 2 on { 1 , . . .
Revised: We sample permutations σ t 1 and σ t 2 on { 1 , . . .

Original: , K } for each task T t 1 and T t 2 respectively and assign class k to the pair { σ t 1 ( k ) , σ t 2 ( k ) } for k = 1 , . .
Revised: , K } for each task T t 1 and T t 2 respectively and assign class k to the pair { σ t 1 ( k ) , σ t 2 ( k ) } for k = 1 , . .

Original: , K .
Revised: , K .

Original: For the newly assigned class k , we pair two instances from classes σ t 1 ( k ) and σ t 2 ( k ) and interpolate their hidden representations with the set function φ λ .
Revised: For the newly assigned class k , we pair two instances from classes σ t 1 ( k ) and σ t 2 ( k ) and interpolate their hidden representations with the set function φ λ .

Original: The class prototypes for class k are computed using the output of φ λ as follows:
Revised: The class prototypes for class k are computed using the output of φ λ as follows:

Original: [Equation]
Revised: [Equation]

Original: where we define ˆ D s to be the set of all the interpolated prototypes ˆ c k for k = 1 , .
Revised: where we define ˆ D s to be the set of all the interpolated prototypes ˆ c k for k = 1 , .

Original: , K .
Revised: , K .

Original: For queries, we do not perform any interpolation.
Revised: For queries, we do not perform any interpolation.

Original: Instead, we use D qt 1 as the query set and compute its hidden representation ˆ f θ,λ ( x qt 1 ,i ) ∈ R D .
Revised: Instead, we use D qt 1 as the query set and compute its hidden representation ˆ f θ,λ ( x qt 1 ,i ) ∈ R D .

Original: We then measure the distance between the query with y qt 1 ,i =
Revised: We then measure the distance between the query with y qt 1 ,i =

Original: σ t 1 ( k ) and the interpolated prototype of class k to compute the loss as follows:
Revised: σ t 1 ( k ) and the interpolated prototype of class k to compute the loss as follows:

Original: [Equation]
Revised: [Equation]

Original: where ˆ T = { ˆ D s , D qt 1 } .
Revised: where ˆ T = { ˆ D s , D qt 1 } .

Original: The intuition behind interpolating only support sets is to construct harder tasks that a meta-learner cannot memorize.
Revised: The intuition behind interpolating only support sets is to construct harder tasks that a meta-learner cannot memorize.

Original: Alternatively, we can interpolate only query sets.
Revised: Alternatively, we can interpolate only query sets.

Original: However, this is computationally more expensive since the size of query sets is usually larger than that of support sets.
Revised: However, this is computationally more expensive since the size of query sets is usually larger than that of support sets.

Original: In Section 5, we empirically show that interpolating either support or query sets achieves higher training loss than interpolating both, which empirically supports the intuition.
Revised: In Section 5, we empirically show that interpolating either support or query sets achieves higher training loss than interpolating both, which empirically supports the intuition.

Original: Lastly, we also use the original task T t 1 to evaluate the loss L singleton ( λ, θ, T t 1 ) in Eq.
Revised: Lastly, we also use the original task T t 1 to evaluate the loss L singleton ( λ, θ, T t 1 ) in Eq.

Original: by passing the corresponding support and query set to ˆ f θ,λ .
Revised: by passing the corresponding support and query set to ˆ f θ,λ .

Original: The additional forward pass enriches the diversity of the augmented tasks and makes meta-training consistent with meta-testing since we do not perform any task augmentation in the meta-testing stage.
Revised: The additional forward pass enriches the diversity of the augmented tasks and makes meta-training consistent with meta-testing since we do not perform any task augmentation in the meta-testing stage.

Original: Optimization
Revised: Optimization

Original: Since jointly optimizing θ , the parameters of ProtoNet, and λ , the parameters of the set function φ λ , with few tasks is prone to overfitting, we consider λ as hyperparameters and perform bilevel optimization with meta-training and meta-validation tasks as follows:
Revised: Since jointly optimizing θ , the parameters of ProtoNet, and λ , the parameters of the set function φ λ , with few tasks is prone to overfitting, we consider λ as hyperparameters and perform bilevel optimization with meta-training and meta-validation tasks as follows:

Original: [Equation]
Revised: [Equation]

Original: where T train t , T val t , ˆ T t denote the meta-training, meta-validation, and interpolated task, respectively.
Revised: where T train t , T val t , ˆ T t denote the meta-training, meta-validation, and interpolated task, respectively.

Original: Since computing the exact gradient w.r.t λ is intractable due to the long inner optimization steps in
Revised: Since computing the exact gradient w.r.t λ is intractable due to the long inner optimization steps in

Original: Eq. 6, we leverage the implicit function theorem to approximate the gradient as Lorraine et al.
Revised: Eq. 6, we leverage the implicit function theorem to approximate the gradient as Lorraine et al.

Original: Moreover, we alternately update θ and λ for computational efficiency as described in Algo.
Revised: Moreover, we alternately update θ and λ for computational efficiency as described in Algo.

Original: 1 and 2.
Revised: 1 and 2.

Original:  Theoretical Analysis
Revised:  Theoretical Analysis

Original: In this section, we theoretically investigate the behavior of the Set Transformer and how it induces a distribution dependent regularization, which is then shown to have the ability to control the
Revised: In this section, we theoretically investigate the behavior of the Set Transformer and how it induces a distribution dependent regularization, which is then shown to have the ability to control the

Original: Rademacher complexity for better generalization.
Revised: Rademacher complexity for better generalization.

Original: To analyze the behavior of the Set Transformer, we first define it concretely with the attention mechanism A ( Q, K, V ) =
Revised: To analyze the behavior of the Set Transformer, we first define it concretely with the attention mechanism A ( Q, K, V ) =

Original: softmax( √ d − 1 QK ⊤ ) V .
Revised: softmax( √ d − 1 QK ⊤ ) V .

Original: Given h, h ′ ∈ R d , define H { h,h ′ }=
Revised: Given h, h ′ ∈ R d , define H { h,h ′ }=

Original: [ h, h ′ ]
Revised: [ h, h ′ ]

Original: ⊤ ∈ R 2 × d
Revised: ⊤ ∈ R 2 × d

Original: and H { h }= h ⊤ ∈ R 1 × d .
Revised: and H { h }= h ⊤ ∈ R 1 × d .

Original: Then, for any r ∈ {{ h, h ′ } , { h }} , the output of the Set Transformer φ λ ( r ) is defined as follows:
Revised: Then, for any r ∈ {{ h, h ′ } , { h }} , the output of the Set Transformer φ λ ( r ) is defined as follows:

Original: [Equation]
Revised: [Equation]

Original: where Q 2 = SW Q 2 + b Q 2 , Q r 1 = H r 1 W Q 1 + 1 2 b Q 1 , K rj
Revised: where Q 2 = SW Q 2 + b Q 2 , Q r 1 = H r 1 W Q 1 + 1 2 b Q 1 , K rj

Original: = H rj W Kj + 1 2 b Kj , V rj
Revised: = H rj W Kj + 1 2 b Kj , V rj

Original: = H rj W Vj + 1 2 b Vj (for j ∈ { 1 , 2 } ), and H r 2 =
Revised: = H rj W Vj + 1 2 b Vj (for j ∈ { 1 , 2 } ), and H r 2 =

Original: A ( Q r 1 , K r 1 , V r 1 ) ∈ R n × d .
Revised: A ( Q r 1 , K r 1 , V r 1 ) ∈ R n × d .

Original: Q j , K j , V j denote query, key, and value for the attention mechanism for j = 1 , 2 , respectively.
Revised: Q j , K j , V j denote query, key, and value for the attention mechanism for j = 1 , 2 , respectively.

Original: Here, 1 2 = [1 , . . .
Revised: Here, 1 2 = [1 , . . .

Original: , 1] ⊤ ∈ R n , W Qj , W Kj , W Vj ∈
Revised: , 1] ⊤ ∈ R n , W Qj , W Kj , W Vj ∈

Original: R d × d , b Qj , b Kj , b Vj ∈ R 1 × d , Q r 1 , K rj , V rj ∈ R n × d , and Q 2 ∈ R 1 × d .
Revised: R d × d , b Qj , b Kj , b Vj ∈ R 1 × d , Q r 1 , K rj , V rj ∈ R n × d , and Q 2 ∈ R 1 × d .

Original: Let l ∈ { 1 , . .
Revised: Let l ∈ { 1 , . .

Original: , L } .
Revised: , L } .

Original: Our analysis will show the importance of the following quantity of the Set Transformer in our method:
Revised: Our analysis will show the importance of the following quantity of the Set Transformer in our method:

Original: [Equation]
Revised: [Equation]

Original: ) where
Revised: ) where

Original: [Equation]
Revised: [Equation]

Original: with h t,i =
Revised: with h t,i =

Original: ϕ lθ ( x st,i ) and ϕ lθ
Revised: ϕ lθ ( x st,i ) and ϕ lθ

Original: = f lθ l ◦ · · · ◦ f 1 θ 1 .
Revised: = f lθ l ◦ · · · ◦ f 1 θ 1 .

Original: For a matrix A ∈ R m × n , A i,j denotes the entry for i -th row and j -th column of the matrix A .
Revised: For a matrix A ∈ R m × n , A i,j denotes the entry for i -th row and j -th column of the matrix A .

Original: We now introduce the additional notation and problem setting to present our results.
Revised: We now introduce the additional notation and problem setting to present our results.

Original: Define W =
Revised: Define W =

Original: ( W V 1 W V 2 )
Revised: ( W V 1 W V 2 )

Original: ⊤ ∈ R d × d , b =
Revised: ⊤ ∈ R d × d , b =

Original: ( b V 1 W V 2 + b V 2 )
Revised: ( b V 1 W V 2 + b V 2 )

Original: ⊤ ∈ R d , L t ( c ) = − 1 n (cid:80) ni
Revised: ⊤ ∈ R d , L t ( c ) = − 1 n (cid:80) ni

Original: =1 log exp( − d
Revised: =1 log exp( − d

Original: [Equation]
Revised: [Equation]

Original: ˆ f θ,λ ( x qt,i ) ,
Revised: ˆ f θ,λ ( x qt,i ) ,

Original: c yqt,i ))
Revised: c yqt,i ))

Original: (cid:80)
Revised: (cid:80)

Original: [Equation]
Revised: [Equation]

Original: ,i ) , c k ′ )) , and I t,k
Revised: ,i ) , c k ′ )) , and I t,k

Original: = { i ∈ [ N ( t ) s ] : y st,i = k } , where N ( t ) s =
Revised: = { i ∈ [ N ( t ) s ] : y st,i = k } , where N ( t ) s =

Original: |D st | .
Revised: |D st | .

Original: We also define the empirical measure µ t,k = 1 | I t,k | (cid:80) i ∈
Revised: We also define the empirical measure µ t,k = 1 | I t,k | (cid:80) i ∈

Original: I t,k δ i over the index i ∈
Revised: I t,k δ i over the index i ∈

Original: [ N ( t ) s ] with the Dirac measures
Revised: [ N ( t ) s ] with the Dirac measures

Original: δ i .
Revised: δ i .

Original: Let U [ K ] be the uniform distribution over { 1 , . . .
Revised: Let U [ K ] be the uniform distribution over { 1 , . . .

Original: , K } .
Revised: , K } .

Original: For any function φ and point u in its domain, we define the j -th order tensor ∂
Revised: For any function φ and point u in its domain, we define the j -th order tensor ∂

Original: j φ ( u ) ∈ R d × d ×···× d by ∂ j φ ( u ) i 1 i 2 ··· i j =
Revised: j φ ( u ) ∈ R d × d ×···× d by ∂ j φ ( u ) i 1 i 2 ··· i j =

Original: [Equation]
Revised: [Equation]

Original: ∂u i 1 u i 2 ··· ∂u ij φ ( u ) .
Revised: ∂u i 1 u i 2 ··· ∂u ij φ ( u ) .

Original: For example, ∂ 1 φ ( u ) and ∂ 2 φ ( u ) are the gradient and the Hessian of φ evaluated at u .
Revised: For example, ∂ 1 φ ( u ) and ∂ 2 φ ( u ) are the gradient and the Hessian of φ evaluated at u .

Original: For any j -th order tensor ∂ j φ ( u ) , we define the vectorization of the tensor by vec[ ∂ j φ ( u )]
Revised: For any j -th order tensor ∂ j φ ( u ) , we define the vectorization of the tensor by vec[ ∂ j φ ( u )]

Original: ∈ R d j .
Revised: ∈ R d j .

Original: For an vector a ∈ R d , we define a ⊗ j = a ⊗ a ⊗ · · · ⊗ a ∈ R d j where ⊗ represents the Kronecker product.
Revised: For an vector a ∈ R d , we define a ⊗ j = a ⊗ a ⊗ · · · ⊗ a ∈ R d j where ⊗ represents the Kronecker product.

Original: We assume that ∂ r g (cid:0) Wϕ lθ l ( x st,i )
Revised: We assume that ∂ r g (cid:0) Wϕ lθ l ( x st,i )

Original: + b (cid:1)
Revised: + b (cid:1)

Original: = 

Revised: =

Original: for all r ≥ 2 , where g :=
Revised: for all r ≥ 2 , where g :=

Original: f Lθ L ◦ · · · ◦ f l +1 θ l +1 .
Revised: f Lθ L ◦ · · · ◦ f l +1 θ l +1 .

Original: This assumption is satisfied, for example, if g represents a deep neural network with ReLU activations.
Revised: This assumption is satisfied, for example, if g represents a deep neural network with ReLU activations.

Original: This assumption is also satisfied in the simpler special case considered in the proposition below.
Revised: This assumption is also satisfied in the simpler special case considered in the proposition below.

Original: The following theorem shows that L mix ( λ, θ, ˆ T t,t ′ ) is approximately L singleton ( λ, θ ; T t ) plus regularization terms on the directional derivatives of ϕ lθ l on the direction of W ( ϕ lθ l ( x st ′ ,j )
Revised: The following theorem shows that L mix ( λ, θ, ˆ T t,t ′ ) is approximately L singleton ( λ, θ ; T t ) plus regularization terms on the directional derivatives of ϕ lθ l on the direction of W ( ϕ lθ l ( x st ′ ,j )

Original: − ϕ lθ l ( x st,i )) :
Revised: − ϕ lθ l ( x st,i )) :

Original: Theorem 1.
Revised: Theorem 1.

Original: For any J ∈ N + , if c (cid:55)→ d ( y, c ) is J -times differentiable for all y , then the J -th order approximation of L mix ( λ, θ, ˆ T t,t ′ ) is given by L singleton ( λ, θ ; T t )
Revised: For any J ∈ N + , if c (cid:55)→ d ( y, c ) is J -times differentiable for all y , then the J -th order approximation of L mix ( λ, θ, ˆ T t,t ′ ) is given by L singleton ( λ, θ ; T t )

Original: + (cid:80)
Revised: + (cid:80)

Original: Jj =1 1 j ! vec[ ∂ j L t ( c )]
Revised: Jj =1 1 j ! vec[ ∂ j L t ( c )]

Original: ⊤ ∆ ⊗ j , where ∆ =
Revised: ⊤ ∆ ⊗ j , where ∆ =

Original: [∆ ⊤ 1 , . . . , ∆ ⊤ K ] ⊤ and
Revised: [∆ ⊤ 1 , . . . , ∆ ⊤ K ] ⊤ and

Original: [Equation]
Revised: [Equation]

Original: To illustrate the effect of this data-dependent regularization, we now consider the following special case that is used by Yao et al.
Revised: To illustrate the effect of this data-dependent regularization, we now consider the following special case that is used by Yao et al.

Original: [50] for ProtoNet: L ( λ, θ ; T t )
Revised: [50] for ProtoNet: L ( λ, θ ; T t )

Original: = 1 n (cid:80) ni
Revised: = 1 n (cid:80) ni

Original: =

Revised: =

Original: L i ( λ, θ ; T t ) where L i ( λ, θ ; T t ) =
Revised: L i ( λ, θ ; T t ) where L i ( λ, θ ; T t ) =

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: st
Revised: st

Original: [Equation]
Revised: [Equation]

Original: ∈D
Revised: ∈D

Original: st
Revised: st

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: } x st
Revised: } x st

Original: [Equation]
Revised: [Equation]

Original: and ⟨· , ·⟩ denotes dot product.
Revised: and ⟨· , ·⟩ denotes dot product.

Original: Define c = 1 n (cid:80) ni
Revised: Define c = 1 n (cid:80) ni

Original: =1 14 ψ ( z t,i )( ψ ( z t,i ) − 0 . 5) 1+exp( z t,i ) , where ψ ( z t,i ) =
Revised: =1 14 ψ ( z t,i )( ψ ( z t,i ) − 0 . 5) 1+exp( z t,i ) , where ψ ( z t,i ) =

Original: exp( z t,i ) 1+exp( z t,i ) and z t,i = ⟨ x qt,i − ( c ′ 1 + c ′ 2 ) / 2 , θ ⟩ .
Revised: exp( z t,i ) 1+exp( z t,i ) and z t,i = ⟨ x qt,i − ( c ′ 1 + c ′ 2 ) / 2 , θ ⟩ .

Original: Note that c > 

Revised: Note that c >

Original: if θ is no worse than the random guess; e.g., L i ( λ, 0; T t ) >
Revised: if θ is no worse than the random guess; e.g., L i ( λ, 0; T t ) >

Original: L i ( λ, θ ; T t ) for all i ∈ [ n ] .
Revised: L i ( λ, θ ; T t ) for all i ∈ [ n ] .

Original: We write ∥ v ∥ 2 M = v ⊤ Mv for any positive semi-definite matrix M .
Revised: We write ∥ v ∥ 2 M = v ⊤ Mv for any positive semi-definite matrix M .

Original: In this special case, we consider that α is balanced: i.e.,
Revised: In this special case, we consider that α is balanced: i.e.,

Original: E t ′ ,σ [ (cid:80) 2 k = 1 | I t ′ ,σ ( k ) | (cid:80)
Revised: E t ′ ,σ [ (cid:80) 2 k = 1 | I t ′ ,σ ( k ) | (cid:80)

Original: j ∈
Revised: j ∈

Original: I t
Revised: I t

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: t,
Revised: t,

Original: [Equation]
Revised: [Equation]

Original: | (cid:80)
Revised: | (cid:80)

Original: [Equation]
Revised: [Equation]

Original: t,
Revised: t,

Original: [Equation]
Revised: [Equation]

Original: t,t
Revised: t,t

Original: [Equation]
Revised: [Equation]

Original: ( ϕ lθ l ( x st ′ ,j ) − ϕ lθ l ( x st,i ))]
Revised: ( ϕ lθ l ( x st ′ ,j ) − ϕ lθ l ( x st,i ))]

Original: = 0 for all t .
Revised: = 0 for all t .

Original: This is used to prevent the Set Transformer from over-fitting to the training sets; i.e., in such simple special cases, the Set Transformer without any restriction is too expressive relative to the rest of the model (and may memorize the training sets without using the rest of the model).
Revised: This is used to prevent the Set Transformer from over-fitting to the training sets; i.e., in such simple special cases, the Set Transformer without any restriction is too expressive relative to the rest of the model (and may memorize the training sets without using the rest of the model).

Original: The following proposition shows that the additional regularization term is simplified to the form of c ∥ θ ∥ 2 M in this special case:
Revised: The following proposition shows that the additional regularization term is simplified to the form of c ∥ θ ∥ 2 M in this special case:

Original: Proposition 1.
Revised: Proposition 1.

Original: In the special case explained above, the second approximation of
Revised: In the special case explained above, the second approximation of

Original: E t ′ ,σ [ L mix ( λ, θ, ˆ T t,t ′ )] is given by L singleton ( λ, θ ; T t )
Revised: E t ′ ,σ [ L mix ( λ, θ, ˆ T t,t ′ )] is given by L singleton ( λ, θ ; T t )

Original: + c ∥ θ ∥ 2 E t ′ ,σ [ δ t,t ′ ,σ δ ⊤ t,t ′ ,σ ] , where δ t,t ′ ,σ =
Revised: + c ∥ θ ∥ 2 E t ′ ,σ [ δ t,t ′ ,σ δ ⊤ t,t ′ ,σ ] , where δ t,t ′ ,σ =

Original: [Equation]
Revised: [Equation]

Original: In the above regularization form, we have an implicit regularization effect on ∥ θ ∥ 2Σ where Σ =
Revised: In the above regularization form, we have an implicit regularization effect on ∥ θ ∥ 2Σ where Σ =

Original: E x , x ′ [( x − x ′ )( x − x ′ ) ⊤ ] .
Revised: E x , x ′ [( x − x ′ )( x − x ′ ) ⊤ ] .

Original: The following theorem shows that this implicit regularization can reduce the Rademacher complexity for better generalization:
Revised: The following theorem shows that this implicit regularization can reduce the Rademacher complexity for better generalization:

Original: Proposition 2.
Revised: Proposition 2.

Original: Let F R = { x (cid:55)→ θ ⊤ x : ∥ θ ∥ 2Σ ≤ R } with E x [ x ] = 0 .
Revised: Let F R = { x (cid:55)→ θ ⊤ x : ∥ θ ∥ 2Σ ≤ R } with E x [ x ] = 0 .

Original: Then, R n ( F R ) ≤ √ R √ rank(Σ) √ n .
Revised: Then, R n ( F R ) ≤ √ R √ rank(Σ) √ n .

Original: All the proofs are presented in Appendix A.
Revised: All the proofs are presented in Appendix A.

Original:  Experiments
Revised:  Experiments

Original: We now demonstrate the efficacy of our set-based task augmentation method on multiple few-task benchmark datasets and compare against the relevant baselines.
Revised: We now demonstrate the efficacy of our set-based task augmentation method on multiple few-task benchmark datasets and compare against the relevant baselines.

Original: Datasets We perform classification on eight datasets to validate our method.
Revised: Datasets We perform classification on eight datasets to validate our method.

Original: Metabolism [17], NCI [31] and Tox21 [18]: these are binary classification datasets for predicting the properties of chemical molecules.
Revised: Metabolism [17], NCI [31] and Tox21 [18]: these are binary classification datasets for predicting the properties of chemical molecules.

Original: For Metabolism, we use three subdatasets for meta-training, meta-validation, and meta-testing, respectively.
Revised: For Metabolism, we use three subdatasets for meta-training, meta-validation, and meta-testing, respectively.

Original: For NCI, we use four subdatasets for meta-training, two for meta-validation and the remaining three for meta-testing.
Revised: For NCI, we use four subdatasets for meta-training, two for meta-validation and the remaining three for meta-testing.

Original: For Tox21, we use six subdatasets
Revised: For Tox21, we use six subdatasets

Original: [Table]
Revised: [Table]

Original: for meta-training, two for meta-validation
Revised: for meta-training, two for meta-validation

Original: [Table]
Revised: [Table]

Original: and four for meta-testing.
Revised: and four for meta-testing.

Original: (4) GLUE-SciTail [30]: it consists of four natural language inference datasets where we predict whether a hypothesis sentence contradicts a premise sentence.
Revised: (4) GLUE-SciTail [30]: it consists of four natural language inference datasets where we predict whether a hypothesis sentence contradicts a premise sentence.

Original: We use MNLI [47] and QNLI [46] for meta-training, SNLI [5] and RTE [46] for meta-validation, and SciTail [20] for meta-testing.
Revised: We use MNLI [47] and QNLI [46] for meta-training, SNLI [5] and RTE [46] for meta-validation, and SciTail [20] for meta-testing.

Original: (5) ESC-50 [34]: this is an automatic speech recognition dataset.
Revised: (5) ESC-50 [34]: this is an environmental sound recognition dataset.

Original: We make a 20/15/15 split out of 50 base classes for metatraining/validation/testing and sample 5 classes from each spilt to construct a 5-way classification task.
Revised: We make a 20/15/15 split out of 50 base classes for metatraining/validation/testing and sample 5 classes from each spilt to construct a 5-way classification task.

Original: (6) Rainbow MNIST (RMNIST) [11]: this is a 10-way classification dataset.
Revised: (6) Rainbow MNIST (RMNIST) [11]: this is a 10-way classification dataset.

Original: Following Yao et al.
Revised: Following Yao et al.

Original: [50], we construct each task by applying compositions of image transformations to the images of the MNIST [9] dataset.
Revised: [50], we construct each task by applying compositions of image transformations to the images of the MNIST [9] dataset.

Original: (7) & (8) Mini-ImageNet-S [45] and CIFAR100-FS [22]: these are 5-way classification datasets where we choose 12/16/20 classes out of 100 base classes for metatraining/validation/testing, respectively and sample 5 classes from each split to construct a task.
Revised: (7) & (8) Mini-ImageNet-S [45] and CIFAR100-FS [22]: these are 5-way classification datasets where we choose 12/16/20 classes out of 100 base classes for metatraining/validation/testing, respectively and sample 5 classes from each split to construct a task.

Original: Note that Metabolism, Tox21, NCI, GLUE-SciTail, and ESC-50 are real-world few-task metalearning datasets with a very small number of tasks.
Revised: Note that Metabolism, Tox21, NCI, GLUE-SciTail, and ESC-50 are real-world few-task metalearning datasets with a very small number of tasks.

Original: For Mini-ImageNet-S and CIFAR100-FS, following Yao et al.
Revised: For Mini-ImageNet-S and CIFAR100-FS, following Yao et al.

Original: [50], we artificially reduce the number of tasks from the original datasets for few-task meta-learning.
Revised: [50], we artificially reduce the number of tasks from the original datasets for few-task meta-learning.

Original: RMNIST is synthetically generated by applying augmentations to MNIST.
Revised: RMNIST is synthetically generated by applying augmentations to MNIST.

Original: Implementation Details For RMNIST, Mini-ImageNet-S, and CIFAR100-FS, we use four convolutional blocks with each block consisting of a convolution, ReLU, batch normalization [19], and max pooling.
Revised: Implementation Details For RMNIST, Mini-ImageNet-S, and CIFAR100-FS, we use four convolutional blocks with each block consisting of a convolution, ReLU, batch normalization [19], and max pooling.

Original: For Metabolism, Tox21, and NCI, we convert the chemical molecules into SMILES format and extract a 1024 bit fingerprint feature using RDKit [15] where each bit captures a fragment of the molecule.
Revised: For Metabolism, Tox21, and NCI, we convert the chemical molecules into SMILES format and extract a 1024 bit fingerprint feature using RDKit [15] where each bit captures a fragment of the molecule.

Original: We use two blocks of affine transformation, batch normalization, and Leaky ReLU, and affine transformation for the last layer.
Revised: We use two blocks of affine transformation, batch normalization, and Leaky ReLU, and affine transformation for the last layer.

Original: For GLUE-SciTail dataset, we stack 3 fully connected layers with ReLU on the pretrained language model ELECTRA [8].
Revised: For GLUE-SciTail dataset, we stack 3 fully connected layers with ReLU on the pretrained language model ELECTRA [8].

Original: For ESC-50 dataset, we pass raw audio signal to the pretrained VGGish [16] feature extractor to obtain an embedding vector.
Revised: For ESC-50 dataset, we pass raw audio signal to the pretrained VGGish [16] feature extractor to obtain an embedding vector.

Original: We use the feature vector as input to the classifier which is exactly the same as the one used for Metabolism,
Revised: We use the feature vector as input to the classifier which is exactly the same as the one used for Metabolism,

Original: Tox21, and NCI.
Revised: Tox21, and NCI.

Original: For our Meta-Interpolation, we use Set Transformer [23] for the set function φ λ .
Revised: For our Meta-Interpolation, we use Set Transformer [23] for the set function φ λ .

Original: Baselines We compare our method against following domain-agnostic baselines.
Revised: Baselines We compare our method against following domain-agnostic baselines.

Original: ProtoNet [40]: Vanilla ProtoNet trained on Eq. 2 by fixing φ λ to be the
Revised: ProtoNet [40]: Vanilla ProtoNet trained on Eq. 2 by fixing φ λ to be the

Original: identity function
Revised: identity function

Original: MetaReg [2]: ProtoNet with ℓ 2 regularization where element-wise coefficients are learned with bilevel optimization.
Revised: MetaReg [2]: ProtoNet with ℓ 2 regularization where element-wise coefficients are learned with bilevel optimization.

Original: MetaMix [49]: ProtoNet trained with support sets and mixed query sets where we interpolate one instance from the support sets and the other from the original query sets with Manifold Mixup.
Revised: MetaMix [49]: ProtoNet trained with support sets and mixed query sets where we interpolate one instance from the support sets and the other from the original query sets with Manifold Mixup.

Original: MLTI [50]: ProtoNet trained with Manifold Mixup based task augmentation.
Revised: MLTI [50]: ProtoNet trained with Manifold Mixup based task augmentation.

Original: We sample two tasks and interpolate two query sets and support sets with Manifold Mixup, respectively.
Revised: We sample two tasks and interpolate two query sets and support sets with Manifold Mixup, respectively.

Original: ProtoNet+ST ProtoNet and Set Transformer ( φ λ ) trained with bilevel optimization but without task augmentation for L mix ( λ, θ, ˆ T t ) in Eq.
Revised: ProtoNet+ST ProtoNet and Set Transformer ( φ λ ) trained with bilevel optimization but without task augmentation for L mix ( λ, θ, ˆ T t ) in Eq.

Original: Meta-Interpolation Our full model learning to interpolate support sets from two tasks using bilevel optimization and training the ProtoNet with both the original and interpolated tasks.
Revised: Meta-Interpolation Our full model learning to interpolate support sets from two tasks using bilevel optimization and training the ProtoNet with both the original and interpolated tasks.

Original: Results As shown in Table 1, Meta-Interpolation consistently outperforms all the domain-agnostic task augmentation and regularization baselines on non-image domains.
Revised: Results As shown in Table 1, Meta-Interpolation consistently outperforms all the domain-agnostic task augmentation and regularization baselines on non-image domains.

Original: Notably, it significantly improves the performance on ESC-50, which is a challenging datatset that only contains 40 examples per class.
Revised: Notably, it significantly improves the performance on ESC-50, which is a challenging datatset that only contains 40 examples per class.

Original: In addition, Meta-Interpolation effectively tackles the Metabolism and GLUE-SciTail datasets which have an extremely small number of meta-training tasks: three and two meta-training tasks, respectively.
Revised: In addition, Meta-Interpolation effectively tackles the Metabolism and GLUE-SciTail datasets which have an extremely small number of meta-training tasks: three and two meta-training tasks, respectively.

Original: Contrarily, the baselines do not achieve consistent improvements across all the
Revised: Contrarily, the baselines do not achieve consistent improvements across all the

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: domains and tasks considered.
Revised: domains and tasks considered.

Original: For example, MetaReg is effective on the speech domain (ESC-50) and Metabolism, but does not work on the chemical (Tox21 and NCI) and text (GLUE-SciTail) domains.
Revised: For example, MetaReg is effective on the sound domain (ESC-50) and Metabolism, but does not work on the chemical (Tox21 and NCI) and text (GLUE-SciTail) domains.

Original: Similarly, MetaMix and MLTI achieve performance improvements on some datasets but degrade the test accuracy on others.
Revised: Similarly, MetaMix and MLTI achieve performance improvements on some datasets but degrade the test accuracy on others.

Original: This empirical evidence supports the hypothesis that the optimal task augmentation strategy varies across domains and justifies the motivation for Meta-Interpolation which learns augmentation strategies tailored to each domain.
Revised: This empirical evidence supports the hypothesis that the optimal task augmentation strategy varies across domains and justifies the motivation for Meta-Interpolation which learns augmentation strategies tailored to each domain.

Original: We provide additional experimental results on the image domain in Table 2.
Revised: We provide additional experimental results on the image domain in Table 2.

Original: Again, Meta-Interpolation outperforms all the baselines.
Revised: Again, Meta-Interpolation outperforms all the baselines.

Original: In contrast to the previous experiments, MetaReg hurts the generalization performance on all the image datasets except on RMNIST.
Revised: In contrast to the previous experiments, MetaReg hurts the generalization performance on all the image datasets except on RMNIST.

Original: Note that Manifold Mixup-based augmentation methods, MetaMix and MLTI, marginally improve the generalization performance for 1-shot classification on Mini-ImageNet-S and CIFAR-100-FS, although they boost the accuracy on 5-shot experiments.
Revised: Note that Manifold Mixup-based augmentation methods, MetaMix and MLTI, marginally improve the generalization performance for 1-shot classification on Mini-ImageNet-S and CIFAR-100-FS, although they boost the accuracy on 5-shot experiments.

Original: This suggests that different task augmentation strategies are required for 1-shot and 5-shot for the same dataset.
Revised: This suggests that different task augmentation strategies are required for 1-shot and 5-shot for the same dataset.

Original: Meta-Interpolation on the other hand learns task augmentation strategies tailored for each task and dataset and consistently improves the performance of the vanilla
Revised: Meta-Interpolation on the other hand learns task augmentation strategies tailored for each task and dataset and consistently improves the performance of the vanilla

Original: ProtoNet for all the experiments on the image datasets.
Revised: ProtoNet for all the experiments on the image datasets.

Original: Moreover, we plot the meta-training and meta-validation loss on RMNIST and NCI dataset in Figure 2.[Figure]
Revised: Moreover, we plot the meta-training and meta-validation loss on RMNIST and NCI dataset in Figure 2.

Original: -Interpolation obtains higher training loss but much lower validation loss than the others on both datasets.
Revised: Meta-Interpolation obtains higher training loss but much lower validation loss than the others on both datasets.

Original: This implies that interpolating only support sets constructs harder tasks that a meta-learner cannot memorize and regularizes the meta-learner for better generalization.
Revised: This implies that interpolating only support sets constructs harder tasks that a meta-learner cannot memorize and regularizes the meta-learner for better generalization.

Original: ProtoNet overfits to the meta-training tasks on both datasets.
Revised: ProtoNet overfits to the meta-training tasks on both datasets.

Original: MLTI mitigates the overfitting issue on RMNIST but is not effective on the NCI dataset where it shows high validation loss in Figure 2d.
Revised: MLTI mitigates the overfitting issue on RMNIST but is not effective on the NCI dataset where it shows high validation loss in Figure 2d.

Original: On the other hand,
Revised: On the other hand,

Original: MetaMix, which constructs a new query set by interpolating a support and query set with Manifold
Revised: MetaMix, which constructs a new query set by interpolating a support and query set with Manifold

Original: Mixup, results in generating overly difficult tasks which causes underfitting on RMNIST where the training loss is not properly minimized in Figure 2a.
Revised: Mixup, results in generating overly difficult tasks which causes underfitting on RMNIST where the training loss is not properly minimized in Figure 2a.

Original: However, this augmentation strategy is effective for tackling meta-overfitting on NCI where the validation loss is lower than ProtoNet.
Revised: However, this augmentation strategy is effective for tackling meta-overfitting on NCI where the validation loss is lower than ProtoNet.

Original: The loss curve of ProtoNet+ST supports the claim that increasing the model size and using bilevel optimization cannot handle the few-task meta-learning problem.
Revised: The loss curve of ProtoNet+ST supports the claim that increasing the model size and using bilevel optimization cannot handle the few-task meta-learning problem.

Original: It shows higher validation loss on both RMNIST and NCI as presented in Figure 2b and 2d.
Revised: It shows higher validation loss on both RMNIST and NCI as presented in Figure 2b and 2d.

Original: Similarly, MetaReg which learns coefficients for ℓregularization fails to prevent meta-overfitting on both datasts.
Revised: Similarly, MetaReg which learns coefficients for ℓregularization fails to prevent meta-overfitting on both datasts.

Original: Lastly, we empirically show that the performance gains mostly come from the task augmentation with
Revised: Lastly, we empirically show that the performance gains mostly come from the task augmentation with

Original: Meta-Interpolation, rather than from bilevel optimization or the introduction of extra parameters with the set function.
Revised: Meta-Interpolation, rather than from bilevel optimization or the introduction of extra parameters with the set function.

Original: As shown in Table 1 and 2, ProtoNet+ST, which is Meta Interpolation but trained without any task augmentation, significantly degrades the performance of ProtoNet on Mini-ImageNet and CIFAR-100-FS.
Revised: As shown in Table 1 and 2, ProtoNet+ST, which is Meta Interpolation but trained without any task augmentation, significantly degrades the performance of ProtoNet on Mini-ImageNet and CIFAR-100-FS.

Original: On the other datasets, the ProtoNet+ST obtains marginal improvement or largely
Revised: On the other datasets, the ProtoNet+ST obtains marginal improvement or largely

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: underperforms the other baselines.
Revised: underperforms the other baselines.

Original: Thus, the task augmentation strategy of interpolating two support sets with the set function φ λ is indeed crucial for tackling the few-task meta-learning problem.
Revised: Thus, the task augmentation strategy of interpolating two support sets with the set function φ λ is indeed crucial for tackling the few-task meta-learning problem.

Original: Ablation Study We further perform ablation studies to verify the effectiveness of each component of Meta-Interpolation.
Revised: Ablation Study We further perform ablation studies to verify the effectiveness of each component of Meta-Interpolation.

Original: In Table
Revised: In Table

Original: [Table]
Revised: [Table]

Original: we show experimental results on the ESC-50 dataset by removing various components of our model.
Revised: we show experimental results on the ESC-50 dataset by removing various components of our model.

Original: Firstly, we train our model without any task interpolation but keep the set function φ λ , denoted as w/o Interpolation.
Revised: Firstly, we train our model without any task interpolation but keep the set function φ λ , denoted as w/o Interpolation.

Original: The model without task interpolation significantly underperforms the full task-augmentation model, Meta-Interpolation, which shows that the improvements come from task interpolation rather than the extra parameters introduced by the set encoding layer.
Revised: The model without task interpolation significantly underperforms the full task-augmentation model, Meta-Interpolation, which shows that the improvements come from task interpolation rather than the extra parameters introduced by the set encoding layer.

Original: Moreover, bilevel optimization is shown to be effective for estimating λ , which are the parameters of the set function.
Revised: Moreover, bilevel optimization is shown to be effective for estimating λ , which are the parameters of the set function.

Original: Jointly training the ProtoNet and the set function without bilevel optimization, denoted as w/o Bilevel, largely degrades the test accuracy by 15% .
Revised: Jointly training the ProtoNet and the set function without bilevel optimization, denoted as w/o Bilevel, largely degrades the test accuracy by 15% .

Original: Lastly, we remove the loss L singleton ( λ, θ, T train t ) for inner optimization in Eq.
Revised: Lastly, we remove the loss L singleton ( λ, θ, T train t ) for inner optimization in Eq.

Original: 6, denoted as w/o
Revised: 6, denoted as w/o

Original: L singleton ( λ, θ, T train t ) .
Revised: L singleton ( λ, θ, T train t ) .

Original: This hurts the generalization performance since it decreases the diversity of tasks and causes inconsistency between meta-training and meta-testing, since we do not perform any interpolation for support sets at meta-test time.
Revised: This hurts the generalization performance since it decreases the diversity of tasks and causes inconsistency between meta-training and meta-testing, since we do not perform any interpolation for support sets at meta-test time.

Original: We also explore an alternative set function such as DeepSets [52] using the ESC50 dataset to show the general effectiveness of our method regardless of the set encoding scheme.
Revised: We also explore an alternative set function such as DeepSets [52] using the ESC50 dataset to show the general effectiveness of our method regardless of the set encoding scheme.

Original: In Table 4, MetaInterpolation with DeepSets improves the generalization performance of ProtoTypical Network and the model with Set Transformer further boost the performance as a consequence of higher-order and pairwise interactions among the set elements via the attention mechanism.
Revised: In Table 4, MetaInterpolation with DeepSets improves the generalization performance of ProtoTypical Network and the model with Set Transformer further boost the performance as a consequence of higher-order and pairwise interactions among the set elements via the attention mechanism.

Original: [Table]
Revised: [Table]

Original: Lastly, we empirically validate our interpolation strategy that mixes only support sets.
Revised: Lastly, we empirically validate our interpolation strategy that mixes only support sets.

Original: We compare our method to various interpolation strategies including one that mixes a support set with a zero mean and unit variance Gaussian noise.
Revised: We compare our method to various interpolation strategies including one that mixes a support set with a zero mean and unit variance Gaussian noise.

Original: In Table 5, we empirically show that the interpolation strategy which mixes only support sets outperforms the other mixing strategies.
Revised: In Table 5, we empirically show that the interpolation strategy which mixes only support sets outperforms the other mixing strategies.

Original: Note that interpolating a support set with gaussian noise works well on ESC50 dataset though we find that it significantly degrades the performance of ProtoNet on
Revised: Note that interpolating a support set with gaussian noise works well on ESC50 dataset though we find that it significantly degrades the performance of ProtoNet on

Original: RMNIST, from 75 .
Revised: RMNIST, from 75 .

Original: 35 ± 1 .
Revised: 35 ± 1 .

Original: 63 to 69 .
Revised: 63 to 69 .

Original: 60 ± 1 .
Revised: 60 ± 1 .

Original: 60 as shown in Table 6, which justifies our approach of mixing two support sets.
Revised: 60 as shown in Table 6, which justifies our approach of mixing two support sets.

Original: Visualization In Figure 3, we present the t-SNE [43] visualizations of the original and interpolated tasks with MLTI and Meta-Interpolation, respectively.
Revised: Visualization In Figure 3, we present the t-SNE [43] visualizations of the original and interpolated tasks with MLTI and Meta-Interpolation, respectively.

Original: Following Yao et al.
Revised: Following Yao et al.

Original: [50], we sample three original tasks from NCI and ESC-50 dataset, where each task is a two-way five-shot and five-way fiveshot classification problem, respectively.
Revised: [50], we sample three original tasks from NCI and ESC-50 dataset, where each task is a two-way five-shot and five-way fiveshot classification problem, respectively.

Original: The tasks are interpolated with MLTI or Meta-Interpolation to construct 300 additional tasks and represented as a set of all class prototypes.
Revised: The tasks are interpolated with MLTI or Meta-Interpolation to construct 300 additional tasks and represented as a set of all class prototypes.

Original: To visualize the prototypes, we first perform Principal Component Analysis [13] (PCA) to reduce the dimension of each prototype.
Revised: To visualize the prototypes, we first perform Principal Component Analysis [13] (PCA) to reduce the dimension of each prototype.

Original: The first 50 principal components are then used to compute the t-SNE visualizations.
Revised: The first 50 principal components are then used to compute the t-SNE visualizations.

Original: As shown in Figure 3b and 3d, Meta-Interpolation successfully learns an expressive neural set function that densifies the task distribution.
Revised: As shown in Figure 3b and 3d, Meta-Interpolation successfully learns an expressive neural set function that densifies the task distribution.

Original: The task augmentations with MLTI, however, do not cover a wide embedding space as shown in Figure 3a and 3c as the mixup strategy allows to generate tasks only on the simplex defined by the given set of tasks.
Revised: The task augmentations with MLTI, however, do not cover a wide embedding space as shown in Figure 3a and 3c as the mixup strategy allows to generate tasks only on the simplex defined by the given set of tasks.

Original: Limitation
Revised: Limitation

Original: Although we have shown promising results in various domains, our method requires extra computation for bilevel optimization to estimate λ , the parameters of the set function φ λ , which makes it challenging to apply our method to gradient based meta-learning methods such as MAML.
Revised: Although we have shown promising results in various domains, our method requires extra computation for bilevel optimization to estimate λ , the parameters of the set function φ λ , which makes it challenging to apply our method to gradient based meta-learning methods such as MAML.

Original: Moreover, our interpolation is limited to classification problem and it is not straightforward to apply it to regression tasks.
Revised: Moreover, our interpolation is limited to classification problem and it is not straightforward to apply it to regression tasks.

Original: Reducing the computational cost for bilevel optimization and extending our framework to regression will be important for future work.
Revised: Reducing the computational cost for bilevel optimization and extending our framework to regression will be important for future work.

Original:  Conclusion
Revised:  Conclusion

Original: We proposed a novel domain-agnostic task augmentation method, Meta Interpolation, to tackle the meta-overfitting problem in few-task meta-learning.
Revised: We proposed a novel domain-agnostic task augmentation method, Meta Interpolation, to tackle the meta-overfitting problem in few-task meta-learning.

Original: Specifically, we leveraged expressive neural set functions to interpolate a given set of tasks and trained the interpolating function using bilevel optimization, so that the meta-learner trained with the augmented tasks generalizes to meta-validation tasks.
Revised: Specifically, we leveraged expressive neural set functions to interpolate a given set of tasks and trained the interpolating function using bilevel optimization, so that the meta-learner trained with the augmented tasks generalizes to meta-validation tasks.

Original: Since the set function is optimized to minimize the loss on the validation tasks, it allows us to tailor the task augmentation strategy to each specific domain.
Revised: Since the set function is optimized to minimize the loss on the validation tasks, it allows us to tailor the task augmentation strategy to each specific domain.

Original: We empirically validated the efficacy of our proposed method on various domains, including image classification, chemical property prediction, textclassification, and automatic speech recognition, showing that Meta-Interpolation achieves consistent improvements across all domains.
Revised: We empirically validated the efficacy of our proposed method on various domains, including image classification, chemical property prediction, text and sound classification, showing that Meta-Interpolation achieves consistent improvements across all domains.

Original: This is in stark contrast to the baselines which improve generalization in certain domains but degenerate performance in others.
Revised: This is in stark contrast to the baselines which improve generalization in certain domains but degenerate performance in others.

Original: Furthermore, our theoretical analysis shed light on how Meta-Interpolation regularizes the meta-learner and improves its generalization performance.
Revised: Furthermore, our theoretical analysis shed light on how Meta-Interpolation regularizes the meta-learner and improves its generalization performance.

Original: Lastly, we discussed the limitation of our method.
Revised: Lastly, we discussed the limitation of our method.

Original: Acknowledgments
Revised: Acknowledgments

Original: This work was supported by Institute of Information & communications Technology Planning &
Revised: This work was supported by Institute of Information & communications Technology Planning &

Original: Evaluation (IITP) grant funded by the Korea government(MSIT)
Revised: Evaluation (IITP) grant funded by the Korea government(MSIT)

Original: (No.2019-0-00075, Artificial Intelligence Graduate School Program(KAIST)), the Engineering Research Center Program through the National Research Foundation of Korea (NRF) funded by the Korean Government MSIT (NRF2018R1A5A1059921), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT)(No. 2021-0-02068, Artificial Intelligence Innovation Hub), the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2021R1F1A1061655), Institute of Information & communications Technology
Revised: (No.2019-0-00075, Artificial Intelligence Graduate School Program(KAIST)), the Engineering Research Center Program through the National Research Foundation of Korea (NRF) funded by the Korean Government MSIT (NRF2018R1A5A1059921), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT)

Original: Planning & Evaluation (IITP) grant funded by the Korea government(MSIT)

Original: (No.2022-0-00713), and Samsung Electronics (IO201214-08145-01). Appendix
Revised: (No. 2021-0-02068, Artificial Intelligence

Original: Organization The appendix is organized as follows:

Original: In Section A, we provide proofs of the theorem and propositions in Section 4.

Original: In Section B, we show additional experimental results —

Original: Meta-interpolation with first order MAML, the effect of the number of meta-training and metavalidation tasks, ablation study for location of interpolation, and the effect of the number of tasks for interpolation.

Original: In Section C, we provide detailed descriptions of the experimental setup used in the main paper together with the exact data splits for meta-training, meta-validation and meta-testing for all the datasets used.

Original: Finally, we specify the exact architecture of the Prototypical Networks used for all the experiments and further describe the Set Transformer in detail in Section C.3.

Original: All hyperparameters are specified in Section C.4.

Original: A Proofs A.1 Proof of Theorem

Original: Proof.

Original: Define ϕ :=

Original: ϕ lθ l = f lθ l ◦ · · · ◦ f 1 θ 1 , and g :=

Original: f Lθ L ◦ · · · ◦ f l +1 θ l +1 .

Original: Define the dimensionality as ϕ ( x st,i ) ∈ R d , and g ( ϕ ( x st,i ))

Original: ∈ R D .

Original: From the definition, since we use ˆ f θ,λ in both training and testing time, we have

Original: [Equation]

Original: where c

Original: [Equation]

Original: In the analysis of the loss functions, without the loss of generality, we can set the permutation σ t on { 1 , . . .

Original: , K } to be the identity since the every combination can be realized by one permutation σ instead of the two permutations σ t , σ t ′ .

Original: Therefore, using the definition of the ˆ T t,t ′ , we can write the corresponding loss by

Original: [Equation]

Original: where

Original: [Equation]

Original: This can be rewritten as:

Original: [Equation]

Original: ′ ,j ,y st ′ ,j ) ∈D st ′ 1 { y st ′ ,i =

Original: σ ( k ) }     (cid:88) ( x st,i ,y st,i ) ∈D

Original: st 1 { y st,i = k } 

Original:  =

Original: N t ′ ,k N t,k

Original: Thus,

Original: [Equation]

Original: 

Original:  .

Original: Define the set

Original: I

Original: [Equation]

Original: k

Original: [Equation]

Original: = { i : y st,i = k } .

Original: Then,

Original: [Equation]

Original: Summarizing the computation so far, we have that

Original: [Equation]

Original: and

Original: L

Original: [Equation]

Original: | I t,k | (cid:88) i ∈

Original: I t,k

Original: ( g ◦ φ λ )( { ϕ ( x st,i ) , ϕ ( x st ′ ,j ) } ) with

Original: I t,k

Original: =

Original: i : y st,i = k } .

Original: We now analyze the relationship of φ λ ( { h, h ′ } ) and φ λ ( { h } ) .

Original: From this definition, we first compute φ λ ( { h, h ′ } ) as follows.

Original: By writing σ 1 = softmax( √ d − 1 Q { h,h ′ }( K { h,h ′ }) ⊤ ) and σ 2 = softmax( √ d

Original: [Equation]

Original: Q 2 ( K { h,h ′ }) ⊤ ) ,

Original: [Equation]

Original: Here, by writing p 1 = p ( t,t ′ ,i,j ), p ′ 1 = ˜ p ( t,t ′ ,i,j ), and p 2 = p ( t,t ′ ,i,j ), we can rewrite that

Original: [Equation]

Original: −

Original: [Equation]

Original: [Equation]

Original: R

Original: [Equation]

Original: Thus, by letting ¯ p = p 2 p 1 + (1 − p 2 ) p ′ 1 ∈ [0 , 1] , σ 2 σ 1 = [ p1 − p 2 ] (cid:20) p

Original: [Equation]

Original: 1 − p

Original: [Equation]

Original: p ′

Original: [Equation]

Original: − p ′

Original: [Equation]

Original: p

Original: [Equation]

Original: p

Original: [Equation]

Original: [Equation]

Original: p

Original: [Equation]

Original: ) p ′

Original: [Equation]

Original: p

Original: [Equation]

Original: p

Original: [Equation]

Original: p

Original: [Equation]

Original: p ′

Original: [Equation]

Original: ] =

Original: [¯ p 1 − ¯ p ] .

Original: Using these,

Original: [Equation]

Original: and

Original: [Equation]

Original: Thus, by letting W =

Original: ( W V 1 W V 2 )

Original: ⊤ ∈ R d × d and b = ( b V 1 W V 2 + b V 2 )

Original: ⊤ ∈ R d , and by defining α = 1 − ¯ p , we have that

Original: [Equation]

Original: For φ λ ( { h } ) , similarly, by writing σ

Original: [Equation]

Original: σ

Original: [Equation]

Original: Here, since σ 1 = softmax( √ d − 1 Q { h } 1 ( K { h }) ⊤ ) ∈ R and σ 2 = softmax

Original: [Equation]

Original: √ d − 1 Q

Original: [Equation]

Original: K { h }) ⊤ ) ∈ R , we have σ 1 = σ 2 = 1 and thus,

Original: [Equation]

Original: λ ( { h

Original: [Equation]

Original: Wh + b.

Original: Therefore, we have that

Original: [Equation]

Original: where W and b does not depend on the ( h, h ′ ) .

Original: Using these and by defining h t,i = ϕ

Original: ( x st,i ) , we have that

Original: [Equation]

Original: With these preparation

Original: [Equation]

Original: we are now ready to prove the regularization form.

Original: Fix t, t ′ and write α ij =

Original: α ( t,t ′ ) ij .

Original: Define a vector α such that α = ( α ij ) i,j .

Original: Then,

Original: [Equation]

Original: Then, from the results of the calculations above

Original: [Equation]

Original: we have that

Original: [Equation]

Original: [-Using the assumptions that ∂ r g ( z ) = 
-]
Revised: [-Using the assumptions that ∂ r g ( z ) = 
-]

Original: for all r ≥ 2 , for any J

Original: [Equation]

Original: the J -th approximation of ˆ c k

Original: ( α ) is given by

Original: [Equation]

Original: Let ¯ γ ≥ 1 to be set later and define

Original: [Equation]

Original: Then,

Original: [Equation]

Original: + ¯ γ ∆ k .

Original: Define c

Original: =

Original: ( c 1 , . .

Original: . , c K ) and ∆ =

Original: (∆ 1 , . . .

Original: , ∆ K ) .

Original: For any given t , define

Original: [Equation]

Original: Then, we have that

Original: [Equation]

Original: The J -th approximation of L mix ( λ, θ, ˆ T t,t ′ ) is given by

Original: [Equation]

Original: where φ ( j ) is the j -th order derivative of φ : γ (

Original: cid:55)→ L t ( c + γ ∆) .

Original: Here, for any j ∈ N + , by defining

Original: [Equation]

Original: KD

Original: [Equation]

Original: Then

Original: [Equation]

Original: by using the vectorization of the tensor vec[ ∂ j L t ( b ′ )]

Original: ∈ R ( KD )

Original: j

Original: [Equation]

Original: we can rewrite this equation as

Original: [Equation]

Original: where ∆ ⊗ j = ∆ ⊗ ∆ ⊗ · · · ⊗ ∆ ∈ R ( KD ) j . By combining these with ¯ γ = [Equation] where ∆
Revised: Innovation Hub), the National Research Foundation of Korea (NRF) funded by the Ministry of

Original: [Equation]

Original: =

Original: [Equation]

Original: A.2 Proof of Proposition

Original: Proof.

Original: We now apply our general regularization form theorem to this special case from the previous paper [50] on the ProtoNet loss.

Original: In this special case, we have that ϕ ( x ) = x , W

Original: =

Original: I, b = 0 , h t,i = x st,i , g ( x ) =

Original: x ⊤ θ, and

Original: L t ( c ) = 1 n n (cid:88) i =  1 + exp( ⟨ x qt,i , θ ⟩ − c 1 / 2 − c 2 / 2) .

Original: Here, for c = ( c 1 , c 2 ) with c 1 =

Original: N t,(cid:88)

Original: ( x st,i ,y st,i ) ∈D

Original: st 1 { y st

Original: [Equation]

Original: i =1 } φ λ ( { x st

Original: [Equation]

Original: i } ) ⊤ θ = θ ⊤ c ′(10) and c 2 =

Original: N t,(cid:88)

Original: ( x st,i ,y st,i ) ∈D

Original: st 1 { y st

Original: [Equation]

Original: i =2 }

Original: [Equation]

Original: ( { x

Original: [Equation]

Original: } ) ⊤ θ = θ ⊤ c ′ 2 , (11) we recover that

Original: L

Original: [Equation]

Original: Thus, to instantiate our general theorem to this special case, we only need to compute the ∂ j L t ( c ) and ∂g ( h ⊤ t,i W + b ) up to the second order approximation.

Original: [Equation]

Original: Define z t,i := ⟨ x qt,i − ( c ′ 1 + c ′ 2 ) / 2 , θ ⟩ .

Original: Since c 1 = θ ⊤ c ′ 1 and c 2 = θ ⊤ c ′ 2 by Equation 10 and 11,

Original: [Equation]

Original: Thus, we have L t ( c )

Original: = 1 n (cid:80) ni = 1 1+exp( z t,i ) .

Original: Then, ∂L

Original: [Equation]

Original: ( c )

Original: [Equation]

Original: Similarly, ∂L

Original: [Equation]

Original: For the second order, by defining the logistic function ψ ( z t,

Original: [Equation]

Original: Similarly,

Original: [Equation]

Original: Therefore

Original: [Equation]

Original: the second approximation of E t ′ ,σ [ L mix ( λ, θ, ˆ T t,t ′ )] is

Original: L singleton ( λ, θ ; T t ) + E t ′ ,σ   2 (cid:88)

Original: j =

Original: [Equation]

Original: where ∆ =

Original: [∆ ⊤

Original: [Equation]

Original: ∆ ⊤ 2 ] ⊤ with

Original: [Equation]

Original: where

Original: [Equation]

Original: For the first order term,

Original: [Equation]

Original: where δ t,t

Original: [Equation]

Original: [Equation]

Original: =

Original: [Equation]

Original: =

Original: [Equation]

Original: [Equation]

Original: For the second order term,

Original: [Equation]

Original: Since E t ′ ,σ [ δ t,t ′ ,σ ] = 0 with the α being balanced

Original: [Equation]

Original: the second approximation of E t ′ ,σ [ L mix ( λ, θ, ˆ T t,t ′ )] becomes:

Original: [Equation]

Original: A.3 Proof of Proposition

Original: Proof.

Original: Let ξ 1 , . . .

Original: , ξ n be independent uniform random variables taking values in {− 1 , 1 } ; i.e.,

Original: Rademacher variables.

Original: We first bound the empirical Rademacher complexity part as follows:

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: where Σ † denotes the Moore–Penrose inverse of Σ .

Original: By taking expectation and using this bound on the empirical Rademacher complexity, we now bound the Rademacher complexity as follows:

Original: [Equation]

Original: x 1 ,..., x n ˆ R n ( F R ) ≤ E x 1 ,..., x n √ R n (cid:118)(cid:117)(cid:117)(cid:116) n (cid:88) i =

Original: E x ′ ( x i − x ′ )

Original: ⊤ Σ † ( x i − x ′ ) ≤ √ R n (cid:118)(cid:117)(cid:117)(cid:116) n (cid:88) i =

Original: E x i , x ′ ( x i − x ′ )

Original: ⊤ Σ † ( x i − x ′ ) =

Original: √ R n (cid:118)(cid:117)(cid:117)(cid:116) n (cid:88) i = (cid:88) k,l (Σ † )

Original: kl E x i , x ′ ( x i − x ′ )

Original: k ( x i − x ′ )

Original: l = √ R n (cid:118)(cid:117)(cid:117)(cid:116) n (cid:88) i = (cid:88) k,l (Σ † ) kl (Σ) kl = √ R n (cid:118)(cid:117)(cid:117)(cid:116) n (cid:88)

Original: i = tr(ΣΣ † ) = √ R n (cid:118)(cid:117)(cid:117)(cid:116) n (cid:88) i =

Original: rank(Σ) =

Original: [Equation]

Original: (cid:112) rank

Original: [Equation]

Original: Σ) √ n

Original: B Additional Experiments B.1 First-order MAML

Original: As stated in the introduction, we focus solely on metric based meta-learning due to their efficiency and better empirical performance over the gradient based methods on the few-task meta-learning problem.

Original: Moreover it is challenging to combine our method with Model-Agnostic Meta-Learning (MAML) [10] since it yields a tri-level optimization problem which requires differentiating through second order derivatives.

Original: Furthermore, tri-level optimization is still known to be a challenging problem [4, 7] and currently an active line of research.

Original: Thus, instead of using the original MAML, we perform additional experiments on the ESC-50 and Metabolism dataset with first-order MAML (FOMAML) [Table] [Table] [Table]
Revised: Education (NRF-2021R1F1A1061655), Institute of Information & communications Technology

Original: which approximates the Hessian with a zero matrix.

Original: As shown in Table

Original: [Table]

Original: the experimental results show that Meta-Interpolation with first-order MAML outperforms MLTI, which again confirms the general effectiveness of our set-based task augmentation scheme.

Original: However, it largely underperforms our original Meta-Interpolation framework with metric-based meta-learning.

Original: B.2 Effect of the number of meta-training and validation tasks

Original: We analyze the effect of the number of meta-training tasks on ESC-50 dataset.

Original: As shown in Table 8,

Original: Meta-Interpolation consistently outperforms the baselines by large margins, regardless of the number of the tasks.

Original: Furthermore, we report the test accuracy as a function of the number of meta-validation tasks in Table 9.

Original: Although the generalization performance of Meta-Interpolation slightly decreases as we reduce the number of meta-validation tasks, it still outperforms the relevant baselines by a large margin.

Original: B.3 Location of interpolation

Original: [Table]

Original: Contrary to Manifold Mixup, we fix the layer of interpolation as shown in Table 17.

Original: Otherwise, we cannot use the same architecture of Set Transformer to interpolate the output of different layers since the hidden dimension of each layer is different.

Original: Moreover, we report the test accuracy by changing the layer of interpolation.

Original: Interpolating hidden representation of support sets from layer 2, which is the model used in the main experiments on ESC dataset, achieves the best performance.

Original: B.4 Effect of Cardinality for Interpolation

Original: [Figure]

Original: Since the set function φ λ can handle sets of arbitrary cardinality n ∈ N , we plot the test accuracy on the ESC-50 dataset with varying set sizes from one to five.

Original: As shown in Figure 4, for sets of cardinality one, where we do not perform any interpolation, we observe significant degradation in performance.

Original: This suggests that the interpolation of instances is crucial for better generalization.

Original: On the other hand, for set sizes from two to five, the gain is marginal with increasing cardinality.

Original: Furthermore, increasing the set size introduces extra computational overhead.

Original: Thus, we set the cardinality to two for task interpolation in all the experiments.

Original: C Experimental Setup C.1 Dataset Description

Original: Metabolism We use the following split for the metabolism dataset:

Original: Meta-Train = { CYP1A2_Veith , CYP3A4_Veith , CYP2C9_Substrate_CarbonMangels } Meta-Validation = { CYP2D6_Veith , CYP2D6_Substrate_CarbonMangels } Meta-Test

Original: =

Original: { CYP2C19_Veith

Original: CYP2C

Original: Veith

Original: CYP3A

Original: Substrate

Original: _CarbonMangels

Original: Following Yao et al.

Original: [50], we balance each subdataset by selecting 1000 samples from each subdataset.

Original: Each data sample is processed by extracting a 1024-bit fingerprint feature from the SMILES representation of each chemical compound using the RDKit [15] library.

Original: [-Tox
-]
Revised: [-Tox
-]

Original: We use the following split for the metabolism dataset 2 :

Original: Meta-Train = { NR-AR , NR-AR-LBD , NR-AhR , NR-Aromatase , NR-ER , NR-ER-LBD } Meta-Validation = { NR-PPAR-gamma , SR-ARE } Meta-Test

Original: =

Original: { SR-ATAD5 , SR-HSE , SR-MMP , SR-p53 }

Original: NCI We download the dataset from the github repository 3 and use the following splits for metatraining, meta-validation, and meta-testing:

Original: Meta-Train = { 41 , 47 , 83 , 109 } Meta-Validation = { 81 , 145 } Meta-Test = { 1 , 33 , 123 }

Original: GLUE-SciTail We use Hugging Face Datasets library [25] to download MNLI, QNLI, SNLI, RTE, and SciTail datasets and tokenize it ELECTRA tokenizer with setting maximum length to 128.

Original: We list meta-train, meta-validation, and meta-test split as follows:

Original: Meta-Train = { MNLI , QNLI } Meta-Validation

Original: =

Original: SNLI , RTE } Meta-Test = { SciTail } ESC 50 We download ESC 50 dataset [34] from the github repository 4 and use the meta-training, meta-validation, and meta-test split as follows:
Revised: Planning & Evaluation (IITP) grant funded by the Korea government(MSIT)

Original: Meta-train set:

Original: dog , rooster , pig , cow , frog , cat , hen , insects , sheep , crow , rain , sea_waves , crackling_fire , crickets , chirping_birds , water_drops , wind , pouring_water , toilet_flush , thunderstorm

Original: Meta-validation set:

Original: crying_baby , sneezing , clapping , breathing , coughing , footsteps , laughing , brushing_teeth , snoring , drinking_sipping , door_wood_knock , mouse_click , keyboard_typing , door_wood_creaks , can_opening

Original: Meta-test set:

Original: washing_machine , vacuum_cleaner , clock_alarm , clock_tick , glass_breaking , helicopter , chainsaw , siren , car_horn , engine , train , church_bells , airplane , fireworks , hand_saw

Original: RMNIST Following Yao et al.

Original: [50], we create RMNIST dataset by changing the size, color, and angle of the images from the original MNIST dataset.

Original: To be specific, we merge training and test split of MNIST and randomly select 5,600 images for each class and split into 56 subdatasets where each class has 100 examples.

Original: We only choose 16/6/10 subdatasets for meta-train, meta-validation, and meta-test split, respectively and do not use the rest of them.

Original: Each subdatset with the corresponding composition of image transformations is considered a distinct task.

Original: Following are the specific splits.

Original: Meta-train set: ( red , full , 90 ◦ ) , ( indigo , full , 0 ◦ ) , ( blue , full , 270 ◦ ) , ( orange , half , 270 ◦ ) , ( green , full , 90 ◦ ) , ( green , full , 270 ◦ ) , ( orange , full , 180 ◦ ) , ( red , full , 180 ◦ ) , ( green , full , 0 ◦ ) , ( orange , full , 0 ◦ ) , ( violet , full , 270 ◦ ) , ( orange , half , 90 ◦ ) , ( violet , half , 180 ◦ ) , ( orange , full , 90 ◦ ) , ( violet , full , 180 ◦ ) , ( blue , full , 90 ◦ )

Original: Meta-

Original: validation

Original: set

Original: : ( indigo , half , 270 ◦ ) , ( blue , full , 0 ◦ ) , ( yellow , half , 180 ◦ ) , ( yellow , half , 0 ◦ ) , ( yellow , half , 90 ◦ ) , ( violet , half , 0 ◦ )

Original: Meta-test set

Original: : ( yellow ,

Original: full

Original: 270 ◦ )

Original: ( red

Original: full

Original: 0 ◦ ) blue half , 270 ◦ ) , (
Revised: (No.2022-0-00713),

Original: blue

Original: , half , 0 ◦ ) , (

Original: blue

Original: , half , 180 ◦ ) , (

Original: red

Original: half

Original: ◦ )

Original: violet

Original: full

Original: , 90 ◦ ) , (

Original: blue

Original: , half , 90 ◦ ) , ( green , half , 270 ◦ ) , ( red , half , 90 ◦ )

Original: Mini-ImageNet-S

Original: Following Yao et al.

Original: [50], we reduce the number of meta-training tasks by choosing subset of the original meta-training classes.

Original: We specify the classes used for meta-training tasks as follows:

Original: n03017168 , n07697537 , n02108915 , n02113712 , n02120079 , n04509417 , n02089867 , n03888605 , n04258138 , n03347037 , n02606052 , n

Original: For meta-validation and meta-testing classes, we use the same classes as the original Mini-ImageNet.

Original: CIFAR-100-FS Similar to Mini-ImageNet-S, we choose 12/16/20 classes for meta-training, metavalidation, meta-test classes.

Original: Followings are the specific classes for each split:

Original: Meta-Train = { 0 , . .

Original: , 11 } Meta-Validation = { 64 , . .

Original: , 79 } Meta-Test = { 80 , . . .

Original: C.2 Prototypical Network Architecture

Original: We summarize neural network architectures for ProtoNet on each datasets in Table 11, 12, 13, 14, 15, and 16. For Glue-SciTail, we use pretrained ELECTRA-small [8] which we download from Hugging Face [48]. [Table]
Revised: Samsung Electronics (IO201214-08145-01), and Google AI Focused Research Award.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: C.

Original: [Table]

Original: Transformer

Original: We describe Set Transformer [23], φ λ , in more detail.

Original: Let X ∈ R n × d be a stack of n ddimensional row vectors.

Original: Let W Q 1 ,j , W K 1 ,j , W V 1 ,j ∈ R d × d k be weight matrices for self-attention and let b Q 1 ,j , b K 1 ,j , b V 1 ,

Original: j ∈ R d

Original: k be bias vectors for j = 1 , . .

Original: For encoding an input X , we compute self-attention as follows:

Original: [Equation]

Original: where 1 = (1 , . . . , 1) ⊤ ∈ R n is a vector of ones, d h = 4 d k , and softmax is applied for each row.

Original: After self-attention

Original: [Equation]

Original: we add a skip connection with layer normalization [1] as follows:

Original: [Equation]

Original: where W 1 ∈ R d h × d h , b 1 ∈ R d h . Similarly [Equation] we compute another self-attention on top of g 1 ( X ) with
Revised: It was also results of a study on the “HPC Support” Project, supported by the ‘Ministry of Science and ICT’ and

Original: W Q 2 ,j , W K 2 ,j , W V 2 ,j ∈ R d k × d k weight matrices for self-attention and let b Q 2 ,j , b K 2 ,j , b V 2 ,

Original: j ∈ R d k be bias vectors, for j = 1 . .

Original: [Equation]

Original: After self-attention, we also add a skip connection after the second self-attention with dropout [41].

Original: [Equation]

Original: where W 2 ∈ R d h × d h , b 2 ∈ R d h .

Original: After encoding X with two-layers of self-attention, we perform pooling with attention.

Original: Let S ∈ R 1 × d h be learnable parameters and W Q 3 ,j , W K 3 ,j , W V 3 ,j ∈ R d

Original: h × d k be weight matrices for the pooling-attention and let b Q 3 ,j , b K 3 ,j , b V 3 ,

Original: j ∈ R d k be bias vectors, for j = 1 , . .

Original: We pool ( g 2 ◦ g 1 )( X ) as follows:

Original: [Equation]

Original: After pooling, we add another skip connection with dropout as follows:

Original: [Equation]

Original: where W 3 ∈ R d h × d h , b 3 ∈ R d h .

Original: Finally, we perform affine-transformation after the pooling as follows:

Original: [Equation]

Original: where W 4 ∈ R d h × d , b 4 ∈ R d .

Original: To summarize, Set Transformer is the set function φ λ :=

Original: g 4 ◦ g 3 ◦ g 2 ◦ g 1 : R n × d → R d that can handle a set with arbitrary cardinality n ∈ N . C.4 Hyperparameters In Table 17 and [Table]
Revised: NIPA.

Original: we summarize all the hyperparameters for each datasets, where MI stands for

Original: Mini-ImageNet-S. For CIFAR-100-FS, we use the same hyperparameters for both 1-hot and 5-shot.

Original: [Table]

Original: [Table]