Original: Set-based Meta-Interpolation for Few-Task Meta-Learning
Revised: Set-based Meta-Interpolation for Few-Task Meta-Learning

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Meta-learning approaches enable machine learning systems to adapt to new tasksgiven few examples by leveraging knowledge from related tasks.
Revised: Meta-learning approaches enable machine learning systems to adapt to new tasks given few examples by leveraging knowledge from related tasks.

Original: However, a largenumber of meta-training tasks are still required for generalization to unseen tasksduring meta-testing, which introduces a critical bottleneck for real-world problemsthat come with only few tasks, due to various reasons including the difficultyand cost of constructing tasks.
Revised: However, a large number of meta-training tasks are still required for generalization to unseen tasks during meta-testing, which introduces a critical bottleneck for real-world problems that come with only few tasks, due to various reasons including the difficulty and cost of constructing tasks.

Original: Recently, several task augmentation methods have been proposed to tackle this issue using domain-specific knowledge to designaugmentation techniques to densify the meta-training task distribution.
Revised: Recently, several task augmentation methods have been proposed to tackle this issue using domain-specific knowledge to design augmentation techniques to densify the meta-training task distribution.

Original: However,such reliance on domain-specific knowledge renders these methods inapplicableto other domains.
Revised: However, such reliance on domain-specific knowledge renders these methods inapplicable to other domains.

Original: While Manifold Mixup based task augmentation methods aredomain-agnostic, we empirically find them ineffective on non-image domains.
Revised: While Manifold Mixup based task augmentation methods are domain-agnostic, we empirically find them ineffective on non-image domains.

Original: Totackle these limitations, we propose a novel domain-agnostic task augmentationmethod, Meta-Interpolation, which utilizes expressive neural set functions to densify the meta-training task distribution using bilevel optimization.
Revised: To tackle these limitations, we propose a novel domain-agnostic task augmentation method, Meta-Interpolation, which utilizes expressive neural set functions to densify the meta-training task distribution using bilevel optimization.

Original: We empiricallyvalidate the efficacy of Meta-Interpolation on eight datasets spanning across variousdomains such as image classification, molecule property prediction, text classification and speech recognition.
Revised: We empirically validate the efficacy of Meta-Interpolation on eight datasets spanning across various domains such as image classification, molecule property prediction, text classification and speech recognition.

Original: Experimentally, we show that Meta-Interpolationconsistently outperforms all the relevant baselines.
Revised: Experimentally, we show that Meta-Interpolation consistently outperforms all the relevant baselines.

Original: Theoretically, we prove thattask interpolation with the set function regularizes the meta-learner to improvegeneralization.
Revised: Theoretically, we prove that task interpolation with the set function regularizes the meta-learner to improve generalization.

Original:  Introduction
Revised:  Introduction

Original: The ability to learn a new task given only a few examples is crucial for artificial intelligence.
Revised: The ability to learn a new task given only a few examples is crucial for artificial intelligence.

Original: Recently,meta-learning [39, 3] has emerged as a viable method to achieve this objective and enables machinelearning systems to quickly adapt to a new task by leveraging knowledge from other related tasks seenduring meta-training.
Revised: Recently, meta-learning [39, 3] has emerged as a viable method to achieve this objective and enables machine learning systems to quickly adapt to a new task by leveraging knowledge from other related tasks seen during meta-training.

Original: Although existing meta-learning methods can efficiently adapt to new taskswith few data samples, a large dataset of meta-training tasks is still required to learn meta-knowledgethat can be transferred to unseen tasks.
Revised: Although existing meta-learning methods can efficiently adapt to new tasks with few data samples, a large dataset of meta-training tasks is still required to learn meta-knowledge that can be transferred to unseen tasks.

Original: For many real-world applications, such extensive collectionsof meta-training tasks may be unavailable.
Revised: For many real-world applications, such extensive collections of meta-training tasks may be unavailable.

Original: Such scenarios give rise to the few-task meta-learningproblem where a meta-learner can easily memorize the meta-training tasks but fail to generalizewell to unseen tasks.
Revised: Such scenarios give rise to the few-task meta-learning problem where a meta-learner can easily memorize the meta-training tasks but fail to generalize well to unseen tasks.

Original: The few-task meta-learning problem usually results from the difficulty in taskgeneration and data collection.
Revised: The few-task meta-learning problem usually results from the difficulty in task generation and data collection.

Original: For instance, in the medical domain, it is infeasible to collect a largeamount of data to construct extensive meta-training tasks due to privacy concerns.
Revised: For instance, in the medical domain, it is infeasible to collect a large amount of data to construct extensive meta-training tasks due to privacy concerns.

Original: Moreover, fornatural language processing, it is not straightforward to split a dataset into tasks, and hence entiredatasets are treated as tasks [30].
Revised: Moreover, for natural language processing, it is not straightforward to split a dataset into tasks, and hence entire datasets are treated as tasks [30].

Original: Several works have been proposed to tackle the few-task meta-learning problem using task augmentation techniques such as clustering a dataset into multiple tasks [30], leveraging strong image

Original: [Figure]
Revised: [Figure]

Original:  augmentation methods such as vertical flipping to construct new classes [32], and the employmentof Manifold Mixup [44] for densifying the meta-training task distribution [49, 50].
Revised: Several works have been proposed to tackle the few-task meta-learning problem using task augmentation techniques such as clustering a dataset into multiple tasks [30], leveraging strong image augmentation methods such as vertical flipping to construct new classes [32], and the employment of Manifold Mixup [44] for densifying the meta-training task distribution [49, 50].

Original: However, majority of these techniques require domain-specific knowledge to design such task augmentationsand hence cannot be applied to other domains.
Revised: However, majority of these techniques require domain-specific knowledge to design such task augmentations and hence cannot be applied to other domains.

Original: While Manifold Mixup based methods [49, 50] aredomain-agnostic, we empirically find them ineffective for mitigating meta-overfitting in few-taskmeta-learning especially in non-image domains such as chemical and text, and that they sometimesdegrade generalization performance.
Revised: While Manifold Mixup based methods [49, 50] are domain-agnostic, we empirically find them ineffective for mitigating meta-overfitting in few-task meta-learning especially in non-image domains such as chemical and text, and that they sometimes degrade generalization performance.

Original: In this work, we focus solely on domain-agnostic task augmentation methods that can densifythe meta-training task distribution to prevent meta-overfitting and improve generalization at metatesting for few-task meta-learning.
Revised: In this work, we focus solely on domain-agnostic task augmentation methods that can densify the meta-training task distribution to prevent meta-overfitting and improve generalization at metatesting for few-task meta-learning.

Original: To tackle the limitations already discussed, we propose a noveldomain-agnostic task augmentation method for metric based meta-learning models.
Revised: To tackle the limitations already discussed, we propose a novel domain-agnostic task augmentation method for metric based meta-learning models.

Original: Our method,
Revised: Our method,

Original: Meta-Interpolation, utilizes expressive neural set functions to interpolate two tasks and the setfunctions are trained with bilevel optimization so that a meta-learner trained on the interpolated tasksgeneralizes to tasks in the meta-validation set.
Revised: Meta-Interpolation, utilizes expressive neural set functions to interpolate two tasks and the set functions are trained with bilevel optimization so that a meta-learner trained on the interpolated tasks generalizes to tasks in the meta-validation set.

Original: As a consequence of end-to-end training, the learnedaugmentation strategy is tailored to each specific domain without the need for specialized domainknowledge.
Revised: As a consequence of end-to-end training, the learned augmentation strategy is tailored to each specific domain without the need for specialized domain knowledge.

Original: For example, for K -way classification, we sample two tasks consisting of support and query setsand assign a new class k to each pair of classes { σ ( k ) , σ ′ ( k ) } for k = 1 , . . .
Revised: For example, for K -way classification, we sample two tasks consisting of support and query sets and assign a new class k to each pair of classes { σ ( k ) , σ ′ ( k ) } for k = 1 , . . .

Original: , K , where σ, σ ′ arepermutations on { 1 , . . .
Revised: , K , where σ, σ ′ are permutations on { 1 , . . .

Original: , K } as depicted in Figure 1a.
Revised: , K } as depicted in Figure 1a.

Original: Hidden representations of the support set withclasses σ ( k ) and σ ′ ( k ) are then transformed into a single support set using a set function that mapsa set of two vectors to a single vector.
Revised: Hidden representations of the support set with classes σ ( k ) and σ ′ ( k ) are then transformed into a single support set using a set function that maps a set of two vectors to a single vector.

Original: We refer to the output of the set function as the interpolatedsupport set and these are used to compute class prototypes.
Revised: We refer to the output of the set function as the interpolated support set and these are used to compute class prototypes.

Original: As shown in Figure 1b, the interpolatedsupport set is paired with a query set (query set 1 in Figure 1a)), randomly selected from the twotasks to obtain a new task.
Revised: As shown in Figure 1b, the interpolated support set is paired with a query set (query set 1 in Figure 1a)), randomly selected from the two tasks to obtain a new task.

Original: Lastly, we optimize the set function so that a meta-learner trained on theaugmented task can minimize the loss on the meta-validation tasks as illustrated in Figure 1c.
Revised: Lastly, we optimize the set function so that a meta-learner trained on the augmented task can minimize the loss on the meta-validation tasks as illustrated in Figure 1c.

Original: To verify the efficacy of our method, we empirically show that it significantly improves the performance of Prototypical Networks [40] on the few-task meta-learning problem across multiple domains.
Revised: To verify the efficacy of our method, we empirically show that it significantly improves the performance of Prototypical Networks [40] on the few-task meta-learning problem across multiple domains.

Original: Our method outperforms the relevant baselines on eight few-task meta-learning benchmark datasetsspanning image classification, chemical property prediction, text classification, and automatic speechrecognition.
Revised: Our method outperforms the relevant baselines on eight few-task meta-learning benchmark datasets spanning image classification, chemical property prediction, text classification, and automatic speech recognition.

Original: Furthermore, our theoretical analysis shows that our task interpolation method with theset function regularizes the meta-learner and improves generalization performance.
Revised: Furthermore, our theoretical analysis shows that our task interpolation method with the set function regularizes the meta-learner and improves generalization performance.

Original: Our contribution is threefold:
Revised: Our contribution is threefold:

Original: • We propose a novel domain-agnostic task augmentation method, Meta-Interpolation, which leverages expressive set functions to densify the meta-training task distribution for thefew-task meta-learning problem.
Revised: • We propose a novel domain-agnostic task augmentation method, Meta-Interpolation, which leverages expressive set functions to densify the meta-training task distribution for the few-task meta-learning problem.

Original: • We theoretically analyze our model and show that it regularizes the meta-learner for bettergeneralization.• Through extensive experiments, we show that Meta-Interpolation significantly improves the performance of Prototypical Network on various domains such as image, text, and chemicalmolecule classification, and speech recognition on the few-task meta-learning problem.
Revised: • We theoretically analyze our model and show that it regularizes the meta-learner for better generalization. • Through extensive experiments, we show that Meta-Interpolation significantly improves the performance of Prototypical Network on various domains such as image, text, and chemical molecule classification, and speech recognition on the few-task meta-learning problem.

Original:  Related Work
Revised:  Related Work

Original: Meta-Learning The two mainstream approaches to meta-learning are gradient based [10, 33, 14,24, 12, 36, 37] and metric based meta-learning [45, 40, 42, 29, 26, 6, 38].
Revised: Meta-Learning The two mainstream approaches to meta-learning are gradient based [10, 33, 14, 24, 12, 36, 37] and metric based meta-learning [45, 40, 42, 29, 26, 6, 38].

Original: The former formulatesmeta-knowledge as meta-parameters such as the initial model parameters and performs bileveloptimization to estimate the meta-parameters so that a meta-learner can generalize to unseen taskswith few gradient steps.
Revised: The former formulates meta-knowledge as meta-parameters such as the initial model parameters and performs bilevel optimization to estimate the meta-parameters so that a meta-learner can generalize to unseen tasks with few gradient steps.

Original: The latter learns an embedding space where classification is performed bymeasuring the distance between a query and a set of class prototypes.
Revised: The latter learns an embedding space where classification is performed by measuring the distance between a query and a set of class prototypes.

Original: In this work, we focus onmetric based meta-learning with fewer number of meta-training tasks, i.e., few-task meta-learning.
Revised: In this work, we focus on metric based meta-learning with fewer number of meta-training tasks, i.e., few-task meta-learning.

Original: We propose a novel task augmentation method that densifies the meta-training task distribution andmitigates overfitting due to the fewer number of meta-training tasks for better generalization to unseentasks.
Revised: We propose a novel task augmentation method that densifies the meta-training task distribution and mitigates overfitting due to the fewer number of meta-training tasks for better generalization to unseen tasks.

Original: Task Augmentation for Few-Task Meta-learning Several methods have been proposed to augmentthe number of meta-training tasks to mitigate overfitting in the context of few-task meta-learning.
Revised: Task Augmentation for Few-Task Meta-learning Several methods have been proposed to augment the number of meta-training tasks to mitigate overfitting in the context of few-task meta-learning.

Original: Ni et al.
Revised: Ni et al.

Original: [32] apply strong data augmentations such as vertical flip to images to create a new class.
Revised: [32] apply strong data augmentations such as vertical flip to images to create a new class.

Original: For text classification, Murty et al.
Revised: For text classification, Murty et al.

Original: [30] split meta-training tasks into latent reasoning categories byclustering data with a pretrained language model.
Revised: [30] split meta-training tasks into latent reasoning categories by clustering data with a pretrained language model.

Original: However, they require domain-specific knowledgeto design such augmentations, and hence the resulting augmentation techniques are inapplicable toother domains where there is no well-defined data augmentation or pretrained model.
Revised: However, they require domain-specific knowledge to design such augmentations, and hence the resulting augmentation techniques are inapplicable to other domains where there is no well-defined data augmentation or pretrained model.

Original: In order to tacklethis limitation, Manifold Mixup-based task augmentations have also been proposed.
Revised: In order to tackle this limitation, Manifold Mixup-based task augmentations have also been proposed.

Original: MetaMix [49]interpolates support and query sets with Manifold Mixup [44] to construct a new query set.
Revised: MetaMix [49] interpolates support and query sets with Manifold Mixup [44] to construct a new query set.

Original: MLTI [50]performs Manifold Mixup [44] on support and query sets from two tasks for task augmentation.
Revised: MLTI [50] performs Manifold Mixup [44] on support and query sets from two tasks for task augmentation.

Original: Although these methods are domain-agnostic, we empirically find that they are not effective in somedomains and can degrade generalization performance.
Revised: Although these methods are domain-agnostic, we empirically find that they are not effective in some domains and can degrade generalization performance.

Original: In contrast, we propose to train an expressiveneural set function to interpolate two tasks with bilevel optimization to find optimal augmentationstrategies tailored specifically to each domain.
Revised: In contrast, we propose to train an expressive neural set function to interpolate two tasks with bilevel optimization to find optimal augmentation strategies tailored specifically to each domain.

Original:  Method
Revised:  Method

Original: Preliminaries In meta-learning, we are given a finite set of tasks {T t } Tt =1 , which are i.i.d samplesfrom an unknown task distribution p ( T ) .
Revised: Preliminaries In meta-learning, we are given a finite set of tasks {T t } Tt =1 , which are i.i.d samples from an unknown task distribution p ( T ) .

Original: Each task T t consists of a support set D st = { ( x st,i , y st,i ) } N s i
Revised: Each task T t consists of a support set D st = { ( x st,i , y st,i ) } N s i

Original: =1and a query set D qt = { ( x qt,i , y qt,i ) } N q i =1 , where x t,i and y t,i denote a data point and its correspondinglabel respectively.
Revised: = and a query set D qt = { ( x qt,i , y qt,i ) } N q i =1 , where x t,i and y t,i denote a data point and its corresponding label respectively.

Original: Given a predictive model ˆ f θ,λ :=
Revised: Given a predictive model ˆ f θ,λ :=

Original: f Lθ L ◦ · · · ◦ f l +1 θ l +1 ◦ φ λ ◦ f lθ l ◦ · · · ◦ f 1 θ 1 with Llayers, we want to estimate the parameter θ that minimizes the meta-training loss and generalizes toquery sets D q ∗ sampled from an unseen task T ∗ using the support set D s ∗ , where λ is a hyperparameterfor the function φ λ .
Revised: f Lθ L ◦ · · · ◦ f l +1 θ l +1 ◦ φ λ ◦ f lθ l ◦ · · · ◦ f 1 θ 1 with L layers, we want to estimate the parameter θ that minimizes the meta-training loss and generalizes to query sets D q ∗ sampled from an unseen task T ∗ using the support set D s ∗ , where λ is a hyperparameter for the function φ λ .

Original: In this work, we primarily focus on metric based meta-learning methods ratherthan gradient based meta-learning methods due to efficiency and empirically higher performance overthe gradient based methods on the tasks we consider.
Revised: In this work, we primarily focus on metric based meta-learning methods rather than gradient based meta-learning methods due to efficiency and empirically higher performance over the gradient based methods on the tasks we consider.

Original: Problem Statement
Revised: Problem Statement

Original: In this work, we focus solely on few-task meta-learning.
Revised: In this work, we focus solely on few-task meta-learning.

Original: Here, the number ofmeta-training tasks drawn from the meta-training distribution is extremely small and the goal of ameta-learner is to learn meta-knowledge from such limited tasks that can be transferred to unseentasks during meta-testing.
Revised: Here, the number of meta-training tasks drawn from the meta-training distribution is extremely small and the goal of a meta-learner is to learn meta-knowledge from such limited tasks that can be transferred to unseen tasks during meta-testing.

Original: The key challenges here are preventing the meta-learner from overfittingon the meta-training tasks and generalizing to unseen tasks drawn from a meta-test set.
Revised: The key challenges here are preventing the meta-learner from overfitting on the meta-training tasks and generalizing to unseen tasks drawn from a meta-test set.

Original: Metric Based Meta-Learning The goal of metric based meta-learning is to learn an embeddingspace induced by ˆ f θ,λ , where we perform classification by computing distances between data pointsand class prototypes.
Revised: Metric Based Meta-Learning The goal of metric based meta-learning is to learn an embedding space induced by ˆ f θ,λ , where we perform classification by computing distances between data points and class prototypes.

Original: We adopt Prototypical Network (ProtoNet) [40] for ˆ f θ,λ , where φ λ is theidentity function.
Revised: We adopt Prototypical Network (ProtoNet) [40] for ˆ f θ,λ , where φ λ is the identity function.

Original: Specifically, for each task T t with its corresponding support D st and query D qt sets,we compute class prototypes { c k } Kk =1 as the average of the hidden representation of the supportsamples belonging to the class k as follows:
Revised: Specifically, for each task T t with its corresponding support D st and query D qt sets, we compute class prototypes { c k } Kk =1 as the average of the hidden representation of the support samples belonging to the class k as follows:

Original: [Equation]
Revised: [Equation]

Original: where N k denotes the number of instances belonging to the class k .
Revised: where N k denotes the number of instances belonging to the class k .

Original: Given a metric d ( · , · ) : R D ×
Revised: Given a metric d ( · , · ) : R D ×

Original: R D (cid:55)→ R , we compute the probability of a query point x qt,i being assigned to the class k by measuringthe distance between the hidden representation ˆ f θ,λ ( x qt,i ) and the class prototype c k followed by
Revised: R D (cid:55)→ R , we compute the probability of a query point x qt,i being assigned to the class k by measuring the distance between the hidden representation ˆ f θ,λ ( x qt,i ) and the class prototype c k followed by

Original: [Table]
Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: dation
Revised: dation

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 9: v 1 − =
Revised: 9: v 1 − =

Original: α · grad ( ∂ L tr ∂θ , θ, grad_outputs = v 1 ) 10: p += v 1 11: end for 12: v 2 ← grad ( L tr ∂θ , λ, grad_outputs = α p ) 13: return ∂ L V ∂λ − v 2 14:  softmax.
Revised: α · grad ( ∂ L tr ∂θ , θ, grad_outputs = v 1 ) 10: p += v 1 11: end for 12: v 2 ← grad ( L tr ∂θ , λ, grad_outputs = α p ) 13: return ∂ L V ∂λ − v 2 14: 15: softmax.

Original: With the class probability, we compute the cross-entropy loss for ProtoNet as follows:
Revised: With the class probability, we compute the cross-entropy loss for ProtoNet as follows:

Original: [Equation]
Revised: [Equation]

Original: where 1 is an indicator function.
Revised: where 1 is an indicator function.

Original: At meta-test time, a test query is assigned a label based on theminimal distance to a class prototype, i.e., y q ∗ = arg min k d
Revised: At meta-test time, a test query is assigned a label based on the minimal distance to a class prototype, i.e., y q ∗ = arg min k d

Original: ( ˆ f θ,λ ( x q ∗ ) , c k ) .
Revised: ( ˆ f θ,λ ( x q ∗ ) , c k ) .

Original: However, optimizing1 T (cid:80) Tt =
Revised: However, optimizing 1 T (cid:80) Tt =

Original: 1 L singleton ( λ, θ ; T t ) w.r.t θ is prone to overfitting since we are given only a small number ofmeta-training tasks.
Revised: 1 L singleton ( λ, θ ; T t ) w.r.t θ is prone to overfitting since we are given only a small number of meta-training tasks.

Original: The meta-learner tends to memorize the meta-training tasks, which limits itsgeneralization to new tasks at meta-test time [51, 35].
Revised: The meta-learner tends to memorize the meta-training tasks, which limits its generalization to new tasks at meta-test time [51, 35].

Original: Meta-Interpolation for Task Augmentation
Revised: Meta-Interpolation for Task Augmentation

Original: In order to tackle the meta-overfitting problem with asmall number of tasks, we propose a novel data-driven domain-agnostic task augmentation frameworkwhich enables the meta-learner trained on few tasks to generalize to unseen few-shot classificationtasks.
Revised: In order to tackle the meta-overfitting problem with a small number of tasks, we propose a novel data-driven domain-agnostic task augmentation framework which enables the meta-learner trained on few tasks to generalize to unseen few-shot classification tasks.

Original: Several methods have been proposed to densify the meta-training tasks.
Revised: Several methods have been proposed to densify the meta-training tasks.

Original: However, theyheavily depend on the augmentation of images [32] or need a pretrained language model for taskaugmentation [30].
Revised: However, they heavily depend on the augmentation of images [32] or need a pretrained language model for task augmentation [30].

Original: Although Manifold Mixup based methods [49, 50] are domain-agnostic, weempirically find them ineffective in certain domains.
Revised: Although Manifold Mixup based methods [49, 50] are domain-agnostic, we empirically find them ineffective in certain domains.

Original: Instead, we optimize expressive neural setfunctions to augment tasks to enhance the generalization of a meta-learner to unseen tasks.
Revised: Instead, we optimize expressive neural set functions to augment tasks to enhance the generalization of a meta-learner to unseen tasks.

Original: As aconsequence of end-to-end training, the learned augmentation strategy is tailored to each domain.
Revised: As a consequence of end-to-end training, the learned augmentation strategy is tailored to each domain.

Original: Specifically, let φ λ : R n × d → R d be a set function which maps a set of d dimensional vectors withcardinality n to a d dimensional vector.
Revised: Specifically, let φ λ : R n × d → R d be a set function which maps a set of d dimensional vectors with cardinality n to a d dimensional vector.

Original: In all our experiments, we use Set Transformer [23] for φ λ .
Revised: In all our experiments, we use Set Transformer [23] for φ λ .

Original: Given a pair of tasks T t 1 = {D st 1 , D qt 1 } and T t 2 = {D st 2 , D qt 2 } with corresponding support and querysets for K way classification, we construct new classes by choosing K pairs of classes from the twotasks.
Revised: Given a pair of tasks T t 1 = {D st 1 , D qt 1 } and T t 2 = {D st 2 , D qt 2 } with corresponding support and query sets for K way classification, we construct new classes by choosing K pairs of classes from the two tasks.

Original: We sample permutations σ t 1 and σ t 2 on { 1 , . . 
Revised: We sample permutations σ t 1 and σ t 2 on { 1 , . . .

Original: , K } for each task T t 1 and T t 2 respectively andassign class k to the pair { σ t 1 ( k ) , σ t 2 ( k ) } for k = 1 , . .
Revised: , K } for each task T t 1 and T t 2 respectively and assign class k to the pair { σ t 1 ( k ) , σ t 2 ( k ) } for k = 1 , . .

Original: , K .
Revised: , K .

Original: For the newly assigned class k , we pairtwo instances from classes σ t 1 ( k ) and σ t 2 ( k ) and interpolate their hidden representations with theset function φ λ .
Revised: For the newly assigned class k , we pair two instances from classes σ t 1 ( k ) and σ t 2 ( k ) and interpolate their hidden representations with the set function φ λ .

Original: The class prototypes for class k are computed using the output of φ λ as follows:
Revised: The class prototypes for class k are computed using the output of φ λ as follows:

Original: [Equation]
Revised: [Equation]

Original: where we define ˆ D s to be the set of all the interpolated prototypes ˆ c k for k = 1 , .
Revised: where we define ˆ D s to be the set of all the interpolated prototypes ˆ c k for k = 1 , .

Original: , K .
Revised: , K .

Original: For queries,we do not perform any interpolation.
Revised: For queries, we do not perform any interpolation.

Original: Instead, we use D qt 1 as the query set and compute its hidden representation ˆ f θ,λ ( x qt 1 ,i ) ∈ R D .
Revised: Instead, we use D qt 1 as the query set and compute its hidden representation ˆ f θ,λ ( x qt 1 ,i ) ∈ R D .

Original: We then measure the distance between the query with y qt 1 ,i =
Revised: We then measure the distance between the query with y qt 1 ,i =

Original: σ t 1 ( k )and the interpolated prototype of class k to compute the loss as follows:
Revised: σ t 1 ( k ) and the interpolated prototype of class k to compute the loss as follows:

Original: [Equation]
Revised: [Equation]

Original: where ˆ T = { ˆ D s , D qt 1 } .
Revised: where ˆ T = { ˆ D s , D qt 1 } .

Original: The intuition behind interpolating only support sets is to construct harder tasksthat a meta-learner cannot memorize.
Revised: The intuition behind interpolating only support sets is to construct harder tasks that a meta-learner cannot memorize.

Original: Alternatively, we can interpolate only query sets.
Revised: Alternatively, we can interpolate only query sets.

Original: However, thisis computationally more expensive since the size of query sets is usually larger than that of supportsets.
Revised: However, this is computationally more expensive since the size of query sets is usually larger than that of support sets.

Original: In Section 5, we empirically show that interpolating either support or query sets achieves highertraining loss than interpolating both, which empirically supports the intuition.
Revised: In Section 5, we empirically show that interpolating either support or query sets achieves higher training loss than interpolating both, which empirically supports the intuition.

Original: Lastly, we also use theoriginal task T t 1 to evaluate the loss L singleton ( λ, θ, T t 1 ) in Eq.
Revised: Lastly, we also use the original task T t 1 to evaluate the loss L singleton ( λ, θ, T t 1 ) in Eq.

Original: by passing the corresponding supportand query set to ˆ f θ,λ .
Revised: by passing the corresponding support and query set to ˆ f θ,λ .

Original: The additional forward pass enriches the diversity of the augmented tasks andmakes meta-training consistent with meta-testing since we do not perform any task augmentation inthe meta-testing stage.
Revised: The additional forward pass enriches the diversity of the augmented tasks and makes meta-training consistent with meta-testing since we do not perform any task augmentation in the meta-testing stage.

Original: Optimization Since jointly optimizing θ , the parameters of ProtoNet, and λ , the parameters of theset function φ λ , with few tasks is prone to overfitting, we consider λ as hyperparameters and performbilevel optimization with meta-training and meta-validation tasks as follows:
Revised: Since jointly optimizing θ , the parameters of ProtoNet, and λ , the parameters of the set function φ λ , with few tasks is prone to overfitting, we consider λ as hyperparameters and perform bilevel optimization with meta-training and meta-validation tasks as follows:

Original: [Equation]
Revised: [Equation]

Original: where T train t , T val t , ˆ T t denote the meta-training, meta-validation, and interpolated task, respectively.
Revised: where T train t , T val t , ˆ T t denote the meta-training, meta-validation, and interpolated task, respectively.

Original: Since computing the exact gradient w.r.t λ is intractable due to the long inner optimization steps in
Revised: Since computing the exact gradient w.r.t λ is intractable due to the long inner optimization steps in

Original: Eq. 6, we leverage the implicit function theorem to approximate the gradient as Lorraine et al.
Revised: Eq. 6, we leverage the implicit function theorem to approximate the gradient as Lorraine et al.

Original: Moreover, we alternately update θ and λ for computational efficiency as described in Algo.
Revised: Moreover, we alternately update θ and λ for computational efficiency as described in Algo.

Original: 1 and 2.
Revised: 1 and 2.

Original:  Theoretical Analysis
Revised:  Theoretical Analysis

Original: In this section, we theoretically investigate the behavior of the Set Transformer and how it inducesa distribution dependent regularization, which is then shown to have the ability to control the
Revised: In this section, we theoretically investigate the behavior of the Set Transformer and how it induces a distribution dependent regularization, which is then shown to have the ability to control the

Original: Rademacher complexity for better generalization.
Revised: Rademacher complexity for better generalization.

Original: To analyze the behavior of the Set Transformer,we first define it concretely with the attention mechanism A ( Q, K, V ) =
Revised: To analyze the behavior of the Set Transformer, we first define it concretely with the attention mechanism A ( Q, K, V ) =

Original: softmax( √ d − 1 QK ⊤ ) V .
Revised: softmax( √ d − 1 QK ⊤ ) V .

Original: Given h, h ′ ∈ R d , define H { h,h ′ }=
Revised: Given h, h ′ ∈ R d , define H { h,h ′ }=

Original: [Equation] , h ′ ]
Revised: [ h, h ′ ]

Original: ⊤ ∈ R 2 × d
Revised: ⊤ ∈ R 2 × d

Original: and H { [Equation] }= h ⊤ ∈ R 1 × d .
Revised: and H { h }= h ⊤ ∈ R 1 × d .

Original: Then, for anyr ∈ {{ h, h ′ } , { h }} , the output of the Set Transformer φ λ ( r ) is defined as follows:
Revised: Then, for any r ∈ {{ h, h ′ } , { h }} , the output of the Set Transformer φ λ ( r ) is defined as follows:

Original: [Equation]
Revised: [Equation]

Original: where Q 2 = SW Q 2 + b Q 2 , Q r 1 = H r 1 W Q 1 + 1 2 b Q 1 , K rj
Revised: where Q 2 = SW Q 2 + b Q 2 , Q r 1 = H r 1 W Q 1 + 1 2 b Q 1 , K rj

Original: = H rj W Kj + 1 2 b Kj , V rj
Revised: = H rj W Kj + 1 2 b Kj , V rj

Original: = H rj W Vj + 1 2 b Vj(for j ∈ { 1 , 2 } ), and H r 2 =
Revised: = H rj W Vj + 1 2 b Vj (for j ∈ { 1 , 2 } ), and H r 2 =

Original: A ( Q r 1 , K r 1 , V r 1 ) ∈ R n × d .
Revised: A ( Q r 1 , K r 1 , V r 1 ) ∈ R n × d .

Original: Q j , K j , V j denote query, key, and value forthe attention mechanism for j = 1 , 2 , respectively.
Revised: Q j , K j , V j denote query, key, and value for the attention mechanism for j = 1 , 2 , respectively.

Original: Here, 1 2 = [1 , . . .
Revised: Here, 1 2 = [1 , . . .

Original: , 1] ⊤ ∈ R n , W Qj , W Kj , W Vj ∈
Revised: , 1] ⊤ ∈ R n , W Qj , W Kj , W Vj ∈

Original: R d × d , b Qj , b Kj , b Vj ∈ R 1 × d , Q r 1 , K rj , V rj ∈ R n × d , and Q 2 ∈ R 1 × d .
Revised: R d × d , b Qj , b Kj , b Vj ∈ R 1 × d , Q r 1 , K rj , V rj ∈ R n × d , and Q 2 ∈ R 1 × d .

Original: Let l ∈ { 1 , . .
Revised: Let l ∈ { 1 , . .

Original: , L } .
Revised: , L } .

Original: Our analysis will show the importance of the following quantity of the Set Transformer in our method:
Revised: Our analysis will show the importance of the following quantity of the Set Transformer in our method:

Original: [Equation]
Revised: [Equation]

Original:  where
Revised: ) where

Original: [Equation]
Revised: [Equation]

Original: with h t,i =
Revised: with h t,i =

Original: ϕ lθ ( x st,i ) and ϕ lθ
Revised: ϕ lθ ( x st,i ) and ϕ lθ

Original: = f lθ l ◦ · · · ◦ f 1 θ 1 .
Revised: = f lθ l ◦ · · · ◦ f 1 θ 1 .

Original: For a matrix A ∈ R m × n , A i,j denotes the entry fori -th row and j -th column of the matrix A .
Revised: For a matrix A ∈ R m × n , A i,j denotes the entry for i -th row and j -th column of the matrix A .

Original: We now introduce the additional notation and problem setting to present our results.
Revised: We now introduce the additional notation and problem setting to present our results.

Original: Define W =
Revised: Define W =

Revised: ( W V 1 W V 2 )

Revised: ⊤ ∈ R d × d , b =

Revised: ( b V 1 W V 2 + b V 2 )

Revised: ⊤ ∈ R d , L t ( c ) = − 1 n (cid:80) ni

Revised: =1 log exp( − d

Original: [Equation]
Revised: [Equation]

Original: ∈
Revised: ˆ f θ,λ ( x qt,i ) , c yqt,i )) (cid:80)

Original: [Equation]
Revised: [Equation]

Original:  and I t,k= { i ∈
Revised: ,i ) , c k ′ )) , and I t,k

Original:  [ N ( t ) s ] : y st,i = k } , where N ( t ) s =
Revised: = { i ∈ [ N ( t ) s ] : y st,i = k } , where N ( t ) s =

Original: |D st | .
Revised: |D st | .

Original: We also define the empirical measureµ t,k = 1 | I t,k | (cid:80) i ∈
Revised: We also define the empirical measure µ t,k = 1 | I t,k | (cid:80) i ∈

Original: I t,k δ i over the index i ∈
Revised: I t,k δ i over the index i ∈

Original: [ N ( t ) s ] with the Dirac measures
Revised: [ N ( t ) s ] with the Dirac measures

Original: δ i .
Revised: δ i .

Original: Let U [ K ] be theuniform distribution over { 1 , . . .
Revised: Let U [ K ] be the uniform distribution over { 1 , . . .

Original: , K } .
Revised: , K } .

Original: For any function φ and point u in its domain, we definethe j -th order tensor ∂
Revised: For any function φ and point u in its domain, we define the j -th order tensor ∂

Original: j φ ( u ) ∈ R d × d ×···× d by ∂ j φ ( u ) i 1 i 2 ··· i j =
Revised: j φ ( u ) ∈ R d × d ×···× d by ∂ j φ ( u ) i 1 i 2 ··· i j =

Original: [Equation]
Revised: [Equation]

Original: ∂u [Equation] [Equation] 2 ··· ∂u ij φ ( u ) .
Revised: ∂u i 1 u i 2 ··· ∂u ij φ ( u ) .

Original: For example,∂ 1 φ ( u ) and ∂ 2 φ ( u ) are the gradient and the Hessian of φ evaluated at u .
Revised: For example, ∂ 1 φ ( u ) and ∂ 2 φ ( u ) are the gradient and the Hessian of φ evaluated at u .

Original: For any j -th order tensor∂ j φ ( u ) , we define the vectorization of the tensor by vec[ ∂ j φ ( u )]
Revised: For any j -th order tensor ∂ j φ ( u ) , we define the vectorization of the tensor by vec[ ∂ j φ ( u )]

Original: ∈ R d j .
Revised: ∈ R d j .

Original: For an vector a ∈ R d , wedefine a ⊗ j = a ⊗ a ⊗ · · · ⊗ a ∈ R d j where ⊗ represents the Kronecker product.
Revised: For an vector a ∈ R d , we define a ⊗ j = a ⊗ a ⊗ · · · ⊗ a ∈ R d j where ⊗ represents the Kronecker product.

Original: We assume that∂ r g (cid:0) Wϕ lθ l ( x st,i )
Revised: We assume that ∂ r g (cid:0) Wϕ lθ l ( x st,i )

Original: + b (cid:1)
Revised: + b (cid:1)

Original: = 

Revised: =

Original: for all r ≥ 2 , where g :=
Revised: for all r ≥ 2 , where g :=

Original: f Lθ L ◦ · · · ◦ f l +1 θ l +1 .
Revised: f Lθ L ◦ · · · ◦ f l +1 θ l +1 .

Original: This assumption is satisfied,for example, if g represents a deep neural network with ReLU activations.
Revised: This assumption is satisfied, for example, if g represents a deep neural network with ReLU activations.

Original: This assumption is alsosatisfied in the simpler special case considered in the proposition below.
Revised: This assumption is also satisfied in the simpler special case considered in the proposition below.

Original: The following theorem shows that L mix ( λ, θ, ˆ T t,t ′ ) is approximately L singleton ( λ, θ ; T t ) plus regularization terms on the directional derivatives of ϕ lθ l on the direction of W ( ϕ lθ l ( x st ′ ,j )
Revised: The following theorem shows that L mix ( λ, θ, ˆ T t,t ′ ) is approximately L singleton ( λ, θ ; T t ) plus regularization terms on the directional derivatives of ϕ lθ l on the direction of W ( ϕ lθ l ( x st ′ ,j )

Original: − ϕ lθ l ( x st,i )) :
Revised: − ϕ lθ l ( x st,i )) :

Original: Theorem 1.
Revised: Theorem 1.

Original: For any J ∈ N + , if c (cid:55)→ d ( y, c ) is J -times differentiable for all y , then the J -th order approximation of L mix ( λ, θ, ˆ T t,t ′ ) is given by L singleton ( λ, θ ; T t )
Revised: For any J ∈ N + , if c (cid:55)→ d ( y, c ) is J -times differentiable for all y , then the J -th order approximation of L mix ( λ, θ, ˆ T t,t ′ ) is given by L singleton ( λ, θ ; T t )

Original: + (cid:80)
Revised: + (cid:80)

Original: Jj =1 1 j ! vec[ ∂ j L t ( c )]
Revised: Jj =1 1 j ! vec[ ∂ j L t ( c )]

Original: ⊤ ∆ ⊗ j , where ∆ =
Revised: ⊤ ∆ ⊗ j , where ∆ =

Original: [∆ ⊤ 1 , . . . , ∆ ⊤ K ] ⊤ and
Revised: [∆ ⊤ 1 , . . . , ∆ ⊤ K ] ⊤ and

Original: [Equation]
Revised: [Equation]

Original: To illustrate the effect of this data-dependent regularization, we now consider the following special case that is used by Yao et al.
Revised: To illustrate the effect of this data-dependent regularization, we now consider the following special case that is used by Yao et al.

Original: [50] for ProtoNet: L ( λ, θ ; T t )
Revised: [50] for ProtoNet: L ( λ, θ ; T t )

Original: = 1 n (cid:80) ni
Revised: = 1 n (cid:80) ni

Original: =

Revised: =

Original: L i ( λ, θ ; T t ) where 
Revised: L i ( λ, θ ; T t ) where L i ( λ, θ ; T t ) = [Equation] =

Original: L i ( λ, θ ; T t ) = 1 1+exp( ⟨ ( x qt,i − ( c ′ 1 + c ′ 2 ) / 2 ,θ ⟩ )
Revised: [Equation] st [Equation] ∈D

Original: where c ′ k :
Revised: st [Equation]

Original:  = 1 N t,k (cid:80) ( xst,i , y st,i ) ∈D st1 { y t,i
Revised: [Equation] = [Equation] } x st

Original: = k } x st,i , and⟨· , ·⟩ denotes dot product.
Revised: [Equation] and ⟨· , ·⟩ denotes dot product.

Original: Define c = 1 n (cid:80) ni
Revised: Define c = 1 n (cid:80) ni

Original: =1 14 ψ ( z t,i )( ψ ( z t,i ) − 0 . 5) 1+exp( z t,i ) , where ψ ( z t,i ) =
Revised: =1 14 ψ ( z t,i )( ψ ( z t,i ) − 0 . 5) 1+exp( z t,i ) , where ψ ( z t,i ) =

Original: exp( z t,i ) 1+exp( z t,i )and z t,i = ⟨ x qt,i − ( c ′ 1 + c ′ 2 ) / 2 , θ ⟩ .
Revised: exp( z t,i ) 1+exp( z t,i ) and z t,i = ⟨ x qt,i − ( c ′ 1 + c ′ 2 ) / 2 , θ ⟩ .

Original: Note that c > 

Revised: Note that c >

Original: if θ is no worse than the randomguess; e.g., L i ( λ, 0; T t ) >
Revised: if θ is no worse than the random guess; e.g., L i ( λ, 0; T t ) >

Original: L i ( λ, θ ; T t ) for all i ∈ [ n ] .
Revised: L i ( λ, θ ; T t ) for all i ∈ [ n ] .

Original: We write ∥ v ∥ 2 M = v ⊤ Mv for anypositive semi-definite matrix M .
Revised: We write ∥ v ∥ 2 M = v ⊤ Mv for any positive semi-definite matrix M .

Original: In this special case, we consider that α is balanced: i.e.,
Revised: In this special case, we consider that α is balanced: i.e.,

Original: E t ′ ,σ [ (cid:80) 2 k = 1 | I t ′ ,σ ( k ) | (cid:80)
Revised: E t ′ ,σ [ (cid:80) 2 k = 1 | I t ′ ,σ ( k ) | (cid:80)

Original: j ∈
Revised: j ∈

Revised: I t

Revised: [Equation]

Revised: [Equation]

Revised: k

Original: I t ′ ,σ ( k ) 1 | I t,k | (cid:80)i ∈
Revised: [Equation] t, [Equation] | (cid:80)

Original: I t,k α ( t,t′ ) ij ( ϕ lθ l ( x st ′ ,j ) − ϕ
Revised: [Equation] t, [Equation] t,t

Original:  lθ l ( x st,i ))]
Revised: [Equation] ( ϕ lθ l ( x st ′ ,j ) − ϕ lθ l ( x st,i ))]

Original: = 0 for all t .
Revised: = 0 for all t .

Original: Thisis used to prevent the Set Transformer from over-fitting to the training sets; i.e., in such simple specialcases, the Set Transformer without any restriction is too expressive relative to the rest of the model(and may memorize the training sets without using the rest of the model).
Revised: This is used to prevent the Set Transformer from over-fitting to the training sets; i.e., in such simple special cases, the Set Transformer without any restriction is too expressive relative to the rest of the model (and may memorize the training sets without using the rest of the model).

Original: The following propositionshows that the additional regularization term is simplified to the form of c ∥ θ ∥ 2 M in this special case:
Revised: The following proposition shows that the additional regularization term is simplified to the form of c ∥ θ ∥ 2 M in this special case:

Original: Proposition 1.
Revised: Proposition 1.

Original: In the special case explained above, the second approximation of
Revised: In the special case explained above, the second approximation of

Original: E t ′ ,σ [ L mix ( λ, θ, ˆ T t,t ′ )] is given by L singleton ( λ, θ ; T t )
Revised: E t ′ ,σ [ L mix ( λ, θ, ˆ T t,t ′ )] is given by L singleton ( λ, θ ; T t )

Original: + c ∥ θ ∥ 2 E t ′ ,σ [ δ t,t ′ ,σ δ ⊤ t,t ′ ,σ ] , where δ t,t ′ ,σ =E [Equation] ( x st
Revised: + c ∥ θ ∥ 2 E t ′ ,σ [ δ t,t ′ ,σ δ ⊤ t,t ′ ,σ ] , where δ t,t ′ ,σ =

Original: [Equation]
Revised: [Equation]

Original: j − x st

Original: [Equation]

Original: i )] .

Original: In the above regularization form, we have an implicit regularization effect on ∥ θ ∥ 2Σ where Σ =
Revised: In the above regularization form, we have an implicit regularization effect on ∥ θ ∥ 2Σ where Σ =

Original: E x , x ′ [( x − x ′ )( x − x ′ ) ⊤ ] .
Revised: E x , x ′ [( x − x ′ )( x − x ′ ) ⊤ ] .

Original: The following theorem shows that this implicit regularization can reducethe Rademacher complexity for better generalization:
Revised: The following theorem shows that this implicit regularization can reduce the Rademacher complexity for better generalization:

Original: Proposition 2.
Revised: Proposition 2.

Original: Let F R = { x (cid:55)→ θ ⊤ x : ∥ θ ∥ 2Σ ≤ R } with E x [ x ] = 0 .
Revised: Let F R = { x (cid:55)→ θ ⊤ x : ∥ θ ∥ 2Σ ≤ R } with E x [ x ] = 0 .

Original: Then, R n ( F [Table]
Revised: Then, R n ( F R ) ≤ √ R √ rank(Σ) √ n .

Original: All the proofs are presented in Appendix B.
Revised: All the proofs are presented in Appendix A.

Original:  Experiments
Revised:  Experiments

Original: We now demonstrate the efficacy of our set-based task augmentation method on multiple few-taskbenchmark datasets and compare against the relevant baselines.
Revised: We now demonstrate the efficacy of our set-based task augmentation method on multiple few-task benchmark datasets and compare against the relevant baselines.

Original: Datasets We perform classification on eight datasets to validate our method.
Revised: Datasets We perform classification on eight datasets to validate our method.

Original: Metabolism [17], NCI [31] and Tox21 [18]: these are binary classification datasets for predictingthe properties of chemical molecules.
Revised: Metabolism [17], NCI [31] and Tox21 [18]: these are binary classification datasets for predicting the properties of chemical molecules.

Original: For Metabolism, we use three subdatasets for meta-training,meta-validation, and meta-testing, respectively.
Revised: For Metabolism, we use three subdatasets for meta-training, meta-validation, and meta-testing, respectively.

Original: For NCI, we use four subdatasets for meta-training,two for meta-validation and the remaining three for meta-testing.
Revised: For NCI, we use four subdatasets for meta-training, two for meta-validation and the remaining three for meta-testing.

Original: For Tox21, we use six subdatasets
Revised: For Tox21, we use six subdatasets

Original: [Table]
Revised: [Table]

Original: for meta-training, two for meta-validation, and four for meta-testing.
Revised: for meta-training, two for meta-validation [Table] and four for meta-testing.

Original: (4) GLUE-SciTail [30]: itconsists of four natural language inference datasets where we predict whether a hypothesis sentencecontradicts a premise sentence.
Revised: (4) GLUE-SciTail [30]: it consists of four natural language inference datasets where we predict whether a hypothesis sentence contradicts a premise sentence.

Original: We use MNLI [47] and QNLI [46] for meta-training, SNLI [5]and RTE [46] for meta-validation, and SciTail [20] for meta-testing.
Revised: We use MNLI [47] and QNLI [46] for meta-training, SNLI [5] and RTE [46] for meta-validation, and SciTail [20] for meta-testing.

Original: (5) ESC-50 [34]: this is anautomatic speech recognition dataset.
Revised: (5) ESC-50 [34]: this is an automatic speech recognition dataset.

Original: We make a 20/15/15 split out of 50 base classes for metatraining/validation/testing and sample 5 classes from each spilt to construct a 5-way classificationtask.
Revised: We make a 20/15/15 split out of 50 base classes for metatraining/validation/testing and sample 5 classes from each spilt to construct a 5-way classification task.

Original: (6) Rainbow MNIST (RMNIST) [11]: this is a 10-way classification dataset.
Revised: (6) Rainbow MNIST (RMNIST) [11]: this is a 10-way classification dataset.

Original: Following Yaoet al.
Revised: Following Yao et al.

Original: [50], we construct each task by applying compositions of image transformations to the imagesof the MNIST [9] dataset.
Revised: [50], we construct each task by applying compositions of image transformations to the images of the MNIST [9] dataset.

Original: (7) & (8) Mini-ImageNet-S [45] and CIFAR100-FS [22]: these are5-way classification datasets where we choose 12/16/20 classes out of 100 base classes for metatraining/validation/testing, respectively and sample 5 classes from each split to construct a task.
Revised: (7) & (8) Mini-ImageNet-S [45] and CIFAR100-FS [22]: these are 5-way classification datasets where we choose 12/16/20 classes out of 100 base classes for metatraining/validation/testing, respectively and sample 5 classes from each split to construct a task.

Original: Note that Metabolism, Tox21, NCI, GLUE-SciTail, and ESC-50 are real-world few-task metalearning datasets with a very small number of tasks.
Revised: Note that Metabolism, Tox21, NCI, GLUE-SciTail, and ESC-50 are real-world few-task metalearning datasets with a very small number of tasks.

Original: For Mini-ImageNet-S and CIFAR100-FS,following Yao et al.
Revised: For Mini-ImageNet-S and CIFAR100-FS, following Yao et al.

Original: [50], we artificially reduce the number of tasks from the original datasets forfew-task meta-learning.
Revised: [50], we artificially reduce the number of tasks from the original datasets for few-task meta-learning.

Original: RMNIST is synthetically generated by applying augmentations to MNIST.
Revised: RMNIST is synthetically generated by applying augmentations to MNIST.

Original: Implementation Details For RMNIST, Mini-ImageNet-S, and CIFAR100-FS, we use four convolutional blocks with each block consisting of a convolution, ReLU, batch normalization [19], andmax pooling.
Revised: Implementation Details For RMNIST, Mini-ImageNet-S, and CIFAR100-FS, we use four convolutional blocks with each block consisting of a convolution, ReLU, batch normalization [19], and max pooling.

Original: For Metabolism, Tox21, and NCI, we convert the chemical molecules into SMILESformat and extract a 1024 bit fingerprint feature using RDKit [15] where each bit captures a fragmentof the molecule.
Revised: For Metabolism, Tox21, and NCI, we convert the chemical molecules into SMILES format and extract a 1024 bit fingerprint feature using RDKit [15] where each bit captures a fragment of the molecule.

Original: We use two blocks of affine transformation, batch normalization, and Leaky ReLU,and affine transformation for the last layer.
Revised: We use two blocks of affine transformation, batch normalization, and Leaky ReLU, and affine transformation for the last layer.

Original: For GLUE-SciTail dataset, we stack 3 fully connectedlayers with ReLU on the pretrained language model ELECTRA [8].
Revised: For GLUE-SciTail dataset, we stack 3 fully connected layers with ReLU on the pretrained language model ELECTRA [8].

Original: For ESC-50 dataset, we pass rawaudio signal to the pretrained VGGish [16] feature extractor to obtain an embedding vector.
Revised: For ESC-50 dataset, we pass raw audio signal to the pretrained VGGish [16] feature extractor to obtain an embedding vector.

Original: We usethe feature vector as input to the classifier which is exactly the same as the one used for Metabolism,
Revised: We use the feature vector as input to the classifier which is exactly the same as the one used for Metabolism,

Original: Tox21, and NCI.
Revised: Tox21, and NCI.

Original: For our Meta-Interpolation, we use Set Transformer [23] for the set function φ λ .
Revised: For our Meta-Interpolation, we use Set Transformer [23] for the set function φ λ .

Original: Baselines We compare our method against following domain-agnostic baselines.
Revised: Baselines We compare our method against following domain-agnostic baselines.

Original: ProtoNet [40]: Vanilla ProtoNet trained on Eq. 2 by fixing φ λ to be the identity function.2.
Revised: ProtoNet [40]: Vanilla ProtoNet trained on Eq. 2 by fixing φ λ to be the identity function

Original: MetaReg [2]: ProtoNet with ℓ 2 regularization where element-wise coefficients are learned withbilevel optimization.3.
Revised: MetaReg [2]: ProtoNet with ℓ 2 regularization where element-wise coefficients are learned with bilevel optimization.

Original: MetaMix [49]: ProtoNet trained with support sets and mixed query sets where we interpolate oneinstance from the support sets and the other from the original query sets with Manifold Mixup.4.
Revised: MetaMix [49]: ProtoNet trained with support sets and mixed query sets where we interpolate one instance from the support sets and the other from the original query sets with Manifold Mixup.

Original: MLTI [50]: ProtoNet trained with Manifold Mixup based task augmentation.
Revised: MLTI [50]: ProtoNet trained with Manifold Mixup based task augmentation.

Original: We sample twotasks and interpolate two query sets and support sets with Manifold Mixup, respectively.5.
Revised: We sample two tasks and interpolate two query sets and support sets with Manifold Mixup, respectively.

Original: ProtoNet+ST ProtoNet and Set Transformer ( φ λ ) trained with bilevel optimization but withouttask augmentation for L mix ( λ, θ, ˆ T t ) in Eq.
Revised: ProtoNet+ST ProtoNet and Set Transformer ( φ λ ) trained with bilevel optimization but without task augmentation for L mix ( λ, θ, ˆ T t ) in Eq.

Original: Meta-Interpolation Our full model learning to interpolate support sets from two tasks usingbilevel optimization and training the ProtoNet with both the original and interpolated tasks.
Revised: Meta-Interpolation Our full model learning to interpolate support sets from two tasks using bilevel optimization and training the ProtoNet with both the original and interpolated tasks.

Original: Results As shown in Table 1, Meta-Interpolation consistently outperforms all the domain-agnostictask augmentation and regularization baselines on non-image domains.
Revised: Results As shown in Table 1, Meta-Interpolation consistently outperforms all the domain-agnostic task augmentation and regularization baselines on non-image domains.

Original: Notably, it significantlyimproves the performance on ESC-50, which is a challenging datatset that only contains 40 examplesper class.
Revised: Notably, it significantly improves the performance on ESC-50, which is a challenging datatset that only contains 40 examples per class.

Original: In addition, Meta-Interpolation effectively tackles the Metabolism and GLUE-SciTaildatasets which have an extremely small number of meta-training tasks: three and two meta-trainingtasks, respectively.
Revised: In addition, Meta-Interpolation effectively tackles the Metabolism and GLUE-SciTail datasets which have an extremely small number of meta-training tasks: three and two meta-training tasks, respectively.

Original: Contrarily, the baselines do not achieve consistent improvements across all the
Revised: Contrarily, the baselines do not achieve consistent improvements across all the

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table][Figure] [Table] [Figure]
Revised: [Table]

Original: [Table]

Original: [Figure]
Revised: [Figure]

Original: domains and tasks considered.
Revised: domains and tasks considered.

Original: For example, MetaReg is effective on the speech domain (ESC-50)and Metabolism, but does not work on the chemical (Tox21 and NCI) and text (GLUE-SciTail)domains.
Revised: For example, MetaReg is effective on the speech domain (ESC-50) and Metabolism, but does not work on the chemical (Tox21 and NCI) and text (GLUE-SciTail) domains.

Original: Similarly, MetaMix and MLTI achieve performance improvements on some datasets butdegrade the test accuracy on others.
Revised: Similarly, MetaMix and MLTI achieve performance improvements on some datasets but degrade the test accuracy on others.

Original: This empirical evidence supports the hypothesis that the optimaltask augmentation strategy varies across domains and justifies the motivation for Meta-Interpolationwhich learns augmentation strategies tailored to each domain.
Revised: This empirical evidence supports the hypothesis that the optimal task augmentation strategy varies across domains and justifies the motivation for Meta-Interpolation which learns augmentation strategies tailored to each domain.

Original: We provide additional experimental results on the image domain in Table 2.
Revised: We provide additional experimental results on the image domain in Table 2.

Original: Again, Meta-Interpolationoutperforms all the baselines.
Revised: Again, Meta-Interpolation outperforms all the baselines.

Original: In contrast to the previous experiments, MetaReg hurts the generalization performance on all the image datasets except on RMNIST.
Revised: In contrast to the previous experiments, MetaReg hurts the generalization performance on all the image datasets except on RMNIST.

Original: Note that Manifold Mixup-basedaugmentation methods, MetaMix and MLTI, marginally improve the generalization performancefor 1-shot classification on Mini-ImageNet-S and CIFAR-100-FS, although they boost the accuracyon 5-shot experiments.
Revised: Note that Manifold Mixup-based augmentation methods, MetaMix and MLTI, marginally improve the generalization performance for 1-shot classification on Mini-ImageNet-S and CIFAR-100-FS, although they boost the accuracy on 5-shot experiments.

Original: This suggests that different task augmentation strategies are required for1-shot and 5-shot for the same dataset.
Revised: This suggests that different task augmentation strategies are required for 1-shot and 5-shot for the same dataset.

Original: Meta-Interpolation on the other hand learns task augmentationstrategies tailored for each task and dataset and consistently improves the performance of the vanilla
Revised: Meta-Interpolation on the other hand learns task augmentation strategies tailored for each task and dataset and consistently improves the performance of the vanilla

Original: ProtoNet for all the experiments on the image datasets.
Revised: ProtoNet for all the experiments on the image datasets.

Original: Moreover, we plot the meta-training and meta-validation loss on RMNIST and NCI dataset in Figure 2. 
Revised: Moreover, we plot the meta-training and meta-validation loss on RMNIST and NCI dataset in Figure 2. [Figure]

Original: Meta-Interpolation obtains higher training loss but much lower validation loss than the others on bothdatasets.
Revised: -Interpolation obtains higher training loss but much lower validation loss than the others on both datasets.

Original: This implies that interpolating only support sets constructs harder tasks that a meta-learnercannot memorize and regularizes the meta-learner for better generalization.
Revised: This implies that interpolating only support sets constructs harder tasks that a meta-learner cannot memorize and regularizes the meta-learner for better generalization.

Original: ProtoNet overfits to themeta-training tasks on both datasets.
Revised: ProtoNet overfits to the meta-training tasks on both datasets.

Original: MLTI mitigates the overfitting issue on RMNIST but is noteffective on the NCI dataset where it shows high validation loss in Figure 2d.
Revised: MLTI mitigates the overfitting issue on RMNIST but is not effective on the NCI dataset where it shows high validation loss in Figure 2d.

Original: On the other hand,
Revised: On the other hand,

Original: MetaMix, which constructs a new query set by interpolating a support and query set with Manifold
Revised: MetaMix, which constructs a new query set by interpolating a support and query set with Manifold

Original: Mixup, results in generating overly difficult tasks which causes underfitting on RMNIST where thetraining loss is not properly minimized in Figure 2a.
Revised: Mixup, results in generating overly difficult tasks which causes underfitting on RMNIST where the training loss is not properly minimized in Figure 2a.

Original: However, this augmentation strategy is effectivefor tackling meta-overfitting on NCI where the validation loss is lower than ProtoNet.
Revised: However, this augmentation strategy is effective for tackling meta-overfitting on NCI where the validation loss is lower than ProtoNet.

Original: The loss curveof ProtoNet+ST supports the claim that increasing the model size and using bilevel optimizationcannot handle the few-task meta-learning problem.
Revised: The loss curve of ProtoNet+ST supports the claim that increasing the model size and using bilevel optimization cannot handle the few-task meta-learning problem.

Original: It shows higher validation loss on both RMNISTand NCI as presented in Figure 2b and 2d.
Revised: It shows higher validation loss on both RMNIST and NCI as presented in Figure 2b and 2d.

Original: Similarly, MetaReg which learns coefficients for ℓ 2regularization fails to prevent meta-overfitting on both datasts.
Revised: Similarly, MetaReg which learns coefficients for ℓregularization fails to prevent meta-overfitting on both datasts.

Original: Lastly, we empirically show that the performance gains mostly come from the task augmentation with
Revised: Lastly, we empirically show that the performance gains mostly come from the task augmentation with

Original: Meta-Interpolation, rather than from bilevel optimization or the introduction of extra parameters withthe set function.
Revised: Meta-Interpolation, rather than from bilevel optimization or the introduction of extra parameters with the set function.

Original: As shown in Table 1 and 2, ProtoNet+ST, which is Meta Interpolation but trainedwithout any task augmentation, significantly degrades the performance of ProtoNet on Mini-ImageNetand CIFAR-100-FS.
Revised: As shown in Table 1 and 2, ProtoNet+ST, which is Meta Interpolation but trained without any task augmentation, significantly degrades the performance of ProtoNet on Mini-ImageNet and CIFAR-100-FS.

Original: On the other datasets, the ProtoNet+ST obtains marginal improvement or largely
Revised: On the other datasets, the ProtoNet+ST obtains marginal improvement or largely

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: underperforms the other baselines.
Revised: underperforms the other baselines.

Original: Thus, the task augmentation strategy of interpolating two supportsets with the set function φ λ is indeed crucial for tackling the few-task meta-learning problem.[Table]
Revised: Thus, the task augmentation strategy of interpolating two support sets with the set function φ λ is indeed crucial for tackling the few-task meta-learning problem.

Original: Ablation Study We further perform ablation studies to verifythe effectiveness of each component of Meta-Interpolation.
Revised: Ablation Study We further perform ablation studies to verify the effectiveness of each component of Meta-Interpolation.

Original: In Table 3, we show experimental results on the ESC-50 dataset whileremoving various components of our model.
Revised: In Table [Table] we show experimental results on the ESC-50 dataset by removing various components of our model.

Original: Firstly, we train ourmodel without any task interpolation but keep the set functionφ λ , denoted as w/o Interpolation.
Revised: Firstly, we train our model without any task interpolation but keep the set function φ λ , denoted as w/o Interpolation.

Original: The model without task interpolation significantly underperforms the full task-augmentationmodel, Meta-Interpolation, which shows that the improvements come from task interpolation, ratherthan the extra parameters introduced by the set encoding layer.
Revised: The model without task interpolation significantly underperforms the full task-augmentation model, Meta-Interpolation, which shows that the improvements come from task interpolation rather than the extra parameters introduced by the set encoding layer.

Original: Moreover, bilevel optimizationis shown to be effective for estimating λ , which are the parameters of the set function.
Revised: Moreover, bilevel optimization is shown to be effective for estimating λ , which are the parameters of the set function.

Original: Jointlytraining ProtoNet and the setthe function without bilevel optimization, denoted as w/o Bilevel,largely degrades the test accuracy by 15% .
Revised: Jointly training the ProtoNet and the set function without bilevel optimization, denoted as w/o Bilevel, largely degrades the test accuracy by 15% .

Original: Lastly, we remove the loss L singleton ( λ, θ, T train t ) for inneroptimization in Eq.
Revised: Lastly, we remove the loss L singleton ( λ, θ, T train t ) for inner optimization in Eq.

Original: 6, denoted as w/o L singleton ( λ, θ, T train t ) .
Revised: 6, denoted as w/o L singleton ( λ, θ, T train t ) .

Original: This hurts the generalization performance since it decreases the diversity of tasks and causes inconsistency between meta-training andmeta-testing, since we do not perform any interpolation for support sets at meta-test time.
Revised: This hurts the generalization performance since it decreases the diversity of tasks and causes inconsistency between meta-training and meta-testing, since we do not perform any interpolation for support sets at meta-test time.

Revised: We also explore an alternative set function such as DeepSets [52] using the ESC50 dataset to show the general effectiveness of our method regardless of the set encoding scheme.

Revised: In Table 4, MetaInterpolation with DeepSets improves the generalization performance of ProtoTypical Network and the model with Set Transformer further boost the performance as a consequence of higher-order and pairwise interactions among the set elements via the attention mechanism.

Revised: [Table]

Revised: Lastly, we empirically validate our interpolation strategy that mixes only support sets.

Revised: We compare our method to various interpolation strategies including one that mixes a support set with a zero mean and unit variance Gaussian noise.

Original: Visualization In Figure 3, we present the t-SNE [43] visualizations of the original and interpolatedtasks with MLTI and Meta-Interpolation, respectively.
Revised: In Table 5, we empirically show that the interpolation strategy which mixes only support sets outperforms the other mixing strategies. Note that interpolating a support set with gaussian noise works well on ESC50 dataset though we find that it significantly degrades the performance of ProtoNet on RMNIST, from 75 . 35 ± 1 .

Revised: 63 to 69 .

Revised: 60 ± 1 .

Revised: 60 as shown in Table 6, which justifies our approach of mixing two support sets.

Revised: Visualization In Figure 3, we present the t-SNE [43] visualizations of the original and interpolated tasks with MLTI and Meta-Interpolation, respectively.

Original: Following Yao et al.
Revised: Following Yao et al.

Original: [50], we sample threeoriginal tasks from NCI and ESC-50 dataset, where each task is a two-way five-shot and five-way fiveshot classification problem, respectively.
Revised: [50], we sample three original tasks from NCI and ESC-50 dataset, where each task is a two-way five-shot and five-way fiveshot classification problem, respectively.

Original: The tasks are interpolated with MLTI or Meta-Interpolationto construct 300 additional tasks and represented as a set of all class prototypes.
Revised: The tasks are interpolated with MLTI or Meta-Interpolation to construct 300 additional tasks and represented as a set of all class prototypes.

Original: To visualize theprototypes, we first perform Principal Component Analysis [13] (PCA) to reduce the dimension ofeach prototype.
Revised: To visualize the prototypes, we first perform Principal Component Analysis [13] (PCA) to reduce the dimension of each prototype.

Original: The first 50 principal components are then used to compute the t-SNE visualizations.
Revised: The first 50 principal components are then used to compute the t-SNE visualizations.

Original: As shown in Figure 3b and 3d, Meta-Interpolation successfully learns an expressive neural set functionthat densifies the task distribution.
Revised: As shown in Figure 3b and 3d, Meta-Interpolation successfully learns an expressive neural set function that densifies the task distribution.

Original: The task augmentations with MLTI, however, do not cover a wideembedding space as shown in Figure 3a and 3c as the mixup strategy allows to generate tasks only onthe simplex defined by the given set of tasks.
Revised: The task augmentations with MLTI, however, do not cover a wide embedding space as shown in Figure 3a and 3c as the mixup strategy allows to generate tasks only on the simplex defined by the given set of tasks.

Revised: Limitation

Revised: Although we have shown promising results in various domains, our method requires extra computation for bilevel optimization to estimate λ , the parameters of the set function φ λ , which makes it challenging to apply our method to gradient based meta-learning methods such as MAML.

Revised: Moreover, our interpolation is limited to classification problem and it is not straightforward to apply it to regression tasks.

Revised: Reducing the computational cost for bilevel optimization and extending our framework to regression will be important for future work.

Original:  Conclusion
Revised:  Conclusion

Original: We proposed a novel domain-agnostic task augmentation method, Meta Interpolation, to tackle themeta-overfitting problem in few-task meta-learning.
Revised: We proposed a novel domain-agnostic task augmentation method, Meta Interpolation, to tackle the meta-overfitting problem in few-task meta-learning.

Original: Specifically, we leveraged expressive neuralset functions to interpolate a given set of tasks and trained the interpolating function using bileveloptimization, so that the meta-learner trained with the augmented tasks generalizes to meta-validationtasks.
Revised: Specifically, we leveraged expressive neural set functions to interpolate a given set of tasks and trained the interpolating function using bilevel optimization, so that the meta-learner trained with the augmented tasks generalizes to meta-validation tasks.

Original: Since the set function is optimized to minimize the loss on the validation tasks, it allows us totailor the task augmentation strategy to each specific domain.
Revised: Since the set function is optimized to minimize the loss on the validation tasks, it allows us to tailor the task augmentation strategy to each specific domain.

Original: We empirically validated the efficacyof our proposed method on various domains, including image classification, chemical propertyprediction, text classification, and automatic speech recognition, showing that Meta-Interpolationachieves consistent improvements across all domains.
Revised: We empirically validated the efficacy of our proposed method on various domains, including image classification, chemical property prediction, text classification, and automatic speech recognition, showing that Meta-Interpolation achieves consistent improvements across all domains.

Original: This is in stark contrast to the baselines whichimprove generalization in certain domains but degenerate performance in others.
Revised: This is in stark contrast to the baselines which improve generalization in certain domains but degenerate performance in others.

Original: Furthermore, ourtheoretical analysis shed light on how Meta-Interpolation regularizes the meta-learner and improvesits generalization performance.
Revised: Furthermore, our theoretical analysis shed light on how Meta-Interpolation regularizes the meta-learner and improves its generalization performance.

Original: Lastly, we discussed the limitation of our method in Appendix C.
Revised: Lastly, we discussed the limitation of our method.

Original: Checklist
Revised: Acknowledgments

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope? [Yes](

Original: b

Original: ) Did you describe the limitations of your work?

Original: [Yes] Please see limitation from

Original: [-Section 
 (c) Did you discuss any potential negative societal impacts of your-] work N
Revised: [-Section 
 (c) Did you discuss any potential negative societal impacts of your-]This work was supported by Institute of Information & communications Technology Planning &

Original: A

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] If you are including theoretical results...
Revised: Evaluation (IITP) grant funded by the Korea government(MSIT)

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?

Original: [Yes]

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?

Original: [Yes]

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they

Original: were chosen)?

Original: [Yes]

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] We run experiments 5 times with different random seeds and show the 95% confidence interval on Table
Revised: (No.2019-0-00075, Artificial Intelligence Graduate School Program(KAIST)), the Engineering Research Center Program through the National Research Foundation of Korea (NRF) funded by the Korean Government MSIT (NRF2018R1A5A1059921), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT)

Original: 1 and 2. (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No]
Revised: (No. 2021-0-02068, Artificial Intelligence

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We describe the

Original: datasets and cite the creators in Section 5.

Original: (b) Did you mention the license of  the assets? [N/A] (c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
Revised: Innovation Hub), the National Research Foundation of Korea (NRF) funded by the Ministry of

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?

Original: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content? [N/A] If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: Education (NRF-2021R1F1A1061655), Institute of Information & communications Technology

Original: applicable? [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: Planning & Evaluation (IITP) grant funded by the Korea government(MSIT)

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
Revised: (No.2022-0-00713), and Samsung Electronics (IO201214-08145-01).

Original: Appendix
Revised: Appendix

Original:  The appendix is organized as follows:
Revised: Organization The appendix is organized as follows:

Original: In Section B, we provide proofs of the theorem and propositionsin Section 4.
Revised: In Section A, we provide proofs of the theorem and propositions in Section 4.

Original: We discucss the limitations of our method in Section C.
Revised: In Section B, we show additional experimental results — Meta-interpolation with first order MAML, the effect of the number of meta-training and metavalidation tasks, ablation study for location of interpolation, and the effect of the number of tasks for interpolation.

Original: In Section D, we providedetailed descriptions of the experimental setup used in the main paper together with the exact datasplits for meta-training, meta-validation and meta-testing for all the datasets used.
Revised: In Section C, we provide detailed descriptions of the experimental setup used in the main paper together with the exact data splits for meta-training, meta-validation and meta-testing for all the datasets used.

Original: Additionally,we specify the exact architecture of the Prototypical Networks used for all the experiments andfurther describe the Set Transformer in detail in Section D.3.
Revised: Finally, we specify the exact architecture of the Prototypical Networks used for all the experiments and further describe the Set Transformer in detail in Section C.3.

Original: All hyperparameters are detailed in Section D.4.
Revised: All hyperparameters are specified in Section C.4.

Original: Finally, in Section E, we explore the effect of the number of tasks for interpolation onthe generalization performance of Meta-Interpolation.

Original: B ProofsB.1 Proof of Theorem1
Revised: A Proofs A.1 Proof of Theorem

Original: Proof.
Revised: Proof.

Original: Define ϕ :=
Revised: Define ϕ :=

Original: ϕ lθ l = f lθ l ◦ · · · ◦ f 1 θ 1 , and g :=
Revised: ϕ lθ l = f lθ l ◦ · · · ◦ f 1 θ 1 , and g :=

Original: f Lθ L ◦ · · · ◦ f l +1 θ l +1 .
Revised: f Lθ L ◦ · · · ◦ f l +1 θ l +1 .

Original: Define the dimensionality asϕ ( x st,i ) ∈ R d , and g ( ϕ ( x st,i ))
Revised: Define the dimensionality as ϕ ( x st,i ) ∈ R d , and g ( ϕ ( x st,i ))

Original: ∈ R D .
Revised: ∈ R D .

Original: From the definition, since we use ˆ f θ,λ in both training andtesting time, we have
Revised: From the definition, since we use ˆ f θ,λ in both training and testing time, we have

Original: [Equation]
Revised: [Equation]

Original: where 
Revised: where c

Original: [Equation]
Revised: [Equation]

Original: In the analysis of the loss functions, without the loss of generality, we can set the permutation σ ton { 1 , . . .
Revised: In the analysis of the loss functions, without the loss of generality, we can set the permutation σ t on { 1 , . . .

Original: , K } to be the identity since the every combination can be realized by one permutation σinstead of the two permutations σ t , σ t ′ .
Revised: , K } to be the identity since the every combination can be realized by one permutation σ instead of the two permutations σ t , σ t ′ .

Original: Therefore, using the definition of the ˆ T t,t ′ , we can write thecorresponding loss by
Revised: Therefore, using the definition of the ˆ T t,t ′ , we can write the corresponding loss by

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: This can be rewritten as:
Revised: This can be rewritten as:

Original: [Equation]
Revised: [Equation]

Original: [Equation] ∈D st 1 { y st,i = k } =   (cid:88) ( x st ′ ,j ,y st ′ ,j ) ∈D st ′ 1 { y st ′ ,i =
Revised: ′ ,j ,y st ′ ,j ) ∈D st ′ 1 { y st ′ ,i =

Original: σ ( k ) }     (cid:88) ( x st,i ,y st,i ) ∈D
Revised: σ ( k ) }     (cid:88) ( x st,i ,y st,i ) ∈D

Original: st 1 { y st,i = k } 
Revised: st 1 { y st,i = k } 

Original:  =
Revised:  =

Original: N t ′ ,k N t,k
Revised: N t ′ ,k N t,k

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original:  .
Revised:  .

Original: Define the set
Revised: Define the set

Original: I
Revised: I

Original: [Equation]
Revised: [Equation]

Revised: k

Revised: [Equation]

Original: = { i : y st,i = k } .
Revised: = { i : y st,i = k } .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Summarizing the computation so far, we have that
Revised: Summarizing the computation so far, we have that

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Revised: L

Original: [Equation]
Revised: [Equation]

Original: ( k ) | I t,k | (cid:88) i ∈
Revised: | I t,k | (cid:88) i ∈

Original: I t,k
Revised: I t,k

Original: ( g ◦ φ λ )( { ϕ ( x st,i ) , ϕ ( x st ′ ,j ) } ) with
Revised: ( g ◦ φ λ )( { ϕ ( x st,i ) , ϕ ( x st ′ ,j ) } ) with

Original: I t,k
Revised: I t,k

Original: =
Revised: =

Original: i : y st,i = k } .
Revised: i : y st,i = k } .

Original: We now analyze the relationship of φ λ ( { h, h ′ } ) and φ λ ( { h } ) .
Revised: We now analyze the relationship of φ λ ( { h, h ′ } ) and φ λ ( { h } ) .

Original: From this definition, we firstcompute φ λ ( { h, h ′ } ) as follows.
Revised: From this definition, we first compute φ λ ( { h, h ′ } ) as follows.

Original: By writing σ 1 = softmax( √ d − 1 Q { h,h ′ }( K { h,h ′ }) ⊤ ) andσ 2 = softmax( √ d − 1 Q 2 ( K { h,h ′ }) ⊤ ) ,
Revised: By writing σ 1 = softmax( √ d − 1 Q { h,h ′ }( K { h,h ′ }) ⊤ ) and σ 2 = softmax( √ d [Equation] Q 2 ( K { h,h ′ }) ⊤ ) ,

Original: [Equation]
Revised: [Equation]

Original: Here, by writing p 1 = p ( t,t ′ ,i,j ), p ′ 1 = ˜ p ( t,t ′ ,i,j ), and p 2 = p ( t,t ′ ,i,j ), we can rewrite that
Revised: Here, by writing p 1 = p ( t,t ′ ,i,j ), p ′ 1 = ˜ p ( t,t ′ ,i,j ), and p 2 = p ( t,t ′ ,i,j ), we can rewrite that

Original: [Equation]
Revised: [Equation]

Revised: −

Original: [Equation]
Revised: [Equation]

Original: ] ∈ R
Revised: [Equation] R

Original: [Equation]
Revised: [Equation]

Original: Thus, by letting ¯ p = p 2 p 1 + (1 − p 2 ) p ′ 1 ∈ [0 , 1] , σ 2 σ 1 = [  p 
Revised: Thus, by letting ¯ p = p 2 p 1 + (1 − p 2 ) p ′ 1 ∈ [0 , 1] , σ 2 σ 1 = [ p1 − p 2 ] (cid:20) p [Equation] 1 − p [Equation]

Revised: p ′

Original: [Equation]
Revised: [Equation]

Original: − p 2 ]
Revised: − p ′

Revised: [Equation]

Revised: p

Original: [Equation]
Revised: [Equation]

Revised: p

Revised: [Equation]

Revised: [Equation]

Revised: p

Revised: [Equation]

Revised: ) p ′

Revised: [Equation]

Revised: p

Revised: [Equation]

Revised: p

Revised: [Equation]

Revised: p

Revised: [Equation]

Revised: p ′

Revised: [Equation]

Revised: ] =

Revised: [¯ p 1 − ¯ p ] .

Original: Using these,
Revised: Using these,

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Thus, by letting W =
Revised: Thus, by letting W =

Original: ( W V 1 W V 2 )
Revised: ( W V 1 W V 2 )

Original: ⊤ ∈ R d × d and b = ( b V 1 W V 2 + b V 2 )
Revised: ⊤ ∈ R d × d and b = ( b V 1 W V 2 + b V 2 )

Original: ⊤ ∈ R d , and by definingα = 1 − ¯ p , we have that
Revised: ⊤ ∈ R d , and by defining α = 1 − ¯ p , we have that

Original: [Equation]
Revised: [Equation]

Original: For φ λ ( { h } ) , similarly, by writing σ
Revised: For φ λ ( { h } ) , similarly, by writing σ

Original: [Equation]
Revised: [Equation]

Original: and
Revised: σ

Original: [Equation]
Revised: [Equation]

Revised: Here, since σ 1 = softmax( √ d − 1 Q { h } 1 ( K { h }) ⊤ ) ∈ R and σ 2 = softmax

Original: [Equation]
Revised: [Equation]

Revised: √ d − 1 Q

Revised: [Equation]

Original: Here, since σ 1 = softmax( √ d − 1 Q { h } 1 ( K { h }) ⊤ ) ∈ R and σ 2 = softmax( √ d − 1 Q 2 ( K { h }) ⊤ ) ∈ R , we have σ 1 = σ 2 = 1 and thus, φ λ ( { h } ) =
Revised: K { h }) ⊤ ) ∈ R , we have σ 1 = σ 2 = 1 and thus, [Equation] λ ( { h [Equation]

Original: Wh + b.
Revised: Wh + b.

Original: Therefore, we have that
Revised: Therefore, we have that

Original: [Equation]
Revised: [Equation]

Original: where W and b does not depend on the ( h, h ′ ) .
Revised: where W and b does not depend on the ( h, h ′ ) .

Original: Using these and by defining h t,i = ϕ
Revised: Using these and by defining h t,i = ϕ

Original: ( x st,i ) , we have that
Revised: ( x st,i ) , we have that

Original: [Equation]
Revised: [Equation]

Original: With these preparation, we are now ready to prove the regularization form.
Revised: With these preparation [Equation] we are now ready to prove the regularization form.

Original: Fix t, t ′ and write α ij =
Revised: Fix t, t ′ and write α ij =

Original: α ( t,t ′ ) ij .
Revised: α ( t,t ′ ) ij .

Original: Define a vector α such that α = ( α ij ) i,j .
Revised: Define a vector α such that α = ( α ij ) i,j .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Then, from the results of the calculations above, we have that
Revised: Then, from the results of the calculations above [Equation] we have that

Original: [Equation]
Revised: [Equation]

Original: Using the assumptions that ∂ r g ( z ) = 

Revised: Using the assumptions that ∂ r g ( z ) =

Original: for all r ≥ 2 , for any J , the J -th approximation of ˆ c k
Revised: for all r ≥ 2 , for any J [Equation] the J -th approximation of ˆ c k

Original: ( α ) isgiven by
Revised: ( α ) is given by

Original: [Equation]
Revised: [Equation]

Original: Let ¯ γ ≥ 1 to be set later and define
Revised: Let ¯ γ ≥ 1 to be set later and define

Original: [Equation]
Revised: [Equation]

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Revised: + ¯ γ ∆ k .

Original: Define c
Revised: Define c

Original: =
Revised: =

Original: ( c 1 , . .
Revised: ( c 1 , . .

Original: . , c K ) and ∆ =
Revised: . , c K ) and ∆ =

Original: (∆ 1 , . . .
Revised: (∆ 1 , . . .

Original: , ∆ K ) .
Revised: , ∆ K ) .

Original: For any given t , define
Revised: For any given t , define

Original: [Equation]
Revised: [Equation]

Original: Then, we have that
Revised: Then, we have that

Original: [Equation]
Revised: [Equation]

Original: The J -th approximation of L mix ( λ, θ, ˆ T t,t ′ ) is given by
Revised: The J -th approximation of L mix ( λ, θ, ˆ T t,t ′ ) is given by

Original: [Equation]
Revised: [Equation]

Original: where φ ( j ) is the j -th order derivative of φ : γ (
Revised: where φ ( j ) is the j -th order derivative of φ : γ (

Original: cid:55)→ L t ( c + γ ∆) .
Revised: cid:55)→ L t ( c + γ ∆) .

Original: Here, for any j ∈ N + , by defining
Revised: Here, for any j ∈ N + , by defining

Original: [Equation]
Revised: [Equation]

Original: KD
Revised: KD

Original: [Equation]
Revised: [Equation]

Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Then, by using the vectorization of the tensor vec[ ∂ j L t ( b ′ )]
Revised: by using the vectorization of the tensor vec[ ∂ j L t ( b ′ )]

Original: ∈ R ( KD ) j , we can rewrite this equationas
Revised: ∈ R ( KD ) j [Equation] we can rewrite this equation as

Original: [Equation]
Revised: [Equation]

Original: where ∆ ⊗ j = ∆ ⊗ ∆ ⊗ · · · ⊗ ∆ ∈ R ( KD ) j .
Revised: where ∆ ⊗ j = ∆ ⊗ ∆ ⊗ · · · ⊗ ∆ ∈ R ( KD ) j .

Original: By combining these with ¯ γ =1 ,
Revised: By combining these with ¯ γ =

Original: L mix ( λ, θ, ˆ T t,t ′ ) J = L singleton ( λ, θ ; T t ) +
Revised: [Equation]

Original: J (cid:88)
Revised: where ∆

Original: j =j ! vec[ ∂ j L t ( c )]
Revised: [Equation] =

Original: ⊤ ∆ ⊗ j , where ∆ k = | I t ′ ,σ ( k ) | (cid:88) j ∈ I t ′ ,σ ( k ) | I t,k | (cid:88) i ∈ I t,k α ( t,t ′ ) ij ∂g ( Wh t,i + b ) W ( h t ′ ,j − h t,i ) .
Revised: [Equation]

Original: B.2 Proof of Proposition
Revised: A.2 Proof of Proposition

Original: Proof.
Revised: Proof.

Original: We now apply our general regularization form theorem to this special case from the previous paper [50] on the ProtoNet loss.
Revised: We now apply our general regularization form theorem to this special case from the previous paper [50] on the ProtoNet loss.

Original: In this special case, we have that ϕ ( x ) = x , W
Revised: In this special case, we have that ϕ ( x ) = x , W

Original: =
Revised: =

Original: I, b = 0 , h t,i = x st,i , g ( x ) =
Revised: I, b = 0 , h t,i = x st,i , g ( x ) =

Original: x ⊤ θ, and
Revised: x ⊤ θ, and

Original: L [Equation] n n (cid:88) i =  [Equation]
Revised: L t ( c ) = 1 n n (cid:88) i =  1 + exp( ⟨ x qt,i , θ ⟩ − c 1 / 2 − c 2 / 2) .

Original: Here, for c = ( c 1 , c 2 ) with [Equation] =[Equation]
Revised: Here, for c = ( c 1 , c 2 ) with c 1 =

Original:  t,(cid:88)(
Revised: N t,(cid:88)

Revised: ( x st,i ,y st,i ) ∈D

Revised: st 1 { y st

Original: [Equation]
Revised: [Equation]

Original: (10) andc
Revised: i =1 } φ λ ( { x st

Original: [Equation]
Revised: [Equation]

Original:  = 
Revised: i } ) ⊤ θ = θ ⊤ c ′(10) and c 2 = N t,(cid:88) ( x st,i ,y st,i ) ∈D

Original: N t
Revised: st 1 { y st

Original: [Equation]
Revised: [Equation]

Original: (cid:88) ( x
Revised: i =2 }

Original: [Equation]
Revised: [Equation]

Original:  (11) we recover that 
Revised: ( { x [Equation] } ) ⊤ θ = θ ⊤ c ′ 2 , (11) we recover that L

Original: [Equation]
Revised: [Equation]

Original: Thus, to instantiate our general theorem to this special case, we only need to compute the ∂ j L t ( c ) and ∂g ( h ⊤ t,i W + b ) up to the second order approximation.
Revised: Thus, to instantiate our general theorem to this special case, we only need to compute the ∂ j L t ( c ) and ∂g ( h ⊤ t,i W + b ) up to the second order approximation.

Original: [Equation]
Revised: [Equation]

Original: Define z t,i := ⟨ x qt,i − ( c ′ 1 + c ′ 2 ) / 2 , θ ⟩ .
Revised: Define z t,i := ⟨ x qt,i − ( c ′ 1 + c ′ 2 ) / 2 , θ ⟩ .

Original: Since c 1 = θ ⊤ c ′ 1 and c 2 = θ ⊤ c ′ 2 by Equation 10 and 11,
Revised: Since c 1 = θ ⊤ c ′ 1 and c 2 = θ ⊤ c ′ 2 by Equation 10 and 11,

Original: [Equation]
Revised: [Equation]

Original: Thus, we have L t ( c )
Revised: Thus, we have L t ( c )

Original: = 1 n (cid:80) ni = [Equation] +exp( z t,i ) .
Revised: = 1 n (cid:80) ni = 1 1+exp( z t,i ) .

Original: Then, ∂L
Revised: Then, ∂L

Original: [Equation]
Revised: [Equation]

Original: ( c )
Revised: ( c )

Original: [Equation]
Revised: [Equation]

Original: Similarly, ∂L
Revised: Similarly, ∂L

Original: [Equation]
Revised: [Equation]

Original: For the second order, by defining the logistic function ψ ( z 
Revised: For the second order, by defining the logistic function ψ ( z t,

Original: [Equation]
Revised: [Equation]

Original: ∂L [Equation] 1 +
Revised: Similarly,

Original: [Equation]
Revised: [Equation]

Original: Similarly, ∂L
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: Therefore, the second approximation of E t ′ ,σ [ L mix ( λ, θ, ˆ T t,t ′ )] is
Revised: the second approximation of E t ′ ,σ [ L mix ( λ, θ, ˆ T t,t ′ )] is

Original: L singleton ( λ, θ ; T t ) + E t ′ ,σ   2 (cid:88) 
Revised: L singleton ( λ, θ ; T t ) + E t ′ ,σ   2 (cid:88) j =

Original: [Equation]
Revised: [Equation]

Original: where ∆ =
Revised: where ∆ =

Original: [∆ ⊤ 1 , ∆ ⊤ 2 ] ⊤ with
Revised: [∆ ⊤ [Equation] ∆ ⊤ 2 ] ⊤ with

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: For the first order term,
Revised: For the first order term,

Original: [Equation]
Revised: [Equation]

Original: where δ t,
Revised: where δ t,t

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Revised: =

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: For the second order term,
Revised: For the second order term,

Original: [Equation]
Revised: [Equation]

Original: Since E t ′ ,σ [ δ t,t ′ ,σ ] = 0 with the α being balanced, the second approximation of E t ′ ,σ [ L mix ( λ, θ, ˆ T t,t ′ )]becomes:
Revised: Since E t ′ ,σ [ δ t,t ′ ,σ ] = 0 with the α being balanced

Original: [Equation]
Revised: [Equation]

Original: t,i ) (cid:33) θ ⊤ E t ′ ,σ (cid:2) δ t,t ′ ,σ δ ⊤ t,t ′ ,σ (cid:3) θ.
Revised: the second approximation of E t ′ ,σ [ L mix ( λ, θ, ˆ T t,t ′ )] becomes: [Equation]

Original: B.3 Proof of Proposition
Revised: A.3 Proof of Proposition

Original: Proof.
Revised: Proof.

Original: Let ξ 1 , . . .
Revised: Let ξ 1 , . . .

Original: , ξ n be independent uniform random variables taking values in {− 1 , 1 } ; i.e.,
Revised: , ξ n be independent uniform random variables taking values in {− 1 , 1 } ; i.e.,

Original: Rademacher variables.
Revised: Rademacher variables.

Original: We first bound the empirical Rademacher complexity  as follows:
Revised: We first bound the empirical Rademacher complexity part as follows:

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]

Original: where Σ † denotes the Moore–Penrose inverse of Σ .
Revised: where Σ † denotes the Moore–Penrose inverse of Σ .

Original: Using this bound on the empirical Rademachercomplexity, we now bound the Rademacher complexity as follows:
Revised: By taking expectation and using this bound on the empirical Rademacher complexity, we now bound the Rademacher complexity as follows:

Original: [Equation]
Revised: [Equation]

Original:  n (cid:88) i =
Revised: x 1 ,..., x n ˆ R n ( F R ) ≤ E x 1 ,..., x n √ R n (cid:118)(cid:117)(cid:117)(cid:116) n (cid:88) i =

Original: [Equation] [Equation] x 
Revised: E x ′ ( x i − x ′ )

Original: [Equation]
Revised: ⊤ Σ † ( x i − x ′ ) ≤ √ R n (cid:118)(cid:117)(cid:117)(cid:116) n (cid:88) i = E x i , x ′ ( x i − x ′ )

Original:  ′ ) ≤ [Equation]
Revised: ⊤ Σ † ( x i − x ′ ) =

Original:  n (cid:118)(cid:117)(cid:117)(cid:116) n (cid:88) [Equation]
Revised: √ R n (cid:118)(cid:117)(cid:117)(cid:116) n (cid:88) i = (cid:88) k,l (Σ † )

Original:  x [Equation] [Equation]
Revised: kl E x i , x ′ ( x i − x ′ )

Original: n (cid:88) i 
Revised: k ( x i − x ′ )

Original: [Equation]
Revised: l = √ R n (cid:118)(cid:117)(cid:117)(cid:116) n (cid:88) i = (cid:88) k,l (Σ † ) kl (Σ) kl = √ R n (cid:118)(cid:117)(cid:117)(cid:116) n (cid:88)

Original: [Equation] (cid:118)(cid:117)(cid:117)(cid:116) n (cid:88) 
Revised: i = tr(ΣΣ † ) = √ R n (cid:118)(cid:117)(cid:117)(cid:116) n (cid:88) i =

Original: i
Revised: rank(Σ) =

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:112) rank

Original: [Equation]
Revised: [Equation]

Original: (cid:118)(cid:117)(cid:117)(cid:116)
Revised: Σ) √ n

Original: n (cid:88)
Revised: As stated in the introduction, we focus solely on metric based meta-learning due to their efficiency and better empirical performance over the gradient based methods on the few-task meta-learning problem. Moreover it is challenging to combine our method with Model-Agnostic Meta-Learning (MAML) [10] since it yields a tri-level optimization problem which requires differentiating through second order derivatives. Furthermore, tri-level optimization is still known to be a challenging problem [4, 7] and currently an active line of research.

Original: i
Revised: Thus, instead of using the original MAML, we perform additional experiments on the ESC-50 and Metabolism dataset with first-order MAML (FOMAML) [Table] [Table] [Table]

Revised: which approximates the Hessian with a zero matrix.

Revised: As shown in Table

Original: [Equation]
Revised: [Table]

Revised: the experimental results show that Meta-Interpolation with first-order MAML outperforms MLTI, which again confirms the general effectiveness of our set-based task augmentation scheme.

Revised: However, it largely underperforms our original Meta-Interpolation framework with metric-based meta-learning.

Revised: B.2 Effect of the number of meta-training and validation tasks

Revised: We analyze the effect of the number of meta-training tasks on ESC-50 dataset.

Original: n
Revised: As shown in Table 8, Meta-Interpolation consistently outperforms the baselines by large margins, regardless of the number of the tasks. Furthermore, we report the test accuracy as a function of the number of meta-validation tasks in Table 9. Although the generalization performance of Meta-Interpolation slightly decreases as we reduce the number of meta-validation tasks, it still outperforms the relevant baselines by a large margin.

Original: C Limitation
Revised: B.3 Location of interpolation

Revised: [Table] Contrary to Manifold Mixup, we fix the layer of interpolation as shown in Table 17. Otherwise, we cannot use the same architecture of Set Transformer to interpolate the output of different layers since the hidden dimension of each layer is different.

Revised: Moreover, we report the test accuracy by changing the layer of interpolation.

Revised: Interpolating hidden representation of support sets from layer 2, which is the model used in the main experiments on ESC dataset, achieves the best performance.

Original: Although we have shown promising results in various domains, our method requires extra computation for bilevel optimization to estimate λ , the parameters of the set function φ λ .
Revised: [Figure] Since the set function φ λ can handle sets of arbitrary cardinality n ∈ N , we plot the test accuracy on the ESC-50 dataset with varying set sizes from one to five. As shown in Figure 4, for sets of cardinality one, where we do not perform any interpolation, we observe significant degradation in performance.

Original: Reducing thecomputational cost for bilevel optimization will be important for future work.
Revised: This suggests that the interpolation of instances is crucial for better generalization. On the other hand, for set sizes from two to five, the gain is marginal with increasing cardinality. Furthermore, increasing the set size introduces extra computational overhead. Thus, we set the cardinality to two for task interpolation in all the experiments.

Original: D Experimental SetupD.1 Dataset Description
Revised: C Experimental Setup C.1 Dataset Description

Original: Metabolism We use the following split for the metabolism dataset:
Revised: Metabolism We use the following split for the metabolism dataset:

Original: Meta-Train = { CYP1A2_Veith , CYP3A4_Veith , CYP2C9_Substrate_CarbonMangels } Meta-Validation = { CYP2D6_Veith , CYP2D6_Substrate_CarbonMangels } 
Revised: Meta-Train = { CYP1A2_Veith , CYP3A4_Veith , CYP2C9_Substrate_CarbonMangels } Meta-Validation = { CYP2D6_Veith , CYP2D6_Substrate_CarbonMangels } Meta-Test =

Original: Meta-Test = { CYP2C19_Veith , CYP2C9_Veith , CYP3A4_Substrate_CarbonMangels }
Revised: { CYP2C19_Veith CYP2C Veith CYP3A

Original:  Following Yao et al.
Revised: Substrate _CarbonMangels Following Yao et al.

Original: [50], we balance each subdataset by selecting 1000 samples from each subdataset.
Revised: [50], we balance each subdataset by selecting 1000 samples from each subdataset.

Original: Each data sample is processed by extracting a 1024-bit fingerprint feature from the SMILESrepresentation of each chemical compound using the RDKit [15] library.
Revised: Each data sample is processed by extracting a 1024-bit fingerprint feature from the SMILES representation of each chemical compound using the RDKit [15] library.

Original: Tox

Revised: Tox

Original: We use the following split for the metabolism dataset 1 :
Revised: We use the following split for the metabolism dataset 2 :

Original: Meta-Train = { NR-AR , NR-AR-LBD , NR-AhR , NR-Aromatase , NR-ER , NR-ER-LBD } Meta-Validation = { NR-PPAR-gamma , SR-ARE } Meta-Test
Revised: Meta-Train = { NR-AR , NR-AR-LBD , NR-AhR , NR-Aromatase , NR-ER , NR-ER-LBD } Meta-Validation = { NR-PPAR-gamma , SR-ARE } Meta-Test

Original: =
Revised: =

Original: { SR-ATAD5 , SR-HSE , SR-MMP , SR-p53 }
Revised: { SR-ATAD5 , SR-HSE , SR-MMP , SR-p53 }

Original: NCI We download the dataset from the github repository 2 and use the following splits for metatraining, meta-validation, and meta-testing:
Revised: NCI We download the dataset from the github repository 3 and use the following splits for metatraining, meta-validation, and meta-testing:

Original: Meta-Train = { 41 , 47 , 83 ,  Meta-Validation = { 81 , 145 } Meta-Test = { 1 , 33 , 123 }
Revised: Meta-Train = { 41 , 47 , 83 , 109 } Meta-Validation = { 81 , 145 } Meta-Test = { 1 , 33 , 123 }

Original: GLUE-SciTail We use Hugging Face Datasets library [25] to download MNLI, QNLI, SNLI, RTE,and SciTail datasets and tokenize it ELECTRA tokenizer with setting maximum length to 128.
Revised: GLUE-SciTail We use Hugging Face Datasets library [25] to download MNLI, QNLI, SNLI, RTE, and SciTail datasets and tokenize it ELECTRA tokenizer with setting maximum length to 128.

Original: Welist meta-train, meta-validation, and meta-test split as follows:
Revised: We list meta-train, meta-validation, and meta-test split as follows:

Original: Meta-Train = { MNLI , QNLI } Meta-Validation
Revised: Meta-Train = { MNLI , QNLI } Meta-Validation

Original: =
Revised: =

Original: SNLI , RTE } Meta-Test = { SciTail } ESC-50 We download ESC-50 dataset [34] from the github repository
Revised: SNLI , RTE } Meta-Test = { SciTail } ESC 50 We download ESC

Original:  and use the meta-training,meta-validation, and meta-test split as follows:
Revised: 50 dataset [34] from the github repository 4 and use the meta-training, meta-validation, and meta-test split as follows:

Original: Meta-train set:dog , rooster , pig , cow , frog , cat , hen , insects , sheep , crow , rain , sea_waves , crackling_fire , crickets , chirping_birds , water_drops , wind , pouring_water , toilet_flush , thunderstorm Meta-validation set:
Revised: Meta-train set: dog , rooster , pig , cow , frog , cat , hen , insects , sheep , crow , rain , sea_waves , crackling_fire , crickets , chirping_birds , water_drops , wind , pouring_water , toilet_flush , thunderstorm Meta-validation set:

Original: crying_baby , sneezing , clapping , breathing , coughing , footsteps , laughing , brushing_teeth , snoring , drinking_sipping , door_wood_knock , mouse_click , keyboard_typing , door_wood_creaks , can_opening Meta-test set:
Revised: crying_baby , sneezing , clapping , breathing , coughing , footsteps , laughing , brushing_teeth , snoring , drinking_sipping , door_wood_knock , mouse_click , keyboard_typing , door_wood_creaks , can_opening Meta-test set:

Original: washing_machine , vacuum_cleaner , clock_alarm , clock_tick , glass_breaking , helicopter , chainsaw , siren , car_horn , engine , train , church_bells , airplane , fireworks , hand_saw
Revised: washing_machine , vacuum_cleaner , clock_alarm , clock_tick , glass_breaking , helicopter , chainsaw , siren , car_horn , engine , train , church_bells , airplane , fireworks , hand_saw

Original: RMNIST Following Yao et al.
Revised: RMNIST Following Yao et al.

Original: [50], we create RMNIST dataset by changing the size, color, andangle of the images from the original MNIST dataset.
Revised: [50], we create RMNIST dataset by changing the size, color, and angle of the images from the original MNIST dataset.

Original: To be specific, we merge training and test splitof MNIST and randomly select 5,600 images for each class and split into 56 subdatasets where eachclass has 100 examples.
Revised: To be specific, we merge training and test split of MNIST and randomly select 5,600 images for each class and split into 56 subdatasets where each class has 100 examples.

Original: We only choose 16/6/10 subdatasets for meta-train, meta-validation, andmeta-test split, respectively and do not use the rest of them.
Revised: We only choose 16/6/10 subdatasets for meta-train, meta-validation, and meta-test split, respectively and do not use the rest of them.

Original: Each subdatset with the correspondingcomposition of image transformations is considered a distinct task.
Revised: Each subdatset with the corresponding composition of image transformations is considered a distinct task.

Original: Following are the specific splits.
Revised: Following are the specific splits.

Original: Meta-train set:( red , full , 90 ◦ ) , ( indigo , full , 0 ◦ ) , ( blue , full , 270 ◦ ) , ( orange , half , 270 ◦ ) , ( green , full , 90 ◦ ) , ( green , full , 270 ◦ ) , ( orange , full , 180 ◦ ) , ( red , full , 180 ◦ ) , ( green , full , 0 ◦ ) , ( orange , full , 0 ◦ ) , ( violet , full , 270 ◦ ) , ( orange , half , 90 ◦ ) , ( violet , half , 180 ◦ ) , ( orange , full , 90 ◦ ) , ( violet , full , 180 ◦ ) , ( blue , full , 90 ◦ ) 
Revised: Meta-train set: ( red , full , 90 ◦ ) , ( indigo , full , 0 ◦ ) , ( blue , full , 270 ◦ ) , ( orange , half , 270 ◦ ) , ( green , full , 90 ◦ ) , ( green , full , 270 ◦ ) , ( orange , full , 180 ◦ ) , ( red , full , 180 ◦ ) , ( green , full , 0 ◦ ) , ( orange , full , 0 ◦ ) , ( violet , full , 270 ◦ ) , ( orange , half , 90 ◦ ) , ( violet , half , 180 ◦ ) , ( orange , full , 90 ◦ ) , ( violet , full , 180 ◦ ) , ( blue , full , 90 ◦ ) Meta- validation

Original: Meta-validation set:( indigo , half , 270 ◦ ) , ( blue , full , 0 ◦ ) , ( yellow , half , 180 ◦ ) , ( yellow , half , 0 ◦ ) , ( yellow , half , 90 ◦ ) , ( violet , half , 0 ◦ ) 
Revised: set : ( indigo , half , 270 ◦ ) , ( blue , full , 0 ◦ ) , ( yellow , half , 180 ◦ ) , ( yellow , half , 0 ◦ ) , ( yellow , half , 90 ◦ ) , ( violet , half , 0 ◦ ) Meta-test set : ( yellow ,

Revised: full

Original: Meta-test set:( yellow , full , 270 ◦ ), ( red, full, 0 ◦ ) , ( blue , half , 270 ◦ ) , ( blue , half , 0 ◦ ), ( blue , half , 180 ◦ ) , ( red , half , 270 ◦ ) , ( violet , full , 90 ◦ ) , ( blue , half , 90 ◦ ) , ( green , half , 270 ◦ ) , ( red , half , 90 ◦ )
Revised: 270 ◦ ) ( red full 0 ◦ )

Revised: blue

Revised: half

Revised: , 270 ◦ ) , (

Revised: blue

Revised: , half , 0 ◦ ) , (

Revised: blue

Revised: , half , 180 ◦ ) , (

Revised: red

Revised: half

Revised: ◦ )

Revised: violet

Revised: full

Revised: , 90 ◦ ) , (

Revised: blue

Revised: , half , 90 ◦ ) , ( green , half , 270 ◦ ) , ( red , half , 90 ◦ )

Original: Mini-ImageNet-S
Revised: Mini-ImageNet-S

Original: Following Yao et al.
Revised: Following Yao et al.

Original: [50], we reduce the number of meta-training tasks bychoosing subset of the original meta-training classes.
Revised: [50], we reduce the number of meta-training tasks by choosing subset of the original meta-training classes.

Original: We specify the classes used for meta-trainingtasks as follows:
Revised: We specify the classes used for meta-training tasks as follows:

Original: n03017168 , n07697537 , n02108915 , n02113712 , n02120079 , n04509417 , n02089867 , n03888605 , n04258138 , n03347037 , n02606052 , n
Revised: n03017168 , n07697537 , n02108915 , n02113712 , n02120079 , n04509417 , n02089867 , n03888605 , n04258138 , n03347037 , n02606052 , n

Original: For meta-validation and meta-testing classes, we use the same classes as the original Mini-ImageNet.
Revised: For meta-validation and meta-testing classes, we use the same classes as the original Mini-ImageNet.

Original: CIFAR-100-FS Similar to Mini-ImageNet-S, we choose 12/16/20 classes for meta-training, metavalidation, meta-test classes.
Revised: CIFAR-100-FS Similar to Mini-ImageNet-S, we choose 12/16/20 classes for meta-training, metavalidation, meta-test classes.

Original: Followings are the specific classes for each split:
Revised: Followings are the specific classes for each split:

Original: Meta-Train = [Equation]
Revised: Meta-Train = { 0 , . .

Original: [Equation] Meta-Validation = [Equation]
Revised: , 11 } Meta-Validation = { 64 , . .

Original: [Equation] Meta- Test = { 80 , . . 
Revised: , 79 } Meta-Test = { 80 , . . .

Original: D.2 Prototypical Network Architecture
Revised: C.2 Prototypical Network Architecture

Original: We summarize neural network architectures for ProtoNet on each datasets in Table 4, 5, 6, 7, 8,and 9.
Revised: We summarize neural network architectures for ProtoNet on each datasets in Table 11, 12, 13, 14, 15, and 16.

Original: For Glue-SciTail, we use pretrained ELECTRA-small [8] which we download from Hugging
Revised: For Glue-SciTail, we use pretrained ELECTRA-small [8] which we download from Hugging

Original: Face [48].
Revised: Face [48].

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: D. 
Revised: C.

Original: [Table]
Revised: [Table]

Original: Transformer
Revised: Transformer

Original: We describe Set Transformer [23], φ λ , in more detail.
Revised: We describe Set Transformer [23], φ λ , in more detail.

Original: Let X ∈ R n × d be a stack of n ddimensional row vectors.
Revised: Let X ∈ R n × d be a stack of n ddimensional row vectors.

Original: Let W Q 1 ,j , W K 1 ,j , W V 1 ,j ∈ R d × d k be weight matrices for self-attentionand let b Q 1 ,j , b K 1 ,j , b V 1 ,
Revised: Let W Q 1 ,j , W K 1 ,j , W V 1 ,j ∈ R d × d k be weight matrices for self-attention and let b Q 1 ,j , b K 1 ,j , b V 1 ,

Original: j ∈ R d
Revised: j ∈ R d

Original: k be bias vectors for j = 1 , . .
Revised: k be bias vectors for j = 1 , . .

Original: For encoding an input X , we compute self-attention as follows:
Revised: For encoding an input X , we compute self-attention as follows:

Original: [Equation]
Revised: [Equation]

Original: where 1 = (1 , . . . , 1) ⊤ ∈ R n is a vector of ones, d h = 4 d k , and softmax is applied for each row.
Revised: where 1 = (1 , . . . , 1) ⊤ ∈ R n is a vector of ones, d h = 4 d k , and softmax is applied for each row.

Original: After self-attention, we add a skip connection with layer normalization [1] as follows:
Revised: After self-attention [Equation] we add a skip connection with layer normalization [1] as follows:

Original: [Equation]
Revised: [Equation]

Original: where W 1 ∈ R d h × d h , b 1 ∈ R d h .
Revised: where W 1 ∈ R d h × d h , b 1 ∈ R d h .

Original: Similarly, we compute another self-attention on top of g 1 ( X ) with
Revised: Similarly [Equation] we compute another self-attention on top of g 1 ( X ) with

Original: W Q 2 ,j , W K 2 ,j , W V 2 ,j ∈ R d k × d k weight matrices for self-attention and let b Q 2 ,j , b K 2 ,j , b V 2 ,
Revised: W Q 2 ,j , W K 2 ,j , W V 2 ,j ∈ R d k × d k weight matrices for self-attention and let b Q 2 ,j , b K 2 ,j , b V 2 ,

Original: j ∈ R d k be biasvectors, for j = 1 . ..
Revised: j ∈ R d k be bias vectors, for j = 1 . .

Original: [Equation]
Revised: [Equation]

Original: After self-attention, we also add a skip connection after the second self-attention with dropout [41].
Revised: After self-attention, we also add a skip connection after the second self-attention with dropout [41].

Original: [Equation]
Revised: [Equation]

Original: where W 2 ∈ R d h × d h , b 2 ∈ R d h .
Revised: where W 2 ∈ R d h × d h , b 2 ∈ R d h .

Original: After encoding X with two-layers of self-attention, we perform pooling with attention.
Revised: After encoding X with two-layers of self-attention, we perform pooling with attention.

Original: Let S ∈ R 1 × d hbe learnable parameters and W Q 3 ,j , W K 3 ,j , W V 3 ,j ∈ R dh × d
Revised: Let S ∈ R 1 × d h be learnable parameters and W Q 3 ,j , W K 3 ,j , W V 3 ,j ∈ R d

Original:  k be weight matrices for the pooling-attentionand let b Q 3 ,j , b K 3 ,j , b V 3 ,
Revised: h × d k be weight matrices for the pooling-attention and let b Q 3 ,j , b K 3 ,j , b V 3 ,

Original: j ∈ R d k be bias vectors, for j = 1 , . .
Revised: j ∈ R d k be bias vectors, for j = 1 , . .

Original: We pool ( g 2 ◦ g 1 )( X ) as follows:
Revised: We pool ( g 2 ◦ g 1 )( X ) as follows:

Original: [Equation]
Revised: [Equation]

Original: After pooling, we add another skip connection with dropout as follows:
Revised: After pooling, we add another skip connection with dropout as follows:

Original: [Equation]
Revised: [Equation]

Original: where W 3 ∈ R d h × d h , b 3 ∈ R d h .
Revised: where W 3 ∈ R d h × d h , b 3 ∈ R d h .

Original: Finally, we perform affine-transformation after the pooling asfollows:
Revised: Finally, we perform affine-transformation after the pooling as follows:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where W 4 ∈ R d h × d , b 4 ∈ R d .
Revised: where W 4 ∈ R d h × d , b 4 ∈ R d .

Original: To summarize, Set Transformer is the set function φ λ :=
Revised: To summarize, Set Transformer is the set function φ λ :=

Original: g 4 ◦ g 3 ◦ g 2 ◦ g 1 : R n × d → R d that canhandle a set with arbitrary cardinality n ∈ N .
Revised: g 4 ◦ g 3 ◦ g 2 ◦ g 1 : R n × d → R d that can handle a set with arbitrary cardinality n ∈ N .

Original: D.4 Hyperparameters
Revised: C.4 Hyperparameters

Original: In Table 10 and 11, we summarize all the hyperparameters for each datasets, where MI stands for

Original: Mini-ImageNet-S. For CIFAR-100-FS, we use the same hyperparameters for both 1-hot and 5-shot.

Original: [Table] [Table] [Table] [Table]
Revised: In Table 17 and

Original: E Effect of Cardinality for Interpolation

Original: [Figure]

Original: of

Original: [Figure]

Original: [Figure]

Original: Since the set function φ λ can handle sets of arbitrary cardinality n ∈ N , we plot the test accuracy on the ESC-50dataset with varying set sizes from one to five.

Original: As shownin Figure 4, for sets of cardinality one, where we do notperform any interpolation, we observe significant degradation in performance.

Original: This suggests that the interpolation ofinstances is crucial for better generalization.

Original: On the otherhand, for set sizes from two to five, the gain is marginalwith increasing cardinality.

Original: Furthermore, increasing the setsize introduces extra computational overhead.

Original: Thus, weset the cardinality to two for task interpolation in all theexperiments.

Original: [Table]

Original: [Table]

Original: F Additional ExperimentsF.1 First-order MAML

Original: [Table]
Revised: [Table]

Original: As stated in introduction, we solely focus on metric basedmeta-learning due to the efficiency and better empirical performance over the gradient based methods.

Original: Moreover it ischallenging to combine our method with Model-Agnostic

Original: Meta-Learning (MAML) [10] since it yields a tri-level optimization problem which requires to differentiate throughsecond order derivatives and tri-level optimizati is still knownto be challengin problem [4, 7].

Original: Thus, instead of using theoriginal MAML, we have performed additional experiments on ESC50 dataset with first-order MAML(FOMAML) which approximates the Hessian with a zero matrix.

Original: As shown in Table 14, the experimental results show that our Meta-Interpolation with the first-order MAML outperforms MLTI,which again confirms the general effectiveness of our set-based augmentation scheme. However, itlargely underperforms our original Meta-Interpolation framework with metric-based meta-learning. F.2 Effect of the number of meta-training and validation tasks We analyze the effect of the number of meta-training tasks on ESC50 dataset.
Revised: we summarize all the hyperparameters for each datasets, where MI stands for

Original: As shown in Table 12,

Original: Meta-Interpolation consistently outperforms the baselines by large margins, regardless of the numberof the tasks. Furthermore, we report the test accuracy as a function of the number of meta-validationtasks in Table 13. Although the generalization performance of Meta-Interpolation slightly decreasesas we reduce the number of meta-validation tasks, it still outperforms the relevant baselines by a largemargin. F.3 Location of interpolation
Revised: Mini-ImageNet-S. For CIFAR-100-FS, we use the same hyperparameters for both 1-hot and 5-shot.

Original: [Table]
Revised: [Table]

Original: Contrary to Manifold Mixup, we fix the location of interpolationas shown in Table 10.

Original: Otherwise we cannot use the same architecture of Set Transformer to interpolate output of different layerssince hidden dimension of each layer is different.

Original: Moreover, wereport the test accuracy for each layer to be interpolated.

Original: Interpolating hidden representation of support sets from layer 2, whichis our model used in the main experiments on ESC50 dataset,achieves the best performance.

Original: F.4 Ablation for set functions

Original: [Table]
Revised: [Table]

Original: We have performed additional experiment on ESC50 datasetwith different set function, DeepSets [52] to show the general effectiveness of our method regardless of the set encoding scheme.

Original: In Table 16, Meta-Interpolation with DeepSets improves thegeneralization performance of ProtoTypical Network and themodel with Set Transformer further boost the performance asa consequence of higher-order pairwise interactions among theset elements via the attention mechanism.

Original: F.5 Ablation for interpolation strategy

Original: [Table]

Original: [Table]

Original: We empirically verify our interpolation strategy that mixes onlysupport sets.

Original: We compare our method to various interpolationstrategy including the one mixing a support set with zero meanand unit variance gaussian noise.

Original: In Table 17, we empiricallyshow that the interpolation strategy which mixes only supportsets outperforms the others.

Original: Note that interpolating a supportset with gaussian noise works well on ESC50 dataset though wefind that it significantly degrades the performance of Prototypical Network on RMNIST, from 75 .

Original: 35 ± 1 .

Original: 63 to 69 .

Original: 60 ± 1 . 60 ,which shows a necessity of interpolating two support sets.