Original: Set-based Meta-Interpolation for Few-Task Meta-Learning
Revised: Set-based Meta-Interpolation for Few-Task Meta-Learning

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Meta-learning approaches enable machine learning systems to adapt to new tasksgiven few examples by leveraging knowledge from related tasks.
Revised: Meta-learning approaches enable machine learning systems to adapt to new tasksgiven few examples by leveraging knowledge from related tasks.

Original: However, a largenumber of meta-training tasks are still required for generalization to unseen tasksduring meta-testing, which introduces a critical bottleneck for real-world problemsthat come with only few tasks, due to various reasons including the difficultyand cost of constructing tasks.
Revised: However, a largenumber of meta-training tasks are still required for generalization to unseen tasksduring meta-testing, which introduces a critical bottleneck for real-world problemsthat come with only few tasks, due to various reasons including the difficultyand cost of constructing tasks.

Original: Recently, several task augmentation methods have been proposed to tackle this issue using domain-specific knowledge to designaugmentation techniques to densify the meta-training task distribution.
Revised: Recently, several task augmentation methods have been proposed to tackle this issue using domain-specific knowledge to designaugmentation techniques to densify the meta-training task distribution.

Original: However,such reliance on domain-specific knowledge renders these methods inapplicableto other domains.
Revised: However,such reliance on domain-specific knowledge renders these methods inapplicableto other domains.

Original: While Manifold Mixup based task augmentation methods aredomain-agnostic, we empirically find them ineffective on non-image domains.
Revised: While Manifold Mixup based task augmentation methods aredomain-agnostic, we empirically find them ineffective on non-image domains.

Original: Totackle these limitations, we propose a novel domain-agnostic task augmentationmethod, Meta-Interpolation, which utilizes expressive neural set functions to densify the meta-training task distribution using bilevel optimization.
Revised: Totackle these limitations, we propose a novel domain-agnostic task augmentationmethod, Meta-Interpolation, which utilizes expressive neural set functions to densify the meta-training task distribution using bilevel optimization.

Original: We empiricallyvalidate the efficacy of Meta-Interpolation on eight datasets spanning across variousdomains such as image classification, molecule property prediction, text classification and speech recognition.
Revised: We empiricallyvalidate the efficacy of Meta-Interpolation on eight datasets spanning across variousdomains such as image classification, molecule property prediction, text classification and speech recognition.

Original: Experimentally, we show that Meta-Interpolationconsistently outperforms all the relevant baselines.
Revised: Experimentally, we show that Meta-Interpolationconsistently outperforms all the relevant baselines.

Original: Theoretically, we prove thattask interpolation with the set function regularizes the meta-learner to improvegeneralization.
Revised: Theoretically, we prove thattask interpolation with the set function regularizes the meta-learner to improvegeneralization.

Original: We provide our source code in the supplementary material.

Original:  Introduction
Revised:  Introduction

Original: The ability to learn a new task given only a few examples is crucial for artificial intelligence.
Revised: The ability to learn a new task given only a few examples is crucial for artificial intelligence.

Original: Recently,meta-learning [37, 3] has emerged as a viable method to achieve this objective and enables machinelearning systems to quickly adapt to a new task by leveraging knowledge from other related tasks seenduring meta-training.
Revised: Recently,meta-learning [39, 3] has emerged as a viable method to achieve this objective and enables machinelearning systems to quickly adapt to a new task by leveraging knowledge from other related tasks seenduring meta-training.

Original: Although existing meta-learning methods can efficiently adapt to new taskswith few data samples, a large dataset of meta-training tasks is still required to learn meta-knowledgethat can be transferred to unseen tasks.
Revised: Although existing meta-learning methods can efficiently adapt to new taskswith few data samples, a large dataset of meta-training tasks is still required to learn meta-knowledgethat can be transferred to unseen tasks.

Original: For many real-world applications, such extensive collectionsof meta-training tasks may be unavailable.
Revised: For many real-world applications, such extensive collectionsof meta-training tasks may be unavailable.

Original: Such scenarios give rise to the few-task meta-learningproblem where a meta-learner can easily memorize the meta-training tasks but fail to generalizewell to unseen tasks.
Revised: Such scenarios give rise to the few-task meta-learningproblem where a meta-learner can easily memorize the meta-training tasks but fail to generalizewell to unseen tasks.

Original: The few-task meta-learning problem usually results from the difficulty in taskgeneration and data collection.
Revised: The few-task meta-learning problem usually results from the difficulty in taskgeneration and data collection.

Original: For instance, in the medical domain, it is infeasible to collect a largeamount of data to construct extensive meta-training tasks due to privacy concerns.
Revised: For instance, in the medical domain, it is infeasible to collect a largeamount of data to construct extensive meta-training tasks due to privacy concerns.

Original: Moreover, fornatural language processing, it is not straightforward to split a dataset into tasks, and hence entiredatasets are treated as tasks [28].
Revised: Moreover, fornatural language processing, it is not straightforward to split a dataset into tasks, and hence entiredatasets are treated as tasks [30].

Original: Several works have been proposed to tackle the few-task meta-learning problem using task augmentation techniques such as clustering a dataset into multiple tasks [28], leveraging strong image
Revised: Several works have been proposed to tackle the few-task meta-learning problem using task augmentation techniques such as clustering a dataset into multiple tasks [30], leveraging strong image

Original: [Figure]
Revised: [Figure]

Original: augmentation methods such as vertical flipping to construct new classes [30], and the employmentof Manifold Mixup [42] for densifying the meta-training task distribution [47, 48].
Revised: augmentation methods such as vertical flipping to construct new classes [32], and the employmentof Manifold Mixup [44] for densifying the meta-training task distribution [49, 50].

Original: However, majority of these techniques require domain-specific knowledge to design such task augmentationsand hence cannot be applied to other domains.
Revised: However, majority of these techniques require domain-specific knowledge to design such task augmentationsand hence cannot be applied to other domains.

Original: While Manifold Mixup based methods [47, 48] aredomain-agnostic, we empirically find them ineffective for mitigating meta-overfitting in few-taskmeta-learning especially in non-image domains such as chemical and text, and that they sometimesdegrade generalization performance.
Revised: While Manifold Mixup based methods [49, 50] aredomain-agnostic, we empirically find them ineffective for mitigating meta-overfitting in few-taskmeta-learning especially in non-image domains such as chemical and text, and that they sometimesdegrade generalization performance.

Original: In this work, we focus solely on domain-agnostic task augmentation methods that can densifythe meta-training task distribution to prevent meta-overfitting and improve generalization at metatesting for few-task meta-learning.
Revised: In this work, we focus solely on domain-agnostic task augmentation methods that can densifythe meta-training task distribution to prevent meta-overfitting and improve generalization at metatesting for few-task meta-learning.

Original: To tackle the limitations already discussed, we propose a noveldomain-agnostic task augmentation method for metric based meta-learning models.
Revised: To tackle the limitations already discussed, we propose a noveldomain-agnostic task augmentation method for metric based meta-learning models.

Original: Our method,
Revised: Our method,

Original: Meta-Interpolation, utilizes expressive neural set functions to interpolate two tasks and the setfunctions are trained with bilevel optimization
Revised: Meta-Interpolation, utilizes expressive neural set functions to interpolate two tasks and the setfunctions are trained with bilevel optimization

Original: so that a meta-learner trained on the interpolated tasksgeneralizes to tasks in the meta-validation set.
Revised: so that a meta-learner trained on the interpolated tasksgeneralizes to tasks in the meta-validation set.

Original: As a consequence of end-to-end training, the learnedaugmentation strategy is tailored to each specific domain without the need for specialized domainknowledge.
Revised: As a consequence of end-to-end training, the learnedaugmentation strategy is tailored to each specific domain without the need for specialized domainknowledge.

Original: For example, for K -way classification, we sample two tasks consisting of support and query setsand assign a new class k to each pair of classes { σ ( k ) , σ ′ ( k ) } for k = 1 , . . .
Revised: For example, for K -way classification, we sample two tasks consisting of support and query setsand assign a new class k to each pair of classes { σ ( k ) , σ ′ ( k ) } for k = 1 , . . .

Original: , K , where σ, σ ′ arepermutations on { 1 , . . .
Revised: , K , where σ, σ ′ arepermutations on { 1 , . . .

Original: , K } as depicted in Figure 1a.
Revised: , K } as depicted in Figure 1a.

Original: Hidden representations of the support set withclasses σ ( k ) and σ ′ ( k ) are then transformed into a single support set using a set function that mapsa set of two vectors to a single vector.
Revised: Hidden representations of the support set withclasses σ ( k ) and σ ′ ( k ) are then transformed into a single support set using a set function that mapsa set of two vectors to a single vector.

Original: We refer to the output of the set function as the interpolatedsupport set and these are used to compute class prototypes.
Revised: We refer to the output of the set function as the interpolatedsupport set and these are used to compute class prototypes.

Original: As shown in Figure 1b, the interpolatedsupport set is paired with a query set (query set 1 in Figure 1a)), randomly selected from the twotasks to obtain a new task.
Revised: As shown in Figure 1b, the interpolatedsupport set is paired with a query set (query set 1 in Figure 1a)), randomly selected from the twotasks to obtain a new task.

Original: Lastly, we optimize the set function so that a meta-learner trained on theaugmented task can minimize the loss on the meta-validation tasks as illustrated in Figure 1c.
Revised: Lastly, we optimize the set function so that a meta-learner trained on theaugmented task can minimize the loss on the meta-validation tasks as illustrated in Figure 1c.

Original: To verify the efficacy of our method, we empirically show that it significantly improves the performance of Prototypical Networks [38] on the few-task meta-learning problem across multiple domains.
Revised: To verify the efficacy of our method, we empirically show that it significantly improves the performance of Prototypical Networks [40] on the few-task meta-learning problem across multiple domains.

Original: Our method outperforms the relevant baselines on eight few-task meta-learning benchmark datasetsspanning image classification, chemical property prediction, text classification, and automatic speechrecognition.
Revised: Our method outperforms the relevant baselines on eight few-task meta-learning benchmark datasetsspanning image classification, chemical property prediction, text classification, and automatic speechrecognition.

Original: Furthermore, our theoretical analysis shows that our task interpolation method with theset function regularizes the meta-learner and improves generalization performance.
Revised: Furthermore, our theoretical analysis shows that our task interpolation method with theset function regularizes the meta-learner and improves generalization performance.

Original: Our contribution is threefold:
Revised: Our contribution is threefold:

Original: • We propose a novel domain-agnostic task augmentation method, Meta-Interpolation, which
Revised: • We propose a novel domain-agnostic task augmentation method, Meta-Interpolation, which

Original: leverages expressive set functions to densify the meta-training task distribution for the few-task meta-learning problem.
Revised: leverages expressive set functions to densify the meta-training task distribution for thefew-task meta-learning problem.

Original: • We theoretically analyze our model and show that it regularizes the meta-learner for bettergeneralization.• Through extensive experiments, we show that Meta-Interpolation significantly improves the
Revised: • We theoretically analyze our model and show that it regularizes the meta-learner for bettergeneralization.• Through extensive experiments, we show that Meta-Interpolation significantly improves the

Original: performance of Prototypical Network on various domains such as image, text, and chemicalmolecule classification, and speech recognition on the few-task meta-learning problem.
Revised: performance of Prototypical Network on various domains such as image, text, and chemicalmolecule classification, and speech recognition on the few-task meta-learning problem.

Original:  Related Work
Revised:  Related Work

Original: Meta-Learning The two mainstream approaches to meta-learning are gradient based [8, 31, 12, 22,10, 34, 35] and metric based meta-learning [43, 38, 40, 27, 24, 5, 36].
Revised: Meta-Learning The two mainstream approaches to meta-learning are gradient based [10, 33, 14,24, 12, 36, 37] and metric based meta-learning [45, 40, 42, 29, 26, 6, 38].

Original: The former formulates metaknowledge as meta-parameters such as the initial model parameters and performs bilevel optimizationto estimate the meta-parameters so that a meta-learner can generalize to unseen tasks with fewgradient steps.
Revised: The former formulatesmeta-knowledge as meta-parameters such as the initial model parameters and performs bileveloptimization to estimate the meta-parameters so that a meta-learner can generalize to unseen taskswith few gradient steps.

Original: The latter learns an embedding space where classification is performed by measuringthe distance between a query and a set of class prototypes.
Revised: The latter learns an embedding space where classification is performed bymeasuring the distance between a query and a set of class prototypes.

Original: In this work, we focus on metric basedmeta-learning with fewer number of meta-training tasks, i.e., few-task meta-learning.
Revised: In this work, we focus onmetric based meta-learning with fewer number of meta-training tasks, i.e., few-task meta-learning.

Original: We proposea novel task augmentation method that densifies the meta-training task distribution and mitigatesoverfitting due to the fewer number of meta-training tasks for better generalization to unseen tasks.
Revised: We propose a novel task augmentation method that densifies the meta-training task distribution andmitigates overfitting due to the fewer number of meta-training tasks for better generalization to unseentasks.

Original: Task Augmentation for Few-Task Meta-learning Several methods have been proposed to augmentthe number of meta-training tasks to mitigate overfitting in the context of few-task meta-learning.
Revised: Task Augmentation for Few-Task Meta-learning Several methods have been proposed to augmentthe number of meta-training tasks to mitigate overfitting in the context of few-task meta-learning.

Original: Ni et al.
Revised: Ni et al.

Original: [30] apply strong data augmentations such as vertical flip to images to create a new class.
Revised: [32] apply strong data augmentations such as vertical flip to images to create a new class.

Original: For text classification, Murty et al.
Revised: For text classification, Murty et al.

Original: [28] split meta-training tasks into latent reasoning categories byclustering data with a pretrained language model.
Revised: [30] split meta-training tasks into latent reasoning categories byclustering data with a pretrained language model.

Original: However, they require domain-specific knowledgeto design such augmentations, and hence the resulting augmentation techniques are inapplicable toother domains where there is no well-defined data augmentation or pretrained model.
Revised: However, they require domain-specific knowledgeto design such augmentations, and hence the resulting augmentation techniques are inapplicable toother domains where there is no well-defined data augmentation or pretrained model.

Original: In order to tacklethis limitation, Manifold Mixup-based task augmentations have also been proposed.
Revised: In order to tacklethis limitation, Manifold Mixup-based task augmentations have also been proposed.

Original: MetaMix [47]interpolates support and query sets with Manifold Mixup [42] to construct a new query set.
Revised: MetaMix [49]interpolates support and query sets with Manifold Mixup [44] to construct a new query set.

Original: MLTI [48]performs Manifold Mixup [42] on support and query sets from two tasks for task augmentation.
Revised: MLTI [50]performs Manifold Mixup [44] on support and query sets from two tasks for task augmentation.

Original: Although these methods are domain-agnostic, we empirically find that they are not effective in somedomains and can degrade generalization performance.
Revised: Although these methods are domain-agnostic, we empirically find that they are not effective in somedomains and can degrade generalization performance.

Original: In contrast, we propose to train an expressiveneural set function to interpolate two tasks with bilevel optimization to find optimal augmentationstrategies tailored specifically to each domain.
Revised: In contrast, we propose to train an expressiveneural set function to interpolate two tasks with bilevel optimization to find optimal augmentationstrategies tailored specifically to each domain.

Original:  Method
Revised:  Method

Original: Preliminaries In meta-learning, we are given a finite set of tasks {T t } Tt =1 , which are i.i.d samplesfrom an unknown task distribution p ( T ) .
Revised: Preliminaries In meta-learning, we are given a finite set of tasks {T t } Tt =1 , which are i.i.d samplesfrom an unknown task distribution p ( T ) .

Original: Each task T t consists of a support set D st = { ( x st,i , y st,i ) } N s i
Revised: Each task T t consists of a support set D st = { ( x st,i , y st,i ) } N s i

Original: =1and a query set D qt = { ( x qt,i , y qt,i ) }
Revised: =1and a query set D qt = { ( x qt,i , y qt,i ) }

Original: N q i =1 , where x t,i and y t,i denote a data point and its correspondinglabel respectively.
Revised: N q i =1 , where x t,i and y t,i denote a data point and its correspondinglabel respectively.

Original: Given a predictive model ˆ f θ,λ :=
Revised: Given a predictive model ˆ f θ,λ :=

Original: f Lθ L ◦ · · · ◦ f l +1 θ l +1 ◦ φ λ ◦ f lθ l ◦ · · · ◦ f 1 θ 1 with Llayers, we want to estimate the parameter θ that minimizes the meta-training loss and generalizes toquery sets D q ∗ sampled from an unseen task T ∗ using the support set D s ∗ , where λ is a hyperparameterfor the function φ λ .
Revised: f Lθ L ◦ · · · ◦ f l +1 θ l +1 ◦ φ λ ◦ f lθ l ◦ · · · ◦ f 1 θ 1 with Llayers, we want to estimate the parameter θ that minimizes the meta-training loss and generalizes toquery sets D q ∗ sampled from an unseen task T ∗ using the support set D s ∗ , where λ is a hyperparameterfor the function φ λ .

Original: In this work, we primarily focus on metric based meta-learning methods ratherthan gradient based meta-learning methods due to efficiency and empirically higher performance overthe gradient based methods on the tasks we consider.
Revised: In this work, we primarily focus on metric based meta-learning methods ratherthan gradient based meta-learning methods due to efficiency and empirically higher performance overthe gradient based methods on the tasks we consider.

Original: Problem Statement
Revised: Problem Statement

Original: In this work, we focus solely on few-task meta-learning.
Revised: In this work, we focus solely on few-task meta-learning.

Original: Here, the number ofmeta-training tasks drawn from the meta-training distribution is extremely small and the goal of ameta-learner is to learn meta-knowledge from such limited tasks that can be transferred to unseentasks during meta-testing.
Revised: Here, the number ofmeta-training tasks drawn from the meta-training distribution is extremely small and the goal of ameta-learner is to learn meta-knowledge from such limited tasks that can be transferred to unseentasks during meta-testing.

Original: The key challenges here are preventing the meta-learner from overfittingon the meta-training tasks and generalizing to unseen tasks drawn from a meta-test set.
Revised: The key challenges here are preventing the meta-learner from overfittingon the meta-training tasks and generalizing to unseen tasks drawn from a meta-test set.

Original: Metric Based Meta-Learning The goal of metric based meta-learning is to learn an embeddingspace induced by ˆ f θ,λ , where we perform classification by computing distances between data pointsand class prototypes.
Revised: Metric Based Meta-Learning The goal of metric based meta-learning is to learn an embeddingspace induced by ˆ f θ,λ , where we perform classification by computing distances between data pointsand class prototypes.

Original: We adopt Prototypical Network (ProtoNet) [38] for ˆ f θ,λ , where φ λ is theidentity function.
Revised: We adopt Prototypical Network (ProtoNet) [40] for ˆ f θ,λ , where φ λ is theidentity function.

Original: Specifically, for each task T t with its corresponding support D st and query D qt sets,we compute class prototypes { c k } Kk =1 as the average of the hidden representation of the supportsamples belonging to the class k as follows:
Revised: Specifically, for each task T t with its corresponding support D st and query D qt sets,we compute class prototypes { c k } Kk =1 as the average of the hidden representation of the supportsamples belonging to the class k as follows:

Original: [Equation]
Revised: [Equation]

Original: where N k denotes the number of instances belonging to the class k .
Revised: where N k denotes the number of instances belonging to the class k .

Original: Given a metric d ( · , · ) : R d L ×
Revised: Given a metric d ( · , · ) : R D ×

Original: R d L (cid:55)→ R , we compute the probability of a query point x qt,i being assigned to the class k by measuringthe distance between the hidden representation ˆ f θ,λ ( x qt,i ) and the class prototype c k followed by dation
Revised: R D (cid:55)→ R , we compute the probability of a query point x qt,i being assigned to the class k by measuringthe distance between the hidden representation ˆ f θ,λ ( x qt,i ) and the class prototype c k followed by [Table] [Table] [Table]

Revised: dation

Original: [Table]
Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original:  − = α · grad ( ∂ L tr ∂θ , θ, grad_outputs = v 1 ) 10: p += v 1 11: end for 12: v 2 ← grad ( L tr ∂θ , λ, grad_outputs = α p ) 13: return ∂ L V ∂λ − v 2 14: softmax.
Revised: [Table] [Table] 9: v 1 − = α · grad ( ∂ L tr ∂θ , θ, grad_outputs = v 1 ) 10: p += v 1 11: end for 12: v 2 ← grad ( L tr ∂θ , λ, grad_outputs = α p ) 13: return ∂ L V ∂λ − v 2 14: softmax.

Original: With the class probability, we compute the cross-entropy loss for ProtoNet as follows:
Revised: With the class probability, we compute the cross-entropy loss for ProtoNet as follows:

Original: [Equation]
Revised: [Equation]

Original: where 1 is an indicator function.
Revised: where 1 is an indicator function.

Original: At meta-test time, a test query is assigned a label based on theminimal distance to a class prototype, i.e., y q ∗ = arg min k d
Revised: At meta-test time, a test query is assigned a label based on theminimal distance to a class prototype, i.e., y q ∗ = arg min k d

Original: ( ˆ f θ,λ ( x q ∗ ) , c k ) .
Revised: ( ˆ f θ,λ ( x q ∗ ) , c k ) .

Original: However, optimizing1 T (cid:80) Tt =
Revised: However, optimizing1 T (cid:80) Tt =

Original: 1 L singleton ( λ, θ ; T t ) w.r.t θ is prone to overfitting since we are given only a small number ofmeta-training tasks.
Revised: 1 L singleton ( λ, θ ; T t ) w.r.t θ is prone to overfitting since we are given only a small number ofmeta-training tasks.

Original: The meta-learner tends to memorize the meta-training tasks, which limits itsgeneralization to new tasks at meta-test time [49, 33].
Revised: The meta-learner tends to memorize the meta-training tasks, which limits itsgeneralization to new tasks at meta-test time [51, 35].

Original: Meta-Interpolation for Task Augmentation
Revised: Meta-Interpolation for Task Augmentation

Original: In order to tackle the meta-overfitting problem with asmall number of tasks, we propose a novel data-driven domain-agnostic task augmentation frameworkwhich enables the meta-learner trained on few tasks to generalize to unseen few-shot classificationtasks.
Revised: In order to tackle the meta-overfitting problem with asmall number of tasks, we propose a novel data-driven domain-agnostic task augmentation frameworkwhich enables the meta-learner trained on few tasks to generalize to unseen few-shot classificationtasks.

Original: Several methods have been proposed to densify the meta-training tasks.
Revised: Several methods have been proposed to densify the meta-training tasks.

Original: However, theyheavily depend on the augmentation of images [30] or need a pretrained language model for taskaugmentation [28].
Revised: However, theyheavily depend on the augmentation of images [32] or need a pretrained language model for taskaugmentation [30].

Original: Although Manifold Mixup based methods [47, 48] are domain-agnostic, weempirically find them ineffective in certain domains.
Revised: Although Manifold Mixup based methods [49, 50] are domain-agnostic, weempirically find them ineffective in certain domains.

Original: Instead, we optimize expressive neural setfunctions to augment tasks to enhance the generalization of a meta-learner to unseen tasks.
Revised: Instead, we optimize expressive neural setfunctions to augment tasks to enhance the generalization of a meta-learner to unseen tasks.

Original: As aconsequence of end-to-end training, the learned augmentation strategy is tailored to each domain.
Revised: As aconsequence of end-to-end training, the learned augmentation strategy is tailored to each domain.

Original: Specifically, let φ λ : R n × d → R d be a set function which maps a set of d dimensional vectors withcardinality n to a d dimensional vector.
Revised: Specifically, let φ λ : R n × d → R d be a set function which maps a set of d dimensional vectors withcardinality n to a d dimensional vector.

Original: In all our experiments, we use Set Transformer [21] for φ λ .
Revised: In all our experiments, we use Set Transformer [23] for φ λ .

Original: Given a pair of tasks T t 1 = {D s t 1 , D qt 1 } and T t 2 = {D s t 2 , D qt 2 } with corresponding support and querysets for K way classification, we construct new classes by choosing K pairs of classes from the twotasks.
Revised: Given a pair of tasks T t 1 = {D st 1 , D qt 1 } and T t 2 = {D st 2 , D qt 2 } with corresponding support and querysets for K way classification, we construct new classes by choosing K pairs of classes from the twotasks.

Original: We sample permutations σ t 1 and σ t 2 on { 1 , . .
Revised: We sample permutations σ t 1 and σ t 2 on { 1 , . .

Original: , K } for each task T t 1 and T t 2 respectively andassign class k to the pair { σ t 1 ( k ) , σ t 2 ( k ) } for k = 1 , . .
Revised: , K } for each task T t 1 and T t 2 respectively andassign class k to the pair { σ t 1 ( k ) , σ t 2 ( k ) } for k = 1 , . .

Original: , K .
Revised: , K .

Original: For the newly assigned class k , we pairtwo instances from classes σ t 1 ( k ) and σ t 2 ( k ) and interpolate their hidden representations with theset function φ λ .
Revised: For the newly assigned class k , we pairtwo instances from classes σ t 1 ( k ) and σ t 2 ( k ) and interpolate their hidden representations with theset function φ λ .

Original: The class prototypes for class k are computed using the output of φ λ as follows:
Revised: The class prototypes for class k are computed using the output of φ λ as follows:

Original: [Equation]
Revised: [Equation]

Original: where we define ˆ D s to be the set of all the interpolated prototypes ˆ c k for k = 1 , .
Revised: where we define ˆ D s to be the set of all the interpolated prototypes ˆ c k for k = 1 , .

Original: , K .
Revised: , K .

Original: Forqueries, we do not perform any interpolation.
Revised: For queries,we do not perform any interpolation.

Original: Instead, we use D qt 1 as the query set and compute its hidden representation ˆ f θ,λ ( x qt 1 ,i ) ∈ R d L .
Revised: Instead, we use D qt 1 as the query set and compute its hidden representation ˆ f θ,λ ( x qt 1 ,i ) ∈ R D .

Original: We then measure the distance between the query withy qt 1 ,i =
Revised: We then measure the distance between the query with y qt 1 ,i =

Original: σ t 1 ( k ) and the interpolated prototype of class k to compute the loss as follows:
Revised: σ t 1 ( k )and the interpolated prototype of class k to compute the loss as follows:

Original: [Equation]
Revised: [Equation]

Original: where ˆ T = { ˆ D s , D qt 1 } .
Revised: where ˆ T = { ˆ D s , D qt 1 } .

Original: The intuition behind interpolating only support sets is to construct hardertasks that a meta-learner cannot memorize.
Revised: The intuition behind interpolating only support sets is to construct harder tasksthat a meta-learner cannot memorize.

Original: Alternatively, we can interpolate only query sets.
Revised: Alternatively, we can interpolate only query sets.

Original: However,this is computationally more expensive since the size of query sets is usually larger than that ofsupport sets.
Revised: However, thisis computationally more expensive since the size of query sets is usually larger than that of supportsets.

Original: In Section 5, we empirically show that interpolating either support or query sets achieveshigher loss than interpolating both.
Revised: In Section 5, we empirically show that interpolating either support or query sets achieves highertraining loss than interpolating both, which empirically supports the intuition.

Original: Lastly, we also use the original task T t 1 to evaluate the loss L singleton ( λ, θ, T t 1 ) in Eq.2 by passing the corresponding support and query set to ˆ f θ,λ .
Revised: Lastly, we also use theoriginal task T t 1 to evaluate the loss L singleton ( λ, θ, T t 1 ) in Eq. by passing the corresponding supportand query set to ˆ f θ,λ .

Original: The additionalforward pass enriches the diversity of the augmented tasks and makes meta-training consistent withmeta-testing since we do not perform any task augmentation in the meta-testing stage.
Revised: The additional forward pass enriches the diversity of the augmented tasks andmakes meta-training consistent with meta-testing since we do not perform any task augmentation inthe meta-testing stage.

Original: Optimization Since jointly optimizing θ , the parameters of ProtoNet, and λ , the parameters of theset function φ λ , with few tasks is prone to overfitting, we consider λ as hyperparameters and performbilevel optimization with meta-training and meta-validation tasks as follows:
Revised: Optimization Since jointly optimizing θ , the parameters of ProtoNet, and λ , the parameters of theset function φ λ , with few tasks is prone to overfitting, we consider λ as hyperparameters and performbilevel optimization with meta-training and meta-validation tasks as follows:

Original: [Equation]
Revised: [Equation]

Original: where T train t , T val t , ˆ T t denote the meta-training, meta-validation, and interpolated task, respectively.
Revised: where T train t , T val t , ˆ T t denote the meta-training, meta-validation, and interpolated task, respectively.

Original: Since computing the exact gradient w.r.t λ is intractable due to the long inner optimization steps in
Revised: Since computing the exact gradient w.r.t λ is intractable due to the long inner optimization steps in

Original: Eq. 6, we leverage the implicit function theorem to approximate the gradient as Lorraine et al.
Revised: Eq. 6, we leverage the implicit function theorem to approximate the gradient as Lorraine et al.

Original: Moreover, we alternately update θ and λ for computational efficiency as described in Algo.
Revised: Moreover, we alternately update θ and λ for computational efficiency as described in Algo.

Original: 1 and 2.
Revised: 1 and 2.

Original:  Theoretical Analysis
Revised:  Theoretical Analysis

Original: In this section, we theoretically investigate the behavior of the Set Transformer and how it inducesa distribution dependent regularization, which is then shown to have the ability to control the
Revised: In this section, we theoretically investigate the behavior of the Set Transformer and how it inducesa distribution dependent regularization, which is then shown to have the ability to control the

Original: Rademacher complexity for better generalization.
Revised: Rademacher complexity for better generalization.

Original: To analyze the behavior of the Set Transformer,we first define it concretely with the attention mechanism A ( Q, K, V ) =
Revised: To analyze the behavior of the Set Transformer,we first define it concretely with the attention mechanism A ( Q, K, V ) =

Original: softmax( √ d − 1 QK ⊤ ) V .
Revised: softmax( √ d − 1 QK ⊤ ) V .

Original: Given h, h ′ ∈ R d , define H { h,h ′ }=
Revised: Given h, h ′ ∈ R d , define H { h,h ′ }=

Original: [ h, h ′ ]
Revised: [Equation] , h ′ ]

Original: ⊤ ∈ R 2 × d
Revised: ⊤ ∈ R 2 × d

Original: and H { h }= h ⊤ ∈ R 1 × d .
Revised: and H { [Equation] }= h ⊤ ∈ R 1 × d .

Original: Then, for anyr ∈ {{ h, h ′ } , { h }} , the output of the Set Transformer φ λ ( r ) is defined as follows:
Revised: Then, for anyr ∈ {{ h, h ′ } , { h }} , the output of the Set Transformer φ λ ( r ) is defined as follows:

Original: [Equation]
Revised: [Equation]

Original: where Q 2 = SW Q 2 + b Q 2 , Q r 1 = H r 1 W Q 1 + 1 2 b Q 1 , K rj
Revised: where Q 2 = SW Q 2 + b Q 2 , Q r 1 = H r 1 W Q 1 + 1 2 b Q 1 , K rj

Original: = H rj W Kj + 1 2 b Kj , V rj
Revised: = H rj W Kj + 1 2 b Kj , V rj

Original: = H rj W Vj + 1 2 b Vj(for j ∈ { 1 , 2 } ), and H r 2 =
Revised: = H rj W Vj + 1 2 b Vj(for j ∈ { 1 , 2 } ), and H r 2 =

Original: A ( Q r 1 , K r 1 , V r 1 ) ∈ R n × d .
Revised: A ( Q r 1 , K r 1 , V r 1 ) ∈ R n × d .

Original: Q j , K j , V j denote query, key, and value forthe attention mechanism for j = 1 , 2 , respectively.
Revised: Q j , K j , V j denote query, key, and value forthe attention mechanism for j = 1 , 2 , respectively.

Original: Here, 1 2 = [1 , . . .
Revised: Here, 1 2 = [1 , . . .

Original: , 1] ⊤ ∈ R n , W Qj , W Kj , W Vj ∈
Revised: , 1] ⊤ ∈ R n , W Qj , W Kj , W Vj ∈

Original: R d × d , b Qj , b Kj , b Vj ∈ R 1 × d , Q r 1 , K rj , V rj ∈ R n × d , and Q 2 ∈ R 1 × d .
Revised: R d × d , b Qj , b Kj , b Vj ∈ R 1 × d , Q r 1 , K rj , V rj ∈ R n × d , and Q 2 ∈ R 1 × d .

Original: Let l ∈ { 1 , . .
Revised: Let l ∈ { 1 , . .

Original: , L } .
Revised: , L } .

Original: Our analysis will show the importance of the following quantity of the Set Transformer in our method:
Revised: Our analysis will show the importance of the following quantity of the Set Transformer in our method:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: with h t,i =
Revised: with h t,i =

Original: ϕ lθ ( x st,i ) and ϕ lθ
Revised: ϕ lθ ( x st,i ) and ϕ lθ

Original: = f lθ l ◦ · · · ◦ f 1 θ 1 .
Revised: = f lθ l ◦ · · · ◦ f 1 θ 1 .

Original: For a matrix A ∈ R m × n , A i,j denotes the entry fori -th row and j -th column of the matrix A .
Revised: For a matrix A ∈ R m × n , A i,j denotes the entry fori -th row and j -th column of the matrix A .

Original: We now introduce the additional notation and problem setting to present our results.
Revised: We now introduce the additional notation and problem setting to present our results.

Original: Define W =
Revised: Define W =

Original: [Equation]
Revised: [Equation]

Revised: ∈

Revised: [Equation]

Original: and I t,k = { i ∈
Revised: and I t,k = { i ∈

Original: [ N ( t ) s ] : y st,i = k } , where N ( t ) s =
Revised: [ N ( t ) s ] : y st,i = k } , where N ( t ) s =

Original: |D st | .
Revised: |D st | .

Original: We also define the empirical measureµ t,k = 1 |
Revised: We also define the empirical measureµ t,k = 1 |

Original: I t,k | (cid:80) i ∈
Revised: I t,k | (cid:80) i ∈

Original: I t,k δ i over the index i ∈
Revised: I t,k δ i over the index i ∈

Original: [ N ( t ) s ] with the Dirac measures
Revised: [ N ( t ) s ] with the Dirac measures

Original: δ i .
Revised: δ i .

Original: Let U [ K ] be theuniform distribution over { 1 , . . .
Revised: Let U [ K ] be theuniform distribution over { 1 , . . .

Original: , K } .
Revised: , K } .

Original: For any function φ and point u in its domain, we definethe j -th order tensor ∂
Revised: For any function φ and point u in its domain, we definethe j -th order tensor ∂

Original: j φ ( u ) ∈ R d × d ×···× d by ∂ j φ ( u ) i 1 i 2 ··· i j =
Revised: j φ ( u ) ∈ R d × d ×···× d by ∂ j φ ( u ) i 1 i 2 ··· i j =

Original: [Equation]
Revised: [Equation]

Original: ∂u
Revised: ∂u

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 2 ··· ∂u ij φ ( u ) .
Revised: 2 ··· ∂u ij φ ( u ) .

Original: For example,∂ 1 φ ( u ) and ∂ 2 φ ( u ) are the gradient and the Hessian of φ evaluated at u .
Revised: For example,∂ 1 φ ( u ) and ∂ 2 φ ( u ) are the gradient and the Hessian of φ evaluated at u .

Original: For any j -th order tensor∂ j φ ( u ) , we define the vectorization of the tensor by vec[ ∂ j φ ( u )]
Revised: For any j -th order tensor∂ j φ ( u ) , we define the vectorization of the tensor by vec[ ∂ j φ ( u )]

Original: ∈ R d j .
Revised: ∈ R d j .

Original: For an vector a ∈ R d , wedefine a ⊗ j = a ⊗ a ⊗ · · · ⊗ a ∈ R d j where ⊗ represents the Kronecker product.
Revised: For an vector a ∈ R d , wedefine a ⊗ j = a ⊗ a ⊗ · · · ⊗ a ∈ R d j where ⊗ represents the Kronecker product.

Original: We assume that∂ r g (cid:0)
Revised: We assume that∂ r g (cid:0)

Original: Wϕ lθ
Revised: Wϕ lθ

Original: l
Revised: l

Original: ( x st,i )
Revised: ( x st,i )

Original: + b (cid:1)
Revised: + b (cid:1)

Original: = 

Revised: =

Original: for all r ≥ 2 , where g :=
Revised: for all r ≥ 2 , where g :=

Original: f Lθ L ◦ · · · ◦ f l +1 θ l +1 .
Revised: f Lθ L ◦ · · · ◦ f l +1 θ l +1 .

Original: This assumption is satisfied,for example, if g represents a deep neural network with ReLU activations.
Revised: This assumption is satisfied,for example, if g represents a deep neural network with ReLU activations.

Original: This assumption is alsosatisfied in the simpler special case considered in the proposition below.
Revised: This assumption is alsosatisfied in the simpler special case considered in the proposition below.

Original: The following theorem shows that L mix ( λ, θ, ˆ T t,t ′ ) is approximately L singleton ( λ, θ ; T t ) plus regularization terms on the directional derivatives of ϕ lθ l on the direction of W ( ϕ lθ l ( x st ′ ,j )
Revised: The following theorem shows that L mix ( λ, θ, ˆ T t,t ′ ) is approximately L singleton ( λ, θ ; T t ) plus regularization terms on the directional derivatives of ϕ lθ l on the direction of W ( ϕ lθ l ( x st ′ ,j )

Original: − ϕ lθ l ( x st,i )) :
Revised: − ϕ lθ l ( x st,i )) :

Original: Theorem 1.
Revised: Theorem 1.

Original: For any J ∈ N + , if c (cid:55)→ d ( y, c ) is J -times differentiable for all y , then the J -th order approximation of L mix ( λ, θ, ˆ T t,t ′ ) is given by L singleton ( λ, θ ; T t )
Revised: For any J ∈ N + , if c (cid:55)→ d ( y, c ) is J -times differentiable for all y , then the J -th order approximation of L mix ( λ, θ, ˆ T t,t ′ ) is given by L singleton ( λ, θ ; T t )

Original: + (cid:80)
Revised: + (cid:80)

Original: Jj =1 1 j ! vec[ ∂ j L t ( c )]
Revised: Jj =1 1 j ! vec[ ∂ j L t ( c )]

Original: ⊤ ∆ ⊗ j , where ∆ =
Revised: ⊤ ∆ ⊗ j , where ∆ =

Original: [∆ ⊤ 1 , . . . , ∆ ⊤ K ] ⊤ and
Revised: [∆ ⊤ 1 , . . . , ∆ ⊤ K ] ⊤ and

Original: [Equation]
Revised: [Equation]

Original: To illustrate the effect of this data-dependent regularization, we now consider the following special case that is used by Yao et al.
Revised: To illustrate the effect of this data-dependent regularization, we now consider the following special case that is used by Yao et al.

Original: [48] for ProtoNet: L ( λ, θ ; T t )
Revised: [50] for ProtoNet: L ( λ, θ ; T t )

Original: = 1 n (cid:80) ni
Revised: = 1 n (cid:80) ni

Original: =

Revised: =

Original: L i ( λ, θ ; T t ) where
Revised: L i ( λ, θ ; T t ) where

Original: L i ( λ, θ ; T t ) = 1 1+exp( ⟨ ( x qt,i − ( c ′ 1 + c ′ 2 ) / 2 ,θ ⟩ )
Revised: L i ( λ, θ ; T t ) = 1 1+exp( ⟨ ( x qt,i − ( c ′ 1 + c ′ 2 ) / 2 ,θ ⟩ )

Original: where c ′ k :
Revised: where c ′ k :

Original: = 1 N t,k (cid:80) ( x st,i , y st,i ) ∈D st 1 { y t,i
Revised: = 1 N t,k (cid:80) ( x st,i , y st,i ) ∈D st 1 { y t,i

Original: = k } x st,i , and⟨· , ·⟩ denotes dot product.
Revised: = k } x st,i , and⟨· , ·⟩ denotes dot product.

Original: Define c = 1 n (cid:80) ni
Revised: Define c = 1 n (cid:80) ni

Original: =1 14 ψ ( z t,i )( ψ ( z t,i ) − 0 . 5) 1+exp( z t,i ) , where ψ ( z t,i ) =
Revised: =1 14 ψ ( z t,i )( ψ ( z t,i ) − 0 . 5) 1+exp( z t,i ) , where ψ ( z t,i ) =

Original: exp( z t,i ) 1+exp( z t,i )and z t,i = ⟨ x qt,i − ( c ′ 1 + c ′ 2 ) / 2 , θ ⟩ .
Revised: exp( z t,i ) 1+exp( z t,i )and z t,i = ⟨ x qt,i − ( c ′ 1 + c ′ 2 ) / 2 , θ ⟩ .

Original: Note that c > 

Revised: Note that c >

Original: if θ is no worse than the randomguess; e.g., L i ( λ, 0; T t ) >
Revised: if θ is no worse than the randomguess; e.g., L i ( λ, 0; T t ) >

Original: L i ( λ, θ ; T t ) for all i ∈ [ n ] .
Revised: L i ( λ, θ ; T t ) for all i ∈ [ n ] .

Original: We write ∥ v ∥ 2 M = v ⊤ Mv for anypositive semi-definite matrix M .
Revised: We write ∥ v ∥ 2 M = v ⊤ Mv for anypositive semi-definite matrix M .

Original: In this special case, we consider that α is balanced: i.e.,
Revised: In this special case, we consider that α is balanced: i.e.,

Original: E t ′ ,σ [ (cid:80) 2 k = 1 | I t ′ ,σ ( k ) | (cid:80)
Revised: E t ′ ,σ [ (cid:80) 2 k = 1 | I t ′ ,σ ( k ) | (cid:80)

Original: j ∈
Revised: j ∈

Original: I t ′ ,σ ( k ) 1 | I t,k | (cid:80) i ∈
Revised: I t ′ ,σ ( k ) 1 | I t,k | (cid:80) i ∈

Original: I t,k α ( t,t ′ ) ij ( ϕ lθ l ( x st ′ ,j ) − ϕ
Revised: I t,k α ( t,t ′ ) ij ( ϕ lθ l ( x st ′ ,j ) − ϕ

Original: lθ l ( x st,i ))]
Revised: lθ l ( x st,i ))]

Original: = 0 for all t .
Revised: = 0 for all t .

Original: Thisis used to prevent the Set Transformer from over-fitting to the training sets; i.e., in such simple specialcases, the Set Transformer without any restriction is too expressive relative to the rest of the model(and may memorize the training sets without using the rest of the model).
Revised: Thisis used to prevent the Set Transformer from over-fitting to the training sets; i.e., in such simple specialcases, the Set Transformer without any restriction is too expressive relative to the rest of the model(and may memorize the training sets without using the rest of the model).

Original: The following propositionshows that the additional regularization term is simplified to the form of c ∥ θ ∥ 2 M in this special case:
Revised: The following propositionshows that the additional regularization term is simplified to the form of c ∥ θ ∥ 2 M in this special case:

Original: Proposition 1.
Revised: Proposition 1.

Original: In the special case explained above, the second approximation of
Revised: In the special case explained above, the second approximation of

Original: E t ′ ,σ [ L mix ( λ, θ, ˆ T t,t ′ )] is given by L singleton ( λ, θ ; T t )
Revised: E t ′ ,σ [ L mix ( λ, θ, ˆ T t,t ′ )] is given by L singleton ( λ, θ ; T t )

Original: + c ∥ θ ∥ 2 E t ′ ,σ [ δ t,t ′ ,σ δ ⊤ t,t ′ ,σ ] , where δ t,t ′ ,σ =
Revised: + c ∥ θ ∥ 2 E t ′ ,σ [ δ t,t ′ ,σ δ ⊤ t,t ′ ,σ ] , where δ t,t ′ ,σ =

Original: E k ∼ U [2] E i ∼ µ t,k ,j ∼ µ t ′ ,σ ( k )
Revised: E [Equation] ( x st [Equation]

Original: [ α ( t,t ′ ) ij ( x st ′ ,j − x st,i )] .
Revised: j − x st [Equation] i )] .

Original: In the above regularization form, we have an implicit regularization effect on ∥ θ ∥ 2Σ where Σ =
Revised: In the above regularization form, we have an implicit regularization effect on ∥ θ ∥ 2Σ where Σ =

Original: E x , x ′ [( x − x ′ )( x − x ′ ) ⊤ ] .
Revised: E x , x ′ [( x − x ′ )( x − x ′ ) ⊤ ] .

Original: The following theorem shows that this implicit regularization can reducethe Rademacher complexity for better generalization:
Revised: The following theorem shows that this implicit regularization can reducethe Rademacher complexity for better generalization:

Original: Proposition 2.
Revised: Proposition 2.

Original: Let F R = { x (cid:55)→ θ ⊤ x : ∥ θ ∥ 2Σ ≤ R } with E x [ x ] = 0 .
Revised: Let F R = { x (cid:55)→ θ ⊤ x : ∥ θ ∥ 2Σ ≤ R } with E x [ x ] = 0 .

Original: Then, R n ( F
Revised: Then, R n ( F

Original: [Table]
Revised: [Table]

Original: All the proofs are presented in Appendix B.
Revised: All the proofs are presented in Appendix B.

Original:  Experiments
Revised:  Experiments

Original: We now demonstrate the efficacy of our set-based task augmentation method on multiple few-taskbenchmark datasets and compare against the relevant baselines.
Revised: We now demonstrate the efficacy of our set-based task augmentation method on multiple few-taskbenchmark datasets and compare against the relevant baselines.

Original: Datasets We perform classification on eight datasets to validate our method.
Revised: Datasets We perform classification on eight datasets to validate our method.

Original: Metabolism [15], NCI [29] and Tox21 [16]: these are binary classification datasets for predictingthe properties of chemical molecules.
Revised: Metabolism [17], NCI [31] and Tox21 [18]: these are binary classification datasets for predictingthe properties of chemical molecules.

Original: For Metabolism, we use three subdatasets for meta-training,meta-validation, and meta-testing, respectively.
Revised: For Metabolism, we use three subdatasets for meta-training,meta-validation, and meta-testing, respectively.

Original: For NCI, we use four subdatasets for meta-training,two for meta-validation and the remaining three for meta-testing.
Revised: For NCI, we use four subdatasets for meta-training,two for meta-validation and the remaining three for meta-testing.

Original: For Tox21, we use six subdatasets
Revised: For Tox21, we use six subdatasets

Original: [Table]
Revised: [Table]

Original: for meta-training, two for meta-validation, and four for meta-testing.
Revised: for meta-training, two for meta-validation, and four for meta-testing.

Original: (4) GLUE-SciTail [28]: itconsists of four natural language inference datasets where we predict whether a hypothesis sentencecontradicts a premise sentence.
Revised: (4) GLUE-SciTail [30]: itconsists of four natural language inference datasets where we predict whether a hypothesis sentencecontradicts a premise sentence.

Original: We use MNLI [45] and QNLI [44] for meta-training, SNLI [4]and RTE [44] for meta-validation, and SciTail [18] for meta-testing.
Revised: We use MNLI [47] and QNLI [46] for meta-training, SNLI [5]and RTE [46] for meta-validation, and SciTail [20] for meta-testing.

Original: (5) ESC-50 [32]: this is anautomatic speech recognition dataset.
Revised: (5) ESC-50 [34]: this is anautomatic speech recognition dataset.

Original: We make a 20/15/15 split out of 50 base classes for metatraining/validation/testing and sample 5 classes from each spilt to construct a 5-way classificationtask.
Revised: We make a 20/15/15 split out of 50 base classes for metatraining/validation/testing and sample 5 classes from each spilt to construct a 5-way classificationtask.

Original: (6) Rainbow MNIST (RMNIST) [9]: this is a 10-way classification dataset.
Revised: (6) Rainbow MNIST (RMNIST) [11]: this is a 10-way classification dataset.

Original: Following Yaoet al.
Revised: Following Yaoet al.

Original: [48], we construct each task by applying compositions of image transformations to the imagesof the MNIST [7] dataset.
Revised: [50], we construct each task by applying compositions of image transformations to the imagesof the MNIST [9] dataset.

Original: (7) & (8) Mini-ImageNet-S [43] and CIFAR100-FS [20]: these are5-way classification datasets where we choose 12/16/20 classes out of 100 base classes for metatraining/validation/testing, respectively and sample 5 classes from each split to construct a task.
Revised: (7) & (8) Mini-ImageNet-S [45] and CIFAR100-FS [22]: these are5-way classification datasets where we choose 12/16/20 classes out of 100 base classes for metatraining/validation/testing, respectively and sample 5 classes from each split to construct a task.

Original: Note that Metabolism, Tox21, NCI, GLUE-SciTail, and ESC-50 are real-world few-task metalearning datasets with a very small number of tasks.
Revised: Note that Metabolism, Tox21, NCI, GLUE-SciTail, and ESC-50 are real-world few-task metalearning datasets with a very small number of tasks.

Original: For Mini-ImageNet-S and CIFAR100-FS,following Yao et al.
Revised: For Mini-ImageNet-S and CIFAR100-FS,following Yao et al.

Original: [48], we artificially reduce the number of tasks from the original datasets forfew-task meta-learning.
Revised: [50], we artificially reduce the number of tasks from the original datasets forfew-task meta-learning.

Original: RMNIST is synthetically generated by applying augmentations to MNIST.
Revised: RMNIST is synthetically generated by applying augmentations to MNIST.

Original: Implementation Details For RMNIST, Mini-ImageNet-S, and CIFAR100-FS, we use four convolutional blocks with each block consisting of a convolution, ReLU, batch normalization [17], andmax pooling.
Revised: Implementation Details For RMNIST, Mini-ImageNet-S, and CIFAR100-FS, we use four convolutional blocks with each block consisting of a convolution, ReLU, batch normalization [19], andmax pooling.

Original: For Metabolism, Tox21, and NCI, we convert the chemical molecules into SMILESformat and extract a 1024 bit fingerprint feature using RDKit [13] where each bit captures a fragmentof the molecule.
Revised: For Metabolism, Tox21, and NCI, we convert the chemical molecules into SMILESformat and extract a 1024 bit fingerprint feature using RDKit [15] where each bit captures a fragmentof the molecule.

Original: We use two blocks of affine transformation, batch normalization, and Leaky ReLU,and affine transformation for the last layer.
Revised: We use two blocks of affine transformation, batch normalization, and Leaky ReLU,and affine transformation for the last layer.

Original: For GLUE-SciTail dataset, we stack 3 fully connectedlayers with ReLU on the pretrained language model ELECTRA [6].
Revised: For GLUE-SciTail dataset, we stack 3 fully connectedlayers with ReLU on the pretrained language model ELECTRA [8].

Original: For ESC-50 dataset, we pass rawaudio signal to the pretrained VGGish [14] feature extractor to obtain an embedding vector.
Revised: For ESC-50 dataset, we pass rawaudio signal to the pretrained VGGish [16] feature extractor to obtain an embedding vector.

Original: We usethe feature vector as input to the classifier which is exactly the same as the one used for Metabolism,
Revised: We usethe feature vector as input to the classifier which is exactly the same as the one used for Metabolism,

Original: Tox21, and NCI.
Revised: Tox21, and NCI.

Original: For our Meta-Interpolation, we use Set Transformer [21] for the set function φ λ .
Revised: For our Meta-Interpolation, we use Set Transformer [23] for the set function φ λ .

Original: Baselines We compare our method against following domain-agnostic baselines.
Revised: Baselines We compare our method against following domain-agnostic baselines.

Original: ProtoNet [38]: Vanilla ProtoNet trained on Eq. 2 by fixing φ λ to be the identity function.2.
Revised: ProtoNet [40]: Vanilla ProtoNet trained on Eq. 2 by fixing φ λ to be the identity function.2.

Original: MetaReg [2]: ProtoNet with ℓ 2 regularization where element-wise coefficients are learned withbilevel optimization.3.
Revised: MetaReg [2]: ProtoNet with ℓ 2 regularization where element-wise coefficients are learned withbilevel optimization.3.

Original: MetaMix [47]: ProtoNet trained with support sets and mixed query sets where we interpolate oneinstance from the support sets and the other from the original query sets with Manifold Mixup.4.
Revised: MetaMix [49]: ProtoNet trained with support sets and mixed query sets where we interpolate oneinstance from the support sets and the other from the original query sets with Manifold Mixup.4.

Original: MLTI [48]: ProtoNet trained with Manifold Mixup based task augmentation.
Revised: MLTI [50]: ProtoNet trained with Manifold Mixup based task augmentation.

Original: We sample twotasks and interpolate two query sets and support sets with Manifold Mixup, respectively.5.
Revised: We sample twotasks and interpolate two query sets and support sets with Manifold Mixup, respectively.5.

Original: ProtoNet+ST
Revised: ProtoNet+ST

Original: ProtoNet and Set Transformer ( φ λ ) trained with bilevel optimization but withouttask augmentation for L mix ( λ, θ, ˆ T t ) in Eq.
Revised: ProtoNet and Set Transformer ( φ λ ) trained with bilevel optimization but withouttask augmentation for L mix ( λ, θ, ˆ T t ) in Eq.

Original: Meta-Interpolation Our full model learning to interpolate support sets from two tasks usingbilevel optimization and training the ProtoNet with both the original and interpolated tasks.
Revised: Meta-Interpolation Our full model learning to interpolate support sets from two tasks usingbilevel optimization and training the ProtoNet with both the original and interpolated tasks.

Original: Results As shown in Table 1, Meta-Interpolation consistently outperforms all the domain-agnostictask augmentation and regularization baselines on non-image domains.
Revised: Results As shown in Table 1, Meta-Interpolation consistently outperforms all the domain-agnostictask augmentation and regularization baselines on non-image domains.

Original: Notably, it significantlyimproves the performance on ESC-50, which is a challenging datatset that only contains 40 examplesper class.
Revised: Notably, it significantlyimproves the performance on ESC-50, which is a challenging datatset that only contains 40 examplesper class.

Original: In addition, Meta-Interpolation effectively tackles the Metabolism and GLUE-SciTaildatasets which have an extremely small number of meta-training tasks: three and two meta-trainingtasks, respectively.
Revised: In addition, Meta-Interpolation effectively tackles the Metabolism and GLUE-SciTaildatasets which have an extremely small number of meta-training tasks: three and two meta-trainingtasks, respectively.

Original: Contrarily, the baselines do not achieve consistent improvements across all the
Revised: Contrarily, the baselines do not achieve consistent improvements across all the

Original: [Table]
Revised: [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Original: [Table] [Figure] 
Revised: [Table] [Figure] [Table]

Original: [Table] [Figure] [Table] [Figure]
Revised: [Figure]

Original: [Table][Figure] [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: domains and tasks considered.
Revised: domains and tasks considered.

Original: For example, MetaReg is effective on the speech domain (ESC-50)and Metabolism, but does not work on the chemical (Tox21 and NCI) and text (GLUE-SciTail)domains.
Revised: For example, MetaReg is effective on the speech domain (ESC-50)and Metabolism, but does not work on the chemical (Tox21 and NCI) and text (GLUE-SciTail)domains.

Original: Similarly, MetaMix and MLTI achieve performance improvements on some datasets butdegrade the test accuracy on others.
Revised: Similarly, MetaMix and MLTI achieve performance improvements on some datasets butdegrade the test accuracy on others.

Original: This empirical evidence supports the hypothesis that the optimaltask augmentation strategy varies across domains and justifies the motivation for Meta-Interpolationwhich learns augmentation strategies tailored to each domain.
Revised: This empirical evidence supports the hypothesis that the optimaltask augmentation strategy varies across domains and justifies the motivation for Meta-Interpolationwhich learns augmentation strategies tailored to each domain.

Original: We provide additional experimental results on the image domain in Table 2.
Revised: We provide additional experimental results on the image domain in Table 2.

Original: Again, Meta-Interpolationoutperforms all the baselines.
Revised: Again, Meta-Interpolationoutperforms all the baselines.

Original: In contrast to the previous experiments, MetaReg hurts the generalization performance on all the image datasets except on RMNIST.
Revised: In contrast to the previous experiments, MetaReg hurts the generalization performance on all the image datasets except on RMNIST.

Original: Note that Manifold Mixup-basedaugmentation methods, MetaMix and MLTI, marginally improve the generalization performancefor 1-shot classification on Mini-ImageNet-S and CIFAR-100-FS, although they boost the accuracyon 5-shot experiments.
Revised: Note that Manifold Mixup-basedaugmentation methods, MetaMix and MLTI, marginally improve the generalization performancefor 1-shot classification on Mini-ImageNet-S and CIFAR-100-FS, although they boost the accuracyon 5-shot experiments.

Original: This suggests that different task augmentation strategies are required for1-shot and 5-shot for the same dataset.
Revised: This suggests that different task augmentation strategies are required for1-shot and 5-shot for the same dataset.

Original: Meta-Interpolation on the other hand learns task augmentationstrategies tailored for each task and dataset and consistently improves the performance of the vanilla
Revised: Meta-Interpolation on the other hand learns task augmentationstrategies tailored for each task and dataset and consistently improves the performance of the vanilla

Original: ProtoNet for all the experiments on the image datasets.
Revised: ProtoNet for all the experiments on the image datasets.

Original: Moreover, we plot the meta-training and meta-validation loss on RMNIST and NCI dataset in Figure 2.
Revised: Moreover, we plot the meta-training and meta-validation loss on RMNIST and NCI dataset in Figure 2.

Original: Meta-Interpolation obtains higher training loss but much lower validation loss than the others on bothdatasets.
Revised: Meta-Interpolation obtains higher training loss but much lower validation loss than the others on bothdatasets.

Original: This implies that interpolating only support sets constructs harder tasks that a meta-learnercannot memorize and regularizes the meta-learner for better generalization.
Revised: This implies that interpolating only support sets constructs harder tasks that a meta-learnercannot memorize and regularizes the meta-learner for better generalization.

Original: ProtoNet overfits to themeta-training tasks on both datasets.
Revised: ProtoNet overfits to themeta-training tasks on both datasets.

Original: MLTI mitigates the overfitting issue on RMNIST but is noteffective on the NCI dataset where it shows high validation loss in Figure 2d.
Revised: MLTI mitigates the overfitting issue on RMNIST but is noteffective on the NCI dataset where it shows high validation loss in Figure 2d.

Original: On the other hand,
Revised: On the other hand,

Original: MetaMix, which constructs a new query set by interpolating a support and query set with Manifold
Revised: MetaMix, which constructs a new query set by interpolating a support and query set with Manifold

Original: Mixup, results in generating overly difficult tasks which causes underfitting on RMNIST where thetraining loss is not properly minimized in Figure 2a.
Revised: Mixup, results in generating overly difficult tasks which causes underfitting on RMNIST where thetraining loss is not properly minimized in Figure 2a.

Original: However, this augmentation strategy is effectivefor tackling meta-overfitting on NCI where the validation loss is lower than ProtoNet.
Revised: However, this augmentation strategy is effectivefor tackling meta-overfitting on NCI where the validation loss is lower than ProtoNet.

Original: The loss curveof ProtoNet+ST supports the claim that increasing the model size and using bilevel
Revised: The loss curveof ProtoNet+ST supports the claim that increasing the model size and using bilevel

Original: optimizationcannot handle the few-task meta-learning problem.
Revised: optimizationcannot handle the few-task meta-learning problem.

Original: It shows higher validation loss on both RMNISTand NCI as presented in Figure 2b and 2d.
Revised: It shows higher validation loss on both RMNISTand NCI as presented in Figure 2b and 2d.

Original: Similarly, MetaReg which learns coefficients for ℓ 2regularization fails to prevent meta-overfitting on both datasts.
Revised: Similarly, MetaReg which learns coefficients for ℓ 2regularization fails to prevent meta-overfitting on both datasts.

Original: Lastly, we empirically show that the performance gains mostly come from the task augmentation with
Revised: Lastly, we empirically show that the performance gains mostly come from the task augmentation with

Original: Meta-Interpolation, rather than from bilevel optimization or the introduction of extra parameters withthe set function.
Revised: Meta-Interpolation, rather than from bilevel optimization or the introduction of extra parameters withthe set function.

Original: As shown in Table 1 and 2, ProtoNet+ST, which is Meta Interpolation but trainedwithout any task augmentation, significantly degrades the performance of ProtoNet on Mini-ImageNetand CIFAR-100-FS.
Revised: As shown in Table 1 and 2, ProtoNet+ST, which is Meta Interpolation but trainedwithout any task augmentation, significantly degrades the performance of ProtoNet on Mini-ImageNetand CIFAR-100-FS.

Original: On the other datasets, the ProtoNet+ST obtains marginal improvement or largely
Revised: On the other datasets, the ProtoNet+ST obtains marginal improvement or largely

Original: [Figure]
Revised: [Figure]

Original: underperforms the other baselines.
Revised: underperforms the other baselines.

Original: Thus, the task augmentation strategy of interpolating two supportsets with the set function φ λ is indeed crucial for tackling the few-task meta-learning problem.
Revised: Thus, the task augmentation strategy of interpolating two supportsets with the set function φ λ is indeed crucial for tackling the few-task meta-learning problem.

Original: [Table]
Revised: [Table]

Original: Ablation Study We further perform ablation studies to verifythe effectiveness of each component of Meta-Interpolation.
Revised: Ablation Study We further perform ablation studies to verifythe effectiveness of each component of Meta-Interpolation.

Original: In Table 3, we show experimental results on the ESC-50 dataset whileremoving various components of our model.
Revised: In Table 3, we show experimental results on the ESC-50 dataset whileremoving various components of our model.

Original: Firstly, we train ourmodel without any task interpolation but keep the set functionφ λ , denoted as w/o Interpolation.
Revised: Firstly, we train ourmodel without any task interpolation but keep the set functionφ λ , denoted as w/o Interpolation.

Original: The model without task interpolation significantly underperforms the full task-augmentationmodel, Meta-Interpolation, which shows that the improvements come from task interpolation, ratherthan the extra parameters introduced by the set encoding layer.
Revised: The model without task interpolation significantly underperforms the full task-augmentationmodel, Meta-Interpolation, which shows that the improvements come from task interpolation, ratherthan the extra parameters introduced by the set encoding layer.

Original: Moreover, bilevel optimizationis shown to be effective for estimating λ , which are the parameters of the set function.
Revised: Moreover, bilevel optimizationis shown to be effective for estimating λ , which are the parameters of the set function.

Original: Jointlytraining ProtoNet and the set the function without bilevel optimization, denoted as w/o Bilevel,largely degrades the test accuracy by 15% .
Revised: Jointlytraining ProtoNet and the set the function without bilevel optimization, denoted as w/o Bilevel,largely degrades the test accuracy by 15% .

Original: Lastly, we remove the loss L singleton ( λ, θ, T train t ) for inneroptimization in Eq.
Revised: Lastly, we remove the loss L singleton ( λ, θ, T train t ) for inneroptimization in Eq.

Original: 6, denoted as w/o L singleton ( λ, θ, T train t ) .
Revised: 6, denoted as w/o L singleton ( λ, θ, T train t ) .

Original: This hurts the generalization performance since it decreases the diversity of tasks and causes inconsistency between meta-training andmeta-testing, since we do not perform any interpolation for support sets at meta-test time.
Revised: This hurts the generalization performance since it decreases the diversity of tasks and causes inconsistency between meta-training andmeta-testing, since we do not perform any interpolation for support sets at meta-test time.

Original: Visualization In Figure 3, we present the t-SNE [41] visualizations of the original and interpolatedtasks with MLTI and Meta-Interpolation, respectively.
Revised: Visualization In Figure 3, we present the t-SNE [43] visualizations of the original and interpolatedtasks with MLTI and Meta-Interpolation, respectively.

Original: Following Yao et al.
Revised: Following Yao et al.

Original: [48], we sample threeoriginal tasks from NCI and ESC-50 dataset, where each task is a two-way five-shot and five-way fiveshot classification problem, respectively.
Revised: [50], we sample threeoriginal tasks from NCI and ESC-50 dataset, where each task is a two-way five-shot and five-way fiveshot classification problem, respectively.

Original: The tasks are interpolated with MLTI or Meta-Interpolationto construct 300 additional tasks and represented as a set of all class prototypes.
Revised: The tasks are interpolated with MLTI or Meta-Interpolationto construct 300 additional tasks and represented as a set of all class prototypes.

Original: To visualize theprototypes, we first perform Principal Component Analysis [11] (PCA) to reduce the dimension ofeach prototype.
Revised: To visualize theprototypes, we first perform Principal Component Analysis [13] (PCA) to reduce the dimension ofeach prototype.

Original: The first 50 principal components are then used to compute the t-SNE visualizations.
Revised: The first 50 principal components are then used to compute the t-SNE visualizations.

Original: As shown in Figure 3b and 3d, Meta-Interpolation successfully learns an expressive neural set functionthat densifies the task distribution.
Revised: As shown in Figure 3b and 3d, Meta-Interpolation successfully learns an expressive neural set functionthat densifies the task distribution.

Original: The task augmentations with MLTI, however, do not cover a wideembedding space as shown in Figure 3a and 3c as the mixup strategy allows to generate tasks only onthe simplex defined by the given set of tasks.
Revised: The task augmentations with MLTI, however, do not cover a wideembedding space as shown in Figure 3a and 3c as the mixup strategy allows to generate tasks only onthe simplex defined by the given set of tasks.

Original:  Conclusion
Revised:  Conclusion

Original: We proposed a novel domain-agnostic task augmentation method, Meta Interpolation, to tackle themeta-overfitting problem in few-task meta-learning.
Revised: We proposed a novel domain-agnostic task augmentation method, Meta Interpolation, to tackle themeta-overfitting problem in few-task meta-learning.

Original: Specifically, we leveraged expressive neuralset functions to interpolate a given set of tasks and trained the interpolating function using bileveloptimization, so that the meta-learner trained with the augmented tasks generalizes to meta-validationtasks.
Revised: Specifically, we leveraged expressive neuralset functions to interpolate a given set of tasks and trained the interpolating function using bileveloptimization, so that the meta-learner trained with the augmented tasks generalizes to meta-validationtasks.

Original: Since the set function is optimized to minimize the loss on the validation tasks, it allows us totailor the task augmentation strategy to each specific domain.
Revised: Since the set function is optimized to minimize the loss on the validation tasks, it allows us totailor the task augmentation strategy to each specific domain.

Original: We empirically validated the efficacyof our proposed method on various domains, including image classification, chemical propertyprediction, text classification, and automatic speech recognition, showing that Meta-Interpolationachieves consistent improvements across all domains.
Revised: We empirically validated the efficacyof our proposed method on various domains, including image classification, chemical propertyprediction, text classification, and automatic speech recognition, showing that Meta-Interpolationachieves consistent improvements across all domains.

Original: This is in stark contrast to the baselines whichimprove generalization in certain domains but degenerate performance in others.
Revised: This is in stark contrast to the baselines whichimprove generalization in certain domains but degenerate performance in others.

Original: Furthermore, ourtheoretical analysis shed light on how Meta-Interpolation regularizes the meta-learner and improvesits generalization performance.
Revised: Furthermore, ourtheoretical analysis shed light on how Meta-Interpolation regularizes the meta-learner and improvesits generalization performance.

Original: Lastly, we discussed the limitation of our method in Appendix C.
Revised: Lastly, we discussed the limitation of our method in Appendix C.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: [Yes]
Revised: [Yes](

Original: (b) Did you describe the limitations of your work?
Revised: b ) Did you describe the limitations of your work?

Original: [Yes] Please see limitation fromSection{+from Section 
+}
Revised: [Yes] Please see limitation {+from Section 
+}

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work

Original: [N/A]
Revised: N A

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you specify all the training details (e g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?

Original: [Yes] We run experiments 5 times with different random seedsand show the 95% confidence interval on Table 1 and 2.
Revised: [Yes] We run experiments 5 times with different random seeds and show the 95% confidence interval on Table 1 and 2.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [No]
Revised: [No]

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We describe the
Revised: [Yes] We describe the

Original: datasets and cite the creators in Section 5.
Revised: datasets and cite the creators in Section 5.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: AppendixA Organization
Revised: Appendix

Revised: A Organization

Original: The appendix is organized as follows:
Revised: The appendix is organized as follows:

Original: In Section B, we provide proofs of the theorem and propositionsin Section 4.
Revised: In Section B, we provide proofs of the theorem and propositionsin Section 4.

Original: We dicsucss the limitations of our method in Section C.
Revised: We discucss the limitations of our method in Section C.

Original: In Section D, we providedetailed descriptions of the experimental setup used in the main paper together with the exact datasplits for meta-training, meta-validation and meta-testing for all the datasets used.
Revised: In Section D, we providedetailed descriptions of the experimental setup used in the main paper together with the exact datasplits for meta-training, meta-validation and meta-testing for all the datasets used.

Original: Additionally,we specify the exact architecture of the Prototypical Networks used for all the experiments andfurther describe the Set Transformer in detail in Section D.3.
Revised: Additionally,we specify the exact architecture of the Prototypical Networks used for all the experiments andfurther describe the Set Transformer in detail in Section D.3.

Original: All hyperparameters are detailed in
Revised: All hyperparameters are detailed in

Original: Section D.4.
Revised: Section D.4.

Original: Finally, in Section E, we explore the effect of the number of tasks for interpolation onthe generalization performance of Meta-Interpolation.
Revised: Finally, in Section E, we explore the effect of the number of tasks for interpolation onthe generalization performance of Meta-Interpolation.

Original: B ProofsB.1 Proof of Theorem 1
Revised: B ProofsB.1 Proof of Theorem 1

Original: Proof.
Revised: Proof.

Original: Define ϕ :=
Revised: Define ϕ :=

Original: ϕ lθ l = f lθ l ◦ · · · ◦ f 1 θ 1 , and g :=
Revised: ϕ lθ l = f lθ l ◦ · · · ◦ f 1 θ 1 , and g :=

Original: f Lθ L ◦ · · · ◦ f l +1 θ l +1 .
Revised: f Lθ L ◦ · · · ◦ f l +1 θ l +1 .

Original: Define the dimensionality asϕ ( x st,i ) ∈ R d , and g ( ϕ ( x st,i ))
Revised: Define the dimensionality asϕ ( x st,i ) ∈ R d , and g ( ϕ ( x st,i ))

Original: ∈ R d L .
Revised: ∈ R D .

Original: From the definition, since we use ˆ f θ,λ in both training andtesting time, we have
Revised: From the definition, since we use ˆ f θ,λ in both training andtesting time, we have

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: In the analysis of the loss functions, without the loss of generality, we can set the permutation σ ton { 1 , . . .
Revised: In the analysis of the loss functions, without the loss of generality, we can set the permutation σ ton { 1 , . . .

Original: , K } to be the identity since the every combination can be realized by one permutation σinstead of the two permutations σ t , σ t ′ .
Revised: , K } to be the identity since the every combination can be realized by one permutation σinstead of the two permutations σ t , σ t ′ .

Original: Therefore, using the definition of the ˆ T t,t ′ , we can write thecorresponding loss by
Revised: Therefore, using the definition of the ˆ T t,t ′ , we can write thecorresponding loss by

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: This can be rewritten as:
Revised: This can be rewritten as:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:  st 1 { y st,i = k } =   (cid:88) ( x st ′ ,j ,y st ′ ,j ) ∈D st ′ 1 { y st ′ ,
Revised: ∈D st 1 { y st,i = k } =   (cid:88) ( x st ′ ,j ,y st ′ ,j ) ∈D st ′ 1 { y st ′ ,i =

Original: i =

Original: σ ( k ) }     (cid:88) ( x st,i ,y st,i ) ∈D
Revised: σ ( k ) }     (cid:88) ( x st,i ,y st,i ) ∈D

Original: st 1 { y st,i = k } 
Revised: st 1 { y st,i = k } 

Original:  =
Revised:  =

Original: N t ′ ,k N t,k
Revised: N t ′ ,k N t,k

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original:  .
Revised:  .

Original: Define the set
Revised: Define the set

Original: It,k
Revised: I

Original: [Equation]
Revised: [Equation]

Original: = { i : y st,i = k } .
Revised: = { i : y st,i = k } .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Summarizing the computation so far, we have that
Revised: Summarizing the computation so far, we have that

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: log [Equation] ,σ ( k ) | I t,k | (cid:88) i ∈
Revised: ( k ) | I t,k | (cid:88) i ∈

Original: I t,k
Revised: I t,k

Original: ( g ◦ φ λ )( { ϕ ( x st,i ) , ϕ ( x st ′ ,j ) } ) with
Revised: ( g ◦ φ λ )( { ϕ ( x st,i ) , ϕ ( x st ′ ,j ) } ) with

Original: I t,k :=
Revised: I t,k =

Original: { i : y st,i = k } .
Revised: i : y st,i = k } .

Original: We now analyze the relationship of φ λ ( { h, h ′ } ) and φ λ ( { h } ) .
Revised: We now analyze the relationship of φ λ ( { h, h ′ } ) and φ λ ( { h } ) .

Original: From this definition, we firstcompute φ λ ( { h, h ′ } ) as follows.
Revised: From this definition, we firstcompute φ λ ( { h, h ′ } ) as follows.

Original: By writing σ 1 = softmax( √ d − 1 Q { h,h ′ }( K { h,h ′ }) ⊤ ) andσ 2 = softmax( √ d − 1 Q 2 ( K { h,h ′ }) ⊤ ) ,
Revised: By writing σ 1 = softmax( √ d − 1 Q { h,h ′ }( K { h,h ′ }) ⊤ ) andσ 2 = softmax( √ d − 1 Q 2 ( K { h,h ′ }) ⊤ ) ,

Original: [Equation]
Revised: [Equation]

Original: Here, by writing p 1 = p ( t,t ′ ,i,j ), p ′ 1 = ˜ p ( t,t ′ ,i,j ), and p 2 = p ( t,t ′ ,i,j ), we can rewrite that[Equation] [ p
Revised: Here, by writing p 1 = p ( t,t ′ ,i,j ), p ′ 1 = ˜ p ( t,t ′ ,i,j ), and p 2 = p ( t,t ′ ,i,j ), we can rewrite that

Original: [Equation]
Revised: [Equation]

Original: 1 − p

Original: [Equation]
Revised: [Equation]

Original: ] ∈ R
Revised: ] ∈ R

Original: [Equation]
Revised: [Equation]

Original: Thus, by letting ¯ p = p 2 p 1 + (1 − p 2 ) p ′ 1 ∈ [0 , 1] , σ [Equation] σ 
Revised: Thus, by letting ¯ p = p 2 p 1 + (1 − p 2 ) p ′ 1 ∈ [0 , 1] , σ 2 σ 1 = [ p

Original: [Equation]
Revised: [Equation]

Original: = [ p [Equation] p 
Revised: − p 2 ]

Original: [Equation]
Revised: [Equation]

Original: Using these,
Revised: Using these,

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Thus, by letting W =
Revised: Thus, by letting W =

Original: ( W V 1 W V 2 )
Revised: ( W V 1 W V 2 )

Original: ⊤ ∈ R d × d and b = ( b V 1 W V 2 + b V 2 )
Revised: ⊤ ∈ R d × d and b = ( b V 1 W V 2 + b V 2 )

Original: ⊤ ∈ R d , and by definingα = 1 − ¯ p , we have that
Revised: ⊤ ∈ R d , and by definingα = 1 − ¯ p , we have that

Original: [Equation]
Revised: [Equation]

Original: For φ λ ( { h } ) , similarly, by writing σ
Revised: For φ λ ( { h } ) , similarly, by writing σ

Original: [Equation]
Revised: [Equation]

Original: andσ
Revised: and

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Here, since σ 1 = softmax( √ d − 1 Q { h } 1 ( K { h }) ⊤ ) ∈ R and σ 2 = softmax( √ d − 1 Q 2 ( K { h }) ⊤ ) ∈ R , we have σ 1 = σ 2 = 1 and thus, φ λ ( { h } ) =
Revised: Here, since σ 1 = softmax( √ d − 1 Q { h } 1 ( K { h }) ⊤ ) ∈ R and σ 2 = softmax( √ d − 1 Q 2 ( K { h }) ⊤ ) ∈ R , we have σ 1 = σ 2 = 1 and thus, φ λ ( { h } ) =

Original: Wh + b.
Revised: Wh + b.

Original: Therefore, we have that
Revised: Therefore, we have that

Original: [Equation]
Revised: [Equation]

Original: where W and b does not depend on the ( h, h ′ ) .
Revised: where W and b does not depend on the ( h, h ′ ) .

Original: Using these and by defining h t,i = ϕ
Revised: Using these and by defining h t,i = ϕ

Original: ( x st,i ) , we have that
Revised: ( x st,i ) , we have that

Original: [Equation]
Revised: [Equation]

Original: With these preparation, we are now ready to prove the regularization form.
Revised: With these preparation, we are now ready to prove the regularization form.

Original: Fix t, t ′ and write α ij =
Revised: Fix t, t ′ and write α ij =

Original: α ( t,t ′ ) ij .
Revised: α ( t,t ′ ) ij .

Original: Define a vector α such that α = ( α ij ) i,j .
Revised: Define a vector α such that α = ( α ij ) i,j .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Then, from the results of the calculations above, we have that
Revised: Then, from the results of the calculations above, we have that

Original: [Equation]
Revised: [Equation]

Original: Using the assumptions that ∂ r g ( z ) = 

Revised: Using the assumptions that ∂ r g ( z ) =

Original: for all r ≥ 2 , for any J , the J -th approximation of ˆ c k
Revised: for all r ≥ 2 , for any J , the J -th approximation of ˆ c k

Original: ( α ) isgiven by
Revised: ( α ) isgiven by

Original: [Equation]
Revised: [Equation]

Original: Let  γ > 1 to be set later and define
Revised: Let ¯ γ ≥ 1 to be set later and define

Original: [Equation]
Revised: [Equation]

Original: Then,ˆ
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Define c
Revised: Define c

Original: =
Revised: =

Original: ( c 1 , . .
Revised: ( c 1 , . .

Original: . , c K ) and ∆ =
Revised: . , c K ) and ∆ =

Original: (∆ 1 , . . .
Revised: (∆ 1 , . . .

Original: , ∆ K ) .
Revised: , ∆ K ) .

Original: For any given t , define
Revised: For any given t , define

Original: [Equation]
Revised: [Equation]

Original: Then, we have that
Revised: Then, we have that

Original: [Equation]
Revised: [Equation]

Original: The J -th approximation of L mix ( λ, θ, ˆ T t,t ′ ) is given by
Revised: The J -th approximation of L mix ( λ, θ, ˆ T t,t ′ ) is given by

Original: [Equation]
Revised: [Equation]

Original: where φ ( j ) is the j -th order derivative of φ : α
Revised: where φ ( j ) is the j -th order derivative of φ : γ (

Original: (cid:55)→ L t ( c + γ ∆) .
Revised: cid:55)→ L t ( c + γ ∆) .

Original: Here, for any j ∈ N + , by defining
Revised: Here, for any j ∈ N + , by defining

Original: [Equation]
Revised: [Equation]

Original: = [Equation] ∈
Revised: KD

Original: [Equation]
Revised: [Equation]

Original: KD ,

Original: [Equation]
Revised: [Equation]

Original: Then, by using the vectorization of the tensor vec[ ∂ j L t ( b  )] ∈ R ( KD ) j , we can rewrite this equationas
Revised: Then, by using the vectorization of the tensor vec[ ∂ j L t ( b ′ )] ∈ R ( KD ) j , we can rewrite this equationas

Original: [Equation]
Revised: [Equation]

Original: where ∆ ⊗ j = ∆ ⊗ ∆ ⊗ · · · ⊗ ∆ ∈ R ( KD ) j .
Revised: where ∆ ⊗ j = ∆ ⊗ ∆ ⊗ · · · ⊗ ∆ ∈ R ( KD ) j .

Original: By combining these with  γ = 1 ,
Revised: By combining these with ¯ γ = 1 ,

Original: [Equation]
Revised: L mix ( λ, θ, ˆ T t,t ′ ) J = L singleton ( λ, θ ; T t ) +

Original:  where ∆ 
Revised: J (cid:88) j =  j ! vec[ ∂ j L t ( c )] ⊤ ∆ ⊗ j , where ∆ k = | I t ′ ,σ ( k ) | (cid:88)

Original: [Equation]
Revised: j ∈ I t ′ ,σ ( k ) | I t,k | (cid:88) i ∈ I t,k α ( t,t ′ ) ij ∂g ( Wh t,i + b ) W ( h t ′ ,j − h t,i ) .

Original: B.2 Proof of Proposition
Revised: B.2 Proof of Proposition

Original: Proof.
Revised: Proof.

Original: We now apply our general regularization form theorem to this special case from the previous paper [48] on the ProtoNet loss.
Revised: We now apply our general regularization form theorem to this special case from the previous paper [50] on the ProtoNet loss.

Original: In this special case, we have that ϕ ( x ) = x , W
Revised: In this special case, we have that ϕ ( x ) = x , W

Original: =
Revised: =

Original: I, b = 0 , h t,i = x st,i , g ( x ) =
Revised: I, b = 0 , h t,i = x st,i , g ( x ) =

Original: x ⊤ θ, and
Revised: x ⊤ θ, and

Original: Lt
Revised: L

Original: [Equation]
Revised: [Equation]

Original: n  (cid:88) 
Revised: n n (cid:88) i =

Original: [Equation]
Revised: [Equation]

Original:  = 
Revised: Here, for c = ( c 1 , c 2 ) with

Original: [Equation]
Revised: [Equation]

Original: Here, for c =( c 1 , c 2 ) with
Revised: =

Original: [Equation]
Revised: [Equation]

Revised: t,(cid:88) (

Original: [Equation]
Revised: [Equation]

Original: (10) and
Revised: (10) andc

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Revised: N t

Original: [Equation]
Revised: [Equation]

Original: (cid:88) 
Revised: (cid:88) ( x

Original: [Equation]
Revised: [Equation]

Original: (11) we recover that
Revised: (11) we recover that

Original: L

Original: [Equation]
Revised: [Equation]

Original: Thus, to instantiate our general theorem to this special case, we only need to compute the ∂ j L t ( c ) and ∂g ( h ⊤ t,i W + b ) up to the second order approximation.
Revised: Thus, to instantiate our general theorem to this special case, we only need to compute the ∂ j L t ( c ) and ∂g ( h ⊤ t,i W + b ) up to the second order approximation.

Original: [Equation]
Revised: [Equation]

Original: Define z t,i := ⟨ x qt,i − ( c ′ 1 + c ′ 2 ) / 2 , θ ⟩ .
Revised: Define z t,i := ⟨ x qt,i − ( c ′ 1 + c ′ 2 ) / 2 , θ ⟩ .

Original: Since c 1 = θ ⊤ c ′ 1 and c 2 = θ ⊤ c ′ 2 by Equation 10 and 11,
Revised: Since c 1 = θ ⊤ c ′ 1 and c 2 = θ ⊤ c ′ 2 by Equation 10 and 11,

Original: [Equation]
Revised: [Equation]

Original: Thus, we have L t ( c )
Revised: Thus, we have L t ( c )

Original: = 1 n (cid:80) ni =
Revised: = 1 n (cid:80) ni =

Original: [Equation]
Revised: [Equation]

Original: +exp( z t,i ) .
Revised: +exp( z t,i ) .

Original: Then, ∂L
Revised: Then, ∂L

Original: [Equation]
Revised: [Equation]

Original: ( c )
Revised: ( c )

Original: [Equation]
Revised: [Equation]

Original: Similarly, ∂L
Revised: Similarly, ∂L

Original: [Equation]
Revised: [Equation]

Original: For the second order, by defining the logistic function ψ ( z
Revised: For the second order, by defining the logistic function ψ ( z

Original: [Equation]
Revised: [Equation]

Original: ∂L
Revised: ∂L

Original: [Equation]
Revised: [Equation]

Original: 1 +
Revised: 1 +

Original: [Equation]
Revised: [Equation]

Original: Similarly, ∂L
Revised: Similarly, ∂L

Original: [Equation]
Revised: [Equation]

Original: c [Equation] Therefore, the second approximation of E t ′ ,σ [ L mix ( λ, θ, ˆ T t,t ′ )] is
Revised: Therefore, the second approximation of E t ′ ,σ [ L mix ( λ, θ, ˆ T t,t ′ )] is

Original: L singleton ( λ, θ ; T t ) + E t ′ ,σ   2 (cid:88)
Revised: L singleton ( λ, θ ; T t ) + E t ′ ,σ   2 (cid:88)

Original: [Equation]
Revised: [Equation]

Original: where ∆ =
Revised: where ∆ =

Original: [∆ ⊤ 1 , ∆ ⊤ 2 ] ⊤ with
Revised: [∆ ⊤ 1 , ∆ ⊤ 2 ] ⊤ with

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: For the first order term,
Revised: For the first order term,

Original: [Equation]
Revised: [Equation]

Original: where δ t,
Revised: where δ t,

Original: [Equation]
Revised: [Equation]

Revised: =

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: For the second order term,
Revised: For the second order term,

Original: [Equation]
Revised: [Equation]

Original: Since E t ′ ,σ [ δ t,t ′ ,σ ] = 0 with the α being balanced, the second approximation of E t ′ ,σ [ L mix ( λ, θ, ˆ T t,t ′ )]becomes:
Revised: Since E t ′ ,σ [ δ t,t ′ ,σ ] = 0 with the α being balanced, the second approximation of E t ′ ,σ [ L mix ( λ, θ, ˆ T t,t ′ )]becomes:

Original: [Equation]
Revised: [Equation]

Original: t,i ) (cid:33) θ
Revised: t,i ) (cid:33) θ

Original: ⊤ E t ′ ,σ (cid:2) δ t,t ′ ,σ δ ⊤ t,t ′ ,σ (cid:3) θ.
Revised: ⊤ E t ′ ,σ (cid:2) δ t,t ′ ,σ δ ⊤ t,t ′ ,σ (cid:3) θ.

Original: B.3 Proof of Proposition
Revised: B.3 Proof of Proposition

Original: Proof.
Revised: Proof.

Original: Let ξ 1 , . . .
Revised: Let ξ 1 , . . .

Original: , ξ n be independent uniform random variables taking values in {− 1 , 1 } ; i.e.,
Revised: , ξ n be independent uniform random variables taking values in {− 1 , 1 } ; i.e.,

Original: Rademacher variables.
Revised: Rademacher variables.

Original: We first bound the empirical Rademacher complexity as follows:
Revised: We first bound the empirical Rademacher complexity as follows:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where Σ † denotes the Moore–Penrose inverse of Σ .
Revised: where Σ † denotes the Moore–Penrose inverse of Σ .

Original: Using this bound on the empirical Rademachercomplexity, we now bound the Rademacher complexity as follows:
Revised: Using this bound on the empirical Rademachercomplexity, we now bound the Rademacher complexity as follows:

Original: [Equation]
Revised: [Equation]

Original: n (cid:88) i =
Revised: n (cid:88) i =

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: ′ ) ≤
Revised: ′ ) ≤

Original: [Equation]
Revised: [Equation]

Original: n (cid:118)(cid:117)(cid:117)(cid:116) n (cid:88)
Revised: n (cid:118)(cid:117)(cid:117)(cid:116) n (cid:88)

Original: [Equation]
Revised: [Equation]

Revised: x

Revised: [Equation]

Revised: [Equation]

Original:  (cid:88)
Revised: n (cid:88)

Revised: i

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:118)(cid:117)(cid:117)(cid:116)
Revised: (cid:118)(cid:117)(cid:117)(cid:116)

Original: n (cid:88)
Revised: n (cid:88)

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Revised: (cid:118)(cid:117)(cid:117)(cid:116)

Original: n (cid:88) 
Revised: n (cid:88) i

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: C Limitation
Revised: C Limitation

Original: Although we have shown promising results in various domains, our method requires extra computation for bilevel optimization to estimate λ , the parameters of the set function φ λ .
Revised: Although we have shown promising results in various domains, our method requires extra computation for bilevel optimization to estimate λ , the parameters of the set function φ λ .

Original: Reducing thecomputational cost for bilevel optimization will be important for future work.
Revised: Reducing thecomputational cost for bilevel optimization will be important for future work.

Original: D Experimental Setup
Revised: D Experimental SetupD.1 Dataset Description

Original: D.1 Dataset Description

Original: Metabolism We use the following split for the metabolism dataset:
Revised: Metabolism We use the following split for the metabolism dataset:

Original: Meta-Train = { CYP1A2_Veith , CYP3A4_Veith , CYP2C9_Substrate_CarbonMangels }
Revised: Meta-Train = { CYP1A2_Veith , CYP3A4_Veith , CYP2C9_Substrate_CarbonMangels }

Original: Meta-Validation = { CYP2D6_Veith , CYP2D6_Substrate_CarbonMangels }
Revised: Meta-Validation = { CYP2D6_Veith , CYP2D6_Substrate_CarbonMangels }

Original: Meta-Test = { CYP2C19_Veith , CYP2C9_Veith , CYP3A4_Substrate_CarbonMangels }
Revised: Meta-Test = { CYP2C19_Veith , CYP2C9_Veith , CYP3A4_Substrate_CarbonMangels }

Original: Following Yao et al.
Revised: Following Yao et al.

Original: [48], we balance each subdataset by selecting 1000 samples from each subdataset.
Revised: [50], we balance each subdataset by selecting 1000 samples from each subdataset.

Original: Each data sample is processed by extracting a 1024-bit fingerprint feature from the SMILESrepresentation of each chemical compound using the RDKit [13] library.
Revised: Each data sample is processed by extracting a 1024-bit fingerprint feature from the SMILESrepresentation of each chemical compound using the RDKit [15] library.

Original: Tox

Revised: Tox

Original: We use the following split for the metabolism dataset 1 :
Revised: We use the following split for the metabolism dataset 1 :

Original: Meta-Train = { NR-AR , NR-AR-LBD , NR-AhR , NR-Aromatase , NR-ER , NR-ER-LBD } Meta-Validation = { NR-PPAR-gamma , SR-ARE } Meta-Test
Revised: Meta-Train = { NR-AR , NR-AR-LBD , NR-AhR , NR-Aromatase , NR-ER , NR-ER-LBD } Meta-Validation = { NR-PPAR-gamma , SR-ARE } Meta-Test

Original: =
Revised: =

Original: { SR-ATAD5 , SR-HSE , SR-MMP , SR-p53 }
Revised: { SR-ATAD5 , SR-HSE , SR-MMP , SR-p53 }

Original: NCI We download the dataset from the github repository 2 and use the following splits for metatraining, meta-validation, and meta-testing:
Revised: NCI We download the dataset from the github repository 2 and use the following splits for metatraining, meta-validation, and meta-testing:

Original: Meta-Train = { 41 , 47 , 83 ,
Revised: Meta-Train = { 41 , 47 , 83 ,

Original: Meta-Validation = { 81 , 145 } Meta-Test = { 1 , 33 , 123 }
Revised: Meta-Validation = { 81 , 145 } Meta-Test = { 1 , 33 , 123 }

Original: GLUE-SciTail We use Hugging Face Datasets library [23] to download MNLI, QNLI, SNLI, RTE,and SciTail datasets and tokenize it ELECTRA tokenizer with setting maximum length to 128.
Revised: GLUE-SciTail We use Hugging Face Datasets library [25] to download MNLI, QNLI, SNLI, RTE,and SciTail datasets and tokenize it ELECTRA tokenizer with setting maximum length to 128.

Original: Welist meta-train, meta-validation, and meta-test split as follows:
Revised: Welist meta-train, meta-validation, and meta-test split as follows:

Original: Meta-Train = { MNLI , QNLI } Meta-Validation
Revised: Meta-Train = { MNLI , QNLI } Meta-Validation

Original: =
Revised: =

Original: SNLI , RTE } Meta-Test = { SciTail } ESC-50 We download ESC-50 dataset [32] from the github repository
Revised: SNLI , RTE } Meta-Test = { SciTail } ESC-50 We download ESC-50 dataset [34] from the github repository

Original: and use the meta-training,meta-validation, and meta-test split as follows:
Revised: and use the meta-training,meta-validation, and meta-test split as follows:

Original: Meta-train set: dog , rooster , pig , cow , frog , cat , hen , insects , sheep , crow , rain , sea_waves , crackling_fire , crickets , chirping_birds , water_drops , wind , pouring_water , toilet_flush , thunderstorm Meta-validation set:
Revised: Meta-train set:dog , rooster , pig , cow , frog , cat , hen , insects , sheep , crow , rain , sea_waves , crackling_fire , crickets , chirping_birds , water_drops , wind , pouring_water , toilet_flush , thunderstorm Meta-validation set:

Original: crying_baby , sneezing , clapping , breathing , coughing , footsteps , laughing , brushing_teeth , snoring , drinking_sipping , door_wood_knock , mouse_click , keyboard_typing , door_wood_creaks , can_opening Meta-test set:
Revised: crying_baby , sneezing , clapping , breathing , coughing , footsteps , laughing , brushing_teeth , snoring , drinking_sipping , door_wood_knock , mouse_click , keyboard_typing , door_wood_creaks , can_opening Meta-test set:

Original: washing_machine , vacuum_cleaner , clock_alarm , clock_tick , glass_breaking , helicopter , chainsaw , siren , car_horn , engine , train , church_bells , airplane , fireworks , hand_saw
Revised: washing_machine , vacuum_cleaner , clock_alarm , clock_tick , glass_breaking , helicopter , chainsaw , siren , car_horn , engine , train , church_bells , airplane , fireworks , hand_saw

Original: RMNIST Following Yao et al.
Revised: RMNIST Following Yao et al.

Original: [48], we create RMNIST dataset by changing the size, color, andangle of the images from the original MNIST dataset.
Revised: [50], we create RMNIST dataset by changing the size, color, andangle of the images from the original MNIST dataset.

Original: To be specific, we merge training and test splitof MNIST and randomly select 5,600 images for each class and split into 56 subdatasets where eachclass has 100 examples.
Revised: To be specific, we merge training and test splitof MNIST and randomly select 5,600 images for each class and split into 56 subdatasets where eachclass has 100 examples.

Original: We only choose 16/6/10 subdatasets for meta-train, meta-validation, andmeta-test split, respectively and do not use the rest of them.
Revised: We only choose 16/6/10 subdatasets for meta-train, meta-validation, andmeta-test split, respectively and do not use the rest of them.

Original: Each subdatset with the correspondingcomposition of image transformations is considered a distinct task.
Revised: Each subdatset with the correspondingcomposition of image transformations is considered a distinct task.

Original: Following are the specific splits.
Revised: Following are the specific splits.

Original: Meta-train set:( red , full , 90 ◦ ) , ( indigo , full , 0 ◦ ) , ( blue , full , 270 ◦ ) , ( orange , half , 270 ◦ ) , ( green , full , 90 ◦ ) , ( green , full , 270 ◦ ) , ( orange , full , 180 ◦ ) , ( red , full , 180 ◦ ) , ( green , full , 0 ◦ ) , ( orange , full , 0 ◦ ) , ( violet , full , 270 ◦ ) , ( orange , half , 90 ◦ ) , ( violet , half , 180 ◦ ) , ( orange , full , 90 ◦ ) , ( violet , full , 180 ◦ ) , ( blue , full , 90 ◦ )
Revised: Meta-train set:( red , full , 90 ◦ ) , ( indigo , full , 0 ◦ ) , ( blue , full , 270 ◦ ) , ( orange , half , 270 ◦ ) , ( green , full , 90 ◦ ) , ( green , full , 270 ◦ ) , ( orange , full , 180 ◦ ) , ( red , full , 180 ◦ ) , ( green , full , 0 ◦ ) , ( orange , full , 0 ◦ ) , ( violet , full , 270 ◦ ) , ( orange , half , 90 ◦ ) , ( violet , half , 180 ◦ ) , ( orange , full , 90 ◦ ) , ( violet , full , 180 ◦ ) , ( blue , full , 90 ◦ )

Original: Meta-validation set:( indigo , half , 270 ◦ ) , ( blue , full , 0 ◦ ) , ( yellow , half , 180 ◦ ) , ( yellow , half , 0 ◦ ) , ( yellow , half , 90 ◦ ) , ( violet , half , 0 ◦ )
Revised: Meta-validation set:( indigo , half , 270 ◦ ) , ( blue , full , 0 ◦ ) , ( yellow , half , 180 ◦ ) , ( yellow , half , 0 ◦ ) , ( yellow , half , 90 ◦ ) , ( violet , half , 0 ◦ )

Original: Meta-test set:( yellow , full , 270 ◦ ) , ( red , full , 0 ◦ ) , ( blue , half , 270 ◦ ) , ( blue , half , 0 ◦ ) , ( blue , half , 180 ◦ ) , ( red , half , 270 ◦ ) , ( violet , full , 90 ◦ ) , ( blue , half , 90 ◦ ) , ( green , half , 270 ◦ ) , ( red , half , 90 ◦ )
Revised: Meta-test set:( yellow , full , 270 ◦ ) , ( red , full , 0 ◦ ) , ( blue , half , 270 ◦ ) , ( blue , half , 0 ◦ ) , ( blue , half , 180 ◦ ) , ( red , half , 270 ◦ ) , ( violet , full , 90 ◦ ) , ( blue , half , 90 ◦ ) , ( green , half , 270 ◦ ) , ( red , half , 90 ◦ )

Original: Mini-ImageNet-S
Revised: Mini-ImageNet-S

Original: Following Yao et al.
Revised: Following Yao et al.

Original: [48], we reduce the number of meta-training tasks bychoosing subset of the original meta-training classes.
Revised: [50], we reduce the number of meta-training tasks bychoosing subset of the original meta-training classes.

Original: We specify the classes used for meta-trainingtasks as follows:
Revised: We specify the classes used for meta-trainingtasks as follows:

Original: n03017168 , n07697537 , n02108915 , n02113712 , n02120079 , n04509417 , n02089867 , n03888605 , n04258138 , n03347037 , n02606052 , n
Revised: n03017168 , n07697537 , n02108915 , n02113712 , n02120079 , n04509417 , n02089867 , n03888605 , n04258138 , n03347037 , n02606052 , n

Original: For meta-validation and meta-testing classes, we use the same classes as the original Mini-ImageNet.
Revised: For meta-validation and meta-testing classes, we use the same classes as the original Mini-ImageNet.

Original: CIFAR-100-FS Similar to Mini-ImageNet-S, we choose 12/16/20 classes for meta-training, metavalidation, meta-test classes.
Revised: CIFAR-100-FS Similar to Mini-ImageNet-S, we choose 12/16/20 classes for meta-training, metavalidation, meta-test classes.

Original: Followings are the specific classes for each split:
Revised: Followings are the specific classes for each split:

Revised: Meta-Train =

Original: [Equation]
Revised: [Equation]

Original: Meta

Original: [Equation]
Revised: [Equation]

Original: Validation
Revised: Meta-Validation =

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Meta-Test
Revised: Meta- Test

Original: [Equation]
Revised: = { 80 , . .

Original: D.2 Prototypical Network Architecture
Revised: D.2 Prototypical Network Architecture

Original: We summarize neural network architectures for ProtoNet on each datasets in Table 4, 5, 6, 7, 8,and 9.
Revised: We summarize neural network architectures for ProtoNet on each datasets in Table 4, 5, 6, 7, 8,and 9.

Original: For Glue-SciTail, we use pretrained ELECTRA-small [6] which we download from Hugging
Revised: For Glue-SciTail, we use pretrained ELECTRA-small [8] which we download from Hugging

Original: Face [46].
Revised: Face [48].

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: D. 
Revised: D.

Original: [Table]
Revised: [Table]

Original: Transformer
Revised: Transformer

Original: We describe Set Transformer [21], φ λ , in more detail.
Revised: We describe Set Transformer [23], φ λ , in more detail.

Original: Let X ∈ R n × d be a stack of n ddimensional row vectors.
Revised: Let X ∈ R n × d be a stack of n ddimensional row vectors.

Original: Let W Q 1 ,j , W K 1 ,j , W V 1 ,j ∈ R d × d k be weight matrices for self-attentionand let b Q 1 ,j , b K 1 ,j , b V 1 ,
Revised: Let W Q 1 ,j , W K 1 ,j , W V 1 ,j ∈ R d × d k be weight matrices for self-attentionand let b Q 1 ,j , b K 1 ,j , b V 1 ,

Original: j ∈ R d
Revised: j ∈ R d

Original: k be bias vectors for j = 1 , . .
Revised: k be bias vectors for j = 1 , . .

Original: For encoding an input X , we compute self-attention as follows:
Revised: For encoding an input X , we compute self-attention as follows:

Original: [Equation]
Revised: [Equation]

Original: where 1 = (1 , . . . , 1) ⊤ ∈ R n is a vector of ones, d h = 4 d k , and softmax is applied for each row.
Revised: where 1 = (1 , . . . , 1) ⊤ ∈ R n is a vector of ones, d h = 4 d k , and softmax is applied for each row.

Original: After self-attention, we add a skip connection with layer normalization [1] as follows:
Revised: After self-attention, we add a skip connection with layer normalization [1] as follows:

Original: [Equation]
Revised: [Equation]

Original: where W 1 ∈ R d h × d h , b 1 ∈ R d h Similarly, we compute another self-attention on top of g 1 ( X ) with
Revised: where W 1 ∈ R d h × d h , b 1 ∈ R d h Similarly, we compute another self-attention on top of g 1 ( X ) with

Original: W Q 2 ,j , W K 2 ,j , W V 2 ,j ∈ R d k × d k weight matrices for self-attention and let b Q 2 ,j , b K 2 ,j , b V 2 ,
Revised: W Q 2 ,j , W K 2 ,j , W V 2 ,j ∈ R d k × d k weight matrices for self-attention and let b Q 2 ,j , b K 2 ,j , b V 2 ,

Original: j ∈ R d k be biasvectors, for j = 1 . . .
Revised: j ∈ R d k be biasvectors, for j = 1 . . .

Original: [Equation]
Revised: [Equation]

Original: After self-attention, we also add a skip connection after the second self-attention with dropout [39].
Revised: After self-attention, we also add a skip connection after the second self-attention with dropout [41].

Original: [Equation]
Revised: [Equation]

Original: where W 2 ∈ R d h × d h , b 2 ∈ R d h
Revised: where W 2 ∈ R d h × d h , b 2 ∈ R d h

Original: After encoding X with two-layers of self-attention, we perform pooling with attention.
Revised: After encoding X with two-layers of self-attention, we perform pooling with attention.

Original: Let S ∈ R 1 × d hbe learnable parameters and W Q 3 ,j , W K 3 ,j , W V 3 ,j ∈ R d h × d
Revised: Let S ∈ R 1 × d hbe learnable parameters and W Q 3 ,j , W K 3 ,j , W V 3 ,j ∈ R d h × d

Original: k be weight matrices for the pooling-attentionand let b Q 3 ,j , b K 3 ,j , b V 3 ,
Revised: k be weight matrices for the pooling-attentionand let b Q 3 ,j , b K 3 ,j , b V 3 ,

Original: j ∈ R d k be bias vectors, for j = 1 , . .
Revised: j ∈ R d k be bias vectors, for j = 1 , . .

Original: We pool ( g 2 ◦ g 1 )( X ) as follows:
Revised: We pool ( g 2 ◦ g 1 )( X ) as follows:

Original: [Equation]
Revised: [Equation]

Original: After pooling, we add another skip connection with dropout as follows:
Revised: After pooling, we add another skip connection with dropout as follows:

Original: [Equation]
Revised: [Equation]

Original: where W 3 ∈ R d h × d h , b 3 ∈ R d h .
Revised: where W 3 ∈ R d h × d h , b 3 ∈ R d h .

Original: Finally, we perform affine-transformation after the pooling asfollows:
Revised: Finally, we perform affine-transformation after the pooling asfollows:

Original: [Equation]
Revised: [Equation]

Original: where W 4 ∈ R d h × d , b 4 ∈ R d .
Revised: where W 4 ∈ R d h × d , b 4 ∈ R d .

Original: To summarize, Set Transformer is the set function φ λ :=
Revised: To summarize, Set Transformer is the set function φ λ :=

Original: g 4 ◦ g 3 ◦ g 2 ◦ g 1 : R n × d → R d that canhandle a set with arbitrary cardinality n ∈ N .
Revised: g 4 ◦ g 3 ◦ g 2 ◦ g 1 : R n × d → R d that canhandle a set with arbitrary cardinality n ∈ N .

Original: D.4 Hyperparameters
Revised: D.4 Hyperparameters

Original: In Table 10 and 11, we summarize all the hyperparameters for each datasets, where MI stands for
Revised: In Table 10 and 11, we summarize all the hyperparameters for each datasets, where MI stands for

Original: Mini-ImageNet-S. For CIFAR-100-FS, we use the same hyperparameters for both 1-hot and 5-shot.
Revised: Mini-ImageNet-S. For CIFAR-100-FS, we use the same hyperparameters for both 1-hot and 5-shot.

Original: [Table]
Revised: [Table]

Original: [Table] 
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: E Effect of Cardinality for Interpolation
Revised: E Effect of Cardinality for Interpolation

Original: [Figure]
Revised: [Figure]

Original: of
Revised: of

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Since the set function φ λ can handle sets of arbitrary cardinality n ∈ N , we plot the test accuracy on the ESC-50dataset with varying set sizes from one to five.
Revised: Since the set function φ λ can handle sets of arbitrary cardinality n ∈ N , we plot the test accuracy on the ESC-50dataset with varying set sizes from one to five.

Original: As shownin Figure 4, for sets of cardinality one, where we do notperform any interpolation, we observe significant degradation in performance.
Revised: As shownin Figure 4, for sets of cardinality one, where we do notperform any interpolation, we observe significant degradation in performance.

Original: This suggests that the interpolation ofinstances is crucial for better generalization.
Revised: This suggests that the interpolation ofinstances is crucial for better generalization.

Original: On the otherhand, for set sizes from two to five, the gain is marginalwith increasing cardinality.
Revised: On the otherhand, for set sizes from two to five, the gain is marginalwith increasing cardinality.

Original: Furthermore, increasing the setsize introduces extra computational overhead.
Revised: Furthermore, increasing the setsize introduces extra computational overhead.

Original: Thus, weset the cardinality to two for task interpolation in all theexperiments. 
Revised: Thus, weset the cardinality to two for task interpolation in all theexperiments. [Table]

Revised: F Additional Experiments

Revised: F.1 First-order MAML

Revised: [Table]

Revised: As stated in introduction, we solely focus on metric based metalearning due to the efficiency and better empirical performanceover the gradient based methods.

Revised: Moreover it is challengingto combine our method with Model-Agnostic Meta-Learning(MAML)

Revised: [10] since it yields a tri-level optimization problemwhich requires to differentiate through second order derivativesand tri-level optimizati is still known to be challengin problem [4, 7].

Revised: Thus, instead of using the original MAML, we haveperformed additional experiments on ESC50 dataset with first-order MAML (FOMAML) whichapproximates the Hessian with a zero matrix.

Revised: As shown in Table 13, the experimental results showthat our Meta-Interpolation with the first-order MAML outperforms MLTI, which again confirms thegeneral effectiveness of our set-based augmentation scheme.

Revised: However, it largely underperforms ouroriginal Meta-Interpolation framework with metric-based meta-learning.

Revised: F.2 Effect of the number of meta-training tasks

Revised: We analyze the effect of the number of meta-training tasks on ESC50 dataset.

Revised: As shown in Table 12,

Revised: Meta-Interpolation consistently outperforms the baselines by large margins, regardless of the numberof the tasks.

Revised: F.3 Location of interpolation

Revised: [Table]

Revised: Contrary to Manifold Mixup, we fix the location of interpolationas shown in Table 10.

Revised: Otherwise we cannot use the same architecture of Set Transformer to interpolate output of different layerssince hidden dimension of each layer is different.

Revised: Moreover, wereport the test accuracy for each layer to be interpolated.

Revised: Interpolating hidden representation of support sets from layer 2, whichis our model used in the main experiments on ESC50 dataset,achieves the best performance.

Revised: F.4 Ablation for set functions

Revised: [Table]

Revised: [Table]

Revised: We have performed additional experiment on ESC50 datasetwith different set function, DeepSets [52] to show the general effectiveness of our method regardless of the set encoding scheme.

Revised: In Table 15, Meta-Interpolation with DeepSets improves thegeneralization performance of ProtoTypical Network and themodel with Set Transformer further boost the performance asa consequence of higher-order pairwise interactions among theset elements via the attention mechanism.

Revised: F.5 Ablation for interpolation strategy

Revised: [Table]

Revised: We empirically verify our interpolation strategy that mixes onlysupport sets.

Revised: We compare our method to various interpolationstrategy including the one mixing a support set with zero meanand unit variance gaussian noise.

Revised: In Table 16, we empiricallyshow that the interpolation strategy which mixes only supportsets outperforms the others.

Revised: Note that interpolating a supportset with gaussian noise works well on ESC50 dataset thoughwe find that it significantly degrades the performance of Prototypical Network on RMNIST, from 75 .

Revised: 35 ± 1 .

Revised: 63 to 69 .

Revised: 60 ± 1 . 60 , which shows a necessity ofinterpolating two support sets.