Original: M ODELLING BIOLOGICAL ASSAYS WITH A DAPTIVE
Revised: M ODELLING BIOLOGICAL ASSAYS WITH A DAPTIVE

Original: D EEP K ERNEL L EARNING
Revised: D EEP K ERNEL L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Due to the signiﬁcant costs of data generation, many prediction tasks within drugdiscovery are by nature few-shot regression (FSR) problems, including accuratemodelling of biological assays.
Revised: Due to the signiﬁcant costs of data generation, many prediction tasks within drugdiscovery are by nature few-shot regression (FSR) problems, including accuratemodelling of biological assays.

Original: Although a number of few-shot classiﬁcation andreinforcement learning methods exist for similar applications,
Revised: Although a number of few-shot classiﬁcation andreinforcement learning methods exist for similar applications,

Original: we ﬁnd relativelyfew FSR methods meeting the performance standards required for such tasks underreal-world constraints.
Revised: we ﬁnd relativelyfew FSR methods meeting the performance standards required for such tasks underreal-world constraints.

Original: Inspired by deep kernel learning, we develop a novel FSRalgorithm that is better suited to these settings.
Revised: Inspired by deep kernel learning, we develop a novel FSRalgorithm that is better suited to these settings.

Original: Our algorithm consists of learninga deep network in combination with a kernel function and a differentiable kernelalgorithm.
Revised: Our algorithm consists of learninga deep network in combination with a kernel function and a differentiable kernelalgorithm.

Original: As the choice of kernel is critical, our algorithm learns to ﬁnd theappropriate one for each task during inference.
Revised: As the choice of kernel is critical, our algorithm learns to ﬁnd theappropriate kernel for each task during inference.

Original: It thus performs more effectivelywith complex task distributions, outperforming current state-of-the-art algorithmson both toy and novel, real-world benchmarks that we introduce herein.
Revised: It thus performs more effectivelywith complex task distributions, outperforming current state-of-the-art algorithmson both toy and novel, real-world benchmarks that we introduce herein.

Original: Byintroducing novel benchmarks derived from biological assays, we hope that thecommunity will progress towards the development of FSR algorithms suitable foruse in noisy and uncertain environments such as drug discovery.
Revised: Byintroducing novel benchmarks derived from biological assays, we hope that thecommunity will progress towards the development of FSR algorithms suitable foruse in noisy and uncertain environments such as drug discovery.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Following breakthroughs in domains including computer vision, autonomous driving, and naturallanguage processing, deep learning methods are now entering the domain of pharmaceutical R&D.
Revised: Following breakthroughs in domains including computer vision, autonomous driving, and naturallanguage processing, deep learning methods are now entering the domain of pharmaceutical R&D.

Original: Recent successes include the deconvolution of biological targets from -omics data (Min et al.,2017), generation of drug-like compounds via de novo molecular design (Xu et al., 2019), chemicalsynthesis planning (Segler and Waller, 2017; Segler et al., 2017), and multi-modal image analysis forquantiﬁcation of cellular response (Min et al., 2017).
Revised: Recent successes include the deconvolution of biological targets from -omics data (Min et al.,2017), generation of drug-like compounds via de novo molecular design (Xu et al., 2019), chemicalsynthesis planning (Segler and Waller, 2017; Segler et al., 2017), and multi-modal image analysis forquantiﬁcation of cellular response (Min et al., 2017).

Original: A common characteristic of these applications,however, is the availability of high quality, high quantity training data.
Revised: A common characteristic of these applications,however, is the availability of high quality, high quantity training data.

Original: Unfortunately, many criticalprediction tasks in the drug discovery pipeline fail to satisfy these requirements, in part due toresource and cost constraints (Cherkasov et al., 2014).
Revised: Unfortunately, many criticalprediction tasks in the drug discovery pipeline fail to satisfy these requirements, in part due toresource and cost constraints (Cherkasov et al., 2014).

Original: We therefore focus this work on modelling biological assays (bio-assays) relevant in the early stagesof drug discovery, primarily binding and cellular readouts.
Revised: We therefore focus this work on modelling biological assays (bio-assays) relevant in the early stagesof drug discovery, primarily binding and cellular readouts.

Original: Under the constraints of an active drugdiscovery program, the data from these assays, consisting of libraries of molecules and their associatedreal-valued activity scores, is often relatively small and noisy (refer to statistics in Section 5).
Revised: Under the constraints of an active drugdiscovery program, the data from these assays, consisting of libraries of molecules and their associatedreal-valued activity scores, is often relatively small and noisy (refer to statistics in Section 5).

Original: Inmany contexts, it can be a struggle to build a training set of even a few dozen samples per individualassay.
Revised: Inmany contexts, it can be challenging to build a training set of even a few dozen samples per individualassay.

Original: Modelling an assay is thus best viewed as a few-shot regression (FSR) problem, with manyvariables (including experimental conditions, readouts, concentrations, and instrument conﬁgurations)accounting for the data distribution  generated.
Revised: Modelling an assay is thus best viewed as a few-shot regression (FSR) problem, with manyvariables (including experimental conditions, readouts, concentrations, and instrument conﬁgurations)accounting for the data distribution being generated.

Original: Practically, these variables make it infeasible tocompare data collected across different assays, thereby making it difﬁcult to learn predictive modelsfrom molecular structures.
Revised: Practically, these variables make it infeasible tocompare data collected across different assays, thereby making it difﬁcult to learn predictive modelsfrom molecular structures.

Original: Furthermore, as bio-assay modelling is intended to be used for prioritizingmolecules for subsequent evaluation (e.g. Bayesian optimization) and efﬁciently exploring the overallchemical space (e.g. active learning), accurate prediction and uncertainty estimation using few data iscritical to successful application in drug discovery.
Revised: Furthermore, as bio-assay modelling is intended to be used for prioritizingmolecules for subsequent evaluation (e.g. Bayesian optimization) and efﬁciently exploring the overallchemical space (e.g. active learning), accurate prediction and uncertainty estimation using fewdatapoints is critical to successful application in drug discovery.

Original: It is our view that robust FSR algorithms are needed to tackle this challenge.
Revised: It is our view that robust FSR algorithms are needed to tackle this challenge.

Original: Speciﬁcally, we arguethat these algorithms should remain accurate in noisy environments, and also provide well-calibrateduncertainty estimates to inform efﬁcient exploration of chemical space during molecular optimization.
Revised: Speciﬁcally, we arguethat these algorithms should remain accurate in noisy environments, and also provide well-calibrateduncertainty estimates to inform efﬁcient exploration of chemical space during molecular optimization.

Original: Fortunately, recent advances in few-shot learning have led to new algorithms that learn efﬁcientlyand generalize adequately from small training data (Wang and Yao, 2019; Chen et al., 2019).
Revised: Fortunately, recent advances in few-shot learning have led to new algorithms that learn efﬁcientlyand generalize adequately from small training data (Wang and Yao, 2019; Chen et al., 2019).

Original: Most have adopted the meta-learning paradigm (Thrun and Pratt, 1998; Vilalta and Drissi, 2002), wheresome prior knowledge is learned across a large collection of tasks and then transferred to new tasksin which there are limited amounts of data.
Revised: Most have adopted the meta-learning paradigm (Thrun and Pratt, 1998; Vilalta and Drissi, 2002), wheresome prior knowledge is learned across a large collection of tasks and then transferred to new tasksin which there are limited amounts of data.

Original: Such algorithms  differ in two aspects: the nature of themeta-knowledge captured and the amount of adaptation performed at test-time for new tasks ordatasets.
Revised: Such algorithms tend to differ in two aspects: the natureof the meta-knowledge captured and the amount of adaptation performed at test-time for newtasks or datasets.

Revised: The meta-knowledge refers to the domain speciﬁc prior needed to solve each taskmost effectively.

Original: Due to the size of the total chemical space accessible when modelling bio-assays (Bohaceket al., 1996), there is a particular need for the meta-knowledge to be sufﬁciently rich so as to allow forextrapolation and uncertainty estimation in unseen regions of chemical space at test-time (i.e. for newtasks).
Revised: Due to the size of the total chemical space accessible when modelling bio-assays(Bohacek et al., 1996), there is a particular need for the meta-knowledge to be sufﬁciently rich so asto allow for extrapolation and uncertainty estimation in unseen regions of chemical space at test-time(i.e. for new tasks).

Original: Given that the same molecule can behave differently across different assays, greater test-timeadaptation is also required and must be accounted for during modelling.
Revised: Given that the same molecule can behave differently across different assays,greater test-time adaptation is also required and must be accounted for during modelling.

Original: In previous work, metric learning methods (Koch et al., 2015; Vinyals et al., 2016; Snell et al.
Revised: In previous work, metric learning methods (Koch et al., 2015; Vinyals et al., 2016; Snell et al.

Original: ,2017; Garcia and Bruna, 2017; Bertinetto et al., 2018) accumulate meta-knowledge in high capacitycovariance/distance functions and use simple base-learners such as k-nearest neighbor (Snell et al.
Revised: ,2017; Garcia and Bruna, 2017; Bertinetto et al., 2018) accumulate meta-knowledge in high capacitycovariance/distance functions and use simple base-learners such as k-nearest neighbor (Snell et al.

Original: ,2017; Vinyals et al., 2016) or low capacity neural networks (Garcia and Bruna, 2017) to produceadequate models for new tasks.
Revised: ,2017; Vinyals et al., 2016) or low capacity neural networks (Garcia and Bruna, 2017) to produceadequate models for new tasks.

Original: However, they do not adapt the covariance functions nor the baselearners at test-time.
Revised: However, they do not adapt the covariance functions nor the baselearners at test-time.

Original: Initialization- and optimization-based methods (Finn et al., 2017; Kim et al.,2018; Ravi and Larochelle, 2016) that learn the initialization points and update rules for gradientdescent-based algorithms, respectively, allow for improved adaptation on new tasks but remain timeconsuming and memory inefﬁcient.
Revised: Initialization- and optimization-based methods (Finn et al., 2017; Kim et al.,2018; Ravi and Larochelle, 2016) that learn the initialization points and update rules for gradientdescent-based algorithms, respectively, allow for improved adaptation on new tasks but remain timeconsuming and memory inefﬁcient.

Original: We therefore argue that to ensure optimal performance whenmodelling bio-assays, it is crucial to combine the strengths of both types of methods while alsoallowing for the incorporation of domain-speciﬁc knowledge when making predictions.
Revised: We therefore argue that to ensure optimal performance whenmodelling bio-assays, it is crucial to combine the strengths of both types of methods while alsoallowing for the incorporation of domain-speciﬁc knowledge when making predictions.

Original: We achievethis by framing FSR as a deep kernel learning (DKL) task, deriving novel algorithms that we apply tomodelling speciﬁc assays and readouts.
Revised: We achievethis by framing FSR as a deep kernel learning (DKL) task, deriving novel algorithms that we apply tomodelling speciﬁc assays and readouts.

Original: Contributions: Our contributions are several-fold.
Revised: Contributions: Our contributions are three-fold.

Original: We ﬁrst frame few-shot regression as a DKLproblem and showcase its advantages relative to classical metric learning methods.
Revised: We ﬁrst frame few-shot regression as a DKLproblem and showcase its advantages relative to classical metric learning methods.

Original: We then derivethe adaptive deep kernel learning (ADKL) framework by learning a conditional kernel function thatis task dependant, allowing for more test-time adaptation than the DKL framework.
Revised: We then derivethe adaptive deep kernel learning (ADKL) framework by learning a conditional kernel function thatis task dependant, allowing for more test-time adaptation than the DKL framework.

Original: Finally, weintroduce two real-world datasets for modelling biological assays using FSR.
Revised: Finally, weintroduce two real-world datasets for modelling biological assays using FSR.

Original: With this contribution,we hope to encourage the development of subsequent few-shot regression methods suitable forreal-world applications (as is the case for few-shot classiﬁcation and reinforcement learning, each ofwhich have received comparatively greater attention in recent years (Wang and Yao, 2019)).
Revised: With this contribution,we hope to encourage the development of subsequent few-shot regression methods suitable forreal-world applications, as is the case for few-shot classiﬁcation and reinforcement learning, each ofwhich have received comparatively greater attention in recent years (Wang and Yao, 2019).

Original:  D EEP K ERNEL L EARNING
Revised:  D EEP K ERNEL L EARNING

Original: In this section, we describe the DKL framework introduced for single tasks by Wilson et al.
Revised: In this section, we describe the DKL framework introduced for single tasks by Wilson et al.

Original: We then extend it to few-shot learning and discuss its advantages overthe metric learning framework.
Revised: We then extend it to few-shot learning and discuss its advantages over metric learning algorithms.

Original: Single Task DKL: Let D ttrn = { ( x i , y i ) } mi =1 ⊂ X × R , a training dataset available for learning 77 the regression task t where
Revised: Single Task DKL: Let D ttrn = { ( x i , y i ) } mi =1 ⊂ X × R , a training dataset available for learning 78 the regression task t where

Original: X is the input space and R is the output space.
Revised: X is the input space and R is the output space.

Original: A DKL algorithm aimsto obtain a non-linear embedding of inputs in the embedding space H , using a deep neural networkφ θ : X → H of parameters θ .
Revised: A DKL algorithm aimsto obtain a non-linear embedding of inputs in the embedding space H , using a deep neural networkφ θ : X → H of parameters θ .

Original: It then ﬁnds the minimal norm regressor h t ∗ in the reproducing kernel 80 Hilbert space (RKHS)
Revised: It then ﬁnds the minimal norm regressor h t ∗ in the reproducing kernel 81 Hilbert space (RKHS)

Original: R on H , that ﬁts the training data, i.e.:
Revised: R on H , that minimize an objective function such as

Original: [Equation]
Revised: [Equation]

Original: (1) where (cid:96) is a non-negative loss function that measures the loss of a regressor h and λ weighs theimportance of the norm minimization against the training loss.
Revised: (1) where (cid:96) is a non-negative loss function that measures the loss of a regressor h and λ weighs theimportance of the norm minimization against the training loss.

Original: Following the representer theorem(Scholkopf and Smola, 2001; Steinwart and Christmann, 2008), h t ∗ can be written as a ﬁnite linearcombination of kernel evaluations on training inputs, i.e.:
Revised: Following the representer theorem(Scholkopf and Smola, 2001; Steinwart and Christmann, 2008), h t ∗ can be written as a ﬁnite linearcombination of kernel evaluations on training inputs, i.e.:

Original: [Equation]
Revised: [Equation]

Original: where α t =
Revised: where α t =

Original: ( α t 1 , · · · , α tm ) are the  combination weights and k ρ : H × H → R + is a  reproducing86 kernel of
Revised: ( α t 1 , · · · , α tm ) are the learned combination weights and k ρ : H × H → R + is a 87 chosen reproducing kernel of

Original: R with hyperparameters ρ .
Revised: R with hyperparameters ρ .

Original: Candidates include the radial basis, polynomial, and linearkernels.
Revised: Candidate kernels include the radial basis,polynomial, and linear kernels.

Original: Depending on the loss function (cid:96) , the weights α t can be obtained by using a differentiablekernel method enabling the computation of the gradients of the loss w.r.t.
Revised: α t can be obtained by using a differentiable kernel method enablingthe computation of the gradients of the loss w.r.t.

Original: the parameters θ .
Revised: the parameters θ .

Original: Suchmethods include Gaussian Process (GP), Kernel Ridge Regression (KRR), and Logistic Regression(LR).
Revised: Such methods include Gaussian Process (GP), Kernel Ridge Regression (KRR), and Logistic Regression (LR).

Original: As DKL inherits from deep learning and kernel methods, it follows that gradient descent algorithmsare required to optimize  θ , which can be high dimensional such that seeing a signiﬁcant amount oftraining samples is essential to  avoid overﬁtting.
Revised: As DKL inherits from deep learning and kernel methods, it follows that gradient descent algorithms arerequired to optimize the network parameters θ . The latter can be high dimensional and a substantialamount of training samples are required to train DKL models and avoid overﬁtting.

Original: However, once the latter condition is met, scalabilityof the kernel method becomes limiting as the running timeof kernel methods scales approximately in O ( m 3 ) for a training set of m samples. Some approximations of the kernel are thus needed for thescalability of the DKL method (see Williams and Seeger (2001); Wilson and Nickisch (2015)).
Revised: However, oncethe latter condition is met, scalability of the kernel method can be limiting (running time in O ( m 3 )for m training samples) and approximations can be needed for scalability (see Williams and Seeger(2001); Wilson and Nickisch (2015)).

Original: Few-Shot DKL:
Revised: Few-Shot DKL:

Original: Inthe setup of episodic meta learning, also known as few-shot learning, one hasaccess to a meta-training collection D meta − trn :=  (cid:110)
Revised: In few-shot learning, one has access to a meta-training collection D meta − trn := 98 (cid:110)

Original: ( D t j trn , D t j val ) (cid:111) T j =1 of T tasksto learn how tolearn from few datapoints. Each task t j has its own training (or support) set D t j trn and validation (orquery) set D t j val .
Revised: ( D t j trn , D t j val ) (cid:111) T j =1 of T tasks Each task t j has its own training (or support) set D t j trn and validation(or query) set D t j val .

Original: A meta-testing collection D meta − tst is also available to assess the generalizationperformance of the few-shot algorithm across unseen tasks.
Revised: A meta-testing collection D meta − tst is also available to assess the generalizationperformance of the few-shot algorithm across unseen tasks.

Original: To obtain a Few-Shot DKL (FSDKL)method for FSR in such settings, one can share the parameters of φ θ across all tasks, similar to metriclearning algorithms.
Revised: To obtain a Few-Shot DKL (FSDKL)method for FSR in such settings, one can share the parameters of φ θ across all tasks, similar to metriclearning algorithms.

Original: Hence, for a given task t j , the inputs are ﬁrst transformed by the function φ θand then a kernel method is used to obtain the regressor h t j ∗ , which will be evaluated on D t j val .
Revised: Hence, for a given task t j , the inputs are ﬁrst transformed by the function φ θand then a kernel method is used to obtain the regressor h t j ∗ , which will be evaluated on D t j val .

Original: Here,
Revised: Here,

Original: KRR and GP are explored as they are the state-of-the-art algorithms for kernel-based regression.
Revised: KRR and GP are explored as they are the state-of-the-art algorithms for kernel-based regression.

Original: Thelatter is used to allow our models to provide accurate predictive uncertainty, which is useful whenmodelling biological assays.
Revised: Thelatter is used to allow our models to provide accurate predictive uncertainty, which is useful whenprioritizing molecules in the context of drug discovery.

Original: KRR:
Revised: KRR:

Original: Using the squared loss and the L2-norm to compute (cid:107) h (cid:107) R , KRR gives the optimal regressor for a task t and its validation loss L t θ , ρ ,λ as follows:[Equation] , with
Revised: Using the squared loss and the L2-norm to compute (cid:107) h (cid:107) R , KRR gives the optimal regressor for a task t and its validation loss L t θ , ρ ,λ as follows:

Original: [Equation]
Revised: [Equation]

Original: where y trn =
Revised: where y trn =

Original: ( y 1 , · · · , y | D ttrn | ) T , K trn,trn is the matrix of kernel evaluations and each entry  isk ρ ( φ θ ( x i ) , φ θ ( x l )) for ( x i , · ) , ( x l , · ) ∈ D ttrn .
Revised: ( y 1 , · · · , y | D ttrn | ) T , K trn,trn is the matrix of kernel evaluations where entry i, l isk ρ ( φ θ ( x i ) , φ θ ( x l )) for pairs of examples in D ttrn .

Original: An equivalent deﬁnition applies to K x ,trn .
Revised: An equivalent deﬁnition applies to K x ,trn .

Original: GP: Using the negative log likelihood loss function instead, the GP algorithm gives a probabilistic regressor for which the predictive mean, covariance, and loss for a task t are:
Revised: GP: When using the negative log likelihood loss function, the GP algorithm gives a probabilistic regressor for which the predictive mean, covariance, and loss for a task t are:

Original: [Equation]
Revised: [Equation]

Original: Finally, the parameters θ of the neural network, along with λ and the kernel hyperparameters ρ , areoptimized using the expected loss on all tasks:
Revised: Finally, the parameters θ of the neural network, along with λ and the kernel hyperparameters ρ , areoptimized using the expected loss on all tasks:

Original: [Equation]
Revised: [Equation]

Original: To summarize, FSDKL ﬁnds a representation common to all tasks such that the kernel method (in ourcase, GP and KRR) will generalize well from a small amount of samples.
Revised: To summarize, FSDKL ﬁnds a representation common to all tasks such that the kernel method (in ourcase, GP and KRR) will generalize well from a small amount of samples.

Original: In doing so, this alleviatestwo of the main limitations of single task DKL: i) the scalability of the kernel method is no longer anissue since we are in the few-shot learning regime 1 , and ii) the parameters θ (and ρ , λ ) are learnedacross a potentially large amount of tasks and samples, providing the opportunity to learn a richrepresentation without overﬁtting.
Revised: In doing so, this alleviatestwo of the main limitations of single task DKL: i) the scalability of the kernel method is no longer anissue since we are in the few-shot learning regime 1 , and ii) the parameters θ (and ρ , λ ) are learnedacross a potentially large amount of tasks and samples, providing the opportunity to learn a richrepresentation without overﬁtting.

Original: Despite shared characteristics with the metric learning framework, the FSDKL framework is morepowerful and ﬂexible.
Revised: Despite shared characteristics with the metric learning framework, the FSDKL framework is morepowerful and ﬂexible.

Original: It provides better task-speciﬁc adaptation due to the inference of the appropriatemodel using the kernel methods compared to shared model parameters in metric learning.
Revised: It provides better task-speciﬁc adaptation due to the inference of the appropriatemodel using the kernel methods compared to shared model parameters in metric learning.

Original: After metatraining, any task-speciﬁc model also inherits the generalization guarantees of kernel-based models,and consequently increasing the number of shots for new tasks can only improve generalizationperformance.
Revised: After metatraining, any task-speciﬁc model also inherits the generalization guarantees of kernel-based models,and consequently increasing the number of shots for new tasks can only improve generalizationperformance.

Original: The incorporation of prior knowledge through user-speciﬁc kernel functions is alsoa major advantage of DKL over metric learning (e.g. use periodic kernels for periodic functionregression tasks).
Revised: The incorporation of prior knowledge through user-speciﬁc kernel functions is alsoa major advantage of DKL over metric learning (e.g. use periodic kernels for periodic functionregression tasks).

Original:  A DAPTIVE D EEP K ERNEL L EARNING
Revised:  A DAPTIVE D EEP K ERNEL L EARNING

Original: FSDKL uses a shared deep kernel function, in which the base kernel k ρ is user-chosen (although itsparameters ρ are learned during meta-training).

Original: Given that the choice of the kernel function dictatesalmost all the generalization properties of any kernel-based method, it may not be optimal to leave itto the user.

Original: Inaddition, for modelling bio-assays, it is not straightforward how to best incorporatetask-speciﬁc prior knowledge in this shared and user-chosen kernel. We overcome these limitationsby developing the Adaptive Deep Kernel Learning (ADKL) framework, illustrated by Fig. ADKL also aims to obtain a non-linear embedding of inputs using a  deep neural network φ θ sharedby all tasks before ﬁnding the minimal norm task-speciﬁc regressor h t ∗ using either GP or KRRas described in Section 2 (ADKL-GP and ADKL-KRR will refer to our algorithm when using GP and KRR, respectively).
Revised: In this section, we present a new algorithm, deemed adapative deep kernel learning (ADKL).

Original: The fundamental difference between FSDKL and ADKL lies in thekernel deﬁnition, which brings signiﬁcantly more ﬂexibility in the latter case relative to the former. Speciﬁcally, during the meta-training, ADKL learns to learn task-speciﬁc kernel functions  instead ofusing one chosen by the user.
Revised: Fundamentally, it differs from FSDKL by having more ﬂexibility in its kernel deﬁnition and by learning toproduce task-speciﬁc kernel functions during the meta-training instead of using one deﬁned by theuser.

Original: It does so by learning how to represent tasks with the task encodingnetwork ψ η and then how to leverage task embeddings to build task-speciﬁc kernels using a multimodal neural network c ρ . 
Revised: It does so by learning a task representation using a task encoding network ψ η and leveraging itto build task-speciﬁc kernels using a multi-modal neural network c ρ . More explicitly, given a task t ,

Original: Given a task t , ADKLthus ﬁrst computes its embedding z t = ψ η ( D ttrn )using its support set D t trn and deduces the adapted kernel with c ρ .
Revised: ADKL ﬁrst computes a task embedding z t = ψ η ( D ttrn ) using its support set D ttrn and then it infersthe adapted kernel with c ρ .

Original: We describe in more detail boththe task encoding network ψ η and the network c
Revised: We describe in more detail both the task encoding network ψ η and thenetwork c

Original: ρ responsible for computing the task-speciﬁc kernelbelow.
Revised: ρ responsible for computing the task-speciﬁc kernel below.

Original: 3. T ASK E NCODING
Revised: 3. T ASK E NCODING

Original: The challenge of the  ψ ηnetwork is to capture complex dependencies in the training set D ttrn toprovide a useful task encoding z  .
Revised: The challenge of the network ψ η is to capture complex dependencies in the training set D ttrn toprovide a useful task encoding z t .

Original: Furthermore, the task encoder should be invariant to permutationsof the training set and be able to encode a variable amount of samples.
Revised: Furthermore, the task encoder should be invariant to permutationsof the training set and be able to encode a variable amount of samples.

Original: After exploring a varietyof architectures, we found that those that are more complex, such as Transformers (Vaswani et al.,2017), tend to underperform.
Revised: After exploring a varietyof architectures, we found that those that are more complex, such as Transformers (Vaswani et al.,2017), tend to underperform.

Original: This is possibly due to overﬁtting or the sensitivity of training sucharchitectures.
Revised: This is possibly due to overﬁtting or the sensitivity of training sucharchitectures.

Original: Consequently, inspired by DeepSets (Zaheer et al., 2017), we propose  the following order invariant network.
Revised: Consequently, inspired by DeepSets (Zaheer et al., 2017), we propose a simple order invariantnetwork that captures the ﬁrst and second order statistics of regression datasets.

Original: It begins its computations by representing each input-target pair xy i = r ( Concat ( φ ( x i ) , v ( y i ))) for all ( x i , y i ) ∈ D ttrn , using neural networks φ , v , and r . The xy icaptures nonlinear interactions between the inputs and  the targets if r is a nonlinear transformation.
Revised: Given a dataset, thisnetwork ﬁrst processes each of its samples individually as follows: a) extract input features usingφ θ (see section2), b) concatenate the input features with the target and embed the obtained vectorusing a simple fully connected neural network r η of parameters η 2 .

Original: Then, by computing µ t xy and σ t xy , the empirical mean and standard deviation of the set{ xy 1 , xy 2 , . . , xy m } , respectively, we obtain the task representation as follows:
Revised: It then computes the ﬁrst andthe second order statistics of the obtained vectors for all samples of the dataset and concatenates themto produce the representation. More formally,

Original: [Equation]
Revised: [Equation]

Revised: where [ · , · ] is the concatenation operator.

Original: As µ txy and σ txy are invariant to permutations in D ttrn , it follows that ψ η is also permutation invariant.
Revised: As µ t and σ t are invariant to permutations in D ttrn , itfollows that ψ η is also permutation invariant.

Original: Overall, ψ η is simply the concatenation of the ﬁrst and second moments of the sample representations, which were nonlinear transformations of the original inputs and targets.
Revised: Overall, ψ η is simply the concatenation of the ﬁrst andsecond moments of the sample representations, which were nonlinear transformations of the originalinputs and targets.

Original: The learnable parameters ηof the task encoder include all the parameters of the networks v and r , and are shared across all tasks. To help the training of these parameters, we maximize the mutual information between D ttrn and 
Revised: To help the training of the parameters η , we add a regularization term I[ D ttrn , D tval ] that maximizes the mutual information between D ttrn and D tval .

Original: D tval i.e. we expect and encourage the network to produce similar task encodings using any datapartitions for a given task.
Revised: This encourage the network to produce similar task encodings when presented with different data partitions for a given task.

Original: More explicitly, we use theMINE algorithm (Belghazi et al., 2018), whichoptimizes a lower bound on the mutual information. For two random variables r, s ∼ p ( r, s ) and asimilarity measure f φ between r and s , parameterized byφ , the following inequality holds: [Equation]
Revised: Concretely, we maximize the lower bound on the mutual information between the task representations given by the support and the query sets instead of the true mutual information (Belghazi et al., 2018).

Original: Using a batch of b tasks2 , and the cosine similarity c as the similarity measure  between two task encodings, one obtains and maximizes I[ D trn , D val ] ≥ ˜ I η , where:
Revised: Using a batch of b tasks and the cosine similarity c as the similarity measure the between two task encodings, the lower bound ˜ I η ≤ I[ D ttrn , D tval ] is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: 3. T ASK -S PECIFIC K ERNEL
Revised: 3. T ASK -S PECIFIC K ERNEL

Revised: Here, we describe how the task-speciﬁc kernels are inferred using the task representations describedpreviously.

Revised: In fact, they are all obtained using a multi-modal neural network c ρ of parameter ρ .

Original: Let H and Z be the output domains of  φ θ andψ η respectively. We deﬁne a pairwise function on H , 169 whose outputs are dependant from the task representations in Z as follows:
Revised: Given any pair of input representations ( φ ( x ) and φ ( x (cid:48) ) ) and a task encoding z t , this network simplycomputes the input pair similarity under the condition given by the task encoding as follows:

Original: [Equation]
Revised: [Equation]

Original: (12) where [ · , · ] is the concatenation operator.
Revised: [Equation] where ( φ ( x ) − φ ( x (cid:48) ))

Original: It bears mentioning that the  parameters ρ are shared acrossall tasks and learned during the meta-training.
Revised: 2 is the element-wise L2 distance between the input representations, [ · , · ] isthe concatenation operator and MLP is a fully connected neural network of parameters ρ which has asingle neuron at its last layer.

Original: Also, c ρ is symmetric and stationary with regard toits inputs (φ x  and φ (cid:48) x  ) as their element-wise L2 distances vector is received as  input of the fullyconnected network.
Revised: It bears mentioning that c ρ is symmetric and stationary with regard toφ ( x ) and φ ( x (cid:48) ) as their element-wise L2 distances vector is received as part of the input of the fullyconnected network.

Original: Further, by simply concatenating the task representation z t to this distancevector at the input, c ρ provides a powerful approach to producing task-speciﬁc kernels.
Revised: Further, by simply concatenating the task representation z t to this distance vectorat the input, c ρ provides a powerful approach to produce task-speciﬁc kernels.

Original: However,these kernels are not positive semi-deﬁnite (PSD) and cannot be directly used for KRR and GP.
Revised: However, these kernelsare not positive semi-deﬁnite (PSD) and cannot be directly used for KRR and GP.

Original: Therefore,without losing any information given by c ρ , we compute task-speciﬁc PSD kernels k ρ ,tas the empirical kernel maps  with regard to the support set inputs, i.e.:
Revised: Therefore, we usethe empirical kernel mapping technique (Schölkopf et al., 1999) to make those kernel PSD and wedenote by k ρ ,t the task-speciﬁc PSD kernel associated with the task representation z t obtained from D ttrn . This kernel can be written as the empirical kernel maps of c ρ ( · , · , z t ) with regard to D ttrn i.e.:

Original: [Equation]
Revised: [Equation]

Original: Using the empirical kernel map of c ρ to compute k ρ ,t offers the opportunity to  improve the kernelevaluations in low data settings using some unlabelled data.
Revised: Using the empirical kernel map of c ρ to compute k ρ ,t offers the opportunity to introduce pseudo-inputrepresentations (or pseudo-representations ) that could improve the kernel evaluations, specially inlow data settings.

Original: More precisely, instead of computing theempirical kernel map with regard toonly D ttrn , we could use ( D ttrn ∪ U ) where U is a set of unlabelledinputs. However, to avoid a signiﬁcant increase in the computation costs of the PSD kernels, | U |should be kept relatively small (in our experiments | U |≤ 50 ). One must also be careful about the 183 composition of U to avoid overﬁtting certain tasks and under-ﬁtting others. Therefore, instead ofasking the user to provide the set U , we propose directly learning them through back-propagation.
Revised: More precisely, instead of computing the empirical kernel map with regard to

Original: Todo so, we introduce pseudo-input representations (or pseudo-representations ) u l ∈ H that are sharedby all tasks and learned during meta-training.
Revised: D ttrn alone, we use ( D ttrn ∪ U ) where U is the set of pseudo-representations .

Original: The function C t , from Eq.
Revised: The function C t , from Eq.

Original: (13), becomes:
Revised: (11), becomes:

Original: [Equation]
Revised: [Equation]

Original: with u l ∈ U ∀ l = 1 , · · · , | U | and ( x i , · ) ∈ D ttrn ∀ i = 1 , · · · , m (14) These pseudo-representations can be thought of as parameters of the adaptive kernel and the numberto be included is a hyperparameter of the algorithm.
Revised: The number of these pseudo-representations is a hyperparameter of ADKL (in our experiments wechoose | U |∈ [0 , 50] ) and all pseudo-representations u l ∈ H are learnable parameters that are shared 174 by all tasks and learned during meta-training.

Original: To prevent their collapse into a single pointand ensure that they are well distributed in the feature space H , we add a regularization term to thetraining loss.
Revised: To prevent their collapse into a single point during thetraining and to ensure that they are well distributed in the feature space H , we add a regularizationterm ( ˜ D u ) to the training loss function.

Original: Let p and q be the distributions that generate the true input representations and thepseudo-input representations, respectively.
Revised: To derive this regularization term, let’s consider p and q to bethe distributions that generate the true input representations and the pseudo-input representations,respectively.

Original: We make the assumption that p and q are both multivariate Gaussian distributions with diagonal covariance matrices and have respective parameters ( µ φ , σ 2 φ ) and ( µ u , σ 2 u ).
Revised: We make the assumption that p and q are both multivariate Gaussian distributions withdiagonal covariance matrices and have respective parameters ( µ φ , σ 2 φ ) and ( µ u , σ 2 u ).

Original: The parameters of p are estimated using the running means and variances of all inputrepresentations computed over batches of tasks.
Revised: The parametersof p are estimated using the running means and variances of all input representations computed overbatches of tasks.

Original: Those of q are estimated using U .
Revised: Those of q are estimated using U .

Original: The training ofthe pseudo-input representations isthen regularized by minimizing the KL distance ˜ D
Revised: As, p and q must be close, the training of thepseudo-representations is regularized by minimizing the KL distance ˜ D

Original: u between pand q , i.e.:
Revised: u between p and q , i.e.:

Original: [Equation]
Revised: [Equation]

Original: Putting it all together, the ADKL training objective is the following:
Revised: Putting it all together, the ADKL training objective is the following:

Original: [Equation]
Revised: [Equation]

Original: [Equation] with γ task ≥ 0 as a tradeoff hyperparameter for the regularization of the task-encoder, γ pseudo ≥ 0as a tradeoff hyperparameter for the regularization of the pseudo-inputs.
Revised: (14) with γ task ≥ 0 as a tradeoff hyperparameter for the regularization of the task-encoder and γ pseudo ≥ 0as a tradeoff hyperparameter for the regularization of the pseudo-inputs.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Across the spectrum of learning approaches, DKL methods lie between neural networks and kernelmethods.
Revised: Across the spectrum of learning approaches, DKL methods lie between neural networks and kernelmethods.

Original: While neural networks can learn from a very large amount of data without much priorknowledge, kernel methods learn from fewer data when given an appropriate covariance functionthat accounts for prior knowledge of the relevant task.
Revised: While neural networks can learn from a very large amount of data without much priorknowledge, kernel methods learn from fewer data when given an appropriate covariance functionthat accounts for prior knowledge of the relevant task.

Original: In the ﬁrst DKL attempt, Wilson et al.
Revised: In the ﬁrst DKL attempt, Wilson et al.

Original: (2016)combined GP with CNN to learn a covariance function adapted to a task from large amounts of data,though the large time and space complexity of kernel methods forced the approximation of the exactkernel using KISS-GP (Wilson and Nickisch, 2015).
Revised: (2016)combined GP with CNN to learn a covariance function adapted to a task from large amounts of data,though the large time and space complexity of kernel methods forced the approximation of the exactkernel using KISS-GP (Wilson and Nickisch, 2015).

Original: Dasgupta et al.
Revised: Dasgupta et al.

Original: (2018) have demonstrated thatsuch approximation is not necessary using ﬁnite rank kernels.
Revised: (2018) have demonstrated thatsuch approximation is not necessary using ﬁnite rank kernels.

Original: Here, we show that learning from acollection of tasks (FSR mode) does not require any approximation when the covariance function isshared across tasks.
Revised: Here, we show that learning from acollection of tasks (FSR mode) does not require any approximation when the covariance function isshared across tasks.

Original: This is an important distinction between our study and other existing studies in
Revised: This is an important distinction between our study and other existing studies in

Original: DKL, which learn their kernel for single task applications instead of multiple task collections.
Revised: DKL, which learn their kernel for single task applications instead of multiple task collections.

Original: On the spectrum between NNs and kernel methods we must also mention metric learning.
Revised: On the spectrum between NNs and kernel methods we must also mention metric learning.

Original: Metriclearning algorithms learn an input covariance function shared across tasks but rely only on theexpressive power of DNNs.
Revised: Metriclearning algorithms learn an input covariance function shared across tasks but rely only on theexpressive power of DNNs.

Original: First, stochastic kernels are built out of shared feature extractors andsimple pairwise metrics (e.g. cosine similarity (Vinyals et al., 2016), Euclidean distance (Snell et al.,2017)), or parametric functions (e.g. relation modules (Sung et al., 2018), graph neural networks(Garcia and Bruna, 2017; Kim et al., 2019a)).
Revised: First, stochastic kernels are built out of shared feature extractors andsimple pairwise metrics (e.g. cosine similarity (Vinyals et al., 2016), Euclidean distance (Snell et al.,2017)), or parametric functions (e.g. relation modules (Sung et al., 2018), graph neural networks(Garcia and Bruna, 2017; Kim et al., 2019a)).

Original: Then, within tasks, the predictions are distanceweighted combinations of the training labels with the stochastic kernel evaluations—no adaptation isdone.
Revised: Then, within tasks, the predictions are distanceweighted combinations of the training labels with the stochastic kernel evaluations—no adaptation isdone.

Original: In connection with the test-time adaptation capabilities of our method, methods that combine metriclearning with initialization-based models are great competitors.
Revised: In connection with the test-time adaptation capabilities of our method, methods that combine metriclearning with initialization-based models are great competitors.

Original: In fact, Proto-MAML (Triantaﬁllouet al., 2019), which captures the best of Prototypical Networks (Snell et al., 2017) and MAML(Finn et al., 2017), allows within-task adaptation using MAML on top of a shared feature extractor.
Revised: In fact, Proto-MAML (Triantaﬁllouet al., 2019), which captures the best of Prototypical Networks (Snell et al., 2017) and MAML(Finn et al., 2017), allows within-task adaptation using MAML on top of a shared feature extractor.

Original: Similarly, Kim et al.
Revised: Similarly, Kim et al.

Original: (2018) have proposed a Bayesian version of MAML where a feature extractor isshared across tasks, while multiple MAML particles are used for the task-level adaptation.
Revised: (2018) have proposed a Bayesian version of MAML where a feature extractor isshared across tasks, while multiple MAML particles are used for the task-level adaptation.

Original: Bertinettoet al.
Revised: Bertinettoet al.

Original: (2018) have also tackled the lack of adaptation for new tasks by using Ridge Regression and
Revised: (2018) have also tackled the lack of adaptation for new tasks by using Ridge Regression and

Original: Logistic Regression to ﬁnd the appropriate weighting of the training samples for classiﬁcation tasks.
Revised: Logistic Regression to ﬁnd the appropriate weighting of the training samples for classiﬁcation tasks.

Original: This study can be considered as an instance of the FSDKL framework, though its contribution waslimited to showing that simple differentiable learning algorithms can increase adaptation in the metriclearning framework.
Revised: This study can be considered as an instance of the FSDKL framework, though its contribution waslimited to showing that simple differentiable learning algorithms can increase adaptation in the metriclearning framework.

Original: Our work goes beyond by formalizing few-shot DKL and proposing ADKL: adata-driven manner for computing the correct kernel for a task.
Revised: Our work goes beyond by formalizing few-shot DKL and proposing ADKL: adata-driven manner to compute the correct kernel for a task.

Original: Since ADKL-GP learns task-speciﬁc stochastic processes, it is related to neural processes (Garneloet al., 2018a) and the ML-PIP framework (Gordon et al., 2018).
Revised: Since ADKL-GP learns task-speciﬁc stochastic processes, it is related to neural processes (Garneloet al., 2018a) and the ML-PIP framework (Gordon et al., 2018).

Original: Both propose a scalable alternativeto learning regression functions by performing inference on stochastic processes.
Revised: Both propose a scalable alternativeto learning regression functions by performing inference on stochastic processes.

Original: In these familiesof methods, both Conditional Neural Processes (CNP) (Garnelo et al., 2018b) and Attentive Neural
Revised: In these familiesof methods, both Conditional Neural Processes (CNP) (Garnelo et al., 2018b) and Attentive Neural

Original: Processes (ANP) (Kim et al., 2019b) learn conditional stochastic processes parameterized by taskspeciﬁc conditions derived from the support sets, but CNP is the most related to ADKL-GP.
Revised: Processes (ANP) (Kim et al., 2019b) learn conditional stochastic processes parameterized by taskspeciﬁc conditions derived from the support sets, but CNP is the most related to ADKL-GP.

Original: CNP isan instance of ML-PIP when the task encoder gives a point estimate of the task parameters insteadof a distribution.
Revised: CNP isan instance of ML-PIP when the task encoder gives a point estimate of the task parameters insteadof a distribution.

Original: Finally, the main differences between ANP and CNP are the architecture of thetask-encoder and the lack of mathematical guarantees associated with stochastic processes in CNP(as it does not impose any consistency with respect to a prior process).
Revised: Finally, the main differences between ANP and CNP are the architecture of thetask-encoder and the lack of mathematical guarantees associated with stochastic processes in CNP(as it does not impose any consistency with respect to a prior process).

Original: By comparison, ADKL-GP also learns conditional stochastic processes but has mathematical guarantees thanks to GP and PSDkernels.
Revised: By comparison, ADKL-GPalso learns conditional stochastic processes but has mathematical guarantees thanks to GP and PSDkernels.

Original:  D ATASETS
Revised:  D ATASETS

Original: Existing FSR methods have been mostly tested on 1D function regression and pixel-wise imagecompletion tasks with MNIST and CelebA (Kim et al., 2018; Garnelo et al., 2018b;a).
Revised: Existing FSR methods have been mostly tested on 1D function regression and pixel-wise imagecompletion tasks with MNIST and CelebA (Kim et al., 2018; Garnelo et al., 2018b;a).

Original: On one hand,the 1D regression tasks are all relatively simple, almost noise-less, and homogeneous.
Revised: On one hand,the 1D regression tasks are all relatively simple, almost noise-less, and homogeneous.

Original: On the otherhand, methods have been successful for image completion tasks only outside the few-shot regime (i.e.when the number of samples is greater than 500) (Garnelo et al., 2018b;a).
Revised: On the otherhand, methods have been successful for image completion tasks only outside the few-shot regime (i.e.when the number of samples is greater than 500) (Garnelo et al., 2018b;a).

Original: For these reasons, weintroduce two task collections from a real-world context.
Revised: For these reasons, weintroduce two task collections from a real-world context.

Original: Deemed Binding and Antibacterial , thesetask collections contain data from bio-assays that are representative of real-world FSR tasks in drugdiscovery.
Revised: Deemed Binding and Antibacterial , thesetask collections contain data from bio-assays that are representative of real-world FSR tasks in drug discovery.

Original: The pre-processed versions of these collections and detailed statistics are available here(anonymized link).
Revised: The pre-processed versions of these collections and detailed statistics are available here(anonymized link).

Original: Binding: All tasks in this collection aim to predict the binding afﬁnity of small molecules to a targetprotein.
Revised: Binding: All tasks in this collection aim to predict the binding afﬁnity of small molecules to a targetprotein.

Original: The characteristics of the proteins thus deﬁne different data distributions over the chemicalspace.
Revised: The characteristics of the proteins thus deﬁne different data distributions over the chemicalspace.

Original: The inputs and the targets for each task are molecules that have been tested in a binding assayand the measured binding afﬁnity of the molecule against a given protein.
Revised: The inputs and the targets for each task are molecules that have been tested in a binding assayand the measured binding afﬁnity of the molecule against a given protein.

Original: The task collection wasextracted from the public database BindingDB and altered by removing bio-assays with correlationsabove 0 .
Revised: The task collection wasextracted from the public database BindingDB and altered by removing bio-assays with correlationsabove 0 .

Original: 8 or those with less than 10 experimental measurements, leaving us with 5 , 717 tasks.
Revised: 8 or those with less than 10 experimental measurements, leaving us with 5 , 717 tasks.

Original: Antibacterial:
Revised: Antibacterial:

Original: Within this collection, the task is to predict the antimicrobial activity of smallmolecules against various bacteria.
Revised: Within this collection, the task is to predict the antimicrobial activity of smallmolecules against various bacteria.

Original: They are characterized by a bacterial strain whose resistance todrug-like molecules was being evaluated.
Revised: They are characterized by a bacterial strain whose resistance todrug-like molecules was being evaluated.

Original: The task collection was extracted from the public database
Revised: The task collection was extracted from the public database

Original: PubChem.
Revised: PubChem.

Original: After also removing bio-assays with correlations above 0 . 8 and those with less than 10samples, we obtain 3 , 255 tasks.
Revised: After also removing bio-assays with correlations above 0 . 8 and those with less than 10samples, we obtain 3 , 255 tasks.

Original: Their meta-test partitions each contain 500 tasks, with the remaining used in the meta-train andmeta-validation.
Revised: Their meta-test partitions each contain 1000 tasks, with the remaining used in the meta-train andmeta-validation.

Original: The molecules (represented as SMILES) are converted into vectors using routinesavailable in the RDKit software (more precisely into ECFP6 binary ﬁngerprint vectors of 4,096dimensions).
Revised: The molecules (represented as SMILES) are converted into vectors using routinesavailable in the RDKit software (more precisely into ECFP6 binary ﬁngerprint vectors of 4,096dimensions).

Original: These inputs were also processed in all methods using the same feature extractorarchitecture, which is a fully-connected network of 256 × 256 × 256 .
Revised: These inputs were also processed in all methods using the same feature extractorarchitecture, which is a fully-connected network of 256 × 256 × 256 .

Original: Due to the high noise-to-signalratio, the targets are ﬁrst log 2 -scaled and then scaled linearly between 0 and 1 to avoid scaling issuesduring training.
Revised: Due to the high noise-to-signalratio, the targets are ﬁrst log 2 -scaled and then scaled linearly between 0 and 1 to avoid scaling issuesduring training.

Original: Fig.
Revised: Fig.

Original: 2 highlights three aspects of the collections that make them  better benchmarks for evaluating thereadiness of FSR methods for real-world applications relative to toy collections.
Revised: 2 highlights three aspects of the collections that make them complementary to existing benchmarks, but better suited for evaluating the readiness of FSR methods for real-world applicationsrelative to toy collections.

Original: First, the distributionsof number of samples per task show that they naturally contain few samples, which we believe reﬂectsthe costs of acquiring labelled data in a drug discovery setting.
Revised: First, the distributions of number of samples per task show that theynaturally contain few samples, which we believe reﬂects the costs of acquiring labelled data in adrug discovery setting.

Original: In comparison, the number of samplesavailable per task is relatively large in previous benchmarks, with the few-shot regime being achievedartiﬁcially through sampling.
Revised: In comparison, the number of samples available per task is relatively large inprevious benchmarks, with the few-shot regime being achieved artiﬁcially through sampling.

Original: Second, as illustrated by their noise-to-signal ratio, real-world tasksare inherently noisy, increasing the difﬁculties associated with few-shot learning.
Revised: Second,as illustrated by their noise-to-signal ratio, real-world tasks are inherently noisy, increasing thedifﬁculties associated with few-shot learning.

Original: Finally, the inputdiversity within each task is reduced relative to the total among tasks.
Revised: Finally, the input diversity within each task is reducedrelative to the total among tasks.

Original: Despite this diversity difference,good models should perform relatively well outside the input region they have seen in the supportset.
Revised: Despite this diversity difference, good models should performrelatively well outside the input region they have seen in the support set.

Original: This situation challenges the methods to learn strong priors about the input space and to beable to generalize after seeing only a small fraction of it.
Revised: This situation challenges themethods to learn strong priors about the input space and to be able to generalize after seeing only asmall fraction of it.

Original: These collections invite researchers toexplore meta-learning with increasingly heterogeneous datasets and in noisy environments, as well asgeneralisation and extrapolation in large input spaces (such as the drug-like chemical space, which isestimated to be approximately 10 33 molecules (Polishchuk et al., 2013)).
Revised: These collections invite researchers to explore meta-learning with increasinglyheterogeneous datasets and in noisy environments, as well as generalisation and extrapolation inlarge input spaces (such as the drug-like chemical space, which is estimated to be approximately 10 33molecules (Polishchuk et al., 2013)).

Original: To test our method in a noise-less environment, we also use the Sinusoids collection introduced by
Revised: To test our method in a noise-less environment, we also use the Sinusoids collection introduced by

Original: Kim et al.
Revised: Kim et al.

Original: This challenging few-shot regression benchmark consists of 5,000 tasks deﬁnedby functions of the form:
Revised: This challenging few-shot regression benchmark consists of 5,000 tasks deﬁnedby functions of the form:

Original: y =
Revised: y =

Original: A sin( wx + b ) + (cid:15) with A ∈ [0 . 1 , 5 . 0] , b ∈ [0 . 0 , 2 π ] , andw ∈ [0 . 5 , 2 . 0] .
Revised: A sin( wx + b ) + (cid:15) with A ∈ [0 . 1 , 5 . 0] , b ∈ [0 . 0 , 2 π ] , andw ∈ [0 . 5 , 2 . 0] .

Original: Sampling inputs x ∈ [ − 5 . 0 , 5 . 0] and observational noise (cid:15) ∈ N (0 , (0 .
Revised: Sampling inputs x ∈ [ − 5 . 0 , 5 . 0] and observational noise (cid:15) ∈ N (0 , (0 .

Original: 01 A ) 2 ) andcomputing y gives the samples for each task.
Revised: 01 A ) 2 ) andcomputing y gives the samples for each task.

Original: Here, the meta-train, meta-validation, and meta-testcontain 56.25%, 18.75% and 25% of all the tasks, respectively, and all methods use the same featureextractor architecture, which is a fully-connected network of 40 × 40 × 40 .
Revised: Here, the meta-train, meta-validation, and meta-testcontain 56.25%, 18.75% and 25% of all the tasks, respectively, and all methods use the same featureextractor architecture, which is a fully-connected network of 40 × 40 × 40 .

Original:  E XPERIMENTS6. B ENCHMARKING ANALYSIS
Revised:  E XPERIMENTS

Revised: 6. B ENCHMARKING ANALYSIS

Original: Performance of ADKL is evaluated against a FSDKL instance(R2-D2 of  Bertinetto et al.
Revised: For all benchmarks, the performances of ADKL is compared against other meta-learning algorithms: R2-D2 (an instance of FSDKL Bertinetto et al.

Original:  CNP (Garnelo et al., 2018b), MAML (Finn et al., 2017), BMAML (Kim et al., 2018),ProtoMAML(Triantaﬁllou et al., 2019) and Learned Basis (Yi Loo, 2019) (all implementations are availablehere (anonymized link)).
Revised: , CNP (Garnelo et al., 2018b), MAML (Finnet al., 2017), BMAML (Kim et al., 2018), and Learned Basis (Yi Loo, 2019) (all implementations areavailable here (anonymized link)).

Original: These algorithms have all proven to have efﬁcient and effective test-timeadaptation routines and therefore constitute strong baselines for benchmarking.
Revised: These algorithms have all proven to have efﬁcient and effectivetest-time adaptation routines and therefore constitute strong baselines for benchmarking.

Revised: However,for bioassay modelling benchmarks, we have also added two methods considered to be state-of-the-artin chemoinformatics to assess performance relative to all meta-learning approaches.

Revised: These methodsare the Random Forest algorithm with ECFP4 (Extended Connectivity FingerPrints of diameter 4) as

Revised: [Figure]

Revised: [Figure]

Revised: [Table]

Original: Tables 1 to 3 reportthe average MSE over all tasks and 20 random support and query partitions for each task, for differentsized support sets.
Revised: molecular input representation, and ECFP4 with KRR and tanimoto similarity as a kernel function(Olier et al., 2018). During meta-test, each task is partitioned into query and support sets, then thesupport set is used to generate a model which is evaluated on the query set to compute the MSE. Thisprocess is repeated 30 times per task and the average MSE over the repetitions per task and over alltasks is reported in Tables 1 to 3.

Original: For the Sinusoids collection, we observe that DKL-based methods signiﬁcantly outperform allother methods despite their test-time adaptation capabilities.
Revised: For the Sinusoids collection, Table 3 shows that DKL-based methods signiﬁcantly outperform allother methods despite their test-time adaptation capabilities.

Original: These results alone demonstrate theeffectiveness of DKL-based methods in FSR relative to the current state-of-the-art.
Revised: These results alone demonstrate theeffectiveness of DKL-based methods in FSR relative to the current state-of-the-art.

Original: Furthermore, ofall DKL-based methods, ADKL-KRR shows consistently stronger performance than others.
Revised: Furthermore, ofall DKL-based methods, ADKL-KRR shows consistently stronger performance than others.

Original: Thisdemonstrates that using ADKL increases test-time performance relative to FS-DKL (as R2-D2 and
Revised: Thisdemonstrates that using ADKL increases test-time performance relative to FS-DKL (as R2-D2 and

Original: ADKL-KRR only differ by the kernel deﬁnition).
Revised: ADKL-KRR only differ by the kernel deﬁnition).

Original: It also indicates that attempting to capture themodel uncertainty using GP in ADKL (instead of KRR) comes with a signiﬁcant cost, especially inlower data regimes.
Revised: It also indicates that attempting to capture themodel uncertainty using GP in ADKL (instead of KRR) comes with a signiﬁcant cost, especially inlower data regimes.

Original: This may be due to the inability of GP to differentiate between the observationalnoise and the model uncertainty as the number of samples get smaller.
Revised: This may be due to the inability of GP to differentiate between the observationalnoise and the model uncertainty as the number of samples get smaller.

Original: It is also important to noticethat all methods using the task representation signiﬁcantly outperform those that do not.
Revised: It is also important to noticethat all methods using the task representation signiﬁcantly outperform those that do not.

Original: This showsthat adequately capturing the task representation is crucial for this task collection, which ADKL-KRRappears to be well-equipped to handle.
Revised: This showsthat adequately capturing the task representation is crucial for this task collection, which ADKL-KRRappears to be well-equipped to handle.

Original: Tables 2 and 3 showthat real-world datasets are challenging for most methods, as their MSE onlymarginally improves when the size of the support set increases.
Revised: Tables 1 and 2 show the performances of all methods on real-world datasets. As a complement,

Original: It should be noted that while thescaling of the targets makes the MSE low for all methods, even a decrease of 0.005 can translateinto large improvements of the modelling accuracy.
Revised: Tables 4 and 5 show the p -value that assesses the statistical signiﬁcance of the difference between each model and ADKL-GP and ADKL-KRR. These p-values were obtained by doing a Wilcoxon ranked test which compared the MSE each algorithm to ADKLs for each task to determine if one is

Original: However, we still observe that ADKL-KRRoutperforms all other methods when the number of samples is greater than 10, again providingevidence of effectiveness of our method for FSR with complex task distributions.
Revised: signiﬁcantly better than the other over all the tasks. Combined together, these tables shows that ADKL methods signiﬁcantly outperforms all other meta-learning methods. They also outperformed the state-

Original: The gaps between
Revised: of the art in chemoinformatic for Antibacterial, but do not on Binding where they are signiﬁcanlty better than all other meta-learning algorithms. These results show that despite all the effort of the community there is still there remains much room to develop meta learning algorithms which are

Original:  ADKL and R2-D2 for these collections alsoconﬁrm that using task speciﬁc kernels can be veryuseful even though inferring the right kernel can become difﬁcult as the size of the support set getssmaller.
Revised: undoubtedly superior to classical methods in computational chemistry for bioassay modelling. If is also worth noticing that ADKL methods are signiﬁcantly better than R2-D2 for these collections also

Original: Finally, it also appears that estimating the model uncertainty using ADKL-GP instead of
Revised: [Table]

Original: ADKL-KRR comes with a marginal accuracy cost.
Revised: [Table] [Table] conﬁrm that using task speciﬁc kernels can be very useful even though inferring the right kernel canbecome difﬁcult as the size of the support set gets smaller

Original: 6. A CTIVE L EARNING
Revised: 6. A CTIVE L EARNING

Original: In this section, we report the results of active learning experiments. Our intent is to measure theeffectiveness of the uncertainty captured by the predictive distribution of ADKL-GP for active
Revised: In this section

Original: [Table]
Revised: [Table]

Original: [Table] [Table] learning, as it is critical to our drug discovery use-cases.
Revised: we report the results of active learning experiments. Our intent is to measure theeffectiveness of the uncertainty captured by the predictive distribution of ADKL-GP for activelearning, as it is critical to our drug discovery use-cases.

Original: CNP, in comparison, serves to measurewhich of CNP and GP better captures the data uncertainty for improving FSR under active sampleselection.
Revised: CNP, in comparison, serves to measurewhich of CNP and GP better captures the data uncertainty for improving FSR under active sampleselection.

Original: For this purpose, we meta-train both algorithms using support and query sets of sizem = 5 .
Revised: For this purpose, we meta-train both algorithms using support and query sets of sizem = 5 .

Original: During meta-test time, ﬁve samples are randomly selected to constitute the support set D trnand build the initial hypothesis for each task.
Revised: During meta-test time, ﬁve samples are randomly selected to constitute the support set D trnand build the initial hypothesis for each task.

Original: Then, from a pool U of unlabeled data, we choose theinput x ∗ of maximum predictive entropy, i.e. x ∗ = argmax x ∈ U E
Revised: Then, from a pool U of unlabeled data, we choose theinput x ∗ of maximum predictive entropy, i.e. x ∗ = argmax x ∈ U E

Original: [log p ( y | x , D trn )] .
Revised: [log p ( y | x , D trn )] .

Original: The latter isremoved from U and added to D trn with its predicted label.
Revised: The latter isremoved from U and added to D trn with its predicted label.

Original: The within-task adaptation is performedon the new support set to obtain a new hypothesis which is evaluated on the query set D val of thetask.
Revised: The within-task adaptation is performedon the new support set to obtain a new hypothesis which is evaluated on the query set D val of thetask.

Original: This process is repeated until we reach the allotted budget of 20 queries.
Revised: This process is repeated until we reach the allotted budget of 20 queries.

Original: Fig.
Revised: Fig.

Original: 3 illustrates, for all collections, the MSE after each sample acquisition iteration and under bothrandom and active learning acquisition strategies.
Revised: 3 illustrates, for all collections, the MSE after each sample acquisition iteration and under bothrandom and active learning acquisition strategies.

Original: Under the active learning strategy, ADKL-GPconsistently outperforms CNP.
Revised: Under the active learning strategy, ADKL-GPconsistently outperforms CNP.

Original: In particular, we observe that very few samples are queried by ADKLGP to capture the data distribution whereas CNP performance remains far from optimal even whenallowed the maximum number of queries.
Revised: In particular, we observe that very few samples are queried by ADKLGP to capture the data distribution whereas CNP performance remains far from optimal even whenallowed the maximum number of queries.

Original: Further, since using the maximum predictive entropystrategy is better than querying samples at random for ADKL-GP (solid vs. dashed line), these resultssuggest that the predictive uncertainty obtained with GP is informative and more accurate than that of
Revised: Further, since using the maximum predictive entropystrategy is better than querying samples at random for ADKL-GP (solid vs. dashed line), these resultssuggest that the predictive uncertainty obtained with GP is informative and more accurate than that of

Original: CNP.
Revised: CNP.

Original: Moreover, when the number of queries is greater than 10 , we observe a performance degradationfor CNP while ADKL-GP remains consistent.
Revised: Moreover, when the number of queries is greater than 10 , we observe a performance degradationfor CNP while ADKL-GP remains consistent.

Original: This observation highlights the generalization capacityof DKL methods, even outside the few-shot regime where they have been trained — this sameproperty does not hold true for CNP.
Revised: This observation highlights the generalization capacityof DKL methods, even outside the few-shot regime where they have been trained — this sameproperty does not hold true for CNP.

Original: We attribute this property of DKL methods to their use of kernelmethods.
Revised: We attribute this property of DKL methods to their use of kernelmethods.

Original: In fact, their role in adaptation and generalization increases as we move away from thefew-shot training regime.[Figure] [Table] [Figure]
Revised: In fact, their role in adaptation and generalization increases as we move away from thefew-shot training regime.

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]
Revised: [Figure]

Original: 6. A BLATION EXPERIMENTS
Revised: 6. A BLATION EXPERIMENTS

Original: In our ﬁnal set of experiments, we more closely evaluate the impact of the task encoder and the pseudoinputs on the generalization during meta-testing.
Revised: In our ﬁnal set of experiments, we more closely evaluate the impact of the task encoder and the pseudoinputs on the generalization during meta-testing.

Original: We do so by training and evaluating ADKL on
Revised: We do so by training and evaluating ADKL on

Original: Sinusoids with different hyperparameter combinations.
Revised: Sinusoids with different hyperparameter combinations.

Original: Figs. 4a to 4d show the relative improvements(negative values) or setbacks (positive values) in the meta-test MSE compared to different baselines(but the joint impact of γ task and γ pseudo is only discussed in Appendix A.3).
Revised: Figs. 4a to 4d show the relative improvements(negative values) or setbacks (positive values) in the meta-test MSE compared to different baselines(but the joint impact of γ task and γ pseudo is only discussed in Appendix A.3).

Original: First, Fig. 4a compares γ task ∈ { 0 . 01 , 0 . 1 } relative to γ task = 0 and consequently demonstrates that 355 regularizing the task encoder by maximizing the mutual information between the support set andthe query set signiﬁcantly improves the generalization performance.
Revised: First, Fig. 4a compares γ task ∈ { 0 . 01 , 0 . 1 } relative to γ task = 0 and consequently demonstrates that 351 regularizing the task encoder by maximizing the mutual information between the support set andthe query set signiﬁcantly improves the generalization performance.

Original: This conclusion holds for allsupport set sizes tested, as shown in Appendix A.1.
Revised: This conclusion holds for allsupport set sizes tested, as shown in Appendix A.1.

Original: Combined with the results from Section 6.1, thisﬁgure shows the importance of good task encoders for generalization in few-shot learning and howusing the regularization term that we introduced is a step forward in that direction.
Revised: Combined with the results from Section 6.1, thisﬁgure shows the importance of good task encoders for generalization in few-shot learning and howusing the regularization term that we introduced is a step forward in that direction.

Original: Then, Fig.
Revised: Then, Fig.

Original: 4c measures the relative differences between γ pseudo ∈ { 0 . 01 , 0 . 1 } and γ pseudo = 0 for 361 different values of hyperparameter combinations.
Revised: 4c measures the relative differences between γ pseudo ∈ { 0 . 01 , 0 . 1 } and γ pseudo = 0 for 357 different values of hyperparameter combinations.

Original: It shows that improving the kernel map evaluationsusing pseudo-input representations can signiﬁcantly help with the generalization performance of
Revised: It shows that improving the kernel map evaluationsusing pseudo-input representations can signiﬁcantly help with the generalization performance of

Original: ADKL.
Revised: ADKL.

Original: This conclusion also holds for all values tested for | D ttrn | ( see Appendix A.2).
Revised: This conclusion also holds for all values tested for | D ttrn | ( see Appendix A.2).

Original: However, theimprovements were more consistent for smaller support sets, which is not surprising as improvingthe kernel map estimations in these cases is more critical.
Revised: However, theimprovements were more consistent for smaller support sets, which is not surprising as improvingthe kernel map estimations in these cases is more critical.

Original: Finally, Figs. 4b and 4d illustrate for ADKL-GP and ADKL-KRR, and different sizes of support sets,how the number of pseudo-representations (i.e | U | ) affects performance.
Revised: Finally, Figs. 4b and 4d illustrate for ADKL-GP and ADKL-KRR, and different sizes of support sets,how the number of pseudo-representations (i.e | U | ) affects performance.

Original: The values for each cellare relative performance using | U |∈ { 20 , 50 } versus | U | = 0 and have been averaged over different 369 hyperparameters and γ pseudo .
Revised: The values for each cellare relative performance using | U |∈ { 20 , 50 } versus | U | = 0 and have been averaged over different 365 hyperparameters and γ pseudo .

Original: In general, we can conﬁrm that increasing the number of pseudorepresentations increases the estimates of the kernel maps and improves generalization.
Revised: In general, we can conﬁrm that increasing the number of pseudorepresentations increases the estimates of the kernel maps and improves generalization.

Original: However, theimprovements are more prominent with KRR in comparison to GP, which may be due to the fact that
Revised: However, theimprovements are more prominent with KRR in comparison to GP, which may be due to the fact that

Original: GP attributes a part of the modelling noise to the kernel evaluations, leading to more constraints onthe optimization of the pseudo-representation parameters.[Figure] [Figure] [Figure]
Revised: GP attributes a part of the modelling noise to the kernel evaluations, leading to more constraints onthe optimization of the pseudo-representation parameters.

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we investigate the modelling of biological assays using few-shot learning methods.
Revised: In this work, we investigate the modelling of biological assays using few-shot learning methods.

Original: We propose a new framework, ADKL, that stores meta-knowledge in kernel functions and adaptsto new tasks using KRR or GP.
Revised: We propose a new framework, ADKL, that stores meta-knowledge in kernel functions and adaptsto new tasks using KRR or GP.

Original: Our experiments provide evidence that the additional adaptationcapacity at test-time provided by these methods increases generalization when modelling bio-assaysand on 1D sinusoidal regression tasks.
Revised: Our experiments provide evidence that the additional adaptation capacity at test-time provided by these methods increases generalization when modelling bio-assaysand on 1D sinusoidal regression tasks.

Original: In a Bayesian setup, they better estimate predictive uncertainty,increasing their utility in real-world applications such as drug discovery.
Revised: In a Bayesian setup, they better estimate predictive uncertainty,increasing their utility in real-world applications such as drug discovery.

Original: Finally, by making ourbio-assay task collections publicly available, we hope that the community will leverage them topropose FSR algorithms that are ready to be deployed under real-world constraints, with the ultimateaim of accurately predicting key molecular properties early in the drug discovery pipeline.
Revised: Finally, by making ourbio-assay task collections publicly available, we hope that the community will leverage them topropose FSR algorithms that are more robust and precise than traditional chemoinformatic methods.

Original: Appendices
Revised: AppendicesA R EGULARIZATION IMPACTA. T ASK REGULARIZATION

Original: A R EGULARIZATION IMPACTA. T ASK REGULARIZATION

Original: Table 4 presents the hyperparameter combinations used in the experiments to assess the impact ofthe trade-off parameter γ task .
Revised: Table 6 presents the hyperparameter combinations used in the experiments to assess the impact ofthe trade-off parameter γ task .

Original: We report the MSE performance obtained on the meta-test for eachcombination.
Revised: We report the MSE performance obtained on the meta-test for eachcombination.

Original: To make reading this table easier, we also repeat the Fig.
Revised: To make reading this table easier, we also repeat the Fig.

Original: 5 showing the improvementof the MSE relative to γ task = 

Revised: 5 showing the improvementof the MSE relative to γ task =

Original: (no regularization).
Revised: (no regularization).

Original: [Table]
Revised: [Table]

Original: For a more in-depth analysis, we show below the similar tables and ﬁgures for different values of K (5, 10 and 20).
Revised: For a more in-depth analysis, we show below the similar tables and ﬁgures for different values of K (5, 10 and 20).

Original: These results conﬁrm that regularizing the task encoder is helpful for any value of
Revised: These results conﬁrm that regularizing the task encoder is helpful for any value of

Original: K , even though the impact seems to become much more important as K increases (observe that themaximum improvement in each ﬁgure increases with K ).
Revised: K , even though the impact seems to become much more important as K increases (observe that themaximum improvement in each ﬁgure increases with K ).

Original: ForK =
Revised: For

Original: [Table]
Revised: [Figure]

Original: For K = {+
+}
Revised: For K = {+
+}

Original: [Table]
Revised: [Table]

Original: For K = 

Revised: For K =

Original: [Table]
Revised: [Table]

Original: A. P SEUDO - INPUT REPRESENTATIONS
Revised: A. P SEUDO - INPUT REPRESENTATIONS

Original: Table 5 presents the hyperparameter combinations used in the experiments to assess the impact ofthe trade-off parameter γ pseudo , which governs the penalty applied to the divergence between thedistribution of learned pseudo-representations and the distribution of actual representations.
Revised: Table 7 presents the hyperparameter combinations used in the experiments to assess the impact ofthe trade-off parameter γ pseudo , which governs the penalty applied to the divergence between thedistribution of learned pseudo-representations and the distribution of actual representations.

Original: We alsorepeat in Fig.
Revised: We alsorepeat in Fig.

Original: 6, the relative improvement of MSE compared to γ pseudo = 0 as shown in the maintext.
Revised: 6, the relative improvement of MSE compared to γ pseudo = 0 as shown in the maintext.

Original: [Table]
Revised: [Table]

Original: Once again, for a more in-depth analysis, we show below the same format of tables and ﬁgures fordifferent values of K , conﬁrming again that regularizing using the pseudo-representation can be veryhelpful for any value of K .
Revised: Once again, for a more in-depth analysis, we show below the same format of tables and ﬁgures fordifferent values of K , conﬁrming again that regularizing using the pseudo-representation can be very helpful for any value of K .

Original: It is worth noticing here that the improvement gain is more consistent for
Revised: It is worth noticing here that the improvement gain is more consistent for

Original: K = 5 compared to K ∈ { 10 , 20 } , supporting the fact that improving kernel maps evaluations using 494 pseudo-representations is critical as size of the support set decreases.
Revised: K = 5 compared to K ∈ { 10 , 20 } , supporting the fact that improving kernel maps evaluations using 495 pseudo-representations is critical as size of the support set decreases.

Original:  For K = 
Revised:  For K = 5

Original: [Table]
Revised: [Table]

Original:  For K = 10
Revised:  For K = 10

Original: [Table]
Revised: [Table]

Revised: [Figure]

Original: For K = {+
+}
Revised: For K = {+
+}

Original: [Figure]
Revised: [Table]

Original: Overall, the effect of the regularization is beneﬁcial, even though we witness a few pathological cases.
Revised: Overall, the effect of the regularization is beneﬁcial, even though we witness a few pathological cases.

Original: [Figure]

Original: A. J OINT IMPACT OF γ task AND γ pseudo
Revised: A. J OINT IMPACT OF γ task AND γ pseudo

Original: Since both γ task and γ pseudo have a high impact on the training and the generalization performance, [Figure] we need to assess the relationship between the two.
Revised: Since both γ task and γ pseudo have a high impact on the training and the generalization performance,we need to assess the relationship between the two.

Original: Fig.
Revised: Fig.

Original: 7 shows, for different values of K , the [Figure] relative improvement of the test MSE compared to the case where no regularization is done, i.e.γ task = 0 and γ pseudo = 0 .
Revised: 7 shows, for different values of K , therelative improvement of the test MSE compared to the case where no regularization is done, i.e.γ task = 0 and γ pseudo = 0 .

Original: Overall, one can see that higher is better in both dimensions but there [Figure] seems to be a sweet spot on the grid for each value of K and therefore we can only advise the user tocross-validate on those hyperparameters.
Revised: Overall, one can see that higher is better in both dimensions but thereseems to be a sweet spot on the grid for each value of K and therefore we can only advise the user tocross-validate on those hyperparameters.

Original: [Figure]

Original: B P REDICTION CURVES ON THE S INUSOIDS COLLECTION
Revised: B P REDICTION CURVES ON THE S INUSOIDS COLLECTION

Original: Figure 8 presents a visualization of the results obtained by each model on three tasks taken randomlyfrom the meta-test set.
Revised: Figure 8 presents a visualization of the results obtained by each model on three tasks taken randomlyfrom the meta-test set.

Original: We provide the model with ten examples from an unseen task consisting ofa slightly noisy sine function (shown in blue), and present in orange the predictions made by thenetwork based on these ten examples.−−  4 − 4 −− 2 −

Revised: We provide the model with ten examples from an unseen task consisting ofa slightly noisy sine function (shown in blue), and present in orange the predictions made by thenetwork based on these ten examples.−−  4 − 4 −− 2 −

Original: BMAML—MSE:1.
Revised: BMAML—MSE:1.

Original: GroundtruthPrediction
Revised: GroundtruthPrediction

Original: LearningexamplesStd −−  4 − 0 .
Revised: LearningexamplesStd −−  4 − 0 .

Original: 6 − 0 .
Revised: 6 − 0 .

Original: 4 − 0 .0 . 

Revised: 4 − 0 .0 .

Original: BMAML—MSE:0.12 GroundtruthPrediction
Revised: BMAML—MSE:0.12 GroundtruthPrediction

Original: LearningexamplesStd −−   −−
Revised: LearningexamplesStd −−   −−

Original: BMAML—MSE:5.
Revised: BMAML—MSE:5.

Original: GroundtruthPrediction
Revised: GroundtruthPrediction

Original: LearningexamplesStd −−  4 −−
Revised: LearningexamplesStd −−  4 −−

Original: CNP—MSE:1.
Revised: CNP—MSE:1.

Original: GroundtruthPrediction
Revised: GroundtruthPrediction

Original: LearningexamplesStd −−   − 1 .− 0 .0 .0 .1 .
Revised: LearningexamplesStd −−   − 1 .− 0 .0 .0 .1 .

Original: CNP—MSE:0.02 GroundtruthPrediction
Revised: CNP—MSE:0.02 GroundtruthPrediction

Original: LearningexamplesStd −−   −−
Revised: LearningexamplesStd −−   −−

Original: CNP—MSE:2.
Revised: CNP—MSE:2.

Original: GroundtruthPrediction
Revised: GroundtruthPrediction

Original: LearningexamplesStd −−  4 − 4 −− 2 −

Revised: LearningexamplesStd −−  4 − 4 −− 2 −

Original: ADKL-KRR—MSE:0.
Revised: ADKL-KRR—MSE:0.

Original: Groundtruth
Revised: Groundtruth

Original: Prediction
Revised: Prediction

Original: Learningexamples −−  4 − 0 .
Revised: Learningexamples −−  4 − 0 .

Original: 6 − 0 .
Revised: 6 − 0 .

Original: 4 − 0 .0 . 

Revised: 4 − 0 .0 .

Original: [Table]
Revised: [Table]

Original: −−   −−
Revised: −−   −−

Original: R2-D2—MSE:5.
Revised: R2-D2—MSE:5.

Original: Groundtruth
Revised: Groundtruth

Original: Prediction
Revised: Prediction

Original: Learningexamples
Revised: Learningexamples

Revised: C S UPPLEMENTARY RESULTS FOR THE REAL - WORLD DATASETS

Revised: [Figure]

Revised: D K

Revised: [Figure]

Revised: G

Revised: [Figure]

Revised: A D K

Revised: [Figure]

Revised: K R R B M A

Revised: [Figure]

Revised: L

Revised: C N P

Revised: E C F

Revised: [Figure]

Revised: K

Revised: [Figure]

Revised: Figure 9 shows the distribution over the random support/query sets generated at test time.

Revised: Note thatthe results presented in the main paper estimate the inﬂuence of the initialisation by using multipleseeds and computing the standard deviation on the average MSE (averaged over the support/querysplits).

Revised: The two pieces of information are important : the results presented above give us a better idea ofthe “ meta-generalisation ” capabilities of each algorithm, while those in the main paper assess thereproducibility and the statistical signiﬁcance of the relative improvements.