Original: DeepPipe : D EEP , M ODULAR AND E XTENDABLE R EPRESENTATIONS OF M ACHINE L EARNING P IPELINES
Revised: DeepPipe : D EEP , M ODULAR AND E XTENDABLE R EPRESENTATIONS OF M ACHINE L EARNING P IPELINES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Finding accurate Machine Learning pipelines is essential in achieving state-of-theart AI predictive performance.
Revised: Finding accurate Machine Learning pipelines is essential in achieving state-of-theart AI predictive performance.

Original: Unfortunately, most existing Pipeline Optimization techniques rely on flavors of Bayesian Optimization that do not explore the deep interaction between pipeline stages/components (e g. between hyperparameters of the deployed preprocessing algorithm and the hyperparameters of a classifier).
Revised: Unfortunately, most existing Pipeline Optimization techniques rely on flavors of Bayesian Optimization that do not explore the deep interaction between pipeline stages/components (e g. between hyperparameters of the deployed preprocessing algorithm and the hyperparameters of a classifier).

Original: In this paper, we are the first to capture the deep interaction between components of a Machine Learning pipeline.
Revised: In this paper, we are the first to capture the deep interaction between components of a Machine Learning pipeline.

Original: We propose embedding pipelines in a deep latent representation through a novel per-component encoder mechanism.
Revised: We propose embedding pipelines in a deep latent representation through a novel per-component encoder mechanism.

Original: Such pipeline embeddings are used with deep kernel Gaussian Process surrogates inside a Bayesian Optimization setup.
Revised: Such pipeline embeddings are used with deep kernel Gaussian Process surrogates inside a Bayesian Optimization setup.

Original: Through extensive experiments on three largescale meta-datasets, including Deep Learning pipelines for computer vision, we demonstrate that learning pipeline embeddings achieves state-of-the-art results in
Revised: Through extensive experiments on three largescale meta-datasets, including Deep Learning pipelines for computer vision, we demonstrate that learning pipeline embeddings achieves state-of-the-art results in

Original: Pipeline Optimization
Revised: Pipeline Optimization

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Machine Learning (ML) has proven to be successful in a wide range of tasks such as image classification, natural language processing, and time series forecasting.
Revised: Machine Learning (ML) has proven to be successful in a wide range of tasks such as image classification, natural language processing, and time series forecasting.

Original: In a supervised learning setup, practitioners need to define a sequence of stages comprising algorithms that transform the data (e.g. imputation, scaling) and produce an estimation (e.g. through a classifier or regressor).
Revised: In a supervised learning setup, practitioners need to define a sequence of stages comprising algorithms that transform the data (e.g. imputation, scaling) and produce an estimation (e.g. through a classifier or regressor).

Original: The selection of the algorithms and their hyperparameters, known as Pipeline Optimization (Olson & Moore, 2016) or pipeline synthesis (Liu et al., 2020; Drori et al., 2021) is challenging.
Revised: The selection of the algorithms and their hyperparameters, known as Pipeline Optimization (Olson & Moore, 2016) or pipeline synthesis (Liu et al., 2020; Drori et al., 2021) is challenging.

Original: Firstly, the search space contains conditional hyperparameters, as only some of them are active depending on the selected algorithms.
Revised: Firstly, the search space contains conditional hyperparameters, as only some of them are active depending on the selected algorithms.

Original: Secondly, this space is arguably bigger than the one for a single algorithm.
Revised: Secondly, this space is arguably bigger than the one for a single algorithm.

Original: Therefore, previous work demonstrates how this pipeline search can be automatized and achieve competitive results (Feurer et al., 2015; Olson & Moore, 2016).
Revised: Therefore, previous work demonstrates how this pipeline search can be automatized and achieve competitive results (Feurer et al., 2015; Olson & Moore, 2016).

Original: Some of these approaches include Evolutionary
Revised: Some of these approaches include Evolutionary

Original: Algorithms (Olson & Moore, 2016), Reinforcement Learning (Rakotoarison et al., 2019; Drori et al., 2021) or Bayesian Optimization (Feurer et al., 2015; Thornton et al., 2012; Alaa & van der Schaar, 2018).
Revised: Algorithms (Olson & Moore, 2016), Reinforcement Learning (Rakotoarison et al., 2019; Drori et al., 2021) or Bayesian Optimization (Feurer et al., 2015; Thornton et al., 2012; Alaa & van der Schaar, 2018).

Original: Pipeline Optimization (PO) techniques need to capture the complex interaction between the algorithms of a Machine Learning pipeline and their hyperparameter configurations.
Revised: Pipeline Optimization (PO) techniques need to capture the complex interaction between the algorithms of a Machine Learning pipeline and their hyperparameter configurations.

Original: Unfortunately, no prior method uses Deep Learning to encapsulate the interaction between pipeline components.
Revised: Unfortunately, no prior method uses Deep Learning to encapsulate the interaction between pipeline components.

Original: Prior work trains performance predictors (a.k.a. surrogates) on the concatenated hyperparameter space of all algorithms (raw search space), for instance, using random forests (Feurer et al., 2015) or finding groups of hyperparameters to use on kernels with additive structure (Alaa & van der Schaar, 2018).
Revised: Prior work trains performance predictors (a.k.a. surrogates) on the concatenated hyperparameter space of all algorithms (raw search space), for instance, using random forests (Feurer et al., 2015) or finding groups of hyperparameters to use on kernels with additive structure (Alaa & van der Schaar, 2018).

Original: On the other hand, transfer learning has been shown to decisively improve PO by transferring efficient pipelines evaluated on other datasets (Fusi et al., 2018; Yang et al., 2019; 2020).
Revised: On the other hand, transfer learning has been shown to decisively improve PO by transferring efficient pipelines evaluated on other datasets (Fusi et al., 2018; Yang et al., 2019; 2020).

Original: Our method is the first to introduce a deep pipeline representation that is meta-learned to achieve state-of-the-art results in terms of the quality of the discovered pipelines.
Revised: Our method is the first to introduce a deep pipeline representation that is meta-learned to achieve state-of-the-art results in terms of the quality of the discovered pipelines.

Original: We introduce DeepPipe , a neural network architecture for embedding pipeline configurations on a latent space.
Revised: We introduce DeepPipe , a neural network architecture for embedding pipeline configurations on a latent space.

Original: Such deep representations are combined with Gaussian Processes (GP) for tuning pipelines with Bayesian Optimization (BO).
Revised: Such deep representations are combined with Gaussian Processes (GP) for tuning pipelines with Bayesian Optimization (BO).

Original: We exploit the knowledge of the hierarchical search space of pipelines by mapping the hyperparameters of every algorithm through per-algorithm encoders to a hidden representation, followed by a Fully Connected Network that receives the concatenated representations as input.
Revised: We exploit the knowledge of the hierarchical search space of pipelines by mapping the hyperparameters of every algorithm through per-algorithm encoders to a hidden representation, followed by a Fully Connected Network that receives the concatenated representations as input.

Original: Additionally, we show that meta-learning this network through evaluations on auxiliary tasks improves the quality of BO.
Revised: Additionally, we show that meta-learning this network through evaluations on auxiliary tasks improves the quality of BO.

Original: Experiments on three large-scale meta-datasets show that our method achieves the new state-of-the-art in Pipeline Optimization.
Revised: Experiments on three large-scale meta-datasets show that our method achieves the new state-of-the-art in Pipeline Optimization.

Original: Our contributions are as follows:
Revised: Our contributions are as follows:

Original: • We introduce DeepPipe , a surrogate for BO that achieves peak performance when optimizing a pipeline for a new dataset through transfer learning.
Revised: • We introduce DeepPipe , a surrogate for BO that achieves peak performance when optimizing a pipeline for a new dataset through transfer learning.

Original: • We present a novel and modular architecture that applies different encoders per stage and yields better generalization in low meta-data regimes, i
Revised: • We present a novel and modular architecture that applies different encoders per stage and yields better generalization in low meta-data regimes, i

Original: e. few/no auxiliary tasks.
Revised: e. few/no auxiliary tasks.

Original: • We conduct extensive evaluations against seven baselines on three large meta-datasets, and we further compare against rival methods in OpenML datasets to assess their performances under time constraints.
Revised: • We conduct extensive evaluations against seven baselines on three large meta-datasets, and we further compare against rival methods in OpenML datasets to assess their performances under time constraints.

Original: • We demonstrate that our pipeline representation helps achieving state-of-the-art results in optimizing pipelines for fine-tuning deep computer vision networks.
Revised: • We demonstrate that our pipeline representation helps achieving state-of-the-art results in optimizing pipelines for fine-tuning deep computer vision networks.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Full Model Selection (FMS) is also referred to as Combined Algorithm Selection and Hyperparameter optimization (CASH) (Hutter et al., 2019; Feurer et al., 2015).
Revised: Full Model Selection (FMS) is also referred to as Combined Algorithm Selection and Hyperparameter optimization (CASH) (Hutter et al., 2019; Feurer et al., 2015).

Original: FMS aims to find the best model and its respective hyperparameter configuration (Hutter et al., 2019).
Revised: FMS aims to find the best model and its respective hyperparameter configuration (Hutter et al., 2019).

Original: A common approach is to use
Revised: A common approach is to use

Original: Bayesian Optimization with surrogates that can handle conditional hyperparameters, such as Random
Revised: Bayesian Optimization with surrogates that can handle conditional hyperparameters, such as Random

Original: Forest (Feurer et al., 2015), tree-structured Parzen estimators (Thornton et al., 2012), or ensembles of neural networks (Schilling et al., 2015).
Revised: Forest (Feurer et al., 2015), tree-structured Parzen estimators (Thornton et al., 2012), or ensembles of neural networks (Schilling et al., 2015).

Original: Pipeline Optimization (PO) is a generalization of FMS where the goal is to find the algorithms and their hyperparameters for different stages of a Machine Learning Pipeline.
Revised: Pipeline Optimization (PO) is a generalization of FMS where the goal is to find the algorithms and their hyperparameters for different stages of a Machine Learning Pipeline.

Original: Common approaches model the search space as a tree structure and use reinforcement learning (Rakotoarison et al., 2019;
Revised: Common approaches model the search space as a tree structure and use reinforcement learning (Rakotoarison et al., 2019;

Original: Drori et al., 2021; de Sá et al., 2017), evolutionary algorithms (Olson & Moore, 2016), Hierarchical
Revised: Drori et al., 2021; de Sá et al., 2017), evolutionary algorithms (Olson & Moore, 2016), Hierarchical

Original: Task Networks (Mohr et al., 2018) for searching pipelines.
Revised: Task Networks (Mohr et al., 2018) for searching pipelines.

Original: Other approaches use Multi-Armed Bandit strategies to optimize the pipeline, and combine them with Bayesian Optimization (Swearingen et al., 2017) or multi-fidelity optimization (Kishimoto et al., 2021).
Revised: Other approaches use Multi-Armed Bandit strategies to optimize the pipeline, and combine them with Bayesian Optimization (Swearingen et al., 2017) or multi-fidelity optimization (Kishimoto et al., 2021).

Original: Alaa & van der Schaar (2018) use additive kernels on a Gaussian Process surrogate to search pipelines with BO that groups the algorithms in clusters and fit their hyperparameters on independent Gaussian Processes, achieving an effectively lower dimensionality per input.
Revised: Alaa & van der Schaar (2018) use additive kernels on a Gaussian Process surrogate to search pipelines with BO that groups the algorithms in clusters and fit their hyperparameters on independent Gaussian Processes, achieving an effectively lower dimensionality per input.

Original: By formulating the Pipeline Optimization as a constrained optimization problem, Liu et al (Liu et al., 2020) introduce a method based on alternating direction method of multipliers (ADMM)
Revised: By formulating the Pipeline Optimization as a constrained optimization problem, Liu et al (Liu et al., 2020) introduce a method based on alternating direction method of multipliers (ADMM)

Original: (Gabay & Mercier, 1976).
Revised: (Gabay & Mercier, 1976).

Original: Transfer-learning for Pipeline Optimization and CASH leverages information from previous (auxiliary) task evaluations.
Revised: Transfer-learning for Pipeline Optimization and CASH leverages information from previous (auxiliary) task evaluations.

Original: Few approaches use dataset meta-features to warm-start BO with good configurations from other datasets (Feurer et al., 2015; Alaa & van der Schaar, 2018).
Revised: Few approaches use dataset meta-features to warm-start BO with good configurations from other datasets (Feurer et al., 2015; Alaa & van der Schaar, 2018).

Original: As extracting meta-features demands computational time, follow-up works find a portfolio based on these auxiliary tasks (Feurer et al., 2020).
Revised: As extracting meta-features demands computational time, follow-up works find a portfolio based on these auxiliary tasks (Feurer et al., 2020).

Original: Another popular approach is to use collaborative filtering with a matrix of pipelines vs task evaluations to learn latent embeddings of pipelines.
Revised: Another popular approach is to use collaborative filtering with a matrix of pipelines vs task evaluations to learn latent embeddings of pipelines.

Original: OBOE obtains the embeddings by applying a QR decomposition of the matrix on a time-constrained formulation (Yang et al., 2019).
Revised: OBOE obtains the embeddings by applying a QR decomposition of the matrix on a time-constrained formulation (Yang et al., 2019).

Original: By recasting the matrix as a tensor, Tensor-OBOE
Revised: By recasting the matrix as a tensor, Tensor-OBOE

Original: (Yang et al., 2020) finds latent representations via the Tucker decomposition.
Revised: (Yang et al., 2020) finds latent representations via the Tucker decomposition.

Original: Furthermore, Fusi et al.
Revised: Furthermore, Fusi et al.

Original: (2018) apply probabilistic matrix factorization for finding the latent pipeline representations.
Revised: (2018) apply probabilistic matrix factorization for finding the latent pipeline representations.

Original: Subsequently, they use the latent representations as inputs for a Gaussian Process, and explore the search space using BO.
Revised: Subsequently, they use the latent representations as inputs for a Gaussian Process, and explore the search space using BO.

Original:  P RELIMINARIES 3. P IPELINE O PTIMIZATION
Revised:  P RELIMINARIES 3. P IPELINE O PTIMIZATION

Original: The pipeline of a ML system consists of a sequence of N stages (e.g. dimensionality reducer, standardizer, encoder, estimator (Yang et al., 2020)).
Revised: The pipeline of a ML system consists of a sequence of N stages (e.g. dimensionality reducer, standardizer, encoder, estimator (Yang et al., 2020)).

Original: At each stage i ∈ { 1 . . .
Revised: At each stage i ∈ { 1 . . .

Original: N } a pipeline includes one algorithm 1 from a set of M i choices (e.g. the estimator stage can include the algorithms
Revised: N } a pipeline includes one algorithm 1 from a set of M i choices (e.g. the estimator stage can include the algorithms

Original: {SVM, MLP, RF}).
Revised: {SVM, MLP, RF}).

Original: Algorithms are tuned through their hyperparameter search spaces, where λ i,j denotes the configuration of the j -th algorithm in the i -th stage.
Revised: Algorithms are tuned through their hyperparameter search spaces, where λ i,j denotes the configuration of the j -th algorithm in the i -th stage.

Original: Furthermore, let us denote a pipeline p as the set of indices for the selected algorithm at each stage, i.e. p :=
Revised: Furthermore, let us denote a pipeline p as the set of indices for the selected algorithm at each stage, i.e. p :=

Original: ( p 1 , . .
Revised: ( p 1 , . .

Original: . , p N ) , where p i ∈ { 1 . . .
Revised: . , p N ) , where p i ∈ { 1 . . .

Original: M i } represents the index of the selected algorithm at the i -th pipeline stage.
Revised: M i } represents the index of the selected algorithm at the i -th pipeline stage.

Original: The hyperparameter configuration of a pipeline is the unified set of the configurations of all the algorithms in a pipeline, concretely λ ( p ) :
Revised: The hyperparameter configuration of a pipeline is the unified set of the configurations of all the algorithms in a pipeline, concretely λ ( p ) :

Original: = ( λ 1 ,p 1 , . . .
Revised: = ( λ 1 ,p 1 , . . .

Original: , λ N,p N ) , λ i,p i ∈
Revised: , λ N,p N ) , λ i,p i ∈

Original: Λ i,p i .
Revised: Λ i,p i .

Original: Pipeline Optimization demands finding the optimal pipeline p ∗ and its optimal configuration λ ( p ∗ ) by minimizing the validation loss of a trained pipeline on a dataset D as shown in Equation 1.
Revised: Pipeline Optimization demands finding the optimal pipeline p ∗ and its optimal configuration λ ( p ∗ ) by minimizing the validation loss of a trained pipeline on a dataset D as shown in Equation 1.

Original: [Equation]
Revised: [Equation]

Original: From now we will use the term pipeline configuration for the combination of a sequence of algorithms p and their hyperparameter configurations λ ( p ) , and denote it simply as p λ :
Revised: From now we will use the term pipeline configuration for the combination of a sequence of algorithms p and their hyperparameter configurations λ ( p ) , and denote it simply as p λ :

Original: = ( p, λ ( p )) .
Revised: = ( p, λ ( p )) .

Original: 3. B AYESIAN O PTIMIZATION
Revised: 3. B AYESIAN O PTIMIZATION

Original: Bayesian optimization (BO) is a mainstream strategy for optimizing ML pipelines (Feurer et al., 2015;
Revised: Bayesian optimization (BO) is a mainstream strategy for optimizing ML pipelines (Feurer et al., 2015;

Original: Hutter et al., 2011; Alaa & van der Schaar, 2018; Fusi et al., 2018;
Revised: Hutter et al., 2011; Alaa & van der Schaar, 2018; Fusi et al., 2018;

Original: Schilling et al., 2015).
Revised: Schilling et al., 2015).

Original: Let us start with defining a history of Q evaluated pipeline configurations as H =
Revised: Let us start with defining a history of Q evaluated pipeline configurations as H =

Original: { ( p (1) λ , y (1) ) , . . .
Revised: { ( p (1) λ , y (1) ) , . . .

Original: , ( p ( Q ) λ , y ( Q ) ) } , where y ( q ) ∼ N ( f ( p ( q ) λ ) , σ 2 q ) is a probabilistic modeling of the validation loss f ( p ( q ) λ ) achieved with the q -th evaluated pipeline configuration p ( q ) λ , q ∈ { 1 . . .
Revised: , ( p ( Q ) λ , y ( Q ) ) } , where y ( q ) ∼ N ( f ( p ( q ) λ ) , σ 2 q ) is a probabilistic modeling of the validation loss f ( p ( q ) λ ) achieved with the q -th evaluated pipeline configuration p ( q ) λ , q ∈ { 1 . . .

Original: Q } .
Revised: Q } .

Original: Such a validation loss is approximated with a surrogate model, which is typically a Gaussian process (GP) regressor.
Revised: Such a validation loss is approximated with a surrogate model, which is typically a Gaussian process (GP) regressor.

Original: We measure the similarity between pipelines via a kernel function k : dom ( p λ ) × dom ( p λ )
Revised: We measure the similarity between pipelines via a kernel function k : dom ( p λ ) × dom ( p λ )

Original: → R > 0 parameterized with θ , and represent similarities as a matrix K ′ q,ℓ := k ( p ( q ) λ , p ( ℓ ) λ ; γ ) ,
Revised: → R > 0 parameterized with θ , and represent similarities as a matrix K ′ q,ℓ := k ( p ( q ) λ , p ( ℓ ) λ ; γ ) ,

Original: K ′ ∈ R Q × Q > 0 .
Revised: K ′ ∈ R Q × Q > 0 .

Original: Since we consider noise, we define K = K ′ + σ p
Revised: Since we consider noise, we define K = K ′ + σ p

Original: I .
Revised: I .

Original: A GP estimates the validation loss f ∗ of a new pipeline configuration p λ ( ∗ ) by computing the posterior mean E [ f ∗ ] and posterior variance V [ f ∗ ] as:
Revised: A GP estimates the validation loss f ∗ of a new pipeline configuration p λ ( ∗ ) by computing the posterior mean E [ f ∗ ] and posterior variance V [ f ∗ ] as:

Original: [Equation]
Revised: [Equation]

Original: where K ∗ ,q = k (cid:16) p λ ( ∗ ) , p ( q ) λ ; γ (cid:17) , K ∗ ∈ R Q> 0 , and K ∗∗ = k (cid:0)
Revised: where K ∗ ,q = k (cid:16) p λ ( ∗ ) , p ( q ) λ ; γ (cid:17) , K ∗ ∈ R Q> 0 , and K ∗∗ = k (cid:0)

Original: p λ ( ∗ ) , p λ ( ∗ ) ; γ (cid:1) , K ∗∗ ∈ R > 0 .
Revised: p λ ( ∗ ) , p λ ( ∗ ) ; γ (cid:1) , K ∗∗ ∈ R > 0 .

Original: BO is an iterative process that alternates between fitting a GP regressor as described above and selecting the next pipeline configuration to evaluate (Snoek et al., 2012).
Revised: BO is an iterative process that alternates between fitting a GP regressor as described above and selecting the next pipeline configuration to evaluate (Snoek et al., 2012).

Original: A description of how BO finds pipelines using a GP surrogate is provided in Appendix I.
Revised: A description of how BO finds pipelines using a GP surrogate is provided in Appendix I.

Original:  DeepPipe : BO WITH D EEP P IPELINE C ONFIGURATIONS
Revised:  DeepPipe : BO WITH D EEP P IPELINE C ONFIGURATIONS

Original: To apply BO to Pipeline Optimization (PO) we must define a kernel function that computes the similarity of pipeline configurations, i.e. k (cid:16) p ( q ) λ , p ( ℓ ) λ ; θ (cid:17) = ? .
Revised: To apply BO to Pipeline Optimization (PO) we must define a kernel function that computes the similarity of pipeline configurations, i.e. k (cid:16) p ( q ) λ , p ( ℓ ) λ ; θ (cid:17) = ? .

Original: Prior work exploring BO for PO use kernel functions directly on the raw concatenated vector space of selected algorithms and their hyperparameters (Alaa & van der Schaar, 2018) or use surrogates without dedicated kernels for the conditional search space (Feurer et al., 2015; Olson & Moore, 2016; Schilling et al., 2015).
Revised: Prior work exploring BO for PO use kernel functions directly on the raw concatenated vector space of selected algorithms and their hyperparameters (Alaa & van der Schaar, 2018) or use surrogates without dedicated kernels for the conditional search space (Feurer et al., 2015; Olson & Moore, 2016; Schilling et al., 2015).

Original: However, we hypothesize that these approaches cannot capture the deep interaction between pipeline stages, between algorithms inside a stage, between algorithms across stages, and between different configurations of these algorithms.
Revised: However, we hypothesize that these approaches cannot capture the deep interaction between pipeline stages, between algorithms inside a stage, between algorithms across stages, and between different configurations of these algorithms.

Original: In order to address this issue we propose a simple, yet powerful solution to PO: learn a deep embedding of a pipeline configuration and apply BO with a deep kernel (Wistuba & Grabocka, 2021; Wilson et al., 2016).
Revised: In order to address this issue we propose a simple, yet powerful solution to PO: learn a deep embedding of a pipeline configuration and apply BO with a deep kernel (Wistuba & Grabocka, 2021; Wilson et al., 2016).

Original: This is done by DeepPipe , which searches pipelines in a latent space using BO with Gaussian
Revised: This is done by DeepPipe , which searches pipelines in a latent space using BO with Gaussian

Original: Processes.
Revised: Processes.

Original: We use a neural network ϕ
Revised: We use a neural network ϕ

Original: ( p λ ; θ ) : dom ( p λ ) → R Z with weights θ to project a pipeline configuration to a Z -dimensional space.
Revised: ( p λ ; θ ) : dom ( p λ ) → R Z with weights θ to project a pipeline configuration to a Z -dimensional space.

Original: Then, we measure the pipelines’ similarity in this latent space as k (cid:16) ϕ
Revised: Then, we measure the pipelines’ similarity in this latent space as k (cid:16) ϕ

Original: ( p ( q ) λ ; θ ) , ϕ ( p ( ℓ ) λ ; θ ) (cid:17) using the popular Matérn 5/2 kernel (Genton, 2002).
Revised: ( p ( q ) λ ; θ ) , ϕ ( p ( ℓ ) λ ; θ ) (cid:17) using the popular Matérn 5/2 kernel (Genton, 2002).

Original: Once we compute the parameters of the kernel similarity function, we can compute the GP’s posterior and conduct PO with BO as specified in Section 3.2.
Revised: Once we compute the parameters of the kernel similarity function, we can compute the GP’s posterior and conduct PO with BO as specified in Section 3.2.

Original: In this work, we exploit existing deep kernel learning machinery (Wistuba & Grabocka, 2021; Wilson et al., 2016) to train the parameters θ of the pipeline embedding neural network ϕ , and the parameters γ of the kernel function k , by maximizing the log-likelihood of the observed validation losses y of the evaluated pipeline configurations p λ .
Revised: In this work, we exploit existing deep kernel learning machinery (Wistuba & Grabocka, 2021; Wilson et al., 2016) to train the parameters θ of the pipeline embedding neural network ϕ , and the parameters γ of the kernel function k , by maximizing the log-likelihood of the observed validation losses y of the evaluated pipeline configurations p λ .

Original: The objective function for training a deep kernel is the log marginal likelihood of the Gaussian Process (Rasmussen & Williams, 2006) with covariance matrix entries k
Revised: The objective function for training a deep kernel is the log marginal likelihood of the Gaussian Process (Rasmussen & Williams, 2006) with covariance matrix entries k

Original: q,ℓ = k (cid:16) ϕ
Revised: q,ℓ = k (cid:16) ϕ

Original: ( p ( q ) λ ; θ ) , ϕ ( p ( ℓ ) λ ; θ ) (cid:17) .
Revised: ( p ( q ) λ ; θ ) , ϕ ( p ( ℓ ) λ ; θ ) (cid:17) .

Original: 4. P IPELINE E MBEDDING N ETWORK
Revised: 4. P IPELINE E MBEDDING N ETWORK

Original: [Figure]
Revised: [Figure]

Original: The main piece of the puzzle is: How to define the pipeline configuration embedding ϕ ?
Revised: The main piece of the puzzle is: How to define the pipeline configuration embedding ϕ ?

Original: Our DeepPipe embedding is composed of two parts (i) per-algorithm encoders, and (ii) a pipeline aggregation network.
Revised: Our DeepPipe embedding is composed of two parts (i) per-algorithm encoders, and (ii) a pipeline aggregation network.

Original: A visualization example of our DeepPipe embedding architecture is provided in Figure 1.
Revised: A visualization example of our DeepPipe embedding architecture is provided in Figure 1.

Original: We define one encoder ξ ( i,j ) for the hyperparameter configurations of each j -th algorithm, in each i -th stage, as plain multi-layer perceptrons (MLP).
Revised: We define one encoder ξ ( i,j ) for the hyperparameter configurations of each j -th algorithm, in each i -th stage, as plain multi-layer perceptrons (MLP).

Original: These encoders, each parameterized by weights θ enc ( i,j ) , map the algorithms’ configurations to a L
Revised: These encoders, each parameterized by weights θ enc ( i,j ) , map the algorithms’ configurations to a L

Original: i -dimensional vector space:
Revised: i -dimensional vector space:

Original: [Equation]
Revised: [Equation]

Original: ,j (cid:1) =
Revised: ,j (cid:1) =

Original: MLP (cid:0) λ i,j
Revised: MLP (cid:0) λ i,j

Original: [Equation]
Revised: [Equation]

Original: θ enc i,j (cid:1) , ξ ( i,j ) : Λ i,j → R L i , ∀ i ∈ { 1 . . .
Revised: θ enc i,j (cid:1) , ξ ( i,j ) : Λ i,j → R L i , ∀ i ∈ { 1 . . .

Original: N } , ∀ j ∈ { 1 . . .
Revised: N } , ∀ j ∈ { 1 . . .

Original: M i } (3)
Revised: M i } (3)

Original: For a pipeline configuration p λ , represented with the indices of its algorithms p , and the configuration vectors of its algorithms λ ( p ) , we project all the pipeline’s algorithms’ configurations to their latent space using the algorithm-specific encoders.
Revised: For a pipeline configuration p λ , represented with the indices of its algorithms p , and the configuration vectors of its algorithms λ ( p ) , we project all the pipeline’s algorithms’ configurations to their latent space using the algorithm-specific encoders.

Original: Then, we concatenate their latent encoder vectors, where our concatenation notation is R L i ⊕ R L k :=
Revised: Then, we concatenate their latent encoder vectors, where our concatenation notation is R L i ⊕ R L k :=

Original: R L i + L k .
Revised: R L i + L k .

Original: Finally, the concatenated representation is embedded to a final R Z space via an aggregating MLP ψ with parameters θ aggr as denoted below:
Revised: Finally, the concatenated representation is embedded to a final R Z space via an aggregating MLP ψ with parameters θ aggr as denoted below:

Original: [Equation]
Revised: [Equation]

Original: Within the i -th stage, only the output of one encoder is concatenated, therefore the output of the
Revised: Within the i -th stage, only the output of one encoder is concatenated, therefore the output of the

Original: Selector corresponds to the active algorithm in the i -th stage and can be formalized as ξ ( i,p i )
Revised: Selector corresponds to the active algorithm in the i -th stage and can be formalized as ξ ( i,p i )

Original: ( λ i,p i  =
Revised: ( λ i,p i ) =

Original: (cid:80) M i
Revised: (cid:80) M i

Original: j =1 I ( j = a ) · ξ ( i,j ) ( λ i,j ) , where a is the index of the active algorithm
Revised: j =1 I ( j = a ) · ξ ( i,j ) ( λ i,j ) , where a is the index of the active algorithm

Original: and I denotes the indicator function.
Revised: and I denotes the indicator function.

Original: Having defined the embedding ϕ in Equations 3-4, we can plug it into the kernel function, optimize it minimizing the negative log likelihood of the GP with respect to θ = { θ enc , θ aggr } , and conduct BO as in Section 3.2.
Revised: Having defined the embedding ϕ in Equations 3-4, we can plug it into the kernel function, optimize it minimizing the negative log likelihood of the GP with respect to θ = { θ enc , θ aggr } , and conduct BO as in Section 3.2.

Original: 4. M ETA - LEARNING OUR PIPELINE EMBEDDING
Revised: 4. M ETA - LEARNING OUR PIPELINE EMBEDDING

Original: In many practical applications, there exist computed evaluations of pipeline configurations on previous datasets, leading to the possibility of transfer learning for PO.
Revised: In many practical applications, there exist computed evaluations of pipeline configurations on previous datasets, leading to the possibility of transfer learning for PO.

Original: Our DeepPipe can be easily meta-learned from such past evaluations by pre-training the pipeline embedding network.
Revised: Our DeepPipe can be easily meta-learned from such past evaluations by pre-training the pipeline embedding network.

Original: Let us denote the meta-dataset of pipeline evaluations on T datasets (a.k.a.
Revised: Let us denote the meta-dataset of pipeline evaluations on T datasets (a.k.a.

Original: auxiliary tasks) as
Revised: auxiliary tasks) as

Original: H t = { ( p λ ( t, 1) , y ( t, 1) ) , . . .
Revised: H t = { ( p λ ( t, 1) , y ( t, 1) ) , . . .

Original: , ( p λ ( t,Q t ) , y ( t,Q t ) ) } , t ∈ { 1 , . .
Revised: , ( p λ ( t,Q t ) , y ( t,Q t ) ) } , t ∈ { 1 , . .

Original: , T } , where Q t is the number of existing evaluations for the t -th dataset.
Revised: , T } , where Q t is the number of existing evaluations for the t -th dataset.

Original: As a result, we meta-learn our method’s parameters to minimize the meta-learning objective of Equation 5.
Revised: As a result, we meta-learn our method’s parameters to minimize the meta-learning objective of Equation 5.

Original: This objective function corresponds to the negative marginal likelihood of the Gaussian Processes using DeepPipe ’s extracted features as input to the kernel (Wistuba & Grabocka, 2021; Patacchiola et al., 2020).
Revised: This objective function corresponds to the negative marginal likelihood of the Gaussian Processes using DeepPipe ’s extracted features as input to the kernel (Wistuba & Grabocka, 2021; Patacchiola et al., 2020).

Original: Further details on the meta-learning procedure of our pipeline configuration weights are provided in Appendix J.
Revised: Further details on the meta-learning procedure of our pipeline configuration weights are provided in Appendix J.

Original: [Equation]
Revised: [Equation]

Original:  E XPERIMENTS AND R ESULTS 5. M ETA -D ATASETS
Revised:  E XPERIMENTS AND R ESULTS 5. M ETA -D ATASETS

Original: A meta-dataset is a collection of pipeline configurations and their respective performance evaluated in different tasks (i.e. datasets).
Revised: A meta-dataset is a collection of pipeline configurations and their respective performance evaluated in different tasks (i.e. datasets).

Original: Information about the meta-data sets is provided on Appendix P, their search spaces are clarified on Appendix R, and the splits of tasks per meta-dataset are found on
Revised: Information about the meta-data sets is provided on Appendix P, their search spaces are clarified on Appendix R, and the splits of tasks per meta-dataset are found on

Original: Appendix T. All the tasks in the meta-dataset correspond to classification.
Revised: Appendix T. All the tasks in the meta-dataset correspond to classification.

Original: We use the meta-training set for pre-training the Pipeline Optimization (PO) methods, the meta-validation set for tuning some of the hyper-parameters of the PO method, and we assess their performance on the meta-test set.
Revised: We use the meta-training set for pre-training the Pipeline Optimization (PO) methods, the meta-validation set for tuning some of the hyper-parameters of the PO method, and we assess their performance on the meta-test set.

Original: In our experiments, we use the following meta-datasets.
Revised: In our experiments, we use the following meta-datasets.

Original: PMF contains 38151 pipelines (after filtering out all pipelines with only NaN entries), anddatasets (Fusi et al., 2018).
Revised: PMF contains 38151 pipelines (after filtering out all pipelines with only NaN entries), anddatasets (Fusi et al., 2018).

Original: Although not all the pipelines were evaluated in all tasks, it has a total of 16M evaluations.
Revised: Although not all the pipelines were evaluated in all tasks, it has a total of 16M evaluations.

Original: The pipeline search space has 2 stages (preprocessing and estimator) with 2 andalgorithms respectively.
Revised: The pipeline search space has 2 stages (preprocessing and estimator) with 2 andalgorithms respectively.

Original: Following the setup in the original paper (Fusi et al., 2018), we taketasks for meta-training and 89 for meta-test.
Revised: Following the setup in the original paper (Fusi et al., 2018), we taketasks for meta-training and 89 for meta-test.

Original: As the authors do not specify a validation meta-dataset, we sample randomly 15 tasks out of the meta-training dataset.
Revised: As the authors do not specify a validation meta-dataset, we sample randomly 15 tasks out of the meta-training dataset.

Original: Tensor-OBOE provides 23424 pipelines and 551 tasks (Yang et al., 2020).
Revised: Tensor-OBOE provides 23424 pipelines and 551 tasks (Yang et al., 2020).

Original: It contains 11M evaluations, as there exist no evaluations for some combinations of pipelines and tasks.
Revised: It contains 11M evaluations, as there exist no evaluations for some combinations of pipelines and tasks.

Original: The pipelines include 5 stages: Imputator (1 algorithm), Dimensionality-Reducer (3 algorithms), Standardizer (1 algorithm),
Revised: The pipelines include 5 stages: Imputator (1 algorithm), Dimensionality-Reducer (3 algorithms), Standardizer (1 algorithm),

Original: Encoder (1 algorithm), and Estimator (11 algorithms).
Revised: Encoder (1 algorithm), and Estimator (11 algorithms).

Original: We assign 331 tasks for meta-training tasks, 110 tasks for meta-validation, and 110 tasks for meta-testing.
Revised: We assign 331 tasks for meta-training tasks, 110 tasks for meta-validation, and 110 tasks for meta-testing.

Original: ZAP is a benchmark that evaluates deep learning pipelines on fine-tuning state-of-the-art computer vision tasks (Ozturk et al., 2022).
Revised: ZAP is a benchmark that evaluates deep learning pipelines on fine-tuning state-of-the-art computer vision tasks (Ozturk et al., 2022).

Original: The meta-dataset contains 275625 evaluated pipeline configurations on 525 datasets and 525 different Deep Learning pipelines (i.e. the best pipeline of a dataset was evaluated also on all other datasets).
Revised: The meta-dataset contains 275625 evaluated pipeline configurations on 525 datasets and 525 different Deep Learning pipelines (i.e. the best pipeline of a dataset was evaluated also on all other datasets).

Original: From the set of datasets, we use 315 for meta-training, 45 for meta-validation and 105 for meta-test, following the protocol of the original paper.
Revised: From the set of datasets, we use 315 for meta-training, 45 for meta-validation and 105 for meta-test, following the protocol of the original paper.

Original: In addition, we use OpenML datasets.
Revised: In addition, we use OpenML datasets.

Original: It comprises 39 curated datasets (Gijsbers et al., 2019) and has been used in previous work for benchmarking (Erickson et al., 2020).
Revised: It comprises 39 curated datasets (Gijsbers et al., 2019) and has been used in previous work for benchmarking (Erickson et al., 2020).

Original: This dataset does not contain pipeline evaluations like the other three meta-datasets above.
Revised: This dataset does not contain pipeline evaluations like the other three meta-datasets above.

Original: However, we use the OpenML collection for evaluating the Pipeline Optimization in time-constrained settings (Ozturk et al., 2022).
Revised: However, we use the OpenML collection for evaluating the Pipeline Optimization in time-constrained settings (Ozturk et al., 2022).

Original: 5. B ASELINES
Revised: 5. B ASELINES

Original: Random Search (RS) selects pipeline configurations by sampling randomly from the search space (Bergstra & Bengio, 2012).
Revised: Random Search (RS) selects pipeline configurations by sampling randomly from the search space (Bergstra & Bengio, 2012).

Original: Probabilistic Matrix Factorization (PMF) uses a surrogate model that learns a latent representation for every pipeline using meta-training tasks (Fusi et al., 2018).
Revised: Probabilistic Matrix Factorization (PMF) uses a surrogate model that learns a latent representation for every pipeline using meta-training tasks (Fusi et al., 2018).

Original: We tuned this latent dimension for the Tensor-OBOE dataset from a grid of { 10 , 15 , 20 } and found 20 to be the best setting.
Revised: We tuned this latent dimension for the Tensor-OBOE dataset from a grid of { 10 , 15 , 20 } and found 20 to be the best setting.

Original: For the
Revised: For the

Original: PMF-Dataset, where the model was introduced, we used the default value of 20.
Revised: PMF-Dataset, where the model was introduced, we used the default value of 20.

Original: We use the original
Revised: We use the original

Original: PMF implementation (Sheth, 2018).
Revised: PMF implementation (Sheth, 2018).

Original: OBOE also uses matrix factorization for optimizing pipelines, but they aim to find fast and informative algorithms to initialize the matrix
Revised: OBOE also uses matrix factorization for optimizing pipelines, but they aim to find fast and informative algorithms to initialize the matrix

Original: (Yang et al., 2019).
Revised: (Yang et al., 2019).

Original: We use the settings provided by the authors.
Revised: We use the settings provided by the authors.

Original: Tensor-OBOE formulates PO as a tensor factorization, where the rank of the tensor is equal to 1+ N , for N being the number of stages in the pipeline (Yang et al., 2020).
Revised: Tensor-OBOE formulates PO as a tensor factorization, where the rank of the tensor is equal to 1+ N , for N being the number of stages in the pipeline (Yang et al., 2020).

Original: We tuned the rank for the
Revised: We tuned the rank for the

Original: Tucker decomposition from a grid of {20,30,40}, resulting in the best value being 30.
Revised: Tucker decomposition from a grid of {20,30,40}, resulting in the best value being 30.

Original: All the other hyper-parameters were set as in the original implementation (Yang et al., 2019).
Revised: All the other hyper-parameters were set as in the original implementation (Yang et al., 2019).

Original: Factorized Multilayer Perceptron (FMLP) creates an ensemble of neural networks with a factorized layer (Schilling et al., 2015).
Revised: Factorized Multilayer Perceptron (FMLP) creates an ensemble of neural networks with a factorized layer (Schilling et al., 2015).

Original: The inputs of the neural network are the one-hot encodings of the algorithms and datasets, in addition to the algorithms’ hyperparameters.
Revised: The inputs of the neural network are the one-hot encodings of the algorithms and datasets, in addition to the algorithms’ hyperparameters.

Original: We tuned the number of base estimators for the ensemble from a grid { 10 , 50 , 100 } , with 100 being the optimal ensemble size.
Revised: We tuned the number of base estimators for the ensemble from a grid { 10 , 50 , 100 } , with 100 being the optimal ensemble size.

Original: Each network layer has 5 neurons and ReLU activations as highlighted in the author’s paper (Schilling et al., 2015).
Revised: Each network layer has 5 neurons and ReLU activations as highlighted in the author’s paper (Schilling et al., 2015).

Original: RGPE builds an ensemble of Gaussian Processes using auxiliary tasks (Feurer et al., 2018).
Revised: RGPE builds an ensemble of Gaussian Processes using auxiliary tasks (Feurer et al., 2018).

Original: The ensemble weights the contributions of every base model and the new model fit on the new task.
Revised: The ensemble weights the contributions of every base model and the new model fit on the new task.

Original: We used the implementation from Botorch (Balandat et al., 2020).
Revised: We used the implementation from Botorch (Balandat et al., 2020).

Original: Gaussian Processes (GP) are a standard and strong baseline in hyperparameter optimization (Snoek et al., 2012).
Revised: Gaussian Processes (GP) are a standard and strong baseline in hyperparameter optimization (Snoek et al., 2012).

Original: We tuned the kernel from {Gaussian, Matérn 5/2}, with Matérn 5/2 performing better.
Revised: We tuned the kernel from {Gaussian, Matérn 5/2}, with Matérn 5/2 performing better.

Original: DNGO uses neural networks as basis functions with a Bayesian linear regressor at the output layer (Snoek et al., 2015).
Revised: DNGO uses neural networks as basis functions with a Bayesian linear regressor at the output layer (Snoek et al., 2015).

Original: We use the implementation provided by Klein & Zela (2020), and its default hyperparameters.
Revised: We use the implementation provided by Klein & Zela (2020), and its default hyperparameters.

Original: SMAC uses Random Forest for predicting uncertainties (Hutter et al., 2011).
Revised: SMAC uses Random Forest for predicting uncertainties (Hutter et al., 2011).

Original: After exploring a grid of { 10 , 50 , 100 } for the number of trees, we found 100 to be the best choice.
Revised: After exploring a grid of { 10 , 50 , 100 } for the number of trees, we found 100 to be the best choice.

Original: TPOT is an AutoML system that conducts PO using evolutionary search (Olson & Moore, 2016).
Revised: TPOT is an AutoML system that conducts PO using evolutionary search (Olson & Moore, 2016).

Original: We use the original implementation but adopted the search space to fit the Tensor-OBOE meta-dataset (see Appendix R).
Revised: We use the original implementation but adopted the search space to fit the Tensor-OBOE meta-dataset (see Appendix R).

Original: 5. R ESEARCH H YPOTHESES AND A SSOCIATED E XPERIMENTS
Revised: 5. R ESEARCH H YPOTHESES AND A SSOCIATED E XPERIMENTS

Original: Hypothesis 1: DeepPipe outperforms standard PO baselines.
Revised: Hypothesis 1: DeepPipe outperforms standard PO baselines.

Original: Experiment 1: We evaluate the performance of DeepPipe when no meta-training data is available.
Revised: Experiment 1: We evaluate the performance of DeepPipe when no meta-training data is available.

Original: We compare against four baselines: Random Search (RS) (Bergstra et al., 2011), Gaussian Processes (GP) (Rasmussen & Williams, 2006), DNGO (Snoek et al., 2015) and SMAC (Hutter et al., 2011).
Revised: We compare against four baselines: Random Search (RS) (Bergstra et al., 2011), Gaussian Processes (GP) (Rasmussen & Williams, 2006), DNGO (Snoek et al., 2015) and SMAC (Hutter et al., 2011).

Original: We evaluate their performances on the aforementioned PMF, Tensor-OBOE and ZAP meta-datasets.
Revised: We evaluate their performances on the aforementioned PMF, Tensor-OBOE and ZAP meta-datasets.

Original: In Experiments 1 and 2 (below) we select 5 initial observations to warm-start the BO, then we runadditional iterations.
Revised: In Experiments 1 and 2 (below) we select 5 initial observations to warm-start the BO, then we runadditional iterations.

Original: The procedure for choosing these configurations is detailed in the Appendix G.
Revised: The procedure for choosing these configurations is detailed in the Appendix G.

Original: Hypothesis 2: Our meta-learned DeepPipe outperforms state-of-the-art transfer-learning PO methods.
Revised: Hypothesis 2: Our meta-learned DeepPipe outperforms state-of-the-art transfer-learning PO methods.

Original: Experiment 2: We compare our proposed method against baselines that use auxiliary tasks (a.k.a.
Revised: Experiment 2: We compare our proposed method against baselines that use auxiliary tasks (a.k.a.

Original: meta-training data) for improving the performance of Pipeline Optimization: Probabilistic Matrix
Revised: meta-training data) for improving the performance of Pipeline Optimization: Probabilistic Matrix

Original: Factorization (PMF) (Fusi et al., 2018), Factorized Multilayer Perceptron (FMLP) (Schilling et al., 2015), OBOE (Yang et al., 2019) and Tensor OBOE
Revised: Factorization (PMF) (Fusi et al., 2018), Factorized Multilayer Perceptron (FMLP) (Schilling et al., 2015), OBOE (Yang et al., 2019) and Tensor OBOE

Original: (Yang et al., 2020).
Revised: (Yang et al., 2020).

Original: Moreover, we compare to
Revised: Moreover, we compare to

Original: RGPE (Feurer et al., 2018), an effective baseline for transfer HPO (Arango et al., 2021).
Revised: RGPE (Feurer et al., 2018), an effective baseline for transfer HPO (Arango et al., 2021).

Original: We evaluate the performances on the PMF and Tensor-OBOE meta-datasets.
Revised: We evaluate the performances on the PMF and Tensor-OBOE meta-datasets.

Original: Hypothesis 3: DeepPipe leads to strong any-time results in a time-constrained PO problem.
Revised: Hypothesis 3: DeepPipe leads to strong any-time results in a time-constrained PO problem.

Original: Experiment 3: Oftentimes practitioners need AutoML systems that discover efficient pipelines within a small time budget.
Revised: Experiment 3: Oftentimes practitioners need AutoML systems that discover efficient pipelines within a small time budget.

Original: To test the convergence speed of our PO method we ran it on the aforementioned
Revised: To test the convergence speed of our PO method we ran it on the aforementioned

Original: OpenML datasets for a budget of 10 minutes, and also 1 hour.
Revised: OpenML datasets for a budget of 10 minutes, and also 1 hour.

Original: We compare against five baselines: (i) TPOT
Revised: We compare against five baselines: (i) TPOT

Original: (Olson & Moore, 2016) adapted to the search space of Tensor-OBOE (see Appendix R), (ii) OBOE and Tensor-OBOE
Revised: (Olson & Moore, 2016) adapted to the search space of Tensor-OBOE (see Appendix R), (ii) OBOE and Tensor-OBOE

Original: (Yang et al., 2019; 2020) using the time-constrained version provided by the authors, (iii)
Revised: (Yang et al., 2019; 2020) using the time-constrained version provided by the authors, (iii)

Original: SMAC (Hutter et al., 2011), and (iv) PMF (Fusi et al., 2018).
Revised: SMAC (Hutter et al., 2011), and (iv) PMF (Fusi et al., 2018).

Original: The last three had the same five initial configurations used to warm-start BO as detailed in Experiment 1.
Revised: The last three had the same five initial configurations used to warm-start BO as detailed in Experiment 1.

Original: Moreover, they were pre-trained with the Tensor-OBOE meta-dataset and all the method-specific settings are the same as in Experiment 2.
Revised: Moreover, they were pre-trained with the Tensor-OBOE meta-dataset and all the method-specific settings are the same as in Experiment 2.

Original: Hypothesis 4: Our novel encoder layers of DeepPipe enable an efficient PO when the pipeline search space changes, i.e. when developers add a new algorithm to a ML system.
Revised: Hypothesis 4: Our novel encoder layers of DeepPipe enable an efficient PO when the pipeline search space changes, i.e. when developers add a new algorithm to a ML system.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Experiment 4: A major obstacle to meta-learning PO solutions is that they do not generalize when the search space changes, especially when the developers of ML systems add new algorithms.
Revised: Experiment 4: A major obstacle to meta-learning PO solutions is that they do not generalize when the search space changes, especially when the developers of ML systems add new algorithms.

Original: Our architecture quickly adapts to newly added algorithms because only an encoder sub-network for the new algorithm should be trained .
Revised: Our architecture quickly adapts to newly added algorithms because only an encoder sub-network for the new algorithm should be trained .

Original: To test the scenario, we ablate the performance of five versions of DeepPipe and try different settings when we remove a specific algorithm (an estimator) either from meta-training, meta-testing
Revised: To test the scenario, we ablate the performance of five versions of DeepPipe and try different settings when we remove a specific algorithm (an estimator) either from meta-training, meta-testing

Original: [Figure]
Revised: [Figure]

Original: or both.
Revised: or both.

Original: Hypothesis 5: The encoders in DeepPipe introduce an inductive bias where latent representation vectors of an algorithm’s configurations are co-located, and located distantly from the representations of other algorithms’ configurations.
Revised: Hypothesis 5: The encoders in DeepPipe introduce an inductive bias where latent representation vectors of an algorithm’s configurations are co-located, and located distantly from the representations of other algorithms’ configurations.

Original: Formally, given three pipelines p ( l ) , p ( m ) , p ( n ) if p ( l ) i = p ( m ) i , p ( l ) i ̸
Revised: Formally, given three pipelines p ( l ) , p ( m ) , p ( n ) if p ( l ) i = p ( m ) i , p ( l ) i ̸

Original: = p ( n ) i then
Revised: = p ( n ) i then

Original: || ϕ
Revised: || ϕ

Original: ( p ( l ) ) − ϕ ( p ( m ) )
Revised: ( p ( l ) ) − ϕ ( p ( m ) )

Original: || <
Revised: || <

Original: || ϕ ( p ( m ) )
Revised: || ϕ ( p ( m ) )

Original: − ϕ ( p ( n ) )
Revised: − ϕ ( p ( n ) )

Original: || with higher probability when using encoder layers, given that p ( n ) i is the index of the algorithm in the i -th stage.
Revised: || with higher probability when using encoder layers, given that p ( n ) i is the index of the algorithm in the i -th stage.

Original: Furthermore, we hypothesize that the less number of tasks during pre-training, the more necessary this inductive bias is.
Revised: Furthermore, we hypothesize that the less number of tasks during pre-training, the more necessary this inductive bias is.

Original: Experiment 5: We sample 2000 pipelines of 5 estimation algorithms on the TensorOBOE dataset.
Revised: Experiment 5: We sample 2000 pipelines of 5 estimation algorithms on the TensorOBOE dataset.

Original: Subsequently, we embed the pipelines using a DeepPipe with 0, 1 and 2 encoder layers, and weights θ , initialized such that θ i ∈
Revised: Subsequently, we embed the pipelines using a DeepPipe with 0, 1 and 2 encoder layers, and weights θ , initialized such that θ i ∈

Original: θ are independently identically distributed θ i ∼ N (0 , 1) .
Revised: θ are independently identically distributed θ i ∼ N (0 , 1) .

Original: Finally, we visualize the embeddings with T-SNE (Van der Maaten & Hinton, 2008) and compute a cluster metric to assess how close pipelines with the same algorithm are in the latent space:
Revised: Finally, we visualize the embeddings with T-SNE (Van der Maaten & Hinton, 2008) and compute a cluster metric to assess how close pipelines with the same algorithm are in the latent space:

Original: E p ( l ) ,p ( m ) ,p ( n ) ( I ( || ϕ ( p ( l ) ) − ϕ ( p ( m ) )
Revised: E p ( l ) ,p ( m ) ,p ( n ) ( I ( || ϕ ( p ( l ) ) − ϕ ( p ( m ) )

Original: || <
Revised: || <

Original: || ϕ ( p ( m ) )
Revised: || ϕ ( p ( m ) )

Original: − ϕ ( p ( n ) ) || )) .
Revised: − ϕ ( p ( n ) ) || )) .

Original: To test the importance of the inductive bias vs the number of pre-training tasks, we ablate the performance of DeepPipe for different percentage of pre-training tasks (0.5%, 1%, 5%, 10%, 50%, 100%) under different values of encoder layers.
Revised: To test the importance of the inductive bias vs the number of pre-training tasks, we ablate the performance of DeepPipe for different percentage of pre-training tasks (0.5%, 1%, 5%, 10%, 50%, 100%) under different values of encoder layers.

Original: 5. E XPERIMENTAL S ETUP FOR DeepPipe
Revised: 5. E XPERIMENTAL S ETUP FOR DeepPipe

Original: The encoders and the aggregated layers are Multilayer Perceptrons with ReLU activations.
Revised: The encoders and the aggregated layers are Multilayer Perceptrons with ReLU activations.

Original: We keep an architecture that is proportional to the input size.
Revised: We keep an architecture that is proportional to the input size.

Original: The number of neurons in the hidden layers for the encoder of algorithm j -th in i -th stage with | Λ i,j | hyperparameters is F ·| Λ i,j | , given an integer factor
Revised: The number of neurons in the hidden layers for the encoder of algorithm j -th in i -th stage with | Λ i,j | hyperparameters is F ·| Λ i,j | , given an integer factor

Original: F .
Revised: F .

Original: The output dimension of the encoders of the i -th stage is defined as Q i = max j | Λ i,j | .
Revised: The output dimension of the encoders of the i -th stage is defined as Q i = max j | Λ i,j | .

Original: The number of total layers (i.e. encoder and aggregation layers) is fixed to 4.
Revised: The number of total layers (i.e. encoder and aggregation layers) is fixed to 4.

Original: The number of encoders is chosen from {0,1,2}.
Revised: The number of encoders is chosen from {0,1,2}.

Original: The specific values of the encoders’ input dimensions are detailed in Appendix R.
Revised: The specific values of the encoders’ input dimensions are detailed in Appendix R.

Original: We choose F ∈ { 4 , 6 , 8 , 10 } based on the performance in the validation split.
Revised: We choose F ∈ { 4 , 6 , 8 , 10 } based on the performance in the validation split.

Original: Specifically, we use the following values for DeepPipe : (i) in Experiment 1: 1 encoder layer (all meta-datasets), F =(PMF and ZAP) and F = 

Revised: Specifically, we use the following values for DeepPipe : (i) in Experiment 1: 1 encoder layer (all meta-datasets), F =(PMF and ZAP) and F =

Original: (Tensor-OBOE), (ii) in Experiment 2: F = 8 , no encoder layer (PMF,
Revised: (Tensor-OBOE), (ii) in Experiment 2: F = 8 , no encoder layer (PMF,

Original: Tensor-OBOE) and one encoder layer (ZAP), (iii) in Experiment 3: F = 8 and no encoder layers, (iv) in Experiment 4 we use F = 8 and { 0 , 1 } encoder layers.
Revised: Tensor-OBOE) and one encoder layer (ZAP), (iii) in Experiment 3: F = 8 and no encoder layers, (iv) in Experiment 4 we use F = 8 and { 0 , 1 } encoder layers.

Original: Finally (iv) in Experiment 5 we use
Revised: Finally (iv) in Experiment 5 we use

Original: F = 8 and { 0 , 1 , 2 } encoder layers.
Revised: F = 8 and { 0 , 1 , 2 } encoder layers.

Original: Additional details on the setup can be found in the Appendix G and our source code 2 .
Revised: Additional details on the setup can be found in the Appendix G and our source code 2 .

Original:  R ESULTS
Revised:  R ESULTS

Original: We present the results for Experiments 1 and 2 in Figures 2 and 3, respectively.
Revised: We present the results for Experiments 1 and 2 in Figures 2 and 3, respectively.

Original: In both cases, we compute the ranks of the classification accuracy achieved by the discovered pipelines of each technique, averaged across the meta-testing datasets.
Revised: In both cases, we compute the ranks of the classification accuracy achieved by the discovered pipelines of each technique, averaged across the meta-testing datasets.

Original: The shadowed lines correspond to the 95% confidence intervals.
Revised: The shadowed lines correspond to the 95% confidence intervals.

Original: Additional results showing the mean regret are included in Appendix L.
Revised: Additional results showing the mean regret are included in Appendix L.

Original: In
Revised: In

Original: Experiment 1 (standard/non-transfer PO) DeepPipe achieved the best performance for both metadatasets, whereas SMAC attained the second place.
Revised: Experiment 1 (standard/non-transfer PO) DeepPipe achieved the best performance for both metadatasets, whereas SMAC attained the second place.

Original: In Experiment 2 DeepPipe strongly outperforms all the transfer-learning PO baselines in all meta-datasets.
Revised: In Experiment 2 DeepPipe strongly outperforms all the transfer-learning PO baselines in all meta-datasets.

Original: Given that DeepPipe yields state-of-the-art
Revised: Given that DeepPipe yields state-of-the-art

Original: PO results on both standard and transfer-learning setups, we conclude that our pipeline embedding network computes efficient representations for PO with Bayesian Optimization.
Revised: PO results on both standard and transfer-learning setups, we conclude that our pipeline embedding network computes efficient representations for PO with Bayesian Optimization.

Original: In particular, the results on the ZAP meta-dataset indicate the efficiency of DeepPipe in discovering state-of-the-art
Revised: In particular, the results on the ZAP meta-dataset indicate the efficiency of DeepPipe in discovering state-of-the-art

Original: Deep Learning pipelines for computer vision.
Revised: Deep Learning pipelines for computer vision.

Original: [Table]
Revised: [Table]

Original: Experiment 3 conducted on the
Revised: Experiment 3 conducted on the

Original: OpenML datasets shows that DeepPipe performs well under restricted budgets (see Table 1).
Revised: OpenML datasets shows that DeepPipe performs well under restricted budgets (see Table 1).

Original: Although our method does not incorporate any direct way to handle time constraints, it is interesting to note that it outperforms other methods that include heuristics for handling a quick convergence, such as OBOE and TensorOBOE.
Revised: Although our method does not incorporate any direct way to handle time constraints, it is interesting to note that it outperforms other methods that include heuristics for handling a quick convergence, such as OBOE and TensorOBOE.

Original: Furthermore, the results of Experiment 4 indicate that our DeepPipe embedding quickly adapts to incrementally-expanding search spaces, e.g. when the developers of a ML system add new algorithms.
Revised: Furthermore, the results of Experiment 4 indicate that our DeepPipe embedding quickly adapts to incrementally-expanding search spaces, e.g. when the developers of a ML system add new algorithms.

Original: In this circumstance, existing transfer-learning PO baselines do not adapt easily, because they assume a static pipeline search space.
Revised: In this circumstance, existing transfer-learning PO baselines do not adapt easily, because they assume a static pipeline search space.

Original: As a remedy, when a new algorithm is added to the system after meta-learning our pipeline embedding network, we train only a new encoder for that new algorithm.
Revised: As a remedy, when a new algorithm is added to the system after meta-learning our pipeline embedding network, we train only a new encoder for that new algorithm.

Original: In this experiment, we run our method on variants of the search space when one algorithm at a time is introduced to the search space (for instance an estimator, e.g. MLP, RF, etc., is not known during meta-training, but added new to the meta-testing).
Revised: In this experiment, we run our method on variants of the search space when one algorithm at a time is introduced to the search space (for instance an estimator, e.g. MLP, RF, etc., is not known during meta-training, but added new to the meta-testing).

Original: In Tables 2 and 5 (in Appendix), we present the values of the average rank among five different configurations for DeepPipe .
Revised: In Tables 2 and 5 (in Appendix), we present the values of the average rank among five different configurations for DeepPipe .

Original: We compare among meta-trained versions (denoted by ✓ in the column
Revised: We compare among meta-trained versions (denoted by ✓ in the column

Original: MTd. )
Revised: MTd. )

Original: that omit specific estimators during meta-training ( MTr.
Revised: that omit specific estimators during meta-training ( MTr.

Original: = ✓ ), or during meta-testing ( MTe.
Revised: = ✓ ), or during meta-testing ( MTe.

Original: = ✓ ).
Revised: = ✓ ).

Original: We also account for versions with one encoder layer denoted by ✓ in the column Enc .
Revised: We also account for versions with one encoder layer denoted by ✓ in the column Enc .

Original: [Figure]
Revised: [Figure]

Original: The best in all cases is the meta-learned model that did not omit the estimator (i.e. algorithm known and prior evaluations with that algorithm exist).
Revised: The best in all cases is the meta-learned model that did not omit the estimator (i.e. algorithm known and prior evaluations with that algorithm exist).

Original: Among the versions that omitted the estimator in the meta-training set (i.e. algorithm added new), the best configuration was the DeepPipe which fine-tuned a new encoder for that algorithm (line Enc = ✓ , MTd.
Revised: Among the versions that omitted the estimator in the meta-training set (i.e. algorithm added new), the best configuration was the DeepPipe which fine-tuned a new encoder for that algorithm (line Enc = ✓ , MTd.

Original: = ✓ , MTr.
Revised: = ✓ , MTr.

Original: = ✓ , MTe.
Revised: = ✓ , MTe.

Original: = ✗ ).
Revised: = ✗ ).

Original: This version of DeepPipe performs better than ablations with no encoder layers (i.e. only aggregation layers ϕ ), or the one omitting the algorithm during meta-testing (i.e. pipelines that do not use the new algorithm at all).
Revised: This version of DeepPipe performs better than ablations with no encoder layers (i.e. only aggregation layers ϕ ), or the one omitting the algorithm during meta-testing (i.e. pipelines that do not use the new algorithm at all).

Original: The message of the results is simple: If we add a new algorithm to a ML system, instead of running PO without meta-learning (because the search space changes and existing transfer
Revised: The message of the results is simple: If we add a new algorithm to a ML system, instead of running PO without meta-learning (because the search space changes and existing transfer

Original: PO baselines are not applicable to the new space), we can use a meta-learned DeepPipe and only fine-tune an encoder for a new algorithm.
Revised: PO baselines are not applicable to the new space), we can use a meta-learned DeepPipe and only fine-tune an encoder for a new algorithm.

Original: [Table]
Revised: [Table]

Original: The effect of the inductive bias introduced by the encoders (Experiment 5) can be appreciated in
Revised: The effect of the inductive bias introduced by the encoders (Experiment 5) can be appreciated in

Original: Figure 4.
Revised: Figure 4.

Original: The pipelines with the same active algorithm in the estimation stage, but with different hyperparameters, lie closer in the embedding space created by a random initialized DeepPipe , forming compact clusters characterized by the defined cluster metric (value below the plots).
Revised: The pipelines with the same active algorithm in the estimation stage, but with different hyperparameters, lie closer in the embedding space created by a random initialized DeepPipe , forming compact clusters characterized by the defined cluster metric (value below the plots).

Original: We formally demonstrate in Appendix S that, in general, a single encoder layer is creating more compact clusters than a fully connected linear layer.
Revised: We formally demonstrate in Appendix S that, in general, a single encoder layer is creating more compact clusters than a fully connected linear layer.

Original: In additional results (Appendix F), we observe that the average rank on the test-tasks improves for DeepPipe versions with deeper encoder layers (keeping the total number of layers fixed), if the number of meta-training tasks gets lower.
Revised: In additional results (Appendix F), we observe that the average rank on the test-tasks improves for DeepPipe versions with deeper encoder layers (keeping the total number of layers fixed), if the number of meta-training tasks gets lower.

Original: This occurs because the objective function (Equation 5) makes possible to learn embeddings where pipelines with similar performance are clustered together (see Appendix O) given enough meta-training data.
Revised: This occurs because the objective function (Equation 5) makes possible to learn embeddings where pipelines with similar performance are clustered together (see Appendix O) given enough meta-training data.

Original: Otherwise, the inductive bias introduced by the encoders becomes more relevant.
Revised: Otherwise, the inductive bias introduced by the encoders becomes more relevant.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we have shown that efficient Machine Learning pipeline representations can be computed with deep modular networks.
Revised: In this paper, we have shown that efficient Machine Learning pipeline representations can be computed with deep modular networks.

Original: Such representations help discovering more accurate pipelines compared to the state-of-art approaches, because they capture the interactions of the different pipelines algorithms and their hyperparameters.
Revised: Such representations help discovering more accurate pipelines compared to the state-of-art approaches, because they capture the interactions of the different pipelines algorithms and their hyperparameters.

Original: Moreover, we show that introducing per-algorithm encoders helps in the case of limited meta-trained data, or when a new algorithm is added to the search space.
Revised: Moreover, we show that introducing per-algorithm encoders helps in the case of limited meta-trained data, or when a new algorithm is added to the search space.

Original: Overall, we demonstrate that our method DeepPipe achieves the new state-of-the-art in Pipeline
Revised: Overall, we demonstrate that our method DeepPipe achieves the new state-of-the-art in Pipeline

Original: Optimization.
Revised: Optimization.

Original: Limitations.
Revised: Limitations.

Original: Our representation network does not model complex parallel pipelines (in that case the embedding will be a graph neural network), and/or pipelines involving ensembles.
Revised: Our representation network does not model complex parallel pipelines (in that case the embedding will be a graph neural network), and/or pipelines involving ensembles.

Original: We plan to investigate these important points in our future work.
Revised: We plan to investigate these important points in our future work.

Original: Reproducibility Statement .
Revised: Reproducibility Statement .

Original: To guarantee the reproducibility of or work, we include an anonymized repository to the related code.
Revised: To guarantee the reproducibility of or work, we include an anonymized repository to the related code.

Original: The code for the baselines is included also, and we reference the original implementations if it is the case.
Revised: The code for the baselines is included also, and we reference the original implementations if it is the case.

Original: All the meta-datasets are publicly available and correspondingly referenced.
Revised: All the meta-datasets are publicly available and correspondingly referenced.

Original: A P OTENTIAL N EGATIVE S OCIETAL I MPACTS
Revised: A P OTENTIAL N EGATIVE S OCIETAL I MPACTS

Original: The meta-training is the most demanding computational step, thus it can incur in high energy consumption.
Revised: The meta-training is the most demanding computational step, thus it can incur in high energy consumption.

Original: Additionally, DeepPipe does not handle fairness, so it may find pipelines that are biased by the data.
Revised: Additionally, DeepPipe does not handle fairness, so it may find pipelines that are biased by the data.

Original: B L ICENCE C LARIFICATION
Revised: B L ICENCE C LARIFICATION

Original: The results of this work (code, data) are under license BSD-3-Clause license.
Revised: The results of this work (code, data) are under license BSD-3-Clause license.

Original: Both PMF dataset
Revised: Both PMF dataset

Original: Sheth (2018) and Tensor-OBOE dataset Akimoto & Yang (2020) hold the same license.
Revised: Sheth (2018) and Tensor-OBOE dataset Akimoto & Yang (2020) hold the same license.

Original: C D ISCUSSION ON N UMBER OF E VALUATED P IPELINES
Revised: C D ISCUSSION ON N UMBER OF E VALUATED P IPELINES

Original: [Table]
Revised: [Table]

Original: Based on results from Experiment 3, we report the average (and standard deviation) of the number of observed pipelines among all the compared methods in 10 and 60 minutes on Table 3.
Revised: Based on results from Experiment 3, we report the average (and standard deviation) of the number of observed pipelines among all the compared methods in 10 and 60 minutes on Table 3.

Original: This is an important metric to understand the optimization overhead introduced by the method.
Revised: This is an important metric to understand the optimization overhead introduced by the method.

Original: For instance, a method that explores few pipelines during a fixed time window, might use expensive computations during the pipeline optimization.
Revised: For instance, a method that explores few pipelines during a fixed time window, might use expensive computations during the pipeline optimization.

Original: We notice that DeepPipe achieves the best results (see Table 1) by using a reasonable amount of pipelines, i.e. the optimization overhead introduced our method is small compared to other approaches such as TPOT and OBOE.
Revised: We notice that DeepPipe achieves the best results (see Table 1) by using a reasonable amount of pipelines, i.e. the optimization overhead introduced our method is small compared to other approaches such as TPOT and OBOE.

Original: D D ISCUSSION ON THE I NTERACTIONS AMONG C OMPONENTS
Revised: D D ISCUSSION ON THE I NTERACTIONS AMONG C OMPONENTS

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Ablation
Revised: Ablation

Original: [Figure]
Revised: [Figure]

Original: The encoder and aggregation layers capture interactions among the pipeline components, and therefore are important to attain good performance.
Revised: The encoder and aggregation layers capture interactions among the pipeline components, and therefore are important to attain good performance.

Original: These interactions are reflected in the features extracted by these layers, i.e. the pipelines representations obtained by DeepPipe .
Revised: These interactions are reflected in the features extracted by these layers, i.e. the pipelines representations obtained by DeepPipe .

Original: These representations lie on a metric space that captures relevant information about the pipelines and which can be used on the kernel for the
Revised: These representations lie on a metric space that captures relevant information about the pipelines and which can be used on the kernel for the

Original: Gaussian Process.
Revised: Gaussian Process.

Original: Using the original input space does not allow to extract rich representations.
Revised: Using the original input space does not allow to extract rich representations.

Original: To test this idea, we meta-train four version of
Revised: To test this idea, we meta-train four version of

Original: DeepPipe with and without encoder and aggregation layers on our TensorOBOE meta-train set and then test on the meta-test split.
Revised: DeepPipe with and without encoder and aggregation layers on our TensorOBOE meta-train set and then test on the meta-test split.

Original: In Figure 5, we show that the best version is obtained when using both encoder ( Enc. )
Revised: In Figure 5, we show that the best version is obtained when using both encoder ( Enc. )

Original: and aggregation ( Agg. )
Revised: and aggregation ( Agg. )

Original: layers (green line), whereas the worst version is obtained when using the original input space, i.e. no encoder and no aggregation layers.
Revised: layers (green line), whereas the worst version is obtained when using the original input space, i.e. no encoder and no aggregation layers.

Original: Having encoder helps more than not having encoder, thus it is important to capture interactions among hyperparameters in the same stage.
Revised: Having encoder helps more than not having encoder, thus it is important to capture interactions among hyperparameters in the same stage.

Original: As having an aggregation layer is better than not, it is important to capture interactions among components from different stages.
Revised: As having an aggregation layer is better than not, it is important to capture interactions among components from different stages.

Original: 5   100 No. of Explored Pipelines 1 .1 .
Revised: 5   100 No. of Explored Pipelines 1 .1 .

Original: TensorOBOE 5   100 No. of Explored Pipelines 1 .2 .2 .
Revised: TensorOBOE 5   100 No. of Explored Pipelines 1 .2 .2 .

Original: PMF 5   100 No. of Explored Pipelines 1 .1 .1 .1 .1 .
Revised: PMF 5   100 No. of Explored Pipelines 1 .1 .1 .1 .1 .

Original: ZAP
Revised: ZAP

Original: DeepPipe
Revised: DeepPipe

Original: SKL
Revised: SKL

Original: E BO
Revised: E BO

Original: Method
Revised: Method

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: E C OMPARISON WITH S TRUCTURED K ERNEL L EARNING (SKL) AND A UTO P ROGNOSIS
Revised: E C OMPARISON WITH S TRUCTURED K ERNEL L EARNING (SKL) AND A UTO P ROGNOSIS

Original: AutoPrognosis (Alaa & van der Schaar, 2018) uses Structured Kernel Learning (SKL) and metalearning to account for the interactions among the pipelines components.
Revised: AutoPrognosis (Alaa & van der Schaar, 2018) uses Structured Kernel Learning (SKL) and metalearning to account for the interactions among the pipelines components.

Original: SKL decomposes the original input space by making up N group of pipelines components, e.g. Random Forest and SVM in a group separated from Linear Regression and Logistic Regression.
Revised: SKL decomposes the original input space by making up N group of pipelines components, e.g. Random Forest and SVM in a group separated from Linear Regression and Logistic Regression.

Original: The hyperparameters of every group of pipelines components is then passed through a kernel, and then the N resulting kernels are added.
Revised: The hyperparameters of every group of pipelines components is then passed through a kernel, and then the N resulting kernels are added.

Original: This effectively builds up a kernel with additive structure (Gardner et al., 2017), however they are not using a feature extractor like DeepPipe .
Revised: This effectively builds up a kernel with additive structure (Gardner et al., 2017), however they are not using a feature extractor like DeepPipe .

Original: We compare SKL against a non-pretrained DeepPipe on Figure 6 on three meta-datasets, where it is noticeable that our method outperforms this strategy.
Revised: We compare SKL against a non-pretrained DeepPipe on Figure 6 on three meta-datasets, where it is noticeable that our method outperforms this strategy.

Original: Additionally we compare DeepPipe with the whole algorithm introduced by AutoPrognosis 2.0 (Imrie et al., 2022) on the Open ML datasets for 50 and 100 BO iterations ( E BO ).
Revised: Additionally we compare DeepPipe with the whole algorithm introduced by AutoPrognosis 2.0 (Imrie et al., 2022) on the Open ML datasets for 50 and 100 BO iterations ( E BO ).

Original: We report the average and standard deviation for the rank, accuracy and time.
Revised: We report the average and standard deviation for the rank, accuracy and time.

Original: DeepPipe achieves the best average rank, ie.
Revised: DeepPipe achieves the best average rank, ie.

Original: lower average rank than AutoPrognosis.
Revised: lower average rank than AutoPrognosis.

Original: This is complemented with the having the highest average accuracy.
Revised: This is complemented with the having the highest average accuracy.

Original: Interestingly, our method is approximately one order of magnitude faster than AutoPrognosis.
Revised: Interestingly, our method is approximately one order of magnitude faster than AutoPrognosis.

Original: We notice this is due to the time overhead introduced by the Gibbs sampling strategy for optimizing the structured kernel, whereas our approach uses gradient-based optimization.
Revised: We notice this is due to the time overhead introduced by the Gibbs sampling strategy for optimizing the structured kernel, whereas our approach uses gradient-based optimization.

Original: Experimental Set-Up for DeepPipe .
Revised: Experimental Set-Up for DeepPipe .

Original: For our comparison with SKL, we use the same hyperparameters and architecture as for the Experiment 1.
Revised: For our comparison with SKL, we use the same hyperparameters and architecture as for the Experiment 1.

Original: When comparing with AutoPrognosis, we use the same hyperparmeters and architecture as for the Experiment 2, pre-trained on the Tensor-OBOE meta-train split.
Revised: When comparing with AutoPrognosis, we use the same hyperparmeters and architecture as for the Experiment 2, pre-trained on the Tensor-OBOE meta-train split.

Original: Experimental Set-Up for SKL and AutoPrognosis For SKL we used the default strategy with
Revised: Experimental Set-Up for SKL and AutoPrognosis For SKL we used the default strategy with

Original: N = 3 (Alaa & van der Schaar, 2018).
Revised: N = 3 (Alaa & van der Schaar, 2018).

Original: For AutoPrognosis, we use the implementation in the respective author’s repository 3 .
Revised: For AutoPrognosis, we use the implementation in the respective author’s repository 3 .

Original: We ran it with the default configuration, but limited the search space of classifiers to match the classifiers on the Tensor-OBOE meta-dataset 4 .
Revised: We ran it with the default configuration, but limited the search space of classifiers to match the classifiers on the Tensor-OBOE meta-dataset 4 .

Original: F D ISCUSSION ON THE I NDUCTIVE B IAS VS . P RE - TRAINING E FFECT
Revised: F D ISCUSSION ON THE I NDUCTIVE B IAS VS . P RE - TRAINING E FFECT

Original: How shallow/deep should the encoder networks be compared to the aggregation network?
Revised: How shallow/deep should the encoder networks be compared to the aggregation network?

Original: We hypothesize that deeper encoders help in the transfer-learning setup where there exist only a few evaluated pipeline configurations on past datasets.
Revised: We hypothesize that deeper encoders help in the transfer-learning setup where there exist only a few evaluated pipeline configurations on past datasets.

Original: To test this hypothesis, we assess the performance of DeepPipe with different network sizes and meta-trained with different percentages of meta-training tasks: 0.5%, 1% , 5%, 10%, 50%, and 100%.
Revised: To test this hypothesis, we assess the performance of DeepPipe with different network sizes and meta-trained with different percentages of meta-training tasks: 0.5%, 1% , 5%, 10%, 50%, and 100%.

Original: As we use the Tensor-OBOE meta-dataset, this effectively means that we use 1, 3, 16, 33, 165, and 330 tasks respectively.
Revised: As we use the Tensor-OBOE meta-dataset, this effectively means that we use 1, 3, 16, 33, 165, and 330 tasks respectively.

Original: [Figure]
Revised: [Figure]

Original: The results of these experiments are shown in Figure 7.
Revised: The results of these experiments are shown in Figure 7.

Original: Here we ablate
Revised: Here we ablate

Original: DeepPipe with different numbers of encoder layers while pre-training on different fractions of the meta-training tasks.
Revised: DeepPipe with different numbers of encoder layers while pre-training on different fractions of the meta-training tasks.

Original: We ran the experiment for three values of F .
Revised: We ran the experiment for three values of F .

Original: The presented scores are the average ranks among the three
Revised: The presented scores are the average ranks among the three

Original: DeepPipe configurations (row-wise).
Revised: DeepPipe configurations (row-wise).

Original: The average rank is computed across all the meta-test tasks and across
Revised: The average rank is computed across all the meta-test tasks and across

Original: BO iterations.
Revised: BO iterations.

Original: The results indicate that deeper encoders achieve a better performance when a small number of meta-training tasks is available.
Revised: The results indicate that deeper encoders achieve a better performance when a small number of meta-training tasks is available.

Original: In contrast, shallower encoders are needed if more meta-training tasks are available.
Revised: In contrast, shallower encoders are needed if more meta-training tasks are available.

Original: Apparently the deep aggregation layers ϕ already capture the interaction between the hyperparameter configurations across algorithms when a large meta-dataset of evaluated pipelines is given.
Revised: Apparently the deep aggregation layers ϕ already capture the interaction between the hyperparameter configurations across algorithms when a large meta-dataset of evaluated pipelines is given.

Original: The smaller the meta-data of evaluated pipeline configurations, the more inductive bias we need to implant in the form of per-algorithm encoders.
Revised: The smaller the meta-data of evaluated pipeline configurations, the more inductive bias we need to implant in the form of per-algorithm encoders.

Original: G A DDITIONAL I NFORMATION ON E XPERIMENTAL S ET - UP
Revised: G A DDITIONAL I NFORMATION ON E XPERIMENTAL S ET - UP

Original: In all experiments (except Experiment 1), we meta-train the surrogate following Algorithm 1 in
Revised: In all experiments (except Experiment 1), we meta-train the surrogate following Algorithm 1 in

Original: Appendix J for 10000 epochs with the Adam optimizer and a learning rate of 10 − 4 , batch size 1000 , and the Matérn Kernel.
Revised: Appendix J for 10000 epochs with the Adam optimizer and a learning rate of 10 − 4 , batch size 1000 , and the Matérn Kernel.

Original: During meta-testing, when we perform BO to search for a pipeline, we fine-tune only the kernel parameters γ for 100 gradient steps.
Revised: During meta-testing, when we perform BO to search for a pipeline, we fine-tune only the kernel parameters γ for 100 gradient steps.

Original: In the non-transfer experiments (Experiment 1) we use an architecture with F = 8 and fine-tuned the network for 10000 iterations.
Revised: In the non-transfer experiments (Experiment 1) we use an architecture with F = 8 and fine-tuned the network for 10000 iterations.

Original: The rest of the training settings are similar to the transfer experiments.
Revised: The rest of the training settings are similar to the transfer experiments.

Original: In Experiment 5 we finetune the whole network for 100 steps when no encoders are used.
Revised: In Experiment 5 we finetune the whole network for 100 steps when no encoders are used.

Original: Otherwise, we fine-tune only the encoder associated with the omitted estimator and freeze the rest of the network.
Revised: Otherwise, we fine-tune only the encoder associated with the omitted estimator and freeze the rest of the network.

Original: We ran all experiments on a CPU cluster, where each node contains two Intel Xeon E5-2630v4 CPUs with 20 CPU cores each, running at 2.2 GHz.
Revised: We ran all experiments on a CPU cluster, where each node contains two Intel Xeon E5-2630v4 CPUs with 20 CPU cores each, running at 2.2 GHz.

Original: We reserved a total maximum memory of 16GB.
Revised: We reserved a total maximum memory of 16GB.

Original: We discuss how we implemented DeepPipe efficiently as a MLP with masked layers 5 in Appendix N.
Revised: We discuss how we implemented DeepPipe efficiently as a MLP with masked layers 5 in Appendix N.

Original: We associate algorithms with no hyperparameters to the same encoder.
Revised: We associate algorithms with no hyperparameters to the same encoder.

Original: We found that adding the
Revised: We found that adding the

Original: One-Hot-Encoding of the selected algorithms per stage as an additional input is helpful.
Revised: One-Hot-Encoding of the selected algorithms per stage as an additional input is helpful.

Original: Therefore, the input dimensionality of the aggregated layers is equal to the dimension after concatenating the encoders output F · (cid:80) i
Revised: Therefore, the input dimensionality of the aggregated layers is equal to the dimension after concatenating the encoders output F · (cid:80) i

Original: ( Q i + M i ) .
Revised: ( Q i + M i ) .

Original: Further details on the architectures for each search space are specified in
Revised: Further details on the architectures for each search space are specified in

Original: Appendix M. Finally, we use the Expected Improvement as acquisition function for
Revised: Appendix M. Finally, we use the Expected Improvement as acquisition function for

Original: DeepPipe and all the baselines.
Revised: DeepPipe and all the baselines.

Original: Initial Configurations For the experiments with the PMF-Dataset, we choose these configurations with the same procedure as the authors Fusi et al.
Revised: Initial Configurations For the experiments with the PMF-Dataset, we choose these configurations with the same procedure as the authors Fusi et al.

Original: (2018), where they use dataset meta-features to find the most similar auxiliary task to initialize the search on the test task.
Revised: (2018), where they use dataset meta-features to find the most similar auxiliary task to initialize the search on the test task.

Original: Since we do not have meta-features for the Tensor-OBOE meta-dataset, we follow a greedy initialization approach Metz et al.
Revised: Since we do not have meta-features for the Tensor-OBOE meta-dataset, we follow a greedy initialization approach Metz et al.

Original: This was also applied to the ZAP-Dataset.
Revised: This was also applied to the ZAP-Dataset.

Original: Specifically, we select the best-performing pipeline configuration by ranking their performances on the meta-training tasks.
Revised: Specifically, we select the best-performing pipeline configuration by ranking their performances on the meta-training tasks.

Original: Subsequently, we iteratively choose four additional configurations that minimize (cid:80) t ∈ Tasks ˆ r t , where ˆ r t = min
Revised: Subsequently, we iteratively choose four additional configurations that minimize (cid:80) t ∈ Tasks ˆ r t , where ˆ r t = min

Original: p ∈X r t,p , given that r t,p is the rank of the pipeline p on task t .
Revised: p ∈X r t,p , given that r t,p is the rank of the pipeline p on task t .

Original: H A DDITIONAL R ELATED W ORK
Revised: H A DDITIONAL R ELATED W ORK

Original: Hyperparameter Optimization (HPO) has been well studied over the past decade (Bergstra &
Revised: Hyperparameter Optimization (HPO) has been well studied over the past decade (Bergstra &

Original: Bengio, 2012).
Revised: Bengio, 2012).

Original: Techniques relying on Bayesian Optimization (BO) employ surrogates to approximate the response function of Machine Learning models, such as Gaussian Processes (Snoek et al., 2012),
Revised: Techniques relying on Bayesian Optimization (BO) employ surrogates to approximate the response function of Machine Learning models, such as Gaussian Processes (Snoek et al., 2012),

Original: Random Forests (Bergstra et al., 2011) or Bayesian Neural Networks (Snoek et al., 2015; Springenberg et al., 2016).
Revised: Random Forests (Bergstra et al., 2011) or Bayesian Neural Networks (Snoek et al., 2015; Springenberg et al., 2016).

Original: Further improvements have been achieved by applying transfer learning, where existing evaluations on auxiliary tasks help pre-training or meta-learning the surrogate.
Revised: Further improvements have been achieved by applying transfer learning, where existing evaluations on auxiliary tasks help pre-training or meta-learning the surrogate.

Original: In this sense, some approaches use pre-trained neural networks with uncertainty outputs (Wistuba & Grabocka, 2021;
Revised: In this sense, some approaches use pre-trained neural networks with uncertainty outputs (Wistuba & Grabocka, 2021;

Original: Perrone et al., 2018; Wei et al., 2021b), or ensembles of Gaussian Processes (Feurer et al., 2018).
Revised: Perrone et al., 2018; Wei et al., 2021b), or ensembles of Gaussian Processes (Feurer et al., 2018).

Original: Deep Kernels propose combining the benefits of stochastic processes such as Gaussian Processes with neural networks (Calandra et al., 2016; Garnelo et al., 2018; Wilson et al., 2016).
Revised: Deep Kernels propose combining the benefits of stochastic processes such as Gaussian Processes with neural networks (Calandra et al., 2016; Garnelo et al., 2018; Wilson et al., 2016).

Original: Follow-up work has applied this combination for training few-shot classifiers (Patacchiola et al., 2020).
Revised: Follow-up work has applied this combination for training few-shot classifiers (Patacchiola et al., 2020).

Original: In the area of Hyperparameter Optimization, (Snoek et al., 2015) achieved success on BO by modeling the output layer of a deep neural network with a Bayesian linear regression.
Revised: In the area of Hyperparameter Optimization, (Snoek et al., 2015) achieved success on BO by modeling the output layer of a deep neural network with a Bayesian linear regression.

Original: Perrone et al.
Revised: Perrone et al.

Original: (2018) extended this work by pre-training the Bayesian network with auxiliary tasks.
Revised: (2018) extended this work by pre-training the Bayesian network with auxiliary tasks.

Original: Recent work proposed to use non-linear kernels, such as the Matérn kernel, on top of the pre-trained network to improve the performance of BO (Wistuba & Grabocka, 2021; Wei et al., 2021a).
Revised: Recent work proposed to use non-linear kernels, such as the Matérn kernel, on top of the pre-trained network to improve the performance of BO (Wistuba & Grabocka, 2021; Wei et al., 2021a).

Original: I B AYESIAN O PTIMIZATION (BO)
Revised: I B AYESIAN O PTIMIZATION (BO)

Original: In BO we fit a surrogate iteratively using the observed configurations and their response.
Revised: In BO we fit a surrogate iteratively using the observed configurations and their response.

Original: Posteriorly, its probabilistic output is used to query the next configuration to evaluate (observe) by maximizing an acquisition function.
Revised: Posteriorly, its probabilistic output is used to query the next configuration to evaluate (observe) by maximizing an acquisition function.

Original: A common choice for the acquisition is Expected Improvement, defined as:
Revised: A common choice for the acquisition is Expected Improvement, defined as:

Original: [Equation]
Revised: [Equation]

Original: 6) where y max is the largest observed response in the history H and µ is the posterior of the mean predicted performance given by the surrogate, computed using Equation 2.
Revised: 6) where y max is the largest observed response in the history H and µ is the posterior of the mean predicted performance given by the surrogate, computed using Equation 2.

Original: A common choice as surrogate is Gaussian Process, but for Pipeline Optimization we introduce DeepPipe .
Revised: A common choice as surrogate is Gaussian Process, but for Pipeline Optimization we introduce DeepPipe .

Original: J DeepPipe M ETA -T RAINING
Revised: J DeepPipe M ETA -T RAINING

Original: Given a task t with observations H t = { ( p λ ( t, 1) , y ( t, 1) ) , . . .
Revised: Given a task t with observations H t = { ( p λ ( t, 1) , y ( t, 1) ) , . . .

Original: , ( p λ ( t,Q t ) , y ( t,Q t ) ) } , t ∈ { 1 , . .
Revised: , ( p λ ( t,Q t ) , y ( t,Q t ) ) } , t ∈ { 1 , . .

Original: , T } , the objective function to minimize can be derived from the negative log marginal likelihood from the
Revised: , T } , the objective function to minimize can be derived from the negative log marginal likelihood from the

Original: Gaussian Process p ( H t ) ∼ N (0 , K T ), where K ( t ) is the covariance matrix induced by DeepPipe with parameters θ, γ .
Revised: Gaussian Process p ( H t ) ∼ N (0 , K T ), where K ( t ) is the covariance matrix induced by DeepPipe with parameters θ, γ .

Original: Specifically, the negative log marginal likelihood is (Rasmussen & Williams, 2006):
Revised: Specifically, the negative log marginal likelihood is (Rasmussen & Williams, 2006):

Original: [Equation]
Revised: [Equation]

Original: The Equation 5 is the multi-task objective function that involves all the meta-learning tasks with indices t ∈ { 1 ..., T } .
Revised: The Equation 5 is the multi-task objective function that involves all the meta-learning tasks with indices t ∈ { 1 ..., T } .

Original: We use auxiliary tasks to learn a good initialization for the surrogate.
Revised: We use auxiliary tasks to learn a good initialization for the surrogate.

Original: We sample batches from the meta-training tasks, and make gradient steps that maximize the marginal log-likelihood in Equation 5, similar to previous work (Wistuba & Grabocka, 2021).
Revised: We sample batches from the meta-training tasks, and make gradient steps that maximize the marginal log-likelihood in Equation 5, similar to previous work (Wistuba & Grabocka, 2021).

Original: The training algorithm for the surrogate is detailed in Algorithm 1.
Revised: The training algorithm for the surrogate is detailed in Algorithm 1.

Original: Additionally, we apply Early Convergence by monitoring the performance on the validation meta-dataset.
Revised: Additionally, we apply Early Convergence by monitoring the performance on the validation meta-dataset.

Original: Every epoch, we perform the following operations for every task t ∈ 1 ...
Revised: Every epoch, we perform the following operations for every task t ∈ 1 ...

Original: T : i) Draw a set of b observations (pipeline configuration and performance), ii) Compute the negative log marginal likelihood (our loss function) as in Equation 7, iii) compute gradient of the loss with respect to the DeepPipe parameters and iv) updated DeepPipe parameters.
Revised: T : i) Draw a set of b observations (pipeline configuration and performance), ii) Compute the negative log marginal likelihood (our loss function) as in Equation 7, iii) compute gradient of the loss with respect to the DeepPipe parameters and iv) updated DeepPipe parameters.

Original: b
Revised: b

Original: θ
Revised: θ

Original: 1 Function
Revised: 1 Function

Original: K
Revised: K

Original: DeepPipe M
Revised: DeepPipe M

Original: ESTING
Revised: ESTING

Original: When a new pipeline is to be optimized on a new dataset (task), we apply BO (see Algorithm 2).
Revised: When a new pipeline is to be optimized on a new dataset (task), we apply BO (see Algorithm 2).

Original: Every iteration we update the surrogate by fine-tuning the kernel parameters.
Revised: Every iteration we update the surrogate by fine-tuning the kernel parameters.

Original: However, the parameters of the MLP layers θ can be also optimized, as we did on the Experiment 1, in which case the parameters were randomly initialized.
Revised: However, the parameters of the MLP layers θ can be also optimized, as we did on the Experiment 1, in which case the parameters were randomly initialized.

Original: L A DDITIONAL R ESULTS
Revised: L A DDITIONAL R ESULTS

Original: In this section, we present further results.
Revised: In this section, we present further results.

Original: Firstly, we show an ablation of the factor that determines the number of hidden units ( F ) in Figure 8.
Revised: Firstly, we show an ablation of the factor that determines the number of hidden units ( F ) in Figure 8.

Original: It shows that F = 8 attains the best performance after exploring 100 pipelines in both datasets.
Revised: It shows that F = 8 attains the best performance after exploring 100 pipelines in both datasets.

Original: Additionally, we present the average regret for the ablation of F , and the results of Experiment 1 and 2 in Figures 9, 10 and 11 respectively.
Revised: Additionally, we present the average regret for the ablation of F , and the results of Experiment 1 and 2 in Figures 9, 10 and 11 respectively.

Original: Table 5 present the extended results of omitting estimators in the PMF Dataset.
Revised: Table 5 present the extended results of omitting estimators in the PMF Dataset.

Original: From these, we draw the same conclusion as in the same paper: having encoders help to obtain better performance when a new algorithm is added to a pipeline.
Revised: From these, we draw the same conclusion as in the same paper: having encoders help to obtain better performance when a new algorithm is added to a pipeline.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: We carry out an ablation to understand the difference between the versions of Deep Pipe with/without encoder and with/without transfer-learning using ZAP Meta-dataset.
Revised: We carry out an ablation to understand the difference between the versions of Deep Pipe with/without encoder and with/without transfer-learning using ZAP Meta-dataset.

Original: As shown in Figure 12, the version with transfer learning and one encoder performs the best, thus, highlighting the importance of encoders in transfer learning our DeepPipe surrogate.
Revised: As shown in Figure 12, the version with transfer learning and one encoder performs the best, thus, highlighting the importance of encoders in transfer learning our DeepPipe surrogate.

Original: M A RCHITECTURE D ETAILS
Revised: M A RCHITECTURE D ETAILS

Original: The input to the kernel has a dimensionality of Z =20.
Revised: The input to the kernel has a dimensionality of Z =20.

Original: We fix it, to be the same as the output dimension for PMFs.
Revised: We fix it, to be the same as the output dimension for PMFs.

Original: The number of neurons per layer, as mentioned in the main paper, depends on F .
Revised: The number of neurons per layer, as mentioned in the main paper, depends on F .

Original: Consider an architecture with with no encoder layers and ℓ a aggregation layers, and hyperparameters Λ i,j , i ∈ { 1 . . .
Revised: Consider an architecture with with no encoder layers and ℓ a aggregation layers, and hyperparameters Λ i,j , i ∈ { 1 . . .

Original: N } , j ∈ { 1 . . .
Revised: N } , j ∈ { 1 . . .

Original: M i } (following the notation in section 4.1) with Q i = max j | Λ i,j | , then the number of weights (omitting biases for the sake of simplicity) will be:
Revised: M i } (following the notation in section 4.1) with Q i = max j | Λ i,j | , then the number of weights (omitting biases for the sake of simplicity) will be:

Original: [Equation]
Revised: [Equation]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: If the architecture has ℓ e encoder layers and ℓ a aggregation layers, then number of weights is given by:
Revised: If the architecture has ℓ e encoder layers and ℓ a aggregation layers, then number of weights is given by:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Figure]
Revised: [Figure]

Original: of
Revised: of

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Encoder
Revised: Encoder

Original: [Figure]
Revised: [Figure]

Original: In other words, the aggregation layers have F · (cid:80) i Q i hidden neurons, whereas every encoder from the i -th stage has F · Q i neurons per layer.
Revised: In other words, the aggregation layers have F · (cid:80) i Q i hidden neurons, whereas every encoder from the i -th stage has F · Q i neurons per layer.

Original: The input sizes are (cid:80) i,j | Λ i,j | and | Λ i,j | for both cases respectively.
Revised: The input sizes are (cid:80) i,j | Λ i,j | and | Λ i,j | for both cases respectively.

Original: The specific values for | Λ i,j | and Q i per search space are specified in Appendix R.
Revised: The specific values for | Λ i,j | and Q i per search space are specified in Appendix R.

Original: In the search space for PMF, we group the algorithms related to Naive Bayers (MultinomialNB,
Revised: In the search space for PMF, we group the algorithms related to Naive Bayers (MultinomialNB,

Original: BernoulliNB, GaussianNB) in a single encoder.
Revised: BernoulliNB, GaussianNB) in a single encoder.

Original: In this search space, we also group LDA and QDA.
Revised: In this search space, we also group LDA and QDA.

Original: In the search space of TensorOboe, we group GaussianNB and Perceptron as they do not have hyperparameters.
Revised: In the search space of TensorOboe, we group GaussianNB and Perceptron as they do not have hyperparameters.

Original: Given these considerations, we can compute the input size and the weights per search space as function of ℓ a , ℓ e , F as follows:
Revised: Given these considerations, we can compute the input size and the weights per search space as function of ℓ a , ℓ e , F as follows:

Original: (i)
Revised: (i)

Original: [Equation]
Revised: [Equation]

Original: (ii) Number of weights for architecture without encoder layers:
Revised: (ii) Number of weights for architecture without encoder layers:

Original: [Equation]
Revised: [Equation]

Original: (iii) Number of weights
Revised: (iii) Number of weights

Original: for architecture with encoder layers:
Revised: for architecture with encoder layers:

Original: [Equation]
Revised: [Equation]

Original: According the previous formulations, Figure 13 shows how many parameters (only weights) the MLP has given a specific value of F and of encoder layers.
Revised: According the previous formulations, Figure 13 shows how many parameters (only weights) the MLP has given a specific value of F and of encoder layers.

Original: We fix the total number of layers to four.
Revised: We fix the total number of layers to four.

Original: Notice that the difference in the number of parameters between an architecture with 1 and 2 encoder layers is small in both search spaces.
Revised: Notice that the difference in the number of parameters between an architecture with 1 and 2 encoder layers is small in both search spaces.

Original: N C OMPUTATIONAL I MPLEMENTATION
Revised: N C OMPUTATIONAL I MPLEMENTATION

Original: DeepPipe ’s architecture (encoder layers + aggregated layers) can be formulated as a Multilayer
Revised: DeepPipe ’s architecture (encoder layers + aggregated layers) can be formulated as a Multilayer

Original: Perceptron (MLP) comprising three parts (Figure 14).
Revised: Perceptron (MLP) comprising three parts (Figure 14).

Original: The first part of the network that builds the
Revised: The first part of the network that builds the

Original: [Figure]
Revised: [Figure]

Original: layers with encoders is implemented as a layer with masked weights.
Revised: layers with encoders is implemented as a layer with masked weights.

Original: We connect the input values corresponding to the hyperparameters
Revised: We connect the input values corresponding to the hyperparameters

Original: λ
Revised: λ

Original: ( i,j ) of the j -th algorithm of the i -th stage to a fraction of the neurons in the following layer, what builds the encoder.
Revised: ( i,j ) of the j -th algorithm of the i -th stage to a fraction of the neurons in the following layer, what builds the encoder.

Original: The fraction of neurons, as explained in section 5.4 is F · max j | λ ( i,j ) | .
Revised: The fraction of neurons, as explained in section 5.4 is F · max j | λ ( i,j ) | .

Original: The rest of the connections are dropped.
Revised: The rest of the connections are dropped.

Original: The second part is a layer that selects the output of the encoders associated with the active algorithms (one per stage), and concatenates their outputs ( Selection & Concatenation ).
Revised: The second part is a layer that selects the output of the encoders associated with the active algorithms (one per stage), and concatenates their outputs ( Selection & Concatenation ).

Original: The layer’s connections are fixed to be either to one or zero during forward and backward pass.
Revised: The layer’s connections are fixed to be either to one or zero during forward and backward pass.

Original: Specifically, they are one if they are connecting outputs of encoders of active algorithms, and zero otherwise.
Revised: Specifically, they are one if they are connecting outputs of encoders of active algorithms, and zero otherwise.

Original: The last part, an aggregation layer , is a fully connected layer that learn interactions between the concatenated output of the encoders.
Revised: The last part, an aggregation layer , is a fully connected layer that learn interactions between the concatenated output of the encoders.

Original: By implementing the architecture as a MLP instead of a multiplexed list of nodes (e.g. with a module list in PyTorch), faster forward and backward passes are obtained.
Revised: By implementing the architecture as a MLP instead of a multiplexed list of nodes (e.g. with a module list in PyTorch), faster forward and backward passes are obtained.

Original: We only need to specify the selected algorithms in the forward-pass so that the weights in the Encoder Layer are masked and the ones in the Selection & Concatenation are accordingly set.
Revised: We only need to specify the selected algorithms in the forward-pass so that the weights in the Encoder Layer are masked and the ones in the Selection & Concatenation are accordingly set.

Original: After this implementation, notice that DeepPipe is a MLP with sparse connections.
Revised: After this implementation, notice that DeepPipe is a MLP with sparse connections.

Original: O V ISUALIZING THE L EARNT R EPRESENTATIONS
Revised: O V ISUALIZING THE L EARNT R EPRESENTATIONS

Original: We train a DeepPipe with 2-layer encoders, 2 aggregation layers, 20 output size and F = 8 .
Revised: We train a DeepPipe with 2-layer encoders, 2 aggregation layers, 20 output size and F = 8 .

Original: To visualize the pipelines embeddings, we apply TSNE (T-distributed Stochastic Neighbor Embedding).
Revised: To visualize the pipelines embeddings, we apply TSNE (T-distributed Stochastic Neighbor Embedding).

Original: As plotted in Figure 15, the pipelines with the same estimator and dimensionality reducer are creating clusters.
Revised: As plotted in Figure 15, the pipelines with the same estimator and dimensionality reducer are creating clusters.

Original: The groups in this latent space are also indicators of the performance on a specific task.
Revised: The groups in this latent space are also indicators of the performance on a specific task.

Original: In
Revised: In

Original: Figure 16 we show the same embeddings of the pipelines with a color marker indicating its accuracy on two meta-testing tasks.
Revised: Figure 16 we show the same embeddings of the pipelines with a color marker indicating its accuracy on two meta-testing tasks.

Original: Top-performing pipelines (yellow color) are relatively close to each other in both tasks, building up regions of good performing pipelines.
Revised: Top-performing pipelines (yellow color) are relatively close to each other in both tasks, building up regions of good performing pipelines.

Original: These groups of good pipelines are different in both cases, which indicates that there is not a single pipeline that works for all tasks.
Revised: These groups of good pipelines are different in both cases, which indicates that there is not a single pipeline that works for all tasks.

Original: DeepPipe maps the pipelines to an embedding space where it is easier to assess the similarity between pipelines and to search for good-performing pipelines.
Revised: DeepPipe maps the pipelines to an embedding space where it is easier to assess the similarity between pipelines and to search for good-performing pipelines.

Original: However, the type of pipeline (good performing pipelines, bad performing pipelines) depends on the task.
Revised: However, the type of pipeline (good performing pipelines, bad performing pipelines) depends on the task.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: P M ETA -D ATASET P REPROCESSING
Revised: P M ETA -D ATASET P REPROCESSING

Original: We obtained the raw data for the meta-datasets from the raw repositories of PMF 6 and TensorOBOE 7 .
Revised: We obtained the raw data for the meta-datasets from the raw repositories of PMF 6 and TensorOBOE 7 .

Original: PMF repo provides an accuracy matrix, while Tensor-OBOE specifies the error.
Revised: PMF repo provides an accuracy matrix, while Tensor-OBOE specifies the error.

Original: We take the pipelines’ configurations and concatenate the hyperparameters in both meta-datasets.
Revised: We take the pipelines’ configurations and concatenate the hyperparameters in both meta-datasets.

Original: Then we proceed with the following steps: 1) One-Hot encode the categorical hyperparameters, 2) apply a log transformation x new =
Revised: Then we proceed with the following steps: 1) One-Hot encode the categorical hyperparameters, 2) apply a log transformation x new =

Original: ln ( x ) to the hyperparameters whose value is greater than 3 standard deviations, 3) scale all the values to be in the range [0,1].
Revised: ln ( x ) to the hyperparameters whose value is greater than 3 standard deviations, 3) scale all the values to be in the range [0,1].

Original: The variables coming from categorical hyperparameters are named original-variable-name_category .
Revised: The variables coming from categorical hyperparameters are named original-variable-name_category .

Original: Q A BBREVIATIONS
Revised: Q A BBREVIATIONS

Original: (i) Abbreviations in Table 2: 1) ET
Revised: (i) Abbreviations in Table 2: 1) ET

Original: 2) GBT
Revised: 2) GBT

Original: (ii) Abbreviations in Table 3:
Revised: (ii) Abbreviations in Table 3:

Original: Neighbours, 5)
Revised: Neighbours, 5)

Original: Analysis/ Linear Discriminant Analysis
Revised: Analysis/ Linear Discriminant Analysis

Original: 8) NB
Revised: 8) NB

Original: Naive Bayes.
Revised: Naive Bayes.

Original: [Equation]
Revised: [Equation]

Original: M ETA -D ATASET S EARCH S PACES
Revised: M ETA -D ATASET S EARCH S PACES

Original: We detail the search spaces composition in Tables 6 and 7.
Revised: We detail the search spaces composition in Tables 6 and 7.

Original: We specify the stages, algorithms, hyperparameters, number of components per stage M i , the number of hyperparameters per algorithm | λ i,j | , and the maximum number of hyperparameters found in an algorithm per stage Q i .
Revised: We specify the stages, algorithms, hyperparameters, number of components per stage M i , the number of hyperparameters per algorithm | λ i,j | , and the maximum number of hyperparameters found in an algorithm per stage Q i .

Original: For the
Revised: For the

Original: ZAP meta-dataset, we defined a pipeline with two stages: (i) Architecture , which specifies the type or architecture used (i e. ResNet
Revised: ZAP meta-dataset, we defined a pipeline with two stages: (i) Architecture , which specifies the type or architecture used (i e. ResNet

Original: EfficientNet-B EfficientNet-B
Revised: EfficientNet-B EfficientNet-B

Original: EfficientNet-B2),
Revised: EfficientNet-B2),

Original: and (ii)
Revised: and (ii)

Original: Optimization-related Hyperparameters that are shared by all the architectures.
Revised: Optimization-related Hyperparameters that are shared by all the architectures.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: S T HEORETICAL I NSIGHT OF H YPOTHESIS
Revised: S T HEORETICAL I NSIGHT OF H YPOTHESIS

Original: Here
Revised: Here

Original: [Table]
Revised: [Table]

Original: we formally demonstrate that the DeepPipe with encoder layers is grouping hyperparameters from the same algorithm in the latent space, better than DeepPipe without encoders, formulated on
Revised: we formally demonstrate that the DeepPipe with encoder layers is grouping hyperparameters from the same algorithm in the latent space, better than DeepPipe without encoders, formulated on

Original: Corollary S.
Revised: Corollary S.

Original: [Table]
Revised: [Table]

Original: which is supported by Proposition S.
Revised: which is supported by Proposition S.

Original: [Table]
Revised: [Table]

Original: Lemma S.1.
Revised: Lemma S.1.

Original: Given w ∈ R M , a vector of weights with independent and identically distributed components w i ∈ { w 1 , ..., w M } such that w i ∼ p ( w ) , the expected value of the square of the norm E p ( w ) ( || w || 2 ) is given by M · ( µ 2 w + σ 2 w ) , where µ w and σ w are the mean and standard deviation of p ( w ) respectively.
Revised: Given w ∈ R M , a vector of weights with independent and identically distributed components w i ∈ { w 1 , ..., w M } such that w i ∼ p ( w ) , the expected value of the square of the norm E p ( w ) ( || w || 2 ) is given by M · ( µ 2 w + σ 2 w ) , where µ w and σ w are the mean and standard deviation of p ( w ) respectively.

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: Lemma S.2.
Revised: Lemma S.2.

Original: Consider a linear function with scalar output z = w T x where w ∈ R M × 1 is the vector of weights with components w i , i ∈ { 1 , ..., M } , x ∈ R M × 1 are the input features.
Revised: Consider a linear function with scalar output z = w T x where w ∈ R M × 1 is the vector of weights with components w i , i ∈ { 1 , ..., M } , x ∈ R M × 1 are the input features.

Original: Moreover, consider the weights are independently and identically distributed w i ∼ p ( w ) .
Revised: Moreover, consider the weights are independently and identically distributed w i ∼ p ( w ) .

Original: The expected value of the norm of the output is given by E p
Revised: The expected value of the norm of the output is given by E p

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: Since w i , w j are independent then E p ( w ) ( w i · w j ) =
Revised: Since w i , w j are independent then E p ( w ) ( w i · w j ) =

Original: E p ( w ) ( w i ) · E p ( w ) ( w j ) = µ 2 w .
Revised: E p ( w ) ( w i ) · E p ( w ) ( w j ) = µ 2 w .

Original: Moreover, with a slight abuse in notation, we denote (cid:80) Mi
Revised: Moreover, with a slight abuse in notation, we denote (cid:80) Mi

Original: =1 (cid:80) i − 1 j =1 x i · x j = x ⊗ x .
Revised: =1 (cid:80) i − 1 j =1 x i · x j = x ⊗ x .

Original: Given lemma S.1, we obtain:
Revised: Given lemma S.1, we obtain:

Original: [Equation]
Revised: [Equation]

Original: where D w ( · ) is introduced as an operation to simplify the notation.
Revised: where D w ( · ) is introduced as an operation to simplify the notation.

Original: Proposition S.3.
Revised: Proposition S.3.

Original: Consider two vectors x ′ , ˆ x ∈ R M , and two weight vectors ˆ w and w ′ , ˆ w T ˆ x ∈ R , w ′ T x ′ ∈ R , such that the weights are iid.
Revised: Consider two vectors x ′ , ˆ x ∈ R M , and two weight vectors ˆ w and w ′ , ˆ w T ˆ x ∈ R , w ′ T x ′ ∈ R , such that the weights are iid.

Original: Then E p ( w ) (cid:16) ( ˆ w T ˆ x − w ′ T x ′ ) 2 (cid:17) >
Revised: Then E p ( w ) (cid:16) ( ˆ w T ˆ x − w ′ T x ′ ) 2 (cid:17) >

Original: E p ( w ) (cid:0) ( ˆ w T ˆ x − ˆ w T x ′ ) 2 (cid:1) .
Revised: E p ( w ) (cid:0) ( ˆ w T ˆ x − ˆ w T x ′ ) 2 (cid:1) .

Original: Proof.
Revised: Proof.

Original: Using lemma S.2 and decomposition the argument within square:
Revised: Using lemma S.2 and decomposition the argument within square:

Original: [Equation]
Revised: [Equation]

Original: Since ˆ w and w ′ are independent, then E p ( w ) ( w j ′ · ˆ w i ) =
Revised: Since ˆ w and w ′ are independent, then E p ( w ) ( w j ′ · ˆ w i ) =

Original: E p ( w ) ( w j ′ ) · E p ( w ) ( ˆ w i ) =
Revised: E p ( w ) ( w j ′ ) · E p ( w ) ( ˆ w i ) =

Original: µ 2 w .
Revised: µ 2 w .

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: j
Revised: j

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: When computing E p ( w ) (cid:0) ( ˆ w
Revised: When computing E p ( w ) (cid:0) ( ˆ w

Original: [Equation]
Revised: [Equation]

Original: Corollary S.4.
Revised: Corollary S.4.

Original: A random initialized DeepPipe with encoder layers induces an assumption that two hyperparameter configurations of an algorithm should have more similar performance than hyperparameter configurations from different algorithms.
Revised: A random initialized DeepPipe with encoder layers induces an assumption that two hyperparameter configurations of an algorithm should have more similar performance than hyperparameter configurations from different algorithms.

Original: Proof.
Revised: Proof.

Original: Given two hyperparameter configurations λ ( l ) , λ ( m ) from an algorithm, and a third hyperparameter configuration λ ( n ) from a different algorithm, every random initialized encoder layer from DeepPipe maps the hyperparameters λ ( l ) , λ ( m ) to latent dimensions z ( l ) , z ( m ) that are closer to each other than to z ( n ) , i.e. the expected distance among the output of the encoder layer will be E p ( w ) ( || z l − z m || ) <
Revised: Given two hyperparameter configurations λ ( l ) , λ ( m ) from an algorithm, and a third hyperparameter configuration λ ( n ) from a different algorithm, every random initialized encoder layer from DeepPipe maps the hyperparameters λ ( l ) , λ ( m ) to latent dimensions z ( l ) , z ( m ) that are closer to each other than to z ( n ) , i.e. the expected distance among the output of the encoder layer will be E p ( w ) ( || z l − z m || ) <

Original: E p ( w ) ( || z l − z n || ) based on Proposition S.3.
Revised: E p ( w ) ( || z l − z n || ) based on Proposition S.3.

Original: Since DeepPipe uses a kernel such that κ ( x , x ′ )
Revised: Since DeepPipe uses a kernel such that κ ( x , x ′ )

Original: = κ ( x − x ′ ) , their similarity will increase, when the distance between two configurations decreases.
Revised: = κ ( x − x ′ ) , their similarity will increase, when the distance between two configurations decreases.

Original: Thus, according to the Equation 2, they will have correlated performance.
Revised: Thus, according to the Equation 2, they will have correlated performance.

Original: T M ETA -D ATASET S PLITS
Revised: T M ETA -D ATASET S PLITS

Original: We specify the IDs of the task used per split.
Revised: We specify the IDs of the task used per split.

Original: The ID of the tasks are taken from the original meta-dataset creators.
Revised: The ID of the tasks are taken from the original meta-dataset creators.

Original: (i) PMF Meta-Dataset Meta-
Revised: (i) PMF Meta-Dataset Meta-

Original: training: 4538, 824, 1544, 1082, 1126, 917, 1153, 1063, 722, 1145, 1106, 1454, 4340, 477, 938, 806, 866, 333, 995, 1125, 924, 298, 755, 336, 820, 1471, 1120, 1520, 1569, 829, 958, 997, 472, 1442, 1122, 868, 313, 928, 921, 1446, 1536, 1025, 4534, 480, 723, 835, 1081, 950, 300, 1162, 821, 469, 933, 343, 766, 936, 1568, 785, 31, 164, 395, 761, 1534, 1056, 685, 1459, 230, 867, 828, 161, 742, 1136, 385, 877, 11, 1066, 1532, 1533, 941, 468, 1542, 795, 329, 792, 782, 1131, 796, 4153, 448, 1508, 1065, 1046, 1014, 54, 780, 748, 1150, 793, 1441, 1531, 717, 819, 1151, 287, 1016, 4135, 874, 162, 1148, 1005, 956, 1528, 23, 1516, 446, 1567, 41, 729, 910, 1156, 32, 1041, 1501, 955, 1129, 827, 937, 180, 1038, 973, 36, 44, 1496, 855, 400, 754, 1557, 1413, 758, 817, 1563, 181, 1127, 43, 444, 277, 1141, 715, 725, 884, 790, 880, 853, 155, 223, 1529, 1535, 6, 1009, 744, 1107, 1158, 830, 859, 947, 1475, 813, 734, 976, 227, 1137, 762, 777, 751, 784, 886, 885, 843, 1055, 1486, 1237, 225, 39, 778, 721, 392, 312, 857, 457, 1450, 209, 779, 479, 718, 801, 770, 1049, 391, 12, 730, 759, 1013, 338, 719, 988, 974, 787, 60, 741, 865, 1050, 735, 1079, 1482, 1143, 954, 1020, 1236, 814, 1048, 892, 879, 745, 971, 913, 1152, 694, 1133, 765, 905, 804, 848, 40477, 846, 334, 791, 923, 377, 1530, 889, 1163, 1006, 749, 922, 10, 59, 1541, 310, 461, 1538, 398, 870, 1481, 970, 1036, 1044, 1068, 187, 476, 1157, 40478, 1124, 1045, 845, 62, 915, 1167, 1059, 458, 815, 28, 797, 462, 21, 952, 467, 1505, 375, 882, 1011, 1460, 964, 1104, 275, 732, 189, 478, 1464, 979, 40474, 772, 720, 1022, 823, 811, 463, 61, 1451, 1067, 1165, 184, 716, 962, 978, 916, 1217, 935, 900, 925, 919, 871, 808, 335, 1457, 799, 983, 1169, 1004, 837, 1507, 4134, 890, 1062, 1510, 818, 728, 1135, 1147, 1019, 450, 1561, 40476, 816, 1562, 740, 864, 942, 151, 713, 953, 737, 1115, 1123, 1545, 1498, 850, 873, 959, 951, 987, 991, 1132, 1154, 294, 1040, 894, 26, 878, 307, 881, 746, 679, 872, 863, 943, 18, 1537, 767, 794, 1121, 1448, 401, 14, 1026, 833, 875, 1488, 383, 914, 20, 1043, 1116, 292, 847, 1540, 1069, 1155, 1015, 1238, 1149, 1546, 841, 1565, 1556, 1527, 682, 465, 1144, 769, 1517, 756, 834, 912, 807, 904, 16, 1061, 386, 805, 3, 775, 464, 50, 1455, 1021, 1160, 1140, 1489, 1519, 946, 994, 46, 22, 1443, 339, 969, 965, 30, 977, 860, 1500, 1064, 776, 822, 182, 743, 934, 1060, 803, 980, 1539, 346, 788, 1444, 1467, 727, 1509, 903, 832.
Revised: training: 4538, 824, 1544, 1082, 1126, 917, 1153, 1063, 722, 1145, 1106, 1454, 4340, 477, 938, 806, 866, 333, 995, 1125, 924, 298, 755, 336, 820, 1471, 1120, 1520, 1569, 829, 958, 997, 472, 1442, 1122, 868, 313, 928, 921, 1446, 1536, 1025, 4534, 480, 723, 835, 1081, 950, 300, 1162, 821, 469, 933, 343, 766, 936, 1568, 785, 31, 164, 395, 761, 1534, 1056, 685, 1459, 230, 867, 828, 161, 742, 1136, 385, 877, 11, 1066, 1532, 1533, 941, 468, 1542, 795, 329, 792, 782, 1131, 796, 4153, 448, 1508, 1065, 1046, 1014, 54, 780, 748, 1150, 793, 1441, 1531, 717, 819, 1151, 287, 1016, 4135, 874, 162, 1148, 1005, 956, 1528, 23, 1516, 446, 1567, 41, 729, 910, 1156, 32, 1041, 1501, 955, 1129, 827, 937, 180, 1038, 973, 36, 44, 1496, 855, 400, 754, 1557, 1413, 758, 817, 1563, 181, 1127, 43, 444, 277, 1141, 715, 725, 884, 790, 880, 853, 155, 223, 1529, 1535, 6, 1009, 744, 1107, 1158, 830, 859, 947, 1475, 813, 734, 976, 227, 1137, 762, 777, 751, 784, 886, 885, 843, 1055, 1486, 1237, 225, 39, 778, 721, 392, 312, 857, 457, 1450, 209, 779, 479, 718, 801, 770, 1049, 391, 12, 730, 759, 1013, 338, 719, 988, 974, 787, 60, 741, 865, 1050, 735, 1079, 1482, 1143, 954, 1020, 1236, 814, 1048, 892, 879, 745, 971, 913, 1152, 694, 1133, 765, 905, 804, 848, 40477, 846, 334, 791, 923, 377, 1530, 889, 1163, 1006, 749, 922, 10, 59, 1541, 310, 461, 1538, 398, 870, 1481, 970, 1036, 1044, 1068, 187, 476, 1157, 40478, 1124, 1045, 845, 62, 915, 1167, 1059, 458, 815, 28, 797, 462, 21, 952, 467, 1505, 375, 882, 1011, 1460, 964, 1104, 275, 732, 189, 478, 1464, 979, 40474, 772, 720, 1022, 823, 811, 463, 61, 1451, 1067, 1165, 184, 716, 962, 978, 916, 1217, 935, 900, 925, 919, 871, 808, 335, 1457, 799, 983, 1169, 1004, 837, 1507, 4134, 890, 1062, 1510, 818, 728, 1135, 1147, 1019, 450, 1561, 40476, 816, 1562, 740, 864, 942, 151, 713, 953, 737, 1115, 1123, 1545, 1498, 850, 873, 959, 951, 987, 991, 1132, 1154, 294, 1040, 894, 26, 878, 307, 881, 746, 679, 872, 863, 943, 18, 1537, 767, 794, 1121, 1448, 401, 14, 1026, 833, 875, 1488, 383, 914, 20, 1043, 1116, 292, 847, 1540, 1069, 1155, 1015, 1238, 1149, 1546, 841, 1565, 1556, 1527, 682, 465, 1144, 769, 1517, 756, 834, 912, 807, 904, 16, 1061, 386, 805, 3, 775, 464, 50, 1455, 1021, 1160, 1140, 1489, 1519, 946, 994, 46, 22, 1443, 339, 969, 965, 30, 977, 860, 1500, 1064, 776, 822, 182, 743, 934, 1060, 803, 980, 1539, 346, 788, 1444, 1467, 727, 1509, 903, 832.

Original: Meta-Test: 906, 789, 1159, 1600, 48, 1453, 876, 929, 1012, 891, 1164, 726, 459, 37, 812, 909, 927, 774, 278, 279, 1054, 918, 763, 394, 948, 40, 1100, 736, 1503, 1071, 1512, 1483, 53, 869, 285, 773,
Revised: Meta-Test: 906, 789, 1159, 1600, 48, 1453, 876, 929, 1012, 891, 1164, 726, 459, 37, 812, 909, 927, 774, 278, 279, 1054, 918, 763, 394, 948, 40, 1100, 736, 1503, 1071, 1512, 1483, 53, 869, 285, 773,

Original: [Table]
Revised: [Table]

Original: cropped, 1-svhn_cropped, 2-svhn_cropped, 3-svhn_cropped, 4svhn_cropped, 5-svhn_cropped, 6-svhn_cropped, 7-svhn_cropped, 8-svhn_cropped, 9svhn_cropped, 10-svhn_cropped, 11-svhn_cropped, 12-svhn_cropped, 13-svhn_cropped, 14svhn_cropped, 0-cycle_gan_apple2orange, 1-cycle_gan_apple2orange, 2-cycle_gan_apple2orange, 3-cycle_gan_apple2orange, 4-cycle_gan_apple2orange, 5-cycle_gan_apple2orange, 6cycle_gan_apple2orange, 7-cycle_gan_apple2orange, 8-cycle_gan_apple2orange, 9cycle_gan_apple2orange, 10-cycle_gan_apple2orange, 11-cycle_gan_apple2orange, 12cycle_gan_apple2orange, 13-cycle_gan_apple2orange, 14-cycle_gan_apple2orange, 0-cats_vs_dogs, 1-cats_vs_dogs, 2-cats_vs_dogs, 3-cats_vs_dogs, 4-cats_vs_dogs, 5-cats_vs_dogs, 6-cats_vs_dogs, 7cats_vs_dogs, 8-cats_vs_dogs, 9-cats_vs_dogs, 10-cats_vs_dogs, 11-cats_vs_dogs, 12-cats_vs_dogs, 13-cats_vs_dogs, 14-cats_vs_dogs, 0-stanford_dogs, 1-stanford_dogs, 2-stanford_dogs, 3stanford_dogs, 4-stanford_dogs, 5-stanford_dogs, 6-stanford_dogs, 7-stanford_dogs, 8stanford_dogs, 9-stanford_dogs, 10-stanford_dogs, 11-stanford_dogs, 12-stanford_dogs, 13stanford_dogs, 14-stanford_dogs, 0-cifar100, 1-cifar100, 2-cifar100, 3-cifar100, 4-cifar100, 5-cifar100, 6-cifar100, 7-cifar100, 8-cifar100, 9-cifar100, 10-cifar100, 11-cifar100, 12cifar100, 13-cifar100, 14-cifar100, 0-coil100, 1-coil100, 2-coil100, 3-coil100, 4-coil100, 5-coil100, 6-coil100, 7-coil100, 8-coil100, 9-coil100, 10-coil100, 11-coil100, 12-coil100, 13coil100, 14-coil100, 0-omniglot, 1-omniglot, 2-omniglot, 3-omniglot, 4-omniglot, 5-omniglot, 6-omniglot, 7-omniglot, 8-omniglot, 9-omniglot, 10-omniglot, 11-omniglot, 12-omniglot, 13-omniglot, 14-omniglot, 0-cars196, 1-cars196, 2-cars196, 3-cars196, 4-cars196, 5-cars196, 6-cars196, 7-cars196, 8-cars196, 9-cars196, 10-cars196, 11-cars196, 12-cars196, 13-cars196,
Revised: cropped, 1-svhn_cropped, 2-svhn_cropped, 3-svhn_cropped, 4svhn_cropped, 5-svhn_cropped, 6-svhn_cropped, 7-svhn_cropped, 8-svhn_cropped, 9svhn_cropped, 10-svhn_cropped, 11-svhn_cropped, 12-svhn_cropped, 13-svhn_cropped, 14svhn_cropped, 0-cycle_gan_apple2orange, 1-cycle_gan_apple2orange, 2-cycle_gan_apple2orange, 3-cycle_gan_apple2orange, 4-cycle_gan_apple2orange, 5-cycle_gan_apple2orange, 6cycle_gan_apple2orange, 7-cycle_gan_apple2orange, 8-cycle_gan_apple2orange, 9cycle_gan_apple2orange, 10-cycle_gan_apple2orange, 11-cycle_gan_apple2orange, 12cycle_gan_apple2orange, 13-cycle_gan_apple2orange, 14-cycle_gan_apple2orange, 0-cats_vs_dogs, 1-cats_vs_dogs, 2-cats_vs_dogs, 3-cats_vs_dogs, 4-cats_vs_dogs, 5-cats_vs_dogs, 6-cats_vs_dogs, 7cats_vs_dogs, 8-cats_vs_dogs, 9-cats_vs_dogs, 10-cats_vs_dogs, 11-cats_vs_dogs, 12-cats_vs_dogs, 13-cats_vs_dogs, 14-cats_vs_dogs, 0-stanford_dogs, 1-stanford_dogs, 2-stanford_dogs, 3stanford_dogs, 4-stanford_dogs, 5-stanford_dogs, 6-stanford_dogs, 7-stanford_dogs, 8stanford_dogs, 9-stanford_dogs, 10-stanford_dogs, 11-stanford_dogs, 12-stanford_dogs, 13stanford_dogs, 14-stanford_dogs, 0-cifar100, 1-cifar100, 2-cifar100, 3-cifar100, 4-cifar100, 5-cifar100, 6-cifar100, 7-cifar100, 8-cifar100, 9-cifar100, 10-cifar100, 11-cifar100, 12cifar100, 13-cifar100, 14-cifar100, 0-coil100, 1-coil100, 2-coil100, 3-coil100, 4-coil100, 5-coil100, 6-coil100, 7-coil100, 8-coil100, 9-coil100, 10-coil100, 11-coil100, 12-coil100, 13coil100, 14-coil100, 0-omniglot, 1-omniglot, 2-omniglot, 3-omniglot, 4-omniglot, 5-omniglot, 6-omniglot, 7-omniglot, 8-omniglot, 9-omniglot, 10-omniglot, 11-omniglot, 12-omniglot, 13-omniglot, 14-omniglot, 0-cars196, 1-cars196, 2-cars196, 3-cars196, 4-cars196, 5-cars196, 6-cars196, 7-cars196, 8-cars196, 9-cars196, 10-cars196, 11-cars196, 12-cars196, 13-cars196,

Original: [Table]
Revised: [Table]

Original: horses_
Revised: horses_

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: horses_
Revised: horses_

Original: [Table]
Revised: [Table]

Original: _humans
Revised: _humans

Original: [Table] [Table]
Revised: [Table] [Table]

Original: _humans
Revised: _humans

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: cycle_
Revised: cycle_

Original: [Table]
Revised: [Table]

Original: _maps
Revised: _maps

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: _gan_
Revised: _gan_

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: cycle_
Revised: cycle_

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: maps
Revised: maps

Original: [Table]
Revised: [Table]

Original: maps
Revised: maps

Original: [Table] [Table]
Revised: [Table] [Table]

Original: rock_paper_scissors,
Revised: rock_paper_scissors,

Original: [Table]
Revised: [Table]

Original: _paper_scissors,
Revised: _paper_scissors,

Original: [Table]
Revised: [Table]

Original: rock_paper_scissors,
Revised: rock_paper_scissors,

Original: [Table]
Revised: [Table]

Original: rock_paper_scissors,
Revised: rock_paper_scissors,

Original: [Table]
Revised: [Table]

Original: rock_paper_scissors,
Revised: rock_paper_scissors,

Original: [Table]
Revised: [Table]

Original: rock_paper_scissors,
Revised: rock_paper_scissors,

Original: [Table]
Revised: [Table]

Original: rock_paper_scissors,
Revised: rock_paper_scissors,

Original: [Table]
Revised: [Table]

Original: rock_paper_scissors,
Revised: rock_paper_scissors,

Original: [Table]
Revised: [Table]

Original: _paper_
Revised: _paper_

Original: [Table] [Table]
Revised: [Table] [Table]

Original: rock_paper_scissors,
Revised: rock_paper_scissors,

Original: [Table]
Revised: [Table]

Original: rock_paper_scissors,
Revised: rock_paper_scissors,

Original: [Table]
Revised: [Table]

Original: rock_paper_
Revised: rock_paper_

Original: [Table] [Table]
Revised: [Table] [Table]

Original: _paper_scissors,
Revised: _paper_scissors,

Original: [Table]
Revised: [Table]

Original: _paper_scissors,
Revised: _paper_scissors,

Original: [Table]
Revised: [Table]

Original: rock_paper_scissors,
Revised: rock_paper_scissors,

Original: [Table]
Revised: [Table]

Original: cassava,
Revised: cassava,

Original: [Table]
Revised: [Table]

Original: cassava, 2-cassava, 3-cassava, 4-cassava, 5-cassava, 6-cassava, 7-cassava, 8-cassava, 9-cassava, 10-cassava, 11-cassava, 12-cassava, 13-cassava, 14-cassava, 0-cmaterdb_devanagari, 1-cmaterdb_devanagari, 2-cmaterdb_devanagari, 3-cmaterdb_devanagari, 4-cmaterdb_devanagari, 5-cmaterdb_devanagari, 6-cmaterdb_devanagari, 7-cmaterdb_devanagari, 8-cmaterdb_devanagari, 9-cmaterdb_devanagari, 10cmaterdb_devanagari, 11-cmaterdb_devanagari, 12-cmaterdb_devanagari, 13-cmaterdb_devanagari, 14-cmaterdb_devanagari, 0-cycle_gan_vangogh2photo, 1-cycle_gan_vangogh2photo, 2cycle_gan_vangogh2photo, 3-cycle_gan_vangogh2photo, 4-cycle_gan_vangogh2photo, 5-cycle_gan_vangogh2photo, 6-cycle_gan_vangogh2photo, 7-cycle_gan_vangogh2photo, 8-cycle_gan_vangogh2photo, 9-cycle_gan_vangogh2photo, 10-cycle_gan_vangogh2photo, 11-cycle_gan_vangogh2photo, 12-cycle_gan_vangogh2photo, 13-cycle_gan_vangogh2photo, 14-cycle_gan_vangogh2photo, 0-cycle_gan_ukiyoe2photo, 1-cycle_gan_ukiyoe2photo, 2-cycle_gan_ukiyoe2photo, 3-cycle_gan_ukiyoe2photo, 4-cycle_gan_ukiyoe2photo, 5cycle_gan_ukiyoe2photo, 6-cycle_gan_ukiyoe2photo, 7-cycle_gan_ukiyoe2photo, 8cycle_gan_ukiyoe2photo, 9-cycle_gan_ukiyoe2photo, 10-cycle_gan_ukiyoe2photo, 11cycle_gan_ukiyoe2photo, 12-cycle_gan_ukiyoe2photo, 13-cycle_gan_ukiyoe2photo, 14cycle_gan_ukiyoe2photo, 0-cifar10, 1-cifar10, 2-cifar10, 3-cifar10, 4-cifar10, 5-cifar10, 6-cifar10, 7-cifar10, 8-cifar10, 9-cifar10, 10-cifar10, 11-cifar10, 12-cifar10, 13-cifar10, 14-cifar10, 0cmaterdb_bangla, 1-cmaterdb_bangla, 2-cmaterdb_bangla, 3-cmaterdb_bangla, 4-cmaterdb_bangla, 5-cmaterdb_bangla, 6-cmaterdb_bangla, 7-cmaterdb_bangla, 8-cmaterdb_bangla, 9-cmaterdb_bangla, 10-cmaterdb_bangla, 11-cmaterdb_bangla, 12-cmaterdb_bangla, 13-cmaterdb_bangla, 14cmaterdb_bangla, 0-cycle_gan_iphone2dslr_flower, 1-cycle_gan_iphone2dslr_flower, 2cycle_gan_iphone2dslr_flower, 3-cycle_gan_iphone2dslr_flower, 4-cycle_gan_iphone2dslr_flower, 5cycle_gan_iphone2dslr_flower, 6-cycle_gan_iphone2dslr_flower, 7-cycle_gan_iphone2dslr_flower, 8cycle_gan_iphone2dslr_flower, 9-cycle_gan_iphone2dslr_flower, 10-cycle_gan_iphone2dslr_flower, 11-cycle_gan_iphone2dslr_flower, 12-cycle_gan_iphone2dslr_flower, 13cycle_gan_iphone2dslr_flower, 14-cycle_gan_iphone2dslr_flower, 0-emnist_mnist, 1-emnist_mnist, 2-emnist_mnist, 3-emnist_mnist, 4-emnist_mnist, 5-emnist_mnist, 6-emnist_mnist, 7-emnist_mnist, 8-emnist_mnist, 9-emnist_mnist, 10-emnist_mnist, 11-emnist_mnist, 12-emnist_mnist, 13emnist_mnist, 14-emnist_mnist, 0-eurosat_rgb, 1-eurosat_rgb, 2-eurosat_rgb, 3-eurosat_rgb, 4-eurosat_rgb, 5-eurosat_rgb, 6-eurosat_rgb, 7-eurosat_rgb, 8-eurosat_rgb, 9-eurosat_rgb, 10eurosat_rgb, 11-eurosat_rgb, 12-eurosat_rgb, 13-eurosat_rgb, 14-eurosat_rgb, 0-colorectal_histology, 1-colorectal_histology, 2-colorectal_histology, 3-colorectal_histology, 4-colorectal_histology, 5-colorectal_histology, 6-colorectal_histology, 7-colorectal_histology, 8-colorectal_histology, 9-colorectal_histology, 10-colorectal_histology, 11-colorectal_histology, 12-colorectal_histology, 13-colorectal_histology, 14-colorectal_histology, 0-cmaterdb_telugu, 1-cmaterdb_telugu, 2cmaterdb_telugu, 3-cmaterdb_telugu, 4-cmaterdb_telugu, 5-cmaterdb_telugu, 6-cmaterdb_telugu, 7cmaterdb_telugu, 8-cmaterdb_telugu, 9-cmaterdb_telugu, 10-cmaterdb_telugu, 11-cmaterdb_telugu, 12-cmaterdb_telugu, 13-cmaterdb_telugu, 14-cmaterdb_telugu, 0-uc_merced, 1-uc_merced, 2-uc_merced, 3-uc_merced, 4-uc_merced, 5-uc_merced, 6-uc_merced, 7-uc_merced, 8-uc_merced, 9-uc_merced, 10-uc_merced, 11-uc_merced, 12-uc_merced, 13-uc_merced, 14-uc_merced, 0-kmnist, 1-kmnist, 2-kmnist, 3-kmnist, 4-kmnist, 5-kmnist, 6-kmnist, 7-kmnist, 8-kmnist, 9-kmnist, 10-kmnist, 11-kmnist, 12-kmnist, 13-kmnist, 14-kmnist
Revised: cassava, 2-cassava, 3-cassava, 4-cassava, 5-cassava, 6-cassava, 7-cassava, 8-cassava, 9-cassava, 10-cassava, 11-cassava, 12-cassava, 13-cassava, 14-cassava, 0-cmaterdb_devanagari, 1-cmaterdb_devanagari, 2-cmaterdb_devanagari, 3-cmaterdb_devanagari, 4-cmaterdb_devanagari, 5-cmaterdb_devanagari, 6-cmaterdb_devanagari, 7-cmaterdb_devanagari, 8-cmaterdb_devanagari, 9-cmaterdb_devanagari, 10cmaterdb_devanagari, 11-cmaterdb_devanagari, 12-cmaterdb_devanagari, 13-cmaterdb_devanagari, 14-cmaterdb_devanagari, 0-cycle_gan_vangogh2photo, 1-cycle_gan_vangogh2photo, 2cycle_gan_vangogh2photo, 3-cycle_gan_vangogh2photo, 4-cycle_gan_vangogh2photo, 5-cycle_gan_vangogh2photo, 6-cycle_gan_vangogh2photo, 7-cycle_gan_vangogh2photo, 8-cycle_gan_vangogh2photo, 9-cycle_gan_vangogh2photo, 10-cycle_gan_vangogh2photo, 11-cycle_gan_vangogh2photo, 12-cycle_gan_vangogh2photo, 13-cycle_gan_vangogh2photo, 14-cycle_gan_vangogh2photo, 0-cycle_gan_ukiyoe2photo, 1-cycle_gan_ukiyoe2photo, 2-cycle_gan_ukiyoe2photo, 3-cycle_gan_ukiyoe2photo, 4-cycle_gan_ukiyoe2photo, 5cycle_gan_ukiyoe2photo, 6-cycle_gan_ukiyoe2photo, 7-cycle_gan_ukiyoe2photo, 8cycle_gan_ukiyoe2photo, 9-cycle_gan_ukiyoe2photo, 10-cycle_gan_ukiyoe2photo, 11cycle_gan_ukiyoe2photo, 12-cycle_gan_ukiyoe2photo, 13-cycle_gan_ukiyoe2photo, 14cycle_gan_ukiyoe2photo, 0-cifar10, 1-cifar10, 2-cifar10, 3-cifar10, 4-cifar10, 5-cifar10, 6-cifar10, 7-cifar10, 8-cifar10, 9-cifar10, 10-cifar10, 11-cifar10, 12-cifar10, 13-cifar10, 14-cifar10, 0cmaterdb_bangla, 1-cmaterdb_bangla, 2-cmaterdb_bangla, 3-cmaterdb_bangla, 4-cmaterdb_bangla, 5-cmaterdb_bangla, 6-cmaterdb_bangla, 7-cmaterdb_bangla, 8-cmaterdb_bangla, 9-cmaterdb_bangla, 10-cmaterdb_bangla, 11-cmaterdb_bangla, 12-cmaterdb_bangla, 13-cmaterdb_bangla, 14cmaterdb_bangla, 0-cycle_gan_iphone2dslr_flower, 1-cycle_gan_iphone2dslr_flower, 2cycle_gan_iphone2dslr_flower, 3-cycle_gan_iphone2dslr_flower, 4-cycle_gan_iphone2dslr_flower, 5cycle_gan_iphone2dslr_flower, 6-cycle_gan_iphone2dslr_flower, 7-cycle_gan_iphone2dslr_flower, 8cycle_gan_iphone2dslr_flower, 9-cycle_gan_iphone2dslr_flower, 10-cycle_gan_iphone2dslr_flower, 11-cycle_gan_iphone2dslr_flower, 12-cycle_gan_iphone2dslr_flower, 13cycle_gan_iphone2dslr_flower, 14-cycle_gan_iphone2dslr_flower, 0-emnist_mnist, 1-emnist_mnist, 2-emnist_mnist, 3-emnist_mnist, 4-emnist_mnist, 5-emnist_mnist, 6-emnist_mnist, 7-emnist_mnist, 8-emnist_mnist, 9-emnist_mnist, 10-emnist_mnist, 11-emnist_mnist, 12-emnist_mnist, 13emnist_mnist, 14-emnist_mnist, 0-eurosat_rgb, 1-eurosat_rgb, 2-eurosat_rgb, 3-eurosat_rgb, 4-eurosat_rgb, 5-eurosat_rgb, 6-eurosat_rgb, 7-eurosat_rgb, 8-eurosat_rgb, 9-eurosat_rgb, 10eurosat_rgb, 11-eurosat_rgb, 12-eurosat_rgb, 13-eurosat_rgb, 14-eurosat_rgb, 0-colorectal_histology, 1-colorectal_histology, 2-colorectal_histology, 3-colorectal_histology, 4-colorectal_histology, 5-colorectal_histology, 6-colorectal_histology, 7-colorectal_histology, 8-colorectal_histology, 9-colorectal_histology, 10-colorectal_histology, 11-colorectal_histology, 12-colorectal_histology, 13-colorectal_histology, 14-colorectal_histology, 0-cmaterdb_telugu, 1-cmaterdb_telugu, 2cmaterdb_telugu, 3-cmaterdb_telugu, 4-cmaterdb_telugu, 5-cmaterdb_telugu, 6-cmaterdb_telugu, 7cmaterdb_telugu, 8-cmaterdb_telugu, 9-cmaterdb_telugu, 10-cmaterdb_telugu, 11-cmaterdb_telugu, 12-cmaterdb_telugu, 13-cmaterdb_telugu, 14-cmaterdb_telugu, 0-uc_merced, 1-uc_merced, 2-uc_merced, 3-uc_merced, 4-uc_merced, 5-uc_merced, 6-uc_merced, 7-uc_merced, 8-uc_merced, 9-uc_merced, 10-uc_merced, 11-uc_merced, 12-uc_merced, 13-uc_merced, 14-uc_merced, 0-kmnist, 1-kmnist, 2-kmnist, 3-kmnist, 4-kmnist, 5-kmnist, 6-kmnist, 7-kmnist, 8-kmnist, 9-kmnist, 10-kmnist, 11-kmnist, 12-kmnist, 13-kmnist, 14-kmnist

Original: Meta-Test 0-cycle_gan_summer2winter_yosemite, 1-cycle_gan_summer2winter_yosemite, 2-cycle_gan_summer2winter_yosemite, 3-cycle_gan_summer2winter_yosemite, 4cycle_gan_summer2winter_yosemite, 5-cycle_gan_summer2winter_yosemite,
Revised: Meta-Test 0-cycle_gan_summer2winter_yosemite, 1-cycle_gan_summer2winter_yosemite, 2-cycle_gan_summer2winter_yosemite, 3-cycle_gan_summer2winter_yosemite, 4cycle_gan_summer2winter_yosemite, 5-cycle_gan_summer2winter_yosemite,

Original: 6- cycle_gan_summer2winter_yosemite, 7-cycle_gan_summer2winter_yosemite, 8cycle_gan_summer2winter_yosemite, 9-cycle_gan_summer2winter_yosemite, 10cycle_gan_summer2winter_yosemite, 11-cycle_gan_summer2winter_yosemite, 12cycle_gan_summer2winter_yosemite, 13-cycle_gan_summer2winter_yosemite, 14cycle_gan_summer2winter_yosemite, 0-malaria, 1-malaria, 2-malaria, 3-malaria, 4-malaria, 5-malaria, 6-malaria, 7-malaria, 8-malaria, 9-malaria, 10-malaria, 11-malaria, 12-malaria, 13-malaria, 14-malaria, 0-cycle_gan_facades, 1-cycle_gan_facades, 2-cycle_gan_facades, 3-cycle_gan_facades, 4-cycle_gan_facades, 5-cycle_gan_facades, 6-cycle_gan_facades, 7cycle_gan_facades, 8-cycle_gan_facades, 9-cycle_gan_facades, 10-cycle_gan_facades, 11cycle_gan_facades, 12-cycle_gan_facades, 13-cycle_gan_facades, 14-cycle_gan_facades, 0-emnist_balanced, 1-emnist_balanced, 2-emnist_balanced, 3-emnist_balanced, 4-emnist_balanced, 5-emnist_balanced, 6-emnist_balanced, 7-emnist_balanced, 8-emnist_balanced, 9-emnist_balanced, 10-emnist_balanced, 11-emnist_balanced, 12-emnist_balanced, 13-emnist_balanced, 14emnist_balanced, 0-imagenette, 1-imagenette, 2-imagenette, 3-imagenette, 4-imagenette, 5-imagenette, 6-imagenette, 7-imagenette, 8-imagenette, 9-imagenette, 10-imagenette, 11imagenette, 12-imagenette, 13-imagenette, 14-imagenette, 0-mnist, 1-mnist, 2-mnist, 3-mnist, 4-mnist, 5-mnist, 6-mnist, 7-mnist, 8-mnist, 9-mnist, 10-mnist, 11-mnist, 12-mnist, 13-mnist, 14-mnist, 0-cycle_gan_horse2zebra, 1-cycle_gan_horse2zebra, 2-cycle_gan_horse2zebra, 3-cycle_gan_horse2zebra, 4-cycle_gan_horse2zebra, 5-cycle_gan_horse2zebra, 6cycle_gan_horse2zebra, 7-cycle_gan_horse2zebra, 8-cycle_gan_horse2zebra, 9cycle_gan_horse2zebra, 10-cycle_gan_horse2zebra, 11-cycle_gan_horse2zebra, 12cycle_gan_horse2zebra, 13-cycle_gan_horse2zebra, 14-cycle_gan_horse2zebra
Revised: 6- cycle_gan_summer2winter_yosemite, 7-cycle_gan_summer2winter_yosemite, 8cycle_gan_summer2winter_yosemite, 9-cycle_gan_summer2winter_yosemite, 10cycle_gan_summer2winter_yosemite, 11-cycle_gan_summer2winter_yosemite, 12cycle_gan_summer2winter_yosemite, 13-cycle_gan_summer2winter_yosemite, 14cycle_gan_summer2winter_yosemite, 0-malaria, 1-malaria, 2-malaria, 3-malaria, 4-malaria, 5-malaria, 6-malaria, 7-malaria, 8-malaria, 9-malaria, 10-malaria, 11-malaria, 12-malaria, 13-malaria, 14-malaria, 0-cycle_gan_facades, 1-cycle_gan_facades, 2-cycle_gan_facades, 3-cycle_gan_facades, 4-cycle_gan_facades, 5-cycle_gan_facades, 6-cycle_gan_facades, 7cycle_gan_facades, 8-cycle_gan_facades, 9-cycle_gan_facades, 10-cycle_gan_facades, 11cycle_gan_facades, 12-cycle_gan_facades, 13-cycle_gan_facades, 14-cycle_gan_facades, 0-emnist_balanced, 1-emnist_balanced, 2-emnist_balanced, 3-emnist_balanced, 4-emnist_balanced, 5-emnist_balanced, 6-emnist_balanced, 7-emnist_balanced, 8-emnist_balanced, 9-emnist_balanced, 10-emnist_balanced, 11-emnist_balanced, 12-emnist_balanced, 13-emnist_balanced, 14emnist_balanced, 0-imagenette, 1-imagenette, 2-imagenette, 3-imagenette, 4-imagenette, 5-imagenette, 6-imagenette, 7-imagenette, 8-imagenette, 9-imagenette, 10-imagenette, 11imagenette, 12-imagenette, 13-imagenette, 14-imagenette, 0-mnist, 1-mnist, 2-mnist, 3-mnist, 4-mnist, 5-mnist, 6-mnist, 7-mnist, 8-mnist, 9-mnist, 10-mnist, 11-mnist, 12-mnist, 13-mnist, 14-mnist, 0-cycle_gan_horse2zebra, 1-cycle_gan_horse2zebra, 2-cycle_gan_horse2zebra, 3-cycle_gan_horse2zebra, 4-cycle_gan_horse2zebra, 5-cycle_gan_horse2zebra, 6cycle_gan_horse2zebra, 7-cycle_gan_horse2zebra, 8-cycle_gan_horse2zebra, 9cycle_gan_horse2zebra, 10-cycle_gan_horse2zebra, 11-cycle_gan_horse2zebra, 12cycle_gan_horse2zebra, 13-cycle_gan_horse2zebra, 14-cycle_gan_horse2zebra

Original: Meta-Validation 0-emnist_byclass, 1-emnist_byclass, 2-emnist_byclass, 3-emnist_byclass, 4-emnist_byclass, 5-emnist_byclass, 6-emnist_byclass, 7-emnist_byclass, 8-emnist_byclass, 9-emnist_byclass, 10-emnist_byclass, 11-emnist_byclass, 12-emnist_byclass, 13-emnist_byclass, 14emnist_byclass, 0-imagenet_resized_32x32, 1-imagenet_resized_32x32, 2-imagenet_resized_32x32, 3-imagenet_resized_32x32, 4-imagenet_resized_32x32, 5-imagenet_resized_32x32, 6imagenet_
Revised: Meta-Validation 0-emnist_byclass, 1-emnist_byclass, 2-emnist_byclass, 3-emnist_byclass, 4-emnist_byclass, 5-emnist_byclass, 6-emnist_byclass, 7-emnist_byclass, 8-emnist_byclass, 9-emnist_byclass, 10-emnist_byclass, 11-emnist_byclass, 12-emnist_byclass, 13-emnist_byclass, 14emnist_byclass, 0-imagenet_resized_32x32, 1-imagenet_resized_32x32, 2-imagenet_resized_32x32, 3-imagenet_resized_32x32, 4-imagenet_resized_32x32, 5-imagenet_resized_32x32, 6imagenet_

Original: resized_32x
Revised: resized_32x

Original: imagenet_resized_32x imagenet_resized_32x
Revised: imagenet_resized_32x imagenet_resized_32x

Original: imagenet_resized_32x imagenet_resized_32x
Revised: imagenet_resized_32x imagenet_resized_32x

Original: [Table]
Revised: [Table]

Original: imagenet_resized_32x
Revised: imagenet_resized_32x

Original: imagenet_resized_32x imagenet_resized_32x
Revised: imagenet_resized_32x imagenet_resized_32x

Original: [Table]
Revised: [Table]

Original: imagenet_resized_32x
Revised: imagenet_resized_32x

Original: fashion_mnist, 1-fashion_mnist, 2-fashion_mnist,
Revised: fashion_mnist, 1-fashion_mnist, 2-fashion_mnist,

Original: [Table]
Revised: [Table]

Original: fashion_mnist, 4-fashion_mnist, 5-fashion_mnist,
Revised: fashion_mnist, 4-fashion_mnist, 5-fashion_mnist,

Original: fashion_mnist, 7-fashion_mnist, 8-fashion_mnist, 9-fashion_mnist, 10-fashion_mnist,
Revised: fashion_mnist, 7-fashion_mnist, 8-fashion_mnist, 9-fashion_mnist, 10-fashion_mnist,

Original: fashion_mnist, 12-fashion_mnist, 13-fashion_mnist, 14-fashion_mnist
Revised: fashion_mnist, 12-fashion_mnist, 13-fashion_mnist, 14-fashion_mnist

Original: iv) OpenML Datasets
Revised: iv) OpenML Datasets

Original: We checked that there is not overlap between the tasks used for meta-training from the TensorOBOE and the tasks used on OpenML Datasets.
Revised: We checked that there is not overlap between the tasks used for meta-training from the TensorOBOE and the tasks used on OpenML Datasets.