Original: D ISTRIBUTION -B ASED I NVARIANT D EEP N ETWORKS FOR L EARNING M ETA -F EATURES
Revised: D ISTRIBUTION -B ASED I NVARIANT D EEP N ETWORKS FOR L EARNING M ETA -F EATURES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recent advances in deep learning from probability distributions successfully achieve classiﬁcation or regression from distribution samples, thus invariant under permutation of the samples.
Revised: Recent advances in deep learning from probability distributions successfully achieve classi ﬁ cation or regression from distribution samples, thus invariant under permutation of the samples.

Original: The ﬁrst contribution of the paper is to extend these neural architectures to achieve invariance under permutation of the features, too.
Revised: The ﬁ rst contribution of the paper is to extend these neural architectures to achieve invariance under permutation of the features, too.

Original: The proposed architecture, called D IDA , inherits the NN properties of universal approximation, and its robustness with respect to Lipschitz-bounded transformations of the input distribution is established.
Revised: The proposed architecture, called D IDA , inherits the NN properties of universal approximation, and its robustness with respect to Lipschitz-bounded transformations of the input distribution is established.

Original: The second contribution is to empirically and comparatively demonstrate the merits of the approach on two tasks deﬁned at the dataset level.
Revised: The second contribution is to empirically and comparatively demonstrate the merits of the approach on two tasks de ﬁ ned at the dataset level.

Original: On both tasks, D IDA learns meta-features supporting the characterization of a (labelled) dataset.
Revised: On both tasks, D IDA learns meta-features supporting the characterization of a (labelled) dataset.

Original: The ﬁrst task consists of predicting whether two dataset patches are extracted from the same initial dataset.
Revised: The ﬁ rst task consists of predicting whether two dataset patches are extracted from the same initial dataset.

Original: The second task consists of predicting whether the learning performance achieved by a hyper-parameter conﬁguration under a ﬁxed algorithm (ranging in k-NN, SVM, logistic regression and linear SGD) dominates that of another conﬁguration, for a dataset extracted from the OpenML benchmarking suite.
Revised: The second task consists of predicting whether the learning performance achieved by a hyper-parameter con ﬁ guration under a ﬁ xed algorithm (ranging in k-NN, SVM, logistic regression and linear SGD) dominates that of another con ﬁ guration, for a dataset extracted from the OpenML benchmarking suite.

Original: On both tasks, D IDA outperforms the state of the art: DSS and D ATASET 2V EC architectures, as well as the models based on the hand-crafted meta-features of the literature.
Revised: On both tasks, D IDA outperforms the state of the art: DSS and D ATASET 2V EC architectures, as well as the models based on the hand-crafted meta-features of the literature.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep networks architectures, initially devised for structured data such as images (Krizhevsky et al., 2012) and speech (Hinton et al., 2012), have been extended to enforce some invariance or equivariance properties (Shawe-Taylor, 1993) for more complex data representations.
Revised: Deep networks architectures, initially devised for structured data such as images (Krizhevsky et al., 2012) and speech (Hinton et al., 2012), have been extended to enforce some invariance or equivariance properties (Shawe-Taylor, 1993) for more complex data representations.

Original: Typically, the network output is required to be invariant with respect to permutations of the input points when dealing with point clouds (Qi et al., 2017), graphs (Henaff et al., 2015) or probability distributions (De Bie et al., 2019).
Revised: Typically, the network output is required to be invariant with respect to permutations of the input points when dealing with point clouds (Qi et al., 2017), graphs (Henaff et al., 2015) or probability distributions (De Bie et al., 2019).

Original: The merit of invariant or equivariant neural architectures is twofold.
Revised: The merit of invariant or equivariant neural architectures is twofold.

Original: On the one hand, they inherit the universal approximation properties of neural nets (Cybenko, 1989; Leshno et al., 1993).
Revised: On the one hand, they inherit the universal approximation properties of neural nets (Cybenko, 1989; Leshno et al., 1993).

Original: On the other hand, the fact that these architectures comply with the requirements attached to the data representation yields more robust and more general models, through constraining the neural weights and/or reducing their number.
Revised: On the other hand, the fact that these architectures comply with the requirements attached to the data representation yields more robust and more general models, through constraining the neural weights and/or reducing their number.

Original: Related works.
Revised: Related works.

Original: Invariance or equivariance properties are relevant to a wide range of applications.
Revised: Invariance or equivariance properties are relevant to a wide range of applications.

Original: In the sequence-to-sequence framework, one might want to relax the sequence order (Vinyals et al., 2016).
Revised: In the sequence-to-sequence framework, one might want to relax the sequence order (Vinyals et al., 2016).

Original: When modelling dynamic cell processes, one might want to follow the cell evolution at a macroscopic level, in terms of distributions as opposed to, a set of individual cell trajectories (Hashimoto et al., 2016).
Revised: When modelling dynamic cell processes, one might want to follow the cell evolution at a macroscopic level, in terms of distributions as opposed to, a set of individual cell trajectories (Hashimoto et al., 2016).

Original: In computer vision, one might want to handle a set of pixels, as opposed to a voxellized representation, for the sake of a better scalability in terms of data dimensionality and computational resources (De Bie et al., 2019).
Revised: In computer vision, one might want to handle a set of pixels, as opposed to a voxellized representation, for the sake of a better scalability in terms of data dimensionality and computational resources (De Bie et al., 2019).

Original: Neural architectures enforcing invariance or equivariance properties have been pioneered by (Qi et al., 2017; Zaheer et al., 2017) for learning from point clouds subject to permutation invariance or equivariance.
Revised: Neural architectures enforcing invariance or equivariance properties have been pioneered by (Qi et al., 2017; Zaheer et al., 2017) for learning from point clouds subject to permutation invariance or equivariance.

Original: These have been extended to permutation equivariance across sets (Hartford et al., 2018).
Revised: These have been extended to permutation equivariance across sets (Hartford et al., 2018).

Original: Characterizations of invariance or equivariance under group actions have been proposed in the ﬁnite (Gens & Domingos, 2014; Cohen & Welling, 2016; Ravanbakhsh et al., 2017) or inﬁnite case (Wood & Shawe-Taylor, 1996; Kondor & Trivedi, 2018).
Revised: Characterizations of invariance or equivariance under group actions have been proposed in the ﬁ nite (Gens & Domingos, 2014; Cohen & Welling, 2016; Ravanbakhsh et al., 2017) or in ﬁ nite case (Wood & Shawe-Taylor, 1996; Kondor & Trivedi, 2018).

Original: On the theoretical side, (Maron et al., 2019a; Keriven & Peyr´e, 2019) have proposed a general characterization of linear layers enforcing invariance or equivariance properties with respect to the whole permutation group on the feature set.
Revised: On the theoretical side, (Maron et al., 2019a; Keriven & Peyr´e, 2019) have proposed a general characterization of linear layers enforcing invariance or equivariance properties with respect to the whole permutation group on the feature set.

Original: The universal approximation properties of such architectures have been established in the case of sets (Zaheer et al., 2017), point clouds (Qi et al., 2017), equivariant point clouds (Segol & Lipman, 2019), discrete measures (De Bie et al., 2019), invariant (Maron et al., 2019b) and equivariant (Keriven & Peyr´e, 2019) graph neural networks.
Revised: The universal approximation properties of such architectures have been established in the case of sets (Zaheer et al., 2017), point clouds (Qi et al., 2017), equivariant point clouds (Segol & Lipman, 2019), discrete measures (De Bie et al., 2019), invariant (Maron et al., 2019b) and equivariant (Keriven & Peyr´e, 2019) graph neural networks.

Original: The approach most related to our work is that of (Maron et al., 2020), handling point clouds and presenting a neural architecture invariant w.r.t.
Revised: The approach most related to our work is that of (Maron et al., 2020), handling point clouds and presenting a neural architecture invariant w.r.t.

Original: the ordering of points and their features.
Revised: the ordering of points and their features.

Original: In this paper, the proposed distribution-based invariant deep architecture (D IDA ) extends (Maron et al., 2020) as it handles (discrete or continuous) probability distributions instead of point clouds.
Revised: In this paper, the proposed distribution-based invariant deep architecture (D IDA ) extends (Maron et al., 2020) as it handles (discrete or continuous) probability distributions instead of point clouds.

Original: This enables to leverage the topology of the Wasserstein distance to provide more general approximation results, covering (Maron et al., 2020) as a special case.
Revised: This enables to leverage the topology of the Wasserstein distance to provide more general approximation results, covering (Maron et al., 2020) as a special case.

Original: Motivations.
Revised: Motivations.

Original: A main motivation for D IDA is the ability to characterize datasets through learned meta-features .
Revised: A main motivation for D IDA is the ability to characterize datasets through learned meta-features .

Original: Meta-features, aimed to represent a dataset as a vector of characteristics, have been mentioned in the ML literature for over 40 years, in relation with several key ML challenges: (i) learning a performance model, predicting a priori the performance of an algorithm (and the hyperparameters thereof) on a dataset (Rice, 1976; Wolpert, 1996; Hutter et al., 2018); (ii) learning a generic model able of quick adaptation to new tasks, e.g. one-shot or few-shot, through the so-called meta-learning approach (Finn et al., 2018; Yoon et al., 2018); (iii) hyper-parameter transfer learning (Perrone et al., 2018), aimed to transfer the performance model learned for a task, to another task.
Revised: Meta-features, aimed to represent a dataset as a vector of characteristics, have been mentioned in the ML literature for over 40 years, in relation with several key ML challenges: (i) learning a performance model, predicting a priori the performance of an algorithm (and the hyperparameters thereof) on a dataset (Rice, 1976; Wolpert, 1996; Hutter et al., 2018); (ii) learning a generic model able of quick adaptation to new tasks, e.g. one-shot or few-shot, through the so-called meta-learning approach (Finn et al., 2018; Yoon et al., 2018); (iii) hyper-parameter transfer learning (Perrone et al., 2018), aimed to transfer the performance model learned for a task, to another task.

Original: A large number of meta-features have been manually designed along the years (Mu ˜ noz et al., 2018), ranging from sufﬁcient statistics to the so-called landmarks (Pfahringer et al., 2000), computing the performance of fast ML algorithms on the considered dataset.
Revised: A large number of meta-features have been manually designed along the years (Mu˜noz et al., 2018), ranging from suf ﬁ cient statistics to the so-called landmarks (Pfahringer et al., 2000), computing the performance of fast ML algorithms on the considered dataset.

Original: Meta-features, expected to describe the joint distribution underlying the dataset, should also be inexpensive to compute.
Revised: Meta-features, expected to describe the joint distribution underlying the dataset, should also be inexpensive to compute.

Original: The learning of meta-features has been ﬁrst tackled by (Jomaa et al., 2019) to our best knowledge, deﬁning the
Revised: The learning of meta-features has been ﬁ rst tackled by (Jomaa et al., 2019) to our best knowledge, de ﬁ ning the

Original: D ATASET 2V EC representation.
Revised: D ATASET 2V EC representation.

Original: Speciﬁcally, D ATASET 2V EC is provided two patches of datasets, (two subsets of examples, described by two (different) sets of features), and is trained to predict whether those patches are extracted from the same initial dataset.
Revised: Speci ﬁ cally, D ATASET 2V EC is provided two patches of datasets, (two subsets of examples, described by two (different) sets of features), and is trained to predict whether those patches are extracted from the same initial dataset.

Original: Contributions.
Revised: Contributions.

Original: The proposed D IDA approach extends the state of the art (Maron et al., 2020;
Revised: The proposed D IDA approach extends the state of the art (Maron et al., 2020;

Original: Jomaa et al., 2019) in two ways.
Revised: Jomaa et al., 2019) in two ways.

Original: Firstly, it is designed to handle discrete or continuous probability distributions, as opposed to point sets (Section 2).
Revised: Firstly, it is designed to handle discrete or continuous probability distributions, as opposed to point sets (Section 2).

Original: As said, this extension enables to leverage the more general topology of the Wasserstein distance as opposed to that of the Haussdorf distance (Section 3).
Revised: As said, this extension enables to leverage the more general topology of the Wasserstein distance as opposed to that of the Haussdorf distance (Section 3).

Original: This framework is used to derive theoretical guarantees of stability under bounded distribution transformations, as well as universal approximation results, extending (Maron et al., 2020) to the continuous setting.
Revised: This framework is used to derive theoretical guarantees of stability under bounded distribution transformations, as well as universal approximation results, extending (Maron et al., 2020) to the continuous setting.

Original: Secondly, the empirical validation of the approach on two tasks deﬁned at the dataset level demonstrates the merit of the approach compared to the state of the art (Maron et al., 2020; Jomaa et al., 2019; Mu˜noz et al., 2018)
Revised: Secondly, the empirical validation of the approach on two tasks de ﬁ ned at the dataset level demonstrates the merit of the approach compared to the state of the art (Maron et al., 2020; Jomaa et al., 2019; Mu˜noz et al., 2018)

Original: (Section 4).
Revised: (Section 4).

Original: Notations.
Revised: Notations.

Original: (cid:74) 1; m (cid:75) denotes the set of integers { 1 , . . .
Revised: � 1; m � denotes the set of integers { 1 , . . .

Original: m } .
Revised: m } .

Original: Distributions, including discrete distributions (datasets) are noted in bold font.
Revised: Distributions, including discrete distributions (datasets) are noted in bold font.

Original: Vectors are noted in italic, with x [ k ] denoting the k -th coordinate of vector x .
Revised: Vectors are noted in italic, with x [ k ] denoting the k -th coordinate of vector x .

Original:  D ISTRIBUTION -B ASED I NVARIANT N ETWORKS FOR M ETA -F EATURE L EARNING
Revised:  D ISTRIBUTION -B ASED I NVARIANT N ETWORKS FOR M ETA -F EATURE L EARNING

Original: This section describes the core of the proposed distribution-based invariant neural architectures, speciﬁcally the mechanism of mapping a point distribution onto another one subject to sample and feature permutation invariance, referred to as invariant layer .
Revised: This section describes the core of the proposed distribution-based invariant neural architectures, speci ﬁ cally the mechanism of mapping a point distribution onto another one subject to sample and feature permutation invariance, referred to as invariant layer .

Original: For the sake of readability, this section focuses on the case of discrete distributions, referring the reader to Appendix A for the general case of continuous distributions.
Revised: For the sake of readability, this section focuses on the case of discrete distributions, referring the reader to Appendix A for the general case of continuous distributions.

Original: 2. I NVARIANT F UNCTIONS OF D ISCRETE D ISTRIBUTIONS
Revised: 2. I NVARIANT F UNCTIONS OF D ISCRETE D ISTRIBUTIONS

Original: Let z = { ( x i , y i )
Revised: Let z = { ( x i , y i )

Original: ∈ R d , i ∈ (cid:74) 1; n (cid:75) } denote a dataset including n labelled samples, with x i ∈ R
Revised: ∈ R d , i ∈ � 1; n � } denote a dataset including n labelled samples, with x i ∈ R

Original: d X an instance and y i ∈ R d Y the associated multi-label.
Revised: d X an instance and y i ∈ R d Y the associated multi-label.

Original: With d X and d Y respectively the dimensions of the instance and label spaces, let d def.
Revised: With d X and d Y respectively the dimensions of the instance and label spaces, let d def.

Original: =
Revised: =

Original: d X + d Y .
Revised: d X + d Y .

Original: By construction, z is invariant under permutation on the sample ordering; it is viewed as an n -size discrete distribution 1 n (cid:80) ni =

Revised: By construction, z is invariant under permutation on the sample ordering; it is viewed as an n -size discrete distribution 1 n � ni =

Original: δ z i in R d with δ z
Revised: δ z i in R d with δ z

Original: i the
Revised: i the

Original: Dirac function at z i .
Revised: Dirac function at z i .

Original: In the following, Z n ( R d ) denotes the space of such n -size point distributions, with Z ( R d ) def.
Revised: In the following, Z n ( R d ) denotes the space of such n -size point distributions, with Z ( R d ) def.

Original: = ∪ n Z n ( R d ) the space of distributions of arbitrary size.
Revised: = ∪ n Z n ( R d ) the space of distributions of arbitrary size.

Original: Let G def.
Revised: Let G def.

Original: =
Revised: =

Original: S d X × S d Y denote the group of permutations independently operating on the feature and label spaces.
Revised: S d X × S d Y denote the group of permutations independently operating on the feature and label spaces.

Original: For σ = ( σ X , σ Y ) ∈ G , the image σ ( z ) of a labelled sample is deﬁned as ( σ X ( x ) , σ Y ( y )) , with x = ( x [ k ] , k ∈ (cid:74) 1; d X (cid:75) ) and σ X ( x ) def.
Revised: For σ = ( σ X , σ Y ) ∈ G , the image σ ( z ) of a labelled sample is de ﬁ ned as ( σ X ( x ) , σ Y ( y )) , with x = ( x [ k ] , k ∈ � 1; d X � ) and σ X ( x ) def.

Original: =
Revised: =

Original: ( x [ σ X ( k )] , k ∈ (cid:74) 1; d X (cid:75) ) .
Revised: ( x [ σ X ( k )] , k ∈ � 1; d X � ) .

Original: For simplicity and by abuse of notations, the operator mapping a distribution z
Revised: For simplicity and by abuse of notations, the operator mapping a distribution z

Original: = ( z i , i ∈ (cid:74) 1; n (cid:75) ) to { σ ( z i ) } def.
Revised: = ( z i , i ∈ � 1; n � ) to { σ ( z i ) } def.

Original: = σ (cid:93) z is still denoted σ .
Revised: = σ � z is still denoted σ .

Original: Let Z (Ω) denote the space of distributions supported on some domain Ω ⊂ R d , with Ω invariant under permutations in G .
Revised: Let Z ( Ω ) denote the space of distributions supported on some domain Ω ⊂ R d , with Ω invariant under permutations in G .

Original: The goal of the paper is to deﬁne and train deep architectures, implementing functions ϕ on Z (Ω ⊂ R d ) that are invariant under G , i.e. such that ∀ σ ∈ G, ϕ ( σ (cid:93) z )
Revised: The goal of the paper is to de ﬁ ne and train deep architectures, implementing functions ϕ on Z ( Ω ⊂ R d ) that are invariant under G , i.e. such that ∀ σ ∈ G, ϕ ( σ � z )

Original: =
Revised: =

Original: ϕ
Revised: ϕ

Original: ( z ) 1 .
Revised: ( z ) 1 .

Original: By construction, a multi-label dataset is invariant under permutations of the samples, of the features, and of the multi-labels.
Revised: By construction, a multi-label dataset is invariant under permutations of the samples, of the features, and of the multi-labels.

Original: Therefore, any meta-feature, that is, a feature describing a multi-label dataset, is required to satisfy the above sample and feature permutation invariance properties.
Revised: Therefore, any meta-feature, that is, a feature describing a multi-label dataset, is required to satisfy the above sample and feature permutation invariance properties.

Original: 2. D ISTRIBUTION -B ASED I NVARIANT 
Revised: 2. D ISTRIBUTION -B ASED I NVARIANT L AYERS

Original: L AYERS

Original: The building block of the proposed architecture, the invariant layer meant to satisfy the feature and label invariance requirements, is deﬁned as follows, taking inspiration from (De Bie et al., 2019).
Revised: The building block of the proposed architecture, the invariant layer meant to satisfy the feature and label invariance requirements, is de ﬁ ned as follows, taking inspiration from (De Bie et al., 2019).

Original: Deﬁnition 1.
Revised: De ﬁ nition 1.

Original: (Distribution-based invariant layers) Let an interaction functional ϕ : R d × R d → R r be G -invariant:
Revised: (Distribution-based invariant layers) Let an interaction functional ϕ : R d × R d → R r be G -invariant:

Original: [Equation]
Revised: [Equation]

Original: The distribution-based invariant layer f ϕ is deﬁned as
Revised: The distribution-based invariant layer f ϕ is de ﬁ ned as

Original: [Equation]
Revised: [Equation]

Original: By construction, f ϕ is G -invariant if ϕ is G -invariant.
Revised: By construction, f ϕ is G -invariant if ϕ is G -invariant.

Original: The construction of f ϕ is extended to the general case of possibly continuous probability distributions by essentially replacing sums by integrals (Appendix A)
Revised: The construction of f ϕ is extended to the general case of possibly continuous probability distributions by essentially replacing sums by integrals (Appendix A)

Original: Remark 1 .
Revised: Remark 1 .

Original: (Varying dimensions d X and d Y ).
Revised: (Varying dimensions d X and d Y ).

Original: Both in practice and in theory, it is important that f ϕ layers (in particular the ﬁrst layer of the neural architecture) handle datasets of arbitrary number of features d X and number of multi-labels d Y .
Revised: Both in practice and in theory, it is important that f ϕ layers (in particular the ﬁ rst layer of the neural architecture) handle datasets of arbitrary number of features d X and number of multi-labels d Y .

Original: The proposed approach, used in the experiments (Section 4), is to deﬁne ϕ as follows.
Revised: The proposed approach, used in the experiments (Section 4), is to de ﬁ ne ϕ as follows.

Original: Letting z
Revised: Letting z

Original: =
Revised: =

Original: ( x, y ) and z (cid:48) = 
Revised: ( x, y ) and z � = ( x � , y � ) be two samples in

Original: ( x (cid:48) , y (cid:48) ) be two samples in R d X × R d Y , let u be deﬁned from R 4 onto R t , consider the sum of u ( x [ k ] , x (cid:48) [ k ] , y [ (cid:96) ] , y (cid:48) [ (cid:96) ]) for k ranging in (cid:74) 1; d X (cid:75) and (cid:96) in (cid:74) 1; d Y (cid:75) , and apply mapping v from R t to R r on this sum:
Revised: R d X × R d Y , let u be de ﬁ ned from R 4 onto R t , consider the sum of u ( x [ k ] , x � [ k ] , y [ � ] , y � [ � ]) for k ranging in � 1; d X � and � in � 1; d Y � , and apply mapping v from R t to R r on this sum:

Original: [Equation]
Revised: [Equation]

Original: Likewise, by construction ϕ is G -invariant, i.e. it is invariant to both feature and label permutations.
Revised: Likewise, by construction ϕ is G -invariant, i.e. it is invariant to both feature and label permutations.

Original: As shown in Section 4, this invariance property is instrumental to a good empirical performance.
Revised: As shown in Section 4, this invariance property is instrumental to a good empirical performance.

Original: Remark 2 .
Revised: Remark 2 .

Original: (Varying sample size n ).
Revised: (Varying sample size n ).

Original: By construction, f ϕ is deﬁned on Z ( R d ) = ∪ n Z n ( R d ) (independent of n ), such that it supports inputs of arbitrary cardinality n .
Revised: By construction, f ϕ is de ﬁ ned on Z ( R d ) = ∪ n Z n ( R d ) (independent of n ), such that it supports inputs of arbitrary cardinality n .

Original: Remark 3 .
Revised: Remark 3 .

Original: (Discussion w.r.t.
Revised: (Discussion w.r.t.

Original: (Maron et al., 2020))
Revised: (Maron et al., 2020))

Original: The above deﬁnition of f ϕ is based on the aggregation of pairwise terms ϕ
Revised: The above de ﬁ nition of f ϕ is based on the aggregation of pairwise terms ϕ

Original: ( z i , z j ) .
Revised: ( z i , z j ) .

Original: The motivation for using a pairwise ϕ is twofold.
Revised: The motivation for using a pairwise ϕ is twofold.

Original: On the one hand, capturing local sample interactions allows to create more expressive architectures, which is important to improve the performance on some complex data sets, as illustrated in the experiments (Section 4).
Revised: On the one hand, capturing local sample interactions allows to create more expressive architectures, which is important to improve the performance on some complex data sets, as illustrated in the experiments (Section 4).

Original: On the other hand, interaction functionals are crucial to design universal architectures
Revised: On the other hand, interaction functionals are crucial to design universal architectures

Original: (Appendix C, theorem 2).
Revised: (Appendix C, theorem 2).

Original: The proposed theoretical framework relies on the Wasserstein distance (corresponding to the convergence in law of probability distributions), which enables to compare distributions with varying number of points or even with continuous densities.
Revised: The proposed theoretical framework relies on the Wasserstein distance (corresponding to the convergence in law of probability distributions), which enables to compare distributions with varying number of points or even with continuous densities.

Original: In contrast, Maron et al.
Revised: In contrast, Maron et al.

Original: (2020) do not use interaction functionals, and establish the universality of their DSS architecture for ﬁxed dimension d and number of points n .
Revised: (2020) do not use interaction functionals, and establish the universality of their DSS architecture for ﬁ xed dimension d and number of points n .

Original: Moreover, DSS happens to resort to max pooling operators, discontinuous w.r.t.
Revised: Moreover, DSS happens to resort to max pooling operators, discontinuous w.r.t.

Original: the Wasserstein topology (see Remark 6).
Revised: the Wasserstein topology (see Remark 6).

Original: Two particular cases are when ϕ only depends on its ﬁrst or second input:
Revised: Two particular cases are when ϕ only depends on its ﬁ rst or second input:

Original: (i) if ϕ ( z, z (cid:48) ) =
Revised: (i) if ϕ ( z, z � ) =

Original: ψ ( z (cid:48) ) , then f ϕ computes a global “moment” descriptor of the input, as f ϕ ( z ) = 1 n (cid:80) nj =1 ψ ( z j ) ∈ R r .
Revised: ψ ( z � ) , then f ϕ computes a global “moment” descriptor of the input, as f ϕ ( z ) = 1 n � nj =1 ψ ( z j ) ∈ R r .

Original: (ii) if ϕ ( z, z (cid:48) ) =
Revised: (ii) if ϕ ( z, z � ) =

Original: ξ ( z ) , then f ϕ transports the input distribution via ξ , as f ϕ ( z ) = { ξ ( z i ) , i ∈ (cid:74) 1; n (cid:75) } ∈ Z ( R r ) .
Revised: ξ ( z ) , then f ϕ transports the input distribution via ξ , as f ϕ ( z ) = { ξ ( z i ) , i ∈ � 1; n � } ∈ Z ( R r ) .

Original: This operation is referred to as a push-forward .
Revised: This operation is referred to as a push-forward .

Original: Remark 4 .
Revised: Remark 4 .

Original: (Localized computation)
Revised: (Localized computation)

Original: In practice, the quadratic complexity of f ϕ w.r.t.
Revised: In practice, the quadratic complexity of f ϕ w.r.t.

Original: the number n of samples can be reduced by only computing ϕ
Revised: the number n of samples can be reduced by only computing ϕ

Original: ( z i , z j ) for pairs z i , z j sufﬁciently close to each other.
Revised: ( z i , z j ) for pairs z i , z j suf ﬁ ciently close to each other.

Original: Layer f ϕ thus extracts and aggregates information related to the neighborhood of the samples.
Revised: Layer f ϕ thus extracts and aggregates information related to the neighborhood of the samples.

Original: Remark 5 .
Revised: Remark 5 .

Original: (Link to kernels)
Revised: (Link to kernels)

Original: The use of an interaction functional ϕ is inspired from kernel ideas, albeit with signiﬁcant differences: (i) in f  ( z i ) , the detail of the pairwise interactions ϕ
Revised: The use of an interaction functional ϕ is inspired from kernel ideas, albeit with signi ﬁ cant differences: (i) in f ϕ ( z i ) , the detail of the pairwise interactions ϕ

Original: ( z i , z j ) is lost through averaging; (ii) ϕ takes into account labels; (iii) ϕ is learnt.
Revised: ( z i , z j ) is lost through averaging; (ii) ϕ takes into account labels; (iii) ϕ is learnt.

Original: Further work will be devoted to investigating the properties of the f  ( z i ) matrix.
Revised: Further work will be devoted to investigating the properties of the f ϕ ( z i ) matrix.

Original: 2. L EARNING M ETA - FEATURES
Revised: 2. L EARNING M ETA - FEATURES

Original: The proposed distributional neural architectures deﬁned on point distributions (D IDA ) are sought as
Revised: The proposed distributional neural architectures de ﬁ ned on point distributions (D IDA ) are sought as

Original: [Equation]
Revised: [Equation]

Original: where ζ are the trainable parameters of the architecture (below).
Revised: where ζ are the trainable parameters of the architecture (below).

Original: Only the case d Y = 1 is considered in the remainder.
Revised: Only the case d Y = 1 is considered in the remainder.

Original: The k -th layer is built on the top of ϕ k , mapping pairs of vectors in R d k onto R d k +1 , with d 1 = d (the dimension of the input samples).
Revised: The k -th layer is built on the top of ϕ k , mapping pairs of vectors in R d k onto R d k +1 , with d 1 = d (the dimension of the input samples).

Original: Last layer is built on ϕ m , only depending on its second argument; it maps the distribution in layer m − 1 onto a vector, whose coordinates are referred to as meta-features.
Revised: Last layer is built on ϕ m , only depending on its second argument; it maps the distribution in layer m − 1 onto a vector, whose coordinates are referred to as meta-features.

Original: The G -invariance and dimension-agnosticity of the whole architecture only depend on the ﬁrst layer f ϕ 1 satisfying these properties.
Revised: The G -invariance and dimension-agnosticity of the whole architecture only depend on the ﬁ rst layer f ϕ 1 satisfying these properties.

Original: In the ﬁrst layer, ϕ 1 is sought as ϕ 1 (( x, y ) , ( x (cid:48) , y (cid:48) ))
Revised: In the ﬁ rst layer, ϕ 1 is sought as ϕ 1 (( x, y ) , ( x � , y � ))

Original: = v ( (cid:80) k u ( x [ k ] , x (cid:48) [ k ] , y, y (cid:48) ))
Revised: = v ( � k u ( x [ k ] , x � [ k ] , y, y � ))

Original: (Remark 1), with u ( x [ k ] , x (cid:48) [ k ] , y, y (cid:48) )
Revised: (Remark 1), with u ( x [ k ] , x � [ k ] , y, y � )

Original: =
Revised: =

Original: ( ρ ( A u · ( x [ k ]; x (cid:48)
Revised: ( ρ ( A u · ( x [ k ]; x � [ k ])

Original: [ k ])+ b u , 1 y (cid:54) = y (cid:48) ) in R t × { 0 , 1 } , where ρ is a non-linear activation function, A u a ( t, 2) matrix, ( x [ k ]; x (cid:48) [ k ]) the 2-dimensional vector concatenating x [ k ] and x (cid:48) [ k ] , and b u a t -dimensional vector.
Revised: + b u , 1 y � = y � ) in R t × { 0 , 1 } , where ρ is a non-linear activation function, A u a ( t, 2) matrix, ( x [ k ]; x � [ k ]) the 2-dimensional vector concatenating x [ k ] and x � [ k ] , and b u a t -dimensional vector.

Original: With e = (cid:80) k u ( x [ k ] , x (cid:48) [ k ] , y, y (cid:48) )) , function v likewise applies a non-linear activation function ρ on an afﬁne transformation of e : v ( e ) 
Revised: With e = � k u ( x [ k ] , x � [ k ] , y, y � )) , function v likewise applies a non-linear activation function ρ on an af ﬁ ne transformation of e : v ( e ) =

Original: = ρ ( A v · e + b v 
Revised: ρ ( A v · e + b v ) , with A v a ( t, r ) matrix and b v a r -dimensional vector.

Original: Note that the subsequent layers need neither be invariant w.r.t.
Revised: Note that the subsequent layers need neither be invariant w.r.t.

Original: the number of samples, nor handle a varying number of dimensions.
Revised: the number of samples, nor handle a varying number of dimensions.

Original: However, maintaining the distributional nature among several layers is shown to improve performance in practice (Section 4).
Revised: However, maintaining the distributional nature among several layers is shown to improve performance in practice (Section 4).

Original: Every ϕ k , k ≥ 2 is deﬁned as ϕ k
Revised: Every ϕ k , k ≥ 2 is de ﬁ ned as ϕ k

Original: = ρ
Revised: = ρ

Original: ( A k · + b k ) , with ρ an activation function, A k a ( d k , d k +1 ) matrix and b k a d k +1 -dimensional vector.
Revised: ( A k · + b k ) , with ρ an activation function, A k a ( d k , d k +1 ) matrix and b k a d k +1 -dimensional vector.

Original: The D IDA neural net thus is parameterized by ζ def.
Revised: The D IDA neural net thus is parameterized by ζ def.

Original: =
Revised: =

Original: ( A u , b u , A v , b v , { A k , b k } k ) , that is classically learned by stochastic gradient descent from the loss function deﬁned after the task at hand (Section 4).
Revised: ( A u , b u , A v , b v , { A k , b k } k ) , that is classically learned by stochastic gradient descent from the loss function de ﬁ ned after the task at hand (Section 4).

Original:  T HEORETICAL A NALYSIS
Revised:  T HEORETICAL A NALYSIS

Original: This section analyzes the properties of invariant-layer based neural architectures, speciﬁcally their robustness w.r.t.
Revised: This section analyzes the properties of invariant-layer based neural architectures, speci ﬁ cally their robustness w.r.t.

Original: bounded transformations of the involved distributions, and their approximation abilities w.r.t.
Revised: bounded transformations of the involved distributions, and their approximation abilities w.r.t.

Original: the convergence in law, which is the natural topology for distributions.
Revised: the convergence in law, which is the natural topology for distributions.

Original: As already said, the discrete distribution case is considered in this section for the sake of readability, referring the reader to Appendix A for the general case of continuous distributions.
Revised: As already said, the discrete distribution case is considered in this section for the sake of readability, referring the reader to Appendix A for the general case of continuous distributions.

Original: 3. O PTIMAL T RANSPORT C OMPARISON OF D ATASETS
Revised: 3. O PTIMAL T RANSPORT C OMPARISON OF D ATASETS

Original: Point clouds vs. distributions.
Revised: Point clouds vs. distributions.

Original: Our claim is that datasets should be seen as probability distributions, rather than point clouds.
Revised: Our claim is that datasets should be seen as probability distributions, rather than point clouds.

Original: Typically, including many copies of a point in a dataset amounts to increasing its importance, which usually makes a difference in a standard machine learning setting.
Revised: Typically, including many copies of a point in a dataset amounts to increasing its importance, which usually makes a difference in a standard machine learning setting.

Original: Accordingly, the topological framework used to deﬁne and learn meta-features in the following is that of the convergence in law, with the distance among two datasets being quantiﬁed using the
Revised: Accordingly, the topological framework used to de ﬁ ne and learn meta-features in the following is that of the convergence in law, with the distance among two datasets being quanti ﬁ ed using the

Original: Wasserstein distance (below).
Revised: Wasserstein distance (below).

Original: In contrast, the point clouds setting (see for instance (Qi et al., 2017)) relies on the Haussdorff distance among sets to theoretically assess the robustness of these architectures.
Revised: In contrast, the point clouds setting (see for instance (Qi et al., 2017)) relies on the Haussdorff distance among sets to theoretically assess the robustness of these architectures.

Original: While it is standard for 2D and 3D data involved in graphics and vision domains, it faces some limitations in higher dimensional domains, e.g. due to max-pooling being a non-continuous operator w.r.t.
Revised: While it is standard for 2D and 3D data involved in graphics and vision domains, it faces some limitations in higher dimensional domains, e.g. due to max-pooling being a non-continuous operator w.r.t.

Original: the convergence in law topology.
Revised: the convergence in law topology.

Original: Wasserstein distance.
Revised: Wasserstein distance.

Original: Referring the reader to (Santambrogio, 2015; Peyr´e & Cuturi, 2019) for a more comprehensive presentation, the standard 1 -Wasserstein distance between two discrete probability distributions z ,[-z (cid:
-]
Revised: Referring the reader to (Santambrogio, 2015; Peyr´e & Cuturi, 2019) for a more comprehensive presentation, the standard 1 -Wasserstein distance between two discrete probability distributions z ,[-z (cid:
-]

Original: ) ∈
Revised: z � ∈

Original: Z n ( R d ) × Z m ( R d ) is deﬁned as:
Revised: Z n ( R d ) × Z m ( R d ) is de ﬁ ned as:

Original: [Equation]
Revised: [Equation]

Original: with Lip 1 ( R d ) the space of 1 -Lipschitz functions f : R d → R .
Revised: with Lip 1 ( R d ) the space of 1 -Lipschitz functions f : R d → R .

Original: To account for the invariance requirement (making indistinguishable z = ( z 1 , . .
Revised: To account for the invariance requirement (making indistinguishable z = ( z 1 , . .

Original: . , z n ) and its permuted image ( σ ( z 1 ) , . .
Revised: . , z n ) and its permuted image ( σ ( z 1 ) , . .

Original: , σ ( z n )) def.
Revised: , σ ( z n )) def.

Original: = σ (cid:93) z under σ ∈ G ), we introduce the G -invariant 1 -Wasserstein distance: for z ∈ Z n ( R d ) , z (
Revised: = σ � z under σ ∈ G ), we introduce the G -invariant 1 -Wasserstein distance: for z ∈ Z n ( R d ) , z � ∈

Original: cid:48) ∈

Original: Z m ( R d ) :
Revised: Z m ( R d ) :

Original: [Equation]
Revised: [Equation]

Original: such that W 1 ( z , z (cid:48) ) = 

Revised: such that W 1 ( z , z � ) =

Original: if and only if z and z (cid:48) belong to the same equivalence class (Appendix A), i.e. are equal in the sense of probability distributions up to sample and feature permutations.
Revised: if and only if z and z � belong to the same equivalence class (Appendix A), i.e. are equal in the sense of probability distributions up to sample and feature permutations.

Original: Lipschitz property.
Revised: Lipschitz property.

Original: In this context, a map f from Z ( R d ) onto Z ( R r ) is continuous for the convergence in law (a.k.a.
Revised: In this context, a map f from Z ( R d ) onto Z ( R r ) is continuous for the convergence in law (a.k.a.

Original: weak convergence on distributions, denoted (cid:42) ) 
Revised: weak convergence on distributions, denoted � ) iff for any sequence

Original: iff for any sequence z ( k ) (cid:42) z , then f ( z ( k ) ) (cid:42) f
Revised: z ( k ) � z , then f ( z ( k ) ) � f

Original: ( z ) .
Revised: ( z ) .

Original: The Wasserstein distance metrizes the convergence in law, in the sense that z ( k ) (cid:42) z is equivalent to W 1 ( z ( k ) , z ) → 0 .
Revised: The Wasserstein distance metrizes the convergence in law, in the sense that z ( k ) � z is equivalent to W 1 ( z ( k ) , z ) → 0 .

Original: Furthermore, map f is said to be C -Lipschitz for the permutation invariant 1 -Wasserstein distance iff
Revised: Furthermore, map f is said to be C -Lipschitz for the permutation invariant 1 -Wasserstein distance iff

Original: [Equation]
Revised: [Equation]

Original: The C -Lipschitz property entails the continuity of f w.r.t.
Revised: The C -Lipschitz property entails the continuity of f w.r.t.

Original: its input: if two input distributions are close in the permutation invariant 1 -Wasserstein sense, the corresponding outputs are close too.
Revised: its input: if two input distributions are close in the permutation invariant 1 -Wasserstein sense, the corresponding outputs are close too.

Original: 3. R EGULARITY OF D ISTRIBUTION -B ASED I NVARIANT L AYERS
Revised: 3. R EGULARITY OF D ISTRIBUTION -B ASED I NVARIANT L AYERS

Original: Assuming the interaction functional to satisfy the Lipschitz property:
Revised: Assuming the interaction functional to satisfy the Lipschitz property:

Original: [Equation]
Revised: [Equation]

Original: ϕ ( z, · ) and ϕ ( · , z ) are C ϕ − Lipschitz.

Original: (5) the robustness of invariant layers with respect to different variations of their input is established (proofs in Appendix B).
Revised: (5) the robustness of invariant layers with respect to different variations of their input is established (proofs in Appendix B).

Original: We ﬁrst show that invariant layers also satisfy Lipschitz property, ensuring that deep architectures of the form (3) map close inputs onto close outputs.
Revised: We ﬁ rst show that invariant layers also satisfy Lipschitz property, ensuring that deep architectures of the form (3) map close inputs onto close outputs.

Original: Proposition 1.
Revised: Proposition 1.

Original: Invariant layer f ϕ of type (1) is (2 rC ϕ ) -Lipschitz in the sense of (4).
Revised: Invariant layer f ϕ of type (1) is (2 rC ϕ ) -Lipschitz in the sense of (4).

Original: A second result regards the case where two datasets z and z (cid:48) are such that z (cid:48) is the image of z through some diffeomorphism τ ( z = ( z 1 , . ..
Revised: A second result regards the case where two datasets z and z � are such that z � is the image of z through some diffeomorphism τ ( z = ( z 1 , . .

Original:  , z n ) and z (cid:48) = 
Revised: . , z n ) and z � = τ

Original: τ (cid:93) z

Original:  = ( τ ( z 1 ) , . . .
Revised: � z = ( τ ( z 1 ) , . . .

Original: , τ ( z n )) .
Revised: , τ ( z n )) .

Original: If τ is close to identity, then the following proposition shows that f ϕ ( τ (cid:93) z ) and f ϕ ( z ) are close too.
Revised: If τ is close to identity, then the following proposition shows that f ϕ ( τ � z ) and f ϕ ( z ) are close too.

Original: More generally, if continuous transformations τ and ξ respectively apply on the input and output space of f ϕ , and are close to identity, then ξ (cid:93) f ϕ ( τ (cid:93) z ) and f ϕ ( z ) are also close.
Revised: More generally, if continuous transformations τ and ξ respectively apply on the input and output space of f ϕ , and are close to identity, then ξ � f ϕ ( τ � z ) and f ϕ ( z ) are also close.

Original: Proposition 2.
Revised: Proposition 2.

Original: Let τ : R d → R d and ξ : R r
Revised: Let τ : R d → R d and ξ : R r

Original: → R r be two Lipschitz maps with respectively Lipschitz constants C τ and C ξ .
Revised: → R r be two Lipschitz maps with respectively Lipschitz constants C τ and C ξ .

Original: Then,
Revised: Then,

Original: [Equation]

Original: ∀ z , z (cid:48)

Original: ∈

Original: Z (Ω) , if τ is equivariant, W 1 ( ξ (cid:93) f

Original: ϕ (

Original: [Equation]

Original: (cid:93) z ) ,

Original: [Equation]

Original: (cid:93)

Original: [Equation]

Original: ϕ

Original: [Equation]
Revised: [Equation]

Original:  z[Equation] [Equation] C
Revised: ∀ z

Original: [Equation]
Revised: [Equation]

Original: C

Original: [Equation] C [Equation] z  , z
Revised: z � ∈ Z ( Ω ) , if τ is equivariant, W

Original: [Equation]
Revised: [Equation]

Original: 3. U NIVERSALITY OF I NVARIANT L AYERS
Revised: 3. U NIVERSALITY OF I NVARIANT L AYERS

Original: Lastly, the universality of the proposed architecture is established, showing that the composition of an invariant layer (1) and a fully-connected layer is enough to enjoy the universal approximation property, over all functions deﬁned on Z ( R d ) with dimension d less than some D (Remark 1).
Revised: Lastly, the universality of the proposed architecture is established, showing that the composition of an invariant layer (1) and a fully-connected layer is enough to enjoy the universal approximation property, over all functions de ﬁ ned on Z ( R d ) with dimension d less than some D (Remark 1).

Original: Theorem 1.
Revised: Theorem 1.

Original: Let F : Z (Ω) → R be a G -invariant map on a compact Ω , continuous for the convergence in law.
Revised: Let F : Z ( Ω ) → R be a G -invariant map on a compact Ω , continuous for the convergence in law.

Original: Then ∀ ε > 0 , there exists two continuous maps ψ, ϕ such that
Revised: Then ∀ ε > 0 , there exists two continuous maps ψ , ϕ such that

Original: [Equation]
Revised: [Equation]

Original: where ϕ is G -invariant and independent of F .
Revised: where ϕ is G -invariant and independent of F .

Original: Proof.
Revised: Proof.

Original: The sketch of the proof is as follows (complete proof in Appendix C).
Revised: The sketch of the proof is as follows (complete proof in Appendix C).

Original: Let us deﬁne ϕ 
Revised: Let us de ﬁ ne ϕ =

Original: = g ◦ h where: (i) h is the collection of d X elementary symmetric polynomials in the features and d Y elementary symmetric polynomials in the labels, which is invariant under G ; (ii) a discretization of h (Ω) on a grid is then considered, achieved thanks to g that aims at collecting integrals over each cell of the discretization; (iii) ψ applies function F on this discretized measure; this requires h to be bijective, and is achieved by ˜ h , through a projection on the quotient space S d /G and a restriction to its image compact Ω (cid:48) .
Revised: g ◦ h where: (i) h is the collection of d X elementary symmetric polynomials in the features and d Y elementary symmetric polynomials in the labels, which is invariant under G ; (ii) a discretization of h ( Ω ) on a grid is then considered, achieved thanks to g that aims at collecting integrals over each cell of the discretization; (iii) ψ applies function F on this discretized measure; this requires h to be bijective, and is achieved by ˜ h , through a projection on the quotient space S d /G and a restriction to its image compact Ω � .

Original: To sum up, f ϕ deﬁned as such computes an expectation which collects integrals over each cell of the grid to approximate measure h (cid:93) z by a discrete counterpart (cid:99) h (cid:93) z .
Revised: To sum up, f ϕ de ﬁ ned as such computes an expectation which collects integrals over each cell of the grid to approximate measure h � z by a discrete counterpart � h � z .

Original: Hence ψ applies F to ˜ h − 1 (cid:93) ( (cid:99) h (cid:93) z ) .
Revised: Hence ψ applies F to ˜ h − 1 � ( � h � z ) .

Original: Continuity is obtained as follows: (i) proximity of h (cid:93) z and (cid:99) h (cid:93) z follows from Lemma 1 in (De Bie et al., 2019)) and gets tighter as the grid discretization step tends to 0; (ii) Map ˜
Revised: Continuity is obtained as follows: (i) proximity of h � z and � h � z follows from Lemma 1 in (De Bie et al., 2019)) and gets tighter as the grid discretization step tends to 0; (ii) Map ˜

Original: h − 1 is 1 /d -H ¨ older, after Theorem 1.3.1 from (Rahman & Schmeisser, 2002)); therefore
Revised: h − 1 is 1 /d -H¨older, after Theorem 1.3.1 from (Rahman & Schmeisser, 2002)); therefore

Original: Lemma 2 entails that W 1 ( z , ˜ h − 1 (cid:93) (cid:99) h (cid:93) z ) can be upper-bounded; (iii) since Ω is compact, by BanachAlaoglu theorem, Z (Ω) also is.
Revised: Lemma 2 entails that W 1 ( z , ˜ h − 1 � � h � z ) can be upper-bounded; (iii) since Ω is compact, by BanachAlaoglu theorem, Z ( Ω ) also is.

Original: Since F is continuous, it is thus uniformly weakly continuous: choosing a discretization step small enough ensures the result.
Revised: Since F is continuous, it is thus uniformly weakly continuous: choosing a discretization step small enough ensures the result.

Original: Remark 6 .
Revised: Remark 6 .

Original: (Comparison with (Maron et al., 2020))
Revised: (Comparison with (Maron et al., 2020))

Original: The above proof holds for functionals of arbitrary input sample size n , as well as continuous distributions, generalizing results in (Maron et al., 2020).
Revised: The above proof holds for functionals of arbitrary input sample size n , as well as continuous distributions, generalizing results in (Maron et al., 2020).

Original: Note that the two types of architectures radically differ (more in Section 4).
Revised: Note that the two types of architectures radically differ (more in Section 4).

Original: Remark 7 .
Revised: Remark 7 .

Original: (Approximation by an invariant NN) After theorem 1, any invariant continuous function deﬁned on distributions with compact support can be approximated with arbitrary precision by an invariant neural network (Appendix C).
Revised: (Approximation by an invariant NN) After theorem 1, any invariant continuous function de ﬁ ned on distributions with compact support can be approximated with arbitrary precision by an invariant neural network (Appendix C).

Original: The proof involves mainly three steps: (i) an invariant layer f ϕ can be approximated by an invariant network; (ii) the universal approximation theorem (Cybenko, 1989; Leshno et al., 1993); (iii) uniform continuity is used to obtain uniform bounds.
Revised: The proof involves mainly three steps: (i) an invariant layer f ϕ can be approximated by an invariant network; (ii) the universal approximation theorem (Cybenko, 1989; Leshno et al., 1993); (iii) uniform continuity is used to obtain uniform bounds.

Original: Remark 8 .
Revised: Remark 8 .

Original: (Extension to different spaces)
Revised: (Extension to different spaces)

Original: Theorem 1 also extends to distributions supported on different spaces, via embedding them into a high-dimensional space.
Revised: Theorem 1 also extends to distributions supported on different spaces, via embedding them into a high-dimensional space.

Original: Therefore, any invariant function on distributions with compact support in R d with d ≤ D can be uniformly approximated by an invariant network (Appendix C).
Revised: Therefore, any invariant function on distributions with compact support in R d with d ≤ D can be uniformly approximated by an invariant network (Appendix C).

Original:  E XPERIMENTAL VALIDATION
Revised:  E XPERIMENTAL VALIDATION

Original: The experimental validation presented in this section considers two goals of experiments: (i) assessing the ability of D IDA to learn accurate meta-features; (ii) assessing the merit of the D IDA invariant layer design, building invariant f ϕ on the top of an interactional function ϕ
Revised: The experimental validation presented in this section considers two goals of experiments: (i) assessing the ability of D IDA to learn accurate meta-features; (ii) assessing the merit of the D IDA invariant layer design, building invariant f ϕ on the top of an interactional function ϕ

Original: (Eq. 1).
Revised: (Eq. 1).

Original: As said, this architecture is expected to grasp contrasts among samples, e.g. belonging to different classes; the proposed experimental setting aims to empirically investigate this conjecture.
Revised: As said, this architecture is expected to grasp contrasts among samples, e.g. belonging to different classes; the proposed experimental setting aims to empirically investigate this conjecture.

Original: Baselines.
Revised: Baselines.

Original: These goals of experiments are tackled by comparing D IDA to three baselines: DSS layers (Maron et al., 2020); hand-crafted meta-features (HC) (Mu˜noz et al., 2018)
Revised: These goals of experiments are tackled by comparing D IDA to three baselines: DSS layers (Maron et al., 2020); hand-crafted meta-features (HC) (Mu˜noz et al., 2018)

Original: (Table 5 in Appendix D); D ATASET 2V EC (Jomaa et al., 2019).
Revised: (Table 5 in Appendix D); D ATASET 2V EC (Jomaa et al., 2019).

Original: We implemented DSS, the code being not available.
Revised: We implemented DSS, the code being not available.

Original: In order to cope with varying dataset dimensions (as required by the UCI and OpenML benchmarks), the original DSS was augmented with an aggregator summing over the features.
Revised: In order to cope with varying dataset dimensions (as required by the UCI and OpenML benchmarks), the original DSS was augmented with an aggregator summing over the features.

Original: Three
Revised: Three

Original: DSS baselines are considered: linear or non-linear invariant layers, possibly preceded by equivariant layers.
Revised: DSS baselines are considered: linear or non-linear invariant layers, possibly preceded by equivariant layers.

Original: Similarly, the original D ATASET 2V EC implementation has been augmented to address our experimental setting.
Revised: Similarly, the original D ATASET 2V EC implementation has been augmented to address our experimental setting.

Original: The baselines are detailed in Appendix D.3.
Revised: The baselines are detailed in Appendix D.3.

Original: Experimental setting.
Revised: Experimental setting.

Original: Two tasks deﬁned at the dataset level are considered: patch identiﬁcation (section 4.1) and performance modelling (section 4.2).
Revised: Two tasks de ﬁ ned at the dataset level are considered: patch identi ﬁ cation (section 4.1) and performance modelling (section 4.2).

Original: The dataset preprocessing protocols are detailed in Appendix D.1.
Revised: The dataset preprocessing protocols are detailed in Appendix D.1.

Original: On both tasks, the same D IDA architecture is considered (Fig 1), involving 2 invariant layers followed by 3 fully connected (FC) layers.
Revised: On both tasks, the same D IDA architecture is considered (Fig 1), involving 2 invariant layers followed by 3 fully connected (FC) layers.

Original: Meta-features F ζ ( z ) consist of the output of the third FC layer, with ζ denoting the trained D IDA () parameters.
Revised: Meta-features F ζ ( z ) consist of the output of the third FC layer, with ζ denoting the trained D IDA () parameters.

Original: All experiments run on 1 NVIDIA-Tesla-V100-SXM2 GPU with 32GB memory, using Adam optimizer with base learning rate 10 − 3 .
Revised: All experiments run on 1 NVIDIA-Tesla-V100-SXM2 GPU with 32GB memory, using Adam optimizer with base learning rate 10 − 3 .

Original: 4. T ASK 1: P ATCH I DENTIFICATION
Revised: 4. T ASK 1: P ATCH I DENTIFICATION

Original: The patch identiﬁcation task consists of detecting whether two blocks of data are extracted from the same original dataset (Jomaa et al., 2019).
Revised: The patch identi ﬁ cation task consists of detecting whether two blocks of data are extracted from the same original dataset (Jomaa et al., 2019).

Original: Letting u denote a n -sample, d -dimensional dataset, an n z , d z patch z is constructed from u by selecting n z examples in u (sampled uniformly with replacement) and retaining their description along d z features (sampled uniformly with replacement).
Revised: Letting u denote a n -sample, d -dimensional dataset, an n z , d z patch z is constructed from u by selecting n z examples in u (sampled uniformly with replacement) and retaining their description along d z features (sampled uniformly with replacement).

Original: The size n z and number d z of features of the patch are uniformly selected in ﬁxed intervals (Table 4, Appendix D).
Revised: The size n z and number d z of features of the patch are uniformly selected in ﬁ xed intervals (Table 4, Appendix D).

Original: To each pair of patches z , z (cid:48) with same number of instances n z = n z (cid:48) , is associated a binary meta-label (cid:96) ( z , z (cid:48) ) set to 1 iff z and z ’ are extracted from the same initial dataset u .
Revised: To each pair of patches z , z � with same number of instances n z = n z � , is associated a binary meta-label � ( z , z � ) set to 1 iff z and z ’ are extracted from the same initial dataset u .

Original: D IDA parameters ζ are trained to minimize the cross-entropy loss of model ˆ (cid:96) ζ ( z , z (cid:48) ) =
Revised: D IDA parameters ζ are trained to minimize the cross-entropy loss of model ˆ � ζ ( z , z � ) =

Original: exp ( −||F ζ ( z ) − F ζ ( z (cid:48) ) || 2 ) , with F ζ ( z ) and F ζ ( z (cid:48) ) the meta-features computed for z and z (cid:48) :Minimize L ( ζ ) = − (cid:88)
Revised: exp ( − ||F ζ ( z ) − F ζ ( z � ) || 2 ) , with F ζ ( z ) and F ζ ( z � ) the meta-features computed for z and z � :

Original: [Equation]
Revised: [Equation]

Original: DIDA and all baselines are trained using a Siamese approach (Figure 1, bottom left): the same
Revised: DIDA and all baselines are trained using a Siamese approach (Figure 1, bottom left): the same

Original: D IDA (or baseline) architecture is used to compute meta-features F ζ ( z ) and F ζ ( z (cid:48) ) from patches z and z (cid:48) , and trained to minimize the cross-entropy loss w.r.t.
Revised: D IDA (or baseline) architecture is used to compute meta-features F ζ ( z ) and F ζ ( z � ) from patches z and z � , and trained to minimize the cross-entropy loss w.r.t.

Original: (cid:96) ( z , z (cid:48) ) .
Revised: � ( z , z � ) .

Original: The classiﬁcation results on toy datasets and UCI datasets (Table 1, detailed in Appendix D) show the pertinence of the D IDA meta-features, particularly so on the UCI datasets where the number of features widely varies from one dataset to another.
Revised: The classi ﬁ cation results on toy datasets and UCI datasets (Table 1, detailed in Appendix D) show the pertinence of the D IDA meta-features, particularly so on the UCI datasets where the number of features widely varies from one dataset to another.

Original: The relevance of the interactional invariant layer design is established on this problem as D IDA outperforms both D ATASET 2V EC , DSS as well as the function learned on the top of the hand-crafted meta-features.
Revised: The relevance of the interactional invariant layer design is established on this problem as D IDA outperforms both D ATASET 2V EC , DSS as well as the function learned on the top of the hand-crafted meta-features.

Original: An ablation study is conducted to assess the impact of (i) the feature permutation invariance; (ii) considering one vs two invariant layers of type (1).
Revised: An ablation study is conducted to assess the impact of (i) the feature permutation invariance; (ii) considering one vs two invariant layers of type (1).

Original: The so-called N O -FI NV -DSS baseline, detailed in Appendix D, is built upon (Zaheer et al., 2017); it only differs from the DSS baseline as it is not feature permutation invariant.
Revised: The so-called N O -FI NV -DSS baseline, detailed in Appendix D, is built upon (Zaheer et al., 2017); it only differs from the DSS baseline as it is not feature permutation invariant.

Original: With ca the same number of parameters as DSS, its performances are
Revised: With ca the same number of parameters as DSS, its performances are

Original: [Table] [Table]
Revised: [Table] [Table]

Original: signiﬁcantly lower (Table 1), showcasing the beneﬁts of enforcing the feature invariance property.
Revised: signi ﬁ cantly lower (Table 1), showcasing the bene ﬁ ts of enforcing the feature invariance property.

Original: Secondly [Table] we compare the 2-invariant layers D IDA , with the 1-invariant layer D IDA (1L-D IDA and 2L-D IDA for short): 1L-D IDA yields signiﬁcantly lower performances, which conﬁrms the advantages of maintaining the distributional nature among several layers, as already noted by (De Bie et al., 2019).
Revised: Secondly, we compare the 2-invariant layers D IDA , with the 1-invariant layer D IDA (1L-D IDA and 2L-D IDA for short): 1L-D IDA yields signi ﬁ cantly lower performances, which con ﬁ rms the advantages of maintaining the distributional nature among several layers, as already noted by (De Bie et al., 2019).

Original: Note that the 1L-D IDA still outperforms the non feature-invariant baseline [Table] while requiring much fewer parameters.
Revised: Note that the 1L-D IDA still outperforms the non feature-invariant baseline, while requiring much fewer parameters.

Original: 4. T ASK 2: P ERFORMANCE MODEL LEARNING
Revised: 4. T ASK 2: P ERFORMANCE MODEL LEARNING

Original: The performance modelling task aims to assess a priori the accuracy of the classiﬁer learned from a given machine learning algorithm with a given conﬁguration θ (vector of hyper-parameters ranging in a hyper-parameter space Θ , Table 6 in Appendix D), on a dataset z (for brevity, the performance of θ on z ) (Rice, 1976).
Revised: The performance modelling task aims to assess a priori the accuracy of the classi ﬁ er learned from a given machine learning algorithm with a given con ﬁ guration θ (vector of hyper-parameters ranging in a hyper-parameter space Θ , Table 6 in Appendix D), on a dataset z (for brevity, the performance of θ on z ) (Rice, 1976).

Original: For each ML algorithm, ranging in Logistic regression (LR), SVM, k-Nearest Neighbours (k-NN), linear classiﬁer learned with stochastic gradient descent (SGD), a set of meta-features is learned to predict whether some conﬁguration θ 1 outperforms some conﬁguration θ 2 on dataset z : to each triplet ( z , θ 1 , θ 2 ) is associated a binary value (cid:96) ( z , θ 1 , θ 2 ) , set to 1 iff θ 2 yields better performance than θ 1 on z .
Revised: For each ML algorithm, ranging in Logistic regression (LR), SVM, k-Nearest Neighbours (k-NN), linear classi ﬁ er learned with stochastic gradient descent (SGD), a set of meta-features is learned to predict whether some con ﬁ guration θ 1 outperforms some con ﬁ guration θ 2 on dataset z : to each triplet ( z , θ 1 , θ 2 ) is associated a binary value � ( z , θ 1 , θ 2 ) , set to 1 iff θ 2 yields better performance than θ 1 on z .

Original: D IDA parameters ζ are trained to build model ˆ (cid:96) ζ , minimizing the (weighted version of) cross-entropy loss (6), where ˆ (cid:96) ζ ( z , θ 1 , θ 2 ) is a 2-layer FC network with input vector [ F ζ ( z ); θ 1 ; θ 2 ] , depending on the considered ML algorithm and its conﬁguration space.
Revised: D IDA parameters ζ are trained to build model ˆ � ζ , minimizing the (weighted version of) cross-entropy loss (6), where ˆ � ζ ( z , θ 1 , θ 2 ) is a 2-layer FC network with input vector [ F ζ ( z ); θ 1 ; θ 2 ] , depending on the considered ML algorithm and its con ﬁ guration space.

Original: In each epoch, a batch made of triplets ( z , θ 1 , θ 2 ) is built, with θ 1 , θ 2 uniformly drawn in the algorithm conﬁguration space (Table 6) and
Revised: In each epoch, a batch made of triplets ( z , θ 1 , θ 2 ) is built, with θ 1 , θ 2 uniformly drawn in the algorithm con ﬁ guration space (Table 6) and

Original: z a n -sample d -dimensional patch of a dataset in the
Revised: z a n -sample d -dimensional patch of a dataset in the

Original: OpenML CC-2018 (Bischl et al., 2019) with n uniformly drawn in [700; 900] and d in [3; 10] .
Revised: OpenML CC-2018 (Bischl et al., 2019) with n uniformly drawn in [700; 900] and d in [3; 10] .

Original: Algorithm 1 summarizes the training procedure.
Revised: Algorithm 1 summarizes the training procedure.

Original: The quality of the D IDA meta-features is assessed from the ranking accuracy (Table 2), showing their relevance.
Revised: The quality of the D IDA meta-features is assessed from the ranking accuracy (Table 2), showing their relevance.

Original: The performance gap compared to the baselines is higher for the k-NN modelling
Revised: The performance gap compared to the baselines is higher for the k-NN modelling

Original: 1: F ζ ← meta-feature extractor (D IDA , DSS, D ATASET 2V EC , or Hand-crafted) 2: MLP ← NN[Linear(64)-ReLU-Linear(32)-ReLU-Linear(1)] 3: CLF ← machine learning classiﬁer (SGD, SVM, LR or k-NN) 4: error ← 3-CV classiﬁcation error function 5: for iteration= 1 , 2 , . . .
Revised: 1: F ζ ← meta-feature extractor (D IDA , DSS, D ATASET 2V EC , or Hand-crafted) 2: MLP ← NN[Linear(64)-ReLU-Linear(32)-ReLU-Linear(1)] 3: CLF ← machine learning classi ﬁ er (SGD, SVM, LR or k-NN) 4: error ← 3-CV classi ﬁ cation error function 5: for iteration= 1 , 2 , . . .

Original: do 6:
Revised: do 6:

Original: Sample ( θ 1 , θ 2 ) , two hyper-parameters of CLF (cid:46) Search space:
Revised: Sample ( θ 1 , θ 2 ) , two hyper-parameters of CLF � Search space:

Original: Table7:
Revised: Table7:

Original: Sample patch z from dataset u (cid:46) Patch dimension: Table8: pred ← softmax(MLP( F ζ ( z ) , θ 1 ), MLP( F ζ ( z ) , θ 2 )) 9:
Revised: Sample patch z from dataset u � Patch dimension: Table8: pred ← softmax(MLP( F ζ ( z ) , θ 1 ), MLP( F ζ ( z ) , θ 2 )) 9:

Original: Backpropagate logloss(pred, 0 if error( z , CLF( θ 1 ))
Revised: Backpropagate logloss(pred, 0 if error( z , CLF( θ 1 ))

Original: < error( z , CLF( θ 2 ))
Revised: < error( z , CLF( θ 2 ))

Original: else 1) 10: end for task; this is explained as the sought performance model only depends on the local geometry of the examples.
Revised: else 1) 10: end for task; this is explained as the sought performance model only depends on the local geometry of the examples.

Original: Still, good performances are observed over all considered algorithms.
Revised: Still, good performances are observed over all considered algorithms.

Original: Note that the 2LD IDA yields signiﬁcantly better (respectively, similar) performances than 1L-D IDA on the k -NN model (resp.
Revised: Note that the 2LD IDA yields signi ﬁ cantly better (respectively, similar) performances than 1L-D IDA on the k -NN model (resp.

Original: on all other models).
Revised: on all other models).

Original: Meta-feature assessment.
Revised: Meta-feature assessment.

Original: A regression setting is thereafter considered, aimed to predict the actual performance of a conﬁguration θ based on the (frozen) meta-features F ζ ( z ) .
Revised: A regression setting is thereafter considered, aimed to predict the actual performance of a con ﬁ guration θ based on the (frozen) meta-features F ζ ( z ) .

Original: The regression accuracy is illustrated for the conﬁgurations of the k -NN algorithm on Figure 2, left (results for other algorithms are presented in Appendix D).
Revised: The regression accuracy is illustrated for the con ﬁ gurations of the k -NN algorithm on Figure 2, left (results for other algorithms are presented in Appendix D).

Original: The comparison with the regression models based on DSS meta-features or hand-crafted features (Figure 2, middle and right) shows the merits of the D IDA architecture; a tentative interpretation for the D IDA better performance is based on the interactional nature of D IDA architecture, better capturing local interactions.
Revised: The comparison with the regression models based on DSS meta-features or hand-crafted features (Figure 2, middle and right) shows the merits of the D IDA architecture; a tentative interpretation for the D IDA better performance is based on the interactional nature of D IDA architecture, better capturing local interactions.

Original: of the metafeatures (i) learned by D IDA , (ii) DSS or (iii) Hand-crafted statistics.
Revised: ) learned by D IDA , (ii) DSS or (iii) Hand-crafted statistics.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: The theoretical contribution of the paper is the D IDA architecture, able to learn from discrete and continuous distributions on R d , invariant w.r.t.
Revised: The theoretical contribution of the paper is the D IDA architecture, able to learn from discrete and continuous distributions on R d , invariant w.r.t.

Original: feature ordering, agnostic w.r.t.
Revised: feature ordering, agnostic w.r.t.

Original: the size and dimension d of the considered distribution sample (with d less than some upper bound D ).
Revised: the size and dimension d of the considered distribution sample (with d less than some upper bound D ).

Original: This architecture enjoys universal approximation and robustness properties, generalizing former results obtained for point clouds (Maron et al., 2020).
Revised: This architecture enjoys universal approximation and robustness properties, generalizing former results obtained for point clouds (Maron et al., 2020).

Original: The merits of D IDA are demonstrated on two tasks deﬁned at the dataset level: patch identiﬁcation and performance model learning, comparatively to the state of the art (Maron et al., 2020; Jomaa et al., 2019; Mu˜noz et al., 2018).
Revised: The merits of D IDA are demonstrated on two tasks de ﬁ ned at the dataset level: patch identi ﬁ cation and performance model learning, comparatively to the state of the art (Maron et al., 2020; Jomaa et al., 2019; Mu˜noz et al., 2018).

Original: The ability to accurately describe a dataset in the landscape deﬁned by ML algorithms opens new perspectives to compare datasets and algorithms, e.g. for domain adaptation (Ben-David et al., 2007; 2010) and meta-learning (Finn et al., 2018; Yoon et al., 2018), in light of kernel methods.
Revised: The ability to accurately describe a dataset in the landscape de ﬁ ned by ML algorithms opens new perspectives to compare datasets and algorithms, e.g. for domain adaptation (Ben-David et al., 2007; 2010) and meta-learning (Finn et al., 2018; Yoon et al., 2018), in light of kernel methods.