Original: Function Classes for Identiﬁable Nonlinear Independent Component Analysis
Revised: Function Classes for Identiﬁable Nonlinear Independent Component Analysis

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Unsupervised learning of latent variable models (LVMs) is widely used to representdata in machine learning.
Revised: Unsupervised learning of latent variable models (LVMs) is widely used to representdata in machine learning.

Original: When such model reﬂects the ground truth factors and themechanisms mapping them to observations, there is reason to expect that such models allow generalisation in downstream tasks.
Revised: When such model reﬂects the ground truth factors and themechanisms mapping them to observations, there is reason to expect that such models allow generalisation in downstream tasks.

Original: It is however well known that suchidentiﬁability guaranties are typically not achievable without putting constraints onthe model class.
Revised: It is however well known that suchidentiﬁability guaranties are typically not achievable without putting constraints onthe model class.

Original: This is notably the case for nonlinear Independent Component
Revised: This is notably the case for nonlinear Independent Component

Original: Analysis, in which the LVM maps statistically independent variables to observations via a deterministic nonlinear function.
Revised: Analysis, in which the LVM maps statistically independent variables to observations via a deterministic nonlinear function.

Original: Several families of spurious solutionsﬁtting perfectly the data, but that do not correspond to the ground truth factors canbe constructed in generic settings.
Revised: Several families of spurious solutionsﬁtting perfectly the data, but that do not correspond to the ground truth factors canbe constructed in generic settings.

Original: However, recent work suggests that constrainingthe function class of such models may promote identiﬁability.
Revised: However, recent work suggests that constrainingthe function class of such models may promote identiﬁability.

Original: Speciﬁcally, functionclasses with constraints on their partial derivatives, gathered in the Jacobian matrix,have been proposed, such as orthogonal coordinate transformations (OCT), whichimpose orthogonality of the Jacobian columns.
Revised: Speciﬁcally, functionclasses with constraints on their partial derivatives, gathered in the Jacobian matrix,have been proposed, such as orthogonal coordinate transformations (OCT), whichimpose orthogonality of the Jacobian columns.

Original: In the present work, we prove thata subclass of these transformations, conformal maps, is identiﬁable and providenovel theoretical results suggesting that OCTs have properties that prevent familiesof spurious solutions to spoil identiﬁability in a generic setting.
Revised: In the present work, we prove thata subclass of these transformations, conformal maps, is identiﬁable and providenovel theoretical results suggesting that OCTs have properties that prevent familiesof spurious solutions to spoil identiﬁability in a generic setting.

Original:  Introduction
Revised:  Introduction

Original: Unsupervised representation learning methods can ﬁt Latent Variables Models (LVM) to complex realworld data.
Revised: Unsupervised representation learning methods can ﬁt Latent Variables Models (LVM) to complex realworld data.

Original: While those latent representations allow to create realistic novel samples or represent thedata in a compact way [23, 12], they are a priori not related to the underlying ground truth generativefactors of the data.
Revised: While those latent representations allow to create realistic novel samples or represent thedata in a compact way [29, 15], they are a priori not related to the underlying ground truth generativefactors of the data.

Original: It is highly desirable to recover the true underlying source distribution becausethis is expected help with various downstream tasks, e.g., out of distribution generalisation [28, 2].
Revised: It is highly desirable to recover the true underlying source distribution becausethis is expected help with various downstream tasks, e.g., out of distribution generalisation [40, 2].

Original: One principled framework for representation learning is Independent Component Analysis (ICA)where one tries to recover unobserved sources s ∈ R d from observations x
Revised: One principled framework for representation learning is Independent Component Analysis (ICA)where one tries to recover unobserved sources s 2 R d from observations x

Original: = f ( s ) and one assumesthat the components s i are independent.
Revised: = f ( s ) and one assumesthat the components s i are independent.

Original: An important result is that for linear functions f it ispossible to recover s from observations x up to certain symmetries, i.e., the model is identiﬁable[6].
Revised: An important result is that for linear functions f it ispossible to recover s from observations x up to certain symmetries, i.e., the model is identiﬁable[7].

Original: In contrast, for general non-linear models f is highly non-identiﬁable [18].
Revised: In contrast, for general non-linear models f is highly non-identiﬁable [24].

Original: This has importantconsequences for representation learning, in particular the learning of disentangled representations isalso unidentiﬁable without some access to the underlying sources [26].
Revised: This has importantconsequences for representation learning, in particular the learning of disentangled representations isalso unidentiﬁable without some access to the underlying sources [34].

Original: Notably, this makes theoreticalanalysis of the large body of methods (see, e.g., [15, 22, 27]) that enforces disentanglement difﬁcult.
Revised: Notably, this makes theoreticalanalysis of the large body of methods (see, e.g., [19, 28, 37]) that enforces disentanglement difﬁcult.

Original: Several additional assumptions were suggested to make the ICA problem identiﬁable.
Revised: Several additional assumptions were suggested to make the ICA problem identiﬁable.

Original: Broadly, thereare two directions.
Revised: Broadly, thereare two directions.

Original: First, some works imposed additional or different restrictions on the distribution ofthe sources.
Revised: First, some works imposed additional or different restrictions on the distribution ofthe sources.

Original: One line of research adds temporal structure by considering time series data [14, 16, 17].
Revised: One line of research adds temporal structure by considering time series data [18, 22, 23].

Original: More recently, Hyvärinen et al.
Revised: More recently, Hyvärinen et al.

Original: [19] proposed to introduce an observed auxiliary variable u , e.g.,a class label, such that the the source distribution has independent components conditional on the auxiliary variable.
Revised: [25] proposed to introduce an observed auxiliary variable u , e.g.,a class label, such that the the source distribution has independent components conditional on theauxiliary variable.

Original: They show that under suitable assumptions on the distribution of u and s arbitrarynonlinear mixing function can be identiﬁed.
Revised: They show that under suitable assumptions on the distribution of u and s arbitrarynonlinear mixing function can be identiﬁed.

Original: Several recent works extended this approach [21, 29, 31].
Revised: Several recent works extended this approach [27, 42, 47].

Original: Another possibility is to restrict the class of admissible functions by considering more ﬂexible classesthan just linear functions but not allowing arbitrary non-linear functions.
Revised: Another possibility is to restrict the class of admissible functions by considering more ﬂexible classesthan just linear functions but not allowing arbitrary non-linear functions.

Original: The general aim of thisapproach is to ﬁnd sufﬁciently small function classes such that ICA is identiﬁable in this class whilemaking them as large as possible to allow ﬂexible representation of complex data and being applicableto real world problems.
Revised: The general aim of thisapproach is to ﬁnd sufﬁciently small function classes such that ICA is identiﬁable in this class whilemaking them as large as possible to allow ﬂexible representation of complex data and being applicableto real world problems.

Original: So far results in this direction are rather limited.
Revised: So far results in this direction are rather limited.

Original: It was shown that thepost-nonlinear model is identiﬁable [30].
Revised: It was shown that thepost-nonlinear model is identiﬁable [43].

Original: Moreover, it has been shown that ICA with conformal mapsin dimension 2 is almost identiﬁable [18] and, more recently, that volume preserving transformationsare identiﬁable (in the auxiliary variable case, combining the two possible restrictions) [31].
Revised: Moreover, it has been shown that ICA with conformal mapsin dimension 2 is almost identiﬁable [24] and, more recently, that volume preserving transformationsare identiﬁable (in the auxiliary variable case, combining the two possible restrictions) [47].

Original: In this work we extend the previous works by proving new identiﬁability results for unconditional
Revised: In this work, we extend the previous works by proving new identiﬁability results for unconditional

Original: ICA.
Revised: ICA.

Original: Our main focus is conformal maps (i.e., maps that locally preserve angles) and Orthogonal
Revised: Our main focus is conformal maps (i.e., maps that locally preserve angles) and Orthogonal

Original: Coordinate Transforms (OCT) (i.e., maps satisfying that Df (cid:62) Df is a diagonal matrix where Dfdenotes the derivative of f ).
Revised: Coordinate Transforms (OCT) (i.e., maps satisfying that Df > Df is a diagonal matrix where Dfdenotes the derivative of f ).

Original: OCTs, that will also call orthogonal maps for simplicty, were recentlyintroduced in the context of representation learning in [13] where they were motivated using theindependence of mechanisms assumption from the causality literature.
Revised: OCTs, that will also call orthogonal maps for simplicty, were recentlyintroduced in the context of representation learning in [16] where they were motivated using theindependence of mechanisms assumption from the causality literature.

Original: The main focus of this workis to prove new identiﬁability and partial identiﬁability results for this class of functions.
Revised: The main focus of this workis to prove new identiﬁability and partial identiﬁability results for this class of functions.

Original: Our maincontributions are the following.
Revised: Our maincontributions are the following

Original: • We prove that ICA with conformal maps is identiﬁable in d ≥ 3 and extend the earlier
Revised: • We prove that ICA with conformal maps is identiﬁable in d  3 and extend the earlier

Original: results in dimension 2 (Theorems 2-3).
Revised: results in dimension 2 (Theorems 2-3).

Original: • We deﬁne a new notion of local identiﬁability (Deﬁnition 5) and prove that ICA with orthogonal maps is locally identiﬁable (Theorem 4).
Revised: • We deﬁne a new notion of local identiﬁability (Deﬁnition 5) and prove that ICA withorthogonal maps is locally identiﬁable (Theorem 4).

Original: • On the contrary we show that ICA with volume preserving maps is not identiﬁable not evenin the local sense (Theorem 6).
Revised: • On the contrary we show that ICA with volume preserving maps is not identiﬁable not evenin the local sense (Theorem 6).

Original: • We introduce new tools to the ICA ﬁeld: our results are based on connections to rigiditytheory, restricting the global structure of functions based on local restrictions.
Revised: • We introduce new tools to the ICA ﬁeld: our results are based on connections to rigidity theory, restricting the global structure of functions based on local restrictions.

Original: Moreover, incontrast to most earlier results that argue locally using results from linear algebra we exploitthe global structure of partial differential equations related to the identiﬁability problem.
Revised: Moreover, incontrast to most earlier results that argue locally using results from linear algebra we exploitthe global structure of partial differential equations related to the identiﬁability problem.

Original: The rest of this paper is structured as follows.
Revised: The rest of this paper is structured as follows.

Original: In Section 2 we introduce the general setting of ICAand identiﬁability.
Revised: In Section 2 we introduce the general setting of ICAand identiﬁability.

Original: Then we discuss our results for different classes of nonlinear maps.
Revised: Then we discuss our results for different classes of nonlinear maps.

Original: We considerconformal (Section 3), orthogonal (Section 4) and volume preserving (Section 5) maps.
Revised: We considerconformal (Section 3), orthogonal (Section 4) and volume preserving (Section 5) maps.

Original: An overviewof our results can be found in Table 1.
Revised: An overviewof our results can be found in Table 1.

Original: Finally, in Section 6, we discuss the relation of identiﬁabilityof ICA to the rigidity of the considered function class.
Revised: Finally, in Section 6, we discuss the relation of identiﬁabilityof ICA to the rigidity of the considered function class.

Original:  Setting
Revised:  Setting

Original: Independent component analysis deals with the problem of identifying underlying sources whenobserving a mixture of the sources.
Revised: Independent component analysis deals with the problem of identifying underlying sources whenobserving a mixture of the sources.

Original: We will consider the following general setting: there exists somerandom hidden vector of sources s ∈ R d and the observed data is generated by
Revised: We will consider the following general setting: there exists somerandom hidden vector of sources s 2 R d and the observed data is generated by

Original: [Equation]
Revised: [Equation]

Original: where f : R d → R d is a smooth invertible function.
Revised: where f : R d ! R d is a smooth invertible function.

Original: The condition on s means that its coordinates(often referred to as factors of variation) are independent.
Revised: The condition on s means that its coordinates(often referred to as factors of variation) are independent.

Original: Formally this means that the distribution ofs which we will denote by P satisﬁes P ∈ M 1 ( R ) ⊗ n where M 1 ( R ) denotes the probability measureson R .
Revised: Formally this means that the distribution ofs which we will denote by P satisﬁes P 2 M 1 ( R ) ⌦ n where M 1 ( R ) denotes the probability measureson R .

Original: The goal of ICA is to ﬁnd an unmixing function g : R d → R d such that g ( x ) has independentcomponents.
Revised: The goal of ICA is to ﬁnd an unmixing function g : R d ! R d such that g ( x ) has independentcomponents.

Original: Ideally, this should recover the true underlying factors of variation and achieve Blind
Revised: Ideally, this should recover the true underlying factors of variation and achieve Blind

Original: Source Separation (BSS), i.e., g = f − 1 up to certain symmetries.
Revised: Source Separation (BSS), i.e., g = f  1 up to certain symmetries.

Original: Identiﬁcation of the true generativefactors of variations of an observed data distribution is of interest also since these provide a causaland interventional understanding of the data.
Revised: Identiﬁcation of the true generativefactors of variations of an observed data distribution is of interest also since these provide a causaland interventional understanding of the data.

Original: An important observation was that in the generality stated above identiﬁcation of s is not possible.
Revised: An important observation was that in the generality stated above identiﬁcation of s is not possible.

Original: In[18] two general constructions of spurious solutions were given, the well known Darmois constructionand a construction based on measure preserving transformations.
Revised: In[24] two general constructions of spurious solutions were given, the well known Darmois constructionand a construction based on measure preserving transformations.

Original: The latter one is closer to our work here and we will discuss those in more detail in Section 4 and Appendix B.
Revised: The latter one is closer to ourwork here and we will discuss those in more detail in Section 4 and Appendix C.

Original: In a nutshell it isbased on the observation that for measures P with smooth density one can construct smooth Measure
Revised: In a nutshell it isbased on the observation that for measures P with smooth density one can construct smooth Measure

Original: Preserving Transformations (MPT), m : R d → R d (that mix the different coordinates), i.e., maps thatleave P invariant, such that m ( s ) D 
Revised: Preserving Transformations (MPT), m : R d ! R d (that mix the different coordinates), i.e., maps thatleave P invariant, such that m ( s ) D =

Original: = s if s ∼ P . {+
+}
Revised: s if s ⇠ P . {+
+}

Original: This implies that all functions ( f ◦ m ) − 1 recoverindependent sources since ( f ◦ m ) − 1 ( x ) D
Revised: This implies that all functions ( f  m )  1 recoverindependent sources since ( f  m )  1 ( x ) D

Original: = s making BSS impossible.
Revised: = s making BSS impossible.

Original: Thus it is a natural question whether additional assumptions on the mixing function f or distributionof s allow us to identify f .
Revised: Thus it is a natural question whether additional assumptions on the mixing function f or distributionof s allow us to identify f .

Original: Let us deﬁne a framework for identiﬁability.
Revised: Let us deﬁne a framework for identiﬁability.

Original: We assume data is generatedaccording to (1) where f ∈ F for some function class of invertible functions which we will alwaysassume to be diffeomorphisms 2 and we assume the source distribution P satisﬁes P ∈ P for some setof probability distributions P ⊂ M 1 ( R ) ⊗ d .
Revised: We assume data is generatedaccording to (1) where f 2 F for some function class of invertible functions which we will alwaysassume to be diffeomorphisms 2 and we assume the source distribution P satisﬁes P 2 P for some setof probability distributions P ⇢ M 1 ( R ) ⌦ d .

Original: Finally, let S be a group of transformations g : R d 
Revised: Finally, let S be a group of transformations g : R d !

Original: → R dthat encodes the allowed symmetries up to which the sources can be identiﬁed as follows.
Revised: R d 94 that encodes the allowed symmetries up to which the sources can be identiﬁed as follows.

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: (Identiﬁability)
Revised: (Identiﬁability)

Original: We say that independent component analysis in ( F , P ) is identiﬁableup to S if for functions f, f (cid:48) ∈ F and distributions P , P (cid:48)
Revised: We say that independent component analysis in ( F , P ) is identiﬁableup to S if for functions f, f 0 2 F and distributions P , P 0 2 P the relation

Original: ∈ P the relationf ( s ) D = f (cid:48) ( s (cid:48) ) where s ∼ P and s (cid:48) ∼ P (cid:48) (2) implies that there is h ∈ S
Revised: [Equation]

Original:  such that h = f (cid:48)− 1 ◦ f on the support of P (cid:48) .
Revised: implies that there is h 2 S such that h = f 0 1  f on the support of P 0 .

Original: Note that we require the identity h
Revised: Note that we require the identity h

Original: = f (cid:48)− 1 ◦ f only to hold on the support of P because for complexclasses F there is in general no unique extension of f beyond the support of P and without data theextension cannot be identiﬁed.
Revised: = f 0 1  f only to hold on the support of P because for complexclasses F there is in general no unique extension of f beyond the support of P and without data theextension cannot be identiﬁed.

Original: We do not always make this explicit in the following.
Revised: We do not always make this explicit in the following.

Original: Put differently,identiﬁability means that given observations of x = f ( s ) and knowledge of ( F , P ) , we can ﬁnd gsuch that g ◦ f ∈ S , in particular the reconstructed sources s (cid:48) = g ( x ) and the true sources s arerelated by a symmetry transformation in S .
Revised: Put differently,identiﬁability means that given observations of x = f ( s ) and knowledge of ( F , P ) , we can ﬁnd gsuch that g  f 2 S , in particular the reconstructed sources s 0 = g ( x ) and the true sources s arerelated by a symmetry transformation in S .

Original: In Appendix B we will discuss how to identify the set Sand how spurious solutions to the identiﬁcation problem can be constructed.
Revised: In Appendix C we will discuss how to identify the set Sand how spurious solutions to the identiﬁcation problem can be constructed.

Original: In the following it willbe convenient to use the notation f ∗ P which denotes the push-forward of the measure P along thefunction f .
Revised: In the following it willbe convenient to use the notation f ⇤ P which denotes the push-forward of the measure P along thefunction f .

Original: For a formal deﬁnition we refer to Appendix A
Revised: For a formal deﬁnition we refer to Appendix A

Original: but we note here that the distribution off ( s ) equals f ∗ P if s ∼ P and (2) can be equivalently written as f ∗ P = f
Revised: but we note here that the distribution off ( s ) equals f ⇤ P if s ⇠ P and (2) can be equivalently written as f ⇤ P = f

Original: (cid:48)∗ P (cid:48) .
Revised: 0⇤ P 0 .

Original: We illustrate Deﬁnition 1 through the well known example of linear maps
Revised: We illustrate Deﬁnition 1 through the well known example of linear maps

Original: [Equation]
Revised: [Equation]

Original: i.e., x = As for some invertible matrix A∈ R n × n .
Revised: i.e., x = As for some invertible matrix A

Original: We further deﬁne

Original: [Equation]
Revised: [Equation]

Original: one component of P
Revised: ⇥ n .

Original: [Equation] permutation [Equation] a diagonal matrix
Revised: We further deﬁne

Original: [Equation]
Revised: [Equation]

Original: It is easy to check that S is a group.
Revised: It is easy to check that S is a group.

Original: Then the following identiﬁability result for F lin is well known.
Revised: Then the following identiﬁability result for F lin is well known.

Original: Theorem 1.
Revised: Theorem 1.

Original: (Theorem 11 in [6])
Revised: (Theorem 11 in [7])

Original: The pair ( F lin , P lin ) is identiﬁable up to S lin .
Revised: The pair ( F lin , P lin ) is identiﬁable up to S lin .

Original: [Table]

Original: Moreover, this result is optimal as the ordering and scale of the s i cannot be identiﬁed and therestriction to at most one Gaussian component is required as rotations of Gaussians are Gaussian.
Revised: Moreover, this result is optimal as the ordering and scale of the s i cannot be identiﬁed and therestriction to at most one Gaussian component is required as rotations of Gaussians are Gaussian.

Original: Forcompleteness we provide a proof of this result in Appendix C as this serves as a preparation for themore involved Theorem 2 below.
Revised: Forcompleteness we provide a proof of this result in Appendix D as this serves as a preparation for themore involved Theorem 2 below.

Revised: An important observation which was also made in [21] is that with minor differences one canalso consider the case where f : R d !

Revised: M maps to a d -dimensional Riemannian manifold M .

Revised: Animportant example for this setting is the case where M ⇢ R n is a submanifold of a higher dimensional

Revised: [Equation]

Revised: [Table]

Original: In the next sections we discuss our results on identiﬁability of ICAfor different function classes.
Revised: Euclidean space. This covers the standard setting of unsupervised representation learning wherehigh dimensional observartions (often images) are created from low dimensional factors of variationmirroring the well known manifold hypothesis [44]. Note that this setting essentially covers the caseof undercomplete ICA where we consider f : R d ! R n with n > d . The only difference is that weassume that we already know the submanifold M that f maps to.

Revised: This manifold can, however, beidentiﬁed from the observations x

Revised: = f ( s ) under minor regularity assumption on f and the support ofthe data distribution.

Revised: To avoid technical difﬁculties we assume that the manifold is already known.

Revised: Note that we restrict our attention to the case where the factors of variations are parametrized by a

Revised: Euclidean space.

Revised: An extension to product manifolds and a combination with the approach in [20] isan interesting question left to future work.

Revised: In the next sections we discuss our results on identiﬁability of ICA for different function classes.

Original: An illustration of the considered classes can be found in Figure 1.
Revised: An illustration of the considered classes can be found in Figure 1.

Revised: They are all characterized by alocal condition on their gradient.

Revised: Previously, in [21] it was shown that the function class of localisometries is identiﬁable.

Original: Results for conformal maps 
Revised: Local isometries have been used frequently in machine learning, and morespeciﬁcally in representation learning [39, 44, 10]. Our main results consider two generalizationsof these function classes, conformal maps and orthogonal coordinate transformations. Conformalmaps preserve angles locally and have been used in computer vision [41, 45, 17]. Note that thisclass contains few functions in ﬁxed dimension (i.e., mapping R d !

Revised: R d ).

Revised: However, those are richclasses when the target space is higher dimensional than the domain (i.e., R d !

Revised: R n , d < n ).

Revised: OCTare an even more general class that were motivated based on the principle of independent mechanismin [16].

Revised: Moreover, many works showed that training VAEs promotes orthogonality of the columnsof the input Jacobian [38, 48, 30] and this has been empirically shown to be a good inductive biasfor disentanglement.

Revised: Indeed, these algorithms are widely used in representation learning and oftenrecover semantically meaningful representations [31, 4, 28, 19].

Revised:  Results for conformal maps

Original: Our ﬁrst main result is an extension of Theorem 1 to conformal maps.
Revised: Our ﬁrst main result is an extension of Theorem 1 to conformal maps.

Original: A conformal map is a mapthat locally preserves angles, i.e. locally it looks like a scaled rotation.
Revised: A conformal map is a mapthat locally preserves angles, i.e. locally it looks like a scaled rotation.

Original: It can be shown that this isequivalent to the following deﬁnition.
Revised: It can be shown that this isequivalent to the following deﬁnition.

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: (Conformal map)
Revised: (Conformal map)

Original: We deﬁne for domains Ω ⊂ R d the set of conformal maps by
Revised: We deﬁne for domains ⌦ ⇢ R d the set of conformal maps by

Original: F conf = { f ∈ C 1 (Ω , R d ) : Df ( x ) = λ ( x ) O ( x ) } where λ : Ω → R \ { 0 } is a scalar function and
Revised: F conf = { f 2 C 1 ( ⌦ , R d ) : Df ( x ) =  ( x ) O ( x ) } where  : ⌦ ! R \ { 0 } is a scalar function and

Original: O : Ω → O( d ) is a map to orthogonal matrices (i.e., O ( x ) − 
 
Revised: O : ⌦ ! O( d ) is a map to orthogonal matrices (i.e., O ( x )  
 = O ( x ) > ).

Revised: All our results also hold for the more general class of conformal maps f : ⌦ !

Original: = O ( x ) (cid:62) ).
Revised: M where M is a Riemannian manifold. The complete deﬁnition can be found in Appendix B.

Original: For convenience we deﬁne signed permutation matrices by
Revised: For convenience wedeﬁne signed permutation matrices by

Original: Perm
Revised: Perm

Original: [Equation]
Revised: [Equation]

Original:  is [Equation] permutation matrix where Q ij
Revised: ⇥ d is a permutation matrix where Q ij

Original: [Equation]

Original: | P

Original: [Equation]
Revised: [Equation]

Original: i.e. the set of matrices whose entry-wise absolute value is a permutation.
Revised: i.e. the set of matrices whose entry-wise absolute value is a permutation.

Original: Later we will also use thenotation Diag( d ) and Perm( d ) for d × d diagonal and permutation matrices, respectively.
Revised: Later we will also use thenotation Diag( d ) and Perm( d ) for d ⇥ d diagonal and permutation matrices, respectively.

Original: We deﬁne
Revised: We deﬁne

Original: S conf = { x → κPx + a where P ∈ Perm ± ( d ) , a ∈ R d , κ ∈ R } [Equation] 7) and
Revised: S conf = { x !  Px + a where P 2 Perm ± ( d ) , a 2 R d ,  2 R } (7) and

Original: P conf =
Revised: P conf =

Original: P [Equation] n∩ P lin , where
Revised: P ⌦ n\ P lin , where

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: R [Equation] , there is ∅ (cid:54) = O ⊂ R open, s.t. µ has positive C 2 density on O } 
Revised: ) , there is ; 6 = O ⇢ R open, s.t. µ has positive C 2 density on O } .

Original: [Equation] While this condition might appear a bit technical it actually only rules out pathological cases likethe cantor measure or densities which are nowhere differentiable and probably it could be relaxedfurther.
Revised: While this condition might appear a bit technical it actually only rules out pathological cases likethe cantor measure or densities which are nowhere differentiable and probably it could be relaxedfurther.

Original: In particular P 1 contains all probability measures with piecewise smooth densities.
Revised: In particular P 1 contains all probability measures with piecewise smooth densities.

Original: Then thefollowing identiﬁability for conformal maps in dimension d > 2 holds.
Revised: Then thefollowing identiﬁability for conformal maps in dimension d > 2 holds.

Original: Theorem 2.
Revised: Theorem 2.

Original: For d > 2 , ICA with respect to the pair ( F conf , P conf ) is identiﬁable up to S conf .
Revised: For d > 2 , ICA with respect to the pair ( F conf , P conf ) is identiﬁable up to S conf .

Original: This means that we can identify conformal maps up to three symmetries, namely constant shifts ofthe distributions, rescaling of all coordinates by a constant factor, and permutations of the coordinates.
Revised: This means that we can identify conformal maps up to three symmetries, namely constant shifts ofthe distributions, rescaling of all coordinates by a constant factor, and permutations of the coordinates.

Original: The proof is in Appendix D. The main ingredient in the proof is that conformal maps in dimensiond > 2 are very rigid and can be characterized explicitly as we will discuss in Section 6.
Revised: The proof is in Appendix E. The main ingredient in the proof is that conformal maps in dimensiond > 2 are very rigid and can be characterized explicitly as we will discuss in Section 6.

Original: For dimension d = 2 it was shown [18] that conformal maps can be identiﬁed up to a rotation whenﬁxing one point of the conformal map (setting f (0) = 0 ).
Revised: For dimension d = 2 it was shown [24] that conformal maps can be identiﬁed up to a rotation whenﬁxing one point of the conformal map (setting f (0) = 0 ).

Original: The authors also claim, without proof,that the remaining ambiguity can be removed for typical probability distributions.
Revised: The authors also claim, without proof,that the remaining ambiguity can be removed for typical probability distributions.

Original: We extend theirresult by removing the condition that one point is ﬁxed and prove full identiﬁability with a minorassumption on the involved densities.
Revised: We extend theirresult by removing the condition that one point is ﬁxed and prove full identiﬁability with a minorassumption on the involved densities.

Original: We deﬁne the following set of probability measures on R 

Revised: We deﬁne the following set of probability measures on R

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: conf
Revised: supp

Original: [Equation]
Revised: [Equation]

Original: t. supp
Revised: P

Original: [Equation]
Revised: [Equation]

Original: P i ) is a bounded interval 
Revised: ) is a bounded interval I

Revised: [Equation]

Original: I i and 
Revised: and [Equation]

Original: P i has density bounded above and below on I
Revised: density bounded above and below on I

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Then we get the following result.
Revised: Then we get the following result.

Original: Theorem 3.
Revised: Theorem 3.

Original: For d = 2 , ICA with respect to the pair ( F conf , P conf2 ) is identiﬁable up to S conf .
Revised: For d = 2 , ICA with respect to the pair ( F conf , P conf2 ) is identiﬁable up to S conf .

Original: This means that we can identify conformal maps on compact domains in dimension 2 up to shifts,permutations of coordinates, and scale.
Revised: This means that we can identify conformal maps on compact domains in dimension 2 up to shifts,permutations of coordinates, and scale.

Original: Note that we can also identify conformal maps if P has fullsupport R 2 using the same proof as for d > 2 (see Lemma 2 in the supplement).
Revised: Note that we can also identify conformal maps if P has fullsupport R 2 using the same proof as for d > 2 (see Lemma 2 in the supplement).

Original: The proof of thisresult is in Appendix D. It relies on the Schwarz-Christoffel mapping that provides a formula forconformal maps from the upper half plane or the unit disc to polygons.
Revised: The proof of thisresult is in Appendix E. It relies on the Schwarz-Christoffel mapping that provides a formula forconformal maps from the upper half plane or the unit disc to polygons.

Original:  Results for orthogonal maps
Revised:  Results for orthogonal maps

Original: Recently, in [13], the more general class of OCTs was considered in the context of ICA.
Revised: Recently, in [16], the more general class of OCTs was considered in the context of ICA.

Original: They referredto orthogonal coordinates as IMA maps, referencing to independent mechanisms.
Revised: They referredto orthogonal coordinates as IMA maps, referencing to independent mechanisms.

Original: This nomenclaturewas motivated by the causality literature and we refer to their paper for an extensive motivation andfurther results.
Revised: This nomenclaturewas motivated by the causality literature and we refer to their paper for an extensive motivation andfurther results.

Original: As we focus on theoretical results for this function class we stick to the more commonterm of OCTs.
Revised: As we focus on theoretical results for this function class we stick to the more commonterm of OCTs.

Original: Orthogonal coordinate transformations are deﬁned as the set of functions whosederivative have orthogonal columns, i.e., the vectors ∂ i f and ∂ j f are orthogonal for i(cid:54) =
Revised: Orthogonal coordinate transformations are deﬁned as the set of functions whosederivative have orthogonal columns, i.e., the vectors @ i f and @ j f are orthogonal for i =

Original: j .
Revised: j .

Original: Deﬁnition 3.
Revised: Deﬁnition 3.

Original: (OCT maps)
Revised: (OCT maps)

Original: We deﬁne for domains Ω ⊂ R d the set of OCT maps (orthogonalcoordinates) by F OCT = { f ∈ C 1 (Ω , R d ) : Df ( x ) (cid:62) Df ( x
Revised: We deﬁne for domains ⌦ ⇢ R d the set of OCT maps (orthogonalcoordinates) by F OCT = { f 2 C 1 ( ⌦ , R d ) : Df ( x ) >

Original:  ) ∈ Diag( d ) } .
Revised: Df ( x ) 2 Diag( d ) } .

Revised: Again, our results apply to maps whose target is a d -dimensional manifold.

Original:  First, we note that we can only hope to identify a mechanism f ∈ F OCT up to coordinate-wisetransformations and permutations.
Revised: The generalized deﬁnitioncan be found in Appendix B. First, we note that we can only hope to identify a mechanism f 2 F OCTup to coordinate-wise transformations and permutations.

Original: Therefore we set
Revised: Therefore we set

Original: S OCT = { g : R d → R d | g =
Revised: S [Equation] = { g [Equation]

Original:  P ◦ h where P ∈ Perm ± ( d ) and h : R d → R d with h ( x ) = 
Revised: R d | g = P  h where P 2 Perm ± ( d ) and h : R d ! R d with h ( x ) = ( h

Original: ( h 1 ( x 1 ) , . ..
Revised: [Equation] ( x 1 ) , . .

Original: , h d ( x d )) (cid:62) for some h i ∈ C 1 ( R , R ) with h (cid:48) i > 0 } .
Revised: , h d ( x d )) > for some h i 2 C 1 ( R , R ) with h 0 i > 0 } .

Original: It is easy to see that S OCT is a group.
Revised: It is easy to see that S OCT is a group.

Original: We note that if f ∈ F OCT and g ∈ S OCT then f ◦ g ∈ F OCT . {+
+}
Revised: We note that if f 2 F OCT and g 2 S OCT then f  g 2 F OCT . {+
+}

Original: Thus, in particular f ∗ P =
Revised: Thus, in particular f ⇤ P =

Original: ( f ◦ g ) ∗ ( g − 1 ) ∗ P .
Revised: ( f  g ) ⇤ ( g  1 ) ⇤ P .

Original: This implies that given observations from f ∗ P we canidentify f and P only up.
Revised: This implies that given observations from f ⇤ P we can 191 identify f and P only up.

Original: to g ∈ S OCT .
Revised: to g 2 S OCT .

Original: More precisely, for any (sufﬁciently smooth) P (cid:48) there is f (cid:48)such that f ∗ P = f (cid:48)∗ P (cid:48) where we pick g such that P (cid:48){+
+}
Revised: More precisely, for any (sufﬁciently smooth) P 0 there is f 0such that f ⇤ P = f 0⇤ P 0 where we pick g such that P {+
+}

Original: = g − 1 ∗ P .[-
-]
Revised: = g  1 ⇤ P .[-
-]

Original: As the distribution of the s i is not identiﬁable we will map it to a ﬁxed reference distribution wherewe will choose the uniform distribution on (0 , 1) d .
Revised: As the distribution of the s i is not identiﬁable we will map it to a ﬁxed reference distribution wherewe will choose the uniform distribution on (0 , 1) d .

Original: We introduce the shorthand C d = (0 , 1) d for thestandard unit cube and denote by ν the uniform (Lebesgue) measure on C d .
Revised: We introduce the shorthand C d = (0 , 1) d for thestandard unit cube and denote by ⌫ the uniform (Lebesgue) measure on C d .

Original: For ﬁxed base measure νthe symmetry group is reduced to permutations and reﬂections, i.e., maps in P ∈ Perm ± ( d ) .
Revised: For ﬁxed base measure ⌫the symmetry group is reduced to permutations and reﬂections, i.e., maps in P 2 Perm ± ( d ) .

Original: We conjecture that for ’typical’ pairs of f ∈ F OCT , P ∈ P
Revised: We conjecture that for ’typical’ pairs of f 2 F OCT , P 2 P

Original: OCT ICA is identiﬁable with respectto S OCT (with a suitable deﬁnition of P OCT , e.g., P OCT = P conf ).
Revised: OCT ICA is identiﬁable with respect 198 to S OCT (with a suitable deﬁnition of P OCT , e.g., P OCT = P conf ).

Original: However, we leave a precisestatement for future work.
Revised: However, we leave a precise 199 statement for future work.

Original: Below we show a weaker notion of identiﬁability for OCTs but ﬁrst weconsider the exceptional classes of spurious solutions OCTs.
Revised: Below we show a weaker notion of identiﬁability for OCTs but ﬁrst weconsider the exceptional classes of spurious solutions OCTs.

Original: Spurious solutions for ICA with OCTs.
Revised: Spurious solutions for ICA with OCTs.

Original: First we remark that just as for linear maps and conformalmaps (see Thm. 1) Gaussian distributions can hamper identiﬁcation.
Revised: First we remark that just as for linear maps and conformalmaps (see Thm. 1) Gaussian distributions can hamper identiﬁcation.

Original: Note that a suitable coordinatewise transformation transforms a measure with independent coordinates to a multivariate Gaussian.
Revised: Note that a suitable coordinatewise transformation transforms a measure with independent coordinates to a multivariate Gaussian.

Original: Fact 1 .
Revised: Fact 1 .

Original: Let P be a probability measure on R with bounded density p and cumulative distributionfunction F P .
Revised: Let P be a probability measure on R with bounded density p and cumulative distributionfunction F P .

Original: Denote the cumulative distribution function of the standard normal by F N .
Revised: Denote the cumulative distribution function of the standard normal by F N .

Original: Leth P = F − 1 N ◦ F P .
Revised: Leth P = F  1 N  F P .

Original: Then ( h P ) ∗ P has standard normal distribution.
Revised: Then ( h P ) ⇤ P has standard normal distribution.

Original: This implies that if P = P 1 ⊗ . . .
Revised: This implies that if P = P 1 ⌦ . . .

Original: ⊗ P d and x = f ( s ) =
Revised: ⌦ P d and x = f ( s ) =

Original: ( h P 1 ( s 1 ) , . . .
Revised: ( h P 1 ( s 1 ) , . . .

Original: , h P d ( s d )) then f ∗ P follows astandard normal distribution.
Revised: , h P d ( s d )) then f ⇤ P follows astandard normal distribution.

Original: In particular, for every A ∈ O( d ) the distribution of Ax is standardnormal and its components are independent.
Revised: In particular, for every A 2 O( d ) the distribution of Ax is standardnormal and its components are independent.

Original: Note that in contrast to conformal and linear maps it isnot sufﬁcient to exclude Gaussian source distribution but due to the ﬂexibility of the function classwe have to exclude that the pair ( f, P ) has a Gaussian observational distribution f ∗ P .
Revised: Note that in contrast to conformal and linear maps it isnot sufﬁcient to exclude Gaussian source distribution but due to the ﬂexibility of the function classwe have to exclude that the pair ( f, P ) has a Gaussian observational distribution f ⇤ P .

Original: Next we showthat a more general construction using OCTs is possible.
Revised: Next we showthat a more general construction using OCTs is possible.

Original: Proposition 1.
Revised: Proposition 1.

Original: Let P be a rotation invariant distribution on R d with smooth density.
Revised: Let P be a rotation invariant distribution on R d with smooth density.

Original: Then there is asmooth and invertible (on its image) function f : C d → R d with f ∈ F OCT such that f ∗ ν = P .
Revised: Then there is asmooth and invertible (on its image) function f : C d ! R d with f 2 F OCT such that f ⇤ ⌫ = P .

Original: The proof of this result can be found in Appendix E.
Revised: The proof of this result can be found in Appendix F.

Original: The main idea in the proof is that d -dimensionalpolar coordinates do the trick up to coordinate-wise rescaling.
Revised: The main idea in the proof is that d -dimensionalpolar coordinates do the trick up to coordinate-wise rescaling.

Original: This proposition implies that theentire family { f R = 
Revised: This proposition implies that theentire family { f R = R  f

Original: R ◦ f | R ∈ O ( n ) } satisﬁes ( f R ) ∗ ν = f ∗ ν and, by deﬁnition of F OCT we havef R ∈ F OCT because f ∈ F OCT implies R ◦ f ∈ F OCT .
Revised: | R 2 O ( n ) } satisﬁes ( f R ) ⇤ ⌫ = f ⇤ ⌫ and, by deﬁnition of F OCT we havef R 2 F OCT because f 2 F OCT implies R  f 2 F OCT .

Original: In particular all inverses g R = f − 1 R recoverindependent sources in the sense that ( g R ) ∗ f∗ ν = ν
Revised: In particular all inverses g R = f  1 R recover 219 independent sources in the sense that ( g R ) ⇤ f

Original:  and BSS is not possible in a meaningful wayin this (special) case.
Revised: ⇤ ⌫ = ⌫ and BSS is not possible in a meaningful wayin this (special) case.

Original: The construction in Proposition 1 gives spurious solutions for substantiallymore observational distributions than just the Gaussian (this is indicated in Table 1).
Revised: The construction in Proposition 1 gives spurious solutions for substantiallymore observational distributions than just the Gaussian (this is indicated in Table 1).

Original: Neverthelesswe view this not as a general obstacle to identiﬁability results for OCTs for two reasons.
Revised: Neverthelesswe view this not as a general obstacle to identiﬁability results for OCTs for two reasons.

Original: Firstly thespurious solutions only apply to carefully chosen pairs of function and base measure such that weobtain a still very non-generic (radial) observational distribution.
Revised: Firstly thespurious solutions only apply to carefully chosen pairs of function and base measure such that weobtain a still very non-generic (radial) observational distribution.

Original: Moreover, the function constructedin Proposition 1 cannot be extended to C d = [0 , 1] d such that it remains invertible (in the language ofdifferential geometry this means that f is only an immersion not an embedding of a submanifold).
Revised: Moreover, the function constructedin Proposition 1 cannot be extended to C d = [0 , 1] d such that it remains invertible (in the language ofdifferential geometry this means that f is only an immersion not an embedding of a submanifold).

Original: Inthis sense the main message of Proposition 1 is that an identiﬁability result for F OCT needs to containassumptions that rule out those spurious solutions (just like the Gaussian is excluded for linear ICA).
Revised: Inthis sense the main message of Proposition 1 is that an identiﬁability result for F OCT needs to containassumptions that rule out those spurious solutions (just like the Gaussian is excluded for linear ICA).

Original: Local Stability of OCTs.
Revised: Local Stability of OCTs.

Original: We now give partial results towards identiﬁability of OCTs.
Revised: We now give partial results towards identiﬁability of OCTs.

Original: While wedo not prove general identiﬁability for OCTs, we demonstrate their local rigidity : OCTs cannot becontinuously deformed to obtain spurious solutions.
Revised: While wedo not prove general identiﬁability for OCTs, we demonstrate their local rigidity : OCTs cannot becontinuously deformed to obtain spurious solutions.

Original: This is in stark contrast to the general nonlinearcase which we will discuss for comparison below.
Revised: This is in stark contrast to the general nonlinearcase which we will discuss for comparison below.

Revised: Actually we will show a slightly stronger statement.

Revised: Suppose we know some initial data generating mechanism f 0 because we have, e.g., access to samples( s, f 0 ( s )) .

Original: The result will be based on the following deﬁnition.
Revised: Now we assume that the mixing f t depends smoothly on some parameter t which couldbe, e.g., time or an environment. Then we can identify f t if f t 2 F OCT for all t and given access tosamples f t ( s ) . This is not true for variations of general nonlinear mixing functions. We now makethese statements precise.

Revised: We ﬁrst deﬁne smooth deformations of a mixing function.

Original: Deﬁnition 4.
Revised: Deﬁnition 4.

Original: (Smooth invariant deformations)  Consider a family of differentiable transformationsΦ ∈ C 1 (( − T, T ) × R d , R d ) for some T > 0  such that Φ t =
Revised: (Smooth invariant deformations) Let F be some function class. Consider a familyof differentiable transformations  2 C 1 ((  T, T ) ⇥ R d , M ) for some T > 0 and a smooth, ddimensional manifold M such that  t =

Original: Φ( t, · ) is diffeomorphism.
Revised:  ( t, · ) is diffeomorphism.

Original: Let

Original: P t ∈ M 1 ( R ) ⊗ d for t ∈ ( − T, T ) be a family of probability measures. We call (Φ t , P t) a smoothinvariant deformation of the pair ( f, P ) if [-Φ 
 = f , P 0 = P and (Φ-] t  ) ∗ P t= f ∗ P .
Revised: We call  t a smooth invariantdeformation if [-Φ 
 = f , P 0 = P and (Φ-] t ( · ) 2 F for all t .

Original: An illustration of this deﬁnition can be found in Figure 3.
Revised: An illustration of this deﬁnition can be found in Figure 3.

Original: Based on invariant deformations we cannow deﬁne a local identiﬁability property of ICA in a given function class.
Revised: Based on invariant deformations we cannow deﬁne a local identiﬁability property of ICA in a given function class.

Original: Deﬁnition 5.
Revised: Deﬁnition 5.

Original: (Local identiﬁability of ICA).
Revised: (Local identiﬁability of ICA).

Original: Consider a function class F .
Revised: Consider a function class F .

Original: Let (Φ t, ν ) be a smoothinvariant deformation of ( f, ν ) (i.e., P t {+such that  
 = f+}
Revised: Let f t be a smooth invariantdeformation in F that is analytic in t . Let  t be another smooth invariant deformation in F analyticin t {+such that  
 = f+}

Original:  =ν is constant).
Revised: and (  t ) ⇤ ⌫ =

Original:  Assume thatΦ t ∈ F and there is an openset Ω ⊂ C d with ¯Ω ⊂ C d such that Φ t ( x ) = f 
Revised: ( f t ) ⇤ ⌫ for all t . Assume that there is " > 0 such that t ( s ) = f t ( s ) if dist( s, @

Original: ( x ) for x / ∈ Ω .
Revised: C d ) < " .

Original: Then we say that ICA in F is locallyidentiﬁable at ( f, ν ) if under these assumptions Φ t = f  for all t .
Revised: Then we say that ICA in F is locally identiﬁable at ( f t , ⌫ ) ifthese assumptions imply  t = f t for all t .

Original: We call F locally identiﬁable if it is locally identiﬁable at ( f, ν ) for all  f ∈ F .
Revised: We call F locally identiﬁable if it is locally identiﬁable at( f t , ⌫ ) for all analytic local deformations f t .

Original: Local identiﬁabilityof a function class means that we cannot smoothly transform a function f ∈ F to any other functionf (cid:48)
Revised: Local identiﬁability of a function class means that we can identify smooth deformations f t 2 F fromsome initial mixing f 0 2 F if f 0 is known on the whole latent domain, and the behaviour of f t closeto the domain’s boundary is known for all t . This provides us with an identiﬁability result in a settingwith concept drift [46, 14].

Original: ∈ F such thatwe stay in F along the transformation path, the  observational distribution remainsthe same, and we ﬁx the functions close to the boundary of C d .
Revised: Formulated differently, it means that an adversary cannot deform thefunction f 0 such that the outcome is ambiguous given the resulting observational distribution. Anexperimental illustration of this setting and Theorem 4 below can be found in Appendix H. Note that

Original: Note that this deﬁnition is local intwo ways.
Revised: Deﬁnition 5 is local in two ways.

Original: We consider smooth transformations of the ground truth, i.e., small changes of the datagenerating function f {+
+}
Revised: We consider smooth transformations of the ground truth, i.e., smallchanges of the data generating function f {+
+}

Original: and in addition we assume that the changes are local in s , i.e., sources s closeto the boundary are kept invariant.
Revised: and in addition we assume that the changes are local in s ,i.e., sources s close to the boundary are kept invariant.

Original: An extension to varying P is possible but not necessary in thecontext of OCTs as explained above.
Revised: An extension to varying P is possible but notnecessary in the context of OCTs as explained above.

Original: We can show that local identiﬁability holds true in F OCT .
Revised: We can show that local identiﬁability holdstrue in F OCT .

Original: Theorem 4.
Revised: Theorem 4.

Original: The function class F OCT is locally identiﬁable at ( f, ν ) for all f ∈ F OCT .
Revised: The function class F OCT is locally identiﬁable.

Original: Let us reiterate what this theorem shows (illustrated in Fig.

Original: 3): we cannot smoothly and locallytransform the function f such that (1) the observational distribution remains invariant, i.e., equal to f ∗ ν , and (2) the deformed functions remain OCTs.

Original: The proof, in Appendix E, introduces new toolsto the ﬁeld of ICA.
Revised: The proof, in Appendix F, introduces new tools to the ﬁeld of ICA.

Original: The main idea is to consider the vector ﬁeld X that generates the deformation Φ tand then rewrite the assumption as systems of partial differential equations for X .
Revised: The main idea is to considerthe vector ﬁeld X that generates the deformation  t and then rewrite the assumption as systems ofpartial differential equations for X .

Original: The proof is thencompleted by showing that the only solution of this system vanishes.
Revised: The proof is then completed by showing that the only solution ofthis system vanishes.

Revised: Let us state one simple consequence of this theorem.

Revised: Corollary 1.

Revised: Let  t be a smooth analytic invariant deformation in F OCT such that (  t ) ⇤ ⌫

Revised: = f ⇤ ⌫ 267 for all t and there is " > 0 such that  t ( s ) =

Revised: f ( s ) if dist( s, @

Original:  At a high level this result suggests that OCTs can be identiﬁed if we know f close to the boundary ofthe support of s , e.g., by having, in addition to unlabelled data f ( s ) , labelled data ( s, f ( s )) for thoses where one coordinate s i is extremal.
Revised: C d ) < " . Then  t = f for all t . Let us reiterate what this corollary shows: we cannot smoothly and locally transform the function fsuch that (1) the observational distribution remains invariant, i.e., equal to f ⇤ ⌫ , and (2) the deformedfunctions remain OCTs. At a high level this result suggests that OCTs can be identiﬁed if we know f close to the boundary ofthe support of s , e.g., by having, in addition to unlabelled data f ( s ) , labelled data ( s, f ( s )) for thoses where one coordinate s i is extremal.

Original: Note that we actually do not show this result as there might befurther solutions which are not connected by smooth transformations.
Revised: Note that we actually do not show this result as there might befurther solutions which are not connected by smooth transformations.

Original: We expect that this result canbe generalised substantially.
Revised: We expect that those resultscan be generalised substantially.

Original: In particular, we conjecture that for “most” functions
Revised: In particular, we conjecture that for “most” functions

Original: f the compactsupport condition can be removed thus giving a stronger local identiﬁability result up to the boundaryof the support of s .
Revised: f the boundarycondition can be removed thus giving a stronger local identiﬁability result up to the boundary of thesupport of s .

Original: As a partial result in this direction we show the following theorem.
Revised: As a partial result in this direction we show the following theorem.

Original: Theorem 5.
Revised: Theorem 5.

Original: Let f : C d → R d be given by f ( x ) = RDx , where R ∈ O ( d ) and D =
Revised: Let f : C d ! R d be given by f ( x ) = RDx , where R 2 O( d ) and D =

Original: Diag( µ 1 , . . . , µ d ) with µ i > 0 and µ − 2 i are linearly independent over the rational numbers Q .
Revised: Diag( µ 1 , . . . , µ d ) with µ i > 0 and µ  2 i are linearly independent over the rational numbers Q .

Original: Suppose that (Φ t , ν ) (i.e., P t= ν is constant) is a smooth invariant deformation of{+such that  
 = f ,+} ( f, ν )  and Φ t isanalytic in t .
Revised: Suppose that  t is a smooth invariant deformation {+such that  
 = f ,+} (  t ) ⇤ ⌫ = f ⇤ ⌫ , and  t isanalytic in t .

Original: Then Φ t ∈ F OCT for all t implies that Φ t = f on C d , i.e., Φ t is constant in time.
Revised: Then  t = f on C d , i.e.,  t is constant in time.

Original: The proof is in Appendix E and similar to Theorem 4.
Revised: The proof is in Appendix F and similar to Theorem 4.

Original: Note that for random µ i whose distributionhave a bounded density the condition is satisﬁed with probability one.
Revised: Note that for random µ i whose distributionhave a bounded density the condition is satisﬁed with probability one.

Original: Moreover, we do not expectnon-constant Φ t to exist when we remove the assumption on the µ i
Revised: Moreover, we do not expectnon-constant  t to exist when we remove the assumption on the µ i

Original: but we are unable to show this.
Revised: but we are unable to show this.

Original: Comparison with ICA for general nonlinear functions.
Revised: Comparison with ICA for general nonlinear functions.

Original: Let us emphasize that those results arenon-trivial as they establish a large difference between ICA with generic nonlinear maps and ICAwith OCTs.
Revised: Let us emphasize that those results arenon-trivial as they establish a large difference between ICA with generic nonlinear maps and ICAwith OCTs.

Original: To clarify this we state that no result similar to Theorem 4 holds without the assumptionthat Φ t ∈ F OCT .
Revised: To clarify this, we state that no result similar to Theorem 4 holds without the assumptionthat  t 2 F OCT .

Original: Put differently, the function class F nonlinear is not locally identiﬁable.
Revised: Put differently, the function class F nonlinear is not locally identiﬁable.

Original: Fact 2 .
Revised: Fact 2 .

Original: Suppose f : C d → R d is a diffeomorphism on its image.
Revised: Suppose f : C d ! R d is a diffeomorphism on its image.

Original: Then there are many smoothinvariant deformations (Φ , P t) of ( f, ν ) such that P t 
Revised: Then there are many smoothdeformations  t of ( f, ⌫ ) such that (  t )

Original:  =ν
Revised: ⇤ ⌫ =

Revised: f ⇤ ⌫

Original: and Φ t | Ω c = f | Ω c for any Ω ⊂ C d open.
Revised: and there is an " > 0 such that  t ( s ) = f ( s ) ifdist( s, @ C d ) < " .

Original: For completeness we provide a general construction that is close to our proof of Theorem 4 in
Revised: For completeness, we provide a general construction that is close to our proof of Theorem 4 in

Original: Appendix B in the supplement.
Revised: Appendix C in the supplement.

Original: A very clear construction for this result was given in [18].
Revised: A very clear construction for this result was given in [24].

Original: Lemma 1 (Smoothly varying radius dependent rotations (see [18])) .
Revised: Lemma 1 (Smoothly varying radius dependent rotations (see [24])) .

Original: Let R : R × R + → O( d ) bea smooth function mapping to orthogonal matrices and let a ∈ C d .
Revised: Let R : R ⇥ R + ! O( d ) bea smooth function mapping to orthogonal matrices and let a 2 C d .

Original: Assume that R ( t, r ) =
Revised: Assume that R ( t, r ) =

Original: id forr ≥ dist( a, ∂C d ) .
Revised: id forr  dist( a, @ C d ) .

Original: Then the map s → h R,a ( t, s ) =
Revised: Then the map s ! h R,a ( t, s ) =

Original: R ( | s − a | , t )( s − a ) + a preserves the uniformmeasure ν on C d for all t so that f ◦ h R,a ( t, s ) D = f ( s ) for all t if s is distributed according to ν .
Revised: R ( | s  a | , t )( s  a ) + a preserves the uniformmeasure ⌫ on C d for all t so that f  h R,a ( t, s ) D = f ( s ) for all t if s is distributed according to ⌫ .

Original: An illustration of this construction is shown in Figure 2 (see App.
Revised: An illustration of this construction is shown in Figure 2 (see App.

Original: B for details).
Revised: C for details).

Original: Clearly, byconcatenation this allows us to create a vast family of spurious solutions.
Revised: Clearly, byconcatenation this allows us to create a vast family of spurious solutions.

Original: Note that those solutionsare excluded when restricting to OCTs which is a corollary of Theorem 4.
Revised: Note that those solutionsare excluded when restricting to OCTs which is a corollary of Theorem 4.

Original: Corollary 1.
Revised: Corollary 2.

Original: Suppose f ∈ F OCT .
Revised: Suppose f 2 F OCT .

Original: Let Φ t be the smooth invariant deformation deﬁned by Φ t =f ◦ h R,a ( t, · ) where h R,a is as in Lemma 1.
Revised: Let  t be the smooth invariant deformation deﬁned by  t = 302 f  h R,a ( t, · ) where h R,a is as in Lemma 1.

Original: If Φ t ∈ F OCT for all t this implies that h R,a ( t, s ) =
Revised: If  t 2 F OCT for all t this implies that h R,a ( t, s ) =

Original: sand Φ t = f for all t .
Revised: s 303 and  t = f for all t .

Original: We summarize the results of this section informally (we do not claim that the statements belowregarding (inﬁnite dimensional) manifolds can be made rigorous).
Revised: We summarize the results of this section informally (we do not claim that the statements belowregarding (inﬁnite dimensional) manifolds can be made rigorous).

Original: For a given data generatingmechanism ( f 0 , ν ) we expect that typically the set of all solutions M OCT = { f ∈ F OCT | f ∗ ν =( f 0 ) ∗ ν } ⊂ F OCT is a zero dimensional submanifold, i.e., consists of isolated spurious solutionsand{+solutions 
 and+} we prove this when ﬁxing the boundary (see Theorem 4) while the corresponding submanifoldof general nonlinear spurious solutions M nonlinear
Revised: For a given data generatingmechanism ( f 0 , ⌫ ) we expect that typically the set of all solutions M OCT = { f 2 F OCT | f ⇤ ⌫ = 307 ( f 0 ) ⇤ ⌫ } ⇢ F OCT is a zero dimensional submanifold, i.e., consists of isolated spurious {+solutions 
 and+} we prove this when ﬁxing the boundary (see Theorem 4) while the corresponding submanifoldof general nonlinear spurious solutions M nonlinear

Original: = { f : C d → R d | f ∗ ν = ( f 0 ) ∗ ν } is inﬁnitedimensional even when requiring f ( s ) =
Revised: = { f : C d ! R d | f ⇤ ⌫ = ( f 0 ) ⇤ ⌫ } is inﬁnite 310 dimensional even when requiring f ( s ) =

Original: f 0 ( s ) close to the boundary of C d .
Revised: f 0 ( s ) close to the boundary of C d .

Original:  Results for volume preserving maps
Revised:  Results for volume preserving maps

Original: Let us ﬁnally consider volume preserving transformations.
Revised: Let us ﬁnally consider volume preserving transformations.

Original: Those are deﬁned as the set of functions
Revised: Those are deﬁned as the set of functions

Original: [Equation]
Revised: [Equation]

Original: Invertible volume preserving deformations have the property that they preserve the standard(Lebesgue)-measure λ in the sense that f ∗ λ Ω = λ f (Ω) .
Revised: Invertible volume preserving deformations have the property that they preserve the standard(Lebesgue)-measure  in the sense that f ⇤  ⌦ =  f ( ⌦ ) .

Original: Recently it was proposed that volumepreserving functions are a suitable function class for ICA.
Revised: Recently it was proposed that volumepreserving functions are a suitable function class for ICA.

Original: Here we show that those functions are not sufﬁciently rigid to allow identiﬁability of ICA in the unconditional case.
Revised: Here we show that those functions are notsufﬁciently rigid to allow identiﬁability of ICA in the unconditional case.

Original: Note that Lemma 1 and
Revised: Note that Lemma 1 and

Original: Fact 2 already show how to construct spurious solutions for the case that the base distribution is theuniform measure ν .
Revised: Fact 2 already show how to construct spurious solutions for the case that the base distribution is theuniform measure ⌫ .

Original: However, for an arbitrary distribution P this is slightly more difﬁcult becausewe need to ﬁnd maps g that preserve P , i.e., g ∗ P = P , and are volume preserving, i.e., preserve thestandard measure.
Revised: However, for an arbitrary distribution P this is slightly more difﬁcult becausewe need to ﬁnd maps g that preserve P , i.e., g ⇤ P = P , and are volume preserving, i.e., preserve thestandard measure.

Original: Nevertheless, we have the following theorem.
Revised: Nevertheless, we have the following theorem.

Original: Theorem 6.
Revised: Theorem 6.

Original: Let p be a twice differentiable probability density with bounded gradient.
Revised: Let p be a twice differentiable probability density with bounded gradient.

Original: Suppose thatx = f
Revised: Suppose thatx = f

Original: ( s ) where the distribution P of s has density p and f is a diffeomorphism with det Df ( x ) =
Revised: ( s ) where the distribution P of s has density p and f is a diffeomorphism with det Df ( x ) =

Original: 1for x ∈ R d .
Revised: 1for x 2 R d .

Original: Then there is a family of functions f t : R × R 
Revised: Then there is a family of functions f t : R ⇥ R d !

Original: d → R d with f 0 = f and f t (cid:54) = f 0 for t (cid:54) = 0such
Revised: R d with f 0 = f and f t 6 = f 0 for t 6 = 0 324 such that det Df t ( x )

Original: that det Df t ( x ) = 1 and ( f t ) ∗ P = f ∗ P .
Revised: = 1 and ( f t ) ⇤ P = f ⇤ P .

Original: The proof and an illustration are in Appendix F.
Revised: The proof and an illustration are in Appendix G.

Original: It is based on the ﬂows generated by suitable explicitvector ﬁelds.
Revised: It is based on the ﬂows generated by suitable explicitvector ﬁelds.

Original: As those ﬂows can be concatenated we obtain a large family of spurious solutions.
Revised: As those ﬂows can be concatenated we obtain a large family of spurious solutions.

Original: Note that while f is not identiﬁable it can be possible to identify certain values of f ( s ) when we thedistribution of s is known, e.g., volume preservation implies that the source value with the largestdensity is mapped to the point with the largest density in the observational distribution.
Revised: Note that while f is not identiﬁable it can be possible to identify certain values of f ( s ) when we thedistribution of s is known, e.g., volume preservation implies that the source value with the largestdensity is mapped to the point with the largest density in the observational distribution.

Revised: We think thatthe approach used here is a powerful technique to construct counter-examples to identiﬁability in

Original:  As local identiﬁability is weaker than identiﬁability we get the following (informal) corollary.
Revised: ICA. As local identiﬁability is weaker than identiﬁability we get the following (informal) corollary.

Original: Corollary 2.
Revised: Corollary 3.

Original: ICA in F vol is not identiﬁable.
Revised: ICA in F vol is not identiﬁable.

Original: Note that this is even true when we know the distribution of s .
Revised: Note that this is even true when we know the distribution of s .

Original: The statement could be made rigorousby showing that if ( F vol , P vol ) is identiﬁable with respect to S vol then S vol contains functions mixingcoordinates s i , s j for i (cid:54) =
Revised: The statement could be made rigorousby showing that if ( F vol , P vol ) is identiﬁable with respect to S vol then S vol contains functions mixing 336 coordinates s i , s j for i 6 =

Original: j (i.e., there is h ∈ S vol such that ∂ i ∂
Revised: j (i.e., there is h 2 S vol such that @ i @

Original: j h (cid:54) = 0 ).
Revised: j h 6 = 0 ).

Original:  Relation to rigidity theory
Revised:  Relation to rigidity theory

Original: Our results rely on rigidity properties of certain function classes.
Revised: Our results rely on rigidity properties of certain function classes.

Original: Rigidity refers to the property that alocal constraint on the derivative of a function implies global restrictions on the shape of the function.
Revised: Rigidity refers to the property that alocal constraint on the derivative of a function implies global restrictions on the shape of the function.

Original: Let us clarify this through a well known example (a special case of Theorem 8 below).

Original: Theorem 7.

Original: Let f : Ω → R d with Ω ⊂ R d connected be a C 1 function such that Df ( x ) ∈ O( n ) forall x ∈ Ω .

Original: Then f

Original: ( x ) =

Original: Rx

Original: + b for some R ∈ O( d ) and b ∈ R d .

Original: This means that when the gradient of a function is pointwise a rotation, then the function is already aconstant rotation.

Original: Results of this type are of considerable interest in continuum mechanics.

Original: Thereit is natural to consider deformation of solids f : Ω → R d which are locally constrained by thestructure of the material.

Original: The condition in Theorem 7 corresponds to non-deformable solids whilethe condition det Df = 1 is used for incompressible ﬂuids.

Original: One important result in statistical mechanics is that results like Theorem 7 come with estimates in aneighbourhood of the function class, in the sense that if Df ( x ) is close to a rotation for all x , then itwill be close to a constant rotation [11, 4].

Original: Results of this type could be important for three reasons.

Original: First, in many real world applications it might be more realistic to assume that f is close to a certainfunction class F but not necessary contained in F .

Original: Secondly, it could rigorously justify the use ofsurrogate losses for the differential constraints (e.g., [13] consider the loss (cid:80)

Original: i ln( | ∂ i f | ) − ln | Det f | )and, thirdly, results of this type would be needed for ﬁnite sample analysis. We leave the further investigation of these matters to future work and return to the  rigidity result forconformal maps relevant for Theorem 2. This is an extension of Theorem 7 where the condition Df ∈ O( d ) is replaced by Df ∈
Revised: For conformal maps the following rigidity result holds.

Original: λ O( d ) for some λ (cid:54)

Original: = 0 .

Original: It turns out that this class is still very rigidin dimension d ≥ 3 in the sense that there is only one additional solution.

Original: Theorem 8 (Liouville) .
Revised: Theorem 7 (Liouville) .

Original: If Ω ⊂ R n is an open connected set and f : Ω → R n is conformal, then
Revised: If ⌦ ⇢ R n is an open, connected set and f : ⌦ ! R n is conformal, then

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original:  where b, a ∈ R n , α ∈ R , A ∈ O( n ) , and ε ∈ { 0 , 2 } .
Revised: 12) where b, a 2 R n , ↵ 2 R , A 2 O( n ) , and " 2 { 0 , 2 } .

Original: Originally this result was shown by Liouville [25], a modern treatment is [20].
Revised: Originally this result was shown by Liouville [33], a modern treatment is [26].

Original: In particular, thisshows that conformal maps are (up to translations) rotations or rotations followed by an inversion.
Revised: In particular, thisshows that conformal maps are (up to translations) rotations or rotations followed by an inversion.

Original: To illustrate the strength of Theorem 8 we compare it to the setting of volume preserving maps whichsatisfy no similar rigidity property.
Revised: To illustrate the strength of Theorem 7 we compare it to the setting of volume preserving maps whichsatisfy no similar rigidity property.

Original: Intuitively the different rigidity properties are already apparent from the connection to solids, which can merely be rotated and shifted, and ﬂuids which can alsobe stirred leading to chaotic deformations.
Revised: Intuitively the different rigidity properties are already apparentfrom the connection to solids, which can merely be rotated and shifted, and ﬂuids which can alsobe stirred leading to chaotic deformations.

Original: Rigorously the different behaviours can be clariﬁed bythe observation that conformal maps have a ﬁnite number of parameters and thus a ﬁnite number ofconstraints (e.g., of the form f ( x ) =
Revised: Rigorously the different behaviours can be clariﬁed bythe observation that conformal maps have a ﬁnite number of parameters and thus a ﬁnite number ofconstraints (e.g., of the form f ( x ) =

Original: y ) allows to identify them.
Revised: y ) allows to identify them.

Original: In contrast volume preserving mapscannot be identiﬁed from ﬁnitely many constraints as the following proposition shows.
Revised: In contrast volume preserving mapscannot be identiﬁed from ﬁnitely many constraints as the following proposition shows.

Original: Proposition 2.
Revised: Proposition 2.

Original: For d > 2 and { x 1 , . . .
Revised: For d > 2 and { x 1 , . . .

Original: , x n , y 1 , . . .
Revised: , x n , y 1 , . . .

Original: , y n } ⊂ R d , all pairwise different there is avolume preserving diffeomorphism f : R d → R d , f ∈ F vol such that f ( x i ) =
Revised: , y n } ⇢ R d , all pairwise different there is avolume preserving diffeomorphism f : R d ! R d , f 2 F vol such that f ( x i ) =

Original: y i .
Revised: y i .

Original: Let us emphasize that the different rigidity properties are not at all surprising when arguing basedon degrees of freedom or numbers of constraints.
Revised: Let us emphasize that the different rigidity properties are not at all surprising when arguing basedon degrees of freedom or numbers of constraints.

Original: While volume preserving maps enforce only asingle scalar constraint on the Jacobian Df the condition for conformal maps gives n ( n + 1) / 2 − 1constraints on the Jacobian.
Revised: While volume preserving maps enforce only asingle scalar constraint on the Jacobian Df the condition for conformal maps gives n ( n + 1) / 2  1constraints on the Jacobian.

Original: Let us ﬁnally comment on OCTs where the picture is not as well understood.
Revised: Let us ﬁnally comment on OCTs where the picture is not as well understood.

Original: There has been a lotof work on orthogonal coordinates quite some time ago [8, 3] and there are some rather implicitlocal characterisations of orthogonal coordinates.
Revised: There has been a lotof work on orthogonal coordinates quite some time ago [9, 3] and there are some rather implicitlocal characterisations of orthogonal coordinates.

Original: In particular, it was shown that OCTs, in contrastto conformal maps, cannot be parametrized by a ﬁnite number of parameters and therefore OCTsare much more ﬂexible than conformal maps.
Revised: In particular, it was shown that OCTs, in contrastto conformal maps, cannot be parametrized by a ﬁnite number of parameters and therefore OCTsare much more ﬂexible than conformal maps.

Original: It is not known if and what rigidity properties can bederived for OCTs.
Revised: It is not known if and what rigidity properties can bederived for OCTs.

Original: However, our results suggest that the additional measure preservation condition inthe context of ICA gives enough rigidity to (almost) give identiﬁability of ICA.
Revised: However, our results suggest that the additional measure preservation condition inthe context of ICA gives enough rigidity to (almost) give identiﬁability of ICA.

Original: In this sense OCTsmight be a good function class for ICA as it is rich enough to allow complex representations of datawhile at the same time being sufﬁciently rigid to still provide (some) identiﬁability.
Revised: In this sense OCTsmight be a good function class for ICA as it is rich enough to allow complex representations of datawhile at the same time being sufﬁciently rigid to still provide (some) identiﬁability.

Original:  Discussion
Revised:  Discussion

Original: ICA is long known to be identiﬁable for linear maps, baring pathological cases, and highly nonidentiﬁable for general nonlinear ones.
Revised: ICA is long known to be identiﬁable for linear maps, baring pathological cases, and highly nonidentiﬁable for general nonlinear ones.

Original: Surprisingly, similar results for function classes of intermediatecomplexity remain scarce.
Revised: Surprisingly, similar results for function classes of intermediatecomplexity remain scarce.

Original: In this work we address this question with several identiﬁability results fordifferent function classes.
Revised: In this work we address this question with several identiﬁability results fordifferent function classes.

Original: Our ﬁrst main result is that ICA is identiﬁable in the class of conformalmaps (up to classical ambiguities).
Revised: Our ﬁrst main result is that ICA is identiﬁable in the class of conformalmaps (up to classical ambiguities).

Original: This considerably extends previous claims, limited to a speciﬁc2D setting [18], and ruling out several families of spurious solutions [13].
Revised: This considerably extends previous claims, limited to a speciﬁc2D setting [24], and ruling out several families of spurious solutions [16].

Original: On the negative side weshow that the ICA problem for volume preserving maps admits a large class of spurious solutions.
Revised: On the negative side weshow that the ICA problem for volume preserving maps admits a large class of spurious solutions.

Original: Finally, we show that OCTs satisfy certain weaker notions of local identiﬁability.
Revised: Finally, we show that OCTs satisfy certain weaker notions of local identiﬁability.

Original: In our proofs, we draw connections to methods and techniques that, to the best of our knowledge,have not been used in the context of ICA before.
Revised: In our proofs, we draw connections to methods and techniques that, to the best of our knowledge,have not been used in the context of ICA before.

Original: We relate the identiﬁability problem in ICA tothe rigidity of the considered function class F and use tools from the theory of partial differentialequations.
Revised: We relate the identiﬁability problem in ICA tothe rigidity of the considered function class F and use tools from the theory of partial differentialequations.

Original: These techniques have been applied very successfully to the analysis of elastic solids[5, 4] and we believe that there are many applications of these methods in the ﬁeld of ICA.
Revised: These techniques have been applied very successfully to the analysis of elastic solids[6, 5] and we believe that there are many applications of these methods in the ﬁeld of ICA.

Original: While the main focus of current research after the seminal work of Hyvärinen et al.
Revised: While the main focus of current research after the seminal work of Hyvärinen et al.

Original: [19] is onthe auxiliary variable case, there are three reasons to consider unconditional ICA.
Revised: [25] is onthe auxiliary variable case, there are three reasons to consider unconditional ICA.

Original: Firstly, it is afundamental research question that is, as illustrated by our results, deeply rooted in functionalanalysis.
Revised: Firstly, it is afundamental research question that is, as illustrated by our results, deeply rooted in functionalanalysis.

Original: Secondly there is high application potential for completely unsupervised learning withoutany auxiliary variables, as the corresponding datasets do not require labelling or speciﬁc experimentalsettings.
Revised: Secondly there is high application potential for completely unsupervised learning withoutany auxiliary variables, as the corresponding datasets do not require labelling or speciﬁc experimentalsettings.

Original: Thirdly, it is very likely that the techniques can be generalised to the auxiliary variable case.
Revised: Thirdly, it is very likely that the techniques can be generalised to the auxiliary variable case.

Original: Another important open problem is assessing the type of constraints on ground truth mechanisms,encoded by function classes, that are relevant for real world data.
Revised: Another important open problem is assessing the type of constraints on ground truth mechanisms,encoded by function classes, that are relevant for real world data.

Original: It is plausible that those mechanisms are typically much more regular than generic nonlinear functions.
Revised: It is plausible that those mechanisms are typically much more regular than generic nonlinear functions.

Original: Recently, Gresele et al.[13] suggested, based on arguments from the causality literature that F OCT is a natural class forrepresentation learning (and our results show it also has favourable theoretical properties), but thiswill require experimental conﬁrmation on real world data.
Revised: Recently, Gresele et al.[16] suggested, based on arguments from the causality literature that F OCT is a natural class forrepresentation learning (and our results show it also has favourable theoretical properties), but thiswill require experimental conﬁrmation on real world data.

Original: To bridge the gap between theory and realworld applications, two questions of considerable interest are (1) an analysis of undercomplete ICA(i.e., f : R d → R d (cid:48) with d < d (cid:48) ) with restricted function classes which is completely open and (2)identiﬁability results when we only know that f is close to some function class F .

Original: Finally, a central question from a machine learning perspective is the ability to design learningalgorithms that can train LVMs with identiﬁable function class constraints.
Revised: Finally, a central question from a machine learning perspective is the ability to design learningalgorithms that can train LVMs with identiﬁable function class constraints.

Original: Interestingly, Gresele et al.[13] showed that OCT maps can be learnt using a closed from regularized likelihood loss, therebyproviding, supported by our result, a full-ﬂedged identiﬁable nonlinear ICA framework.
Revised: Interestingly, Gresele et al.[16] showed that OCT maps can be learnt using a closed from regularized likelihood loss, therebyproviding, supported by our result, a full-ﬂedged identiﬁable nonlinear ICA framework.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See discussion (Section 7) and
Revised: [Yes] See discussion (Section 7) and

Original: assumptions of our results
Revised: assumptions of our results

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: Due to
Revised: Due to

Original: the theoretical focus of this work no direct negative impacts are expected.
Revised: the theoretical focus of this work no direct negative impacts are expected.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] All results are
Revised: [Yes] All results are

Original: stated including all required assumptions
Revised: stated including all required assumptions

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] The proofs can befound in the supplementary material (Appendix C to Appendix F)
Revised: [Yes] The proofs can be found in the supplementary material (Appendix D to Appendix G)

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [N/A]
Revised: [No] Code will bemade available upon publication

Original: (b) Did you specify all the training details (e g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [N/A]
Revised: [Yes] See end of Appendix H

Original: (c) Did you report error bars (e
Revised: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experi-
Revised: g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [N/A]
Revised: [Yes] See Figure 7 and

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?

Original: [N/A]
Revised: [Yes] See end of Appendix H

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g.

Original: , code,

Original: data , models or curating /releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [N/A]
Revised: [N/A](b)

Original: (b) Did you mention the license of the assets?
Revised: Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A](c)

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: [Table] (d) Did you discuss whether and how consent was obtained from people whose data you’re [Table] using/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?

Original: [N/A]
Revised: [N/A]

Original: [Table] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable [Table] information or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁableinformation or offensive content?

Original: [N/A]
Revised: [N/A]

Original: [Table]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: [Table] (a) Did you include the full text of instructions given to participants and screenshots, if[Table]
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: [Table]
Revised: [Table]

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: [Table]
Revised: [Table]

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A] [Table]
Revised: [N/A] [Table]