Original: D ENSITY ESTIMATION ON LOW - DIMENSIONAL MANIFOLDS : AN INFLATION - DEFLATION APPROACH
Revised: D ENSITY ESTIMATION ON LOW - DIMENSIONAL MANIFOLDS : AN INFLATION - DEFLATION APPROACH

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Normalizing Flows (NFs) are universal density estimators based on Neuronal Networks.
Revised: Normalizing Flows (NFs) are universal density estimators based on Neuronal Networks.

Original: However, this universality is limited: the density’s support needs to be diffeomorphic to a Euclidean space.
Revised: However, this universality is limited: the density’s support needs to be diffeomorphic to a Euclidean space.

Original: In this paper, we propose a novel method to overcome this limitation without sacriﬁcing the universality.
Revised: In this paper, we propose a novel method to overcome this limitation without sacriﬁcing the universality.

Original: The proposed method inﬂates the data manifold by adding noise in the normal space, trains a
Revised: The proposed method inﬂates the data manifold by adding noise in the normal space, trains an

Original: NF on this inﬂated manifold and, ﬁnally, deﬂates the learned density.
Revised: NF on this inﬂated manifold and, ﬁnally, deﬂates the learned density.

Original: Our main result provides sufﬁcient conditions on the manifold and the speciﬁc choice of noise under which the corresponding estimator is exact.
Revised: Our main result provides sufﬁcient conditions on the manifold and the speciﬁc choice of noise under which the corresponding estimator is exact.

Original: Our method has the same computational complexity as NFs, and does not require to compute an inverse.
Revised: Our method has the same computational complexity as NFs, and does not require to compute an inverse ﬂow.

Original: We also show that, if the embedding dimension is much larger than the manifold dimension, noise in the normal space can be well approximated by some Gaussian noise.
Revised: We also show that, if the embedding dimension is much larger than the manifold dimension, noise in the normal space can be well approximated by some Gaussian noise.

Original: This allows using our method for approximating arbitrary densities on non-ﬂat manifolds provided that the manifold dimension is known.
Revised: This allows using our method for approximating arbitrary densities on non-ﬂat manifolds provided that the manifold dimension is known.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Many modern problems involving high dimensional data are formulated probabilistically.
Revised: Many modern problems involving high dimensional data are formulated probabilistically.

Original: Key concepts, such as Bayesian Classiﬁcation, Denoising or Anomaly Detection, make use of the data generating density p ∗ ( x ) .
Revised: Key concepts, such as Bayesian Classiﬁcation, Denoising or Anomaly Detection, rely on the data generating density p ∗ ( x ) .

Original: Therefore, a main research area and of crucial importance is learning this data generating density p ∗ ( x ) from samples.
Revised: Therefore, a main research area and of crucial importance is learning this data generating density p ∗ ( x ) from samples.

Original: For the case where the corresponding random variable X ∈ R D takes values on a manifold diffeomorphic to R D , a Normalizing Flow (NF) can be used to learn p ∗ ( x ) exactly (Huang et al., 2018).
Revised: For the case where the corresponding random variable X ∈ R D takes values on a manifold diffeomorphic to R D , a Normalizing Flow (NF) can be used to learn p ∗ ( x ) exactly (Huang et al., 2018).

Original: Recently, a few attempts have been made to overcome this topological constraint.
Revised: Recently, a few attempts have been made to overcome this topological constraint.

Original: However, to do so, all of these methods either need to know the manifold beforehand (Gemici et al.
Revised: However, to do so, all of these methods either need to know the manifold beforehand (Gemici et al.

Original: (2016), Rezende et al.
Revised: (2016), Rezende et al.

Original: or they sacriﬁce the exactness of the estimate (Cornish et al.
Revised: or they sacriﬁce the exactness of the estimate (Cornish et al.

Original: (2019), Dupont et al.
Revised: (2019), Dupont et al.

Original: Our goal in this paper is to overcome both the aforementioned limitations of using NFs for density estimation on Riemannian manifolds.
Revised: Our goal in this paper is to overcome both the aforementioned limitations of using NFs for density estimation on Riemannian manifolds.

Original: Given data points from a d − dimensional Riemannian manifold embedded in R D , d < D , we ﬁrst inﬂate the manifold by adding a speciﬁc noise in the normal space direction of the manifold, then train a NF on this inﬂated manifold, and, ﬁnally, deﬂate the trained density by exploiting the choice of noise and the geometry of the manifold.
Revised: Given data points from a d − dimensional Riemannian manifold embedded in R D , d < D , we ﬁrst inﬂate the manifold by adding a speciﬁc noise in the normal space direction of the manifold, then train an NF on this inﬂated manifold, and, ﬁnally, deﬂate the trained density by exploiting the choice of noise and the geometry of the manifold.

Original: See Figure 1 for a schematic overview of these points.
Revised: See Figure 1 for a schematic overview of these points.

Original: Our main theorem states sufﬁcient conditions on the manifold and the type of noise we use for the inﬂation step such that the deﬂation becomes exact.
Revised: Our main theorem states sufﬁcient conditions on the manifold and the type of noise we use for the inﬂation step such that the deﬂation becomes exact.

Original: To guarantee the exactness, we do need to know the manifold as in e.g. Rezende et al.
Revised: To guarantee the exactness, we do need to know the manifold as in e.g. Rezende et al.

Original: (2020) because we need to be able to sample in the manifold’s normal space.
Revised: (2020) because we need to be able to sample in the manifold’s normal space.

Original: However, as we will show, for the special case where D (cid:29) d , the usual Gaussian noise is an excellent approximation for a noise in the normal space component.
Revised: However, as we will show, for the special case where D (cid:29) d , the usual Gaussian noise is an excellent approximation for a noise in the normal space component.

Original: This allows using our method for approximating arbitrary densities on Riemannian manifolds provided that the manifold dimension is known.
Revised: This allows using our method for approximating arbitrary densities on Riemannian manifolds provided that the manifold dimension is known.

Original: In addition, our method is based on a single NF without the necessity to invert it.
Revised: In addition, our method is based on a single NF without the necessity to invert it.

Original: Hence, we don’t add any additional complexity to the usual training procedure of NFs.
Revised: Hence, we don’t add any additional complexity to the usual training procedure of NFs.

Original: Notations: We denote the determinant of the Gram matrix of f as g f ( x ) := | det (cid:0) J f ( x ) T J f ( x ) (cid:1) | where J f ( x ) is the Jacobian of f .
Revised: Notations: We denote the determinant of the Gram matrix of f as g f ( x ) := | det (cid:0) J f ( x ) T J f ( x ) (cid:1) | where J f ( x ) is the Jacobian of f .

Original: We denote the Lebesque measure in R n as λ n .
Revised: We denote the Lebesque measure in R n as λ n .

Original: Random variables will be denoted with a capital letter, say X , and its corresponding state space with the calligraphical
Revised: Random variables will be denoted with a capital letter, say X , and its corresponding state space with the calligraphical

Original: ←− X ⊂ R d 2 F − 1 θ
Revised: ←− X ⊂ R d 2 F − 1 θ

Original: R D x ∼ p ∗ ( x ) x ˜ x ∼ q (˜ x ) + σ
Revised: R D x ∼ p ∗ ( x ) x ˜ x ∼ q (˜ x ) + σ

Original: R D ←− ˜ X ⊂ R D
Revised: R D ←− ˜ X ⊂ R D

Original: R D = U
Revised: R D = U

Original: ˜ x u = F − 1 θ (˜ x ) − σ
Revised: ˜ x u = F − 1 θ (˜ x ) − σ

Original: Theorem←− X ⊂ R d
Revised: Theorem←− X ⊂ R d

Original: R D x ∼ ˆ p ( x ) x   u ∼ p
Revised: R D x ∼ ˆ p ( x ) x   u ∼ p

Original: U ( u ) version, i.e. X .
Revised: U ( u ) version, i.e. X .

Original: Small letters correspond to vectors with dimensionality given by context.
Revised: Small letters correspond to vectors with dimensionality given by context.

Original: The letters d, D, n , and N are always natural numbers.
Revised: The letters d, D, n , and N are always natural numbers.

Original:  B ACKGROUND AND PROBLEM STATEMENT
Revised:  B ACKGROUND AND PROBLEM STATEMENT

Original: A NF transforms a known auxiliary random variable by using bijective mappings parametrized by Neuronal Networks such that the given data points are samples from this transformed random variable, see Papamakarios et al.
Revised: An NF transforms a known auxiliary random variable by using bijective mappings parametrized by Neuronal Networks such that the given data points are samples from this transformed random variable, see Papamakarios et al.

Original: Formally, a NF is a diffeomorphism F θ : U → X and induces a density on X through p θ ( x ) =
Revised: Formally, an NF is a diffeomorphism F θ : U → X and induces a density on X through p θ ( x ) =

Original: ( g F ( u ) − 12 p U ( u ) where p U ( u ) is known and u = F − 1 θ ( x ) .
Revised: ( g F ( u ) − 12 p U ( u ) where p U ( u ) is known and u = F − 1 θ ( x ) .

Original: The parameters θ are updated such that the KL-divergence between p ∗ ( x ) and p θ ( x ) ,
Revised: The parameters θ are updated such that the KL-divergence between p ∗ ( x ) and p θ ( x ) ,

Original: [Equation]
Revised: [Equation]

Original: 1) is minimized.
Revised: is minimized.

Original: If F θ is expressive enough, it was proven that in the limit of inﬁnitely many samples, updating θ to minimize this objective function converges to a θ ∗ such that it holds P X -almost
Revised: If F θ is expressive enough, it was proven that in the limit of inﬁnitely many samples, updating θ to minimize this objective function converges to a θ ∗ such that it holds P X -almost

Original: surely p ∗ ( x ) =
Revised: surely p ∗ ( x ) =

Original: p θ ∗ ( x ) , see (Huang et al., 2018).
Revised: p θ ∗ ( x ) , see (Huang et al., 2018).

Original: More generally, let X ∈ X
Revised: More generally, let X ∈ X

Original: ⊂ R D be generated by an unobserved random variable Z ∈ Z ⊂ R d with density π ( z ) , that is X
Revised: ⊂ R D be generated by an unobserved random variable Z ∈ Z ⊂ R d with density π ( z ) , that is X

Original: = f ( Z ) for some function f : Z → X where typically d < D .
Revised: = f ( Z ) for some function f : Z → X where typically d < D .

Original: In
Revised: In

Original: Gemici et al.
Revised: Gemici et al.

Original: (2016), f is an embedding 1 , and it was shown that one can calculate probabilities such as P X ( A ) for measurable A ⊂ X using a density p ∗ ( x ) with respect to the volume form dV f induced by f , that is
Revised: (2016), f is an embedding 1 , and it was shown that one can calculate probabilities such as P X ( A ) for measurable A ⊂ X using a density p ∗ ( x ) with respect to the volume form dV f induced by f , that is

Original: [Equation]
Revised: [Equation]

Original: with p ∗ ( x ) =
Revised: with p ∗ ( x ) =

Original: π ( z ) g f ( z ) − 12 and dV f ( x ) =
Revised: π ( z ) g f ( z ) − 12 and dV f ( x ) =

Original: (cid:112) g f ( z ) dz where z
Revised: (cid:112) g f ( z ) dz where z

Original: = f − 1 ( x ) .
Revised: = f − 1 ( x ) .

Original: Hence, given an explicit mapping f and samples from p ∗ ( x ) , we can learn the unknown density π ( z ) using a usual NF in R d .
Revised: Hence, given an explicit mapping f and samples from p ∗ ( x ) , we can learn the unknown density π ( z ) using a usual NF in R d .

Original: However, in general, the generating function f is either unknown or not an embedding creating numerical instabilities for training inputs close to singularity points.
Revised: However, in general, the generating function f is either unknown or not an embedding creating numerical instabilities for training inputs close to singularity points.

Original: In Brehmer & Cranmer (2020), f and the unknown density π are learned simultaneously.
Revised: In Brehmer & Cranmer (2020), f and the unknown density π are learned simultaneously.

Original: The main idea is to deﬁne f as a level set of a usual ﬂow in R D and train it together with the ﬂow in R d used to learn π ( z ) .
Revised: The main idea is to deﬁne f as a level set of a usual ﬂow in R D and train it together with the ﬂow in R d used to learn π ( z ) .

Original: To evaluate the density, one needs to invert f and thus this approach may be very slow for high-dimensional data.
Revised: To evaluate the density, one needs to invert f and thus this approach may be very slow for high-dimensional data.

Original: Besides, to guarantee that f learns the manifold they proposed several ad hoc training strategies.
Revised: Besides, to guarantee that f learns the manifold they proposed several ad hoc training strategies.

Original: We tie in with the idea to use a NF for learning p ∗ ( x ) with unknown f and study the following problem.[Figure]
Revised: We tie in with the idea to use an NF for learning p ∗ ( x ) with unknown f and study the following problem.

Original: Thus, a regular continuously differentiable mapping (called immersion) which is, restricted to its image, ahomeomorphism.
Revised: regular continuously differentiable mapping (called immersion) which is, restricted to its image a

Original: Problem 1 Let X be a d − dimensional manifold embedded in R D .
Revised: Problem 1 Let X be a d − dimensional manifold embedded in R D .

Original: Let X = f
Revised: Let X = f

Original: ( Z ) be a random variable generated by an embedding f : R d → R D and a random variable Z ∼ π ( z ) in R d .
Revised: ( Z ) be a random variable generated by an embedding f : R d → R D and a random variable Z ∼ π ( z ) in R d .

Original: Given
Revised: Given

Original: N samples from p ∗ ( x ) as described above, ﬁnd an estimator ˆ p of p ∗ such that in the limit of inﬁnitely many samples we have that ˆ p ( x ) =
Revised: N samples from p ∗ ( x ) as described above, ﬁnd an estimator ˆ p of p ∗ such that in the limit of inﬁnitely many samples we have that ˆ p ( x ) =

Original: p ∗ ( x ) , P X − almost surely.
Revised: p ∗ ( x ) , P X − almost surely.

Original:  M ETHODS
Revised:  M ETHODS

Original: To solve Problem 1, we want to exploit the universality of NFs.
Revised: To solve Problem 1, we want to exploit the universality of NFs.

Original: We want to inﬂate X such that the inﬂated manifold ˜ X becomes diffeomorphic to a set U on which a simple density exists.
Revised: We want to inﬂate X such that the inﬂated manifold ˜ X becomes diffeomorphic to a set U on which a simple density exists.

Original: Doing so, allows us to learn the inﬂated density p (˜ x ) exactly using a single NF, see Section 2.
Revised: Doing so, allows us to learn the inﬂated density p (˜ x ) exactly using a single NF, see Section 2.

Original: Then, given such an estimator for the modiﬁed density, we approximate p ∗ ( x ) and give sufﬁcient conditions when this approximation is exact.
Revised: Then, given such an estimator for the modiﬁed density, we approximate p ∗ ( x ) and give sufﬁcient conditions when this approximation is exact.

Original: 3. T HE I NFLATION STEP
Revised: 3. T HE I NFLATION STEP

Original: Given a sample x of X , if we add some noise ε ∈ R D to it, the resulting new random variable ˜
Revised: Given a sample x of X , if we add some noise ε ∈ R D to it, the resulting new random variable ˜

Original: X = X
Revised: X = X

Original: + ε has the following density
Revised: + ε has the following density

Original: [Equation]
Revised: [Equation]

Original: Denote the tangent space in x as T x and the normal space as N x .
Revised: Denote the tangent space in x as T x and the normal space as N x .

Original: By deﬁnition, N x is the orthogonal complement of T x .
Revised: By deﬁnition, N x is the orthogonal complement of T x .

Original: Therefore, we can decompose the noise ε into its tangent and normal component, ε = ε t + ε n .
Revised: Therefore, we can decompose the noise ε into its tangent and normal component, ε = ε t + ε n .

Original: In the following, we consider noise in the normal space only, i.e. ε t = 0 , and denote the density of the resulting random variable as q n (˜ x ) .
Revised: In the following, we consider noise in the normal space only, i.e. ε t = 0 , and denote the density of the resulting random variable as q n (˜ x ) .

Original: The corresponding noise density q n (˜ x | x ) has mean x and domain N x .
Revised: The corresponding noise density q n (˜ x | x ) has mean x and domain N x .

Original: We denote the support of q n ( ·| x ) by N q n ( ·| x ) .
Revised: We denote the support of q n ( ·| x ) by N q n ( ·| x ) .

Original: The random variable ˜ X
Revised: The random variable ˜ X

Original: = X + ε n is now deﬁned on (cid:101) X
Revised: = X + ε n is now deﬁned on (cid:101) X

Original: = (cid:83) x ∈X N q n ( ·| x ) .
Revised: = (cid:83) x ∈X N q n ( ·| x ) .

Original: We want (cid:101) X to be diffeomorphic to a set U on which a known density can be deﬁned.
Revised: We want (cid:101) X to be diffeomorphic to a set U on which a known density can be deﬁned.

Original: Example 1 (a) Let X
Revised: Example 1 (a) Let X

Original: = S 1 = { x ∈ R 2 |
Revised: = S 1 = { x ∈ R 2 |

Original: || x
Revised: || x

Original: || = 1 } be the unit circle.
Revised: || = 1 } be the unit circle.

Original: For each x ∈ S 1 there exists z ∈ [0 , 2 π ) such that x = e r ( x ) =
Revised: For each x ∈ S 1 there exists z ∈ [0 , 2 π ) such that x = e r ( x ) =

Original: (cos( z ) , sin( z )) T .
Revised: (cos( z ) , sin( z )) T .

Original: To sample a point ˜ x in N x , which is spanned by e r ( x ) , we sample a scalar value γ and set ˜ x
Revised: To sample a point ˜ x in N x , which is spanned by e r ( x ) , we sample a scalar value γ and set ˜ x

Original: = x + γe r ( x ) .
Revised: = x + γe r ( x ) .

Original: With γ ∼
Revised: With γ ∼

Original: Uniform [ − 1 , 1) , we have that
Revised: Uniform [ − 1 , 1) , we have that

Original: [Equation]
Revised: [Equation]

Original: which is the open disk with radius 2 .
Revised: which is the open disk with radius 2 .

Original: The open disk is diffeomorphic to (0 , 1) × (0 , 1) .
Revised: The open disk is diffeomorphic to (0 , 1) × (0 , 1) .

Original: Thus, q n (˜ x ) can be learned by a single NF F − 1 and p U ( u ) =
Revised: Thus, q n (˜ x ) can be learned by a single NF F − 1 and p U ( u ) =

Original: Uniform ((0 , 1) × (0 , 1)) as reference.
Revised: Uniform ((0 , 1) × (0 , 1)) as reference.

Original: (b) As in (a), we consider the unit circle.
Revised: (b) As in (a), we consider the unit circle.

Original: Now we set γ to be a χ 2 − distribution with support [ − 1 , ∞ ) .
Revised: Now we set γ to be a χ 2 − distribution with support [ − 1 , ∞ ) .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Thus, q n (˜ x ) can be learned by a single NF F − 1 and p U ( u ) =
Revised: Thus, q n (˜ x ) can be learned by a single NF F − 1 and p U ( u ) =

Original: N ( u ; 0 , I D ) as reference.
Revised: N ( u ; 0 , I D ) as reference.

Original: Both cases can be analogously extended to higher dimensions.
Revised: Both cases can be analogously extended to higher dimensions.

Original: Remark 1 To be precise, the random variable ε n is generated by a random variable in R D − d , say Γ , with measure P Γ .
Revised: Remark 1 To be precise, the random variable ε n is generated by a random variable in R D − d , say Γ , with measure P Γ .

Original: Then, q n (˜ x | x ) is the density of the pushforward of the noise measure P Γ with regard to the mapping h : R D − d → N x .
Revised: Then, q n (˜ x | x ) is the density of the pushforward of the noise measure P Γ with regard to the mapping h : R D − d → N x .

Original: Hence, formally, the density q (˜ x | x ) is with respect to the induced volume form dV h , see Section 2.
Revised: Hence, formally, the density q (˜ x | x ) is with respect to the induced volume form dV h , see Section 2.

Original: However, if we choose an orthonormal basis for N x , say n (1) , . .
Revised: However, if we choose an orthonormal basis for N x , say n (1) , . .

Original: . , n ( D − d ) , then we have that ˜ x
Revised: . , n ( D − d ) , then we have that ˜ x

Original: = h ( γ ) =
Revised: = h ( γ ) =

Original: Aγ + x where the columns of A ∈ R D × ( D − d ) are given by these basis vectors, i.e. A = [ n (1) , . . . , n ( D − d ) ] .
Revised: Aγ + x where the columns of A ∈ R D × ( D − d ) are given by these basis vectors, i.e. A = [ n (1) , . . . , n ( D − d ) ] .

Original: Thus, the Gram determinant of h is g h = det( A T A )
Revised: Thus, the Gram determinant of h is g h = det( A T A )

Original: = 

Revised: =

Original: and we have that dV h (˜ x ) =
Revised: and we have that dV h (˜ x ) =

Original: dγ where dγ denotes the volume form with respect to λ D − d .
Revised: dγ where dγ denotes the volume form with respect to λ D − d .

Original: In this case, we can think of q n (˜ x | x ) as a density with respect to λ D − d .
Revised: In this case, we can think of q n (˜ x | x ) as a density with respect to λ D − d .

Original: If q n (˜ x | x ) depends only on || ˜ x − x || , as it is for the Gaussian distribution, we have that q n (˜ x | x ) =
Revised: If q n (˜ x | x ) depends only on || ˜ x − x || , as it is for the Gaussian distribution, we have that q n (˜ x | x ) =

Original: q n ( || ˜ x − x || )
Revised: q n ( || ˜ x − x || )

Original: =
Revised: =

Original: q n ( || γ || ) because h is an isomorphism.
Revised: q n ( || γ || ) because h is an isomorphism.

Original: Thus, for this case it holds that q n (˜ x | x ) dV h
Revised: Thus, for this case it holds that q n (˜ x | x ) dV h

Original: (˜ x ) =
Revised: (˜ x ) =

Original: q n ( || γ || ) dγ .
Revised: q n ( || γ || ) dγ .

Original: Then, for convenience, we may abuse notation by writing γ ∼ q (˜ x | x ) or ε n ∼ r ( γ ) where r ( γ ) is the density of P Γ with respect to λ D − d .
Revised: Then, for convenience, we may abuse notation by writing γ ∼ q (˜ x | x ) or ε n ∼ r ( γ ) where r ( γ ) is the density of P Γ with respect to λ D − d .

Original: 3. T HE D EFLATION STEP
Revised: 3. T HE D EFLATION STEP

Original: Our main idea is to ﬁnd conditions such that
Revised: Our main idea is to ﬁnd conditions such that

Original: [Equation]
Revised: [Equation]

Original: for almost surely all ˜ x ∈ ˜ X and for an almost surely unique x ∈ X .
Revised: for almost surely all ˜ x ∈ ˜ X and for an almost surely unique x ∈ X .

Original: Because then, given an exact estimator of q n (˜ x ) , say ˆ q n (˜ x ) , we have for ˜ x
Revised: Because then, given an exact estimator of q n (˜ x ) , say ˆ q n (˜ x ) , we have for ˜ x

Original: = x that p ∗ ( x ) =
Revised: = x that p ∗ ( x ) =

Original: ˆ q n ( x ) /q
Revised: ˆ q n ( x ) /q

Original: n ( x | x ) .
Revised: n ( x | x ) .

Original: For equation (6) to be true, we need to guarantee that almost every ˜ x corresponds to only one x ∈ X .
Revised: For equation (6) to be true, we need to guarantee that almost every ˜ x corresponds to only one x ∈ X .

Original: This is certainly the case whenever all the normal spaces have no intersections at all (think of a simple line in R 2 ).
Revised: This is certainly the case whenever all the normal spaces have no intersections at all (think of a simple line in R 2 ).

Original: We can relax this assumption by allowing null-set intersections.
Revised: We can relax this assumption by allowing null-set intersections.

Original: Moreover, only those subsets of the normal spaces are of interest which are generated by the speciﬁc choice of noise q n (˜ x | x ) .
Revised: Moreover, only those subsets of the normal spaces are of interest which are generated by the speciﬁc choice of noise q n (˜ x | x ) .

Original: Thus, only the support of q n (˜ x | x ) , N q n ( ·| x ) , matters.
Revised: Thus, only the support of q n (˜ x | x ) , N q n ( ·| x ) , matters.

Original: The key concept for our main result is expressed in the following deﬁnition:
Revised: The key concept for our main result is expressed in the following deﬁnition:

Original: Deﬁnition 1 Let X be a d − dimensional manifold and N x the normal space in x ∈ X .
Revised: Deﬁnition 1 Let X be a d − dimensional manifold and N x the normal space in x ∈ X .

Original: Let q n ( ·| x ) be a density deﬁned on N x and denote by N q n ( ·| x ) the domain of q n ( ·| x ) .
Revised: Let q n ( ·| x ) be a density deﬁned on N x and denote by N q n ( ·| x ) the domain of q n ( ·| x ) .

Original: Denote the collection of all such densities as Q := { q n ( ·| x ) } x ∈X .
Revised: Denote the collection of all such densities as Q := { q n ( ·| x ) } x ∈X .

Original: For ˜ x ∈ (cid:101) X , we deﬁne the set of all possible generators of ˜ x as A (˜ x ) =
Revised: For ˜ x ∈ (cid:101) X , we deﬁne the set of all possible generators of ˜ x as A (˜ x ) =

Original: { x (cid:48) ∈ X| N q n ( ·| x (cid:48) )
Revised: { x (cid:48) ∈ X| N q n ( ·| x (cid:48) )

Original: (cid:51)
Revised: (cid:51)

Original: ˜ x } .
Revised: ˜ x } .

Original: We say X is Q − normally separated if for all x ∈ X holds that P ˜ X | X = x
Revised: We say X is Q − normally separated if for all x ∈ X holds that P ˜ X | X = x

Original: [˜ x ∈ N x | # A (˜ x ) > 1] = 0 where # A (˜ x ) is the cardinality of the set A (˜ x ) .
Revised: [˜ x ∈ N x | # A (˜ x ) > 1] = 0 where # A (˜ x ) is the cardinality of the set A (˜ x ) .

Original: In words, every ˜ x ∈ N x is P ˜ X
Revised: In words, every ˜ x ∈ N x is P ˜ X

Original: | X = x -almost surely determined by x .
Revised: | X = x -almost surely determined by x .

Original: To familiarize with this concept, consider Figure 2 and 
Revised: To familiarize with this concept, consider Figure 2 and the following example:

Original: Example 2 For the circle in example 1, we choose ε n to be uniformly distributed on the half-open interval [ − 1 , 1) .
Revised: Example 2 For the circle in example 1, we choose ε n to be uniformly distributed on the half-open interval [ − 1 , 1) .

Original: The point (0 , 0) T is contained in N q n ( ·| x ) for all x ∈ X and thus N q n ( ·| x (cid:48) )
Revised: The point (0 , 0) T is contained in N q n ( ·| x ) for all x ∈ X and thus N q n ( ·| x (cid:48) )

Original: ∩
Revised: ∩

Original: N q n ( ·| x ) = { (0 , 0) T } for all x (cid:54) =
Revised: N q n ( ·| x ) = { (0 , 0) T } for all x (cid:54) =

Original: x (cid:48) , seethe middle image in Figure 2.
Revised: x (cid:48) , see Figure 2 (middle).

Original: Hence, for any given ˜ x ∈ N x we have that A (˜ x ) =
Revised: Hence, for any given ˜ x ∈ N x we have that A (˜ x ) =

Original: (cid:26) X if ˜ x = (0 , 0) T , x else, and therefore # A (˜ x ) = (cid:26) ∞
Revised: (cid:26) X if ˜ x = (0 , 0) T , x else, and therefore # A (˜ x ) = (cid:26) ∞

Original: if ˜ x = (0 , 0) T , else.
Revised: if ˜ x = (0 , 0) T , else.

Original: Thus, P ˜ X | X = x (cid:104) ˜ x ∈ ˜ X|
Revised: Thus, P ˜ X | X = x (cid:104) ˜ x ∈ ˜ X|

Original: # A (˜ x ) > 1 (cid:105) =
Revised: # A (˜ x ) > 1 (cid:105) =

Original: P ˜ X | X
Revised: P ˜ X | X

Original: = x (cid:2) ˜ x
Revised: = x (cid:2) ˜ x

Original: = (0 , 0) T (cid:3)
Revised: = (0 , 0) T (cid:3)

Original: = 

Revised: =

Original: for all x ∈ X .
Revised: for all x ∈ X .

Original: What follows is that X is Q − normally seperated.
Revised: What follows is that X is Q − normally seperated.

Original: If we were to choose ε n to be uniformly distributed on [ − 1 . 5 , 1) (right image in Figure 2), the normal spaces  overlap and we  have that P ˜ X 
Revised: If we were to choose ε n to be uniformly distributed on [ − 1 . 5 , 1) , see Figure 2 (right), the normal spaces would overlap and we would have that P ˜ X | X

Original: | X = x (cid:104) 
Revised: = x (cid:104) ˜ x ∈ ˜

Original: ˜ x ∈ ˜ X| # A (˜ x ) > 1 (cid:105) > 0 .
Revised: X| # A (˜ x ) > 1 (cid:105) > 0 .

Original: In this case, X is not  Q − normally seperated.
Revised: In this case, X would not be Q − normally seperated.

Original: n Uniform [ 0.5,1)          n Uniform [ 1,1)          n Uniform [ 1.5,1)
Revised: n Uniform [ 0.5,1)          n Uniform [ 1,1)          n Uniform [ 1.5,1)

Original: Theorem 1 Let X be a d − dimensional manifold.
Revised: Theorem 1 Let X be a d − dimensional manifold.

Original: For each x ∈ X , let q n ( ·| x ) denote a distribution on the normal space of x .
Revised: For each x ∈ X , let q n ( ·| x ) denote a distribution on the normal space of x .

Original: Let X be Q − normally separated where Q :=
Revised: Let X be Q − normally separated where Q :=

Original: { q n ( ·| x ) } x ∈X .
Revised: { q n ( ·| x ) } x ∈X .

Original: Assume that we can learn the density q n (˜ x ) as deﬁned in equation (3), by using a single NF F − 1 , thus q n (˜ x ) =
Revised: Assume that we can learn the density q n (˜ x ) as deﬁned in equation (3), by using a single NF F − 1 , thus q n (˜ x ) =

Original: ( g F ( F − 1 (˜ x ))) − 12 p U ( F − 1 (˜ x )) for some known density p U .
Revised: ( g F ( F − 1 (˜ x ))) − 12 p U ( F − 1 (˜ x )) for some known density p U .

Original: Then, for P ˜ X − almost all ˜ x ∈ (cid:101) X holds that q n (˜ x ) =
Revised: Then, for P ˜ X − almost all ˜ x ∈ (cid:101) X holds that q n (˜ x ) =

Original: p ∗ ( x ) q n (˜ x | x ) . Evaluated at ˜ x
Revised: p ∗ ( x ) q n (˜ x | x ) , therefore this equation, when evaluated at ˜ x

Original: = x  yields
Revised: = x , yields

Original: [Equation]
Revised: [Equation]

Original: The proof can be found in Appendix A.1.
Revised: The proof can be found in Appendix A.1.

Original: 3. G AUSSIAN NOISE AS NORMAL NOISE AND THE CHOICE OF σ
Revised: 3. G AUSSIAN NOISE AS NORMAL NOISE AND THE CHOICE OF σ

Original: Our proposed method depends on three critical points.
Revised: Our proposed method depends on three critical points.

Original: First, we need to be able to sample in the normal space of X .
Revised: First, we need to be able to sample in the normal space of X .

Original: Second, we need to determine the magnitude and type of noise.
Revised: Second, we need to determine the magnitude and type of noise.

Original: Third, we need to make sure that the conditions of Theorem 1 are fulﬁlled.
Revised: Third, we need to make sure that the conditions of Theorem 1 are fulﬁlled.

Original: We address (partially) those three points.
Revised: We address (partially) those three points.

Original: For the special case where D (cid:29) d , we show that a full Gaussian noise is an excellent approximation for a Gaussian noise restricted to the normal space.
Revised: For the special case where D (cid:29) d , we show that a full Gaussian noise is an excellent approximation for a Gaussian noise restricted to the normal space.

Original: Consider ε = ε t + ε n , ε ∼ N (0 , σ 2 I D ) .
Revised: Consider ε = ε t + ε n , ε ∼ N (0 , σ 2 I D ) .

Original: Then, the expected absolute squared error when approximating normal noise with full Gaussian noise is E (cid:2) |ε
Revised: Then, the expected absolute squared error when approximating normal noise with full Gaussian noise is E (cid:2) |

Original: [Equation]
Revised: [Equation]

Original: | 2 (cid:3) = 
Revised: = E (cid:2) |

Original: E (cid:2)
Revised: [Equation]

Original: |
Revised: |

Original: [Equation]
Revised: [Equation]

Original: | 2 (cid:3) =
Revised: (cid:3) =

Original: dσ 2 .
Revised: dσ 2 .

Original: The expected relative squared error is therefore
Revised: The expected relative squared error is therefore

Original: [Equation]
Revised: [Equation]

Original: ) because ε t and ε n are independent and D − d | ε n | 2 follows a scaled inverse χ 2 − distribution with D − d degrees of freedom and scale parameter 1 /σ 2 .Thus
Revised: because ε t and ε n are independent and D − d | ε n | 2 follows a scaled inverse χ 2 − distribution with D − d degrees of freedom and scale parameter 1 /σ 2 .

Original: , if

Original: D

Original:  (cid:29) d , Gaussian noise is an excellent approximation for a Gaussian in the normal space.
Revised: Thus, if D (cid:29) d , Gaussian noise is an excellent approximation for a Gaussian in the normal space.

Original: We denote the inﬂated density with Gaussian noise by q σ 2 (˜ x ) in the following
Revised: We denote the inﬂated density with Gaussian noise by q σ 2 (˜ x ) in the following.

Original: The inﬂation must not garble the manifold too much.
Revised: The inﬂation must not garble the manifold too much.

Original: For instance, adding Gaussian noise with magnitude σ ≥ r to S 1 will blurthe circle.
Revised: For instance, adding Gaussian noise with magnitude σ ≥ r to S 1 will blur

Original:  Since the curvature of the circle is 1 /r , intuitively, we want σ to scale with the second derivative of the generating function f .
Revised: the circle Since the curvature of the circle is r , intuitively, we want σ to scale with the second derivative of the generating function f .

Original: Additionally, we do not want to lose the information of p ∗ ( x ) by inﬂating the manifold
Revised: Additionally, we do not want to lose the information of p ∗ ( x ) by inﬂating the manifold.

Original: If the generating distribution π ( z ) makes a sharp transition at z 0 , π ( z 0 − ∆ z o ) (cid:28) π ( z 0 + ∆ z o ) for | ∆ z o | (cid:28) 1 , adding to much noise in x 0 = f
Revised: If the generating distribution π ( z ) makes a sharp transition at z 0 , π ( z 0 − ∆ z o ) (cid:28) π ( z 0 + ∆ z o ) for | ∆ z o | (cid:28) 1 , adding to much noise in x 0 = f

Original: ( z 0 ) will smooth out that transition.
Revised: ( z 0 ) will smooth out that transition.

Original: Hence, we want σ to inversely scale with π (cid:48)(cid:48)
Revised: Hence, we want σ to inversely scale with π (cid:48)(cid:48)

Original: ( z ) .
Revised: ( z ) .

Original: We formalize these intuitions in Proposition 1 and prove them in Appendix A.2.
Revised: We formalize these intuitions in Proposition 1 and prove them in Appendix A.2.

Original: In accordance with Theorem 1, we say p σ (˜ x ) approximates well p ∗ ( x ) if lim σ 2 → 0 p σ ( x ) /q n ( x | x ) =
Revised: In accordance with Theorem 1, we say p σ (˜ x ) approximates well p ∗ ( x ) if lim σ 2 → 0 p σ ( x ) /q n ( x | x ) =

Original: p ∗ ( x ) for all x ∈ X where q n ( x | x ) is the normalization constant of a ( D − d ) − dimensional Gaussian distribution.
Revised: p ∗ ( x ) for all x ∈ X where q n ( x | x ) is the normalization constant of a ( D − d ) − dimensional Gaussian distribution.

Original: Proposition 1 Let X ∈ R D be generated by Z ∼ π ( z ) through an embedding f : R d → R D , i.e. f ( Z ) =
Revised: Proposition 1 Let X ∈ R D be generated by Z ∼ π ( z ) through an embedding f : R d → R D , i.e. f ( Z ) =

Original: X .
Revised: X .

Original: Let π ∈ C 2 ( R D ) .
Revised: Let π ∈ C 2 ( R d ) .

Original: For q σ 2 (˜ x ) to approximate well p ∗ ( x ) , in the sense that lim σ 2 → 0 q σ 2 ( x ) /q n ( x | x ) = p ∗ ( x ) for x ∈ X , a necessary condition is that:
Revised: For q σ 2 (˜ x ) to approximate well p ∗ ( x ) , in the sense that lim σ 2 → 0 q σ 2 ( x ) /q n ( x | x ) = p ∗ ( x ) for x ∈ X , a necessary condition is that:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where ||
Revised: where ||

Original: A ||
Revised: A ||

Original: + = (cid:80) di,j =1 A ij for A ∈ R d × R d and (cid:12) denotes the elementwise product, and ( π (cid:48)(cid:48) ( z 0 )) ij = ∂
Revised: + = (cid:80) di,j =1 A ij for A ∈ R d × R d and (cid:12) denotes the elementwise product, and ( π (cid:48)(cid:48) ( z 0 )) ij = ∂

Original: 2 π ( z ) ∂z
Revised: 2 π ( z ) ∂z

Original: i ∂z j | z
Revised: i ∂z j | z

Original: = z 0 is the Hessian of the prior evaluated at z 0 =
Revised: = z 0 is the Hessian of the prior evaluated at z 0 =

Original: f − 1 ( x ) .
Revised: f − 1 ( x ) .

Original: Intuitively, a second necessary condition is that the noise magnitude should be much smaller than the radius of the curvature of the manifold which directly depends on the second-order derivatives of f .
Revised: Intuitively, a second necessary condition is that the noise magnitude should be much smaller than the radius of the curvature of the manifold which directly depends on the second-order derivatives of f .

Original: This can be illustrated in the following example:
Revised: This can be illustrated in the following example:

Original: Example 3 For the circle 2 in R 2 generated by f ( z )
Revised: Example 3 For the circle 2 in R 2 generated by f ( z )

Original: =
Revised: =

Original: (cos( z ) , sin( z )) T and a von Mises distribution π ( z ) ∝ exp( κ cos( z )) , we get that σ 2 (cid:28) min (cid:16)(cid:12) (cid:12)(cid:12)
Revised: (cos( z ) , sin( z )) T and a von Mises distribution π ( z ) ∝ exp( κ cos( z )) , we get that σ 2 (cid:28) min (cid:16)(cid:12) (cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst condition comes from Proposition 1 and the second one comes from the curvature argument.
Revised: where the ﬁrst condition comes from Proposition 1 and the second one comes from the curvature argument.

Original: Even though this bound may not be usefull as such in practice when f and π are unknown, it can still be used if f and π are estimated locally with nearest neighbor statistics.
Revised: Even though this bound may not be usefull as such in practice when f and π are unknown, it can still be used if f and π are estimated locally with nearest neighbor statistics.

Original: From a numerical perspective, inﬂating a manifold using Gaussian noise circumvents degeneracy problems when training a vanilla NF for low-dimensional manifolds.
Revised: From a numerical perspective, inﬂating a manifold using Gaussian noise circumvents degeneracy problems when training an vanilla NF for low-dimensional manifolds.

Original: In particular, the ﬂows Jacobian determinant becomes numerically unstable, see equation (1).
Revised: In particular, the ﬂows Jacobian determinant becomes numerically unstable, see equation (1).

Original: This determinant is essentially a volume changing factor for balls.
Revised: This determinant is essentially a volume changing factor for balls.

Original: From a sampling perspective, these volumes can be estimated with the number of samples falling into the ball divided by the total number of points.
Revised: From a sampling perspective, these volumes can be estimated with the number of samples falling into the ball divided by the total number of points.

Original: Therefore, we suggest to lower bound σ with the average nearest neighbor obtained from the training set to make sure that these volumes are not empty and thus avoid numerical instabilities.
Revised: Therefore, we suggest to lower bound σ with the average nearest neighbor obtained from the training set to make sure that these volumes are not empty and thus avoid numerical instabilities.

Original: Intuitively, if the curvature of the manifold is not too high and if the manifold is not too entangled,
Revised: Intuitively, if the curvature of the manifold is not too high and if the manifold is not too entangled,

Original: Q − normal separability is satisﬁed for a sufﬁciently small magnitude of noise.
Revised: Q − normal separability is satisﬁed for a sufﬁciently small magnitude of noise.

Original: Also in the manifold learning literature, the entangling must not be too strong.
Revised: Also in the manifold learning literature, the entangling must not be too strong.

Original: Informally, the reach number provides a necessary condition on the manifold such that it is learnable through samples, see Chapter 2.3 in
Revised: Informally, the reach number provides a necessary condition on the manifold such that it is learnable through samples, see Chapter 2.3 in

Original: Berenfeld & Hoffmann (2019).
Revised: Berenfeld & Hoffmann (2019).

Original: Formally, the reach number is the maximum distance τ X such that for all ˜ x in a τ X − neighbourhood of X the projection onto X is unique.
Revised: Formally, the reach number is the maximum distance τ X such that for all ˜ x in a τ X − neighbourhood of X the projection onto X is unique.

Original: In Appendix A.3 we prove
Revised: In Appendix A.3 we prove

Original: Theorem 2 which states that any closed manifold X with τ X > 0 is Q − normally separable.
Revised: Theorem 2 which states that any closed manifold X with τ X > 0 is Q − normally separable.

Original: Theorem 2 Let X ⊂ R D be a closed d -dimensional manifold.
Revised: Theorem 2 Let X ⊂ R D be a closed d -dimensional manifold.

Original: If X has a positive reach number τ X , then X is Q − normally separable where Q := { q n ( ·| x ) } x ∈X is the collection of uniform distributions on a ball with radius τ X , i.e. q n (˜ x | x ) =1 /
Revised: If X has a positive reach number τ X , then X is Q − normally separable where Q := { q n ( ·| x ) } x ∈X is the collection of uniform distributions on a ball with radius τ X , i.e. q n (˜ x | x ) =

Original: Vol ( D − d, τ X )  where Vol ( D − d, τ X ) denotesthe volume of a( D −d ) − dimensional ball with radius τ X  .
Revised: Uniform (˜ x ; B ( x, τ X ) ∩ N x ) where B ( x, τ X ) denotes a D − dimensional ball with radius τ X and center x .

Original:  R ESULTS
Revised:  R ESULTS

Original: We have three goals in this section: First, we numerically conﬁrm the scaling factor in equation (7).
Revised: We have three goals in this section: First, we numerically conﬁrm the scaling factor in equation (7).

Original: Second, we verify that Gaussian noise can be used to approximate a Gaussian noise restricted to the normal space.
Revised: Second, we verify that Gaussian noise can be used to approximate a Gaussian noise restricted to the normal space.

Original: Third, we numerically test our bound for σ 2 derived in Section 3.3.
Revised: Third, we numerically test our bound for σ 2 derived in Section 3.3.

Original: Finally, we show that we can learn complicated distributions on S 2 without using explicit charts.
Revised: Finally, we show that we can learn complicated distributions on S 2 without using explicit charts.

Original: For training details, we refer to Appendix B.1 and B.2, respectively.
Revised: For training details, we refer to Appendix B.1 and B.2, respectively.

Original: 4. V ON M ISES ON A CIRCLE
Revised: 4. V ON M ISES ON A CIRCLE

Original: Let X be a circle with radius 3 and let π ( z ) ∝ exp(8 cos( z )) be a 1 D von Mises distribution.
Revised: Let X be a circle with radius 3 and let π ( z ) ∝ exp(8 cos( z )) be a 1 D von Mises distribution.

Original: Given z ∼ π ( z ) , we generate a point in X according to the mapping f ( z ) =
Revised: Given z ∼ π ( z ) , we generate a point in X according to the mapping f ( z ) =

Original: 3(cos( z ) , sin( z )) .
Revised: 3(cos( z ) , sin( z )) .

Original: We want to learn the induced density p ∗ ( x ) .
Revised: We want to learn the induced density p ∗ ( x ) .

Original: Note that 3 p ∗ ( x ) =
Revised: Note that 3 p ∗ ( x ) =

Original: π ( z ) since 1 / 3 is the volume form induced by f .
Revised: π ( z ) since 1 / 3 is the volume form induced by f .

Original: To benchmark our performance, we use the idea in Gemici et al.
Revised: To benchmark our performance, we use the idea in Gemici et al.

Original: (2016) to ﬁrst embed the circle into R , using e.g. f − 1 , learn the density there with a NF, and transform this learned density back to
Revised: (2016) to ﬁrst embed the circle into R , using e.g. f − 1 , learn the density there with an NF, and transform this learned density back to

Original: S 1 .
Revised: S 1 .

Original: In Brehmer & Cranmer (2020), this method is named Flow on manifolds (FOM) and we stick to this notation in the following.
Revised: In Brehmer & Cranmer (2020), this method is named Flow on manifolds (FOM) and we stick to this notation in the following.

Original: Note that f is not injective and to illustrate the beneﬁt of our method we choose the singularity point to be (1 , 0) T .
Revised: Note that f is not injective and to illustrate the beneﬁt of our method we choose the singularity point to be (1 , 0) T .

Original: By moving points close to (1 , 0) T slightly away from (1 , 0) T , we numerically ensure that f is an embedding.
Revised: By moving points close to (1 , 0) T slightly away from (1 , 0) T , we numerically ensure that f is an embedding.

Original: The Inﬂation step: We inﬂate
Revised: The Inﬂation step: We inﬂate

Original: X using 3 types of noise: Gaussian in the normal space (NG), Gaussian in the full ambient space (FG), and χ 2 -noise as described in examples 1(b) with scale parameter 3 .
Revised: X using 3 types of noise: Gaussian in the normal space (NG), Gaussian in the full ambient space (FG), and χ 2 -noise as described in examples 1(b) with scale parameter 3 .

Original: Technically, NG violates the Q − normal separability assumption.
Revised: Technically, NG violates the Q − normal separability assumption.

Original: However, if σis small and the scale parameter for the von Mises distribution is large enough, this is practically fulﬁlled.
Revised: However, if σis small and the scale parameter for the von Mises distribution is large enough, this is practically fulﬁlled.

Original: Learning q n (˜ x ) : For the NFs we use a Block Neural Autoregressive Flow (BNAF)
Revised: Learning q n (˜ x ) : For the NFs we use a Block Neural Autoregressive Flow (BNAF)

Original: (De Cao et al., 2019).
Revised: (De Cao et al., 2019).

Original: We use the same NF architecture and training procedures across all models.
Revised: We use the same NF architecture and training procedures across all models.

Original: Deﬂation
Revised: Deﬂation

Original: : Given an estimator for q n (˜ x ) , we use equation (7) to calculate p ∗ ( x ) .
Revised: : Given an estimator for q n (˜ x ) , we use equation (7) to calculate p ∗ ( x ) .

Original: For FG and NG, we have that q n ( x | x ) = 1 / √ 2 πσ 2 and for the normal χ 2 − noise is q n ( x | x ) = √ 3 e − 3 / 2 / ( √ 8Γ( 32 )) .
Revised: For FG and NG, we have that q n ( x | x ) = 1 / √ 2 πσ 2 and for the normal χ 2 − noise is q n ( x | x ) = √ 3 e − 3 / 2 / ( √ 8Γ( 32 )) .

Original: In Figure 3, we show the results for σ 2 = 0 .
Revised: In Figure 3, we show the results for σ 2 = 0 .

Original: 01 and σ 2 = 1 .
Revised: 01 and σ 2 = 1 .

Original: In the respective plot, the ﬁrst row shows training samples from the inﬂated distributions q σ 2 (˜ x ) (left), and q n (˜ x ) (middle), respectively.
Revised: In the respective plot, the ﬁrst row shows training samples from the inﬂated distributions q σ 2 (˜ x ) (left), and q n (˜ x ) (middle), respectively.

Original: We color code a sample ˜ x
Revised: We color code a sample ˜ x

Original: = x + ε according to p ∗ ( x ) to illustrate the impact of noise on the inﬂated density.
Revised: = x + ε according to p ∗ ( x ) to illustrate the impact of noise on the inﬂated density.

Original: Note that the FOM model (top right) does not need any inﬂation and therefore is trained on samples from p ∗ ( x ) only.
Revised: Note that the FOM model (top right) does not need any inﬂation and therefore is trained on samples from p ∗ ( x ) only.

Original: In the respective plot, the second row shows the learned density for the different models and compares it to the ground truth von Mises distribution π ( z ) depicted in black.
Revised: In the respective plot, the second row shows the learned density for the different models and compares it to the ground truth von Mises distribution π ( z ) depicted in black.

Original: As we can see, for σ 2 = 0 .
Revised: As we can see, for σ 2 = 0 .

Original: 01 all models perform very well, although the FOM model slightly fails to capture p ( z ) for z close to 0 which corresponds to the chosen singularity point.
Revised: 01 all models perform very well, although the FOM model slightly fails to capture p ( z ) for z close to 0 which corresponds to the chosen singularity point.

Original: For σ 2 = 1 , we see a signiﬁcant drop in the performance of the Gaussian model.
Revised: For σ 2 = 1 , we see a signiﬁcant drop in the performance of the Gaussian model.

Original: Although the manifold is signiﬁcantly disturbed, the normal noise model still learns the density almost perfectly 3 , so does the normal χ 2 − noise model, as predicted by Theorem
Revised: Although the manifold is signiﬁcantly disturbed, the normal noise model still learns the density almost perfectly 3 , so does the normal χ 2 − noise model, as predicted by Theorem

Original: [Figure]
Revised: [Figure]

Original: To measure the dependence of our method on the magnitude of noise we iterate this experiment for various values of σ 2 and estimate the Kolmogorov-Smirnov (KS) statistics.
Revised: To measure the dependence of our method on the magnitude of noise we iterate this experiment for various values of σ 2 and estimate the Kolmogorov-Smirnov (KS) statistics.

Original: The KS statistic is deﬁned as KS = sup x ∈X | F ( x ) − G ( x ) | , where F and G are the cumulative distribution functions associated with the probability densities p ( x ) and q ( x ) , respectively.
Revised: The KS statistic is deﬁned as KS = sup x ∈X | F ( x ) − G ( x ) | , where F and G are the cumulative distribution functions associated with the probability densities p ( x ) and q ( x ) , respectively.

Original: By deﬁnition, KS ∈ [0 , 1] and
Revised: By deﬁnition, KS ∈ [0 , 1] and

Original: KS
Revised: KS

Original: = 

Revised: =

Original: if and only if p ( x ) =
Revised: if and only if p ( x ) =

Original: q ( x ) for almost every x ∈ X .
Revised: q ( x ) for almost every x ∈ X .

Original: However, equation (6) is only valid if the conditions of Theorem 1 are fulﬁlled.
Revised: However, equation (6) is only valid if the conditions of Theorem 1 are fulﬁlled.

Original: There is no reason why using equation (7) for the full
Revised: There is no reason why using equation (7) for the full

Original: Gaussian noise would lead to a density on the manifold X .
Revised: Gaussian noise would lead to a density on the manifold X .

Original: The KS statistics is ill-posed in this case.
Revised: The KS statistics is ill-posed in this case.

Original: Nevertheless, we are interested in measuring the sensitivity to the noise, and thus consider the KS statistics as a relative performance measure.
Revised: Nevertheless, we are interested in measuring the sensitivity to the noise, and thus consider the KS statistics as a relative performance measure.

Original: In Figure 4, we display the KS values depending on different levels of noise, for the NG (blue) and FG (orange) noise compared with the ground truth von Mises distribution.
Revised: In Figure 4, we display the KS values depending on different levels of noise, for the NG (blue) and FG (orange) noise compared with the ground truth von Mises distribution.

Original: Also, we embed the circle into higher dimensions D = 5 , 10 , 15 , 20 and repeat this experiment.
Revised: Also, we embed the circle into higher dimensions D = 5 , 10 , 15 , 20 and repeat this experiment.

Original: The result for D =and D = 20 are shown in the ﬁrst row (left and right).
Revised: The result for D =and D = 20 are shown in the ﬁrst row (left and right).

Original: For D = 2 , we add the FOM model (which is independent of σ 2 ) horizontally.
Revised: For D = 2 , we add the FOM model (which is independent of σ 2 ) horizontally.

Original: Besides, we depict the lower and upper bound for σ 2 from Chapter 3.3 with dashed vertical lines.
Revised: Besides, we depict the lower and upper bound for σ 2 from Chapter 3.3 with dashed vertical lines.

Original: In the lower-left image, we show the optimal KS values obtained for both models depending on D .
Revised: In the lower-left image, we show the optimal KS values obtained for both models depending on D .

Original: The lower-right image shows the corresponding σ 2 for those optimal KS .
Revised: The lower-right image shows the corresponding σ 2 for those optimal KS .

Original: In bright, the optimal average σ 2 is shown whereas the dark regions are the minimum respectively maximum values for σ 2 such that we outperformed the FOM benchmark.
Revised: In bright, the optimal average σ 2 is shown whereas the dark regions are the minimum respectively maximum values for σ 2 such that we outperformed the FOM benchmark.

Original: We note that for both cases, the averaged optimal σ 2 is within the predicted bounds for σ 2 .
Revised: We note that for both cases, the averaged optimal σ 2 is within the predicted bounds for σ 2 .

Original: [Figure]
Revised: [Figure]

Original: Several aspects are remarkable.
Revised: Several aspects are remarkable.

Original: The ﬂat course of the KS vs. σ 2 plot is an indicator that the method is not very sensitive to noise and this does not change with the dimensionality of the embedding space.
Revised: The ﬂat course of the KS vs. σ 2 plot is an indicator that the method is not very sensitive to noise and this does not change with the dimensionality of the embedding space.

Original: Also, the optimal KS values do not change much depending on D and the NS and FG model approach each other, as predicted
Revised: Also, the optimal KS values do not change much depending on D and the NS and FG model approach each other, as predicted

Original: Interestingly, the onset for the increase in the KS value for the NS-noise is roughly 3 which is the radius of the circle.
Revised: Interestingly, the onset for the increase in the KS value for the NS-noise is roughly 3 which is the radius of the circle.

Original: For increasing σ 2 , ˜ X resembles more and more a double cone which is not diffeomorphic to R 2 and thus the NF used to train the inﬂated distribution may not be able to capture the density close to the circle’s center correctly.
Revised: For increasing σ 2 , ˜ X resembles more and more a double cone which is not diffeomorphic to R 2 and thus the NF used to train the inﬂated distribution may not be able to capture the density close to the circle’s center correctly.

Original: 4. M IXTURES ON S
Revised: 4. M IXTURES ON S

Original: We show that we can learn a complicated distribution, a mixture of 2-dimensional von Mises distributions on a sphere with radius 1, without using any knowledge about the manifold except for its intrinsic dimension.
Revised: We show that we can learn a complicated distribution, a mixture of 2-dimensional von Mises distributions on a sphere with radius 1, without using any knowledge about the manifold except for its intrinsic dimension.

Original: For certain magnitudes of σ 2 , we obtain similar estimates as the FOM benchmark as we can see in the direct comparison of the learned densities, see Figure 5 (top right), and the KS statistics (bottom right).
Revised: For certain magnitudes of σ 2 , we obtain similar estimates as the FOM benchmark as we can see in the direct comparison of the learned densities, see Figure 5 (top right), and the KS statistics (bottom right).

Original: As for the circle, the Gaussian restricted in the normal space allows for a greater range of noise magnitude without sacriﬁcing the quality of the estimate.
Revised: As for the circle, the Gaussian restricted in the normal space allows for a greater range of noise magnitude without sacriﬁcing the quality of the estimate.

Original: [Figure]
Revised: [Figure]

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: To overcome the limitations of NFs to learn a density p ∗ ( x ) deﬁned on a low-dimensional manifold, we proposed to embed the manifold into the ambient space such that it becomes diffeomorphic to R D , learn this inﬂated density using a NF, and, ﬁnally, deﬂate the inﬂated density according to Theorem 1.
Revised: To overcome the limitations of NFs to learn a density p ∗ ( x ) deﬁned on a low-dimensional manifold, we proposed to embed the manifold into the ambient space such that it becomes diffeomorphic to R D , learn this inﬂated density using an NF, and, ﬁnally, deﬂate the inﬂated density according to Theorem 1.

Original: There, we provided sufﬁcient conditions on the choice of inﬂation such that we can compute p ∗ ( x ) exactly.
Revised: There, we provided sufﬁcient conditions on the choice of inﬂation such that we can compute p ∗ ( x ) exactly.

Original: Notably, we don’t need to assume that p ∗ ( x ) is supported on a ﬂat manifold.
Revised: Notably, we don’t need to assume that p ∗ ( x ) is supported on a ﬂat manifold.

Original: Our method depends on some critical points which we addressed in Section 3.3.
Revised: Our method depends on some critical points which we addressed in Section 3.3.

Original: So far, the magnitude of noise σ 2 when using NFs on real-world data is somewhat chosen arbitrarily.
Revised: So far, the magnitude of noise σ 2 when using NFs on real-world data is somewhat chosen arbitrarily.

Original: As a ﬁrst step to overcome this arbitrariness, we derived an upper bound for σ 2 in Proposition 1 and established an interesting connection to the manifold learning literature in Theorem 2.
Revised: As a ﬁrst step to overcome this arbitrariness, we derived an upper bound for σ 2 in Proposition 1 and established an interesting connection to the manifold learning literature in Theorem 2.

Original: We hope that our theoretical results motivate some new research directions.
Revised: We hope that our theoretical results motivate some new research directions.

Original: Using full Gaussian noise to learn the inﬂated distribution smears information on p ∗ ( x ) , in particular if p ∗ ( x ) has many local extrema.
Revised: Using full Gaussian noise to learn the inﬂated distribution smears information on p ∗ ( x ) , in particular if p ∗ ( x ) has many local extrema.

Original: This loss of information may be especially impactful in out of distribution (OOD) detection or when it comes to adversarial robustness.
Revised: This loss of information may be especially impactful in out of distribution (OOD) detection or when it comes to adversarial robustness.

Original: Therefore, developing methods which allow to generate noise in the manifold’s normal space could improve the performance of NFs on such tasks.
Revised: Therefore, developing methods which allow to generate noise in the manifold’s normal space could improve the performance of NFs on such tasks.

Original: Another interesting direction is to exploit the product form of equation (6) and learn low-dimensional representations by forcing the NF to be noise insensitive in the ﬁrst d -components and noise sensitive in the remaining ones.
Revised: Another interesting direction is to exploit the product form of equation (6) and learn low-dimensional representations by forcing the NF to be noise insensitive in the ﬁrst d -components and noise sensitive in the remaining ones.

Original: Inverting the corresponding ﬂow allows to sample directly from the manifold which has the potential to improve the generative ability of NFs.
Revised: Inverting the corresponding ﬂow allows to sample directly from the manifold which has the potential to improve the generative ability of NFs.

Original: A A PPENDIX A. P ROOF OF T HEOREM
Revised: A A PPENDIX A. P ROOF OF T HEOREM

Original: We denote the probability measure of the random variable X as P X
Revised: We denote the probability measure of the random variable X as P X

Original: and it is deﬁned on ( X , B ( X ))
Revised: and it is deﬁned on ( X , B ( X ))

Original: where B ( X ) is the set of borel measure in R D intersected with X .
Revised: where B ( X ) is the set of borel measure in R D intersected with X .

Original: For a realisation of X , say x , we denote the probability measure of the shifted random variable x + ε n as P ˜ X | X = x
Revised: For a realisation of X , say x , we denote the probability measure of the shifted random variable x + ε n as P ˜ X | X = x

Original: and it is deﬁned on ( N x , B ( N x )) .
Revised: and it is deﬁned on ( N x , B ( N x )) .

Original: We extend both measures to ( R D , B ( R D )) by setting the probabilities to 0 whenever a set A ∈ B ( R D ) has no intersection with X or N x , respectively.
Revised: We extend both measures to ( R D , B ( R D )) by setting the probabilities to 0 whenever a set A ∈ B ( R D ) has no intersection with X or N x , respectively.

Original: For instance, that means for ˜ x ∈ N x that
Revised: For instance, that means for ˜ x ∈ N x that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where (˜ x, ˜ x + d ˜ x ) denotes an inﬁnitesimal volume element around ˜ x .
Revised: where (˜ x, ˜ x + d ˜ x ) denotes an inﬁnitesimal volume element around ˜ x .

Original: The mapping ( x, (cid:15) )
Revised: The mapping ( x, (cid:15) )

Original: (cid:55)→ x + (cid:15) is B ( R D )
Revised: (cid:55)→ x + (cid:15) is B ( R D )

Original: × B ( R D ) − measurable because R D is a topological vector space and B ( R D ) the Borel σ − algebra.
Revised: × B ( R D ) − measurable because R D is a topological vector space and B ( R D ) the Borel σ − algebra.

Original: What follows is that ˜ X
Revised: What follows is that ˜ X

Original: = X
Revised: = X

Original: + ε n is a random variable on ( R D , B ( R D )) and has the pushforward of P ( X,ε n ) with regard to the mapping ( x, ε n )
Revised: + ε n is a random variable on ( R D , B ( R D )) and has the pushforward of P ( X,ε n ) with regard to the mapping ( x, ε n )

Original: → x + ε n as probability measure where P ( X,ε n ) is the joint measure of X and ε n .
Revised: → x + ε n as probability measure where P ( X,ε n ) is the joint measure of X and ε n .

Original: Thus, for A ∈ B ( (cid:101) X ) , we have that
Revised: Thus, for A ∈ B ( (cid:101) X ) , we have that

Original: [Equation]
Revised: [Equation]

Original: Now let ˜ x ∈ N x for an x ∈ X .
Revised: Now let ˜ x ∈ N x for an x ∈ X .

Original: Since X is Q − normally separated, P ˜ X − almost all ˜ x are uniquely determined by ( x, (cid:15) )
Revised: Since X is Q − normally separated, P ˜ X − almost all ˜ x are uniquely determined by ( x, (cid:15) )

Original: such that ˜ x
Revised: such that ˜ x

Original: = x + (cid:15) .
Revised: = x + (cid:15) .

Original: Therefore, we have for P ˜ X − almost all ˜ x
Revised: Therefore, we have for P ˜ X − almost all ˜ x

Original: = x + (cid:15)
Revised: = x + (cid:15)

Original: that
Revised: that

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: X
Revised: X

Original: [Equation]
Revised: [Equation]

Original: where for the ﬁrst equality we used equation (11) and for the third the fact that x and (cid:15) are almost surely uniquely determined by ˜ x .
Revised: where for the ﬁrst equality we used equation (11) and for the third the fact that x and (cid:15) are almost surely uniquely determined by ˜ x .

Original: Both probability measures on the right-hand side have a density.
Revised: Both probability measures on the right-hand side have a density.

Original: For P X with respect to dV f , see
Revised: For P X with respect to dV f , see

Original: Section 2, this density is p ∗ ( x ) .
Revised: Section 2, this density is p ∗ ( x ) .

Original: Similarly, since N x is a linear subspace of R D , q n (˜ x | x ) is the density of P ˜ X | X
Revised: Similarly, since N x is a linear subspace of R D , q n (˜ x | x ) is the density of P ˜ X | X

Original: = x with respect to the volume form dV h where h is the mapping from R D − d to N x , see Remark 1.
Revised: = x with respect to the volume form dV h where h is the mapping from R D − d to N x , see Remark 1.

Original: Then, the corresponding density of P ˜ X is with respect to the product measure dV h · dV f .
Revised: Then, the corresponding density of P ˜ X is with respect to the product measure dV h · dV f .

Original: However, this product measure is equivalent to λ D when restricted to subsets of (cid:101) X and thus q n is the density of P ˜ X with respect to λ D . 

Revised: However, this product measure is equivalent to λ D when restricted to subsets of (cid:101) X and thus q n is the density of P ˜ X with respect to λ D .

Original: Therefore, we can write equation (12) in terms of densities as follows:
Revised: Therefore, we can write equation (12) in terms of densities as follows:

Original: [Equation]
Revised: [Equation]

Original: and it holds that
Revised: and it holds that

Original: [Equation]
Revised: [Equation]

Original: as needed for a density on (cid:101) X .
Revised: as needed for a density on (cid:101) X .

Original: By setting ˜ x to x in equation (13), we obtain equation (7).
Revised: By setting ˜ x to x in equation (13), we obtain equation (7).

Original: This ends the proof.
Revised: This ends the proof.

Original: Remark 2 Note that in Theorem 1, we need that ˜ X is diffeomorphic to R D .
Revised: Remark 2 Note that in Theorem 1, we need that ˜ X is diffeomorphic to R D .

Original: This requires that the noise distribution q n ( ·| x ) is continuous for all xand that x ∈ N q n ( ·| x ) .
Revised: This requires that the noise distribution q n ( ·| x ) is continuous for all x .

Original: A. P ROOF OF P ROPOSITION
Revised: A. P ROOF OF P ROPOSITION

Original: The generating function f is an embedding for X and X = f
Revised: The generating function f is an embedding for X and X = f

Original: ( Z ) has the density p ∗ ( x ) for x ∈ X .
Revised: ( Z ) has the density p ∗ ( x ) for x ∈ X .

Original: We may extend the domain of p ∗ ( x ) to include all points x ∈ R D using the Dirac-delta function as follows
Revised: We may extend the domain of p ∗ ( x ) to include all points x ∈ R D using the Dirac-delta function as follows

Original: [Equation]
Revised: [Equation]

Original: After inﬂating X , we have that
Revised: After inﬂating X , we have that

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: (16) with covariance matrix Σ ∈ R D × D .
Revised: with covariance matrix Σ ∈ R D × D .

Original: Assume ˜
Revised: Assume ˜

Original: x
Revised: x

Original: = x for some x ∈ X .
Revised: = x for some x ∈ X .

Original: We Taylor expand f ( x ) around z 

Revised: We Taylor expand f ( x ) around z

Original: = f − 1 ( x ) up to ﬁrst order,
Revised: = f − 1 ( x ) up to ﬁrst order,

Original: [Equation]
Revised: [Equation]

Original: 17) and π ( z ) up to second order,
Revised: 17) and π ( z ) up to second order,

Original: [Equation]
Revised: [Equation]

Original: where π ( z 0 ) (cid:48) denotes the gradient and π (cid:48)(cid:48)
Revised: where π ( z 0 ) (cid:48) denotes the gradient and π (cid:48)(cid:48)

Original: ( z 0 ) the Hessian of π evaluated at z 0 , thus π ( z 0 ) (cid:48) ∈ R d and π (cid:48)(cid:48)
Revised: ( z 0 ) the Hessian of π evaluated at z 0 , thus π ( z 0 ) (cid:48) ∈ R d and π (cid:48)(cid:48)

Original: ( z 0 ) ∈ R d × d .
Revised: ( z 0 ) ∈ R d × d .

Original: Then, we can approximate p Σ ( x ) as follows:
Revised: Then, we can approximate p Σ ( x ) as follows:

Original: [Equation]
Revised: [Equation]

Original: Now deﬁne ˆΣ − 1 =
Revised: Now deﬁne ˆΣ − 1 =

Original: J Tf Σ − 1 J f .
Revised: J Tf Σ − 1 J f .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: we can exploit this Gaussian in Z -space and get
Revised: we can exploit this Gaussian in Z -space and get

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where (cid:12) stands for the elementwise multiplication and
Revised: where (cid:12) stands for the elementwise multiplication and

Original: ||
Revised: ||

Original: A ||
Revised: A ||

Original: + = (cid:80) di,j =1 A ij for a R d × R d matrix
Revised: + = (cid:80) di,j =1 A ij for a R d × R d matrix

Original: A .
Revised: A .

Original: For the special case where Σ = σ 2 I D , we can simplify this expression by exploiting that
Revised: For the special case where Σ = σ 2 I D , we can simplify this expression by exploiting that

Original: [Equation]
Revised: [Equation]

Original: Thus, in total, we get for this special choice of Σ
Revised: Thus, in total, we get for this special choice of Σ

Original: [Equation]
Revised: [Equation]

Original: We assume now
Revised: We assume now

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Note that 1 / (2 πσ 2 )
Revised: Note that 1 / (2 πσ 2 )

Original: D − dfrom equation (23) is exactly the normalization constant obtained when inﬂating the manifold with Gaussian noise in the normal space, q n ( x | x )
Revised: D − dfrom equation (23) is exactly the normalization constant obtained when inﬂating the manifold with Gaussian noise in the normal space, q n ( x | x )

Original: = 1 / (2 πσ 2 ) D − d 2 .
Revised: = 1 / (2 πσ 2 ) D − d 2 .

Original: What follows is that lim σ 2 → 0 p σ ( x ) /q
Revised: What follows is that lim σ 2 → 0 p σ ( x ) /q

Original: n ( x | x ) =
Revised: n ( x | x ) =

Original: p ∗ ( x ) as we wanted to show.
Revised: p ∗ ( x ) as we wanted to show.

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: The theorem follows directly from the deﬁnition of the reach number τ X of X .
Revised: The theorem follows directly from the deﬁnition of the reach number τ X of X .

Original: It is deﬁned as the supremum of all r ≥ 0 such that the orthogonal projection pr X on X is well-deﬁned on the r − neighbourhood
Revised: It is deﬁned as the supremum of all r ≥ 0 such that the orthogonal projection pr X on X is well-deﬁned on the r − neighbourhood

Original: X r of X ,
Revised: X r of X ,

Original: [Equation]
Revised: [Equation]

Original: where dist (˜ x, X ) denotes the distance of ˜ x to X .
Revised: where dist (˜ x, X ) denotes the distance of ˜ x to X .

Original: Thus,
Revised: Thus,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: see Deﬁnition 2.1.
Revised: see Deﬁnition 2.1.

Original: in Berenfeld & Hoffmann (2019).
Revised: in Berenfeld & Hoffmann (2019).

Original: By assumption τ X > 0 .
Revised: By assumption τ X > 0 .

Original: Thus for all ˜ x ∈ X τ X we have that x := pr X (˜ x ) is unique.
Revised: Thus for all ˜ x ∈ X τ X we have that x := pr X (˜ x ) is unique.

Original: Since X is a manifold, it must hold that ˜ x ∈ N x where N x denotes the normal space in x .
Revised: Since X is a manifold, it must hold that ˜ x ∈ N x where N x denotes the normal space in x .

Original: Let the noise generating distributions be a uniform distribution on the ball with radius τ X , thus
Revised: Let the noise generating distributions be a uniform distribution on the ball with radius τ X , thus

Original: [Equation]
Revised: [Equation]

Original: where Vol ( D − d, τ X ) denotesthe volume of a( D −d ) − dimensional ball with radius τ X .
Revised: where B ( x, τ X ) denotes a D − dimensional ball with radius τ X and center x

Original:  Then, we have for (cid:101) X
Revised: .. Then, we have for (cid:101) X

Original: = (cid:83) x ∈X N q n ( ·| x ) that
Revised: = (cid:83) x ∈X N q n ( ·| x ) that

Original: [Equation]
Revised: [Equation]

Original: Thus, X is Q − normally separable where Q :=
Revised: Thus, X is Q − normally separable where Q :=

Original: { q n ( ·| x ) } x ∈X .
Revised: { q n ( ·| x ) } x ∈X .

Original: B E XPERIMENTS B. T ECHNICAL D ETAILS FOR CIRCLE EXPERIMENTS
Revised: B E XPERIMENTS B. T ECHNICAL D ETAILS FOR CIRCLE EXPERIMENTS

Original: For the Normalizing Flow T − 1 ( x ) we use a BNAF (Block Neural Autoregressive Flow) for the circle experiment.
Revised: For the Normalizing Flow T − 1 ( x ) we use a BNAF (Block Neural Autoregressive Flow) for the circle experiment.

Original: The number of hidden dimensions was adapted to the dimensionality of the data and the difﬁculty of the target density.
Revised: The number of hidden dimensions was adapted to the dimensionality of the data and the difﬁculty of the target density.

Original: These details are reported in the corresponding tabular.
Revised: These details are reported in the corresponding tabular.

Original: For the optimization scheme, we used Adam optimizer with an initial learning rate 0.1, a learning rate decay of 0.5 after 2000 optimization steps without improvement (learning rate patience).
Revised: For the optimization scheme, we used Adam optimizer with an initial learning rate 0.1, a learning rate decay of 0.5 after 2000 optimization steps without improvement (learning rate patience).

Original: The batch size was set to 200 .
Revised: The batch size was set to 200 .

Original: The total number of iterations (one iteration corresponds to updating the parameters using one batch sample) used is also reported in the tabular.
Revised: The total number of iterations (one iteration corresponds to updating the parameters using one batch sample) used is also reported in the tabular.

Original: No hyperparameter ﬁne-tuning was done.
Revised: No hyperparameter ﬁne-tuning was done.

Original: For the FOM and χ 2 − noise models, we use the same architecture as for the D = 2 case.
Revised: For the FOM and χ 2 − noise models, we use the same architecture as for the D = 2 case.

Original: [Table]
Revised: [Table]

Original: B. T ECHNICAL D ETAILS FOR MIXTURE OF VON M ISE DISTRIBUTIONS ON S
Revised: B. T ECHNICAL D ETAILS FOR MIXTURE OF VON M ISE DISTRIBUTIONS ON S

Original: For the Normalizing Flow T − 1 ( x ) we use rational-quadratic neural spline ﬂows, alternating coupling layers and random feature permutations, see Durkan et al.
Revised: For the Normalizing Flow T − 1 ( x ) we use rational-quadratic neural spline ﬂows, alternating coupling layers and random feature permutations, see Durkan et al.

Original: For the optimization scheme, we used AdamW optimizer with an initial learning rate 0.0003, a learning rate cosine decay to 0 after every 2000 optimization steps, see Loshchilov & Hutter (2016), a weight decay of 0.
Revised: For the optimization scheme, we used AdamW optimizer with an initial learning rate 0.0003, a learning rate cosine decay to 0 after every 2000 optimization steps, see Loshchilov & Hutter (2016), a weight decay of 0.

Original: and a dropout probability of 0.
Revised: and a dropout probability of 0.

Original: The batch size was set to 200 .
Revised: The batch size was set to 200 .

Original: No hyperparameter ﬁne-tuning was done.
Revised: No hyperparameter ﬁne-tuning was done.

Original: See table 2 for more details.
Revised: See table 2 for more details.

Original: We use the same architecture for the FOM model and 3 seeds for the error bars.
Revised: We use the same architecture for the FOM model and 3 seeds for the error bars.

Original: [Table]
Revised: [Table]

Original: The distribution is distributions, p ∗ 1 ( φ 1 , θ 1 ) , p ∗ 2 ( φ 2 , θ 2 ) , p ∗ 3 ( φ 3 , θ 3 ) and p ∗ 4 ( φ 4 , θ 4 ) .
Revised: The distribution is distributions, p ∗ 1 ( φ 1 , θ 1 ) , p ∗ 2 ( φ 2 , θ 2 ) , p ∗ 3 ( φ 3 , θ 3 ) and p ∗ 4 ( φ 4 , θ 4 ) .

Original: Each of those distributions has the same product form
Revised: Each of those distributions has the same product form

Original: [Equation]
Revised: [Equation]

Original: We set κ = 6 .
Revised: We set κ = 6 .

Original: However, they differ in their mean values µ i and m i , see table 3.
Revised: However, they differ in their mean values µ i and m i , see table 3.

Original: We used 3 different seeds in total to obtain the conﬁdence intervals.
Revised: We used 3 different seeds in total to obtain the conﬁdence intervals.

Original: [Table]
Revised: [Table]