Original: P URE AND S PURIOUS C RITICAL P OINTS : A G EOMETRIC S TUDY OF L INEAR N ETWORKS
Revised: P URE AND S PURIOUS C RITICAL P OINTS : A G EOMETRIC S TUDY OF L INEAR N ETWORKS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The critical locus of the loss function of a neural network is determined by the geometry of the functional space and by the parameterization of this space by the network’s weights.
Revised: The critical locus of the loss function of a neural network is determined by the geometry of the functional space and by the parameterization of this space by the network’s weights.

Original: We introduce a natural distinction between pure critical points, which only depend on the functional space, and spurious critical points, which arise from the parameterization.
Revised: We introduce a natural distinction between pure critical points, which only depend on the functional space, and spurious critical points, which arise from the parameterization.

Original: We apply this perspective to revisit and extend the literature on the loss function of linear neural networks.
Revised: We apply this perspective to revisit and extend the literature on the loss function of linear neural networks.

Original: For this type of network, the functional space is either the set of all linear maps from input to output space or a determinantal variety , i e. , a set of linear maps with bounded rank.
Revised: For this type of network, the functional space is either the set of all linear maps from input to output space, or a determinantal variety , i.e. , a set of linear maps with bounded rank.

Original: We use geometric properties of determinantal varieties to derive new results on the landscape of linear networks with different loss functions and different parameterizations
Revised: We use geometric properties of determinantal varieties to derive new results on the landscape of linear networks with different loss functions and different parameterizations.

Revised: Our analysis clearly illustrates that the absence of “bad” local minima in the loss landscape of linear networks is due to two distinct phenomena that apply in different settings: it is true for arbitrary smooth convex losses in the case of architectures that can express all linear maps (“ﬁlling architectures”) but it holds only for the quadratic loss when the functional space is a determinantal variety (“non-ﬁlling architectures”).

Revised: Without any assumption on the architecture, smooth convex losses may lead to landscapes with many bad minima.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: A fundamental goal in the theory of deep learning is to explain why the optimization of the nonconvex loss function of a neural network does not seem to be affected by the presence of nonglobal local minima.
Revised: A fundamental goal in the theory of deep learning is to explain why the optimization of the nonconvex loss function of a neural network does not seem to be affected by the presence of nonglobal local minima.

Original: Many papers have addressed this issue by studying the landscape of the loss function (Baldi & Hornik, 1989; Choromanska et al., 2015; Kawaguchi, 2016;
Revised: Many papers have addressed this issue by studying the landscape of the loss function (Baldi & Hornik, 1989; Choromanska et al., 2015; Kawaguchi, 2016;

Original: Venturi et al., 2018).
Revised: Venturi et al., 2018).

Original: These papers have shown that, in certain situations, any local minimum for the loss is in fact always a global minimum.
Revised: These papers have shown that, in certain situations, any local minimum for the loss is in fact always a global minimum.

Original: Unfortunately, it is also known that this property does not apply in more general realistic settings (Yun et al., 2018; Venturi et al., 2018).
Revised: Unfortunately, it is also known that this property does not apply in more general realistic settings (Yun et al., 2018; Venturi et al., 2018).

Original: More recently, researchers have begun to search for explanations based on the dynamics of optimization.
Revised: More recently, researchers have begun to search for explanations based on the dynamics of optimization.

Original: For example, in certain limit situations, the gradient ﬂow of over-parameterized networks will avoid local minimizers (Chizat &
Revised: For example, in certain limit situations, the gradient ﬂow of over-parameterized networks will avoid local minimizers (Chizat &

Original: Bach, 2018; Mei et al., 2018).
Revised: Bach, 2018; Mei et al., 2018).

Original: We believe however that the study of the static properties of the loss function (the structure of its critical locus) is not settled.
Revised: We believe however that the study of the static properties of the loss function (the structure of its critical locus) is not settled.

Original: Even in the case of linear networks , the existing literature paints a purely analytical picture of the loss, and provides no sort of explanation as to “why” such architectures exhibit no bad local minima.
Revised: Even in the case of linear networks , the existing literature paints a purely analytical picture of the loss, and provides no sort of explanation as to “why” such architectures exhibit no bad local minima.

Original: A complete understanding of the critical locus should be a prerequisite for investigating the dynamics of the optimization.
Revised: A complete understanding of the critical locus should be a prerequisite for investigating the dynamics of the optimization.

Original: The goal of this paper is to revisit the loss function of neural networks from a geometric perspective, focusing on the relationship between the functional space of the network and its parameterization.
Revised: The goal of this paper is to revisit the loss function of neural networks from a geometric perspective, focusing on the relationship between the functional space of the network and its parameterization.

Original: In particular, we view the loss as a composition
Revised: In particular, we view the loss as a composition

Original: [Equation]
Revised: [Equation]

Original: In this setting, the function (cid:96) is almost always convex, however the composition L = (cid:96) ◦ µ is not.
Revised: In this setting, the function (cid:96) is almost always convex, however the composition L = (cid:96) ◦ µ is not.

Original: Critical points for L can in fact arise for two distinct reasons: either because we are applying (cid:96) to a non-convex functional space, or because the parameterizing map µ is locally degenerate.
Revised: Critical points for L can in fact arise for two distinct reasons: either because we are applying (cid:96) to a non-convex functional space, or because the parameterizing map µ is locally degenerate.

Original: We distinguish these two types of critical points by referring to them, respectively, as pure and spurious .
Revised: We distinguish these two types of critical points by referring to them, respectively, as pure and spurious .

Original:  Intuitively, pure critical points actually reﬂect the geometry of the functional space associated with the network, while spurious critical points arise as “artifacts” from the parameterization.
Revised: ∗ Equal contribution. [Table] Intuitively, pure critical points actually reﬂect the geometry of the functional space associated with the network, while spurious critical points arise as “artifacts” from the parameterization.

Original: After deﬁning pure and critical points for arbitrary networks, we investigate in detail the classiﬁcation of critical points in the case of linear networks.
Revised: After deﬁning pure and critical points for arbitrary networks, we investigate in detail the classiﬁcation of critical points in the case of linear networks.

Original: The functional space for such networks can be identiﬁed with a family of linear maps, and we can describe its geometry using algebraic tools.
Revised: The functional space for such networks can be identiﬁed with a family of linear maps, and we can describe its geometry using algebraic tools.

Original: Many of our statements rely on a careful analysis of the differential of the matrix multiplication map.
Revised: Many of our statements rely on a careful analysis of the differential of the matrix multiplication map.

Original: In particular, we prove that non-global local minima are necessarily pure critical points for convex losses, which means that many properties of the loss landscape can be read from the functional space.
Revised: In particular, we prove that non-global local minima are necessarily pure critical points for convex losses, which means that many properties of the loss landscape can be read from the functional space.

Original: On the other hand, we emphasize that even for linear networks it is possible to ﬁnd many smooth convex losses with non-global local minima.
Revised: On the other hand, we emphasize that even for linear networks it is possible to ﬁnd many smooth convex losses with non-global local minima.

Original: This happens when the functional space is a determinantal variety , i.e. , a (non-smooth and non-convex) family of matrices with bounded rank.
Revised: This happens when the functional space is a determinantal variety , i.e. , a (non-smooth and non-convex) family of matrices with bounded rank.

Original: In this setting, the absence of non-global minima actually holds in the particular case of the quadratic loss, because of very special geometric properties of determinantal varieties that we discuss.
Revised: In this setting, the absence of non-global minima actually holds in the particular case of the quadratic loss, because of very special geometric properties of determinantal varieties that we discuss.

Original: Related Work.
Revised: Related Work.

Original: Baldi & Hornik (1989) ﬁrst proved the absence of non-global (“bad”)
Revised: Baldi & Hornik (1989) ﬁrst proved the absence of non-global (“bad”)

Original: local minima for linear networks with one hidden layer (autoencoders).
Revised: local minima for linear networks with one hidden layer (autoencoders).

Original: Their result was generalized to the case of deep linear networks by Kawaguchi (2016).
Revised: Their result was generalized to the case of deep linear networks by Kawaguchi (2016).

Original: Many papers have since then studied the loss landscape of linear networks under different assumptions (Hardt & Ma, 2016; Yun et al., 2017; Zhou & Liang, 2017; Laurent & von Brecht, 2017; Lu & Kawaguchi, 2017; Zhang, 2019).
Revised: Many papers have since then studied the loss landscape of linear networks under different assumptions (Hardt & Ma, 2016; Yun et al., 2017; Zhou & Liang, 2017; Laurent & von Brecht, 2017; Lu & Kawaguchi, 2017; Zhang, 2019).

Original: In particular, Laurent & von Brecht (2017) showed that linear networks with “no bottlenecks” have no bad local minima for arbitrary smooth loss functions.
Revised: In particular, Laurent & von Brecht (2017) showed that linear networks with “no bottlenecks” have no bad local minima for arbitrary smooth loss functions.

Original: Lu & Kawaguchi (2017) and Zhang (2019) argued that “depth does not create local minima”, meaning that the absence of local minima of deep linear networks is implied by the same property of shallow linear networks.
Revised: Lu & Kawaguchi (2017) and Zhang (2019) argued that “depth does not create local minima”, meaning that the absence of local minima of deep linear networks is implied by the same property of shallow linear networks.

Original: Our study of pure and spurious critical points can be used as a framework for explaining all these results in a uniﬁedand intuitive way.
Revised: Our study of pure and spurious critical points can be used as a framework for explaining all these results in a uniﬁed way.

Revised: The optimization dynamics of linear networks are also an active area of research (Arora et al., 2019; 2018), and our analysis of the landscape in function space sets the stage for studying gradient dynamics on determinantal varieties, as in Bah et al.

Original: Our analysis is also closely related to objects of study in applied algebraic geometry, particularly determinantal varieties and ED discriminants (Draisma et al., 2013; Ottaviani et al., 2013) .
Revised: Our work is also closely related to objects of study in applied algebraic geometry, particularly determinantal varieties and ED discriminants (Draisma et al., 2013; Ottaviani et al., 2013).

Revised: Finally, we mention other recent works that study neural networks using algebraic-geometric tools (Mehta et al., 2018; Kileel et al., 2019; Jaffali & Oeding, 2019).

Original: Main contributions.
Revised: Main contributions.

Original: • We introduce a natural distinction between “pure” and “spurious” critical points for the loss function of networks.
Revised: • We introduce a natural distinction between “pure” and “spurious” critical points for the loss function of networks.

Revised: These notions provide an intuitive and useful language for studying a central aspect in the theory of neural networks, namely the (over)parameterization of the functional space and its effect on the optimization landscape.

Original: While most of the paper focuses on linear networks, this viewpoint applies to more general settings as well (see also our discussion in Appendix A.3).
Revised: While most of the paper focuses on linear networks, this viewpoint applies to more general settings as well (see also our discussion in Appendix A.

Original: • We study the pure and critical locus for linear networks and arbitrary loss functions.
Revised: • We study the pure and critical locus for linear networks and arbitrary loss functions.

Original: We show that non-global local minima are always pure for convex losses, unifying many known properties on the landscape of linear networks.
Revised: We show that non-global local minima are always pure for convex losses, unifying many known properties on the landscape of linear networks.

Revised: • We explain that the absence of “bad” local minima in the loss landscape of linear networks is due to two distinct phenomena and does not hold in general: it is true for arbitrary smooth convex losses in the case of architectures that can express all linear maps (“ﬁlling architectures”) and it holds for the quadratic loss when the functional space is a determinantal variety (“non-ﬁlling architectures”).

Original:  We also prove other new results, including a precise description of the number of topologically connected components of the variety of global minima. 
Revised: Without any assumption on the architecture, smooth convex losses may lead to many local minima See Table 1. • We provide a precise description of the number of topologically connected components of the set of global minima. This relates to recent work on “mode connectivity” in loss landscapes of neural networks (Garipov et al., 2018).

Original: • We spell out connections between the loss landscape and classical geometric objects such as caustics and ED discriminants.
Revised: • We spell out connections between the loss landscape and classical geometric objects such as caustics and ED discriminants.

Original: We believe that these concepts may be useful in the study of more general functional spaces.
Revised: We believe that these concepts may be useful in the study of more general functional spaces.

Original: Differential notation.
Revised: Differential notation.

Original: Our functional spaces will be manifolds with singularities, so we will make use of elementary notions from differential geometry.
Revised: Our functional spaces will be manifolds with singularities, so we will make use of elementary notions from differential geometry.

Original: If M and N are manifolds and g : M → N is a smooth map, then we write dg ( x ) for the differential of g at the point x .
Revised: If M and N are manifolds and g : M → N is a smooth map, then we write dg ( x ) for the differential of g at the point x .

Original: This means that dg ( x ) : T x M → T g ( x ) N is the ﬁrst order linear approximation of g at the point x ∈ M .
Revised: This means that dg ( x ) : T x M → T g ( x ) N is the ﬁrst order linear approximation of g at the point x ∈ M .

Original: If M and
Revised: If M and

Original: N have singularities, then the same deﬁnitions apply if we restrict g to smooth points in M whose image is also smooth in N .
Revised: N have singularities, then the same deﬁnitions apply if we restrict g to smooth points in M whose image is also smooth in N .

Original: For most of our analysis, manifolds will be embedded in Euclidean spaces, say M ⊂ R m and N ⊂ R n , so we can view the tangent spaces T x M and T g ( x ) N as also embedded in R m and R n .
Revised: For most of our analysis, manifolds will be embedded in Euclidean spaces, say M ⊂ R m and N ⊂ R n , so we can view the tangent spaces T x M and T g ( x ) N as also embedded in R m and R n .

Original: When N = R , the critical locus of a map g : M → R is deﬁned as
Revised: When N = R , the critical locus of a map g : M → R is deﬁned as

Original: Crit ( g ) = { x ∈ Smooth ( M ) | dg ( x ) = 0 } .
Revised: Crit ( g ) = { x ∈ Smooth ( M ) | dg ( x ) = 0 } .

Original:  P RELIMINARIES 2. P URE AND SPURIOUS CRITICAL POINTS
Revised:  P RELIMINARIES 2. P URE AND SPURIOUS CRITICAL POINTS

Original: A neural network (or any general “parametric learning model”) is deﬁned by a continuous mapping Φ : R d θ × R d x → R d y that associates an input vector x ∈ R d x and a set of parameters θ ∈ R
Revised: A neural network (or any general “parametric learning model”) is deﬁned by a continuous mapping Φ : R d θ × R d x → R d y that associates an input vector x ∈ R d x and a set of parameters θ ∈ R

Original: d θ to an output vector y =
Revised: d θ to an output vector y =

Original: Φ( θ, x ) ∈ R d y .
Revised: Φ( θ, x ) ∈ R d y .

Original: In other words, Φ determines a family of continuous functions parameterized by θ ∈ R d θ :
Revised: In other words, Φ determines a family of continuous functions parameterized by θ ∈ R d θ :

Original: [Equation]
Revised: [Equation]

Original: Even though M Φ is naturally embedded in an inﬁnite-dimensional functional space, it is itself ﬁnite dimensional.
Revised: Even though M Φ is naturally embedded in an inﬁnite-dimensional functional space, it is itself ﬁnite dimensional.

Original: In fact, if the mapping Φ is smooth, then M Φ is a ﬁnite-dimensional manifold with singularities, and its intrinsic dimension is upper bounded by d θ .
Revised: In fact, if the mapping Φ is smooth, then M Φ is a ﬁnite-dimensional manifold with singularities, and its intrinsic dimension is upper bounded by d θ .

Original: It is also important to note that neural networks are often non-identiﬁable models , which means that different parameters can represent the same function ( i.e. , f θ = f θ (cid:48) does not imply θ = θ (cid:48) ).
Revised: It is also important to note that neural networks are often non-identiﬁable models , which means that different parameters can represent the same function ( i.e. , f θ = f θ (cid:48) does not imply θ = θ (cid:48) ).

Original: The manifold M Φ is sometimes known as a neuromanifold (Amari, 2016).
Revised: The manifold M Φ is sometimes known as a neuromanifold (Amari, 2016).

Original: We now consider a general loss function of the form
Revised: We now consider a general loss function of the form

Original: L = (cid:96) ◦ µ , where µ : R d θ → M Φ is the (over)parameterization of M Φ by θ and (cid:96) is a functional deﬁned on a subset of C ( R d x , R d y ) containing MΦ :
Revised: L = (cid:96) ◦ µ , where µ : R d θ → M Φ is the (over)parameterization of M Φ by θ and (cid:96) is a functional deﬁned on a subset of C ( R d x , R d y ) containing M

Original: [Equation]
Revised: [Equation]

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: A critical point θ ∗ ∈ Crit ( L ) is a pure critical point if µ ( θ ∗ ) is a critical point for the restriction (cid:96)
Revised: A critical point θ ∗ ∈ Crit ( L ) is a pure critical point if µ ( θ ∗ ) is a critical point for the restriction (cid:96)

Original: | M Φ (note that this implicitly requires µ ( θ ∗ ) to be a smooth point of M Φ ).
Revised: | M Φ (note that this implicitly requires µ ( θ ∗ ) to be a smooth point of M Φ ).

Original: If θ ∗ ∈ Crit ( L ) but µ ( θ ∗ ) (cid:54)∈ Crit ( (cid:96) | M Φ ) , we say that θ ∗ is a spurious critical point .
Revised: If θ ∗ ∈ Crit ( L ) but µ ( θ ∗ ) (cid:54)∈ Crit ( (cid:96) | M Φ ) , we say that θ ∗ is a spurious critical point .

Original: It is clear from this deﬁnition that pure critical points reﬂect the geometry of the functional space, while spurious critical points do not have an intrinsic functional interpretation.
Revised: It is clear from this deﬁnition that pure critical points reﬂect the geometry of the functional space, while spurious critical points do not have an intrinsic functional interpretation.

Original: For example, if θ ∗ ∈ Crit ( L ) is a spurious critical point, then it may be possible to ﬁnd another parameter θ (cid:48) that represents the same function f θ ∗ =
Revised: For example, if θ ∗ ∈ Crit ( L ) is a spurious critical point, then it may be possible to ﬁnd another parameter θ (cid:48) that represents the same function f θ ∗ =

Original: f θ (cid:48) and is not a critical point for L (see Figure 1).
Revised: f θ (cid:48) and is not a critical point for L (see Figure 1).

Original: In contrast, if θ ∗ is a pure critical point, then all parameters θ (cid:48) such that µ ( θ (cid:48) )
Revised: In contrast, if θ ∗ is a pure critical point, then all parameters θ (cid:48) such that µ ( θ (cid:48) )

Original: = µ ( θ ∗ ) are automatically in
Revised: = µ ( θ ∗ ) are automatically in

Original: Crit ( L ) , simply because dL ( θ (cid:48) )
Revised: Crit ( L ) , simply because dL ( θ (cid:48) )

Original: = d(cid:96) | M Φ ( µ ( θ (cid:48) ))
Revised: = d(cid:96) | M Φ ( µ ( θ (cid:48) ))

Original: ◦ dµ ( θ (cid:48) ) .
Revised: ◦ dµ ( θ (cid:48) ) .

Original: This will motivate us to study the ﬁber { θ | µ ( θ ) =
Revised: This will motivate us to study the ﬁber { θ | µ ( θ ) =

Original: f } of all parameters mapped to the same function f (particularly when the function f is a critical point of (cid:96) | M Φ ).
Revised: f } of all parameters mapped to the same function f (particularly when the function f is a critical point of (cid:96) | M Φ ).

Original: We note that a sufﬁcient condition for θ ∗ ∈ Crit ( L ) to be a pure critical point is that the differential dµ ( θ ∗ ) at θ ∗ has maximal rank (namely dim M Φ ), i.e. , that µ is locally a submersion at θ ∗ .
Revised: We note that a sufﬁcient condition for θ ∗ ∈ Crit ( L ) to be a pure critical point is that the differential dµ ( θ ∗ ) at θ ∗ has maximal rank (namely dim M Φ ), i.e. , that µ is locally a submersion at θ ∗ .

Original: Indeed, we have in this case
Revised: Indeed, we have in this case

Original: [Equation]
Revised: [Equation]

Original: so µ ( θ ∗ ) is critical for the restriction of (cid:96) to M Φ .
Revised: so µ ( θ ∗ ) is critical for the restriction of (cid:96) to M Φ .

Original: We also point out a special situation when M Φ is a convex set (as a subset of C ( R d x , R d y ) ) and (cid:96) is a smooth convex functional .
Revised: We also point out a special situation when M Φ is a convex set (as a subset of C ( R d x , R d y ) ) and (cid:96) is a smooth convex functional .

Original: In this case, the only critical points of (cid:96) | M Φ are global minima, so we deduce that any critical point of L = (cid:96) ◦ µ is either a global minimum or a spurious critical point.
Revised: In this case, the only critical points of (cid:96) | M Φ are global minima, so we deduce that any critical point of L = (cid:96) ◦ µ is either a global minimum or a spurious critical point.

Original: The following simple observation gives a sufﬁcient condition for critical points to be saddles ( i.e. , they are not local minima or local maxima).
Revised: The following simple observation gives a sufﬁcient condition for critical points to be saddles ( i.e. , they are not local minima or local maxima).

Original: Lemma 2.
Revised: Lemma 2.

Original: Let θ ∗ ∈ Crit ( L ) be a (necessarily spurious) critical point with the following property: for any open neighborhood U of θ , there exists θ (cid:48) in U such that µ ( θ (cid:48) )
Revised: Let θ ∗ ∈ Crit ( L ) be a (necessarily spurious) critical point with the following property: for any open neighborhood U of θ , there exists θ (cid:48) in U such that µ ( θ (cid:48) )

Original: =
Revised: =

Original: µ ( θ ) and θ (cid:48) (cid:54)∈ Crit ( L ) .
Revised: µ ( θ ) and θ (cid:48) (cid:54)∈ Crit ( L ) .

Original: Then θ ∗ is a saddle for L .
Revised: Then θ ∗ is a saddle for L .

Original: Proof.
Revised: Proof.

Original: Assume that θ ∗ is a local minimum (the reasoning is analogous if θ ∗ is a local maximum).
Revised: Assume that θ ∗ is a local minimum (the reasoning is analogous if θ ∗ is a local maximum).

Original: This means that there exists a neighborhood U of θ ∗ such that L ( θ ) ≥ L ( θ ∗ ) for all θ ∈ U .
Revised: This means that there exists a neighborhood U of θ ∗ such that L ( θ ) ≥ L ( θ ∗ ) for all θ ∈ U .

Original: In particular, if θ (cid:48) ∈
Revised: In particular, if θ (cid:48) ∈

Original: U is such that µ ( θ (cid:48) )
Revised: U is such that µ ( θ (cid:48) )

Original: = µ ( θ ) , then θ (cid:48) must also be a local minimum.
Revised: = µ ( θ ) , then θ (cid:48) must also be a local minimum.

Original: This contradicts θ (cid:48) (cid:54)∈ Crit ( L ) .
Revised: This contradicts θ (cid:48) (cid:54)∈ Crit ( L ) .

Original: This general discussion on pure and spurious critical points applies to any smooth network map Φ (with possible extensions to the case of piece-wise smooth mappings), and we believe that the distinction can be a useful tool in the study of the optimization landscape of general networks.
Revised: This general discussion on pure and spurious critical points applies to any smooth network map Φ (with possible extensions to the case of piece-wise smooth mappings), and we believe that the distinction can be a useful tool in the study of the optimization landscape of general networks.

Original: In the remaining part of the paper, we use this perspective for an in-depth study of the critical points of linear networks .
Revised: In the remaining part of the paper, we use this perspective for an in-depth study of the critical points of linear networks .

Original: For this type of network, the functional set M Φ can be embedded in a ﬁnite dimensional ambient space, namely the space of all linear maps R d x → R d y .
Revised: For this type of network, the functional set M Φ can be embedded in a ﬁnite dimensional ambient space, namely the space of all linear maps R d x → R d y .

Original: Furthermore, M Φ is an algebraic variety (a manifold that can have singularities and that can be described by algebraic equations).
Revised: Furthermore, M Φ is an algebraic variety (a manifold that can have singularities and that can be described by algebraic equations).

Original: We will use basic tools from algebraic geometry to provide a complete description of pure and spurious critical points, and to prove new results on the landscape of linear networks.
Revised: We will use basic tools from algebraic geometry to provide a complete description of pure and spurious critical points, and to prove new results on the landscape of linear networks.

Original: 2. L INEAR NETWORKS AND DETERMINANTAL VARIETIES
Revised: 2. L INEAR NETWORKS AND DETERMINANTAL VARIETIES

Original: A linear network is a map Φ : R d θ × R d x → R d y of the form
Revised: A linear network is a map Φ : R d θ × R d x → R d y of the form

Original: [Equation]
Revised: [Equation]

Original: where W i ∈ R d i × d i − 1 are matrices (so d 0 = d x , d h = d y , and d θ = d 0 d 1 + d 1 d 2 + . . .
Revised: where W i ∈ R d i × d i − 1 are matrices (so d 0 = d x , d h = d y , and d θ = d 0 d 1 + d 1 d 2 + . . .

Original: + d h − 1 d h ).
Revised: + d h − 1 d h ).

Original: The functional space is in this case a subset of the space of all linear maps R d 0 → R d h .
Revised: The functional space is in this case a subset of the space of all linear maps R d 0 → R d h .

Original: As in (1), we can decompose a loss function L for a linear network Φ as
Revised: As in (1), we can decompose a loss function L for a linear network Φ as

Original: [Equation]
Revised: [Equation]

Original: Here µ d is the matrix multiplication map for the sequence of widths d = ( d h , . . .
Revised: Here µ d is the matrix multiplication map for the sequence of widths d = ( d h , . . .

Original: , d 0 ) , and (cid:96) is a functional on the space of ( d h × d 0 ) -matrices.
Revised: , d 0 ) , and (cid:96) is a functional on the space of ( d h × d 0 ) -matrices.

Original: In practice, it is typically a functional that depends on the training data, e.g. (cid:96) ( W )
Revised: In practice, it is typically a functional that depends on the training data, e.g. (cid:96) ( W )

Original: =
Revised: =

Original: (cid:107) WX − Y (cid:107) 2 for ﬁxed matrices X, Y .
Revised: (cid:107) WX − Y (cid:107) 2 for ﬁxed matrices X, Y .

Original: 1 Note that even if (cid:96) is a convex functional, the set M Φ will often not be a convex set.
Revised: 2 Note that even if (cid:96) is a convex functional, the set M Φ will often not be a convex set.

Original: In fact, it is easy to see that the image of µ d is the space M r of ( d h × d 0 ) -matrices of rank at most r = min { d 0 , . . .
Revised: In fact, it is easy to see that the image of µ d is the space M r of ( d h × d 0 ) -matrices of rank at most r = min { d 0 , . . .

Original: , d h } .
Revised: , d h } .

Original: If r < min( d 0 , d h ) , this set is known as a determinantal variety , a classical object of study in algebraic geometry (Harris, 1995).
Revised: If r < min( d 0 , d h ) , this set is known as a determinantal variety , a classical object of study in algebraic geometry (Harris, 1995).

Original: It is in fact an algebraic variety , i.e. , it is described by polynomial equations in the matrix entries (namely, it is the zero-set of all ( r + 1) × ( r + 1) -minors), and it is well known that the dimension of M r is r ( m + n − r ) .
Revised: It is in fact an algebraic variety , i.e. , it is described by polynomial equations in the matrix entries (namely, it is the zero-set of all ( r + 1) × ( r + 1) -minors), and it is well known that the dimension of M r is r ( m + n − r ) .

Original: Furthermore, for r > 0 , the variety M r has many singularities: its singular locus is exactly M r − 1 ⊂ M r , the set of all matrices with rank strictly smaller than r .
Revised: Furthermore, for r > 0 , the variety M r has many singularities: its singular locus is exactly M r − 1 ⊂ M r , the set of all matrices with rank strictly smaller than r .

Original: We refer the reader tothe Appendix  for more details on determinantal varieties.
Revised: We refer the reader to Appendix A.1 for more details on determinantal varieties.

Original:  M AIN RESULTS
Revised:  M AIN RESULTS

Original: In this section, we investigate the critical locus Crit ( L ) of general functions L : R d θ → R of the form L = (cid:96) ◦ µ d where (cid:96) :
Revised: In this section, we investigate the critical locus Crit ( L ) of general functions L : R d θ → R of the form L = (cid:96) ◦ µ d where (cid:96) :

Original: R d h × d 0 → R is a (often convex) smooth map, and µ d is the matrix multiplication map introduced in (3).
Revised: R d h × d 0 → R is a (often convex) smooth map, and µ d is the matrix multiplication map introduced in (3).

Original: By studying the differential of µ d , we will characterize pure and spurious critical points of L .
Revised: By studying the differential of µ d , we will characterize pure and spurious critical points of L .

Original: As previously noted, the image of µ d is M r ⊂ R d h × d 0 where r = min { d i } .
Revised: As previously noted, the image of µ d is M r ⊂ R d h × d 0 where r = min { d i } .

Original: In particular, we distinguish between two cases:
Revised: In particular, we distinguish between two cases:

Original: • We say that the map µ d is ﬁlling if r = min { d 0 , d h } , so
Revised: • We say that the map µ d is ﬁlling if r = min { d 0 , d h } , so

Original: M r = R d h × d 0 .
Revised: M r = R d h × d 0 .

Original: In this case, the functional space is smooth and convex .
Revised: In this case, the functional space is smooth and convex .

Original: • We say that the map µ d is non-ﬁlling if r < min { d 0 , d h } , so M r (cid:40) R d h × d 0 is a determinantal variety.
Revised: • We say that the map µ d is non-ﬁlling if r < min { d 0 , d h } , so M r (cid:40) R d h × d 0 is a determinantal variety.

Original: In this case, the functional space is non-smooth and non-convex .
Revised: In this case, the functional space is non-smooth and non-convex .

Original: 3. P ROPERTIES OF THE MATRIX MULTIPLICATION MAP
Revised: 3. P ROPERTIES OF THE MATRIX MULTIPLICATION MAP

Original: We present some general results on the matrix multiplication map µ d , which we will apply to linear networks in the next subsection.
Revised: We present some general results on the matrix multiplication map µ d , which we will apply to linear networks in the next subsection.

Original: These facts may also be useful in other settings, for example, to study the piece-wise linear behavior of ReLU networks.
Revised: These facts may also be useful in other settings, for example, to study the piece-wise linear behavior of ReLU networks.

Original: We begin by noting that the differential map of µ d can be written explicitly as
Revised: We begin by noting that the differential map of µ d can be written explicitly as

Original: [Equation]
Revised: [Equation]

Original: Given a matrix M ∈ R m × n , we denote by Row ( M ) ⊂ R n and Col ( M ) ⊂
Revised: Given a matrix M ∈ R m × n , we denote by Row ( M ) ⊂ R n and Col ( M ) ⊂

Original: R m the vector spaces spanned by the rows and columns of M , respectively.
Revised: R m the vector spaces spanned by the rows and columns of M , respectively.

Original: Writing W >i
Revised: Writing W >i

Original: =
Revised: =

Original: W h
Revised: W h

Original: W h − 1 . . .
Revised: W h − 1 . . .

Original: W i +

Revised: W i +

Original: and
Revised: and

Original: W <i
Revised: W <i

Original: =
Revised: =

Original: W i − 1 W i − 1 . . .
Revised: W i − 1 W i − 1 . . .

Original: W 1 , the image of dµ d ( θ ) in (4) is
Revised: W 1 , the image of dµ d ( θ ) in (4) is

Original: [Equation]
Revised: [Equation]

Original: From this expression, we deduce the following useful fact.
Revised: From this expression, we deduce the following useful fact.

Original: Lemma 3.
Revised: Lemma 3.

Original: The dimension of the image of the differential dµ d at θ =
Revised: The dimension of the image of the differential dµ d at θ =

Original: ( W h , . . .
Revised: ( W h , . . .

Original: , W 1 ) is given by
Revised: , W 1 ) is given by

Original: [Equation]
Revised: [Equation]

Original: where we use the convention that W < 1 = I d 0 , W >h
Revised: where we use the convention that W < 1 = I d 0 , W >h

Original: = I
Revised: = I

Original: d h are the identity matrices of size d 0 , d h .
Revised: d h are the identity matrices of size d 0 , d h .

Original: We can use Lemma 3 to characterize all cases when the differential dµ d at θ =
Revised: We can use Lemma 3 to characterize all cases when the differential dµ d at θ =

Original: ( W h , . . .
Revised: ( W h , . . .

Original: , W 1 ) has full rank ( i.e. , when the matrix multiplication map is a local submersion onto M r ).
Revised: , W 1 ) has full rank ( i.e. , when the matrix multiplication map is a local submersion onto M r ).

Original: Theorem 4.
Revised: Theorem 4.

Original: Let r = min
Revised: Let r = min

Original: { d i } , θ =
Revised: { d i } , θ =

Original: ( W h , . . .
Revised: ( W h , . . .

Original: , W 1 ) , and W = µ d ( θ ) .
Revised: , W 1 ) , and W = µ d ( θ ) .

Original: • (Filling case)
Revised: • (Filling case)

Original: If r = min { d h , d 0 } , the differential dµ d ( θ ) has maximal rank equal to dim M r =
Revised: If r = min { d h , d 0 } , the differential dµ d ( θ ) has maximal rank equal to dim M r =

Original: d h d 0 if and only if, for every i ∈ { 1 , 2 , . .
Revised: d h d 0 if and only if, for every i ∈ { 1 , 2 , . .

Original: , h − 1 } , either rk( W >i ) =
Revised: , h − 1 } , either rk( W >i ) =

Original: d h or rk( W <i +1 )
Revised: d h or rk( W <i +1 )

Original: =
Revised: =

Original: d 0 holds.
Revised: d 0 holds.

Original: • (Non-ﬁlling case)
Revised: • (Non-ﬁlling case)

Original: If r < min { d h , d 0 } , the differential dµ d ( θ ) has maximal rank equal to dim M r = r
Revised: If r < min { d h , d 0 } , the differential dµ d ( θ ) has maximal rank equal to dim M r = r

Original: ( d h + d 0 − r )
Revised: ( d h + d 0 − r )

Original: if and only if rk( W ) = r .
Revised: if and only if rk( W ) = r .

Original: Furthermore, in both situations, if rk( W ) =
Revised: Furthermore, in both situations, if rk( W ) =

Original: e < r , then the image of dµ d ( θ ) always contains the tangent space T W M e of M e ⊂ M r at W .
Revised: e < r , then the image of dµ d ( θ ) always contains the tangent space T W M e of M e ⊂ M r at W .

Original: We note that dµ d ( θ ) has always maximal rank when rk( W ) =
Revised: We note that dµ d ( θ ) has always maximal rank when rk( W ) =

Original: r = min { d i } , however in the ﬁlling case it is possible to obtain a local submersion even when rk( W ) <
Revised: r = min { d i } , however in the ﬁlling case it is possible to obtain a local submersion even when rk( W ) <

Original: r (see Example 19 in appendix).
Revised: r (see Example 19 in appendix).

Original: We next describe the ﬁber of the matrix multiplication map, that is, the set
Revised: We next describe the ﬁber of the matrix multiplication map, that is, the set

Original: [Equation]
Revised: [Equation]

Original: It will be convenient to refer to µ − 1 d ( W ) as the set of d -factorizations of W .
Revised: It will be convenient to refer to µ − 1 d ( W ) as the set of d -factorizations of W .

Original: We are interested in understanding the structure of µ − 1 d ( W ) since, as argued in Section 2.1, pure critical loci consist of ﬁbers of “critical functions”.
Revised: We are interested in understanding the structure of µ − 1 d ( W ) since, as argued in Section 2.1, pure critical loci consist of ﬁbers of “critical functions”.

Original: The following result completely describes the connectivity of µ − 1 d ( W ) .
Revised: The following result completely describes the connectivity of µ − 1 d ( W ) .

Original: Theorem 5.
Revised: Theorem 5.

Original: Let r = min
Revised: Let r = min

Original: { d i } .
Revised: { d i } .

Original: If rk( W ) =
Revised: If rk( W ) =

Original: r , then the set of d -factorizations µ − 1 d ( W ) of W has exactly 2 b path-connected components, where b
Revised: r , then the set of d -factorizations µ − 1 d ( W ) of W has exactly 2 b path-connected components, where b

Original: = # { i | d i = r, 0 <
Revised: = # { i | d i = r, 0 <

Original: i < h } .
Revised: i < h } .

Original: If rk( W ) <
Revised: If rk( W ) <

Original: r , then µ − 1 d ( W ) is always path-connected.
Revised: r , then µ − 1 d ( W ) is always path-connected.

Original: 3. A PPLICATION TO LINEAR NETWORKS
Revised: 3. A PPLICATION TO LINEAR NETWORKS

Original: We now apply the general results from the previous subsection to study the critical locus Crit ( L ) with L = (cid:96) ◦ µ d , where (cid:96) is any smooth function.
Revised: We now apply the general results from the previous subsection to study the critical locus Crit ( L ) with L = (cid:96) ◦ µ d , where (cid:96) is any smooth function.

Original: In the following, we always use r = min { r i } and
Revised: In the following, we always use r = min { r i } and

Original: W = µ d ( θ ) .
Revised: W = µ d ( θ ) .

Original: The next two facts follow almost immediately from Theorem 4.
Revised: The next two facts follow almost immediately from Theorem 4.

Original: Proposition 6.
Revised: Proposition 6.

Original: If θ is such that dµ d ( θ ) has maximal rank (see Theorem 4), then θ ∈ Crit ( L )
Revised: If θ is such that dµ d ( θ ) has maximal rank (see Theorem 4), then θ ∈ Crit ( L )

Original: if and only if W ∈ Crit ( (cid:96) | M r ) , and θ is a minimum (resp., saddle, maximum) for L if and only if W is a minimum (resp., saddle, maximum) for (cid:96)
Revised: if and only if W ∈ Crit ( (cid:96) | M r ) , and θ is a minimum (resp., saddle, maximum) for L if and only if W is a minimum (resp., saddle, maximum) for (cid:96)

Original: | M r .
Revised: | M r .

Original: If rk( W ) =
Revised: If rk( W ) =

Original: r (which implies that dµ d ( θ ) has maximal rank) and θ ∈ Crit ( L ) , then all d -factorizations of W also belong to Crit ( L ) .
Revised: r (which implies that dµ d ( θ ) has maximal rank) and θ ∈ Crit ( L ) , then all d -factorizations of W also belong to Crit ( L ) .

Original: Proposition 7.
Revised: Proposition 7.

Original: If θ ∈ Crit ( L ) with rk( W ) =
Revised: If θ ∈ Crit ( L ) with rk( W ) =

Original: e ≤ r , then W ∈ Crit ( (cid:96) | M e ) .
Revised: e ≤ r , then W ∈ Crit ( (cid:96) | M e ) .

Original: In other words, if rk( W ) < r , then θ ∈ Crit ( L ) implies that W is a critical point for the restriction of (cid:96) to a smaller determinantal variety M e (which is in the singular locus of the functional space M r in the non-ﬁlling case).
Revised: In other words, if rk( W ) < r , then θ ∈ Crit ( L ) implies that W is a critical point for the restriction of (cid:96) to a smaller determinantal variety M e (which is in the singular locus of the functional space M r in the non-ﬁlling case).

Original: Note that if d h = 1 , then either W = 0 or rk( W ) = 1 , and in the latter case Proposition 7 implies that W ∈ Crit ( (cid:96) | R d 0 \{ 0 } ) .
Revised: Note that if d h = 1 , then either W = 0 or rk( W ) = 1 , and in the latter case Proposition 7 implies that W ∈ Crit ( (cid:96) | R d 0 \{ 0 } ) .

Original: If (cid:96) is convex, we immediately obtain that all critical points (not just local minima, as in Laurent & von Brecht (2017)) below a certain energy level are global minima .
Revised: If (cid:96) is convex, we immediately obtain that all critical points (not just local minima, as in Laurent & von Brecht (2017)) below a certain energy level are global minima .

Original: Corollary 8.
Revised: Corollary 8.

Original: Assume that (cid:96) is a smooth convex function and that d h = 1 .
Revised: Assume that (cid:96) is a smooth convex function and that d h = 1 .

Original: If θ ∈ Crit ( L ) , then either W = µ d ( θ ) = 0 or θ is global minimum for L .
Revised: If θ ∈ Crit ( L ) , then either W = µ d ( θ ) = 0 or θ is global minimum for L .

Original: Proposition 7 shows that critical points for L such that rk( W ) <
Revised: Proposition 7 shows that critical points for L such that rk( W ) <

Original: r correspond to critical points for (cid:96) restricted to a smaller determinantal variety.
Revised: r correspond to critical points for (cid:96) restricted to a smaller determinantal variety.

Original: Using Lemma 2, it is possible to show that these points are essentially always saddles for L .
Revised: Using Lemma 2, it is possible to show that these points are essentially always saddles for L .

Original: Proposition 9.
Revised: Proposition 9.

Original: Let θ ∈ Crit ( L ) be such that rk( W ) < r , and assume that d(cid:96) ( W ) (cid:54) = 0 .
Revised: Let θ ∈ Crit ( L ) be such that rk( W ) < r , and assume that d(cid:96) ( W ) (cid:54) = 0 .

Original: Then, for any neighborhood U of θ , there exists θ (cid:48) in U such that µ d ( θ (cid:48) )
Revised: Then, for any neighborhood U of θ , there exists θ (cid:48) in U such that µ d ( θ (cid:48) )

Original: =
Revised: =

Original: W but θ (cid:48) (cid:54)∈ Crit ( L ) .
Revised: W but θ (cid:48) (cid:54)∈ Crit ( L ) .

Original: In particular, θ is a saddle point.
Revised: In particular, θ is a saddle point.

Original: Proposition 10.
Revised: Proposition 10.

Original: Let (cid:96) be any smooth convex function, and let L = (cid:96) ◦ µ d .
Revised: Let (cid:96) be any smooth convex function, and let L = (cid:96) ◦ µ d .

Original: If θ is a non-global local minimum for L , then necessarily rk( W ) =
Revised: If θ is a non-global local minimum for L , then necessarily rk( W ) =

Original: r (so θ is a pure critical point).
Revised: r (so θ is a pure critical point).

Original: In particular, L has non-global minima if and only if (cid:96) | M r has non-global minima.
Revised: In particular, L has non-global minima if and only if (cid:96) | M r has non-global minima.

Original: This statement succinctly explains many known facts on the landscape of linear networks.
Revised: This statement succinctly explains many known facts on the landscape of linear networks.

Original: For example, we recover the main result from (Laurent & von Brecht, 2017), which states that when (cid:96) is a smooth convex function and µ d is ﬁlling ( r = min { d h , d 0 } ), then all local minima for L are global minima: indeed, this is because M r = R d h × d 0 is a linear space, so (cid:96) | M r does not have non-global minima.
Revised: For example, we recover the main result from (Laurent & von Brecht, 2017), which states that when (cid:96) is a smooth convex function and µ d is ﬁlling ( r = min { d h , d 0 } ), then all local minima for L are global minima: indeed, this is because M r = R d h × d 0 is a linear space, so (cid:96) | M r does not have non-global minima.

Original: On the other hand, when µ d is not ﬁlling, the functional space is not convex, and multiple local minima may exist even when (cid:96) is a convex function.
Revised: On the other hand, when µ d is not ﬁlling, the functional space is not convex, and multiple local minima may exist even when (cid:96) is a convex function.

Original: We will in fact present many examples of smooth convex functions (cid:96) such that L = (cid:96) ◦ µ d has non-global local minima (see Figure 3).
Revised: We will in fact present many examples of smooth convex functions (cid:96) such that L = (cid:96) ◦ µ d has non-global local minima (see Figure 3).

Original: In the special case that (cid:96) is a quadratic loss (for any data distribution), then it is a remarkable fact that there are no non-global local minima even when µ d is not ﬁlling (Baldi & Hornik, 1989; Kawaguchi, 2016).
Revised: In the special case that (cid:96) is a quadratic loss (for any data distribution), then it is a remarkable fact that there are no non-global local minima even when µ d is not ﬁlling (Baldi & Hornik, 1989; Kawaguchi, 2016).

Original: In the next section, we will provide an intrinsic geometric justiﬁcation for this property.
Revised: In the next section, we will provide an intrinsic geometric justiﬁcation for this property.

Original: Remark 11.
Revised: Remark 11.

Original: In Laurent & von Brecht (2017), the authors observe that their “structural hypothesis” ( i.e. , for us, the fact that the network is ﬁlling) is a necessary assumption for their main result, as otherwise critical points of (cid:96) might not lie in the functional space of the network.
Revised: In Laurent & von Brecht (2017), the authors observe that their “structural hypothesis” ( i.e. , for us, the fact that the network is ﬁlling) is a necessary assumption for their main result, as otherwise critical points of (cid:96) might not lie in the functional space of the network.

Original: This last observation however does not imply the necessity of the ﬁlling assumption, and indeed in the case of the quadratic loss there are no local bad minima despite the fact M r (cid:40) R d h × d 0 .
Revised: This last observation however does not imply the necessity of the ﬁlling assumption, and indeed in the case of the quadratic loss there are no local bad minima despite the fact M r (cid:40) R d h × d 0 .

Original: Finally, we conclude this section by pointing out that although the pure critical locus is determined by the geometry of the functional space, the “lift” from function space to parameter space is not completely trivial.
Revised: Finally, we conclude this section by pointing out that although the pure critical locus is determined by the geometry of the functional space, the “lift” from function space to parameter space is not completely trivial.

Original: In particular, there is always a large positive-dimensional set of critical parameters associated with a critical linear function W (all possible d -factorizations of W ).
Revised: In particular, there is always a large positive-dimensional set of critical parameters associated with a critical linear function W (all possible d -factorizations of W ).

Original: More interestingly, this set may be topologically disconnected into a large number of components that are all functionally equivalent (see Theorem 5).
Revised: More interestingly, this set may be topologically disconnected into a large number of components that are all functionally equivalent (see Theorem 5).

Original: This observation agrees with the folklore knowledge that neural networks can have many disconnected valleys where the loss function achieves the same value.
Revised: This observation agrees with the folklore knowledge that neural networks can have many disconnected valleys where the loss function achieves the same value.

Original: 3. T HE QUADRATIC LOSS
Revised: 3. T HE QUADRATIC LOSS

Original: We now assume that (cid:96) : R d y × d x → R is of the form (cid:96) ( W )
Revised: We now assume that (cid:96) : R d y × d x → R is of the form (cid:96) ( W )

Original: = (cid:107) WX − Y (cid:107) 2 , where X ∈ R d x × s and
Revised: = (cid:107) WX − Y (cid:107) 2 , where X ∈ R d x × s and

Original: Y ∈ R d y × s are ﬁxed data matrices.
Revised: Y ∈ R d y × s are ﬁxed data matrices.

Original: As mentioned above, it is known that L = (cid:96) ◦ µ d has no non-global local minima, even when µ d is non-ﬁlling (Baldi & Hornik, 1989; Kawaguchi, 2016).
Revised: As mentioned above, it is known that L = (cid:96) ◦ µ d has no non-global local minima, even when µ d is non-ﬁlling (Baldi & Hornik, 1989; Kawaguchi, 2016).

Original: In this section, we discuss the intrinsic geometric reasons for this special behavior.
Revised: In this section, we discuss the intrinsic geometric reasons for this special behavior.

Original: It is easy to relate the landscape of L with the Euclidean distance function from a determinantal variety (or, equivalently, to the problem low-rank matrix approximation ).
Revised: It is easy to relate the landscape of L with the Euclidean distance function from a determinantal variety (or, equivalently, to the problem low-rank matrix approximation ).

Original: Indeed, we know from
Revised: Indeed, we know from

Original: Proposition 10 that L has non-global local minima if and only if the same is true for (cid:96) | M r .
Revised: Proposition 10 that L has non-global local minima if and only if the same is true for (cid:96) | M r .

Original: Furthermore, assuming  XX T = Id (“whitened data”), we have that
Revised: Furthermore, assuming that XX T has full rank, we use its square root P = ( XX T ) 1 / 2 as a positive deﬁnite matrix to derive

Original: [Equation]
Revised: [Equation]

Original: In other words, we are interested in studying the landscape of the function h Q 0 ( W )

Original: = (cid:107) W − Q 0 (cid:107) 2 , where Q 0 ∈ R d y ×
Revised: where Q 0 = Y X T P − 1 and “const.”

Original: d x is ﬁxed and W is restricted to the determinantal variety M r .
Revised: only depends on the data matrices X and Y .

Original: Although (6) required XX T = Id , our general analysis actually still applies if XX T only has full rank.
Revised: Hence, minimizing (cid:96) ( W ) is equivalent to minimizing (cid:107) WP − Q 0 (cid:107) 2 . Since the bijection M r → M r , W (cid:55)→ WP is also a bijective on the tangent spaces, it provides a one-to-one correspondence from the critical points of min W ∈M r (cid:107) WP − Q 0 (cid:107) 2 to the critical points of min W ∈M r (cid:107) W − Q 0 (cid:107) 2 . All in all, studying the critical points of (cid:96) | M r is equivalent to studying the critical points of the function h Q 0 ( W ) = (cid:107) W − Q 0 (cid:107) 2 where W is restricted to the determinantal variety M r .

Original: The function h Q 0 ( W ) is described by following generalization of the classical Eckart-Young Theorem.
Revised: The function h Q 0 ( W ) is described by following generalization of the classical Eckart-Young Theorem.

Original: The formulation we prove is an extension of Example 2.3 in Draisma & Horobet (2014) and
Revised: The formulation we prove is an extension of Example 2.3 in Draisma & Horobet (2014) and

Original: Theorem 2.9 in Ottaviani et al.
Revised: Theorem 2.9 in Ottaviani et al.

Original: We consider a ﬁxed matrix Q 0 ∈ R d y × d x and a singular value decomposition (SVD) Q 0 = U Σ V T , where we assume Σ ∈ R d y ×
Revised: We consider a ﬁxed matrix Q 0 ∈ R d y × d x and a singular value decomposition (SVD) Q 0 = U Σ V T , where we assume Σ ∈ R d y ×

Original: d x has decreasing diagonal entries σ 1 , . . .
Revised: d x has decreasing diagonal entries σ 1 , . . .

Original: , σ m , with m = min( d y , d x ) .
Revised: , σ m , with m = min( d y , d x ) .

Original: For any I ⊂ { 1 , 2 , . . .
Revised: For any I ⊂ { 1 , 2 , . . .

Original: , m } we write Σ I ∈ R d y × d x for the diagonal matrix with entries σ I , 1 , . . .
Revised: , m } we write Σ I ∈ R d y × d x for the diagonal matrix with entries σ I , 1 , . . .

Original: , σ I ,m where σ I ,i =
Revised: , σ I ,m where σ I ,i =

Original: σ i if i ∈ I and σ I ,i = 0 otherwise.
Revised: σ i if i ∈ I and σ I ,i = 0 otherwise.

Original: Theorem 12.
Revised: Theorem 12.

Original: If the singular values of Q 0 are pairwise distinct and positive, h Q 0 | M r has exactly (cid:0) mr (cid:1) critical points, namely the matrices Q
Revised: If the singular values of Q 0 are pairwise distinct and positive, h Q 0 | M r has exactly (cid:0) mr (cid:1) critical points, namely the matrices Q

Original: I = U Σ
Revised: I = U Σ

Original: I V T with #( I ) =
Revised: I V T with #( I ) =

Original: r .
Revised: r .

Original: Moreover, its unique local and global minimum is Q { 1 ,...,r } .
Revised: Moreover, its unique local and global minimum is Q { 1 ,...,r } .

Original: More precisely, the index of Q I as a critical point of h Q 0 | M r ( i.e. , the number of negative eigenvalues of the Hessian matrix for any local parameterization) isindex( Q I ) =
Revised: More precisely, the index of Q I as a critical point of h Q 0 | M r ( i.e. , the number of negative eigenvalues of the Hessian matrix for any local parameterization) is

Original: # { ( j, i ) ∈
Revised: [Equation]

Original: I × I c | j > i } , where I c = { 1 , . ..
Revised: where I c = { 1 , . .

Original: , m } \ I .
Revised: , m } \ I .

Original: In the appendix we present a more general version of this statement without the assumption that the singular values of Q 0 are pairwise distinct and positive.
Revised: In the appendix we present a more general version of this statement without the assumption that the singular values of Q 0 are pairwise distinct and positive.

Original: The surprising aspect of this result is that the structure of the critical points is the same for almost all choices of Q 0 .
Revised: The surprising aspect of this result is that the structure of the critical points is the same for almost all choices of Q 0 .

Original: We want to emphasize that this is a special behavior of determinantal varieties with respect to the Euclidean distance, and the situation changes drastically if we apply even inﬁnitesimal changes to the quadratic loss function.
Revised: We want to emphasize that this is a special behavior of determinantal varieties with respect to the Euclidean distance, and the situation changes drastically if we apply even inﬁnitesimal changes to the quadratic loss function.

Original: More precisely, any linear perturbation of the Euclidean norm will result in a totally different landscape, as the following example shows (more details are given in Appendix A.2).
Revised: More precisely, any linear perturbation of the Euclidean norm will result in a totally different landscape, as the following example shows (more details are given in Appendix A.2).

Original: Example 13.
Revised: Example 13.

Original: Let us consider the variety M 1 ⊆ R 3 × 3 of rank-one (3 × 3) -matrices.
Revised: Let us consider the variety M 1 ⊆ R 3 × 3 of rank-one (3 × 3) -matrices.

Original: By Theorem 12, for almost all Q 0 , the function h Q 0 | M 1 has three (real) critical points.
Revised: By Theorem 12, for almost all Q 0 , the function h Q 0 | M 1 has three (real) critical points.

Original: Applying a linear change of coordinates to R d y × d x ∼ =
Revised: Applying a linear change of coordinates to R d y × d x ∼ =

Original: R d y d x yields a different quadratic loss ˜ h Q 0 .
Revised: R d y d x yields a different quadratic loss ˜ h Q 0 .

Original: Using tools from algebraic geometry, it is possible to show that for almost all linear coordinate changes (an open dense set), the function ˜ h Q 0 | M 1 has 39 critical points over the complex numbers.
Revised: Using tools from algebraic geometry, it is possible to show that for almost all linear coordinate changes (an open dense set), the function ˜ h Q 0 | M 1 has 39 critical points over the complex numbers.

Original: The number of real critical points however varies, depending on whether Q 0 belongs to different open regions separated by a caustic hypersurface in R 3 × 3 .
Revised: The number of real critical points however varies, depending on whether Q 0 belongs to different open regions separated by a caustic hypersurface in R 3 × 3 .

Original: Furthermore, the number of local minima varies as well; in particular, it is no longer true that all Q 0 admit a unique local minimum.
Revised: Furthermore, the number of local minima varies as well; in particular, it is no longer true that all Q 0 admit a unique local minimum.

Original: Figure 3 presents some simple computational experiments illustrating this behavior.
Revised: Figure 3 presents some simple computational experiments illustrating this behavior.

Original: For all determinantal varieties, the situation is similar to the description in Example 13.
Revised: For all determinantal varieties, the situation is similar to the description in Example 13.

Original: More generally, given an algebraic variety V ⊂ R n and a point u ∈ R n , the number of (real) critical points of the distance function h u ( p ) =
Revised: More generally, given an algebraic variety V ⊂ R n and a point u ∈ R n , the number of (real) critical points of the distance function h u ( p ) =

Original: (cid:107) p − u (cid:107) 2 restricted to V is usually not constant as u varies: the behavior changes when u crosses the caustic hypersurface , or ED (Euclidean distance) discriminant , of V ; see Figure 2.
Revised: (cid:107) p − u (cid:107) 2 restricted to V is usually not constant as u varies: the behavior changes when u crosses the caustic hypersurface , or ED (Euclidean distance) discriminant , of V ; see Figure 2.

Original: In the case of determinantal varieties with the standard Euclidean distance, this caustic hypersurface (more precisely its real locus) degenerates to a set of codimension 2 , which does not partition the space into different regions.
Revised: In the case of determinantal varieties with the standard Euclidean distance, this caustic hypersurface (more precisely its real locus) degenerates to a set of codimension 2 , which does not partition the space into different regions.

Original: This is analogous to the case of the circle in Figure 2. 
Revised: This is analogous to the case of the circle in Figure 2. [Figure]

Revised: ple 13.

Revised: The size of each disk is proportional to the number of instances we found with that number of critical points and local minima.

Revised: This shows that linear networks with a convex loss may indeed have multiple non-global local minima.

Revised: More details in Appendix A.2 (Table 2 and Experiment 1).

Original: 3. U SING DIFFERENT PARAMETERIZATIONS : NORMALIZED NETWORKS
Revised: 3. U SING DIFFERENT PARAMETERIZATIONS : NORMALIZED NETWORKS

Original: In the simple linear network model (2), the functional space M r ⊂ R d h × d 0 is parameterized using the matrix multiplication map µ d .
Revised: In the simple linear network model (2), the functional space M r ⊂ R d h × d 0 is parameterized using the matrix multiplication map µ d .

Original: On the other hand, one can envision many variations of this model that are network architectures with the same functional space but parameterized differently.
Revised: On the other hand, one can envision many variations of this model that are network architectures with the same functional space but parameterized differently.

Original: Examples include linear networks with skip connections, or convolutional linear networks.
Revised: Examples include linear networks with skip connections, or convolutional linear networks.

Original: In this subsection, we take a look at a model for normalized linear networks : these are maps of the form
Revised: In this subsection, we take a look at a model for normalized linear networks : these are maps of the form

Original: [Equation]
Revised: [Equation]

Original: where W i ∈ R d i × d i − 1 as before.
Revised: where W i ∈ R d i × d i − 1 as before.

Original: This is a simple model for different types of weight normalization schemes often used in practice.
Revised: This is a simple model for different types of weight normalization schemes often used in practice.

Original: It is easy to see that the difference between (7) and our previous[Figure] nu m be r o f c r i t i c a l po i n t s linear network lies only in the parameterization of linear maps, since for normalized networks the matrix multiplication map is replaced by
Revised: It is easy to see that the difference between (6) and our previous linear network lies only in the parameterization of linear maps, since for normalized networks the matrix multiplication map is replaced by

Original: [Equation]
Revised: [Equation]

Original: where Ω = { ( W h , . . .
Revised: where Ω = { ( W h , . . .

Original: , W 1 ) | W i (cid:54)
Revised: , W 1 ) | W i (cid:54)

Original: = 0 ,
Revised: = 0 ,

Original: i = 1 , . .
Revised: i = 1 , . .

Original: , h − 1 } ⊂ R d θ .
Revised: , h − 1 } ⊂ R d θ .

Original: According to our deﬁnitions, if
Revised: According to our deﬁnitions, if

Original: L = (cid:96) ◦ µ d and L (cid:48)
Revised: L = (cid:96) ◦ µ d and L (cid:48)

Original: = (cid:96) ◦ ν d are losses respectively for linear networks and normalized linear networks, then the pure critical loci of L and L (cid:48) will correspond to each other (since these only depend on the functional space), but a priori the spurious critical loci induced by the two parameterizations may be different.
Revised: = (cid:96) ◦ ν d are losses respectively for linear networks and normalized linear networks, then the pure critical loci of L and L (cid:48) will correspond to each other (since these only depend on the functional space), but a priori the spurious critical loci induced by the two parameterizations may be different.

Original: In this particular setting, however, we show that this is not the case: the new paramerization effectively does not introduce different critical points, and in fact makes the critical locus slightly smaller.
Revised: In this particular setting, however, we show that this is not the case: the new paramerization effectively does not introduce different critical points, and in fact makes the critical locus slightly smaller.

Original: Proposition 14.
Revised: Proposition 14.

Original: If L (cid:48) =
Revised: If L (cid:48) =

Original: (cid:96) ◦
Revised: (cid:96) ◦

Original: ν
Revised: ν

Original: d and L = (cid:96) ◦ µ d , then the critical locus Crit ( L (cid:48) ) is in “correspondence” with Crit ( L )
Revised: d and L = (cid:96) ◦ µ d , then the critical locus Crit ( L (cid:48) ) is in “correspondence” with Crit ( L )

Original: ∩ Ω , meaning that
Revised: ∩ Ω , meaning that

Original: [Equation]
Revised: [Equation]

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We have introduced the notions of pure and spurious critical points as general tools for a geometric investigation of the landscape of neural networks.
Revised: We have introduced the notions of pure and spurious critical points as general tools for a geometric investigation of the landscape of neural networks.

Original: In particular, they provide a basic language for describing the interplay between a convex loss function and an overparameterized, non-convex functional space .
Revised: In particular, they provide a basic language for describing the interplay between a convex loss function and an overparameterized, non-convex functional space .

Original: In this paper, we have focused on the landscape of linear networks.
Revised: In this paper, we have focused on the landscape of linear networks.

Original: This simple model is useful for illustrating our geometric perspective, but also exhibits several interesting (and surprisingly subtle) features.
Revised: This simple model is useful for illustrating our geometric perspective, but also exhibits several interesting (and surprisingly subtle) features.

Original: For example, the absence of non-global minima in the loss landscape is a rather general property when the architecture is “ﬁlling”, while in the “non-ﬁlling” setting it is a special property that holds for the quadratic loss.
Revised: For example, the absence of non-global minima in the loss landscape is a rather general property when the architecture is “ﬁlling”, while in the “non-ﬁlling” setting it is a special property that holds for the quadratic loss.

Original: Furthermore, we have observed that even in this simple framework global minima can have (possibly exponentially) many disconnected components.
Revised: Furthermore, we have observed that even in this simple framework global minima can have (possibly exponentially) many disconnected components.

Original: In the future, we hope to extend our analysis to different network models.
Revised: In the future, we hope to extend our analysis to different network models.

Original: For example, we can use our framework to study networks with polynomial activations (Kileel et al., 2019), which are a direct generalization of the linear model.
Revised: For example, we can use our framework to study networks with polynomial activations (Kileel et al., 2019), which are a direct generalization of the linear model.

Revised: We expect that an analysis of pure and spurious critical points in this context can be used to address a conjecture in Venturi et al.

Revised: (2018) regarding the gap between “upper” and “lower” dimensions in functional space.

Original: A geometric investigation of networks with smooth non-polynomial activations is also possible; in that setting, the parameter space and the functional space are usually of the same dimension ( i.e. , d θ = dim( M Φ ) ), however there is still an interesting stratiﬁcation of singular loci, as explained for example in (Amari, 2016, Section 12.2.2).
Revised: A geometric investigation of networks with smooth non-polynomial activations is also possible; in that setting, the parameter space and the functional space are usually of the same dimension ( i.e. , d θ = dim( M Φ ) ), however there is still an interesting stratiﬁcation of singular loci, as explained for example in (Amari, 2016, Section 12.2.2).

Original: General “discriminant hypersurfaces” can also be used to describe qualitative changes in the landscape as the data distribution varies.
Revised: General “discriminant hypersurfaces” can also be used to describe qualitative changes in the landscape as the data distribution varies.

Original: Finally, extending our analysis to networks with ReLU activations will require some care because of the non-differentiable setting.
Revised: Finally, extending our analysis to networks with ReLU activations will require some care because of the non-differentiable setting.

Original: On the other hand, it is clear that ReLU networks behave as linear networks when restricted to appropriate regions of input space: this suggests that our study of ranks of differentials may be a useful building block for pursuing in this important direction. 
Revised: On the other hand, it is clear that ReLU networks behave as linear networks when restricted to appropriate regions of input space: this suggests that our study of ranks of differentials may be a useful building block for pursuing in this important direction. Acknowledgements.

Revised: We thank James Mathews for many helpful discussions in the beginning of this project.

Revised: We are gratuful to ICERM (NSF DMS-1439786 and the Simons Foundation grant 507536) for the hospitality during the academic year 2018/2019 where many ideas for this project were developed.

Revised: MT and JB were partially supported by the Alfred P. Sloan Foundation, NSF RI1816753, NSF CAREER CIF 1845360, and Samsung Electronics.

Revised: KK was partially supported by the Knut and Alice Wallenbergs Foundation within their WASP AI/Math initiative.

Original: A A PPENDIX A. D ETERMINANTAL VARIETIES
Revised: A A PPENDIX A. D ETERMINANTAL VARIETIES

Original: We present some additional properties of determinantal varieties.
Revised: We present some additional properties of determinantal varieties.

Original: For proofs and more details, we refer the reader to Harris (1995).
Revised: For proofs and more details, we refer the reader to Harris (1995).

Original: Given r < min( m, n ) , the r -th determinantal variety M r ⊂ R m × n is deﬁned as the set of matrices with rank at most r :
Revised: Given r < min( m, n ) , the r -th determinantal variety M r ⊂ R m × n is deﬁned as the set of matrices with rank at most r :

Original: [Equation]
Revised: [Equation]

Original:  As mentioned in the main part of the paper, M r is an algebraic variety of dimension r ( m + n − r ) , that can be described as the zero-set of all ( r + 1) ×
Revised: } ⊂ R m × n . As mentioned in the main part of the paper, M r is an algebraic variety of dimension r ( m + n − r ) , that can be described as the zero-set of all ( r + 1) ×

Original: ( r + 1) minors.
Revised: ( r + 1) minors.

Original: For r > 0 , the the singular locus of M r is exactly M r − 1 ⊂ M r .
Revised: For r > 0 , the the singular locus of M r is exactly M r − 1 ⊂ M r .

Original: Some of our proofs will rely on the following explicit characterization of tangent space of determinantal varieties: given a a matrix P ∈ R m × n of rank exactly r
Revised: Some of our proofs will rely on the following explicit characterization of tangent space of determinantal varieties: given a a matrix P ∈ R m × n of rank exactly r

Original: (so P is a smooth point on M r )
Revised: (so P is a smooth point on M r )

Original: we have that
Revised: we have that

Original: [Equation]
Revised: [Equation]

Original: We will also make use of the normal space to the tangent space T P M r at P , with respect to the
Revised: We will also make use of the normal space to the tangent space T P M r at P , with respect to the

Original: Frobenius inner product.
Revised: Frobenius inner product.

Original: This is given by
Revised: This is given by

Original: [Equation]
Revised: [Equation]

Original: where Col ( P ) ⊥ and Row ( P ) ⊥ are the orthogonal spaces to Col ( P ) and Row ( P ) , respectively.
Revised: where Col ( P ) ⊥ and Row ( P ) ⊥ are the orthogonal spaces to Col ( P ) and Row ( P ) , respectively.

Original: A. E UCLIDEAN DISTANCE DEGREES AND DISCRIMINANTS
Revised: A. E UCLIDEAN DISTANCE DEGREES AND DISCRIMINANTS

Original: In this section, we informally discuss some algebraic notions related to ED (Euclidean distance) degrees and discriminants.
Revised: In this section, we informally discuss some algebraic notions related to ED (Euclidean distance) degrees and discriminants.

Original: A detailed presentation can be found in Draisma et al.
Revised: A detailed presentation can be found in Draisma et al.

Original: Given an algebraic variety V ⊂ R n and a point u ∈ R n , the number of real critical points of the distance function h u ( p ) =
Revised: Given an algebraic variety V ⊂ R n and a point u ∈ R n , the number of real critical points of the distance function h u ( p ) =

Original: (cid:107) p − u (cid:107) 2 restricted to V is only locally constant as u varies.
Revised: (cid:107) p − u (cid:107) 2 restricted to V is only locally constant as u varies.

Original: In general, the behavior changes when u crosses the caustic hypersurface , or ED (Euclidean distance) discriminant , of V .
Revised: In general, the behavior changes when u crosses the caustic hypersurface , or ED (Euclidean distance) discriminant , of V .

Original: The ED discriminant can be deﬁned over the complex numbers, and in this setting it is indeed always a hypersurface ( i.e. , it has codimension one), however it can have higher codimension over the real numbers.
Revised: The ED discriminant can be deﬁned over the complex numbers, and in this setting it is indeed always a hypersurface ( i.e. , it has codimension one), however it can have higher codimension over the real numbers.

Original: For instance, for a circle in the complex plane with the origin as its center, a point ( u 1 , u 2 ) ∈
Revised: For instance, for a circle in the complex plane with the origin as its center, a point ( u 1 , u 2 ) ∈

Original: C 2 is on the ED discriminant if and only if u 21 + u 22 = 0 .
Revised: C 2 is on the ED discriminant if and only if u 21 + u 22 = 0 .

Original: This deﬁnes a curve in the complex plane whose real locus is a point (see right side of Figure 2).
Revised: This deﬁnes a curve in the complex plane whose real locus is a point (see right side of Figure 2).

Original: By the Eckart-Young Theorem (Theorem 12), the ED discriminant of the determinantal variety M r is the locus of all matrices Qwith at least two coinciding singular values, so it is deﬁned by the discriminant of Q 0 Q T 0 .
Revised: By the Eckart-Young Theorem (Theorem 12), the ED discriminant of the determinantal variety M r is the locus of all matrices Q with at least two coinciding singular values, so it is deﬁned by the discriminant of Q 0 Q T 0 .

Original: As in the case of the circle, the ED discriminant of M r has codimension two in R d y × d x .
Revised: As in the case of the circle, the ED discriminant of M r has codimension two in R d y × d x .

Original: Over the complex numbers, the number of critical points of the distance function h u restricted to V is actually the same for every point u ∈ C n not on the ED discriminant of V .
Revised: Over the complex numbers, the number of critical points of the distance function h u restricted to V is actually the same for every point u ∈ C n not on the ED discriminant of V .

Original: This quantity is known as the ED degree of the variety V .
Revised: This quantity is known as the ED degree of the variety V .

Original: For instance, a circle has ED degree two whereas an ellipse has ED degree four (on the left side of Figure 2, points u outside of the caustic curve yield two real critical points and two imaginary critical points).
Revised: For instance, a circle has ED degree two whereas an ellipse has ED degree four (on the left side of Figure 2, points u outside of the caustic curve yield two real critical points and two imaginary critical points).

Original: The Eckart-Young Theorem (Theorem 12) tells us that
Revised: The Eckart-Young Theorem (Theorem 12) tells us that

Original: the
Revised: the

Original: ED degree of the determinantal variety M r ⊂ R d y × d x is (cid:0) mr (cid:1) where m = min( d x , d y ) .
Revised: ED degree of the determinantal variety M r ⊂ R d y × d x is (cid:0) mr (cid:1) where m = min( d x , d y ) .

Original: As argued in the main part of the paper, this does not hold any longer after perturbing either the determinantal variety or the Euclidean distance slightly, even using only a linear change of coordinates.
Revised: As argued in the main part of the paper, this does not hold any longer after perturbing either the determinantal variety or the Euclidean distance slightly, even using only a linear change of coordinates.

Original: For an algebraic variety V ⊂ C n , a linear change of coordinates is given by an automorphism ϕ : C n → C n .
Revised: For an algebraic variety V ⊂ C n , a linear change of coordinates is given by an automorphism ϕ : C n → C n .

Original: For almost all such automorphisms ( i.e. , for all ϕ except those lying in some subvariety of
Revised: For almost all such automorphisms ( i.e. , for all ϕ except those lying in some subvariety of

Original: GL( n, C ) )
Revised: GL( n, C ) )

Original: the ED degree of ϕ ( V ) is the same ; see Theorem 5.4 in Draisma et al.
Revised: the ED degree of ϕ ( V ) is the same ; see Theorem 5.4 in Draisma et al.

Original: This quantity is known as the general ED degree of V .
Revised: This quantity is known as the general ED degree of V .

Original: For instance, almost all linear coordinate changes will deform a circle into an ellipse, such that the general ED degree of the circle is four.
Revised: For instance, almost all linear coordinate changes will deform a circle into an ellipse, such that the general ED degree of the circle is four.

Original: In the above deﬁnition of the general ED degree, we ﬁxed the standard Euclidean distance and perturbed the variety.
Revised: In the above deﬁnition of the general ED degree, we ﬁxed the standard Euclidean distance and perturbed the variety.

Original: Alternatively, we can ﬁx the variety and change the standard Euclidean distance (cid:107) · (cid:107) to dist ϕ
Revised: Alternatively, we can ﬁx the variety and change the standard Euclidean distance (cid:107) · (cid:107) to dist ϕ

Original: = (cid:107) ϕ
Revised: = (cid:107) ϕ

Original: ( · ) (cid:107) .
Revised: ( · ) (cid:107) .

Original: The new distance function h ϕ,u ( p ) =
Revised: The new distance function h ϕ,u ( p ) =

Original: dist ϕ ( p − u ) 2 from u satisﬁes h ϕ ( u ) ( ϕ ( p ))
Revised: dist ϕ ( p − u ) 2 from u satisﬁes h ϕ ( u ) ( ϕ ( p ))

Original: =
Revised: =

Original: h id ,ϕ ( u ) ( ϕ ( p ))
Revised: h id ,ϕ ( u ) ( ϕ ( p ))

Original: =
Revised: =

Original: h ϕ,u ( p ) .
Revised: h ϕ,u ( p ) .

Original: Hence, the ED degree of ϕ ( V ) with respect to the standard Euclidean distance dist id = (cid:107) · (cid:107) equals the ED degree of V with respect to the perturbed
Revised: Hence, the ED degree of ϕ ( V ) with respect to the standard Euclidean distance dist id = (cid:107) · (cid:107) equals the ED degree of V with respect to the perturbed

Original: Euclidean distance dist ϕ .
Revised: Euclidean distance dist ϕ .

Original: In particular, the general ED degree of V can be obtained by computing the ED degree after applying a sufﬁciently random linear change of coordinates on either the
Revised: In particular, the general ED degree of V can be obtained by computing the ED degree after applying a sufﬁciently random linear change of coordinates on either the

Original: Euclidean distance or the variety V itself.
Revised: Euclidean distance or the variety V itself.

Original: As in the case of a circle, the general ED degree of the determinantal variety M r is not equal to the
Revised: As in the case of a circle, the general ED degree of the determinantal variety M r is not equal to the

Original: ED degree of M r .
Revised: ED degree of M r .

Original: Furthermore, there is no known closed formula for the general ED degree of M r only involving the parameters d x , d y and r .
Revised: Furthermore, there is no known closed formula for the general ED degree of M r only involving the parameters d x , d y and r .

Original: In the special case of rank-one matrices, one can derive a closed expression from the Catanese-Trifogli formula (Theorem 7.8 in Draisma et al.
Revised: In the special case of rank-one matrices, one can derive a closed expression from the Catanese-Trifogli formula (Theorem 7.8 in Draisma et al.

Original: ): the general ED degree of M 1 is
Revised: ): the general ED degree of M 1 is

Original: [Equation]
Revised: [Equation]

Original: This expression yields 39 for d x = d y = 3 , as mentioned in Example 13.
Revised: This expression yields 39 for d x = d y = 3 , as mentioned in Example 13.

Original: For general r , formulas for the general ED degree of M r involving Chern and polar classes can be found in Ottaviani et al.
Revised: For general r , formulas for the general ED degree of M r involving Chern and polar classes can be found in Ottaviani et al.

Original: ; Draisma et al.
Revised: ; Draisma et al.

Original: A short algorithm to compute the general ED degree of M r is given in Example 7.11 of Draisma et al.
Revised: A short algorithm to compute the general ED degree of M r is given in Example 7.11 of Draisma et al.

Original: ; it uses a package for advanced intersection theory in the algebro-geometric software Macaulay

Revised: ; it uses a package for advanced intersection theory in the algebro-geometric software Macaulay

Original: (Grayson & Stillman, 2019).
Revised: (Grayson & Stillman, 2019).

Original: This discussion shows that the Eckart-Young Theorem is indeed very special.
Revised: This discussion shows that the Eckart-Young Theorem is indeed very special.

Original: The intrinsic reason for this is that the determinantal variety M r intersects the “isotropic quadric” associated with the standard Euclidean distance ( i.e. , zero locus of X 2 1 , 1 + . . .
Revised: The intrinsic reason for this is that the determinantal variety M r intersects the “isotropic quadric” associated with the standard Euclidean distance ( i.e. , zero locus of X 21 , 1 + . . .

Original: + X 2 d x ,d y in C d x × d y ) in a particular way ( i.e. , non-transversely).
Revised: + X 2 d x ,d y in C d x × d y ) in a particular way ( i.e. , non-transversely).

Original: Performing a random linear change of coordinates on either M r or the isotropic quadric makes the intersection transverse.
Revised: Performing a random linear change of coordinates on either M r or the isotropic quadric makes the intersection transverse.

Original: So the ED degree after the linear change of coordinates is the general ED degree of M r , and the Eckart-Young Theorem does not apply.
Revised: So the ED degree after the linear change of coordinates is the general ED degree of M r , and the Eckart-Young Theorem does not apply.

Original: In summary, we have observed that the degeneration from an ellipse to a circle is analogous to the degeneration from a determinantal variety with a perturbed Euclidean distance to the determinantal variety with the standard Euclidean distance: in both cases, the ED degree drops because the situation becomes degenerate.
Revised: In summary, we have observed that the degeneration from an ellipse to a circle is analogous to the degeneration from a determinantal variety with a perturbed Euclidean distance to the determinantal variety with the standard Euclidean distance: in both cases, the ED degree drops because the situation becomes degenerate.

Original: Moreover, the ED discriminant drops dimension, which causes the special phenomenon that the number of real critical points is almost everywhere the same.
Revised: Moreover, the ED discriminant drops dimension, which causes the special phenomenon that the number of real critical points is almost everywhere the same.

Original: Experiment 1.
Revised: Experiment 1.

Original: In general, it is very difﬁcult to describe the open regions in R n that are separated by the ED discriminant of a variety V ⊂ R n .
Revised: In general, it is very difﬁcult to describe the open regions in R n that are separated by the ED discriminant of a variety V ⊂ R n .

Original: Also ﬁnding the “typical” number of real critical points for the distance function h u restricted to V , requires the computation of the volumes of these open regions.
Revised: Finding the “typical” number of real critical points for the distance function h u restricted to V , requires the computation of the volumes of these open regions.

Original: In the current state of the art in real algebraic geometry, this is only possible for very particular varieties V .
Revised: In the current state of the art in real algebraic geometry, this is only possible for very particular varieties V .

Original: Even for the average number of real critical points there is no general computational technique.

Original: For these reasons, and to get more insights on the typical number of real critical points of determinantal varieties with a perturbed Euclidean distance, we performed computational experiments with Macaulay2 (Grayson & Stillman, 2019) in the situation of Example 13.
Revised: For these reasons, and to get more insights on the typical number of real critical points of determinantal varieties with a perturbed Euclidean distance, we performed computational experiments with Macaulay2 (Grayson & Stillman, 2019) in the situation of Example 13.

Original: We ﬁxed the  determinantal variety M 1 ⊆ R 3 × 3 of rank-one (3 × 3)
Revised: We ﬁxed the [Table] determinantal variety M 1 ⊆ R 3 × 3 of rank-one (3 × 3)

Original: -matrices.
Revised: -matrices.

Original: In each iteration of the experiment, we picked a random automorphism ϕ :
Revised: In each iteration of the experiment, we picked a random automorphism ϕ :

Original: R 3 × 3 → R 3 × 3 and a random matrix Q 0 ∈ R 3 × 3 .
Revised: R 3 × 3 → R 3 × 3 and a random matrix Q 0 ∈ R 3 × 3 .

Original: We ﬁrst veriﬁed that the number of complex critical points of the perturbed quadratic distance function h ϕ,Q 0 restricted to M 1 is the expected number 39 .
Revised: We ﬁrst veriﬁed that the number of complex critical points of the perturbed quadratic distance function h ϕ,Q 0 restricted to M 1 is the expected number 39 .

Original: After that, we computed the number of real critical points and the number of local minima among them.
Revised: After that, we computed the number of real critical points and the number of local minima among them.

Original: Our results foriterations can be found in Table 1 and Figure 3.
Revised: Our results for 2000 iterations can be found in Table 2 and Figure 3.

Original: Although this is a very rudimentary experiment in an extremely simple setting, it provides clear evidence that the number of local minima of the perturbed distance function is generally not one.
Revised: Although this is a very rudimentary experiment in an extremely simple setting, it provides clear evidence that the number of local minima of the perturbed distance function is generally not one.

Original: Implementation details: We note that our computations of real critical points and local minima involved numerical methods and might thus be affected by numerical errors.
Revised: Implementation details: We note that our computations of real critical points and local minima involved numerical methods and might thus be affected by numerical errors.

Original: In our implementation we used several basic tests to rule out numerically bad iterations, so that we can report our results with high conﬁdence.
Revised: In our implementation we used several basic tests to rule out numerically bad iterations, so that we can report our results with high conﬁdence.

Original: The entries of the random matrix Q 0 are independently and uniformly chosen among the integers in Z = {− 10 , − 9 , . .
Revised: The entries of the random matrix Q 0 are independently and uniformly chosen among the integers in Z = {− 10 , − 9 , . .

Original: The random automorphism ϕ is given by a matrix in Z 9 × 9 whose entries are also chosen independently and uniformly at random.
Revised: The random automorphism ϕ is given by a matrix in Z 9 × 9 whose entries are also chosen independently and uniformly at random.

Original: A. P URE AND SPURIOUS CRITICAL POINTS IN PREDICTOR SPACE
Revised: A. P URE AND SPURIOUS CRITICAL POINTS IN PREDICTOR SPACE

Original: We illustrate a variation of our functional setting where the notions of pure and spurious can also be naturally applied.
Revised: We illustrate a variation of our functional setting where the notions of pure and spurious can also be naturally applied.

Original: We consider a training sample x 1 , . . .
Revised: We consider a training sample x 1 , . . .

Original: , x N ∈ R d x , y 1 , . . .
Revised: , x N ∈ R d x , y 1 , . . .

Original: , y N ∈ R (for notational simplicity we useR d y = 

Revised: , y N ∈ R (for notational simplicity we use d y =

Original: but this is not necessary).
Revised: but this is not necessary).

Original: We then write an empirical risk of the form
Revised: We then write an empirical risk of the form

Original: L ( θ ) =

Original: g ( ˆ Y ( θ ) , Y ) ,

Original: [Table]
Revised: [Equation]

Original: where ˆ Y ( θ ) = (Φ( θ, x 1 ) , . .
Revised: where ˆ Y ( θ ) = (Φ( θ, x 1 ) , . .

Original: Φ( θ, x N )) ∈ R N , Y =
Revised: Φ( θ, x N )) ∈ R N , Y =

Original: ( y 1 , . . .
Revised: ( y 1 , . . .

Original: , y N ) ∈ R N
Revised: , y N ) ∈ R N

Original: and g : R N × R N → R is a convex function.
Revised: and g : R N × R N → R is a convex function.

Original: As θ varies, ˆ Y ( θ ) deﬁnes a “predictor manifold” Y ⊂ R N , which depends only on the input data x 1 , . .
Revised: As θ varies, ˆ Y ( θ ) deﬁnes a “predictor manifold” Y ⊂ R N , which depends only on the input data x 1 , . .

Original: , x N , but not on θ .
Revised: , x N , but not on θ .

Original: The function L ( θ ) can be naturally seen as a composition
Revised: The function L ( θ ) can be naturally seen as a composition

Original: R d θ η → Y g
Revised: [Equation]

Original: → R , where η ( θ )
Revised: where η ( θ )

Original: =
Revised: =

Original: ˆ Y ( θ ) ∈ Y .
Revised: ˆ Y ( θ ) ∈ Y .

Original: We may now distinguish again between “pure” and “spurious” critical points for L .
Revised: We may now distinguish again between “pure” and “spurious” critical points for L .

Original: In an underparameterized regime d θ < N , or if the input data x 1 , . . .
Revised: In an underparameterized regime d θ < N , or if the input data x 1 , . . .

Original: , x N is in some way special, then Y (cid:40)
Revised: , x N is in some way special, then Y (cid:40)

Original: R N is a submanifold (with singularities), and critical points may arise because we are restricting g to Y (pure), or because of the parameterization map
Revised: R N is a submanifold (with singularities), and critical points may arise because we are restricting g to Y (pure), or because of the parameterization map

Original: η (spurious).
Revised: η (spurious).

Original: In a highly overparameterized regime d θ (cid:29) N (which is usually the case in practice), we expect
Revised: In a highly overparameterized regime d θ (cid:29) N (which is usually the case in practice), we expect

Original: Y = R N .
Revised: Y = R N .

Original: This can be viewed as analogous to the “ﬁlling” situation described for linear networks in this paper.
Revised: This can be viewed as analogous to the “ﬁlling” situation described for linear networks in this paper.

Original: In particular, all critical points that are not global minima for L are necessarily spurious, since g | Y = g is convex.
Revised: In particular, all critical points that are not global minima for L are necessarily spurious, since g | Y = g is convex.

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: We ﬁrst show Lemma 3 with help of the following general observation:
Revised: We ﬁrst show Lemma 3 with help of the following general observation:

Original: Proposition 15.
Revised: Proposition 15.

Original: Let V +1 ⊆
Revised: Let V +1 ⊆

Original: V +2 ⊆ . . .
Revised: V +2 ⊆ . . .

Original: ⊆
Revised: ⊆

Original: V + h and V − 1 ⊇ V − 2 ⊇ . . .
Revised: V + h and V − 1 ⊇ V − 2 ⊇ . . .

Original: ⊇ V − h be vector spaces with dimensions r
Revised: ⊇ V − h be vector spaces with dimensions r

Original: + i :=
Revised: + i :=

Original: dim( V + i ) and r − i :=
Revised: dim( V + i ) and r − i :=

Original: dim( V − i ) for i = 1 , . .
Revised: dim( V − i ) for i = 1 , . .

Original: , h .
Revised: , h .

Original: Then we have dim (cid:0)(cid:0) [Equation] [Equation]
Revised: Then we have dim (cid:0)(cid:0) V +1 ⊗ V − 1 (cid:1)

Original: V
Revised: +

Original: [Equation]
Revised: [Equation]

Original: 1 (cid:1) +  (cid:0) V + 
Revised: V +2 ⊗ V − 2 (cid:1) + . . . + (cid:0) V + h ⊗ V

Original: [Equation]
Revised: [Equation]

Original: V
Revised: = h

Original: [Equation]
Revised: [Equation]

Original: + (cid:0) [Equation] + h ⊗ V − h
Revised: i =

Original: (cid:1)(cid:1) = h (cid:88) i =
Revised: r

Original: [Equation]
Revised: [Equation]

Original:  r
Revised: i r

Original: [Equation]
Revised: [Equation]

Original: − h
Revised: − h

Original: [Equation]
Revised: [Equation]

Original: (cid:88) i =
Revised: i [Equation]

Original: r + i r − i +1 .
Revised: r + [Equation] r − i +1 .

Original: Proof.
Revised: Proof.

Original: We prove this assertion by induction on h .
Revised: We prove this assertion by induction on h .

Original: The base case ( h = 1 ) is clear: dim( V +1 ⊗
Revised: The base case ( h = 1 ) is clear: dim( V +1 ⊗

Original: V − 1 ) =
Revised: V − 1 ) =

Original: r +1 r − 1 .
Revised: r +1 r − 1 .

Original: For the induction step, we set V :=
Revised: For the induction step, we set V :=

Original: ( V +1 ⊗ V − 1 )
Revised: ( V +1 ⊗ V − 1 )

Original: + . . .
Revised: + . . .

Original: + ( V + h − 1 ⊗ V − h − 1 ) .
Revised: + ( V + h − 1 ⊗ V − h − 1 ) .

Original: The key observation is that the inclusions V +1 ⊆
Revised: The key observation is that the inclusions V +1 ⊆

Original: V +2 ⊆ . . .
Revised: V +2 ⊆ . . .

Original: ⊆
Revised: ⊆

Original: V + h and V − 1 ⊇ V − 2 ⊇ . . .
Revised: V + h and V − 1 ⊇ V − 2 ⊇ . . .

Original: ⊇ V − h imply that
Revised: ⊇ V − h imply that

Original: V ∩ ( V + h ⊗ V − h )
Revised: V ∩ ( V + h ⊗ V − h )

Original: =
Revised: =

Original: V + h − 1 ⊗ V − h .
Revised: V + h − 1 ⊗ V − h .

Original: Hence, applying the induction hypothesis to V , we derive dim(cid:0) V + (cid:0) V + h ⊗ V − h (cid:1)(cid:1) = dim( V ) + dim( V + h ⊗ V − h ) − dim( V + h − 1 ⊗ V − h ) = (cid:32) h − 1 (cid:88) i =
Revised: Hence, applying the induction hypothesis to V , we derive dim

Original: r + i r − i − h − 2 (cid:88) i = r
Revised: [Equation]

Original: + i r − i + (cid:33) + r + h r − h − r + h − 1 r − h

Original: = h (cid:88) i =

Original: r + i r − i − h − 1 (cid:88) i

Original: =

Original: r + i r − i +1 .

Original: Lemma 3.
Revised: Lemma 3.

Original: The dimension of the image of the differential dµ d at θ =
Revised: The dimension of the image of the differential dµ d at θ =

Original: ( W h , . . .
Revised: ( W h , . . .

Original: , W 1 ) is given by
Revised: , W 1 ) is given by

Original: [Equation]
Revised: [Equation]

Original: where we use the convention that W < 1 = I d 0 , W >h
Revised: where we use the convention that W < 1 = I d 0 , W >h

Original: = I
Revised: = I

Original: d h are the identity matrices of size d 0 , d h .
Revised: d h are the identity matrices of size d 0 , d h .

Original: Proof.
Revised: Proof.

Original: The image of the differential dµ d ( θ ) is given in (5).
Revised: The image of the differential dµ d ( θ ) is given in (5).

Original: Due to
Revised: Due to

Original: [Equation]
Revised: [Equation]

Original: we can apply Proposition 15, which concludes the proof.
Revised: we can apply Proposition 15, which concludes the proof.

Original: Now we provide a proof for Theorem 4, starting from a reﬁnement of the last statement.
Revised: Now we provide a proof for Theorem 4, starting from a reﬁnement of the last statement.

Original: Proposition 16.
Revised: Proposition 16.

Original: Let r = min
Revised: Let r = min

Original: { d i } , θ =
Revised: { d i } , θ =

Original: ( W h , . . .
Revised: ( W h , . . .

Original: , W 1 ) , W = µ d ( θ ) , and e = rk W .
Revised: , W 1 ) , W = µ d ( θ ) , and e = rk W .

Original: The image of the differential dµ d at θ contains the tangent space T W M e of M e at W .
Revised: The image of the differential dµ d at θ contains the tangent space T W M e of M e at W .

Original: Furthermore, for every
Revised: Furthermore, for every

Original: W ∈ M e \M e − 1 there exists θ (cid:48)
Revised: W ∈ M e \M e − 1 there exists θ (cid:48)

Original: such that µ d ( θ (cid:48) )
Revised: such that µ d ( θ (cid:48) )

Original: =
Revised: =

Original: W and the image of dµ d ( θ (cid:48) ) is exactly T W M e .
Revised: W and the image of dµ d ( θ (cid:48) ) is exactly T W M e .

Original: Proof.
Revised: Proof.

Original: Due to (8) the image (5) of dµ d ( θ ) always contains R d h ⊗ Row ( W ) + Col ( W ) ⊗ R d 0 =
Revised: Due to (7) the image (5) of dµ d ( θ ) always contains R d h ⊗ Row ( W ) + Col ( W ) ⊗ R d 0 =

Original: T W M e .
Revised: T W M e .

Original: Furthermore, there always exists ( W h , . . .
Revised: Furthermore, there always exists ( W h , . . .

Original: , W 1 ) ∈ µ − 1 d ( W ) such that each W i has rank exactly r and the containments in (8) are all equalities.
Revised: , W 1 ) ∈ µ − 1 d ( W ) such that each W i has rank exactly r and the containments in (7) are all equalities.

Original: For example, one way to achieve this is to consider any decomposition W = UV T where U ∈ R d h × r and V = R d 0 × r
Revised: For example, one way to achieve this is to consider any decomposition W = UV T where U ∈ R d h × r and V = R d 0 × r

Original: and then set
Revised: and then set

Original: W 1 =
Revised: W 1 =

Original: [ V | 0] T , W h
Revised: [ V | 0] T , W h

Original: =
Revised: =

Original: [ U | 0] , and W i =
Revised: [ U | 0] , and W i =

Original: (cid:20) I r
Revised: (cid:20) I r

Original:   (cid:21) for 2 ≤ i ≤ h − 1 , where I r is the ( r × r ) -identity matrix and the zeros ﬁll in the dimensions ( d i × d i − 1 ) of W i .
Revised:   (cid:21) for 2 ≤ i ≤ h − 1 , where I r is the ( r × r ) -identity matrix and the zeros ﬁll in the dimensions ( d i × d i − 1 ) of W i .

Original: The next two propositions discuss the ﬁrst part of Theorem 4, which distinguishes between the ﬁlling and the non-ﬁlling case.
Revised: The next two propositions discuss the ﬁrst part of Theorem 4, which distinguishes between the ﬁlling and the non-ﬁlling case.

Original: Proposition 17.
Revised: Proposition 17.

Original: Let r = min
Revised: Let r = min

Original: { d i } and θ =
Revised: { d i } and θ =

Original: ( W h , . . .
Revised: ( W h , . . .

Original: , W 1 ) .
Revised: , W 1 ) .

Original: In the non-ﬁlling case ( i.e. , if r < min { d h , d 0 } ) we have that rk( dµ d ( θ )) <
Revised: In the non-ﬁlling case ( i.e. , if r < min { d h , d 0 } ) we have that rk( dµ d ( θ )) <

Original: dim M r if and only if rk( µ d ( θ )) <
Revised: dim M r if and only if rk( µ d ( θ )) <

Original: r .
Revised: r .

Original: Proof.
Revised: Proof.

Original: If rk( µ d ( θ ))
Revised: If rk( µ d ( θ ))

Original: =
Revised: =

Original: r , then Proposition 16 implies that the image of the differential dµ d ( θ ) is the whole tangent space of M r at µ d ( θ ) .
Revised: r , then Proposition 16 implies that the image of the differential dµ d ( θ ) is the whole tangent space of M r at µ d ( θ ) .

Original: To prove the other direction of the assertion, we assume that rk( µ d ( θ )) < r .
Revised: To prove the other direction of the assertion, we assume that rk( µ d ( θ )) < r .

Original: Since r < min { d h , d 0 } , there is some i ∈ { 1 , . . .
Revised: Since r < min { d h , d 0 } , there is some i ∈ { 1 , . . .

Original: , h − 1 } such that d i = r .
Revised: , h − 1 } such that d i = r .

Original: We view µ d as the following concatenation of the matrix multiplication maps:
Revised: We view µ d as the following concatenation of the matrix multiplication maps:

Original: [Equation]
Revised: [Equation]

Original: where µ i, 1 = µ ( d h ,...,d i ) × µ ( d i ,...,d 0 ) and µ i, 2 = µ ( d h ,d i ,d 0 ) .
Revised: where µ i, 1 = µ ( d h ,...,d i ) × µ ( d i ,...,d 0 ) and µ i, 2 = µ ( d h ,d i ,d 0 ) .

Original: Since rk( µ d ( θ )) < r , we have that rk( W >i ) <
Revised: Since rk( µ d ( θ )) < r , we have that rk( W >i ) <

Original: r or rk( W <i +1 )
Revised: r or rk( W <i +1 )

Original: < r .
Revised: < r .

Original: Without loss of generality, we may assume the latter.
Revised: Without loss of generality, we may assume the latter.

Original: So applying
Revised: So applying

Original: Lemma 3 to µ i, 2 and θ (cid:48) :
Revised: Lemma 3 to µ i, 2 and θ (cid:48) :

Original: = µ i, 1 ( θ ) yields
Revised: = µ i, 1 ( θ ) yields

Original: [Equation]
Revised: [Equation]

Original: Proposition 18.
Revised: Proposition 18.

Original: Let r = min
Revised: Let r = min

Original: { d i } and θ =
Revised: { d i } and θ =

Original: ( W h , . . .
Revised: ( W h , . . .

Original: , W 1 ) .
Revised: , W 1 ) .

Original: In the ﬁlling case ( i.e. , if r = min { d h , d 0 } ) we have that rk( dµ d ( θ )) <
Revised: In the ﬁlling case ( i.e. , if r = min { d h , d 0 } ) we have that rk( dµ d ( θ )) <

Original: d h d 0 if and only if there is some i ∈ { 1 , . . .
Revised: d h d 0 if and only if there is some i ∈ { 1 , . . .

Original: , h − 1 } with rk( W >i ) <
Revised: , h − 1 } with rk( W >i ) <

Original: d h and rk( W <i +1 ) <
Revised: d h and rk( W <i +1 ) <

Original: d 0 .
Revised: d 0 .

Original: Proof.
Revised: Proof.

Original: Let us ﬁrst assume that rk( W >i ) <
Revised: Let us ﬁrst assume that rk( W >i ) <

Original: d h and rk( W <i +1 )
Revised: d h and rk( W <i +1 )

Original: <
Revised: <

Original: d 0 for some i ∈ { 1 , . . .
Revised: d 0 for some i ∈ { 1 , . . .

Original: , h − 1 } .
Revised: , h − 1 } .

Original: We view µ d as the concatenation of the matrix multiplication maps in (9).
Revised: We view µ d as the concatenation of the matrix multiplication maps in (8).

Original: Applying Lemma 3 to µ i, 2 and θ (cid:48) :
Revised: Applying Lemma 3 to µ i, 2 and θ (cid:48) :

Original: = µ i, 1 ( θ ) yields
Revised: = µ i, 1 ( θ ) yields

Original: [Equation]
Revised: [Equation]

Original: Secondly, we assume the contrary, i.e. , that every i ∈ { 1 , . .
Revised: Secondly, we assume the contrary, i.e. , that every i ∈ { 1 , . .

Original: , h − 1 } satisﬁes rk( W >i ) =
Revised: , h − 1 } satisﬁes rk( W >i ) =

Original: d h or rk( W <i +1 ) =
Revised: d h or rk( W <i +1 ) =

Original: d 0 .
Revised: d 0 .

Original: We observe the following 2 key properties which hold for all i ∈ { 1 , . . .
Revised: We observe the following 2 key properties which hold for all i ∈ { 1 , . . .

Original: , h − 1 } :
Revised: , h − 1 } :

Original: [Equation]
Revised: [Equation]

Original: We consider the index set I :=
Revised: We consider the index set I :=

Original: { i ∈ { 1 , . . .
Revised: { i ∈ { 1 , . . .

Original: , h − 1 } | rk( W <i +1 )
Revised: , h − 1 } | rk( W <i +1 )

Original: =
Revised: =

Original: d 0 } .
Revised: d 0 } .

Original: If I = ∅ , our assumption implies that rk( W >i ) =
Revised: If I = ∅ , our assumption implies that rk( W >i ) =

Original: d h for every i ∈ { 1 , . . .
Revised: d h for every i ∈ { 1 , . . .

Original: , h } .
Revised: , h } .

Original: So due to Lemma 3 we haverk
Revised: So due to Lemma 3 we have

Original: [Equation]
Revised: [Equation]

Original: If I (cid:54) = ∅ , we deﬁne k := max I .
Revised: If I (cid:54) = ∅ , we deﬁne k := max I .

Original: So for every i ∈ { k + 1 , . .
Revised: So for every i ∈ { k + 1 , . .

Original: , h − 1 } we have rk( W <i +1 ) <
Revised: , h − 1 } we have rk( W <i +1 ) <

Original: d 0 , and thus rk( W >i ) =
Revised: d 0 , and thus rk( W >i ) =

Original: d h by our assumption.
Revised: d h by our assumption.

Original: Moreover, due to (10), every j ∈ { 0 , . . . , k } satisﬁes rk( W <j +1 ) =
Revised: Moreover, due to (9), every j ∈ { 0 , . . . , k } satisﬁes rk( W <j +1 ) =

Original: d 0 .
Revised: d 0 .

Original: Hence, Lemma 3 yields
Revised: Hence, Lemma 3 yields

Original: [Equation]
Revised: [Equation]

Original: Example 19.
Revised: Example 19.

Original: According to Proposition 18, the differential of the matrix multiplication map is surjective whenever rk( W ) =
Revised: According to Proposition 18, the differential of the matrix multiplication map is surjective whenever rk( W ) =

Original: r , but also for certain θ when rk( W ) < r .
Revised: r , but also for certain θ when rk( W ) < r .

Original: For example, let us consider the map µ (2 , 2 , 2) :
Revised: For example, let us consider the map µ (2 , 2 , 2) :

Original: R 2 × 2 × R 2 × 2 → R 2 × 2 and the two factorizations θ =
Revised: R 2 × 2 × R 2 × 2 → R 2 × 2 and the two factorizations θ =

Original: ([ 1 1 1 1 ] , [ 1 0 0 1 ]) and θ (cid:48) =
Revised: ([ 1 1 1 1 ] , [ 1 0 0 1 ]) and θ (cid:48) =

Original: ([ 1 0 1 0 ] , [ 1 1 0 0 ]) of the rank-one matrix [ 1 1 1 1 ] .
Revised: ([ 1 0 1 0 ] , [ 1 1 0 0 ]) of the rank-one matrix [ 1 1 1 1 ] .

Original: According to Proposition 18, the differential dµ (2 , 2 , 2) ( θ ) has maximal rank 4 .
Revised: According to Proposition 18, the differential dµ (2 , 2 , 2) ( θ ) has maximal rank 4 .

Original: So it is surjective, whereas dµ (2 , 2 , 2) ( θ (cid:48) ) is not.
Revised: So it is surjective, whereas dµ (2 , 2 , 2) ( θ (cid:48) ) is not.

Original: In fact, by
Revised: In fact, by

Original: Lemma 3, we have rk( dµ (2 , 2 , 2) ( θ (cid:48) ))
Revised: Lemma 3, we have rk( dµ (2 , 2 , 2) ( θ (cid:48) ))

Original: = 3 .
Revised: = 3 .

Original: Theorem 4.
Revised: Theorem 4.

Original: Let r = min
Revised: Let r = min

Original: { d i } , θ =
Revised: { d i } , θ =

Original: ( W h , . . .
Revised: ( W h , . . .

Original: , W 1 ) , and W = µ d ( θ ) 
Revised: , W 1 ) , and W = µ d ( θ ) .

Original: • (Filling case)
Revised: • (Filling case)

Original: If r = min { d h , d 0 } , the differential dµ d ( θ ) has maximal rank equal to dim M r =
Revised: If r = min { d h , d 0 } , the differential dµ d ( θ ) has maximal rank equal to dim M r =

Original: d h d 0 if and only if, for every i ∈ { 1 , 2 , . .
Revised: d h d 0 if and only if, for every i ∈ { 1 , 2 , . .

Original: , h − 1 } , either rk( W >i ) =
Revised: , h − 1 } , either rk( W >i ) =

Original: d h or rk( W <i +1 )
Revised: d h or rk( W <i +1 )

Original: =
Revised: =

Original: d 0 holds.
Revised: d 0 holds.

Original: • (Non-ﬁlling case)
Revised: • (Non-ﬁlling case)

Original: If r < min { d h , d 0 } , the differential dµ d ( θ ) has maximal rank equal to dim M r = r
Revised: If r < min { d h , d 0 } , the differential dµ d ( θ ) has maximal rank equal to dim M r = r

Original: ( d h + d 0 − r )
Revised: ( d h + d 0 − r )

Original: if and only if rk( W ) = r .
Revised: if and only if rk( W ) = r .

Original: Furthermore, in both situations, if rk( W ) =
Revised: Furthermore, in both situations, if rk( W ) =

Original: e < r , then the image of dµ d ( θ ) always contains the tangent space T W M e of M e ⊂ M r at W .
Revised: e < r , then the image of dµ d ( θ ) always contains the tangent space T W M e of M e ⊂ M r at W .

Original: Proof.
Revised: Proof.

Original: This is an amalgamation of Propositions 16, 17 and 18.
Revised: This is an amalgamation of Propositions 16, 17 and 18.

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: In the following we use the notation from Theorem 5:
Revised: In the following we use the notation from Theorem 5:

Original: Theorem 5.
Revised: Theorem 5.

Original: Let r = min
Revised: Let r = min

Original: { d i } .
Revised: { d i } .

Original: If rk( W ) =
Revised: If rk( W ) =

Original: r , then the set of d -factorizations µ − 1 d ( W ) of W has exactly 2 b path-connected components, where b
Revised: r , then the set of d -factorizations µ − 1 d ( W ) of W has exactly 2 b path-connected components, where b

Original: = # { i | d i = r, 0 <
Revised: = # { i | d i = r, 0 <

Original: i < h } .
Revised: i < h } .

Original: If rk( W ) <
Revised: If rk( W ) <

Original: r , then µ − 1 d ( W ) is always path-connected.
Revised: r , then µ − 1 d ( W ) is always path-connected.

Original: We also write GL + ( r ) for the set of matrices in GL( r ) with positive determinant.
Revised: We also write GL + ( r ) for the set of matrices in GL( r ) with positive determinant.

Original: Analogously, we set GL − ( r ) :
Revised: Analogously, we set GL − ( r ) :

Original: = { G ∈ GL( r ) | det( G ) < 0 } .
Revised: = { G ∈ GL( r ) | det( G ) < 0 } .

Original: We ﬁrst prove Theorem 5 in the case that b = 0 .
Revised: We ﬁrst prove Theorem 5 in the case that b = 0 .

Original: To show that µ − 1 d ( W ) is path-connected in this case, we show the following stronger assertion: given two matrices W and W (cid:48) of arbitrary rank and factorizations θ ∈ µ − 1 d ( W ) and θ (cid:48) ∈ µ − 1 d ( W (cid:48) ) , each path in the codomain of µ d from W to W (cid:48) can be lifted to a path in the domain of µ d from θ to θ (cid:48) .
Revised: To show that µ − 1 d ( W ) is path-connected in this case, we show the following stronger assertion: given two matrices W and W (cid:48) of arbitrary rank and factorizations θ ∈ µ − 1 d ( W ) and θ (cid:48) ∈ µ − 1 d ( W (cid:48) ) , each path in the codomain of µ d from W to W (cid:48) can be lifted to a path in the domain of µ d from θ to θ (cid:48) .

Original: Proposition 20 (Path Lifting Property) .
Revised: Proposition 20 (Path Lifting Property) .

Original: If b = 0 , then for every W, W (cid:

Revised: If b = 0 , then for every W, W (cid:

Original: ) ∈ R d y × d x , every θ ∈ µ − 1 d ( W ) , every θ (cid:48) ∈ µ − 1 d ( W (cid:48) ) and every continuous function f : [0 , 1]
Revised: ) ∈ R d y × d x , every θ ∈ µ − 1 d ( W ) , every θ (cid:48) ∈ µ − 1 d ( W (cid:48) ) and every continuous function f : [0 , 1]

Original: → R d y × d x with f (0) =
Revised: → R d y × d x with f (0) =

Original: W and f (1) = W (cid:48) , there is a continuous function F : [ − 1 , 2] → R d y ×
Revised: W and f (1) = W (cid:48) , there is a continuous function F : [ − 1 , 2] → R d y ×

Original: d
Revised: d

Original: h − 1 × . . .
Revised: h − 1 × . . .

Original: ×
Revised: ×

Original: R d 1 × d x such that
Revised: R d 1 × d x such that

Original: F ( − 1)
Revised: F ( − 1)

Original: = θ , F (2)
Revised: = θ , F (2)

Original: = θ (cid:48) , µ d ( F ( t )) =
Revised: = θ (cid:48) , µ d ( F ( t )) =

Original: W for every t ∈ [ − 1 , 0] , µ d ( F ( t ))
Revised: W for every t ∈ [ − 1 , 0] , µ d ( F ( t ))

Original: =
Revised: =

Original: W (cid:48) for every t ∈ [1 , 2] and µ d ( F ( t ))
Revised: W (cid:48) for every t ∈ [1 , 2] and µ d ( F ( t ))

Original: =
Revised: =

Original: f ( t ) for every t ∈ [0 , 1] .
Revised: f ( t ) for every t ∈ [0 , 1] .

Original: Proof.
Revised: Proof.

Original: Without loss of generality, we may assume that d y ≤ d x .
Revised: Without loss of generality, we may assume that d y ≤ d x .

Original: Then the assumption b = 0 means that d i >
Revised: Then the assumption b = 0 means that d i >

Original: d
Revised: d

Original: y for all i = 1 , . . .
Revised: y for all i = 1 , . . .

Original: , h − 1 .
Revised: , h − 1 .

Original: We prove the assertion by induction on h .
Revised: We prove the assertion by induction on h .

Original: For the induction beginning, we consider the cases
Revised: For the induction beginning, we consider the cases

Original: h
Revised: h

Original: =and h = 2 .
Revised: =and h = 2 .

Original: If h = 1 , then µ d is the identity and Proposition 20 is trivial.
Revised: If h = 1 , then µ d is the identity and Proposition 20 is trivial.

Original: For h = 2 , we construct explicit lifts of the given paths.
Revised: For h = 2 , we construct explicit lifts of the given paths.

Original: We ﬁrst show that there is a path in µ − 1 d ( W ) from θ =
Revised: We ﬁrst show that there is a path in µ − 1 d ( W ) from θ =

Original: ( W 2 , W 1 ) to some ( B 2 , B 1 ) such that B 2 has full rank.
Revised: ( W 2 , W 1 ) to some ( B 2 , B 1 ) such that B 2 has full rank.

Original: Claim 1 .
Revised: Claim 1 .

Original: Let h
Revised: Let h

Original: = 2 , W ∈ R d y × d x and ( W 2 , W 1 ) ∈ µ − 1 d ( W ) .
Revised: = 2 , W ∈ R d y × d x and ( W 2 , W 1 ) ∈ µ − 1 d ( W ) .

Original: Then there is ( B 2 , B 1 ) ∈ µ − 1 d ( W ) with rk( B 2 ) =
Revised: Then there is ( B 2 , B 1 ) ∈ µ − 1 d ( W ) with rk( B 2 ) =

Original: d y and a continuous function g : [0 , 1] → µ − 1 d ( W ) such that g (0)
Revised: d y and a continuous function g : [0 , 1] → µ − 1 d ( W ) such that g (0)

Original: =
Revised: =

Original: ( W 2 , W 1 ) and g (1) =
Revised: ( W 2 , W 1 ) and g (1) =

Original: ( B 2 , B 1 ) .
Revised: ( B 2 , B 1 ) .

Original: Proof.
Revised: Proof.

Original: If rk( W 2 ) =
Revised: If rk( W 2 ) =

Original: d y , we have nothing to show.
Revised: d y , we have nothing to show.

Original: So we assume that s :
Revised: So we assume that s :

Original: = rk( W 2 ) <
Revised: = rk( W 2 ) <

Original: d y .
Revised: d y .

Original: Without loss of generality, we may further assume that the ﬁrst s rows of W 2 have rank s .
Revised: Without loss of generality, we may further assume that the ﬁrst s rows of W 2 have rank s .

Original: As s < d 1 , we ﬁnd a matrix G ∈ GL + ( d 1 ) such that W 2 G = (cid:2)
Revised: As s < d 1 , we ﬁnd a matrix G ∈ GL + ( d 1 ) such that W 2 G = (cid:2)

Original: I s 0 M 0 (cid:3) , where I s ∈ R s × s is the identity matrix and M ∈
Revised: I s 0 M 0 (cid:3) , where I s ∈ R s × s is the identity matrix and M ∈

Original: R ( d y − s ) × s .
Revised: R ( d y − s ) × s .

Original: Since GL + ( d 1 ) is path-connected, there is a continuous function g (cid:48) 1 : [0 , 1] → GL + ( d 1 ) with g (cid:48) 1 (0) =
Revised: Since GL + ( d 1 ) is path-connected, there is a continuous function g (cid:48) 1 : [0 , 1] → GL + ( d 1 ) with g (cid:48) 1 (0) =

Original: I d 1 and g (cid:48) 1 (1) =
Revised: I d 1 and g (cid:48) 1 (1) =

Original: G .
Revised: G .

Original: Concatenation with GL( d 1 )
Revised: Concatenation with GL( d 1 )

Original: → µ − 1 d ( W ) , H (cid:55)→ ( W 2 H, H − 1 W 1 ) yields a continuous path g 1 in µ − 1 d ( W ) from ( W 2 , W 1 ) to ( W 2 G, G − 1 W 1 ) .
Revised: → µ − 1 d ( W ) , H (cid:55)→ ( W 2 H, H − 1 W 1 ) yields a continuous path g 1 in µ − 1 d ( W ) from ( W 2 , W 1 ) to ( W 2 G, G − 1 W 1 ) .

Original: Since ( W 2 G )( G − 1 W 1 ) =
Revised: Since ( W 2 G )( G − 1 W 1 ) =

Original: W , we see that G − 1 W 1 =
Revised: W , we see that G − 1 W 1 =

Original: (cid:104) W ( s ) N (cid:105) , where W ( s ) ∈ R s ×
Revised: (cid:104) W ( s ) N (cid:105) , where W ( s ) ∈ R s ×

Original: d x is the ﬁrst s rows of W and N ∈ R ( d y − s ) ×
Revised: d x is the ﬁrst s rows of W and N ∈ R ( d y − s ) ×

Original: d x .
Revised: d x .

Original: Replacing N by an arbitrary matrix N (cid:48) still yields that
Revised: Replacing N by an arbitrary matrix N (cid:48) still yields that

Original: W 2 G (cid:104) W ( s ) N (cid:48) (cid:105)
Revised: W 2 G (cid:104) W ( s ) N (cid:48) (cid:105)

Original: =
Revised: =

Original: W .
Revised: W .

Original: Hence, we ﬁnd a continuous path g 2 in µ − 1 d ( W ) from ( W 2 G, G − 1 W 1 ) to ( W 2 G, B 1 :=
Revised: Hence, we ﬁnd a continuous path g 2 in µ − 1 d ( W ) from ( W 2 G, G − 1 W 1 ) to ( W 2 G, B 1 :=

Original: (cid:104) W ( s ) 0 (cid:105) ) .
Revised: (cid:104) W ( s ) 0 (cid:105) ) .

Original: Finally, we can replace the 0 -columns in W 2 G by arbitrary matrices M 1 ∈ R s × ( d 1 − s ) and M 2 ∈
Revised: Finally, we can replace the 0 -columns in W 2 G by arbitrary matrices M 1 ∈ R s × ( d 1 − s ) and M 2 ∈

Original: R ( d y − s ) × ( d 1 − s ) such that (cid:2)
Revised: R ( d y − s ) × ( d 1 − s ) such that (cid:2)

Original: I s M 1 M M 2 (cid:3) B 1 =
Revised: I s M 1 M M 2 (cid:3) B 1 =

Original: W still holds.
Revised: W still holds.

Original: In particular, we can pick M 1 = 0 and
Revised: In particular, we can pick M 1 = 0 and

Original: M 

Revised: M

Original: = [ I dy − s 0 ] such that B 2 := (cid:2)
Revised: = [ I dy − s 0 ] such that B 2 := (cid:2)

Original: I s 0 M M 2 (cid:3) has full rank d y , and we ﬁnd a continuous path g 3 in µ − 1 d ( W ) from ( W 2 G, B 1 ) to ( B 2 , B 1 ) .
Revised: I s 0 M M 2 (cid:3) has full rank d y , and we ﬁnd a continuous path g 3 in µ − 1 d ( W ) from ( W 2 G, B 1 ) to ( B 2 , B 1 ) .

Original: Putting g 1 , g 2 and g 3 together yields a path g as desired in
Revised: Putting g 1 , g 2 and g 3 together yields a path g as desired in

Original: Claim 1.
Revised: Claim 1.

Original: ♦
Revised: ♦

Original: As B 2 has full rank, we ﬁnd a matrix H ∈ GL
Revised: As B 2 has full rank, we ﬁnd a matrix H ∈ GL

Original: + ( d 1 ) such that B 2 H
Revised: + ( d 1 ) such that B 2 H

Original: =
Revised: =

Original: [ I dy 0 ] .
Revised: [ I dy 0 ] .

Original: As in the proof of Claim 1, we construct a continuous path in µ − 1 d ( W ) from ( B 2 , B 1 ) to ( B 2 H, H − 1 B 1 ) .
Revised: As in the proof of Claim 1, we construct a continuous path in µ − 1 d ( W ) from ( B 2 , B 1 ) to ( B 2 H, H − 1 B 1 ) .

Original: Since H − 1 B 1 = [ WN ] for some N ∈ R ( d 1 − d y ) ×
Revised: Since H − 1 B 1 = [ W N ] for some N ∈ R ( d 1 − d y ) ×

Original: d x , we also ﬁnd a continuous path in µ − 1 d ( W ) from ( B 2 H, H − 1 B 1 ) to ( B 2 H, [ W 0 ]) .
Revised: d x , we also ﬁnd a continuous path in µ − 1 d ( W ) from ( B 2 H, H − 1 B 1 ) to ( B 2 H, [ W 0 ]) .

Original: All in all, we have constructed a continuous path F 1 in µ − 1 d ( W ) from ( W 2 , W 1 ) to ([ I dy 0 ] , [ W 0 ]) .
Revised: All in all, we have constructed a continuous path F 1 in µ − 1 d ( W ) from ( W 2 , W 1 ) to ([ I dy 0 ] , [ W 0 ]) .

Original: Analogously, we ﬁnd a continuous path F 3 in µ − 1 d ( W (cid:48) ) between ( W (cid:48) 2 , W (cid:48) 1 ) and ([ I dy 0 ] , (cid:2) W (cid:48) 0 (cid:3) ) .
Revised: Analogously, we ﬁnd a continuous path F 3 in µ − 1 d ( W (cid:48) ) between ( W (cid:48) 2 , W (cid:48) 1 ) and ([ I dy 0 ] , (cid:2) W (cid:48) 0 (cid:3) ) .

Original: Finally, we deﬁne F 2 : [0 , 1] → [Equation] y 
Revised: Finally, we deﬁne F 2 : [0 , 1] → R d y ×

Original: [Equation] [Equation] , t [Equation]
Revised: d 1 × R d 1 × d x , t (cid:55)→ ([ I dy 0 ] , (cid:2)

Original: ([ I dy 0 ] , (cid:2) f ( t ) 0 (cid:3) ) such that putting F 1 , F 2 and F 3 together yields a path F as desired in Proposition 20.
Revised: f ( t ) 0 (cid:3) ) such that putting F 1 , F 2 and F 3 together yields a path F as desired in Proposition 20.

Original: For the induction step, we view µ d as the concatenation of the following two matrix multiplication maps:
Revised: For the induction step, we view µ d as the concatenation of the following two matrix multiplication maps:

Original: [Equation]
Revised: [Equation]

Original: We consider θ =
Revised: We consider θ =

Original: ( W h , . . .
Revised: ( W h , . . .

Original: , W 1 ) and θ (cid:48) =
Revised: , W 1 ) and θ (cid:48) =

Original: ( W (cid:48) h , . . .
Revised: ( W (cid:48) h , . . .

Original: , W (cid:48) 1 ) , as well as W > 1 = W h · · · W 2 and W (cid:48) > 1 =
Revised: , W (cid:48) 1 ) , as well as W > 1 = W h · · · W 2 and W (cid:48) > 1 =

Original: W (cid:48) h · · · W (cid:48) 2 .
Revised: W (cid:48) h · · · W (cid:48) 2 .

Original: Given a path f : [0 , 1] → R d y ×
Revised: Given a path f : [0 , 1] → R d y ×

Original: d
Revised: d

Original: x from W = W > 1 W 1 to W (cid:48)
Revised: x from W = W > 1 W 1 to W (cid:48)

Original: =
Revised: =

Original: W (cid:48) > 1 W (cid:48) 1 , we apply the induction beginning ( h = 2 ) to µ ( d y ,d 1 ,d x ) to get a path F 2 : [ − 0 .
Revised: W (cid:48) > 1 W (cid:48) 1 , we apply the induction beginning ( h = 2 ) to µ ( d y ,d 1 ,d x ) to get a path F 2 : [ − 0 .

Original: → R d y × d 1 × R d 1 × d x such that F 2 ( − 0 .
Revised: → R d y × d 1 × R d 1 × d x such that F 2 ( − 0 .

Original: =
Revised: =

Original: ( W > 1 , W 1 ) , F 2 (1 . 5) =( W
Revised: ( W > 1 , W 1 ) , F 2 (1 . 5) =

Original: [Equation] > 1 , W (cid:48) 1 ) , µ ( d y ,d 1 ,d x ) ( F 2 ( t ))
Revised: ( W (cid:48) > 1 , W (cid:48) 1 ) , µ ( d y ,d 1 ,d x ) ( F 2 ( t ))

Original: =
Revised: =

Original: W for all t ∈ [ − 0 .
Revised: W for all t ∈ [ − 0 .

Original: 5 , 0] , µ ( d y ,d 1 ,d x ) ( F 2 ( t ))
Revised: 5 , 0] , µ ( d y ,d 1 ,d x ) ( F 2 ( t ))

Original: =
Revised: =

Original: W (cid:48) for all t ∈ [1 , 1 . 5] and µ ( d y ,d 1 ,d x ) ( F 2 ( t )) =
Revised: W (cid:48) for all t ∈ [1 , 1 . 5] and µ ( d y ,d 1 ,d x ) ( F 2 ( t )) =

Original: f ( t ) for all t ∈ [0 , 1] .
Revised: f ( t ) for all t ∈ [0 , 1] .

Original: Now we apply the induction hypothesis on µ ( d h ,...,d 1 ) and the path from W > 1 to W (cid:48) > 1 given by the ﬁrst factor of F 2 .
Revised: Now we apply the induction hypothesis on µ ( d h ,...,d 1 ) and the path from W > 1 to W (cid:48) > 1 given by the ﬁrst factor of F 2 .

Original: This yields a path F 1 : [ − 1 , 2] → R d y ×
Revised: This yields a path F 1 : [ − 1 , 2] → R d y ×

Original: d
Revised: d

Original: h − 1 × . . .
Revised: h − 1 × . . .

Original: ×
Revised: ×

Original: R d 2 × d 1 with F 1 ( − 1) =
Revised: R d 2 × d 1 with F 1 ( − 1) =

Original: ( W h , . . .
Revised: ( W h , . . .

Original: W 2 ) ,
Revised: W 2 ) ,

Original: F 1 (2) =
Revised: F 1 (2) =

Original: ( W (cid:48) h , . . .
Revised: ( W (cid:48) h , . . .

Original: W (cid:48) 2 ) , µ ( d h ,...,d 1 ) ( F 1 ( t )) =
Revised: W (cid:48) 2 ) , µ ( d h ,...,d 1 ) ( F 1 ( t )) =

Original: W > 1 for all t ∈ [ − 1 , − 0 .
Revised: W > 1 for all t ∈ [ − 1 , − 0 .

Original: 5] , µ ( d h ,...,d 1 ) ( F 1 ( t ))
Revised: 5] , µ ( d h ,...,d 1 ) ( F 1 ( t ))

Original: =
Revised: =

Original: W (cid:48) >for all t ∈ [1 . 5 , 2] and µ ( d h ,...,d 1 ) ( F 1 ( t )) is the ﬁrst factor of F 2 ( t ) for all t ∈ [ − 0 .
Revised: W (cid:48) >for all t ∈ [1 . 5 , 2] and µ ( d h ,...,d 1 ) ( F 1 ( t )) is the ﬁrst factor of F 2 ( t ) for all t ∈ [ − 0 .

Original: This allows us to deﬁne a continuous path F : [ − 1 , 2] → R d y ×
Revised: This allows us to deﬁne a continuous path F : [ − 1 , 2] → R d y ×

Original: d
Revised: d

Original: h − 1 × . . .
Revised: h − 1 × . . .

Original: ×
Revised: ×

Original: R d 1 × d x from θ to θ (cid:48) by setting F ( t ) =
Revised: R d 1 × d x from θ to θ (cid:48) by setting F ( t ) =

Original: ( F 1 ( t ) , W 1 ) for all t ∈ [ − 1 , − 0 .
Revised: ( F 1 ( t ) , W 1 ) for all t ∈ [ − 1 , − 0 .

Original: 5] , F ( t ) =
Revised: 5] , F ( t ) =

Original: ( F 1 ( t ) , W (cid:48) 1 ) for all t ∈ [1 . 5 , 2] and for all t ∈ [ − 1 , − 0 . 5]
Revised: ( F 1 ( t ) , W (cid:48) 1 ) for all t ∈ [1 . 5 , 2] and for all t ∈ [ − 1 , − 0 . 5]

Original: we let F ( t ) consist of F 1 ( t ) and the second factor of F 2 ( t ) .
Revised: we let F ( t ) consist of F 1 ( t ) and the second factor of F 2 ( t ) .

Original: Corollary 21.
Revised: Corollary 21.

Original: If b = 0 , then µ − 1 d ( W ) is path-connected for each W ∈ R d y × d x .
Revised: If b = 0 , then µ − 1 d ( W ) is path-connected for each W ∈ R d y × d x .

Original: Proof.
Revised: Proof.

Original: Apply Proposition 20 to the constant function f : [0 , 1] → R d y × d x , t (cid:55)→ W .
Revised: Apply Proposition 20 to the constant function f : [0 , 1] → R d y × d x , t (cid:55)→ W .

Original: Now we study the case b > 0 .
Revised: Now we study the case b > 0 .

Original: We write 0 < i 1 < . . .
Revised: We write 0 < i 1 < . . .

Original: <
Revised: <

Original: i b < h for those indices
Revised: i b < h for those indices

Original: i j such that d i j = r .
Revised: i j such that d i j = r .

Original: Then we view µ d as the concatenation of the following two matrix multiplication maps:
Revised: Then we view µ d as the concatenation of the following two matrix multiplication maps:

Original: [Equation]
Revised: [Equation]

Original: ) where µ 1 = µ ( d h ,...,d ib ) × µ ( d ib ,...,d ib − 1 ) × . . .
Revised: where µ 1 = µ ( d h ,...,d ib ) × µ ( d ib ,..., d ib − 1 ) × . . .

Original: × µ ( d i 1 ,...,d 0 ) and µ 2 = µ ( d y ,d ib ,...,d i 1 ,d x ) .
Revised: × µ ( d i 1 ,...,d 0 ) and µ 2 = µ ( d y ,d ib ,...,d i 1 ,d x ) .

Original: Applying the path lifting property described above to the map µ 1 , we will show in Proposition 26 that µ − 1 2 ( W ) and µ − 1 d ( W ) have the same number of (path-)connected components.
Revised: Applying the path lifting property described above to the map µ 1 , we will show in Proposition 26 that µ − 1 2 ( W ) and µ − 1 d ( W ) have the same number of (path-)connected components.

Original: So it remains to study the connected components of µ − 1 2 ( W ) .
Revised: So it remains to study the connected components of µ − 1 2 ( W ) .

Original: We can shortly write the map µ 2 as
Revised: We can shortly write the map µ 2 as

Original: [Equation]
Revised: [Equation]

Original: In the case that rk( W ) =
Revised: In the case that rk( W ) =

Original: r , we use the following natural action of GL( r ) b on µ − 1 2 ( W ) :
Revised: r , we use the following natural action of GL( r ) b on µ − 1 2 ( W ) :

Original: [Equation]
Revised: [Equation]

Original: In fact, we show now that µ − 1 2 ( W ) is the orbit of any element under this action if rk( W ) =
Revised: In fact, we show now that µ − 1 2 ( W ) is the orbit of any element under this action if rk( W ) =

Original: r .
Revised: r .

Original: From this we will deduce in Corollaries 23 and 24 that µ − 1 2 ( W ) is homeomorphic to GL( r ) b and thus has 2 b (path-)connected components if the matrix W has maximal rank r .
Revised: From this we will deduce in Corollaries 23 and 24 that µ − 1 2 ( W ) is homeomorphic to GL( r ) b and thus has 2 b (path-)connected components if the matrix W has maximal rank r .

Original: Proposition 22.
Revised: Proposition 22.

Original: Let b > 

Revised: Let b >

Original: and θ =
Revised: and θ =

Original: ( A b +1 , . . .
Revised: ( A b +1 , . . .

Original: , A 1 ) ∈ R d y × r ×
Revised: , A 1 ) ∈ R d y × r ×

Original: ( R r × r ) b − 1 × R r × d x such that
Revised: ( R r × r ) b − 1 × R r × d x such that

Original: W = µ 2 ( θ ) has maximal rank r .
Revised: W = µ 2 ( θ ) has maximal rank r .

Original: Then µ − 1 2 ( W ) is the orbit of θ under the action deﬁned in (12) , i.e. ,
Revised: Then µ − 1 2 ( W ) is the orbit of θ under the action deﬁned in (11) , i.e. ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: One inclusion, namely “ ⊇ ”, is trivial.
Revised: One inclusion, namely “ ⊇ ”, is trivial.

Original: We prove the other inclusion “ ⊆ ” by induction on b .
Revised: We prove the other inclusion “ ⊆ ” by induction on b .

Original: For the induction beginning ( b = 1 ), we write W = (cid:2) W 11 W 12 W 21 W 22 (cid:3) where W 11 ∈ R r × r .
Revised: For the induction beginning ( b = 1 ), we write W = (cid:2) W 11 W 12 W 21 W 22 (cid:3) where W 11 ∈ R r × r .

Original: Without loss of generality, we may assume that rk( W 11 ) =
Revised: Without loss of generality, we may assume that rk( W 11 ) =

Original: r .
Revised: r .

Original: Similarly, we write A 2 = (cid:2)
Revised: Similarly, we write A 2 = (cid:2)

Original: A 21 A 22 (cid:3) and A 1 =
Revised: A 21 A 22 (cid:3) and A 1 =

Original: [ A 11 A 12 ] where A i 1 ∈ R r × r for i = 1 , 2 .
Revised: [ A 11 A 12 ] where A i 1 ∈ R r × r for i = 1 , 2 .

Original: For ( A (cid:48) 2 , A (cid:48) 1 ) ∈ µ − 1 2 ( W ) , we write analogously A (cid:48) 2 = (cid:104) A (cid:48)
Revised: For ( A (cid:48) 2 , A (cid:48) 1 ) ∈ µ − 1 2 ( W ) , we write analogously A (cid:48) 2 = (cid:104) A (cid:48)

Original: 21 A (cid:48) 22 (cid:105) and A (cid:48) 

Revised: 21 A (cid:48) 22 (cid:105) and A (cid:48)

Original: =
Revised: =

Original: [ A (cid:48) 11 A (cid:48) 12 ] .
Revised: [ A (cid:48) 11 A (cid:48) 12 ] .

Original: Due to rk( W 11 ) =
Revised: Due to rk( W 11 ) =

Original: r , we have that rk( A 21 ) =
Revised: r , we have that rk( A 21 ) =

Original: r = rk( A (cid:48) 21 ) .
Revised: r = rk( A (cid:48) 21 ) .

Original: Hence, there is a matrix G ∈ GL( r ) such that A (cid:48) 

Revised: Hence, there is a matrix G ∈ GL( r ) such that A (cid:48)

Original: =
Revised: =

Original: A 21 G .
Revised: A 21 G .

Original: This implies that A 21 GA (cid:48) 

Revised: This implies that A 21 GA (cid:48)

Original: =
Revised: =

Original: W 11 =
Revised: W 11 =

Original: A 21 A 11 , so A (cid:48) 

Revised: A 21 A 11 , so A (cid:48)

Original: =
Revised: =

Original: G − 1 A 11 .
Revised: G − 1 A 11 .

Original: Due to A 21 A 12 = W 12 =
Revised: Due to A 21 A 12 = W 12 =

Original: A 21 GA (cid:48) 12 , we get that A (cid:48) 

Revised: A 21 GA (cid:48) 12 , we get that A (cid:48)

Original: =
Revised: =

Original: G − 1 A 12 .
Revised: G − 1 A 12 .

Original: Finally, rk( W 11 ) =
Revised: Finally, rk( W 11 ) =

Original: r implies that rk( A 11 ) =
Revised: r implies that rk( A 11 ) =

Original: r , so
Revised: r , so

Original: A 
 A 

Revised: A 
 A

Original: =
Revised: =

Original: W 

Revised: W

Original: = A (cid:48) 22 G − 

Revised: = A (cid:48) 22 G −

Original: A 11 shows A (cid:48) 

Revised: A 11 shows A (cid:48)

Original: =
Revised: =

Original: A 22 G .
Revised: A 22 G .

Original: Thus we have shown that A (cid:48) 

Revised: Thus we have shown that A (cid:48)

Original: = A 2 G and A (cid:48) 

Revised: = A 2 G and A (cid:48)

Original: = G − 1 A 1 .
Revised: = G − 1 A 1 .

Original: For the induction step ( b > 1 ), we consider ( A (cid:48)
Revised: For the induction step ( b > 1 ), we consider ( A (cid:48)

Original: b +1 , . . .
Revised: b +1 , . . .

Original: , A (cid:48) 1 ) ∈ µ − 1 2 ( W ) and A > 1 =
Revised: , A (cid:48) 1 ) ∈ µ − 1 2 ( W ) and A > 1 =

Original: A b +1 · · · A 2 , A (cid:48) > 1 = A (cid:48)
Revised: A b +1 · · · A 2 , A (cid:48) > 1 = A (cid:48)

Original: b +1 · · · A (cid:48) 2 ∈ R d y × r .
Revised: b +1 · · · A (cid:48) 2 ∈ R d y × r .

Original: Now we can apply the induction beginning to ﬁnd G 1 ∈ GL( r ) such that A (cid:48) >
Revised: Now we can apply the induction beginning to ﬁnd G 1 ∈ GL( r ) such that A (cid:48) >

Original: 1 =
Revised: 1 =

Original: A > 1 G 1 and A (cid:48) 1 =
Revised: A > 1 G 1 and A (cid:48) 1 =

Original: G − 1 1 A 1 .
Revised: G − 1 1 A 1 .

Original: As A (cid:48) > 1 has rank r and
Revised: As A (cid:48) > 1 has rank r and

Original: A (cid:48)
Revised: A (cid:48)

Original: b +1 · · · A (cid:48) 

Revised: b +1 · · · A (cid:48)

Original: = A (cid:48) > 1 =
Revised: = A (cid:48) > 1 =

Original: A b +1 · · · ( A 2 G 1 ) , we can apply the induction hypothesis on the map which multiplies the left-most b matrices.
Revised: A b +1 · · · ( A 2 G 1 ) , we can apply the induction hypothesis on the map which multiplies the left-most b matrices.

Original: This yields G b , . . .
Revised: This yields G b , . . .

Original: G 2 ∈ GL( r ) such that A (cid:48)
Revised: G 2 ∈ GL( r ) such that A (cid:48)

Original: b +1 =
Revised: b +1 =

Original: A b +1 G b , . . .
Revised: A b +1 G b , . . .

Original: , A (cid:48)
Revised: , A (cid:48)

Original: 3 =
Revised: 3 =

Original: G − 1 3 A 3 G 2 , A (cid:48) 2 =
Revised: G − 1 3 A 3 G 2 , A (cid:48) 2 =

Original: G − 1 2 ( A 2 G 1 ) .
Revised: G − 1 2 ( A 2 G 1 ) .

Original: Corollary 23.
Revised: Corollary 23.

Original: If b > 0 and W ∈ R d y ×
Revised: If b > 0 and W ∈ R d y ×

Original: d x has maximal rank r , then µ − 1 2 ( W ) is homeomorphic to
Revised: d x has maximal rank r , then µ − 1 2 ( W ) is homeomorphic to

Original: GL( r ) b .
Revised: GL( r ) b .

Original: Proof.
Revised: Proof.

Original: We ﬁx θ =
Revised: We ﬁx θ =

Original: ( A b +1 , . . .
Revised: ( A b +1 , . . .

Original: , A 1 ) ∈ µ − 1 2 ( W ) .
Revised: , A 1 ) ∈ µ − 1 2 ( W ) .

Original: The map ϕ : GL( r ) b → µ − 1 2 ( W ) given by the action (12) on θ is continuous.
Revised: The map ϕ : GL( r ) b → µ − 1 2 ( W ) given by the action (11) on θ is continuous.

Original: We now construct its inverse.
Revised: We now construct its inverse.

Original: As rk( W ) =
Revised: As rk( W ) =

Original: r , we have that rk( A i ) =
Revised: r , we have that rk( A i ) =

Original: r for all i = 1 , . . .
Revised: r for all i = 1 , . . .

Original: , b + 1 .
Revised: , b + 1 .

Original: Without loss of generality, we may assume that the ﬁrst r rows of A b +1 have rank r .
Revised: Without loss of generality, we may assume that the ﬁrst r rows of A b +1 have rank r .

Original: We write π : R d y × r → R r × r for the projection which forgets the last d y − r rows of a given matrix.
Revised: We write π : R d y × r → R r × r for the projection which forgets the last d y − r rows of a given matrix.

Original: For θ (cid:48) =
Revised: For θ (cid:48) =

Original: ( A (cid:48)
Revised: ( A (cid:48)

Original: b +1 , . . .
Revised: b +1 , . . .

Original: , A (cid:48) 1 ) ∈ µ − 1 2 ( W ) , Proposition 22 shows that θ (cid:48) =
Revised: , A (cid:48) 1 ) ∈ µ − 1 2 ( W ) , Proposition 22 shows that θ (cid:48) =

Original: ϕ ( G b , . .
Revised: ϕ ( G b , . .

Original: , G 1 ) for some ( G b , . .
Revised: , G 1 ) for some ( G b , . .

Original: , G 1 ) ∈ GL( r ) b .
Revised: , G 1 ) ∈ GL( r ) b .

Original: So we have that
Revised: So we have that

Original: G b =π ( A b +1 ) − 1 π ( A (cid:48) b +1 ) , G b − 1 = A − 1 b G b A (cid:48) b , . . . , G 1 =
Revised: G b =

Original: A − 1 2 G 2 A (cid:48) 2 , which deﬁnes a map ψ

Original: [Equation]
Revised: [Equation]

Original: W

Original: [Equation]
Revised: [Equation]

Original: b , (cid:0) A

Original: [Equation] b +1 , . . . ,A (cid:48) 1 (cid:1) (cid:55)−→ (cid:0) G( A (cid:48) b
Revised: . , G

Original: [Equation]
Revised: [Equation]

Original: A
Revised: =

Original: [Equation]
Revised: [Equation]

Original: 1 b G ( A (cid:48) b
Revised: 2 , which deﬁnes a map ψ

Original: [Equation]
Revised: [Equation]

Original: ) A (cid:48) b , . . . A [Equation]
Revised: , where G ( A (cid:48) b +1 ) :

Original: [Equation] A  − 1 b G ( A (cid:48)b
Revised: = π ( A b +1 ) − 1 π ( A (cid:48)

Original: [Equation]

Original: 1 ) A (cid:48) b · · · A (cid:48) 2 (cid:1) , where G ( A (cid:48) b +1 )

Original: = π ( A b +1 ) − 1 π ( A (cid:48)

Original: b +1 ) .
Revised: b +1 ) .

Original: By construction, ψ is the inverse of ϕ .
Revised: By construction, ψ is the inverse of ϕ .

Original: Since ψ is continuous, it is a homeomorphism between µ − 1 2 ( W ) and GL( r ) b .
Revised: Since ψ is continuous, it is a homeomorphism between µ − 1 2 ( W ) and GL( r ) b .

Original: Corollary 24.
Revised: Corollary 24.

Original: If b > 0 and W ∈ R d y ×
Revised: If b > 0 and W ∈ R d y ×

Original: d x has maximal rank r , then µ − 1 2 ( W ) has 2 b connected components.
Revised: d x has maximal rank r , then µ − 1 2 ( W ) has 2 b connected components.

Original: Each of these components is path-connected.
Revised: Each of these components is path-connected.

Original: Proof.
Revised: Proof.

Original: The group GL( r ) has two connected components, namely GL + ( r ) and GL − ( r ) .
Revised: The group GL( r ) has two connected components, namely GL + ( r ) and GL − ( r ) .

Original: Both components are path-connected.
Revised: Both components are path-connected.

Original: Hence, GL( r ) b has 2 b connected components, each of them pathconnected.
Revised: Hence, GL( r ) b has 2 b connected components, each of them pathconnected.

Original: By Corollary 23, the same holds for µ − 1 2 ( W ) .
Revised: By Corollary 23, the same holds for µ − 1 2 ( W ) .

Original: To complete our understanding of the connected components of µ − 1 2 ( W ) , we consider the case that the matrix W ∈ R d y × d x does not have maximal rank r .
Revised: To complete our understanding of the connected components of µ − 1 2 ( W ) , we consider the case that the matrix W ∈ R d y × d x does not have maximal rank r .

Original: In that case, it turns out that µ − 1 2 ( W ) is pathconnected, which we show by constructing explicit paths between any two elements of µ − 1 2 ( W ) .
Revised: In that case, it turns out that µ − 1 2 ( W ) is pathconnected, which we show by constructing explicit paths between any two elements of µ − 1 2 ( W ) .

Original: Proposition 25.
Revised: Proposition 25.

Original: Let
Revised: Let

Original: W ∈ R d
Revised: W ∈ R d

Original: y × d x .
Revised: y × d x .

Original: If b > 0 and rk( W ) < r , then µ − 1 2 ( W ) is path-connected.
Revised: If b > 0 and rk( W ) < r , then µ − 1 2 ( W ) is path-connected.

Original: Proof.
Revised: Proof.

Original: We write W = (cid:104) W 1 W 2 (cid:105) where W 1 ∈ R r × d x , and denote by e the rank of W .
Revised: We write W = (cid:104) W 1 W 2 (cid:105) where W 1 ∈ R r × d x , and denote by e the rank of W .

Original: If rk( W 1 ) =
Revised: If rk( W 1 ) =

Original: e , then W 2 = MW 1 for some M ∈ R ( d y − r ) × r .
Revised: e , then W 2 = MW 1 for some M ∈ R ( d y − r ) × r .

Original: Claim 2 .
Revised: Claim 2 .

Original: If b = 1 , ( A 2 , A 1 ) ∈ µ − 1 2 ( W ) , rk( W 1 )
Revised: If b = 1 , ( A 2 , A 1 ) ∈ µ − 1 2 ( W ) , rk( W 1 )

Original: =
Revised: =

Original: e and W 2 = MW 1 , then there is a continuous function f : [0 , 1] → µ − 1 2 ( W ) with f (0) =
Revised: e and W 2 = MW 1 , then there is a continuous function f : [0 , 1] → µ − 1 2 ( W ) with f (0) =

Original: ( A 2 , A 1 ) and f (1) =
Revised: ( A 2 , A 1 ) and f (1) =

Original: ( (cid:2) I r M (cid:3) , W 1 ) .
Revised: ( (cid:2) I r M (cid:3) , W 1 ) .

Original: Proof.
Revised: Proof.

Original: Since rk( W ) < r , we have that rk( A 2 ) <
Revised: Since rk( W ) < r , we have that rk( A 2 ) <

Original: r or rk( A 1 ) < r .
Revised: r or rk( A 1 ) < r .

Original: If rk( A 2 ) < r , we proceed as in the proof of Claim 1 to ﬁnd a path in µ − 1 2 ( W ) from ( A 2 , A 1 ) to ( A (cid:48) 2 , A (cid:48) 1 ) such that rk( A (cid:48) 2 ) =
Revised: If rk( A 2 ) < r , we proceed as in the proof of Claim 1 to ﬁnd a path in µ − 1 2 ( W ) from ( A 2 , A 1 ) to ( A (cid:48) 2 , A (cid:48) 1 ) such that rk( A (cid:48) 2 ) =

Original: r .
Revised: r .

Original: Hence, we may assume that rk( A 2 ) =
Revised: Hence, we may assume that rk( A 2 ) =

Original: r .
Revised: r .

Original: This implies that rk( A 1 ) =
Revised: This implies that rk( A 1 ) =

Original: e .
Revised: e .

Original: So K := ker
Revised: So K := ker

Original: ( A T 1 ) ⊆
Revised: ( A T 1 ) ⊆

Original: R r has positive dimension r − e .
Revised: R r has positive dimension r − e .

Original: We write A 2 = (cid:2)
Revised: We write A 2 = (cid:2)

Original: A 21 A 22 (cid:3) where A 21 ∈ R r × r , and denote by r 2 the rank of A 21 .
Revised: A 21 A 22 (cid:3) where A 21 ∈ R r × r , and denote by r 2 the rank of A 21 .

Original: So the rowspace R ⊆ R r of A 21 has dimension r 2 .
Revised: So the rowspace R ⊆ R r of A 21 has dimension r 2 .

Original: We now show that K + R = R r .
Revised: We now show that K + R = R r .

Original: To see this we set δ := dim( K ∩ R ) .
Revised: To see this we set δ := dim( K ∩ R ) .

Original: Without loss of generality, we may assume that the ﬁrst e rows of
Revised: Without loss of generality, we may assume that the ﬁrst e rows of

Original: W 1 are linearly independent.
Revised: W 1 are linearly independent.

Original: Then the ﬁrst e rows of A 21 must also be linearly independent, so we might further assume that the ﬁrst r 2 rows of A 21 are linearly independent.
Revised: Then the ﬁrst e rows of A 21 must also be linearly independent, so we might further assume that the ﬁrst r 2 rows of A 21 are linearly independent.

Original: We denote by A 211 and
Revised: We denote by A 211 and

Original: W 11 the matrices formed by the ﬁrst r 2 rows of A 21 and W 1 , respectively.
Revised: W 11 the matrices formed by the ﬁrst r 2 rows of A 21 and W 1 , respectively.

Original: In particular, we have that
Revised: In particular, we have that

Original: A 

Revised: A

Original: A 1 =
Revised: A 1 =

Original: W 11 .
Revised: W 11 .

Original: Now we choose a basis ( b 1 , . . .
Revised: Now we choose a basis ( b 1 , . . .

Original: , b r 2 ) for R such that ( b 1 , . . .
Revised: , b r 2 ) for R such that ( b 1 , . . .

Original: , b δ ) is a basis for
Revised: , b δ ) is a basis for

Original: K ∩ R .
Revised: K ∩ R .

Original: Since R is the rowspace of A 211 , there is some G ∈ GL( r 2 ) such that the i -th row of GAis b i .
Revised: Since R is the rowspace of A 211 , there is some G ∈ GL( r 2 ) such that the i -th row of GAis b i .

Original: So the ﬁrst δ rows of GW 11 = GA 

Revised: So the ﬁrst δ rows of GW 11 = GA

Original: A 1 are zero, which shows that e = rk( GW 11 ) ≤ r 2 − δ .
Revised: A 1 are zero, which shows that e = rk( GW 11 ) ≤ r 2 − δ .

Original: Thus, dim( K + R ) =
Revised: Thus, dim( K + R ) =

Original: ( r − e ) + r 2 − δ ≥ r , which proves that K + R = R r .
Revised: ( r − e ) + r 2 − δ ≥ r , which proves that K + R = R r .

Original: If r 2 < r , we now show that there is a path in µ − 1 2 ( W ) from ( A 2 , A 1 ) to ( A (cid:48)(cid:48) 

Revised: If r 2 < r , we now show that there is a path in µ − 1 2 ( W ) from ( A 2 , A 1 ) to ( A (cid:48)(cid:48)

Original: = (cid:104) A (cid:48)(cid:48) 

Revised: = (cid:104) A (cid:48)(cid:48)

Original: A 22 (cid:105) , A 1 ) such that rk( A (cid:48)(cid:48) 21 ) =
Revised: A 22 (cid:105) , A 1 ) such that rk( A (cid:48)(cid:48) 21 ) =

Original: r .
Revised: r .

Original: We may assume again that the ﬁrst r 2 rows a 1 , . .
Revised: We may assume again that the ﬁrst r 2 rows a 1 , . .

Original: . , a r 2 of A 21 are linearly independent, i.e. , that they form a basis for R .
Revised: . , a r 2 of A 21 are linearly independent, i.e. , that they form a basis for R .

Original: Since K + R = R r , we can extend this basis to a basis ( a 1 , . .
Revised: Since K + R = R r , we can extend this basis to a basis ( a 1 , . .

Original: . , a r ) for R r such that a i ∈ K for all i > r 2 .
Revised: . , a r ) for R r such that a i ∈ K for all i > r 2 .

Original: We deﬁne A (cid:48)(cid:48) 21 such that its ﬁrst r 2 rows are a 1 , . . .
Revised: We deﬁne A (cid:48)(cid:48) 21 such that its ﬁrst r 2 rows are a 1 , . . .

Original: , a r 2 and such that its i -th row, for r 2 <
Revised: , a r 2 and such that its i -th row, for r 2 <

Original: i ≤ r , is the sum of a i ∈ K and the i -th row of A 21 .
Revised: i ≤ r , is the sum of a i ∈ K and the i -th row of A 21 .

Original: Then A (cid:48)(cid:48) 2 = (cid:104) A (cid:48)(cid:48) 

Revised: Then A (cid:48)(cid:48) 2 = (cid:104) A (cid:48)(cid:48)

Original: A 22 (cid:105) satisﬁes A (cid:48)(cid:48) 

Revised: A 22 (cid:105) satisﬁes A (cid:48)(cid:48)

Original: A 1 = W .
Revised: A 1 = W .

Original: Moreover, the straight line from ( A 2 , A 1 ) to ( A (cid:48)(cid:48) 2 , A 1 ) is a path in µ − 1 2 ( W ) .
Revised: Moreover, the straight line from ( A 2 , A 1 ) to ( A (cid:48)(cid:48) 2 , A 1 ) is a path in µ − 1 2 ( W ) .

Original: Since the last r − r 2 rows of A 21 are contained in the linear span R of the ﬁrst rrows of A 21 , the linearity of the determinant implies that det( A (cid:48)(cid:48) 21 ) =
Revised: Since the last r − r 2 rows of A 21 are contained in the linear span R of the ﬁrst rrows of A 21 , the linearity of the determinant implies that det( A (cid:48)(cid:48) 21 ) =

Original: det([ a 1 ··· a r ]) (cid:54) = 0 .
Revised: det([ a 1 ··· a r ]) (cid:54) = 0 .

Original: Thus, we may assume that r 2 = r .
Revised: Thus, we may assume that r 2 = r .

Original: If det( A 21 ) < 0 , we now construct a path in µ − 1 2 ( W ) from ( A 2 , A 1 ) to ( A (cid:48)(cid:48)(cid:48) 

Revised: If det( A 21 ) < 0 , we now construct a path in µ − 1 2 ( W ) from ( A 2 , A 1 ) to ( A (cid:48)(cid:48)(cid:48)

Original: = (cid:104) A (cid:48)(cid:48)(cid:48) 

Revised: = (cid:104) A (cid:48)(cid:48)(cid:48)

Original: A 22 (cid:105) , A 1 ) such that det( A (cid:48)(cid:48)(cid:48) 21 ) > 0 .
Revised: A 22 (cid:105) , A 1 ) such that det( A (cid:48)(cid:48)(cid:48) 21 ) > 0 .

Original: For this, we pick a vector v ∈ K \ { 0 } .
Revised: For this, we pick a vector v ∈ K \ { 0 } .

Original: Since the rows of A 21 form a basis for R r , there is an index i ∈ { 1 , . .
Revised: Since the rows of A 21 form a basis for R r , there is an index i ∈ { 1 , . .

Original: , r } such that the matrix
Revised: , r } such that the matrix

Original: D ∈ R r × r obtained from A 21 by replacing its i -th row with v has full rank.
Revised: D ∈ R r × r obtained from A 21 by replacing its i -th row with v has full rank.

Original: We pick µ ∈ R such that det( A 21 ) + µ det( D ) > 0 and deﬁne A (cid:48)(cid:48)(cid:48) 21 to be the matrix obtained from A 21 by adding µv to its i -th row.
Revised: We pick µ ∈ R such that det( A 21 ) + µ det( D ) > 0 and deﬁne A (cid:48)(cid:48)(cid:48) 21 to be the matrix obtained from A 21 by adding µv to its i -th row.

Original: Then det( A (cid:48)(cid:48)(cid:48) 21 ) = det( A 21 ) + µ det( D ) > 0 and A (cid:48)(cid:48)(cid:48) 

Revised: Then det( A (cid:48)(cid:48)(cid:48) 21 ) = det( A 21 ) + µ det( D ) > 0 and A (cid:48)(cid:48)(cid:48)

Original: =
Revised: =

Original: (cid:104) A (cid:48)(cid:48)(cid:48) 

Revised: (cid:104) A (cid:48)(cid:48)(cid:48)

Original: A 22 (cid:105) satisﬁes A (cid:48)(cid:48)(cid:48) 

Revised: A 22 (cid:105) satisﬁes A (cid:48)(cid:48)(cid:48)

Original: A 1 = W .
Revised: A 1 = W .

Original: Moreover, the straight line from ( A 2 , A 1 ) to ( A (cid:48)(cid:48)(cid:48) 2 , A 1 ) is a path in µ − 1 2 ( W ) .
Revised: Moreover, the straight line from ( A 2 , A 1 ) to ( A (cid:48)(cid:48)(cid:48) 2 , A 1 ) is a path in µ − 1 2 ( W ) .

Original: Therefore, we may assume that det( A 21 ) > 0 , so G :
Revised: Therefore, we may assume that det( A 21 ) > 0 , so G :

Original: =
Revised: =

Original: A − 1 21 ∈ GL + ( r ) .
Revised: A − 1 21 ∈ GL + ( r ) .

Original: Any path in GL + ( r ) from I r to G yields a path in µ − 1 2 ( W ) from ( A 2 , A 1 ) to ( A 2 G, G − 1 A 1 ) =
Revised: Any path in GL + ( r ) from I r to G yields a path in µ − 1 2 ( W ) from ( A 2 , A 1 ) to ( A 2 G, G − 1 A 1 ) =

Original: ( (cid:2)
Revised: ( (cid:2)

Original: I r A 22 G (cid:3) , W 1 ) .
Revised: I r A 22 G (cid:3) , W 1 ) .

Original: Since ( A 22 G ) W 1 = W 2 = MW 1 , the straight line from ( (cid:2) I r A 22 G (cid:3) , W 1 ) to ( (cid:2)
Revised: Since ( A 22 G ) W 1 = W 2 = MW 1 , the straight line from ( (cid:2) I r A 22 G (cid:3) , W 1 ) to ( (cid:2)

Original: I r M (cid:3) , W 1 ) is a path in µ − 1 2 ( W ) .
Revised: I r M (cid:3) , W 1 ) is a path in µ − 1 2 ( W ) .

Original: ♦
Revised: ♦

Original: Claim 3 .
Revised: Claim 3 .

Original: If θ =
Revised: If θ =

Original: ( A b +1 , . . .
Revised: ( A b +1 , . . .

Original: , A 1 ) ∈ µ − 1 2 ( W ) ,
Revised: , A 1 ) ∈ µ − 1 2 ( W ) ,

Original: rk( W 1 )
Revised: rk( W 1 )

Original: =
Revised: =

Original: e and W 2 = MW 1 , then there is a continuous function F : [0 , 1] → µ − 1 2 ( W ) with F (0)
Revised: e and W 2 = MW 1 , then there is a continuous function F : [0 , 1] → µ − 1 2 ( W ) with F (0)

Original: = θ and F (1) =
Revised: = θ and F (1) =

Original: ( (cid:2) I r M (cid:3) , I r , . . .
Revised: ( (cid:2) I r M (cid:3) , I r , . . .

Original: , I r , W 1 ) .
Revised: , I r , W 1 ) .

Original: Proof.
Revised: Proof.

Original: As e < r , at least one of the A i has rank smaller than r .
Revised: As e < r , at least one of the A i has rank smaller than r .

Original: We set k :=
Revised: We set k :=

Original: min { i ∈ { 1 , . . .
Revised: min { i ∈ { 1 , . . .

Original: , b +1 } | rk( A i ) < r } .
Revised: , b +1 } | rk( A i ) < r } .

Original: If k < b + 1 , we ﬁrst show that there is a path in µ − 1 2 ( W ) from θ to ( A (cid:48)
Revised: If k < b + 1 , we ﬁrst show that there is a path in µ − 1 2 ( W ) from θ to ( A (cid:48)

Original: b +1 , . . .
Revised: b +1 , . . .

Original: , A (cid:48) 1 ) such that min { i ∈ { 1 , . . .
Revised: , A (cid:48) 1 ) such that min { i ∈ { 1 , . . .

Original: , b + 1 } | rk( A (cid:48) i ) < r } =
Revised: , b + 1 } | rk( A (cid:48) i ) < r } =

Original: b + 1 .
Revised: b + 1 .

Original: Since rk( A k ) < r , the rank of
Revised: Since rk( A k ) < r , the rank of

Original: W (cid:48) :
Revised: W (cid:48) :

Original: = A k +

Revised: = A k +

Original: A k is smaller than r .
Revised: A k is smaller than r .

Original: We write W (cid:48) =
Revised: We write W (cid:48) =

Original: [ W (cid:48) 1 W (cid:48) 2 ] where W (cid:48) 1 has r columns.
Revised: [ W (cid:48) 1 W (cid:48) 2 ] where W (cid:48) 1 has r columns.

Original: Without loss of generality, we may assume that rk( W (cid:48) 1 ) =
Revised: Without loss of generality, we may assume that rk( W (cid:48) 1 ) =

Original: rk( W (cid:48) ) .
Revised: rk( W (cid:48) ) .

Original: Then there is a matrix N such that
Revised: Then there is a matrix N such that

Original: W (cid:48) 2 = W (cid:48) 1 N .
Revised: W (cid:48) 2 = W (cid:48) 1 N .

Original: Hence, we can apply the transposed version of Claim 2, which yields a path from ( A k +1 , A k ) to ( W (cid:48) 1 , [ I r N ]) in the set of factorizations of W (cid:48) .
Revised: Hence, we can apply the transposed version of Claim 2, which yields a path from ( A k +1 , A k ) to ( W (cid:48) 1 , [ I r N ]) in the set of factorizations of W (cid:48) .

Original: Deﬁning ˜
Revised: Deﬁning ˜

Original: A k +1 :
Revised: A k +1 :

Original: = W (cid:48) 1 , ˜
Revised: = W (cid:48) 1 , ˜

Original: A k :=
Revised: A k :=

Original: [ I r N ] and ˜ A i :=
Revised: [ I r N ] and ˜ A i :=

Original: A i for all i ∈ { 1 , . . .
Revised: A i for all i ∈ { 1 , . . .

Original: , b + 1 } \ { k, k + 1 } , extends this path to a path in µ − 1 2 ( W ) from θ to ( ˜ A b +1 , . .
Revised: , b + 1 } \ { k, k + 1 } , extends this path to a path in µ − 1 2 ( W ) from θ to ( ˜ A b +1 , . .

Original: ˜ A 1 ) such that min { i ∈ { 1 , . . .
Revised: ˜ A 1 ) such that min { i ∈ { 1 , . . .

Original: , b + 1 } | rk( ˜ A i ) < r } = k + 1 .
Revised: , b + 1 } | rk( ˜ A i ) < r } = k + 1 .

Original: We note that this construction increased the number k .
Revised: We note that this construction increased the number k .

Original: So we can repeat the construction until we reach ( A (cid:48)
Revised: So we can repeat the construction until we reach ( A (cid:48)

Original: b +1 , . . .
Revised: b +1 , . . .

Original: , A (cid:48) 1 ) as desired.
Revised: , A (cid:48) 1 ) as desired.

Original: Hence, we may assume that k = b + 1 .
Revised: Hence, we may assume that k = b + 1 .

Original: Since rk( A b +1 ) < r , the rank of W (cid:48)(cid:48) :
Revised: Since rk( A b +1 ) < r , the rank of W (cid:48)(cid:48) :

Original: = A b +

Revised: = A b +

Original: A b is smaller than r .
Revised: A b is smaller than r .

Original: We write W (cid:48)(cid:48) =
Revised: We write W (cid:48)(cid:48) =

Original: (cid:104) W (cid:48)(cid:48) 1 W (cid:48)(cid:48) 2 (cid:105) where W (cid:48)(cid:48) 1 has r rows.
Revised: (cid:104) W (cid:48)(cid:48) 1 W (cid:48)(cid:48) 2 (cid:105) where W (cid:48)(cid:48) 1 has r rows.

Original: Since W 1 = W (cid:48)(cid:48) 1 A b − 1 · · · Aand the matrices
Revised: Since W 1 = W (cid:48)(cid:48) 1 A b − 1 · · · Aand the matrices

Original: A b − 1 , . .
Revised: A b − 1 , . .

Original: , A 1 have rank r , we have that rk( W (cid:48)(cid:48) 1 )
Revised: , A 1 have rank r , we have that rk( W (cid:48)(cid:48) 1 )

Original: =
Revised: =

Original: rk( W 1 ) =
Revised: rk( W 1 ) =

Original: e .
Revised: e .

Original: Analogously, rk( W (cid:48)(cid:48) ) =
Revised: Analogously, rk( W (cid:48)(cid:48) ) =

Original: e .
Revised: e .

Original: So there is matrix M (cid:48) such that W (cid:48)(cid:48) 2 = M (cid:48) W (cid:48)(cid:48) 1 .
Revised: So there is matrix M (cid:48) such that W (cid:48)(cid:48) 2 = M (cid:48) W (cid:48)(cid:48) 1 .

Original: Applying Claim 2 yields a path from ( A b +1 , A b ) to ( (cid:2)
Revised: Applying Claim 2 yields a path from ( A b +1 , A b ) to ( (cid:2)

Original: I r M (cid:48) (cid:3) , W (cid:48)(cid:48) 1 ) in the set of factorizations of W (cid:48)(cid:48) .
Revised: I r M (cid:48) (cid:3) , W (cid:48)(cid:48) 1 ) in the set of factorizations of W (cid:48)(cid:48) .

Original: This path can be extended to a path in µ − 1 2 ( W ) from θ to θ (cid:48) :
Revised: This path can be extended to a path in µ − 1 2 ( W ) from θ to θ (cid:48) :

Original: = ( (cid:2) I r M (cid:48) (cid:3) , W (cid:48)(cid:48) 1 , A b − 1 , . . .
Revised: = ( (cid:2) I r M (cid:48) (cid:3) , W (cid:48)(cid:48) 1 , A b − 1 , . . .

Original: , A 1 ) .
Revised: , A 1 ) .

Original: Applying the same construction on
Revised: Applying the same construction on

Original: W (cid:48)(cid:48)(cid:48) :
Revised: W (cid:48)(cid:48)(cid:48) :

Original: = W (cid:48)(cid:48) 

Revised: = W (cid:48)(cid:48)

Original: A b − 1 yields a path in µ − 1 2 ( W ) from θ (cid:48) to θ (cid:48)(cid:48) :
Revised: A b − 1 yields a path in µ − 1 2 ( W ) from θ (cid:48) to θ (cid:48)(cid:48) :

Original: = ( (cid:2) I r M (cid:48) (cid:3) , I r , W (cid:48)(cid:48) 

Revised: = ( (cid:2) I r M (cid:48) (cid:3) , I r , W (cid:48)(cid:48)

Original: A b − 1 , . . .
Revised: A b − 1 , . . .

Original: , A 1 ) .
Revised: , A 1 ) .

Original: We repeat the contruction until ( (cid:2)
Revised: We repeat the contruction until ( (cid:2)

Original: I r M (cid:48) (cid:3) , I r , . .
Revised: I r M (cid:48) (cid:3) , I r , . .

Original: , I r , W 1 ) is reached.
Revised: , I r , W 1 ) is reached.

Original: Since M (cid:48)
Revised: Since M (cid:48)

Original: W 1 =
Revised: W 1 =

Original: W 2 = MW 1 , the straight line from ( (cid:2)
Revised: W 2 = MW 1 , the straight line from ( (cid:2)

Original: I r M (cid:48) (cid:3) , I r , . .
Revised: I r M (cid:48) (cid:3) , I r , . .

Original: , I r , W 1 ) to ( (cid:2)
Revised: , I r , W 1 ) to ( (cid:2)

Original: I r M (cid:3) , I r , . .
Revised: I r M (cid:3) , I r , . .

Original: , I r , W 1 ) is a path in µ − 1 2 ( W ) .
Revised: , I r , W 1 ) is a path in µ − 1 2 ( W ) .

Original: ♦
Revised: ♦

Original: Now we ﬁnally show that µ − 1 2 ( W ) is path-connected.
Revised: Now we ﬁnally show that µ − 1 2 ( W ) is path-connected.

Original: Without loss of generality, we may assume that rk( W 1 ) =
Revised: Without loss of generality, we may assume that rk( W 1 ) =

Original: e , and we write W 2 = MW 1 .
Revised: e , and we write W 2 = MW 1 .

Original: For θ, θ (cid:48) ∈ µ − 1 2 ( W ) , there are paths in µ − 1 2 ( W ) from θ resp.
Revised: For θ, θ (cid:48) ∈ µ − 1 2 ( W ) , there are paths in µ − 1 2 ( W ) from θ resp.

Original: θ (cid:48) to ( (cid:2)
Revised: θ (cid:48) to ( (cid:2)

Original: I r M (cid:3) , I r , . .
Revised: I r M (cid:3) , I r , . .

Original: , I r , W 1 ) , so there is a path from θ to θ (cid:48) in µ − 1 2 ( W ) .
Revised: , I r , W 1 ) , so there is a path from θ to θ (cid:48) in µ − 1 2 ( W ) .

Original: To settle the proof of Theorem 5, it is left to show that µ − 1 2 ( W ) and µ − 1 d ( W ) have indeed the same number of (path-)connected components, as we promised earlier.
Revised: To settle the proof of Theorem 5, it is left to show that µ − 1 2 ( W ) and µ − 1 d ( W ) have indeed the same number of (path-)connected components, as we promised earlier.

Original: Proposition 26.
Revised: Proposition 26.

Original: Let b > 0 and W ∈ R d y × d x .
Revised: Let b > 0 and W ∈ R d y × d x .

Original: Then µ − 1 d ( W ) and µ − 1 2 ( W ) have the same number of connected components.
Revised: Then µ − 1 d ( W ) and µ − 1 2 ( W ) have the same number of connected components.

Original: Moreover, each of these components is path-connected.
Revised: Moreover, each of these components is path-connected.

Original: Proof.
Revised: Proof.

Original: Let us denote the connected components of µ − 1 2 ( W ) be C 1 , . .
Revised: Let us denote the connected components of µ − 1 2 ( W ) be C 1 , . .

Original: C k .
Revised: C k .

Original: By Corollary 24 and
Revised: By Corollary 24 and

Original: Proposition 25, each of these components is path-connected.
Revised: Proposition 25, each of these components is path-connected.

Original: Using the notation in (11), we have that µ − 1 d ( W ) =
Revised: Using the notation in (10), we have that µ − 1 d ( W ) =

Original: (cid:83) ki =1 µ − 1 1 ( C i ) .
Revised: (cid:83) ki =1 µ − 1 1 ( C i ) .

Original: Since the µ − 1 1 ( C 1 ) , . . . , µ − 1 1 ( C k ) are pairwise disconnected, we see that µ − 1 d ( W ) has at least k disconnected components.
Revised: Since the µ − 1 1 ( C 1 ) , . . . , µ − 1 1 ( C k ) are pairwise disconnected, we see that µ − 1 d ( W ) has at least k disconnected components.

Original: It is left to show that each µ − 1 1 ( C i ) is path-connected.
Revised: It is left to show that each µ − 1 1 ( C i ) is path-connected.

Original: For this, let θ, θ (cid:48) ∈ µ − 1 1 ( C i ) and σ := µ 1 ( θ ) , σ (cid:48) :
Revised: For this, let θ, θ (cid:48) ∈ µ − 1 1 ( C i ) and σ := µ 1 ( θ ) , σ (cid:48) :

Original: = µ 1 ( θ (cid:48) )
Revised: = µ 1 ( θ (cid:48) )

Original: ∈ C i .
Revised: ∈ C i .

Original: As C i is path-connected, there is a path in C i from σ to σ (cid:48) .
Revised: As C i is path-connected, there is a path in C i from σ to σ (cid:48) .

Original: The map µ 1 is a direct product of b + 1 matrix multiplication maps.
Revised: The map µ 1 is a direct product of b + 1 matrix multiplication maps.

Original: To each factor we can apply Proposition 20, which yields a path in µ − 1 1 ( C i ) from θ to θ (cid:48) .
Revised: To each factor we can apply Proposition 20, which yields a path in µ − 1 1 ( C i ) from θ to θ (cid:48) .

Original: Corollary 27.
Revised: Corollary 27.

Original: Let b > 0 and W ∈ R d y × d x .
Revised: Let b > 0 and W ∈ R d y × d x .

Original: If rk( W ) =
Revised: If rk( W ) =

Original: r , then µ − 1 d ( W ) has 2 b connected components, and each of these components is path-connected.
Revised: r , then µ − 1 d ( W ) has 2 b connected components, and each of these components is path-connected.

Original: If rk( W 
Revised: If rk( W ) <

Revised: r , then µ − 1 d ( W ) is path-connnected.

Original: Proof.
Revised: Proof.

Original: Combine Corollary 24 and Propositions 25 and 26.
Revised: Combine Corollary 24 and Propositions 25 and 26.

Original: Proof of Theorem 5.
Revised: Proof of Theorem 5.

Original: Theorem 5 is an amalgamation of Corollaries 21 and 27.
Revised: Theorem 5 is an amalgamation of Corollaries 21 and 27.

Original: A. P ROOFS OF P ROPOSITIONS 6, 7, 9, 10 AND
Revised: A. P ROOFS OF P ROPOSITIONS 6, 7, 9, 10 AND

Original: Proposition 6.
Revised: Proposition 6.

Original: If θ is such that dµ d ( θ ) has maximal rank (see Theorem 4), then θ ∈ Crit ( L )
Revised: If θ is such that dµ d ( θ ) has maximal rank (see Theorem 4), then θ ∈ Crit ( L )

Original: if and only if W ∈ Crit ( (cid:96) | M r ) , and θ is a minimum (resp., saddle, maximum) for L if and only if W is a minimum (resp., saddle, maximum) for (cid:96)
Revised: if and only if W ∈ Crit ( (cid:96) | M r ) , and θ is a minimum (resp., saddle, maximum) for L if and only if W is a minimum (resp., saddle, maximum) for (cid:96)

Original: | M r .
Revised: | M r .

Original: If rk( W ) =
Revised: If rk( W ) =

Original: r (which implies that dµ d ( θ ) has maximal rank) and θ ∈ Crit ( L ) , then all d -factorizations of W also belong to Crit ( L ) .
Revised: r (which implies that dµ d ( θ ) has maximal rank) and θ ∈ Crit ( L ) , then all d -factorizations of W also belong to Crit ( L ) .

Original: Proof.
Revised: Proof.

Original: If µ d is a local submersion at θ onto M r , then there exists an open neighborhood U of W in M r and an open neighborhood V of θ with the property that µ d acts as a projection from V onto U (see, e.g. , (Lee, 2003, Theorem 7.3)).
Revised: If µ d is a local submersion at θ onto M r , then there exists an open neighborhood U of W in M r and an open neighborhood V of θ with the property that µ d acts as a projection from V onto U (see, e.g. , (Lee, 2003, Theorem 7.3)).

Original: From this, we easily deduce that θ is a minimum (resp.
Revised: From this, we easily deduce that θ is a minimum (resp.

Original: saddle, maximum) for L if and only if W = µ d ( θ ) is a minimum (resp.
Revised: saddle, maximum) for L if and only if W = µ d ( θ ) is a minimum (resp.

Original: saddle, maximum) for (cid:96) | M r .
Revised: saddle, maximum) for (cid:96) | M r .

Original: Finally, if rk( W ) =
Revised: Finally, if rk( W ) =

Original: r , then dµ d ( θ ) has maximal rank for all θ ∈ µ − 1 d ( W ) by Theorem 4.
Revised: r , then dµ d ( θ ) has maximal rank for all θ ∈ µ − 1 d ( W ) by Theorem 4.

Original: Proposition 7.
Revised: Proposition 7.

Original: If θ ∈ Crit ( L ) with rk( W ) =
Revised: If θ ∈ Crit ( L ) with rk( W ) =

Original: e ≤ r , then W ∈ Crit ( (cid:96) | M e ) .
Revised: e ≤ r , then W ∈ Crit ( (cid:96) | M e ) .

Original: In other words, if rk( W ) < r , then θ ∈ Crit ( L ) implies that W is a critical point for the restriction of (cid:96) to a smaller determinantal variety M e (which is in the singular locus of the functional space M r in the non-ﬁlling case).
Revised: In other words, if rk( W ) < r , then θ ∈ Crit ( L ) implies that W is a critical point for the restriction of (cid:96) to a smaller determinantal variety M e (which is in the singular locus of the functional space M r in the non-ﬁlling case).

Original: Proof.
Revised: Proof.

Original: According to Theorem 4, if µ d ( θ )
Revised: According to Theorem 4, if µ d ( θ )

Original: =
Revised: =

Original: W with rk( W ) =
Revised: W with rk( W ) =

Original: e , then Im ( dµ d ( θ ))
Revised: e , then Im ( dµ d ( θ ))

Original: ⊃ T W M e .
Revised: ⊃ T W M e .

Original: This means that θ ∈ Crit ( L ) implies that W is critical for (cid:96) | M e .
Revised: This means that θ ∈ Crit ( L ) implies that W is critical for (cid:96) | M e .

Original: Proposition 9.
Revised: Proposition 9.

Original: Let θ ∈ Crit ( L ) be such that rk( W ) < r , and assume that d(cid:96) ( W ) (cid:54) = 0 .
Revised: Let θ ∈ Crit ( L ) be such that rk( W ) < r , and assume that d(cid:96) ( W ) (cid:54) = 0 .

Original: Then, for any neighborhood U of θ , there exists θ (cid:48) in U such that µ d ( θ (cid:48) )
Revised: Then, for any neighborhood U of θ , there exists θ (cid:48) in U such that µ d ( θ (cid:48) )

Original: =
Revised: =

Original: W but θ (cid:48) (cid:54)∈ Crit ( L ) .
Revised: W but θ (cid:48) (cid:54)∈ Crit ( L ) .

Original: In particular, θ is a saddle point.
Revised: In particular, θ is a saddle point.

Original: Proof.
Revised: Proof.

Original: Our proof is a modiﬁcation of an argument used in Zhang (2019).
Revised: Our proof is a modiﬁcation of an argument used in Zhang (2019).

Original: Let us ﬁrst consider the case that µ d is ﬁlling, so r = min { d h , d 0 } .
Revised: Let us ﬁrst consider the case that µ d is ﬁlling, so r = min { d h , d 0 } .

Original: Without loss of generality, we assume r
Revised: Without loss of generality, we assume r

Original: =
Revised: =

Original: d 0 .
Revised: d 0 .

Original: Recall that the image of dµ d ( θ ) is given by
Revised: Recall that the image of dµ d ( θ ) is given by

Original: [Equation]
Revised: [Equation]

Original: We ﬁrst note that Row ( W <h ) (cid:54) = R d 0 , for otherwise dµ d ( θ ) would be surjective, implying that d(cid:96) ( W ) = 0 .
Revised: We ﬁrst note that Row ( W <h ) (cid:54) = R d 0 , for otherwise dµ d ( θ ) would be surjective, implying that d(cid:96) ( W ) = 0 .

Original: We deﬁne i = max { j | Row ( W <j ) =
Revised: We deﬁne i = max { j | Row ( W <j ) =

Original: R d 0 } , so 1 ≤ i < h
Revised: R d 0 } , so 1 ≤ i < h

Original: (writing W < 1 = I d 0 ).
Revised: (writing W < 1 = I d 0 ).

Original: We have that 
Revised: We have that d(cid:96)

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Since Row ( W <i +1 ) (cid:40) R d 0 , we may ﬁnd w i ∈
Revised: Since Row ( W <i +1 ) (cid:40) R d 0 , we may ﬁnd w i ∈

Original: R
Revised: R

Original: d
Revised: d

Original: i , w i (cid:54) = 

Revised: i , w i (cid:54) =

Original: such that w Ti W i . .
Revised: such that w Ti W i . .

Original: W 1 = 0 .
Revised: W 1 = 0 .

Original: We now ﬁx (cid:15) > 0 and v i +1 ∈
Revised: We now ﬁx (cid:15) > 0 and v i +1 ∈

Original: R d
Revised: R d

Original: i +1 arbitrarily, and deﬁne ˜ W
Revised: i +1 arbitrarily, and deﬁne ˜ W

Original: i +

Revised: i +

Original: = W i +

Revised: = W i +

Original: + (cid:15) ( v i +1 ⊗ w i ) .
Revised: + (cid:15) ( v i +1 ⊗ w i ) .

Original: Clearly, we have that ˜ W
Revised: Clearly, we have that ˜ W

Original: i +1 W i . . .
Revised: i +1 W i . . .

Original: W 

Revised: W

Original: = W i +1 W i . . .
Revised: = W i +1 W i . . .

Original: W 1 .
Revised: W 1 .

Original: If ( W h , . . . ,
Revised: If ( W h , . . . ,

Original: ˜ W
Revised: ˜ W

Original: i +1 , W i , . . .
Revised: i +1 , W i , . . .

Original: , W 1 ) is also a critical point of L , then d(cid:96)
Revised: , W 1 ) is also a critical point of L , then d(cid:96)

Original: [Equation]
Revised: [Equation]

Original: Col

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Combining (13) and (14), we have that
Revised: Combining (12) and (13), we have that

Original: [Equation]
Revised: [Equation]

Original: v [Equation] If this were true for all v i +1 , then it would imply 
Revised: If this were true for all v i +1 , then it would imply [Equation]

Original: [Equation]
Revised: [Equation]

Original: Hence, we have either found an arbitrarily small perturbation θ (cid:48) of θ as required in Proposition 9, or (15) must hold.
Revised: Hence, we have either found an arbitrarily small perturbation θ (cid:48) of θ as required in Proposition 9, or (14) must hold.

Original: In the latter case, we reapply the same argument for ˜ W
Revised: In the latter case, we reapply the same argument for ˜ W

Original: i
Revised: i

Original: +

Revised: +

Original: =
Revised: =

Original: W i +

Revised: W i +

Original: + (cid:15) ( v i +2 ⊗ w i +1 ) where w i +1 (cid:54)
Revised: + (cid:15) ( v i +2 ⊗ w i +1 ) where w i +1 (cid:54)

Original: = 0 and w Ti +1 W
Revised: = 0 and w Ti +1 W

Original: i +1 . . .
Revised: i +1 . . .

Original: W 1 = 0 .
Revised: W 1 = 0 .

Original: Again, we can either construct an arbitrarily small perturbation θ (cid:48) of θ as required in Proposition 9, or we have d(cid:96) ( W )( Col ( W >i +2 ) ⊗ R d 0 )
Revised: Again, we can either construct an arbitrarily small perturbation θ (cid:48) of θ as required in Proposition 9, or we have d(cid:96) ( W )( Col ( W >i +2 ) ⊗ R d 0 )

Original: = 0 .
Revised: = 0 .

Original: Proceeding this way we eventually arrive at d(cid:96) ( W )( R d h ⊗ R d 0 )
Revised: Proceeding this way we eventually arrive at d(cid:96) ( W )( R d h ⊗ R d 0 )

Original: = 

Revised: =

Original: so d(cid:96) ( W ) = 0 , which contradicts the hypothesis.
Revised: so d(cid:96) ( W ) = 0 , which contradicts the hypothesis.

Original: Thus, at some point we must ﬁnd an arbitrarily small perturbation θ (cid:48) of θ as required in Proposition 9, which concludes the proof in the case that µ d is ﬁlling.
Revised: Thus, at some point we must ﬁnd an arbitrarily small perturbation θ (cid:48) of θ as required in Proposition 9, which concludes the proof in the case that µ d is ﬁlling.

Original: We now consider the case that µ d is not ﬁlling.
Revised: We now consider the case that µ d is not ﬁlling.

Original: We pick i ∈ { 1 , . . .
Revised: We pick i ∈ { 1 , . . .

Original: , h − 1 } such that d i = r , and write for simplicity
Revised: , h − 1 } such that d i = r , and write for simplicity

Original: A = W h . . .
Revised: A = W h . . .

Original: W i +1 and B =
Revised: W i +1 and B =

Original: W i . . .
Revised: W i . . .

Original: W 1 .
Revised: W 1 .

Original: The assumption rk( W ) <
Revised: The assumption rk( W ) <

Original: r implies that rk( A ) <
Revised: r implies that rk( A ) <

Original: r or rk( B ) < r , and we assume without loss of generality that rk( A )
Revised: r or rk( B ) < r , and we assume without loss of generality that rk( A )

Original: < r .
Revised: < r .

Original: We deﬁne the map L B ( W (cid:48) h , . . .
Revised: We deﬁne the map L B ( W (cid:48) h , . . .

Original: , W (cid:48)
Revised: , W (cid:48)

Original: i +1 )
Revised: i +1 )

Original: = (cid:96) ( W (cid:48)
Revised: = (cid:96) ( W (cid:48)

Original: h . . .
Revised: h . . .

Original: W (cid:48)
Revised: W (cid:48)

Original: i +1 B ) .
Revised: i +1 B ) .

Original: We also introduce the map (cid:96) B ( A (cid:48) )
Revised: We also introduce the map (cid:96) B ( A (cid:48) )

Original: = (cid:96) ( A (cid:48) B ) and the matrix multiplication map µ d A
Revised: = (cid:96) ( A (cid:48) B ) and the matrix multiplication map µ d A

Original: where d A
Revised: where d A

Original: =
Revised: =

Original: ( d h , . . .
Revised: ( d h , . . .

Original: d i +1 ) , so that L B = (cid:96) B ◦ µ d A .
Revised: d i +1 ) , so that L B = (cid:96) B ◦ µ d A .

Original: If θ is a critical point for L , then θ A
Revised: If θ is a critical point for L , then θ A

Original: = ( W h , . .
Revised: = ( W h , . .

Original: , W i +1 ) must be a critical point for L B .
Revised: , W i +1 ) must be a critical point for L B .

Original: We are thus in the position to apply the analysis carried out in the ﬁlling case.
Revised: We are thus in the position to apply the analysis carried out in the ﬁlling case.

Original: In particular, we have that either θ A can be perturbed to ˜ θ
Revised: In particular, we have that either θ A can be perturbed to ˜ θ

Original: A such that µ d A ( θ A ) = µ d A (˜ θ A ) but ˜ θ A / ∈ Crit ( L B ) , or d(cid:96) B ( A ) = 0 .
Revised: A such that µ d A ( θ A ) = µ d A (˜ θ A ) but ˜ θ A / ∈ Crit ( L B ) , or d(cid:96) B ( A ) = 0 .

Original: In the former case, we have that θ (cid:48) =
Revised: In the former case, we have that θ (cid:48) =

Original: (˜ θ A , θ B ) is not a critical point for L , and we are done.
Revised: (˜ θ A , θ B ) is not a critical point for L , and we are done.

Original: If instead d(cid:96) B ( A ) = 0 , then we have that
Revised: If instead d(cid:96) B ( A ) = 0 , then we have that

Original: [Equation]
Revised: [Equation]

Original: because the image of the differential of the map A (cid:48) (cid:55)→ A (cid:48) B is given by R d h ⊗ Row ( B ) .
Revised: because the image of the differential of the map A (cid:48) (cid:55)→ A (cid:48) B is given by R d h ⊗ Row ( B ) .

Original: We now proceed in a similar manner as before.
Revised: We now proceed in a similar manner as before.

Original: We have that Col ( W >i ) (cid:40) R d h , because we assumed that W >i
Revised: We have that Col ( W >i ) (cid:40) R d h , because we assumed that W >i

Original: = A had rank less than r ≤ d h .
Revised: = A had rank less than r ≤ d h .

Original: Thus, we may ﬁnd w i +1 ∈
Revised: Thus, we may ﬁnd w i +1 ∈

Original: R d
Revised: R d

Original: i , w i +1 (cid:54) = 

Revised: i , w i +1 (cid:54) =

Original: such that
Revised: such that

Original: W h . . .
Revised: W h . . .

Original: W i +

Revised: W i +

Original: w i +

Revised: w i +

Original: = 0 .
Revised: = 0 .

Original: We ﬁx (cid:15) > 0 and v i arbitrarily, and deﬁne ˜
Revised: We ﬁx (cid:15) > 0 and v i arbitrarily, and deﬁne ˜

Original: W i
Revised: W i

Original: =
Revised: =

Original: W i + (cid:15) ( w i +1 ⊗ v i ) .
Revised: W i + (cid:15) ( w i +1 ⊗ v i ) .

Original: We have that W h . . .
Revised: We have that W h . . .

Original: W i +

Revised: W i +

Original: ˜ W
Revised: ˜ W

Original: i
Revised: i

Original: =
Revised: =

Original: W h . . .
Revised: W h . . .

Original: W i +

Revised: W i +

Original: W i .
Revised: W i .

Original: If for all choices of v i we have that ( W h , . . .
Revised: If for all choices of v i we have that ( W h , . . .

Original: , ˜ W i , . . .
Revised: , ˜ W i , . . .

Original: , W 1 ) is still a critical point for L , then we can deduce that
Revised: , W 1 ) is still a critical point for L , then we can deduce that

Original: [Equation]
Revised: [Equation]

Original: Repeating this reasoning, we obtain our result as before.
Revised: Repeating this reasoning, we obtain our result as before.

Original: Proposition 10.
Revised: Proposition 10.

Original: Let (cid:96) be any smooth convex function, and let L = (cid:96) ◦ µ d .
Revised: Let (cid:96) be any smooth convex function, and let L = (cid:96) ◦ µ d .

Original: If θ is a non-global local minimum for L , then necessarily rk( W ) =
Revised: If θ is a non-global local minimum for L , then necessarily rk( W ) =

Original: r (so θ is a pure critical point).
Revised: r (so θ is a pure critical point).

Original: In particular, L has non-global minima if and only if (cid:96) | M r has non-global minima.
Revised: In particular, L has non-global minima if and only if (cid:96) | M r has non-global minima.

Original: Proof.
Revised: Proof.

Original: The ﬁrst statement follows immediately from Proposition 9: if θ ∈ Crit ( L ) is a non-global local minimum, then necessarily d(cid:96) ( W ) (cid:54) = 0 , and we conclude that rk( W )
Revised: The ﬁrst statement follows immediately from Proposition 9: if θ ∈ Crit ( L ) is a non-global local minimum, then necessarily d(cid:96) ( W ) (cid:54) = 0 , and we conclude that rk( W )

Original: =
Revised: =

Original: r .
Revised: r .

Original: For the second statement, we observe that if (cid:96) is a convex function, then θ is a local minimum for L if and only if
Revised: For the second statement, we observe that if (cid:96) is a convex function, then θ is a local minimum for L if and only if

Original: W = µ d ( θ ) is a local minimum for (cid:96) | M r .
Revised: W = µ d ( θ ) is a local minimum for (cid:96) | M r .

Original: Indeed, if W = µ d ( θ ) is a local minimum for (cid:96) | M r , then it is always true that any θ ∈ µ − 1 d ( W ) is a local minimum.
Revised: Indeed, if W = µ d ( θ ) is a local minimum for (cid:96) | M r , then it is always true that any θ ∈ µ − 1 d ( W ) is a local minimum.

Original: Conversely, if θ is a local minimum, then from Proposition 9 we see that either d(cid:96) ( W ) = 0 , in which case W is a (global) minimum because (cid:96) is convex, or W must have maximal rank.
Revised: Conversely, if θ is a local minimum, then from Proposition 9 we see that either d(cid:96) ( W ) = 0 , in which case W is a (global) minimum because (cid:96) is convex, or W must have maximal rank.

Original: In the latter case, dµ d ( θ ) would be surjective (by Theorem 4), so W would also be a local minimum for (cid:96) | M r (see Proposition 6).
Revised: In the latter case, dµ d ( θ ) would be surjective (by Theorem 4), so W would also be a local minimum for (cid:96) | M r (see Proposition 6).

Original: Finally, it is clear that θ is also a global minimum for L if and only if W is a global minimum for (cid:96) | M r .
Revised: Finally, it is clear that θ is also a global minimum for L if and only if W is a global minimum for (cid:96) | M r .

Original: Proposition 14.
Revised: Proposition 14.

Original: If L (cid:48) =
Revised: If L (cid:48) =

Original: (cid:96) ◦
Revised: (cid:96) ◦

Original: ν
Revised: ν

Original: d and L = (cid:96) ◦ µ d , then the critical locus Crit ( L (cid:48) ) is in “correspondence” with Crit ( L )
Revised: d and L = (cid:96) ◦ µ d , then the critical locus Crit ( L (cid:48) ) is in “correspondence” with Crit ( L )

Original: ∩ Ω , meaning that
Revised: ∩ Ω , meaning that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let us deﬁne
Revised: Let us deﬁne

Original: [Equation]
Revised: [Equation]

Original: The image of both of these maps is N = { ( W h , . . .
Revised: The image of both of these maps is N = { ( W h , . . .

Original: , W 1 ) | (cid:107) W i (cid:107) =
Revised: , W 1 ) | (cid:107) W i (cid:107) =

Original: 1 , i = 1 , . . .
Revised: 1 , i = 1 , . . .

Original: , h − 1 } .
Revised: , h − 1 } .

Original: In fact, both maps are submersions onto N .
Revised: In fact, both maps are submersions onto N .

Original: Since ν d = µ d ◦ p and µ d ◦ q = µ d | Ω = ν d ◦ q , it is enough to show the following two assertions: 1) θ (cid:

Revised: Since ν d = µ d ◦ p and µ d ◦ q = µ d | Ω = ν d ◦ q , it is enough to show the following two assertions: 1) θ (cid:

Original: ) ∈ Crit ( L (cid:48) )
Revised: ) ∈ Crit ( L (cid:48) )

Original: if and only if p ( θ (cid:48) )
Revised: if and only if p ( θ (cid:48) )

Original: ∈ Crit ( L ) ; and 2) θ ∈ Crit ( L ) ∩ Ω
Revised: ∈ Crit ( L ) ; and 2) θ ∈ Crit ( L ) ∩ Ω

Original: if and only if q ( θ ) ∈ Crit ( L (cid:48) ) .
Revised: if and only if q ( θ ) ∈ Crit ( L (cid:48) ) .

Original: For 1), we deduce from L (cid:48)
Revised: For 1), we deduce from L (cid:48)

Original: =
Revised: =

Original: L ◦ p that dL (cid:48)
Revised: L ◦ p that dL (cid:48)

Original: ( θ (cid:48) )
Revised: ( θ (cid:48) )

Original: =
Revised: =

Original: dL ( p ( θ (cid:48) )) ◦ dp ( θ (cid:48) )
Revised: dL ( p ( θ (cid:48) )) ◦ dp ( θ (cid:48) )

Original: = 0 if dL ( p ( θ (cid:48) ))
Revised: = 0 if dL ( p ( θ (cid:48) ))

Original: = 0 , but this also holds conversely: if dL (cid:48)
Revised: = 0 , but this also holds conversely: if dL (cid:48)

Original: ( θ (cid:48) )
Revised: ( θ (cid:48) )

Original: = 0 , then Im ( dp ( θ (cid:48) )) is contained in Ker ( dL ( p ( θ (cid:48) ))) .
Revised: = 0 , then Im ( dp ( θ (cid:48) )) is contained in Ker ( dL ( p ( θ (cid:48) ))) .

Original: Since q ◦ p = p and both maps p and q are submersions, we have that Im ( dp ( θ (cid:48) ))
Revised: Since q ◦ p = p and both maps p and q are submersions, we have that Im ( dp ( θ (cid:48) ))

Original: =
Revised: =

Original: T p ( θ (cid:48) )
Revised: T p ( θ (cid:48) )

Original: N =
Revised: N =

Original: Im ( dq ( p ( θ (cid:48) ))) .
Revised: Im ( dq ( p ( θ (cid:48) ))) .

Original: Now it follows from L ◦ q =
Revised: Now it follows from L ◦ q =

Original: L | Ω that dL ( p ( θ (cid:48) ))
Revised: L | Ω that dL ( p ( θ (cid:48) ))

Original: =
Revised: =

Original: dL ( p ( θ (cid:48) )) ◦ dq ( p ( θ (cid:48) ))
Revised: dL ( p ( θ (cid:48) )) ◦ dq ( p ( θ (cid:48) ))

Original: = 0 .
Revised: = 0 .

Original: For 2), we can argue analogously, exchanging the roles of L and L (cid:48) as well as p and q .
Revised: For 2), we can argue analogously, exchanging the roles of L and L (cid:48) as well as p and q .

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: We consider a ﬁxed matrix Q 0 ∈ R d y × d x and a singular value decomposition (SVD) Q 0 = U Σ V T .
Revised: We consider a ﬁxed matrix Q 0 ∈ R d y × d x and a singular value decomposition (SVD) Q 0 = U Σ V T .

Original: Here U ∈ R d y × d y and V ∈ R d x × d x are orthogonal and Σ ∈ R d y ×
Revised: Here U ∈ R d y × d y and V ∈ R d x × d x are orthogonal and Σ ∈ R d y ×

Original: d x is diagonal with decreasing diagonal entries σ 1 , σ 2 , . .
Revised: d x is diagonal with decreasing diagonal entries σ 1 , σ 2 , . .

Original: , σ m where m = min( d x , d y ) .
Revised: , σ m where m = min( d x , d y ) .

Original: We also write shortly [ m ] = { 1 , 2 , . . .
Revised: We also write shortly [ m ] = { 1 , 2 , . . .

Original: , m } and denote by [ m ] r the set of all subsets of [ m ] of cardinality r .
Revised: , m } and denote by [ m ] r the set of all subsets of [ m ] of cardinality r .

Original: For I ∈ [ m ] r , we deﬁne Σ
Revised: For I ∈ [ m ] r , we deﬁne Σ

Original: I ∈ R d y ×
Revised: I ∈ R d y ×

Original: d x to be the diagonal matrix with entries σ
Revised: d x to be the diagonal matrix with entries σ

Original: I , 1 , σ I , 2 , . . .
Revised: I , 1 , σ I , 2 , . . .

Original: , σ I ,m where σ I ,i =
Revised: , σ I ,m where σ I ,i =

Original: σ i if i ∈ I and σ I ,i = 

Revised: σ i if i ∈ I and σ I ,i =

Original: otherwise.
Revised: otherwise.

Original: These matrices yield the critical points of the function h Q 0 ( P )
Revised: These matrices yield the critical points of the function h Q 0 ( P )

Original: = (cid:107) P − Q 0 (cid:107)restricted to the determinantal variety M r .
Revised: = (cid:107) P − Q 0 (cid:107)restricted to the determinantal variety M r .

Original: Theorem 28.
Revised: Theorem 28.

Original: If Q 0 / ∈ M r , the critical points of h Q 0 | M r are all matrices of the form U Σ
Revised: If Q 0 / ∈ M r , the critical points of h Q 0 | M r are all matrices of the form U Σ

Original: I V T where Q 0 =
Revised: I V T where Q 0 =

Original: U Σ V T is a SVD and I ∈ [ m ] r .
Revised: U Σ V T is a SVD and I ∈ [ m ] r .

Original: The local minima are the critical points with I =
Revised: The local minima are the critical points with I =

Original: [ r ] .
Revised: [ r ] .

Original: They are all global minima.
Revised: They are all global minima.

Original: Proof.
Revised: Proof.

Original: A matrix P ∈
Revised: A matrix P ∈

Original: M r is a critical point if and only if ( Q 

Revised: M r is a critical point if and only if ( Q

Original: − P ) ∈ T P M ⊥ r = Col ( P ) ⊥ ⊗
Revised: − P ) ∈ T P M ⊥ r = Col ( P ) ⊥ ⊗

Original: Row ( P ) ⊥ .
Revised: Row ( P ) ⊥ .

Original: If P = (cid:80) ri =1 σ (cid:48)
Revised: If P = (cid:80) ri =1 σ (cid:48)

Original: i ( u (cid:48)
Revised: i ( u (cid:48)

Original: i ⊗ v (cid:48) i ) and Q 0 − P
Revised: i ⊗ v (cid:48) i ) and Q 0 − P

Original: =
Revised: =

Original: (cid:80) ej
Revised: (cid:80) ej

Original: =1 σ (cid:48)(cid:48)
Revised: =1 σ (cid:48)(cid:48)

Original: j ( u (cid:48)(cid:48)
Revised: j ( u (cid:48)(cid:48)

Original: j ⊗ v (cid:48)(cid:48) j ) are SVD decompositions with σ (cid:48)
Revised: j ⊗ v (cid:48)(cid:48) j ) are SVD decompositions with σ (cid:48)

Original: i (cid:54) = 0 and σ (cid:48)(cid:48)
Revised: i (cid:54) = 0 and σ (cid:48)(cid:48)

Original: j (cid:54) = 0 , the column spaces of P and Q 0 − P are spanned by the u (cid:48)
Revised: j (cid:54) = 0 , the column spaces of P and Q 0 − P are spanned by the u (cid:48)

Original: i and u (cid:48)(cid:48) j , respectively.
Revised: i and u (cid:48)(cid:48) j , respectively.

Original: Similarly, the row spaces of P and Q 0 − P are spanned by the v (cid:48)
Revised: Similarly, the row spaces of P and Q 0 − P are spanned by the v (cid:48)

Original: i and v (cid:48)(cid:48) j , respectively.
Revised: i and v (cid:48)(cid:48) j , respectively.

Original: So P is a critical point if and only if the vectors u (cid:48)
Revised: So P is a critical point if and only if the vectors u (cid:48)

Original: i , u (cid:48)(cid:48)
Revised: i , u (cid:48)(cid:48)

Original: j and v (cid:48)
Revised: j and v (cid:48)

Original: i , v(cid:48)(cid:48) j are orthonormal, i.e. , if
Revised: i , v

Original: [Equation]
Revised: [Equation]

Original: +
Revised: j are orthonormal, i.e. , if

Original: [Equation]
Revised: [Equation]

Original: ) is a SVD of Q 0 .
Revised: is a SVD of Q 0 .

Original: This proves that the critical points are of the form U Σ I V T where Q 0 = U Σ V T is a SVD and I ∈ [ m ] r .
Revised: This proves that the critical points are of the form U Σ I V T where Q 0 = U Σ V T is a SVD and I ∈ [ m ] r .

Original: Since h Q 0 ( U Σ I V T ) = (cid:107) U Σ [ m ] \I V T (cid:107) 2 = (cid:107) Σ [ m ] \I (cid:107) 2 = (cid:80) i/ ∈I σ 2 i , we see that the global minima are exactly the critical points selecting r of the largest singular values of Q 0 , i.e. , with I =
Revised: Since h Q 0 ( U Σ I V T ) = (cid:107) U Σ [ m ] \I V T (cid:107) 2 = (cid:107) Σ [ m ] \I (cid:107) 2 = (cid:80) i/ ∈I σ 2 i , we see that the global minima are exactly the critical points selecting r of the largest singular values of Q 0 , i.e. , with I =

Original: [ r ] .
Revised: [ r ] .

Original: It is left to show that there are no other local minima.
Revised: It is left to show that there are no other local minima.

Original: For this, we consider a critical point P = U Σ
Revised: For this, we consider a critical point P = U Σ

Original: I V T such that at least one selected singular value σ i for i ∈
Revised: I V T such that at least one selected singular value σ i for i ∈

Original: I is strictly smaller than σ r .
Revised: I is strictly smaller than σ r .

Original: We will show now that P cannot be a local minimum.
Revised: We will show now that P cannot be a local minimum.

Original: Since σ i < σ r , there is some j ∈ [ r ] such that j / ∈ I .
Revised: Since σ i < σ r , there is some j ∈ [ r ] such that j / ∈ I .

Original: As above, we write u k and v k for the columns of U and V T such that Q 

Revised: As above, we write u k and v k for the columns of U and V T such that Q

Original: = (cid:80) mk =1 σ k ( u k ⊗ v k ) and P = (cid:80) k ∈I σ k ( u k ⊗ v k ) .
Revised: = (cid:80) mk =1 σ k ( u k ⊗ v k ) and P = (cid:80) k ∈I σ k ( u k ⊗ v k ) .

Original: We consider rotations in the planes spanned by u i , u j and v i , v j , respectively: for a ∈ [0 , π 2 ] , we set u ( α ) =
Revised: We consider rotations in the planes spanned by u i , u j and v i , v j , respectively: for a ∈ [0 , π 2 ] , we set u ( α ) =

Original: cos( α ) u i + sin( α ) u j and v ( α ) =
Revised: cos( α ) u i + sin( α ) u j and v ( α ) =

Original: cos( α ) v i + sin( α ) v j .
Revised: cos( α ) v i + sin( α ) v j .

Original: Note that u (0) =
Revised: Note that u (0) =

Original: u i and u ( π 2 ) =
Revised: u i and u ( π 2 ) =

Original: u j ; analogously for v ( α ) .
Revised: u j ; analogously for v ( α ) .

Original: Next we deﬁne σ ( α ) =
Revised: Next we deﬁne σ ( α ) =

Original: cos 2 ( α ) σ i + sin 2 ( α) σ j and
Revised: cos 2 ( α [Equation] σ i + sin 2 ( α

Original: P α =
Revised: [Equation] σ j and

Original: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: We note that P 0 =
Revised: We note that P 0 =

Original: P and P π 2 = U Σ I\{ i }∪{ j } V T are both critical points of h Q 

Revised: P and P π 2 = U Σ I\{ i }∪{ j } V T are both critical points of h Q

Original: | M r .
Revised: | M r .

Original: It remains to show that h Q 0 ( P α ) as a function in α is strictly decreasing on the interval [0 , π 2 ] .
Revised: It remains to show that h Q 0 ( P α ) as a function in α is strictly decreasing on the interval [0 , π 2 ] .

Original: From h Q
Revised: From h Q

Original: [Equation]
Revised: [Equation]

Revised: P

Original:  and u ( α ) ⊗ v ( α ) =
Revised: [Equation] ) = [Equation] and u ( α ) ⊗ v ( α ) =

Original: cos 2 ( α )( u i ⊗ v i ) 
Revised: cos 2 ( α )( u i ⊗ v i ) + cos [Equation] u

Original: + cos( α ) sin( α )( u i ⊗ v j + u j ⊗ v i )
Revised: [Equation] + u [Equation] ⊗ v i )

Original: + sin 2 ( α )( u j ⊗ v j ) , we deduce that
Revised: + sin 2 ( α )( u j ⊗ v j ) , we deduce that

Original: [Equation]
Revised: [Equation]

Original: The graph of the function f ( x ) =
Revised: The graph of the function f ( x ) =

Original: σ 2 i +2 σ j ( σ j − σ i ) x − ( σ j − σ i ) 2 x 2 , for x ∈ R , is a parabola with a unique local and global maximum at x 0 = [Equation] − σ i .
Revised: σ 2 i +2 σ j ( σ j − σ i ) x − ( σ j − σ i ) 2 x 2 , for x ∈ R , is a parabola with a unique local and global maximum at x 0 = σ j σ j − σ i .

Original: Since x 0 ≥ 1 , the function f is strictly increasing on the interval [0 , 1] .
Revised: Since x 0 ≥ 1 , the function f is strictly increasing on the interval [0 , 1] .

Original: Hence, h Q 0 ( P α )
Revised: Hence, h Q 0 ( P α )

Original: = (cid:80) k/ ∈I ,k (cid:54) =
Revised: = (cid:80) [Equation] [Equation]

Original:  j σ 2 k + f (cos 2 ( α )) is strictly decreasing on [0 , π 2 ] , which concludes the proof.
Revised: = j σ 2 k + f (cos 2 ( α )) is strictly decreasing on [0 , π 2 ] , which concludes the proof.

Original: If the singular values of Q 0 are pairwise distinct and positive, the singular vectors of Q 0 are unique up to sign.
Revised: If the singular values of Q 0 are pairwise distinct and positive, the singular vectors of Q 0 are unique up to sign.

Original: So for each index set I ∈
Revised: So for each index set I ∈

Original: [ m ] r the matrix Q I = U Σ
Revised: [ m ] r the matrix Q I = U Σ

Original: I V T is the unique critical point of h Q 0 | M r whose singular values are the σ i for i ∈
Revised: I V T is the unique critical point of h Q 0 | M r whose singular values are the σ i for i ∈

Original: I .
Revised: I .

Original: Hence, Theorem 28 implies immediately the following:
Revised: Hence, Theorem 28 implies immediately the following:

Original: Corollary 29.
Revised: Corollary 29.

Original: If the singular values of Q 0 are pairwise distinct and positive, h Q 0 | M r has exactly (cid:0) mr (cid:1) critical points, namely the Q I =
Revised: If the singular values of Q 0 are pairwise distinct and positive, h Q 0 | M r has exactly (cid:0) mr (cid:1) critical points, namely the Q I =

Original: U Σ
Revised: U Σ

Original: I V T for I ∈ [ m ] r .
Revised: I V T for I ∈ [ m ] r .

Original: Moreover, its unique local and global minimum is Q [ r ] .
Revised: Moreover, its unique local and global minimum is Q [ r ] .

Original: We can strengthen this result by explicitly calculating the index of each critical point, i.e. , the number of negative eigenvalues of the Hessian matrix.
Revised: We can strengthen this result by explicitly calculating the index of each critical point, i.e. , the number of negative eigenvalues of the Hessian matrix.

Original: Theorem 30.
Revised: Theorem 30.

Original: If the singular values of Q 0 are pairwise distinct and positive, the index of Q I as a critical point of h Q 0 | M r is 
Revised: If the singular values of Q 0 are pairwise distinct and positive, the index of Q I as a critical point of h Q 0 | M r is index

Original: [Equation]
Revised: [Equation]

Original:  To prove this assertion, we may assume without loss of generality that d y ≤ d x , so m = d y .
Revised: { ( j, i ) ∈ I × ([ m ] \ I ) | j > i } . To prove this assertion, we may assume without loss of generality that d y ≤ d x , so m = d y .

Original: We may further assume that Q 0 is a diagonal matrix, so Q 0 = Σ .
Revised: We may further assume that Q 0 is a diagonal matrix, so Q 0 = Σ .

Original: Let µ ( d y ,r,d x ) : R d y × r × R r × d x → R d y × d x be the matrix multiplication map, and L = h Σ ◦ µ ( d y ,r,d x ) .
Revised: Let µ ( d y ,r,d x ) : R d y × r × R r × d x → R d y × d x be the matrix multiplication map, and L = h Σ ◦ µ ( d y ,r,d x ) .

Original: For ( A, B ) ∈ µ − 1 ( d y ,r,d x ) (Σ I ) , Theoremimplies that the condition Σ I ∈ Crit ( h Σ | M r ) is equivalent to dL ( A, B ) = 0 .
Revised: For ( A, B ) ∈ µ − 1 ( d y ,r,d x ) (Σ I ) , Theoremimplies that the condition Σ I ∈ Crit ( h Σ | M r ) is equivalent to dL ( A, B ) = 0 .

Original: Moreover, the number of negative eigenvalues of the Hessian of L at any such factorization ( A, B ) of Σ
Revised: Moreover, the number of negative eigenvalues of the Hessian of L at any such factorization ( A, B ) of Σ

Original: I is the same.
Revised: I is the same.

Original: This number is the index of Σ I .
Revised: This number is the index of Σ I .

Original: So we can compute it by ﬁxing one speciﬁc factorization ( A, B ) of Σ I .
Revised: So we can compute it by ﬁxing one speciﬁc factorization ( A, B ) of Σ I .

Original: To compute the Hessian of L at ( A, B ) , we compute the partial derivatives of ﬁrst and second order of L : ∂L
Revised: To compute the Hessian of L at ( A, B ) , we compute the partial derivatives of ﬁrst and second order of L : [Equation]

Original: [Equation]
Revised: [Equation]

Original: To assemble these second order partial derivatives into a matrix, we choose the following order of the entries of ( A, B ) :a 11 , . .
Revised: To assemble these second order partial derivatives into a matrix [Equation] we choose the following order of the entries of ( A, B ) :

Original: . , a 1 r , a 21 , . . , a 2 r , . . . , a d y 1 , . . . , a d y r , b 11 , . .
Revised: [Equation]

Original: , b r 1 , b 12 , . . .

Original: , b r 2 , . . . , b 1 d x , . . . , b rd x . We denote by H the Hessian matrix of L with respect to this ordering at the following speciﬁcally chosen matrices ( A 0 , B 0 ) : denoting by i 1 , i 2 , . . .
Revised: We denote by H the Hessian matrix of L with respect to this ordering at the following speciﬁcally chosen matrices ( A 0 , B 0 ) : denoting by i 1 , i 2 , . . .

Original: , i r the entries of I in decreasing order, we pick the j -th column of A 0 to be the i j -th standard basis vector in R d y and the j -th row of B 0 to be the σ i j -multiple of the i j -th standard basis vector in R d x .
Revised: , i r the entries of I in decreasing order, we pick the j -th column of A 0 to be the i j -th standard basis vector in R d y and the j -th row of B 0 to be the σ i j -multiple of the i j -th standard basis vector in R d x .

Original: Note that A 0 B 0 = Σ
Revised: Note that A 0 B 0 = Σ

Original: I , A T 

Revised: I , A T

Original: A 

Revised: A

Original: = I r is the r × r -identity matrix, and B 0 B T 0 is the r × r -diagonal matrix with entries σ 2 i 1 , σ 2 i 2 , . . .
Revised: = I r is the r × r -identity matrix [Equation] and B 0 B T 0 is the r × r -diagonal matrix with entries σ 2 i 1 , σ 2 i 2 , . . .

Original: , σ 2 i r .
Revised: , σ 2 i r .

Original: We write
Revised: We write

Original: H = (cid:20) D
Revised: H = (cid:20) D

Original: M
Revised: M

Original: M T
Revised: M [Equation]

Original: N (cid:21) , where D ∈ R rd y ×
Revised: N (cid:21) , where D ∈ R rd y ×

Original: rd
Revised: rd

Original: y , N ∈ R
Revised: y , N ∈ R

Original: rd x × rd x , M ∈ R rd
Revised: rd x × rd x , M ∈ R rd

Original: y × rd x .
Revised: y × rd x .

Original: Our ﬁrst observation is that N , whose entries are described by (17), is twice the identity matrix, so N = 

Revised: Our ﬁrst observation is that N , whose entries are described by (16), is twice the identity matrix, so N =

Original: I rd x .
Revised: I rd x .

Original: Similarly, we see from (16) that D is a diagonal matrix.
Revised: Similarly, we see from (15) that D is a diagonal matrix.

Original: According to our ﬁxed ordering, the entries of D are indexed by pairs ( ij, kl ) of integers i, k ∈ [ d y ] and j, l ∈ [ r ] .
Revised: According to our ﬁxed ordering, the entries of D are indexed by pairs ( ij, kl ) of integers i, k ∈ [ d y ] and j, l ∈ [ r ] .

Original: With this, the diagonal entries of D are D ij,ij = 2 σ 2 i j .
Revised: With this, the diagonal entries of D are D ij,ij = 2 σ 2 i j .

Original: Analogously, the entries of M are indexed by pairs ( ij, kl ) of integers i ∈
Revised: Analogously, the entries of M are indexed by pairs ( ij, kl ) of integers i ∈

Original: [ d y ] , j, k ∈ [ r ] and l ∈ [ d x ] .
Revised: [ d y ] , j, k ∈ [ r ] and l ∈ [ d x ] .

Original: Lemma 31.
Revised: Lemma 31.

Original: Let i ∈
Revised: Let i ∈

Original: [ d y ] and j ∈ [ r ] .
Revised: [ d y ] and j ∈ [ r ] .

Original: The ij -th row of M has exactly one non-zero entry.
Revised: The ij -th row of M has exactly one non-zero entry.

Original: If i ∈ I , there is some k ∈ [ r ] with i =
Revised: If i ∈ I , there is some k ∈ [ r ] with i =

Original: i k and the non-zero entry is M ij,ki j = 2 σ i j .
Revised: i k and the non-zero entry is M ij,ki j = 2 σ i j .

Original: Otherwise, so if i / ∈ I , the non-zero entry is M ij,ji = − 2 σ i .
Revised: Otherwise, so if i / ∈ I , the non-zero entry is M ij,ji = − 2 σ i .

Original: Proof.
Revised: Proof.

Original: The entries of M are given by (18).
Revised: The entries of M are given by (17).

Original: We ﬁrst observe that ( A 0 )
Revised: We ﬁrst observe that ( A 0 )

Original: ik ( B 0 ) jl is non-zero if and only if i = i k and l
Revised: ik ( B 0 ) jl is non-zero if and only if i = i k and l

Original: = i j .
Revised: = i j .

Original: Moreover, we have that ( A 0 )
Revised: Moreover, we have that ( A 0 )

Original: i k k
Revised: i k k

Original: ( B 0 )
Revised: ( B 0 )

Original: ji j = σ i j .
Revised: ji j = σ i j .

Original: Similarly, [ A 0 B 0 − Σ] il is non-zero if and only i = l / ∈ I .
Revised: Similarly, [ A 0 B 0 − Σ] il is non-zero if and only i = l / ∈ I .

Original: For i / ∈
Revised: For i / ∈

Original: I we have that [ A 0 B 0 − Σ] ii = − σ i .
Revised: I we have that [ A 0 B 0 − Σ] ii = − σ i .

Original: Now we ﬁx i and j and consider the ij -th row of M .
Revised: Now we ﬁx i and j and consider the ij -th row of M .

Original: We apply our observations above to the following three cases.
Revised: We apply our observations above to the following three cases.

Original: If i = i j , then M ij,kl is non-zero if and only if k = j and l = i .
Revised: If i = i j , then M ij,kl is non-zero if and only if k = j and l = i .

Original: In that case, M ij,ji = 2 σ i .
Revised: In that case, M ij,ji = 2 σ i .

Original: If i ∈ I , but i (cid:54)
Revised: If i ∈ I , but i (cid:54)

Original: =
Revised: =

Original: i j , then there is some n (cid:54) =
Revised: i j , then there is some n (cid:54) =

Original: j such that i = i n .
Revised: j such that i = i n .

Original: Now M ij,kl is non-zero if and only if k = n and l = i j .
Revised: Now M ij,kl is non-zero if and only if k = n and l = i j .

Original: In that case, M ij,ki j = 2 σ i j .
Revised: In that case, M ij,ki j = 2 σ i j .

Original: Finally, if i / ∈ I , then M ij,kl is non-zero if and only if k = j and l = i .
Revised: Finally, if i / ∈ I , then M ij,kl is non-zero if and only if k = j and l = i .

Original: In that case, we have that
Revised: In that case, we have that

Original: M ij,ji = − 2 σ i .
Revised: M ij,ji = − 2 σ i .

Original: Corollary 32.
Revised: Corollary 32.

Original: The square matrix ∆ := MM T ∈ R rd
Revised: The square matrix ∆ := MM T ∈ R rd

Original: y ×
Revised: y ×

Original: rd y is a diagonal matrix.
Revised: rd y is a diagonal matrix.

Original: For i ∈ [ d y ] and j ∈ [ r ] , its ij -th diagonal entry is ∆ ij,ij = 4 σ 2 i j if i ∈ I and ∆ ij,ij = 4 σ 2 i if i / ∈ I .
Revised: For i ∈ [ d y ] and j ∈ [ r ] , its ij -th diagonal entry is ∆ ij,ij = 4 σ 2 i j if i ∈ I and ∆ ij,ij = 4 σ 2 i if i / ∈ I .

Original: Proof.
Revised: Proof.